When I configured my single-user Lemmy instance, I marked the server as "private" in the settings. I thought that was a good idea, but it actually prevents my avatar from being accessible from the Internet. I'm not entirely sure what else it impacted.
Once I made my server "public" in my server setting, I was able to pull my avatar from my Mastodon instance.