Selfhosted @lemmy.world gabriele97 @lemmy.g97.top 1y ago

Spawned my own instance of lemmy: now I've got a lot of questions about federation

cross-posted from: https://lemmy.g97.top/post/761

cross-posted from: https://lemmy.g97.top/post/723

Hi! I spawned my own instance of lemmy on my server and I discovered new things about how lemmy and federation works, and I have a lot of doubt. I don't know exactly if those doubts are problems of my implementation of if they are normal, so!

My main account is on lemmy.world and I see that new posts from communities I follow show up before on lemmy.world and then on my instance. Is it normal?

With comments happens the same thing and they are slower to "sync". Why?

If a community has been never discovered from the search form with the full format !community@instance, it will never appear on my instance. This means that is not possible to search for an argument (i.e. steam deck) and finding all the posts and communities about it. Is this normal or a feature that we/you would like to see in future/is adaptable to the concept of the fediverse? Because if I am on a big instance with a lot of users maybe I found that specific community or post, but on smaller instances like mine it will never appear If I don't know the exact name.

I created a community on my instance and subscribed it from lemmy.world but I don't see any post nor are they in sync. Why? https://lemmy.world/c/[email protected] vs https://lemmy.g97.top/c/announcements.

From my instance I am unable to follow lemmy.ml communities (they are pending, usually on lemmy.world the pending status is faster)

I am unable to search for communities on Kbin.social, and when I try I see this log message of type "couldnt_find_object: error decoding response body: missing field properties at line 1 column 206" from my docker instance:

2023-06-20T22:02:16.056226139Z 2023-06-20T22:02:16.055937Z ERROR HTTP request{http.method=GET http.scheme="https" http.host=lemmy.g97.top http.target=/api/v3/ws otel.kind="server" request_id=8211e6a4-2b30-4f8c-98b3-d93843a0e293 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: error decoding response body: missing fieldpropertiesat line 1 column 206 2023-06-20T22:02:16.056276976Z 0: lemmy_apub::fetcher::search::search_query_to_object_id 2023-06-20T22:02:16.056286500Z at crates/apub/src/fetcher/search.rs:17 2023-06-20T22:02:16.056293804Z 1: lemmy_apub::api::resolve_object::perform 2023-06-20T22:02:16.056300316Z with self=ResolveObject { q: "[email protected]", auth: Some(Sensitive) } 2023-06-20T22:02:16.056307712Z at crates/apub/src/api/resolve_object.rs:21 2023-06-20T22:02:16.056314152Z 2: lemmy_server::root_span_builder::HTTP request 2023-06-20T22:02:16.056320693Z with http.method=GET http.scheme="https" http.host=lemmy.g97.top http.target=/api/v3/ws otel.kind="server" request_id=8211e6a4-2b30-4f8c-98b3-d93843a0e293 http.status_code=101 otel.status_code="OK" 2023-06-20T22:02:16.056351870Z at src/root_span_builder.rs:16

I have a lot of warnings in the lemmy log of type "Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Header is expired" such as:

2023-06-20T21:58:12.484449111Z 2023-06-20T21:58:12.484275Z WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Header is expired 2023-06-20T21:58:12.484510012Z 0: lemmy_server::root_span_builder::HTTP request 2023-06-20T21:58:12.484517559Z with http.method=POST http.scheme="https" http.host=lemmy.g97.top http.target=/inbox otel.kind="server" request_id=caf194c5-cac3-4c37-a29c-577d65deb050 http.status_code=400 otel.status_code="OK" 2023-06-20T21:58:12.484525578Z at src/root_span_builder.rs:16 2023-06-20T21:58:12.484530286Z LemmyError { message: None, inner: Header is expired, context: "SpanTrace" }

I have more questions/doubt but for now this is enough I think! Thank you!

You're viewing a single thread.

45 comments

I can't even see this post on my selfhosted instance, so that's fun.

With regard to #6, you apparently have to search for the URL of the Kbin magazine (e.g. https://kbin.social/m/RedditMigration) and then you can subscribe as normal from that point on.

And yeah, the discovery and comments thing is super annoying. There will be completely different conversations based on where you're viewing the post, which kind of kills the idea of all these instances federating in the first place.

That being said, it appears to be feeds coming from lemmy.world and lemmy.ml that have the majority of the problems with comment/post sync. Unfortunately that's the selfhosted community.
- Until 5 minutes ago I was seeing 6 comments. Now Lemmy says there are 11 comments (as I was saying on lemmy.world until minutes ago) but I only count 6 comments by hand 😂
  
  Yep the problem is that we all see different things so we are not having a consistent conversation between us.
  
  I still don't know and I would like to, how the activityPub and federation protocol works and where the fault is (in the activityPub or in the Lemmy's implementation). As I understood everything that happens on Lemmy.ml or lemmy.world is sent to all the connected instances with HTTP. This means that if there are 10k connected instances, a like produces 10k HTTP request and this is multiplied for each actions done on these servers. Is this right?
  
  It sounds like you know more about ActivityPub than I do, but that seems right. My guess is that .ml and .world are so overloaded that there's just a massive delay coming from them. It is kind of funny that, like you mentioned above, with my Lemmy instance I have almost perfect sync with Kbin servers but problems with Lemmy.
  
  I'm hopeful that this is like Mastodon, where things calm down in a few weeks and there's a smaller but more sustainable userbase.
  
  You’re close!
  
  Instance must send outbound notifications to every federated server where at least 1 user subscribed to the community the activity happened on.
  
  Take /c/selfhosted on lemmy.world for example. If of the 2K+ servers only 200 had people subscribed to it, then lemmy.world server will only need to send 200 messages for an upvote. However, given the nature of this community, it’s probably closer to 2000.
  
  To exacerbate the problem, if people take their server offline, because of TCP protocol being under pinning of HTTPS, lemmy.world server must wait until the socket completes or times out.
  
  To circumvent this, lemmy.world has bumped the federation worker count to 10K… but if every action on this community creates 2K messages, it could get filled up pretty quickly. And that’s just this community, without regard to other communities.
  
  There is an increase coming per the link I’ve shared with you in another reply, I’m hoping it will alleviate some pain, but I’d still like to see something more scalable to enable support for network that’s much larger.
  
  How does kbin implements ActivityPUB? Isn't it better from this point of view? Because I see that kbin's actions are synchronized better and faster with my instance
  
  I don't know how kbin implements it, as I've not seen / reviewed their code. There are several ways around it; perhaps a longer expiry, perhaps signing each message as it is going out, perhaps a better out bound queuing system. It is important for Lemmy dev team to think these approaches, and implement solutions that makes sense for Lemmy, and enable better scaling.
- The suggestioni for kbin works flawless, thanks! I don't get why Lemmy and kbin instances have a different way to do this.
  
  Anyway, seems like my instance is more in sync (by order of magnitude, I still have to verify it better) with kbin then with Lemmy.world or .ml.

45 comments