[Architecture] Hub-Spoke model for federation? · Issue #3245 · LemmyNet/lemmy
github.com
Question One of the biggest issues I am currently seeing with Lemmy, is federation. Either... Federation between instances having issues. Federation is backed up. Federation / Syncing is not scalin...

https://github.com/LemmyNet/lemmy/issues/3245

I posted far more details on the issue then I am putting here-

But, just to bring some math in- with the current full-mesh federation model, assuming 10,000 instances-

That will require nearly 50 million connections.

Each comment. Each vote. Each post, will have to be sent 50 million seperate times.

In the purposed hub-spoke model, We can reduce that by over 99%, so that each post/vote/comment/etc, only has to be sent 10,000 times (plus n*(n-1)/2 times, where n = number of hub servers).

The current full mesh architecture will not scale. I predict, exponential growth will continue to occur.

Let’s work on a solution to this problem together.

  •  Fauxreigner   ( @Fauxreigner@beehaw.org ) English
    1•1 year ago

    I’m just not seeing a benefit here, I think this is a solution to the wrong problem. Your proposal in theory cuts outbound updates from the big hubs, but in reality they’re only updating a subset of other instances for any given update, and it doesn’t do anything to help with inbound updates. And to do that, you have to solve a pretty tricky problem.

    If my instance gets an update from Beehaw, I can validate that they’re allowed to do so, because Beehaw has a TLS certificate that says “Yep, this is actually Beehaw.” If you introduce a hub system, I need some way to determine that the hub system that’s telling me “Beehaw has an update for you” is allowed to send updates on behalf of Beehaw.