Lemmy.world down, probably following the upgrade. A reminder to move to smaller instances for a better experience

A reminder to move to smaller instances for a better experience

A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.

Github Issue 2910 is the kind of PostgreSQL problems that the developers ignored for months and people still defend the developer choices to have the code doing real-time counting of every single comment and post for numbers nobody needs to needs done in real-time.

PostgreSQL is voodoo to this project, they do everything they can to avoid going to [email protected] community and asking for help, learning 101 about how to fix their SQL TRIGGER logic like Github Issue 2910 spelled out June 4.

A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.

I am always advocating for any of the top 25 instances that are not Lemmy.world or Lemmy.ml

For the rest of your post, I don't know what that has to do with people aggreating on LW.
- For the rest of your post, I don’t know what that has to do with people aggreating on LW.
  
  aggregation refers to the lemmy database tables, site_ aggregates, community, person. The SQL TRIGGER logic lemmy_server uses that has been the source of so many crashes the past 60+ days.
A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.

And this will always.. always be the biggest problem in the FOSS community.

"I dont like X, so I'm going to leave and make my own version of X"

So userbases get spread thin, manpower gets spread thin, developers get spread thin, and the user experiences degrades for everyone until it pushes them back to the bullshit websites and products.
- This is exactly what federation is meant to solve: power in numbers without the centralization. Is that so hard to understand?
- For the rest of your post, I don’t know what that has to do with people aggreating on LW.
  
  And, factually, the project leaders telling everyone to create 1000 new instances and shutting down sign-up on Lemmy.ml caused more performance problems.
  
  They had a bug in their PostgreSQL TRIGGER logic where 1500 instances were updating + 1 comment and +1 post counting instead of WHERE site_id = 1, a single database row. So each new Lemmy server that went online made the table larger and crashes more frequent on lemmy.ml
  
  The amount of disk writing by lemmy was ignored
- Your answer didn't justify lemmy.world being treated the same as Lemmy as a whole. It's just a bunch of people who don't understand federation.
- The developers of Lemmy seem to make every effort they can to avoid using Lemmy itself to discuss their [email protected] learning 101. They have made massive mistakes in SQL TRIGGER logic that they avoided to such a degree that their social motives are in question now. Github Issue 2910 was opened June 4, almost a month before the Reddit API deadline, and they ignored it. Just like they hang out on Matrix Chat and don't use Lemmy their own self to discuss code.
  
  They have cultivated a kind of voodoo attitude towards PostgreSQL where people using Lemmy won't actually scrutinize the Rust code or PostgreSQL tuning parameters.
I know you are salty about how you are getting treated over at GitHub, but you should look at it objectively, Blaze is clearly advocating that people join top instances that's not lemmy.world or lemmy.ml, not nobody instances that only have 1-2 users. They certainly aren't going offline as quickly as they come online.
- I know you are salty about how you are getting treated over at GitHub
  
  No, it isn't about my personal treatment. It's about the cultist attitude you have towards Lemmy and the leaders without any ability to see what they are doing behind the scenes with the code. I know cults and religious faith is how many people enjoy the world.
  
  A 2-line SQL TRIGGER removal takes about minutes to fix. It was crashing the entire site constantly. They sat by and asked for donations of money.

You can also just have multiple accounts. I have one on lemm.ee just for this reason.

Definitely!
Is there a readily available tool that can export/import community subs? I have two accounts on different instances but would like them to have to same subscriptions
- I've not used it myself but heard good things about https://github.com/CMahaff/lasim
I learnt my lesson to have multiple accounts after vlemmy
- That was my main. The instance vanished faster than cotton candy being held by a raccoon standing in a pond.
Yeah, I have .world where I first signed up. It occasionally had some issues so I added another that I just mostly read on if the other isn't working

It'd be nice to export/import subscriptions but I'm sure eventually the servers will get past it
- You can use that script https://github.com/CMahaff/lasim

when I start writing this comment, the post is 47 minutes old. if I understand the linked page properly, lemmy.world has been functional (all green checkmarks) for the past 10 minutes which is the furthest back the data goes. All the other instances are all green except for lemmy.one which is all red. I am assuming that 47 minutes ago, lemmy.world had red boxes?

Maybe a different link would have explained the point better but I don't really see how a 30 minute (??) server outage during an upgrade is compelling to avoid a large instance. Are you suggesting it's better to use a server whos admins don't upgrade? If not, is there really any size of server that would meaningfully avoid this kind of occasional disruption? Seems to me that the dynamism of the environment will inevitably lead to various problems. That's part of the experience. TBH threadiverse uptime on the whole is pretty impressive for such a ragtag groups of admins and devs.

I have accounts on some smaller servers but they have their drawbacks too. Using a bigger server is more convenient because the people and content is already there. It's easier. I didn't plan to use lemmy.world but I ended up making account there to use sometimes.

I think in a year or so the situation might be different. I see the ideological point and I would like it to be true. Maybe the technology will catch up. I think it would be nice to be able to programmatically seed content, but maybe that would be obnoxious to admins.

I didn't track the timing too closely, but in the last few weeks there have been quite long disruption of service due to DDoS attacks on the largest instances.

I am personally quite tolerant towards Lemmy as a platform in its very infancy, but some other users might want to quit it due to this kind of annoyances, hence my comment about moving to a smaller instance.

the people and content is already there. It’s easier

What do you mean? You can access all of the content in the Threadiverse from whatever instance, modulo defederation, but Lemmy.world defederates quite a few instances too, so that's valid for both big and small instances. If you are talking about the "All" feed, which will indeed be empty if you are in a 10 people instances (communities need to be subscribed by an instance member for the instance to get the community content), then it's a valid issue, and that's why I suggest people to move to one of the 24 biggest instances that are not LW or Lemmy.ml

Lemm.ee, sh.itjust.works, sopuli.xyz, reddthat.com, lemmy.one, your country instance (if it's big enough). I'm on sopuli since a while now, and I'm very happy with the experience. 680 monthly active users, so the All feed is pretty much identical to the one on Lemmy.world, except the vey niche community I either don't care about or would already know by myself.

The biggest issue I see with having everyone on LW is that at some point the costs will be too high for the admins. It's quite a big risk, and that why I'm advocating to use smaller instances.

The DB migration at the end of this upgrade is significant, I was surprised how long it took when I upgraded my instance. Lots of room for things to go wrong considering the size of their DB.

That makes sense, especially how large the instance is. Just curious, how long did the migration take for you? Half an hour?
- 10-15 minutes here on a 32gb database.

Just make a second account, the one I run, lemmy.myserv.one is so underutilized its a joke. Smaller instances like mine basically have to beg for users and the server goes unused while bigger instances struggle under the constant traffic.

As always, I have to ask: is there a second admin, what would happen to the instance if something happened to you tomorrow (which I really wish will not!)

The Vlemmy.net disaster is still fresh in people's mind
- Ultimately I am the one paying the bill currently so if I die nobody elses credit card is being charged.
  
  In terms of other admins, this is actually happening. Some smaller instances like mine are in the process of setting up a sharing admin work between instances so that if someone is on holiday, the instance still has an admin who can login. This was only just started and is in the process. We have to create a lot of documentation and basic stuff to get it fully functional where another admin can login and fix something. Its not at that stage yet and will be a couple more weeks before it is. We did a test last night where another instance admin (boulder.ly) could connect to my instance via ssh but without documentation on what to do and check anything more than the basics of rebooting or restarting something isnt going to happen. Eventually we will get it to what to do if site has a critical vulnerability or is being attacked but not ready yet. Its a work in progress unfortunately.
  
  P
- I think without some kind of "incorporation" (or whatever the tech/FOSS equivalent of that is), most of these kinds of thing will be vulnerable to issues with the owner's payment methods failing. Even with donation options available it's almost always still being used to pay to the server owner in parallel to them paying server / domain costs out of pocket (and then reimbursing themselves with the donations)
  
  That said, I have to assume there's some way to set up some kind of automated payment option where community donations actually fill a fund that is used to pay costs directly in case the maintainer drops off the face of the earth.
Because using random tiny servers is worse in other ways. With all due respect, nobody knows you and they don't know how committed you are or how much time you have. When your server gets DDoSed or hits a bug causing data loss, what will you do? Do you have the technological know-how to recover and quickly? If your server suddenly grew and it became more expensive to run, how does anyone know if you will keep paying the bills? If Lemmy has a bad zero day, will you upgrade quickly?

There's no need to answer these questions. I'm not actually asking you personally. But these are the kinds of questions that users have to worry about from random, small, unproven instances.

(Also, Lemmy does not favour small instances because the "all" feed, searching, and going to new communities are all better the more diverse users you have.)
- Yes obviously the barrier to entry is high. But nobody knows for the big servers either since they are basically just small instances that happened to get big. Thats why lemmy.fmhy.ml just died one day due to domain seizure. End of the day all you can do is look at how long a server has been around and if it has be online a reasonable amount of time. That kind of reputation just increases slowly and nobody can make it happen faster.

Of course I have one or two other accounts, but I personally like Lemmy.world. They serve as a necessary stress test that shows the devs and admins how to optimize further, and I just like learning admin practices at this scale of a userbase from a work perspective. Plus I don't want to be on an instance so small they can't or don't know how to handle compliance stuff and evaporate if something like that comes up. Not saying I know how to handle all of those situations, that's the job of someone else at work.

I think that's one of the issues that the rest of the instances are facing to appear as trustful as LW. LW admins have a long established reputation and experience managing Fediverse services, and provide very good transparency and a large team.

Other instances are usually nowhere close to that (some will be in the future I hope). The question I usually raise when someone start promoting their instance is "how many admins do you have?´What happens if you run under a bus tomorrow (hopefully you'll stay safe of course)? Is there a back up plan in place?"
- Yeah, longevity and name recognition are why I went with sdf.org. They've been running many-user services for decades, even if the Lemmy service is pretty new.
  
  ETA: they've been around since BBSes. I'm on a wicked nerdy old-school geek instance, and I love my local communities.

Quoting myself from a previous post:

First of all, it's really fine to stay on LW for now, no need to rush anything. But if at some point you have some time for this, then read the following.

So, to pick your instance, you can have a look at https://lemmy.fediverse.observer/list, filter by "1m" to see what are the most popular ones. As you can see, with a 27433 monthly users, Lemmy.world is by far the most popular, which is why you might experience some issues from time to time.

You should have a look at the next instances on the list. Short story: lemm.ee, sh.itjust.works, lemmy.one, sopuli.xyz and reddthat.com are solid choices.

You are looking at instances with quite a lot of people (the more people help with filling your "All" feed), just not the most populous one (lemmy.world), the original one (lemmy.ml), and instances that are too specific, either due to country or specific focus.

Long story:

spoiler

lemmy.ml is the original insance, also quite crowded, not really the best choice
lemm.ee can be nice, you can have a look at it and see how fast it is for you. The admin communicates a lot and is very helpful.
sh.itjust.works had some rough time in the last few days. You might also not like the name, that's okay.
beehaw.org does not federate with the big instances, so if you go there, you will be in their own space. It can a valid choice, but please have a look at their guidelines first, they tend to moderate a lot. Can work for you, or not.
feddit.de, lemmy.ca, discuss.tchncs.de, feddit.uk, aussie.zone are country specific instances, so probably not interesting to you if you are not from there
lemmynsfw is a NSFW instance, probably not the one you want to move to
programming.dev is an instance focused on programming
lemmy.blahaj.zone is a pro queer instance

.

To migrate your settings (including subscriptions and blocked instances), you can use that script: https://github.com/CMahaff/lasim

I think there is a huge misconception many people have that a larger instance is more likely to stay around, but due to the nonlinear costs involved in hosting fediverse instances this is not true.

Basically there is a sweet-spot around a few thousand (~2500) members where costs are low enough for a single admin paying things out of their own pocket long term is possible, but also enough members willing to occasionally donate or contribute otherwise to cover costs.
- To be honest, I know it's a controversial view, but I would almost like to see Ruud and the LW admins block registrations for a while, along with a communication "Have a look at those other instances, they are well managed, you can access all of Lemmy just as well from there"
LASIM author here, ironically on my own alt: Just an FYI that support for Lemmy 0.18.3 is not yet out, but keep an eye out for it soon (I have it working on a branch but I need to test it more before release).

This is the first breaking API change since it's creation, so here are the limitations:

Old version (0.1.2) will only support API 0.18.1 and 0.18.2

New version (0.2.0) will only support 0.18.3 (and above until there are more breaking API changes)

Profiles downloaded with 0.1.2 (and below) will automatically be converted to work with 0.2.0.

So that all means:

You can use the old LASIM to migrate between 0.18.2 Lemmy instances

You can use the new LASIM to migrate between 0.18.3 Lemmy instances

You can use the old LASIM to download from an 0.18.2 instance then use the new LASIM to upload to a 0.18.3 instance

You cannot use the new LASIN to download from a 0.18.3 instance and then the old LASIM to upload to a 0.18.2 instance (unless you are comfortable doing some manual work editing the JSON file so "old LASIM" understands it).

This will be true of every release with breaking API changes.

EDIT: PR is out. Once it builds, I'll publish a new release! https://github.com/CMahaff/lasim/pull/21

EDIT 2: Release is published! https://github.com/CMahaff/lasim/releases/tag/v0.2.0
- Thank you for the heads up! You might want to add that to the ReadMe on your Github
- Thanks for all your hard work on this. I’ve used lasim to transfer my subscriptions to my other accounts on other instances when the lemmy.world downtimes happened last week. I love the portability of the tool.
Fwiw, sh.itjust.works is a horrible name, but a great instance lol.

What's funny though, I'm getting beehaw posts in my all feed since yesterday. No idea if they've refederated with us, or if it's an artifact of connect (my app of choice), or what,
- Interesting. You are still on the list of their blocked instances, so I'm not sure: https://beehaw.org/instances

Lemm.ee is amazing with a top admin

💯 checking out the admin of a Fediverse server you want to join is a good idea. Do they communicate with their users? Do they seem responsible ? Do they have any previous experience as a network admin? Lemm.ee was an obvious yes on all counts.

The admin for vlemmy.net was the complete opposite and look how that turned out.

working fine for me , actually I have to say lemmy.world has been very stable

Perhaps I open Jerboa too often but in the last week lemmy.world has been down around 4 times for me, sometimes for a short time sometimes for much longer. When I was on lemmy.fmhy.ml it was only down once for me until their domain was seized and that was for the server upgrade.
- You are not the only one, the last few weeks have been rough for LW. That's why I suggest people to moving to other instances, I detailed the process in another comment in this thead (can't get the comment link to work for some reason)

Has anyone any idea what's going on with lemmy.one? It's been down for quite a while now

Probably some hiccup in upgrading to 18.3

As I said elsewhere

There are certain things that are memory intensive and CPU intensive. If you have 10k on one server doing that it really adds up. However having them across a wide range of smaller servers, its not such a big deal.

As a user, you literally lose out on nothing not being on lemmy.world. You can partake in all the same conversations, communities and everything. In fact when lemmy.world is down, you can still see everything and when it comes back up, your posts will synchronize. There's genuinely no upside to being on lemmy.world. That's the way the system was designed.

Not sure people will listen though. I will always talk up the amazing admin I have on lemmy.tf, but it's also worth mentioning that I have a bunch of communities hosted on other instances and each and every one of them is amazing.

Definitely.

Just waiting patiently to get approved on midwest.social

Hopefully they'll get back to you soon!

The instance lemmy.kya.moe that I created my alt on performs well too if anyone else is looking for an alternative :D

I moved to self hosting mine instance :)

Well done!

We're back!

I was on lemmy.world and had a really bad experience. The I moved to aussie.zone and its better than reddit. Never down, commenting, posts everything feels so fast and snappy.

https://1up2down.social/lemmy

Getting server errors?
- Are you saying you’re getting server errors accessing https://1up2down.social/lemmy ?
Server errors here too.
- Thanks for letting me know. The issue was related to the incorrect configuration of pctrs. It has now been resolved and the Server Error should have subsided.
Seems like a nice instance, thanks for sharing! You might want to create a local community to discuss your instance management, who you are, etc.
- I thought I’d seize the opportunity for a little self promotion, haha. You know what that’s a good shout. I might just do that. Thank you :)

Please come to lemmy.today if you like a small fast instance with no downvotes.

Also federates with all other instances.

I don't understand the appeal of no downvotes. Do you really think it's a good thing that trolls, bigots, dangerously wrong answers, general assholes, spam, etc can't be downvoted? I won't pretend downvotes aren't misused sometimes, but their existence is critical for quality control.

Edit: wait, I just saw you post in another thread as an "enlightened centrist", so I guess that explains it.
- Not to mention the other instances have it enabled. So anytime federated with them can downvote them, but they can't downvoted each other.
- I saw it being misused on reddit a lot so I like it. It was common for people to use it as a "i disagree" button, which created a big echo chamber. I read that many people felt that there was no point even posting to a discussion since they knew they would be downvoted.
  
  I think reddit really turned to shit, and I was hoping Lemmy could take another direction here. You can still upvote to support your opinions riding to the top without needing to downvote someone.
  
  I understand this is a hot topic for many though. I guess it depends on what your previous experiences at reddit has been.
  
  And also you mention the word quality control. I'm not sure the majority is some kind of a quality control. I rather hear people's opinions and make up my own mind. It would feel weird to have other people push down comments I may want to read.
@Blaze @1984 Federating with ALL other instances isn't a good prospect either...
Thanks for sharing this. Do you have a second admin for your instance?
- It's not my instance, I just use it and like it. The instance lists only one admin. Is it common with several?

Lemmy's machine-generated ORM SQL and hand-made flawed PostgreSQL TRIGGER logic is so bad, bloated. The developers on GitHub brag about "high performance". It's unbeliable.

In reality, small instances work because it has so many SQL performance problems that it mostly only is stable with little posts and comments in the database. They dd everything they could to avoid using Lemmy itself to discuss [email protected] topics and hang out on Matrix Chat to avoid using the constantly-crashing servers they created.

If you go to a server with no users creating comments and posts and only has a tiny amount of data, it does crash a lot less.

Oh geez, why would the Lemmy developers want to do any kind of discussions with you over at the [email protected] community, which you moderate?
- I see you think that their behavior has only been since June 2, 2023 when I created my account.
  
  The problems with them avoiding Lemmy, "eating their own dog food", to discuss [email protected] have noting to do with me. They hang out on Matrix Chat and do not ask for Rust or PostgreSQL help to their constantly crashing code.
  
  Your style of arguing is to say I wear glasses and have "4 eyes", childish. You obviously can't go see they created a new Rust front-end on Github all on your own .... and that it was not me personally who created GitHub issue 2910 on June 4, 2023 - almost a month before the Reddit API change.,
  
  Is it the pro-China stance that you like about the developers?
That's an interesting community.
You're rocketderp based on your instability I'm guessing? Looks like your the only one that is nonsensical in that thread. Your PR isn't a PR. You were rushing to someone that was being helpful but you still acted like a child with your advise of GitHub bugs and PRs and not using them properly. The other person in agreement with you at least is calm and rational. I'm guessing you never worked on a group project before or have had anyone disagree with you. Your commits broke the pipeline. Someone even tried calling you down, but you wouldn't listen.

You aren't the victim here. You're the bully.
- You’re rocketderp based on your instability I’m guessing?
  
  Yes. My name is Stephen Alfred Gutknecht. It is clearly detailed here on my profile that I am "RocketDerp" on GitHub.
  
  Looks like your the only one that is nonsensical in that thread.
  
  Issue 2910? The word "nonsense" is being discussed about a Lemmy TRIGGER on June 4, 2023.
  
  I’m guessing you never worked on a group project before or have had anyone disagree with you.
  
  You would be guessing wrong. Since late 2019 I am in constant mental anguish and barely surviving. My communications is a constant struggle and I openly disclose this so that it is understood, I am not attempting to cover it up.
  
  I’m guessing you never worked on a group project before
  
  The "i'm guessing" game. And I note you can't actually cite the Github issues by number (2910) or keep details here in fact. A lot of "guessing" going on.
  
  based on your instability I’m guessing?
  
  "guessing" again. Should I guess you do not know how difficult or easy it is to remove a TRIGGER in PostgreSQL that is causing server crashes?
  
  You aren’t the victim here.
  
  Why do you think this is about "me"? I haven't donated money to the project or had blind faith in the developers.
  
  You’re the bully.
  
  I'm beyond frustrated that they seem to be going so far out of their way to have servers crash that their motives for doing so need explanation. "Social hazing" is the best answer I can come up with. I've now outright asked why they are dong it, because the pattern of behavior with trying to let server crashes has been going on now for months.
  
  I have to admit that it's amazing the number of people they have attracted with the server crashing. It's been a social experience in some ways like Elon Musk is doing with Twitter. Perhaps it bothers me when other people actually seem to enjoy it. That seems to be the hiveMind reaction here. In that sense, if it is "social hazing", it has worked very well.
You're spamming this all over this thread. Why don't you go create a PR instead? If you think you have a better solution then go discuss it with the people who have the full context and try to get it fixed instead of complaining here.
- Why don’t you go create a PR instead?
  
  It's amazing how you have fallen hook line and sinker into believing that the problem is difficult to solve. It's the agenda that is the problem.
  
  They have people like you who will not read actual code to see that they only care about the fact that "Rust is cool programming language" and crashing code doesn't get any priority.
  
  They even started a new front-end Rust application this month, because they don't care to bother with the core of the site, PostgreSQL doing INSERT and SELECT statements to load comments.