Lemmy.world status update 2023-07-04
Lemmy.world status update 2023-07-04
Status update July 4th
Just wanted to let you know where we are with Lemmy.world.
Issues
As you might have noticed, things still won't work as desired.. we see several issues:
Performance
- Loading is mostly OK, but sometimes things take forever
- We (and you) see many 502 errors, resulting in empty pages etc.
- System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Bugs
- Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
- 2FA seems to be a problem for many people. It doesn't always work as expected.
Troubleshooting
We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.
So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!
I'm just excited to be back in the Wild West again -- all of the big players had bumps, at least this one is working to fix them.
And giving updates!
I’d rather have to deal with hiccups and bumps along the way, because the community only grows more each time.
I still remember early reddit days of 502 it went through, 504 try once more.
Early? Haha, reddit still shits the bed regularly
Yee-haw, mother fuckers.
That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Lemmy has a memory leak? Or, should I say, a "lemmory leak"?
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A lemmory meek, obviously!
On some of the latest release candidates I think so.
Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn't happen without you!
I want this to succeed so badly. I truly feel like it’s going to be sink or swim and will reflect how all enshitification efforts will play out.
Band together now and people see there’s a chance. Fail and we are doomed to corporate greed in every facet of our lives.
Wow. Drama! Chill bruh. They'll get there. If it takes too long for some, they can switch to other instances.
That’s just a lemming living up to their name!
I'm here for the drama too!
Fuck the cleanliness of the corporate web.
Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.
Seriously, you're doing a great job <3
Great distraction from the collapse
i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.
but please, don't feel forced to overwork yourselves. i understand you want to do it soon so more people can move from Reddit, but i wouldn't like that Lemmy software and community developers overwork and feel miserable, as those things are some of the very motives you escaped from Reddit in first place.
in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.
this applies to all lemmy instances and communities, ofc. have a nice day you all! ^^
Plus, slow steady growth means eventual success. Burnout is very real if you never take a break.
so true, pal! slowly, with patience, no rushing, putting love into it, organizing ourselves, working smart is better than working hard and fast.
because of the federated nature of fediverses like Lemmy, it is very possible that many people are doing the very same task without even knowing they are duping each other's efforts.
and that's sad because if they knew, they could be teaming up, or splitting the task in two, in order to avoid wasting different efforts into dupe results.
i have learnt a thing or two about burnout, it's better for me to make 40% planning, and 40 % self-care and so the 20 % of execution becomes piece of cake.
but this is just my opinion. anyway, please take care, pals <3
As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You're doing great with such fast exponential growth
💙 Thanks for your hard work!
Amen. So happy to be done with Reddit.
As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I'm remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.
I remember when Voat came out and the slight exedous that brought. I made an account and everything but it never properly took off. I checked on it two or three years later and it was just filled with alt-right/racist/transphobic garbage. Sad it never took off as a reddit alternative, reddit likely would have greatly benefited from a proper alternative, not sad it closed down after I saw what it ended up.
So far the fediverse feels really different tho, very explicitly anti that type of shit. I'm sure it will pop up, they always do, but maybe now people know how to deal with it. Block it, defederate, deplatform.
Lemmy is a distributed system that can't quite be perma-killed, kind of like ye olde usenet. Anyone can create a new instance at will.
This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.
You said it: passion projects. While being paid is surely a motivator, seeing your pet project take off the way Lemmy is can be so intoxicating and rewarding! I plan to donate as soon as I get paid on Friday! I want to see this succeed, even if it is just to spite Reddit, and I am willing to pay for the pleasure.
What was that? We're going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?
Yeah, that sounds pretty reasonable, we can work with that.
Just signed up for the 8€/month tier, hopefully my blue checkmark will arrive soon!
It’s here!
✅
(Please do note that the blue checkmark is for our 1€ “Bean” tier subscribers.)
We’re very pleased to provide you with our premium deluxe green checkmark for your generous 8€ subscription to our “Bacon” tier!
Feel free to reach out to our Lemmy Cares team @ 1-800-FUC-SPEZ for any questions or concerns that you may have.
Could I get a discord invite? I’m an ex sysadmin with a. Lot of free time
@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻
We have 128GB of RAM. It just skyrockets after a while!
@ruud Oh damn. This spontaneously sounds crazy but I’m admittedly a novice at servers on this scale.
It’s a ton better this afternoon! Thank you!!
I’ve noticed that too!
Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.
Not just that, but the code contributed to Lemmy by this debugging will make Lemmy run faster for everyone on every instance, which is makes the ecosystem that much better.
The work you're doing is greatly appreciated! It's like you invited half the internet into your house. I feel like I should've brought a cake or something
You should consider setting up a small donation to the site, if you can afford it. If you throw $5 a month at them, you're not only covering your own hosting fees, but covering for some of the folks who can't afford to as well. And maybe Ruud gets to buy some beans.
It appears I have no idea how to imbed an image, but good call. Subscribed.
Huge respect for what you've built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here
Yes he's one of the other admins in our Discord, he's very helpful!
Of course these performance issues are a bit annoying, but I gotta say that I love these updates and explanations here. Great communication, keep it up, please!
Keep up the good work!
I created an account in lemm.ee until the issues are fixed. Then I will happily go back to my lemmy.world account.
Lemm.ee is also a good choice!
If I have an account at lemmy.world, but have another instance selected, does that take some of the burden off the admins or do I need to do something more?
.world is definitely running smoother than when I joined 3 days ago, back then it was impossible to comment and the lag was immense, right now I just have to occasionally reload the page, but that's nothing in comparison.
You guys are doing an amazing work! I'm broke, so here are some
coins 🪙🪙🪙🪙beans 🫘🫘🫘🫘That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
who'd have thought memory leaks would be possible in Rust 🤯
(sorry not sorry Rust devs)
c/programmingcirclejerk when?
Ackchually, leaking memory is totally possible in Rust! But so is hogging way too much of it because you're hanging on to too many things in memory at once🤷
Cloud architect here— I’m sure someone’s probably already brought it up, but I’m curious if any cloud native services have been considered to take the place of what I’m sure are wildly expensive server machines. E.g. serve frontends from cloudfront, host the read-side API on Lambda@Edge so you can aggressively and regionally cache API responses, anything other than an SQL for the database — model it in DynamoDB for dirt cheap wicked speed, or Neptune for a graph database that’s more expensive but more featureful. Drop sync jobs for federated connections into SQS, have a lambda process that too, and it will scale as horizontally as you need to clear the queue in reasonable time.
It’s not quite as simple to develop and deploy as docker containers you can throw anywhere, but the massive scale you can achieve with that for fractions of the cost of servers or fargate with that much RAM is pretty great.
Or maybe you already tried/modeled this and discovered it’s terrible for you use case, in which case ignore me ;-)
You were so close until you mentioned trying to ditch SQL. Lemmy is 100% tied hard to it, and trying to replicate what it does without ACID and Joins is going to require a massive rewrite. More importantly - Lemmy's docs suggest a docker-compose stack, not even k8s for now, it's trying really hard not to tie into a single cloud provider and avoid having three cloud deployment scripts. Which means SQS, lambdas and cloudfront out in the short term. Quick question, are there any STOMP compliant vendors for SQS and lambda equivalent yet?
Also, the growth lemmy.world has seen has been far outside what any team could handle ime. Most products would have closed signups to handle current load and scale, well done to all involved!
cloudfront helps a lot with the client and is absolutely compatible with lemmy if you set it up correctly. possibly it could also help cache api responses, i haven't looked into that part yet.
the database, on the other hand, would need a nearly full rewrite. lemmy uses postgres and dumping it for something else would be a huge pain for the entire federated community. it could probably tear it in half.
there's also the issue of pictrs, which uses a stateful container and isn't yet able to use an external database which would allow you to scale it horizontally. resolving that one is on the roadmap though, and for the most part you can aggressively cache the pictrs get requests to alleviate the read-side load.
but whatever the solution is, it kinda needs to be as simple as developing and deploying docker containers you can throw anywhere. the vendor-agnostic setup is a very important part of the open-source setup of lemmy. it's fine to build on top of that, but currently anyone with docker-compose installed can run the service and that really should be retained.
Staying cloud agnostic is very important and CDN services like cloudflare/cloudfront have inherrent privacy issues. IMO the stack should remain hostable on anyones home server environment.
Please keep working on it, thank you for your effort.
good news, a fix might be in the works: https://github.com/LemmyNet/lemmy/pull/3482
Yes phiresky is working with us improving performance, has helped a lot so far! (We're now running a custom build with some of his improvements)
Just tried upvoting most of the comments here, and this time all of the upvotes went through and were extremely fast! No more 502 errors!
I've noticed lemmy.world has been much more responsive today so something seems to be working!
It’s working, it’s working!
Thanks for sharing. It was very interesting to see the graphs showing the before and after.
Thank you for your effort!
You're welcome! (testing comments now... )
It looks like it's working.
I am very forgiving of the bugs I encounter on Lemmy instances because Lemmy is still growing and it's essentially still in beta. I am totally unforgiving of Reddit crashing virtually every day after almost two decades.
Still a better product than Reddit Mobile lol
System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Shouldn't we be discussing closing registrations?
There's a lot of momentum to move away from reddit right now, and closing registrations would be a wet blanket. Personally, I'll take the performance issues and transparency in the process over closing registrations.
This. Don't stop the train. People need to be able to come over freely.
Does Lemmy have the ability to replace default links?
Basically, replace signup link with one that redirects to a page that gives a very simple as possible explanation what's going on, what fediverse is and gives s list of other instances to try.
Reinforce "All are viable and can browse lemmy.world subs"... Or communities or whatever term we use here for lemmy equivalent of subreddits.
Federation-wise it would be better if new users spread out. Between clueless redditors and impossible ideal, I prefer if they at least made an account and check out what Lemmy has to offer. The curious ones will eventually settle down and even redistribute into smaller instances.
The curious ones will eventually settle down and even redistribute into smaller instances
Absolutely. I migrated from lemmy.ml when that was having too many sign-ups, and I'm not opposed to migrating from lemmy.world to help with their load. I'm sure I'm one of many
The need to restart server every so often to avoid excessive ram usage bit is very interesting to me. This sounds like some issue with memory management. Not necessarily a leak, but maybe something like server keeping unnecessary references so the object cannot be dropped.
Anyway, from my experience Rust developers love debugging such kind of problems. Are Lemmy Devs aware of this issue? And do you publish server usage logs somewhere to look deeper into that?
server keeping unnecessary references so the object cannot be dropped
You indeed just described a memory leak :D
Oh they're well aware, there's a few tickets open.
For example there was some stuff about rate limiting using a lot of memory: https://github.com/LemmyNet/lemmy/pull/3111
How can I donate to lemmy.world?
Just look at the sidebar:
Donations
If you would like to make a donation to support the cost of running this platform, please do so at the mastodon. world donation URLs:https://opencollective.com/mastodonworld
https://patreon.com/mastodonworldNot an expert, but here is where I set up a recurring donation. Apparently Mastadon.world and Lemmy.world are run by the same admins. Took me a while to understand this, please correct me if I'm wrong.
By moving to another instance
People of lemmy.world should encourage the new users joining up on lemmy.world to other create accounts on other instances instead
Would donations help in the long run? How can we donate?
If you would like to make a donation to support the cost of running this platform, please do so at the mastodon. world donation URLs:
https://opencollective.com/mastodonworld https://patreon.com/mastodonworld
Thank you, done!
this is interesting indeed! could we help via donations?
Donations definitely help. They are a real motivator and allow people to make more time for contributing. For instances it is also a way to cover hosting costs.
You can donate directly to the Lemmy development or you can see if your instance accepts donations. Personally I just signed up to the Lemmy Patreon.
Donations invested into REITs would fund Lemmy indefinitely, has anyone pondered this idea?
To all the folks that are worried, don't be. Let me tell you, Mastodon was a wreck when Musk took over Twitter and that all got sorted out within a week or so. The mods and sysadmin are obviously working hard to get things up and running, but growing pains are growing pains. To paraphrase an old adage: "Facebook wasn't built in a day." In the beginning, Zuck and Co literally limited signups to only people with college email accounts and only added universities a few domains at a time... scaling is very difficult, but it's not impossible. The way things are going, Lemmy is going to thrive!
Thank you for everything! Can we donate to cover the costs? If more people throw in $5, we all will benefit. Can Lemmy server scale up without needing RAM exponentially growing with the user number? I hope the system will get better optimized for bigger user base as the time goes on..
There are two donation options listed in the sidebar, idk if one is better than the other from their pov
It has become way smoother the last few hours. The work you guys from .world and Lemmy's devs have been doing is amazing!
Thanks for giving us all these updates all the time!
Thanks team, love your work! Appreciate the updates and improvements but happy to have this place despite the bugs and quirks :)
I'm very much so okay putting up with hiccups right now. I know things will improve in the future.
Thank you and everyone for all your hard work!
Thanks u/ruud!
How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.
Same happened with mastodon.world in November. Family goes first, then work, and then all of my hobbies, of which this is one. (But the one taking up most time at the moment...)
Small remark instead of /u/.... you should use the at-sign
@to mention somebody. For example: Hey @[email protected].
The good thing about the fediverse is we can sign up on federated servers and still interact with everything on lemmy.world without the errors, then come back to our lemmy.world accounts when everything is smoother
Yeah, when I understood this I joined another small local instance, it does improve performance by a lot!
I feel that lemmy runs smoothly the past few hours with very few hiccups and mostly on the upvoting and commenting side. I encountered no issues yet on the loading of the post.
Thank you so much for all your hard work.
Agreed, it seems fine now, makes such a difference to whether I want to keep interacting with Lemmy.
I love the community here and am excited to see it grow. Thanks for giving a report, it awesome how invested y'all are and makes me want to stick with this instance even through the hardships. Keep up the good stuff!
I can't imagine how hard it is to rua server that has influx of large users. I Thank you for your hard work to run and maintain this instance. Hope it works out well and future will be a smooth sail.
Its amazing this site hasn't completely melted down yes it sucks trying to use it and having nothing load but the fact that the site has mostly worked when I want to use it and the fact you guys have less than 1% of 1% of reddits resources is mind blowing
Thank you for your hard work! ♥️
The redirects to baseball games glitch is gone. I am unsure if this is because the glitch is fixed or if baseball season is over.
😆 sorry about that! Baseball season is still very much on.
This issue was caused by the websockets replacing the current thread you were on with newly updated or created posts. Our game threads get updated frequently so they were constantly popping up in people's faces. We decreased the frequency of the updates pretty substantially in order to be less annoying to unrelated instances but it was still happening occasionally.
Now that we're all through with websockets it won't be an issue anymore!
I am sure the stress is unreal, but It is appreciated. You all stepped up when everyone was stepping down (or trying to sellout). I am hopeful this will turn into something great.
Thank you for all of your effort on this. Adjusting to rapid growth is a huge challenge, and we all appreciate the effort you and everyone else is putting in to trying to make this instance run smoothly.
Can anyone give me a link to donate to help out? Ever since discovering lemmy I hardly use reddit and would like to contribute to this community more!
Yep! Donating to both their Open Collective page for Mastodon.World (didn't grab that link by accident), as well as their Patreon page for Mastodon.World works!
They have a note up on Patreon that donations there also contribute to World.Lemmy, as well as another Lemmy instance!
Here's the link to their Patreon; https://patreon.com/mastodonworld
Thanks for the transparent update!
Thanks for the update, you're leading the way for Lemmy.
I was prepared for servers crashing and lagging, but you're keeping things moving thus far, despite receiving 85k new users in a month. Heroic stuff.
Thanks for the transparency and keep up the good work!
Thanks for the transparency, and communication. I think it's always better when the userbase is able to understand what is going on rather than being left in the dark wondering what is going wrong. Keep up the good work, but also be sure you guys get enough rest, and take care of yourselves too.
Y’all doing amazing things keeping us going 🔥
Thank you for the transparency @[email protected] It is rare for platforms to scale this quickly so issues like this are inevitable. Good luck for the troubleshooting!
Thanks for all the work you, and all the others do to welcome so many new users, and thank you for keeping us posted on your efforts to smooth things over.
From what I see the general sentiment is very positive and understanding, and it feels special to be part of a community that's more upbeat than what you see in most online places.
Thanks for keeping us apprised. Hopefully you will find the resource leak.
Thank you to you and everyone behind the scenes working hard to make Lemmy a great place. I am turkey enjoying the experience even with the bugs. Brings back a ton of nostalgia with modern mixed in.
This has been such a refreshing change from reddit! I'm willing to deal with occasional hiccups, and I know it will only get better with time. Keep it up!
Appreciate the update, best of luck.
What sort of container orchestration and database tools are you using? Maybe you need some sys admin help?
I expect it is federation outbound activity growing in memory usage, it's all held in RAM for every post/comment/like going out to subscribed servers. [email protected] is a community for the focus on scaling Lemmy.
Lemmy also buffers failed posts/votes in memory, such as if there's server issues like Lemmy.world and Lemmy.ml are having, to retry later.
Couple that with the server having to deal with an exponentially increasing amount of servers to federate with (since our Federation model currently links every server with every other server in a worst-case scenario), and that's probably doing bad things to the performance.
around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
25 GB of RAM????? Looks like your instance is the edge case with Lemmy's scalability, when there's really big amount of active users. Looks like some problems won't fade with really high-grade hardware. But I have to appreciate, that's one gargantuan job!
I think that’s a premature conclusion. Sounds like a memory leak of some sort.
Fixed in software.
I’ve had instances with only a dozen or so Wordpress sites that will chew through 8gb and go into swap when certain tasks were poorly scheduled (like a couple of sites clearing their cache at the same time).
So I’m not surprised they are hitting 25gb but the gradual nature (from the sounds of it) off the memory build up does sound like a leak.
Don’t envy the devs but I’m sure some of them are enjoying it 😂
I can come whine in the discord if that'll help.
Thank you for everything you do. With any luck, you'll get all the support you need, and have it running smooth as silk.
Thanks for making this possible in the first place 🤘
Yall are navigating waters that you probably have little professional experience with. Keep at it and don't get discouraged.
Its nice to have a place where the admins and developers aren't trying to scheme and manipulate the community to get a big payout. Thanks for all that you guys do. Curating this community is no doubt selfless hard work but we need a place like this free from the influence or potential take over by bad actors and malignant narcissists.
Amazing work team. I am already seeing improvements. Hope you are not killing yourselves though, I'm sure everyone realises how difficult it is and that it will take time to fix. We're here for the long haul! Thanks again
It’s a ton better this afternoon! Thank you!!
Do you need any spare server hardware? @[email protected] I would be happy to donate some!
No thanks, it's all at Hetzner, and thanks to all donations we can extend when needed!
If you need any Python QA test automation help I’m happy to invest. Thanks for the new home.
I switched to a regional instance to take the load off for everyone, Lemmy is even better!
Thanks y'all.
Are there any plans to make user up and down votes not viewable publicly? I know for a lot of new adopters this can be a deal breaker.
If someone cares about their privacy that badly why vote at all? I'm sure votes are tracked by the sites on other websites?
On reddit only reddit knows your up and down votes which are never made public unless you check an option in settings. The fediverse is already a target for brigading due to the decentralized nature, allowing bad actors to figure out who to target seems like a terrible idea. Imagine what people like /r/againsthatesubreddits would do with that info.
Thanks for the update!
Thanks for your work, you are making the Internet a better place!
Thanks for putting in all the hard work. Especially on a holiday. We appreciate you guys!
It’s not a holiday here… just a normal work day
Thanks! Love updates. I moved to lemm.ee for now, I'll come back to your world when you got room for my fat arse.
Can’t go wrong with either, both admins are very communicative.
Keep up the great work! We believe in you! Just a reminder to everyone to help sustain lemmy.world by donating in the sidebar! If 1000 people all donated $2/month then that's 24K a year to keep the hardware running! Reddit is what happens when you rely on Corporate Overlords to provide the services you love.
All these updates are really much appreciated. I would love to see the conversations happening in this discord but I understand that you would want to limit that to only a small group: do you have someone making sure this is archived somewhere, and available in a few years? That would be a nice read then.
Encouraging to hear about how much help you're getting. A happy lemmyverse is a happy fediverse?
As a former programmer, I appreciate the hard work you folks are putting in so the rest of us can putter away our free time. You're doing great. Thank you.
Keep up the good work. It's nice to feel like we are back on the bleeding edge.
Thanks a LOT for your work. It's amazing seeing what people can do when they work together for a good cause.
That DM sending bug was already fixed in a later
rcversion. Although inrc.9(and possiblyrc.10) there is a weird bug where you can't sign out. Fun stuff.🎵99 little bugs on the wall🎶
Can we configure the website to not automatically collapse deep replies? Or at least to collapse them only past six or seven layers deep?
Thanks for the hard work and transparency. It gives confidence that you guys will get through this.
It is hard not to root for your success in this. I'm temporarily off .world, but would love to come back when you guys get this under control more.
Thank you for everything you’ve done and continue to do!
Is there anything that people can do to help if they’re unable assist in troubleshooting?
Get me coffee. I prefer cappuccino.
I’ll bring you a couple. I don’t think one is gonna cut it for as busy as you’ve been the past few days.
That’s cool. Appreciate the work to you all!
More props for you and the other admins as you ride the waves. It’s for a great cause.
Keep up the good work! Things have been really snappy the past few hours 😎
Huge heartfelt thank you to everyone involved - given the huge growth you're doing awesome!
It would be great to see plans on how to grow in a controlled and safe way and plans for improving the performance. Really hope you have the funds needed and continue to have the income to support everything needed.
Thank you again, it's wonderful to get an update!
Again, thanks for all your hard work, and your transparency about the whole ordeal. Hopefully, the userbase in general is being patient and understanding given the situation.
I recall that previously we couldn't update jerboa due to it not being compatible with the last server version. Is that updated now, can we update Jerboa?
Yes, Jerboa now works with lemmy.world
I installed Jerboa, but I can't login with my account on there.
Thanks, just updated and all good.
I am using Jerboa right now and I installed just a couple days ago. Must've been silently fixed.
lemmy.world is on 0.18.1 since a couple days ago, that was the problem.
0.17->0.18 changed some APIs, and lemmy.world was sticking with 0.17 for a while due to broken stuff in 0.18, which broke Jerboa which just expects 0.18.
I appreciate the transparency. Keep it up! Everyone is loving it here (◍•ᴗ•◍)❤!
It’s really good to know that there are more people helping sorting everything out.
I’d just like to suggest publishing the workarounds and fixes public so that other small instances can get it sorted faster when they grow.
Glad to see these progress updates. Thanks for the transparency.
Thanks for all of the updates and work on this instance! As a fellow sys admin, I sympathize with the issues trying to get this update going.
Keep up the good work @Ruud and you all wonderful admins, mods and lemmings for making this possible! 💙💚
I really appreciate the updates. As someone in the tech field, it's been fascinating getting the inside scoop on how it's all coming together.
Edit: I made another account off of your instance to ease up on the pressure, but so many communities and users I care about are on your instance, so I'm using it too!
The biggest frustration is 502 errors on submit. Sometimes th8ngs go through, sometimes not
My workaround is to submit/browse from federated servers until lemmy.world is running well
Using the connect for Lemmy app produces constant Under Maintenance or html code or 504 errors. It seems to load better on Chrome.
Thank you for all your hard work!
Just wanted to say thank you all for you efforts and dedication.
Thank you. Y'all are doing great.
I'm sure it's stressful but I know I'm here to stay. It's been a blast so far
Appreciate the clarity and hope things keep moving in the right direction
Nice! It feels better now.
I hope hosting all these massive communities is enough a portfolio to land Ruud a crazy job offer one day that pays him stupid levels of money while allowing him plenty of time off to maintain these massive hobby projects.
Thanks for all your efforts.
I'm sorry to hear you're using Discord, for me it fits right in with the likes of Reddit, Twitter and Facebook. Why not use something independant and federated like Matrix or XMPP?
I think I use all chat software there is. I'm in hundreds of Matrix rooms. But I think one of the team at least didn't like or use Matrix. Don't remember. And I have Discord anyway for the Mastodon channels..
Thanks for your hard work. Noticing janky bits getting ironed out everyday. Has anyone had any luck making Lemmy.world work in Qutebrowser? It just hangs on loading comments/posts..Any work arounds? I'd love that..working quite well in Firefox and mobile though.
It seems fine right now. Thank you.
Great work! We believe in you guys.
Thanks for working so hard on this and keeping lemmy going. This instance seems to have become synonymous with lemmy as a whole for some of the new comers so keeping it up and running is really helping to keep the new comers on board, this can only be good for all of lemmy in the long run!
Really appreciate the transparency about what's going on behind the scenes too. It's nice to know what my donation is doing and I hope you have time for cappuccinos.
It's not perfect but becoming better everyday.
Don't know about anyone else but I keep getting a "cannot load comments"error and the whole post goes blank.
Still happening? What browser/app and OS?
I think most of the users can understand the current situation. Take care guys and girls!
Thanks u/ruud!
How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.
Curious about the server info. Is there more info available somewhere else?
Is lemmy.world running on a single server? What's the CPU and memory capacity? You mentioned a percentage of CPU usage and an amount for ram, so that made me curious about what's available for each.
Does restarting take down all services or is the application clustered or not monolithic somehow?
It's a single server with 32core/64 thread AMD EPYC and 128GB RAM. At the moment we run multiple containers for lemmy so restarting doesn't mean outage.
Out of interest what's the network 95 percentile like over a day? Keep up the great work :-). Thanks for your hard work.
Are you running the database on the same server as the backend?
Your work is much appreciated!
Thanks for the update, glad you guys are being transparent about the issues everyone seems to be facing. Looking forward to the coming improvements. 😁
You lot are internet heroes. Thank you for putting in the work and creating this community!
Thanks again for the hard work that you and the others are putting in Ruud. It can't be easy trying to patch the boat while the boat keeps growing ridiculously fast at the same time. You are very much appreciated!
Hi, am I correct that typing on iPad currently doesn’t work? As in the keypad won’t show up. I have to type in google or notepad and paste (the paste popup still works) to lemmy.world lol! Anyone else?
As always, thanks for the hard work everybody puts into this!
Had that happen earlier this week. Update your Lemmy client, then update your iPad, then restart your iPad. That fixed it for me with WefWef.
You are the boss! Turning the iPad off and on again worked :)
Thanks a lot for addressing these issues. It's really nice to see how this platform keeps growing and growing.
Yup, I'm having the 2FA problem. I can't log into mobile, because no code gets generated. Luckily I'll just stay logged into a browser for now.
But thanks for all the hard work! :) You're probably getting a flux of Reddit migrants, such as myself now
I removed the 2fa for you
Oh thanks so much!
Thanks for the update!
So seems that Reddit refugees influx indeed hit this instance.
Connect for Lemmy, Android and yeah still happening.
Thanks for the update and keep up the great work! :)
Appreciate all the work. I’ve only been on Lemmy a few days but the openness is refreshing and I think this place has been pretty cool.
Anything to be said for using me cache or redis?
Hi, am I correct that typing on iPad currently doesn’t work? As in the keypad won’t show up. I have to type in google or notepad and paste (the paste popup still works) to lemmy.world lol! Anyone else?
Thank you for your hard work and transparency
In which country is this server located?
Finland
Growing Pains
Thanks for all your hard work. Wow that is a lot of CPU usage. I hope it can be tuned more. Is it mostly postgres causing the load?
As much as I hope that Lemmy grows, I wouldn't want lemmy.world to suddenly get 1 million signups and multiply your headaches.
Perhaps we need more well run servers by other people to take the load.
