Programming.dev Meta @programming.dev snowe @programming.dev 1y ago

[Completed] Upgrade to version 18

I will be updating the instance to v18 at ~~20:00~~ 22:00 UTC.

See https://programming.dev/post/181191 for the changes

edit: Lemmy.ml updated and seems to have gone down. We're going to wait and see what the outage was caused by and then proceed from there.

edit 2: lemmy was down due to a ddos attack. We will upgrade at 22:00 UTC

edit 3: we had issues with the email setup getting overridden again. If you tried to sign up in the past 8 hours please try to just log in. If you can't, please message me (discord, matrix, or mastodon)

25 comments

After logging a daily cycle of the system base load, could you post some before and after metrics for resource usage for comparison? That'd be fun to see.
- Of course, though I think a single image just of the past few hours would demonstrate the difference pretty well lol:
  
  I'm not sure if you intended to do this, but you've doxxed yourself in this image.
  
  If this was deliberate, please ignore this comment.
  
  Question for sysadmins: if you were looking at those metrics, without any context about an upgrade to production, what would you first suspect had happened?
  
  Before cross-check unique visitor or page count remained steady, I'd have guessed the site got yeet'ed of every search platform or red flagged and blacklisted. 🚩💣☠️
Phew I hope lemmy.ml is doing okay...
- RIP
  
  Looks like their back up and running using UI: 0.18.0-rc.6 BE: 0.18.0.
  
  I apologize for the stability issues everyone.
  People found an exploit and are using it to DDOS several lemmy instances.
  https://lemmy.ml/comment/909960
  
  Relevant ticket for patch fix:
  
  300 comment limit. #3306
  
  https://github.com/LemmyNet/lemmy/pull/3306
I seem to be facing some issues after the 0.18 update. I'm unable to report comments on subs on lemmy.world, I made myself as a mod from my other account on lemmy.world, but I'm not seeing that reflect here. Are these known issues?
- Hm. I'm not sure I understand the problem. I don't see anything like this in the current issues list. Maybe if you could describe it a bit more I might be able to log something?
  
  The report issue seems to have resolved itself now, but the moderator issue is still present. I created a bunch of communities on lemmy.world, and I want to see if I can move over here as my main instance. I thought I'd make myself mod on those communities. I did that for 4 communities, but I only see two of them in my profile as being moderator. However, if I open my programming.dev profile on lemmy.world, it shows that I'm moderating 4 communities. Does that make sense?
  
  One other thing I noticed now, is that if I mark a message as read, it doesn't disappear like it used to. It goes away if I refresh the page, but it used to disappear right away when I marked a comment as read.
Yikes, was wondering why it was down. It loaded just once stating a rate limiting error. Mostly just seeing 504 nginx errors.
If you don't mind me asking, how much does it cost to run all of the servers?
- we're at like $48 a month for the main server, we have a staging server at about $6 a month, the domain is quite expensive, $580 or something for the first year, $427 for the following years. Email is $35 a month, but we had a snafu with it so we had actually gone with a different provider and that was $20 for 3 months. We also have cloudflare and I expect I'll need to set up CSAM protection eventually but that will be a bit of money too, so I'm working on getting a donation box set up first and then we'll move on with that.
  
  Thanks for letting me know. I plan to contribute to the source code, as well as running my own server just to play around with it and getting more familiar with Rust.
  
  I agree with the donation box, I'll donate when it's ready.
  
  FYI, others commenting on hosting Lemmy in an AskLemmy thread:
  
  Why do people host Lemmy instances and how do they pay for them?
  
  https://lemmy.ml/post/1579182

25 comments