Update on SJW upgrade to v19.3
Update on SJW upgrade to v19.3
Hello sh.itjust.works community,
Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.
In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:
- Dual Xeon 2.9Ghz CPUs (32 cores total)
- 256GB ram
- 4 x 1TB SSD in raid 10 (with room to add 6 more disks)
- 10gbit networking
While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.
- Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
- There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.
Here's the tentative timeline:
| Task | Date | Expected Downtime |
|---|---|---|
| Migration to new server | Tuesday February 27 2024 @ 8:00PM ET | 90 Minutes |
| Upgrade to V19.3 | Thursday February 29 2024 @ 8:00PM ET | Up to 120 Minutes |
- If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
- If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.
During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel
=========================================================
Update February 29 2024
We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.
=========================================================
Update February 27 2024
We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you
So you bought a server for us? I really need to contribute financially
Is there a way to donate yet?
Highly doubt it. What he has said he needs are volunteers to help run the place
Every time I see SJW, I think "Social Justice Warrior" not "Shit Just Works."
Omnissiah's blessing on the upgrade @TheDude!
TIL this instance is not "Shhh, it just works!" and it is in fact "Shit just works"
I love your alternate interpretation. There are few things in my life which work but I'm not sure how, and I don't want to investigate too much for fear that I break them.
You're just too classy
I like it. I'm just waiting for someone to bring it up as a mark against our instance somehow
I'm happy to be affiliated with both so it works for me.
We should have a Dude Day every year for the anniversary of this instance. I've been very happy with their handling of a group of strongly and diversely opinionated interweb users.
Omnissiah's blessing indeed.
Dude Day
idk that one might be taken
https://www.dudeism.com/holidays/
Some reason it never crossed my mind that someone could run their own instance on their own hardware and not like a cloud provider. That's super cool!
Like Batman and Bruce Wayne, TheDude and TheCloud are never seen together.
Big D Energy
(Dude)
(Decentralized)
(inDependent)
Good luck with the upgrade!
Thanks for running this place, and thanks for the info.
What do you expect me to do for that 90 and 120 minutes??
Prepare the memes for deployment!
ST:TNG isn't going to frame-advance through itself.
Looks like I have no choice but to be productive tonight. Maybe I'll write a SJW back online checker. ;)
Good luck. Yours brought me to Lemmy and then I made my own soon after.
With a url like that how could you not?
Godspeed, f.ukwad.
(Hmm...not as cute as sh.it.head. I still kinda like it, though. Excellent domain name choice in general.)
Thank you for all your time and effort.
The Dude, as per usual, abides.
Thank you again for all you do.
Thank you for everything you are doing. Take a few days if you have to fit this migration and update. I've hosted many servers... I always feel I need to do so much for everyone. Curb that thought, if you can. We genuinely appreciate what you're doing here.
Not on your instance but good luck!
I appreciate everything you do for this instance. Also, thanks for posting the specs of the new build. It’s pretty neat to see what’s going on behind the curtain.
Good luck - I previously moved Feddit.dk to a new server and recently upgraded to 0.19.3 without any problems during either upgrade - hope you have the same experience!
Good luck ! We did it for jlai.lu, it was a long week-end :)
We can't thank you enough and we all love you. You're a rock star.
Wow, so new and snappy, smells like fresh cut mahogany in here
Thank you. We love you very much!
Question. Why would you go with 1tb ssds instead of larger hdds? Isn't the space and price more important than the speed for this use?
You could get double the space (2tb hdd) for the same price as a 1 tb ssd.
Just wondering.
The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn't want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don't plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn't cutting it anymore I'll probably have come across another opportunities on getting a new server with better NVME support.
You let us know when you need some help with new NVMes. We're more than willing to contribute ;)
Speed is usually the reason. SSDs in general are faster, enterprise SSDs are not only faster but much more write-tolerant and last a very long time in comparison to consumer SSDs.
They can also (in many cases) do write caching at the speed of a DRAM buffer, making the bottleneck the SATA or SAS bus itself (SAS is like enterprise SATA, 12Gb/sec as opposed to 6). NVMe can be even faster. This means that programs (ie Lemmy and its database) that write data aren’t waiting around for the drive to acknowledge the write before that program can move on to other things. Shaving off a few milliseconds per write can make a massive difference when you realize there might be millions of IOPS (Input/Output operations Per Second) under load. The requirement for low latency is everything in servers.
When you are running a public service and requests are coming in constantly and at a high rate, you really really do not want storage latency to bottleneck you, as that is a problem that will compound extremely quickly. This is a big issue with HDDs as well, as even disk seek times add to the problem, let alone caching/buffering writes.
We could talk all day about if four SSDs in a RAID 10 are optimal, but sometimes you have to think about budget and complexity as well. For the load that a popular Lemmy instance might currently draw, I’d make an educated guess that this might be sufficient for now. Room to expand was also mentioned, which is the second most important part of a storage plan.
Probably for faster loading times when playing games on the hardware during off peak hours jk jk lol
Speed is important. All else equal, the database will work faster with SSDs. Raid also makes the storage be under heavier load so SSDs make even more sense here as well. You want response times to be as low as possible for a good user experience.
But also SSDs are kinda standard now and you can get a decent amount of storage for not that much higher price. Especially for server hardware that is more or less constantly under load, SSDs just make a lot more sense.
Thank you for everything you do!
Dude, you're the Scotty of the fediverse, you rock.
Thanks Dude
this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected
Woohoo, that's awesome! Thank you so much!
Not on your instance, but wishing you luck on the migration and updates!
Good luck from one small instance that just moved VPS providers and upgraded to 0.19.3. I ho no issues except for email died and I had to figure out the new hosts email gateway that was lacking lacking in some of it's documentation
Woohoo!
Now, can anyone point out what -if anything- changed for us end users?
This change only brings speed & stability, which is essential, but hard to see for us, end users. The bigger one is going to happen on Thursday, where Lemmy itself is going to be updated. After Thursday's update, any users will be able to block entire instances and see our upvotes, along with many other Lemmy updates.
Just want to say Good Luck and we're all counting on you!
Looks like the site was down this morning. Was the migration done earlier ? or just some downtime ? Good luck for the migration, It's an opportunity to use my alt account on fedia, so not that bad
There was a little hiccup this morning unrelated to the migration tonight. The Lemmy services like to be restarted every once in a while. Once we move to the new hardware I'll be able to look into implementing a better logging system and hopefully be more proactive when situations like these happen.
Tysm for your work ! I appreciate it a lot!
Thanks for the update.
Looking forward to the new update.
Please let us throw monies at you
I appreciate everything being done on our behalf. Thanks Dude!
Nice job on the migration! Excited for the update!
Félicitation à toute l'équipe, merci pour votre travail
Dang, I was looking forward to being productive today, but you were just too quick.
Lets go! Congrats on the upgrade of the hardware!
Good luck with the update in (if I'm right in the timezone calculation) a couple of hours...
Good luck, Dude! I'm sooo looking forward to seeing what I previously upvoted.
Thank you!!!
That's a relatively low spec server, I thought it would be worse. What does the co-location cost?
So how much do you pay for the server?
I've encountered a problem, not sure if it's related to the update or not. On both Eternity and browser my Subscribed feed shows posts from All. From Eternity only upvoting posts fails with a 401 code, but I seem to be able to do it on the browser. Any one else encountered some like that?
Others have reported making Eternity work by removing their account and re-adding it.
Thanks, that seems to have fixed it!
How does one go about the blocking whole instances in 19.3?
Profile > settings > blocks
Be aware that this will only block communities of that instance from your feed. You'll still see the users from the blocked instance posting in other communities.
right on, thanks
How can I help, my friend?
I know Scaled isn't selectable in Sync so you have to set it as default in Lemmy settings to use it. Does anyone know what other new features have no/limited compatibility with Sync? Or any other app for that matter?
Edit: I already had the sort for subscribed and everything overridden so I had to figure out how to clear that (since scaled isn't an option when I try to change the override) but now it's good.
The trick is to hit the correct cancel btw 🙄
Pictures don't seem to show up... but I'm guessing this is a caching issue, so it should be resolved by itself.
Wow congrats ! I'm glad it went well :D
Can we please unpin this now? It’s the top of all of my feeds still.
GG