important instance shit @awful.systems self @awful.systems 12mo ago

upgrade: lemmy 0.19.2 + infrastructure [clear your browser cache and cookies!]

today's (later than planned) upgrade to lemmy 0.19.2 provisionally appears to have gone alright. if you see excessive amounts of jank (and your page footer can't decide what version of lemmy it's running on, IE it shows separate FE and BE versions), clear your browser cache and cookies since lemmy doesn't seem to do that cleanly on its own

next up I'm planning to deploy the Proton frontend as an alternative to the default and I'm also going to start pushing code to codeberg (most likely) so stay tuned for that

You're viewing a single thread.

40 comments

In today's episode of awful systems I found outt my phone's browser can't manage individual site cookies. Deleting them is an all-or-nothing deal.

Tfw the mono no aware hits you when you have to log out of everything just to make fun of techbros.
- christ I’m sorry. my phone probably needs the same — the mother fucking cache-control immutable header some lemmy dev decided to hardcode into lemmy-ui’s app server seems to have triggered a bug in my phone where I’m both logged in and not simultaneously, and switching tab groups logs me out until I open a new tab
  
  since all this unpleasantness is due to a bad cache-control header, hopefully it should all clear up when caches expire and pull in the new version of the site and corrected cache settings… in like 20 hours. fuck
  
  They were using immutable cache control headers on resources that were, in fact, mutable? Uh oh.
  
  oh yeah! it’s been causing a ridiculous number of problems for our instance and others (I thought beehaw was down for a long time but I’m realizing it was this cache issue) and basically makes upgrading to a backend that isn’t completely compatible with the previous backend impossible to do cleanly. and yet, I think I might have been the only one who’s discovered this issue, judging by the lack of relevant GitHub issues or other chatter about it.
  
  as for why some lemmy dev chose immutable, all I can figure is they saw that rust is immutable (and I like rust a lot, but these are not good rust devs) and figured that meant immutability is a good thing everywhere regardless of the actual implementation. or maybe Google’s SEO tools insisted they needed it, and they didn’t read on or understand the versioned path requirement for cache busting? you’d think the devs of a federated platform would focus on literally anything other than SEO, but none of these folks had a plan.
  
  anyway… only 7 hours remain until everyone’s browser hopefully marks the old content stale
  
  I saw a few closed issues relating to cache control stuff. They've fixed some bugs related to it and also saw at least one where more aggressive cache control was implemented to improve performance.
  
  oh hey speak of the devil https://github.com/LemmyNet/lemmy-ui/issues/2325 so this isn’t even purely a cache issue, somehow they also broke upgrading credentials between lemmy versions and nobody noticed
  
  fucking brilliant. I might see if I can push a new version of the error page that doesn’t lie about what’s wrong now that caching hopefully isn’t busted
  
  Lemmy-ui immutable static files have a cache busting hash in their path:
  
  On awful.systems, I see you have replaced this hash with an undefined:
  
  By the way, clean upgrades are completely possible if you know what you're doing. I frequently roll out new upgrades on lemm.ee without any downtime (so some servers running 0.19.1, some running 0.19.2 for example).
  
  By the way, clean upgrades are completely possible if you know what you’re doing.
  
  oh do tell
  
  https://join-lemmy.org/docs/administration/horizontal_scaling.html#rolling-upgrades
  
  no thx but while we’re here
  
  the reason why the cache busting broke is because I’m building for an immutable environment (NixOS sans docker) and my builds don’t have .git cause it’s a source of mutability. lemmy’s cache busting utterly breaks without git but it still incorrectly marks the files as immutable, breaking a ton of users. utterly broken behavior.
  
  but the very funny part is the cache behavior isn’t even the bug that’s breaking everything right now. that’s this bug: https://github.com/LemmyNet/lemmy-ui/issues/2325 which you didn’t mention as a factor while you were trying your best to sound smart
  
  also, have you ever gotten banned from an instance for being a shithead reply guy before? I’m betting this ain’t the first time
  
  I frequently roll out new upgrades on lemm.ee without any downtime (so some servers running 0.19.1, some running 0.19.2 for example).
  
  also motherfucker, those versions are compatible! no shit there’s no downtime for a hotfix upgrade. we got into this mess upgrading from 0.18.4 to 0.19.2 cause nobody seemed to think the upgrade process through in any detail
  
  for anyone interested, the hack that NixOS uses to fix the asset path is here. the config we’re running predates that, and we may not be able to use that hack with a flakeified lemmy-ui (which is what makes it a hack — using src.rev like that is brittle and feels like it breaks Nix’s abstractions. maybe using git hashes like this is actually a bad fucking idea, huh? and the real fucked up part is, the UI knows its version so they could use that, but they woke up and chose jank)
  
  Haha wow, now it's hitting me with the same thing I think. Keeps logging me out randomly and putting me in a state of Schrödinger's session cookie.
  
  All hail the jank!
  
  Edit: seems to happen if I refresh the page.
  
  yep, exactly what I’m seeing except it stops when I open awful.systems in a new tab, but happens again if I switch tab groups. I don’t know how they managed this
  
  Only two hard problems in CS: cache invalidation, naming things and off-by-one errors. But especially cache invalidation and especially especially when you make it even harder lol.
  
  no such luck lol
- this is also in Chromium for desktop
- This happened to me with mobile Firefox too, RIP my sessions

40 comments