I feel like this was another one of those "google does something shitty to specifically target Firefox users for degraded performance" situations.
The problem goes away when you set network.http.http3.enable to false.