This was a very nerve racking experience as I'd never gone through a major version Proxmox update before and I had spent a lot of time getting everything just so with lots of config around disk and VLANs. The instructions were also a big long page, which never fills me with confidence as it normally means there's a lot of holes to fall in to.
My initial issue was that it says to perform the upgrade with no VM's running, but it requires an internet connection and my router is Opnsense in a VM. Thankfully apt dist-upgrade --download-only, shutdown the Opnsense VM and then apt dist-upgrade did the trick.
A few config files changed and I always hate this part of Debian upgrades, but nothing major or of importance was impacted.
A nervous reboot and everything was back up running the new Proxmox with the new kernel. Surprisingly smooth overall and the most time consuming part by far was backing up my VM's just in case. The upgrade itself including reboot was probably 15 mins, the backups and making sure I was prepared and mentally ready was about an hour.
Compared to upgrading ESXi on old hardware like I was doing last year, it was a breeze.
I've really come to appreciate having test systems working as a systems engineer. A simple virtualised install of Proxmox that replicates some small part of your environment is great to simply go through the upgrade once or twice.
Like you I have OPNsense in a VM on one of my PVEs. But I only made sure the nigthly VM back up ran and didnt even bother shutting down the VMs during the upgrade.
The VMs got restarted during the final reboot, as the would with every other reboot, and I was back in business.
But seriously, I was wondering about the requirement to shutdown the VM's and couldn't come up with a solid reason? I mean, even if QEMU/KVM/Kernel get replaced during a version upgrade or a more common update, all of these kick in only after the reboot? And how's me shutting down VMs manually different from the OS shutting down during a reboot?
I know I am speculating and may not have the fill picture, probably a question for the Proxmox team, there may be some corner case where this is indeed important.
Best of luck. FWIW I did the upgrade over an SSH connection and didn't have any issues. Just had to be careful not to wipe my custom sshd_config during the update.
I did the same a few months ago and was extremely nervous. I have a 4 node cluster running 30 VMs in production. After migrating the VMS off of one node I quickly realized what a pleasure it was to do it. No muss no fuss. Migrated the VMs back and continued on with the other 3.
That's pretty cool that it worked so well. Does migrating the VM's result in any downtime or is it a seamless cross over?
I waited a few days before upgrading as I wanted to make sure I wasn't going to get stung by any teething troubles. Would have ideally waited longer but had an ideal few hours available to do it without the family being annoyed by any downtime.
Pve8 was surprisingly painless on my homelab. Only thing that bit me was a left over ceph package that was installed caused apt to want to uninstall pve. After I figured that out and upgraded ceph I was in business.
I just did mine yesterday. One stopped responding mid-upgrade and I wasn't able to reconnect, but I was able to log in at the console and run dpkg --reconfigure -a until I got the network back, then apt install --reinstall proxmox-ve pve-manager got those packages to finish installing, then everything worked.
I always feared the same, from PVE 1.2 on, and was almost everytime painless (after you have all the needed parachutes if anything goes wrong in the process), I have two nodes (not clustered) one for prod env and the other for firewall and VoIP services, and only the jump from 1.2 to 1.5 (if I remember well) was a bit of pain due to added local-LVM, but was a lot of years ago. Now it is a breeze.