Docker configs destroyed after update [resolved!]
Docker configs destroyed after update [resolved!]
Hello selfhosted community, something weird just happened to my setup while running a routine update.
I'm running docker containers on a couple Debian LXCs through Proxmox, and a regular apt-get upgrade just wiped all my configurations. Somehow it seems to have gutted my databases and deleted the compose.yml files without a trace remaining. Thankfully all my data seems to be intact as far as I can tell.
Did I royally mess something up in all of my configurations or in doing the update? This has never happened to me before. Thankfully I have a backup for the configs that's about 6 days old, but it's still extremely annoying. Any hints? Thanks
Update in case anyone is interested: I figured out what caused the problem. When I mounted the new drive I used to store my configurations onto Proxmox, I completely forgot to make the relevant /etc/fstab entry. The drive mounted successfully so I didn't realize at the time that I had forgotten to do that step. The update I ran from apt-get included a kernel update, so I restarted the machine to complete it. Since I hadn't modified fstab, my new drive was not mounted when the system started up again. Even though the drive wasn't mounted correctly, I still somehow had access to some incomplete version of the files in its directory (no idea how that works). So no fault of Docker, LXCs or Proxmox, purely PEBKAC.
Despite getting the files back I will still work towards a more resilient system and more regular backups.
I'm not sure. All I know is that Proxmox doesn't recommend using Docker in LXCs. It seems Proxmox has much more control over the internals of the container and can break stuff.
https://pve.proxmox.com/wiki/Linux_Container
If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox QEMU VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.
It was just a matter of time then until something broke? Guess I'll need to do some more research on how to best manage the services I want to run. Good thing I didn't come into this hobby hoping to reduce the amount of headaches
I run a debian VM for hosting docker things.
Its like 500mb extra disk space, and a small ram/cpu hit. But its been solid for me
You didn’t happen to change an unprivileged container to privileged, or vice versa, after creating it, right? Doing so can break filesystem permissions, which could have resulted in something like this.
Interesting, I think I did actually do that. I guess my best bet for now is to just nuke the LXCs then and move everything to a new VM. Thanks for the advice!
That's pretty weird, where were the files stored that went missing? Just in your home folder?
Definitely recommend setting up regular daily backups, you can use the built in proxmox backup system, or run whatever tool you want inside each VM/CT.
The files were on a mountpoint, completely separate drive. This has also been the case for all the previous times I ran an update, though I did recently move these files to a different drive mounted the same way. I got some sort of permission wrong maybe?
Will definitely set up a better backup system as soon as I can
Very odd, I think something else is going on not related to LXC/Proxmox. I've got LXC containers with external mounts and internal storage and haven't seen any data loss over the years they've been running.
New Lemmy Post: Docker configs destroyed after update (https://lemmy.world/post/12089867)
Tagging: #SelfHosted(Replying in the OP of this thread (NOT THIS BOT!) will appear as a comment in the lemmy discussion.)
I am a FOSS bot. Check my README: https://github.com/db0/lemmy-tagginator/blob/main/README.md
Hints? Don't use Docker for your own sake. Why would you, you're already running LXC containers, just setup whatever you need inside those and you're good to go with way less overhead and bloat.
While you're at that, did you know that the creators of LXC have a solution called LXD/Incus that is way better at managing LXD containers and can also create and VMs? For what's worth is a 100% free and open-source solution that can be installed on any clean Debian 12 setup from their repository and doesn't require 1000 different daemons like Proxmox does nor does it constantly nags for a license. :)