How should I do backups?
How should I do backups?
I have a server running Debian with 24 TB of storage. I would ideally like to back up all of it, though much of it is torrents, so only the ones with low seeders really need backed up. I know about the 321 rule but it sounds like it would be expensive. What do you do for backups? Also if anyone uses tape drives for backups I am kinda curious about that potentially for offsite backups in a safe deposit box or something.
TLDR: title.
Edit: You have mentioned borg and rsync, and while borg looks good, I want to go with rsync as it seems to be more actively maintained. I would like to also have my backups encrypted, but rsync doesn't seem to have that built in. Does anyone know what to do for encrypted backups?
You're viewing a single thread.
I am simple man s I use rsync.
Setup a mergerfs drive pool of about 60 TiB and rsync weekly.
Rsync seems daunting at first but then you realize how powerful and most importantly reliable it is.
It's important that you try to restore your backups from time to time.
One of the main reasons why I avoid softwares such as Kopia or Borg or Restic or whatever is in fashion:
- they go unmantained
- they are not simple: so many of my frienda struggled restoring backups because you are not dealing with files anymore, but encrypted or compressed blobs
- rsync has an easy mental model and has extremely good defaults
As long as you understand that simply syncing files does not protect against accidental or malicious data loss like incremental backups do.
I also hope you're not using
--deletebecause I've heard plenty of horror stories about the source dir becoming unmounted and rsync happily erasing everything on the target.I used to use rsync for years, thinking just like you, that having plain old files beats having them in fancy obscure formats. I'm switching to Borg nowadays btw, but that's my choice, you gotta make yours.
rsync can work incrementally, it just takes a bit more fiddling. Here's what I did. First of all, no automatic
--delete. I did run it every once in a while but only manually. The sync setup was:- Nightly sync source into nightly dir.
- Weekly sync nightly dir into weekly dir.
- Monthly tarball the weekly dir into monthly dir.
It's not bad but limited in certain ways, and of course you need lots of space for backups — or you have to pick and choose what you backup.
Borg can't really get around the space for backups requirement, but it's always incremental and between compression and deduplication can save you a ton of space.
Borg also has built-in backup checking and recovery parity which rsync doesn't, you'd have to figure out your own manual solution like par2 checksums (and those take up space too).
Re needing lots of space: you can use --link-dest to make a new directory with hard links to unchanged files in a previous backup. So you end up with de-duplicated incremental backups. But borg handles all that transparently, with rsync you need to carefully plan relative target directory paths to get it to work correctly.
Yeah Borg will see the duplicate chunks even if you move files around.
As long as you understand that simply syncing files does not protect against accidental or malicious data loss like incremental backups do.
Can you show me a scenario? I don't understand how incremental backups cover malicious data loss cases
Let's say you're syncing your personal files into another location once a day.
On Monday you delete files. On Tuesday you edit a file. On Wednesday you maybe get some malware that (unknown to you) encrypts some files (or all of them).
A week later you realize that things went wrong and you want the deleted files back, or the old versions of the file you edited, and of course you'd want back the files that the ransomware has encrypted.
If you simply sync files you have no way to get back deleted files. Every day it synced whatever was in there, overwriting what was there before. If you also sync deletions then sync deletes the files. If you don't sync deletions then files keep piling up when you delete them or you move them around.
An incremental backup system like borg looks at small file chunks, not at files. Whenever a file changes, it makes a copy of only the chunks in it that changed. That way it can give you the latest version of the file but also all the versions before, and it doesn't store the same file over and over, only the chunks that really changed, and only one of each chunk. If you move a file to another folder it still has the same chunks so borg stores that it moved but it doesn't store the chunks twice. Also if several files have identical chunks, those chunks are only stored once each. And of course it never deletes files unless you explicitly tell it to.
Borg can give you perfect recall of all past versions of every file, and can do it in a way that saves tremendous amounts of space (between avoiding the duplication of chunks and compression).
FWIW restic repository format already has two independent implementations. Restic (in Go) and Rustic (Rust), so the chances of both going unmaintained is hopefully pretty low.
I was heavily considering borg but I just looked up rsync and it looks like everything I need. Thank you.
Edit: Actually encryption would also be nice. Is there any way to do that with rsync?
what other people are saying, is that you rsync over an encrypted file system or other type of storages. What are your backup targets? in my case I own the disks so I use LUKS partition -> ext4 -> mergerfs to end up with a single volume I can mount on a folder
Yes. You compose a crypted vault over your storage vault. I pay about $1/mo for B2 Backblaze. Around 150G last I checked.
One of the main reasons why I avoid softwares such as Kopia or Borg or Restic or whatever is in fashion:
- they go unmantained
- they are not simple: so many of my frienda struggled restoring backups because you are not dealing with files anymore, but encrypted or compressed blobs
- rsync has an easy mental model and has extremely good defaults
Going unmaintained is a non issue, since you can still restore from your backup. It is not like a subscription or proprietary software which is no longer usable when you stop to pay for it or the company owning goes down.
The design of restic is quite simple and easy to understand. The original dev gave multiple talks about it, quite interesting.
Imho the additional features of dedup, encryption and versioning outweigh the points you mentioned by far.
Going unmaintained is a non issue, since you can still restore from your backup. It is not like a subscription or proprietary software which is no longer usable when you stop to pay for it or the company owning goes down.
Until they hit a hard bug or don't support newer transport formats or scenarios. Also the community dries up eventually
Until they hit a hard bug or don't support newer transport formats or scenarios. Also the community dries up eventually
That is why you test your backuo. It is unrealiatic, that in a stable software release there is suddenly, after you tested your backup a hard bug which prevents recovery.
Yes unmaintained software will not support new featueres.
I think you misunderstood me. You should not use unmaintained software as your backup tool, but IMO it is no problem when it suddenly goes unmaintained, your backup will most likely still work. Same with any other software, that goes unmaintained, look for an alternative.
It is unrealiatic, that in a stable software release there is suddenly, after you tested your backup a hard bug which prevents recovery.
How is unrealistic? Think of this:
- day 1: you backup your files, test the backup and everything is fine
- day 2: you store a new file that triggers a bug in the compression/encryption algorithm of whatever software you use, now backups are corrupted at least for this file Unless you test every backup you do, and consequently can't backup fast enough, I don't see how you can predict that future files and situations won't trigger bugs in a software
We talk about software that is considered stable. That has verification checks for the backup. Used by thousands of ppl. It is unrealistic.
Two questions, and please don't take it as criticism, I am just curious about rsync but also one point you make.
"They go unmaintained" seeing as Borg is in use for quite some time, how does this look safer for rsync? For me it looks like the risk for that is similar, but I might not know background of development for these.
Second question more something I am asking myself, a lot of people seem to use rsync for backing up, but it is not incremental backup, or is it? I saw some mention of a "time machine" like implementation of rsync, but then we are again at your argument it might go unmaintained as its a separate niche implementation, or does that main rsync support incremental backup? If not, are you not missing that, how do you deal with it when just a file changes? New copy of it being transferred or somehow else?
how does this look safer for rsync? For me it looks like the risk for that is similar, but I might not know background of development for these.
Rsync is available out of the box in most linux distro and is used widely not only for backups, but a lot of other things, such as repository updates and transfers from file hosts. This means a lot more people are interested in it. Also the implementation, looking at the source code, is cleaner and easier to understand.
how do you deal with it when just a file changes?
I think you should consider that not all files are equal. Rsync for me is great because I end up with a bunch of disks that contain an exact copy of the files I have on my own server. Those files don't change frequently, they are movies, pictures, songs and so on.
Other files such as code, configuration, files on my smartphone, etc... are backup up differently. I use git for most stuff that fits its model, syncthing for my temporary folders and my mobile phone.
Not every file can suit the same backup model. I trust that files that get corrupted or lost are in my weekly rsync backup. A configuration file I messed up two minutes ago is on git.
Thanks for elaborating, the part about the pictures and movies not changing makes a lot of sense actually. Thanks for sharing!
One method depends on your storage provider. Rsync may have incremental snapshots, but I haven't looked because my storage provider has it.
Sometimes a separate tool like rsnapshot (but probably not rsnapshot itself as I dont think its hard links interact well with rsync) might be used to manage snapshots locally that are then rsynced.
On to storage providers or back ends. I use B2 Backblaze configured to never delete. When a file changes it uploads the new version and renames the old version with a timestamp and hides it. Rsync has tools to recover the old file versions or delete any history. Again, it only uploads the changed files so its not full snapshots.