1. Important things I care about: these go into a folder that gets backed up to Hetzner (examples include photos in Immich, code pushed to Forgejo, etc)
2. Smaller files I sort-of care about: these usually just go into Proton Drive (examples include design documents)
3. Large files I don't care about losing: these go into a folder on a hard drive that is not backed up (examples include media)
This took me about a day to set up and I just check in on it every once in a while. Once a year I will do a practice disaster recovery run.
One thing I would add to a modern backup strategy: a deferred offline copy
Sure it's bothersome to use the sneakernet once a month (manually copy data to USB stick and stash it somewhere), but when that ransomware hits (can happen on personal devices also), it literally can't access the offline copy, and try to encrypt that one also.
Immutable backup solutions are code and thus eventually defeatable from the remote attackers point of view. Offline requires physical access, and maybe a big enough wrench to get the USB stick's disk encryption PIN out of you.
Tape + Iron Mountain is difficult to beat for offline copies.
Surrendering just a little bit of absolute control over the physical media can dramatically increase the likelihood that it survives whatever apocalyptic event. I don't know that I've ever heard of tapes being stolen from Iron Mountain. Even if this happened, what are the odds that they would get all the tapes. Your tapes? It probably looks like the warehouse from Raiders of the Lost Ark in there.
> One of the other materials that I could not figure out how to back up properly is emails. The reason is that there is no clear way to back it up systematically
Doesn't that depend entirely on the storage format? For example if you use thunderbird as a client it has database files that you can copy. Since you say that one of your devices is running arch you should check out some of the email options in the repos. There are at least a few of options that can act as a middleman to bridge between your external accounts and your clients.
> So having a plan to roll back to a stable state might be helfpul. I installed and set up Timeshift for this
Snapshots make this trivial. Particularly in the case of btrfs if you configure your bootloader to always use the default subvolume then rollback becomes just `btrfs subvolme set-default`.
It sounds like most of your frustration can be eliminated by actually using your NAS as the ground-truth for all data, using something like SMB/NFS instead of SyncThing.
Oh interesting I hadn’t considered this. I guess the only trouble would be if I am out with my laptop and I have no internet connection but this seems like a good tradeoff for simplicity
Beware of interactions with git and syncthing. It's fine for straightforward repos where all you do is commit but as soon as you start doing more complicated branching and re-basing you will start to generate lots of `sync-conflict` files. I haven't really found a reliable way around this so I've decided to just manually rsync from my desktop onto my laptop when I want to work remotely (or more recently, SSH into my desktop directly instead and work off that).
Apple's Time Machine still backs up your Mac even when you don't have access to the backup target (USB isn't connected, network isn't available). It just stores the backup information on your local storage and then transfers it over to the actual backup target when it's back online. Don't know if there's similar solutions for non-macOS systems though.
What the author describes here is hard because it's "simple."
What's simple because it's "hard" is replacing parts 2 & 3 with a network appliance like TrueNAS running a zfs pool that syncs to backblaze every night. Yeah you have to learn a bit but it won't fall in weird ways like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice. My 2¢
> like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice
I think the author is having trouble because he is conflating concepts and roles that should be distinct. Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity.
So he's got sync but he's missing some sort of rotating snapshot system which would solve the stated concern of guarding against syncthing replicating corrupted data. Such automated snapshots can then be used as the source to feed the backup pipeline.
That hard drive doesn't make a good backup because it seems that it is always online. You need an offline backup that you manually plug in to run the job once every so often.
He's also making this more difficult than it needs to be by insisting that the backup drive be compatible with windows. Plug the drives for both snapshots and backups into a linux box, format them with a modern filesystem, and get on with life.
Sync is its own clusterfuck and I have yet to arrive at a satisfactory solution myself despite wasting inordinate amounts of time on it. IMO you either go with a network share or you make due with the "least bad" option of syncthing. Personally I've more or less settled on sshfs at this point not because it's particularly good but because it works well enough and doesn't add any additional complexity.
Personally I use btrfs snapshots on all my devices, those get streamed across the network to a NAS, and the contents of the NAS are periodically (every few months) stuffed into borgbackup on redundant offline devices. Aside from sync the other problem you'll run into if you're a data hoarder is how to split backups across multiple drives once you exceed a few TB. Because external drives only get so large but the NAS will inevitably keep ballooning.
I agree I think one of the main things I learned from all this was that I should probably buy / set up a real NAS. I’ll look into zfs pool thanks for the comment!
I use the wonderful https://github.com/garethgeorge/backrest as a web UI around restic.
Every night, I back the data up to a Hetzner storage box https://www.hetzner.com/storage/storage-box/ which is only ~$3.50/mo USD for 1TB of data.
I have three "tiers" of data for myself:
1. Important things I care about: these go into a folder that gets backed up to Hetzner (examples include photos in Immich, code pushed to Forgejo, etc)
2. Smaller files I sort-of care about: these usually just go into Proton Drive (examples include design documents)
3. Large files I don't care about losing: these go into a folder on a hard drive that is not backed up (examples include media)
This took me about a day to set up and I just check in on it every once in a while. Once a year I will do a practice disaster recovery run.
Sure it's bothersome to use the sneakernet once a month (manually copy data to USB stick and stash it somewhere), but when that ransomware hits (can happen on personal devices also), it literally can't access the offline copy, and try to encrypt that one also.
Immutable backup solutions are code and thus eventually defeatable from the remote attackers point of view. Offline requires physical access, and maybe a big enough wrench to get the USB stick's disk encryption PIN out of you.
Surrendering just a little bit of absolute control over the physical media can dramatically increase the likelihood that it survives whatever apocalyptic event. I don't know that I've ever heard of tapes being stolen from Iron Mountain. Even if this happened, what are the odds that they would get all the tapes. Your tapes? It probably looks like the warehouse from Raiders of the Lost Ark in there.
Doesn't that depend entirely on the storage format? For example if you use thunderbird as a client it has database files that you can copy. Since you say that one of your devices is running arch you should check out some of the email options in the repos. There are at least a few of options that can act as a middleman to bridge between your external accounts and your clients.
> So having a plan to roll back to a stable state might be helfpul. I installed and set up Timeshift for this
Snapshots make this trivial. Particularly in the case of btrfs if you configure your bootloader to always use the default subvolume then rollback becomes just `btrfs subvolme set-default`.
Then you only need to backup the NAS.
https://support.apple.com/en-us/102154
What's simple because it's "hard" is replacing parts 2 & 3 with a network appliance like TrueNAS running a zfs pool that syncs to backblaze every night. Yeah you have to learn a bit but it won't fall in weird ways like the hard drive part here will just fail to mount one night and not back things up for 3 months until you notice. My 2¢
I think the author is having trouble because he is conflating concepts and roles that should be distinct. Sync, rotating snapshots, and deduplicated backups need to be kept entirely separate if you want any hope of maintaining your sanity.
So he's got sync but he's missing some sort of rotating snapshot system which would solve the stated concern of guarding against syncthing replicating corrupted data. Such automated snapshots can then be used as the source to feed the backup pipeline.
That hard drive doesn't make a good backup because it seems that it is always online. You need an offline backup that you manually plug in to run the job once every so often.
He's also making this more difficult than it needs to be by insisting that the backup drive be compatible with windows. Plug the drives for both snapshots and backups into a linux box, format them with a modern filesystem, and get on with life.
Sync is its own clusterfuck and I have yet to arrive at a satisfactory solution myself despite wasting inordinate amounts of time on it. IMO you either go with a network share or you make due with the "least bad" option of syncthing. Personally I've more or less settled on sshfs at this point not because it's particularly good but because it works well enough and doesn't add any additional complexity.
Personally I use btrfs snapshots on all my devices, those get streamed across the network to a NAS, and the contents of the NAS are periodically (every few months) stuffed into borgbackup on redundant offline devices. Aside from sync the other problem you'll run into if you're a data hoarder is how to split backups across multiple drives once you exceed a few TB. Because external drives only get so large but the NAS will inevitably keep ballooning.
Ntfs equivalent of Chmod/chown is a "go have a long lunch" type of operation.