Back up incremental snapshots to another volume

Question

I'm looking for a way to implement something close to the following backup scheme:

1. Initially, a full image is copied to the backup target.
2. Periodically (e.g. nightly), only blocks that have changed since the last backup are copied to the backup target.
3. Ideally, it should be possible to mount snapshots from any point in time, or delete (flatten) some snapshots selectively.

Can this be implemented using LVM (or some other way)? It needs tracking which blocks have become dirty since the last backup, which I'm not sure LVM can do... I'd rather avoid the permanent performance cost of running on an LVM snapshot at all times.

Solution 1

A newcomer to the scene is Attic https://attic-backup.org/

We used rdiff-backup for a few years as our primary backup method. It was great for what it did, but created tens/hundreds of thousands of small diff files across the course of a year. Most file systems and disks are going to struggle to deal with a million-plus file count. Backing up our 90GB Maildir-based IMAP store would take a few hours. I had to constantly lower the number of weeks/diffs that we would keep for history.

In comparison, once we switched to Attic, nightly backups ran in only 15-20 minutes. That means it's much more viable to keep a year's worth of incremental backups to let you go back to any day within the past year.

Main features that drew me to Attic:

• It doesn't create thousands of files on the destination server
• Deduplication using variable block sizes
• Has built-in compression
• Effective at backing up virtual machine image files
• Efficient over WAN connections

After using it for 6-9 months, I'm fairly confident that it's as stable as rdiff-backup. I still do a multi-generation copy of the Attic directories using removable media, but each removable media has a full copy of the Attic repository.

Solution 2

An alternative to LVM snapshot is to use the datto block driver (aka. dattobd).

From the dattobd GitHub page:

The Datto Block Driver (dattobd) solves the above problems and brings functionality similar to VSS on Windows to a broad range of Linux kernels. Dattobd is an open source Linux kernel module for point-in-time live snapshotting. Dattobd can be loaded onto a running Linux machine (without a reboot) and creates a COW file on the original volume representing any block device at the instant the snapshot is taken. After the first snapshot, the driver tracks incremental changes to the block device and therefore can be used to efficiently update existing backups by copying only the blocks that have changed. Dattobd is a true live-snapshotting system that will leave your root volume running and available, without requiring a reboot.

I tried it and it works as expected on ext4 fs. There is also a working example (with scripts) given in the wiki.

Finally, note that UrBackup has built-in support for snapshot backups on Linux using either LVM or dattobd.

Comments

[tgharold]

Update: There is also a fork of Attic, which sees more development activity at borgbackup.github.io/borgbackup

[Faheem Mitha]

I see from your blog you've been trying borgbackup. What do you think of it?

[tgharold]

borgbackup is working very well for me, I'm using it on a few different Windows and Linux boxes.