Backup ZFS pool using rsync

Question

I currently have a FreeNAS box for storing my personal files. I'd like to have an offsite backup, but I'm not willing to spend the money for a second computer capable of running ZFS properly. Therefore I was planning to take the remote backups using rsync.

I would like all the files in the backup to be consistent, which I thought I could do by taking a recursive snapshot first and then transferring that using rsync. It turns out however that a separate snapshot is taken for each dataset.

Now I'm wondering if there is any way to view a recursive snapshot, including all the datasets, or whether there is some other recommended way to rsync an entire zpool. I don't think simply symlinking to the .zfs folders in the datasets will work as I'd like rsync to keep any symlinks that are present in the datasets themselves.

Based on the comments I received, I think some details on my desired configuration are in place. I'm looking to have a NAS at home that I can comfortably put data on, knowing that it's unlikely that I'll ever lose it. For me this means having multiple copies on-site, multiple copies offsite, an offline copy in case things go really bad, periodic snapshots of the data in case of accidental deletion and a means to prevent data errors (e.g. bit rot). The less likely the event is to occur, the more relaxed I am with not having multiple copies of the data after a catastrophe and the less I care about snapshots. Also I care about old data more than I care about new data as I usually have a copy on another device. Finally I should note most files do not get updated too often. Most of the transfers will be new files.

My previous setup was a set of two Raspberry Pi's with attached 4TB external hard drives. I lost trust in this strategy, but had the hardware readily available. After some research it seemed that the only way to prevent errors from sneaking in over time was to go with a checksumming file system such as ZFS combined with server grade components such as ECC RAM and a UPS. For my local copy I went this route. I use 2x4TB disks in mirror and make regular snapshots here.

This machine should cover all cases except for the offsite and offline backups. Since I most likely won't need these backups, I'm not willing to invest too much in it. I therefore figured I could go with the Raspberry Pi's and external disks I already had lying around. I could make it such that one of the disks is always offline, while the other is receiving the backups. Changing the disks at regular intervals would then allow me to have an offline backup of my older data.

The straightforward route would be to use zfs send and receive to two pools, one on each disk. The Raspberry Pi, combined with the USB connection to the hard drive, would however not provide zfs (or any filesystem for that matter) a very reliable environment to operate in. Therefore I'm expecting errors to occur fairly regularly in this setup. Since I'll only be using one disk, zfs would not have any reliable means to recover from failure.

That is the reason I would like to go with ext3 or ext4 combined with rsync. Sure, some bad bits might be written to disk. In case of metadata, there are tools to fix most of these issues. In case of data blocks, that would result in the loss of a single file. Also, the file could be recovered using rsync -c as that would find an incorrect checksum and would transfer the file again from the known-good copy on the local machine. Given the less than ideal hardware, this seems like the best solution possible.

That is my reasoning for using rsync, which led me to the original question of how to rsync a recursive zfs snapshot. If I did not address any of your advice please let me know as I am really open to alternatives. I just do not currently see how they provide any advantage for me.

Solution 1

You seem pretty set on using rsync and a RaspberryPi, so here's another answer with a bit of a brain dump that will hopefully help you come to a solution.

---

Now I'm wondering if there is any way to view a recursive snapshot, including all the datasets, or whether there is some other recommended way to rsync an entire zpool.

Not that I know of... I expect that the recommendations would be along the lines of my other answer.

---

If you were content with simply running rsync on the mounted ZFS pool, then you could either exclude the .zfs directories (if they're visible to you) using rsync --exclude='/.zfs/', or set the snapdir=hidden property.

This causes issues though, as each dataset can be mounted anywhere, and you probably don't want to miss any...

---

You'll want to manage snapshots, and will want to create a new snapshot for "now", back it up, and probably delete it afterwards. Taking this approach (rather than just using the "live" mounted filesystems) will give you a consistent backup of a point in time. It will also ensure that you don't backup any strange hierarchies or miss any filesystems that may be mounted elsewhere.

$ SNAPSHOT_NAME="rsync_$(date +%s)"
$ zfs snapshot -r ${ROOT}@${SNAPSHOT_NAME}
$ # do the backup...
$ zfs destroy -r ${ROOT}@${SNAPSHOT_NAME}

---

Next you'll need to get a full list of datasets that you'd like to back up by running zfs list -Hrt filesystem -o name ${ROOT}. For example I might like to backup my users tree, below is an example:

$ zfs list -Hrt filesystem -o name ell/users
ell/users
ell/users/attie
ell/users/attie/archive
ell/users/attie/dropbox
ell/users/attie/email
ell/users/attie/filing_cabinet
ell/users/attie/home
ell/users/attie/photos
ell/users/attie/junk
ell/users/nobody
ell/users/nobody/downloads
ell/users/nobody/home
ell/users/nobody/photos
ell/users/nobody/scans

This gives you a recursive list of the filesystems that you are interested in...

You may like to skip certain datasets though, and I'd recommend using a property to achieve this - for example rsync:sync=false would prevent syncing that dataset. This is the same approach that I've recently added to syncoid.

The fields below are separated by a tab character.

$ zfs list -Hrt filesystem -o name,rsync:sync ell/users
ell/users   -
ell/users/attie -
ell/users/attie/archive -
ell/users/attie/dropbox -
ell/users/attie/email   -
ell/users/attie/filing_cabinet  -
ell/users/attie/home    -
ell/users/attie/photos  -
ell/users/attie/junk    false
ell/users/nobody    -
ell/users/nobody/downloads  -
ell/users/nobody/home   -
ell/users/nobody/photos -
ell/users/nobody/scans  -

---

You also need to understand that because ZFS datasets can be mounted anywhere (as pointed out above), it is not really okay to think of them as they are presented in the VFS... They are separate entities, and you should handle them as such.

To achieve this, we'll flatten out the filesystem names by replacing any forward slash / with three underscores ___ (or some other delimiter that won't typically appear in a filesystem's name).

$ filesystem="ell/users/attie/archive"
$ echo "${filesystem//\//___}"
ell___users___attie___archive

---

This can all come together into a simple bash script... something like this:

NOTE: I've only briefly tested this... and there should be more error handling.

#!/bin/bash -eu

ROOT="${ZFS_ROOT}"
SNAPSHOT_NAME="rsync_$(date +%s)"
TMP_MNT="$(mktemp -d)"

RSYNC_TARGET="${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_PATH}"

# take the sanpshots
zfs snapshot -r "${ROOT}"@"${SNAPSHOT_NAME}"

# push the changes... mounting each snapshot as we go
zfs list -Hrt filesystem -o name,rsync:sync "${ROOT}" \
    | while read filesystem sync; do
        [ "${sync}" == "false" ] && continue
        echo "Processing ${filesystem}..."

        # make a safe target for us to use... flattening out the ZFS hierarchy
        rsync_target="${RSYNC_TARGET}/${filesystem//\//___}"

        # mount, rsync, umount
        mount -t zfs -o ro "${filesystem}"@"${SNAPSHOT_NAME}" "${TMP_MNT}"
        rsync -avP --exclude="/.zfs/" "${TMP_MNT}/" "${rsync_target}"
        umount "${TMP_MNT}"
    done

# destroy the snapshots
zfs destroy -r "${ROOT}"@"${SNAPSHOT_NAME}"

# double check it's not mounted, and get rid of it
umount "${TMP_MNT}" 2>/dev/null || true
rm -rf "${TMP_MNT}"

Solution 2

I'd highly recommend using zfs send and zfs receive over rsync - it'll be significantly faster and comes with other major benefits (e.g: not missing changes, encryption without needing the keys).

There are storage services who will provide you with the ability to push datasets to them (similar to using a service which supports rsync).

There is even a nice tool - syncoid (part of the sanoid project) - that I would highly recommend. It manages snapshots and allows push or pull operations.

This talk discusses the differences between zfs send/recv and rsync.

---

As a follow up, I have just migrated away from Obnam (which is now retired), and have settled on ZFS with snapshots. I've also just been through the process of investigating off-site storage services, and (for the amount of storage I require) have concluded that building and hosting a machine at a remote location will work out cheaper than using a dedicated storage service before the ~1 year mark... though of course, make your own decision.

---

To address some of your statements:

I'm not willing to spend the money for a second computer capable of running ZFS properly.

It's worth noting that ZFS doesn't have to use ECC RAM, and that you can easily run ZFS on a single disk - it's an off-site backup, so this could well be acceptable to you.

For me building my own machine was about the same price as cloud storage.

As I noted above, I ran some calculations and concluded that building a cheap off-site machine would work out less expensive than paying for one year of "cloud storage" from a service provider... so I paid upfront by building such machines, and within a year I'll start to see savings. "Cloud storage" isn't something that you purchase - you have to keep paying for it.

There are also further benefits - I can offer services and off-site backups to the person hosting my machine... something that in this case they didn't have at all.

Solution 3

I agree with other answers that in general you’re better off using zfs send.

However, if you’re determined to use rsync instead, and all you want is a consistent snapshot of the whole pool, you can do that using recursive zfs snapshot. Although the snapshots appear separately in the output of zfs list for each affected dataset / volume, they are taken at a consistent point in time (i.e. they are "atomic" - all have the same txg, in ZFS-internals lingo).

Comments

[Octaviour]

For me building my own machine was about the same price as cloud storage. Since I like tinkering and since I'll be able to get some extra functionality by building my own machine (Plex, VPN, git repos etc), I decided to go that way. I do realize that ideally I would backup to a similar machine, but that would be too expensive (just as backing up to the cloud is). In case my place burns down or something I'd still like to have two off site backups. No need for versioning or anything, I just want my files in that extreme case. That's why I'd really like to use rsync or similar for this.

[Attie]

I'm not sure I follow your logic. I've updated my answer to clarify some points.

[Octaviour]

I'm not comfortable running ZFS on improper hardware. A single memory error in a bad location can corrupt too much data, potentially without me noticing. Also, I currently have a Raspberry Pi and external hard disk available from another project. Going the rsync road therefore involves zero cost. Also, formatting in ext4 makes it easier to read out the drive in case things go bad.

[Octaviour]

Do you also think zfs send is a good option on very cheap equipment?

[Octaviour]

I understand that a recursive snapshot will be consistent. However, I do not see how I can transfer this snapshot to the remote machine using rsync. Initially I planned on doing a for loop in bash, iterating over all datasets. As the datasets are nested, subsequent runs of rsync will remove data written previously.

[Dan]

rsync will use more resources than zfs send to determine what blocks have changed (more IOs, more memory, more CPU), so I don’t see how using cheap equipment will matter in this case. ZFS usually consumes a lot of RAM because of the cache, but you wouldn’t care about that for a system that’s just receiving backups.

[Attie]

"improper hardware"?... did you read the links? I'll grant you that an RPi isn't going to be a great ZFS host...

[Octaviour]

To be honest I read the article a couple of months ago. I just reread it. To me it seems like no file system can work reliably without ECC RAM. One might always get a bit flip just before writing to disk. zfs not worse, but also not better in this regard. The problem with zfs in this case is that a single bit error can be much harder to recover. Please see my edited, original post for details on how I was planning to provide some resilience using ext.

[Octaviour]

Resources on the remote host are not an issue as its sole purpose will be the backups. The local host will do a bit more work probably, but comes with the option of correcting the remote copy when combined with rsync -c as explained above. To me this is worth the extra computation. Please see the edited, original post for more details.

[Attie]

I've said it before, and I'll say it again... "I'm constantly surprised that anything works at all"

[Octaviour]

Thank you for all the trouble of even writing a number of scripts when you initially did not even agree to my point of view. Iterating over the datasets seems to be the best solution. Is there a specific reason for mounting the snapshot and not just copying the .zfs/snapshot directory?

[Attie]

No problem, I hope it's useful :-)

[Attie]

The main reason for explicitly mounting it somewhere would be the "where is it mounted?" question (as mentioned above)... without inspecting the filesystem's mountpoint property, and dealing with all the issues that entails you cannot be sure where the filesystem appears in the VFS.

[Martin Sugioarto]

Don't forget that there are filesystems with canmount parameter set to off. I cannot tell what happens when you try to mount them. Anyway, I'd still use zfs send for ZFS. It's more reliable for restorable backups than anything else. For recovery I'd use a live CD. Been there and done that.