rsync useful w/ encrypted files?

Question

Is rsync efficient for transferring encrypted files? More specifically:

• I encrypt 'x' with my public key and call the result 'y'.
• I rsync 'y' to my backup server.
• 'x' changes slightly
• I encrypt the modified 'x' and rsync the modified 'y' to my backup server.

Is this efficient? I know a small change in 'x' yields a large change in 'y', but is the change localized? Or has 'y' changed so thoroughly that rsync is not much better than scp?

I currently backup my "critical" files by tarring/bzipping them nightly, then encrypting the .tar.bz file and rsync'ing it to my backup server.

Many of the individual files don't change, but, of course, the tar file changes if even one of the files change.

Is this efficient? Should I be encrypting and backing up each file individually? That way, unchanged files will take no time to rsync.

Edit to add more information:

• The source is my home machine. I own it and consider it secure.

• The target is a colocated server. It's mine, but the colocation company could see any/all network traffic and anything on that machine's hard drive.

• The source has very little free space. I don't want to halve its space by storing an encrypted backup on it.

• The target has a lot of space.

Any thoughts based on this new info?

Solution 1

See this answer Backing up 80G hard drive 1G per day regarding rsyncrypto as discussed at Linux.com

Also offsite backup using rsync and aes

Solution 2

First of all, congratulations for taking backups seriously.

Looked at purely from an rsync-transfer-efficiency point of view, you would be a lot better off compressing and encrypting individual files and then creating tarballs of those for which you do no further encryption or compression.

You would be even better off (again, looking only at transfer time) by skipping the tarballs and just rsyncing a hierarchy of encrypted/compressed files.

With any given compressed and encrypted file, I believe that up to the point of change (or really, the first changed block in a block-compressor like bzip2) you will get the same file, but after that point you will never sync up on the old stream; i.e., the rest of the tarball will be different.

OTOH, from a security standpoint, encrypting the tarball will hide filenames and modification patterns, preventing traffic analysis.

I don't see a way using just rsync to get both incremental backups and complete security.

Update:

If you are in control of both ends you could use rsync -e ssh -a ...; this would use an ssh tunnel (which would also supply compression with your ssh_config set up right) and then you might really have the best of both worlds.

And for a final thought, check out rdiff-backup.

Solution 3

If you're using GNU tar (and possibly others) you could create a tar file of all your files to be backed up and keep it stored locally then use "--append" to extend the file with any updates. Because all of the updates to the file are at the end the encryption doesn't hide the unchanged data from rsync so it runs very efficiently. Downside is that it requires a lot of space in both locations that grows with each backup. Upside is that it does sequential backups with a history of changes.

If you're using OS X there's a simple, more efficient, solution to your problem: create an encrypted sparse bundle disk image and rsync that.

The bundle is a directory tree that contains metadata and small segments of encrypted data in individual files where the contents of the disk image are stored. The number of fixed size segments increases as the disk image is filled up (think of them as dynamic hard drive blocks) and only those that contain data that is changed will change on disk allowing rsync to be efficient. I doubt you can have rw access when using public key encryption though.

Security... You're using public key cryptography so if an attacker gets into your box which is being backed up they won't have access to historical data? I can't think of any other reason to use public key over AES. Be aware that rsync allows an attacker to destroy all backups or overwrite them with anything of their choice when they have access to the box to be backed up.

Comments

[nc3b]

"I encrypt 'x' with my public key and call the result 'y'" I think you meant the server's public key.

[Admin]

Well, I'm actually using my own public key, but I suppose I could give my source server a key and use that... same difference, no?

[CodeEnthusiast]

Thanks. Encrypting filenames is important to me and is one of the things that many file backup systems don't do. Of course, I could rsync a file with the salted sha1 of its name or something (or even its contents I suppose), and keep track of salted sha1 to filename mappings in another file (which I'd also need to encrypt and transfer). I'm trying to do this without "keeping state"-- ideally, I don't have to know which files I've backed up (and if they've changed) before. Or, find a really good, reliable way of keeping state.

[CodeEnthusiast]

I have root access to both sides, but the backup server is on someone else's network and belongs to someone else. In other words, the backups on the backup server have to be encrypted. rdiff-backup may be the way to go. Somehow, I sense encrypting many small files (even just to check that they're the same on the other side!) is more painful than encrypting one large file. Wonder if there's a hybrid approach.

[user71]

Thanks for the note on "--append", I never knew that; might be very attractive for those that want history without reverting to a full blown "mercurial / git / subversion" repository.