How can I speed up my Duplicity backup?

Question

I need to perform onsite backups of hundreds of gigabytes from a few Xen VMs to some storage available on a dedicated server in the same network, with a Gigabit connection. The data is mostly MySQL data - I use Percona XtraDB Cluster - backed up locally on the servers with Xtrabackup, so I guess this data should be highly compressable.

At the moment I am using duplicity 0.6.08b with encryption with passphrase (I am not using keys), since I also rsync the backup volumes created with duplicity to some offsite storage. Compression level is currently 6 and volsize is 250. Backups take longer than a day, which is the reason I am looking for recommended duplicity settings that would result in speedy backups to the local network-shared storage without taking too much space.

Any idea?

Solution

You say in a comment that you are seeing about 50 MB/s throughput on those backups.

50 MB/s is on the order of what you can expect for semi-random disk throughput with single spinning-rust disks (i.e., not mirrored or striped RAID to allow reads to be spread across disks to increase throughput). Note that some RAID configurations effectively limit even the best-case throughput to that of the slowest drive. Yes, many HDDs are rated at up to ~200 MB/s, but keep in mind that those figures are best-case sequential access numbers. 50 MB/s is also about 400 Mbit/s, which with some fudging for IP overhead etc. comes out to 500-600 Mbit/s on the network wire, so while you aren't saturating the gigabit link with only that, you are getting fairly close to collisions-likely territory.

You don't give any numbers for CPU utilization while backups are running except saying that you "have three hypervisors with a bunch of VMs on each, more or less busy". But copying data and compressing it isn't terribly CPU-intensive, and if while the backups are running you have any CPU time to spare, then you are not CPU-bound. The only way to actually answer this question is to figure out what factor is limiting the throughput and then focus your efforts there.

My guess would be that you are I/O bound, either on reads or writes, and that you might be network bound. You talk about a dedicated backup storage server with a gigabit Ethernet connection, but you don't say anything about the nature of that connection. Is the for-backups network connection between the physical hosts shared or dedicated? (A separate physical network connecting each of the HVs to the backup server would be acceptable, if only one VM or HV pushes backup data at any one time.)

If the physical network connection to the backup server is shared with other network traffic, you could move to a dedicated connection architecture. How much benefit you get from this depends very much on where the data is compressed and how many collisions you really are seeing currently, but if you do this and nothing else you might be able to double the network throughput and thus, if you are network bound, cut backup times in half.

If you are I/O bound on the reads and/or writes, then moving to a mirrored or striped setup that allows disk I/O to be spread out to multiple disks might help increase throughput; it would increase the total disk bus throughput. Of course, that comes with its own disadvantages. Depending on how much data you push at any one time, adding more fast disk cache to the backup storage server might also help, but my suspicion would be that if you are I/O bound, it's on the read side because the writes are probably more-or-less sequential, in which case adding cache won't help you much.

You could also consider either moving to a file system on the VMs or HVs, and/or on the backup storage server, that does on-the-fly compression of the data as written to the disk, or enabling such compression if it is supported. It will cost CPU time, but increases the effective disk data transfer rate because less data has to be moved to and from the physical platters for the same amount of userspace data stored. Whether or not that would be a net gain in any one situation is basically a coin toss, and needs to be evaluated on a case-by-case basis, but it is certainly one possibility for the situation where you are I/O bound particularly if the data is highly compressible to begin with. Even if the data can only be compressed 20% (equivalent to a compression ratio of 1.25:1, and definitely achievable with for example natural language text; for comparison, ZFS with gzip-9 compression gives me 1.20:1 compression on a sampling of Internet web sites, including images), those same 50 MB/s platter transfer rates suddenly give you over 60 MB/s useful data transferred, assuming the host CPU can keep up with the compression and decompression. Note that encrypted data is supposed to compress extremely poorly, as it should resemble random noise; you'd generally compress before encryption, if you plan to encrypt the data, in which case file-system-level compression on the encrypted side won't do you any good.

Comments

[EEAA]

This is un-answerable. It depends entirely on the nature of your implemention, and that of your data.

[user]

Are your backup processes currently I/O bound on the VM, CPU bound, network bound or I/O bound on the backup storage server side? To illustrate: adjusting the compression ratio won't do anything if you are I/O bound on the reads (IMO, you're much more likely to be read I/O bound than CPU bound), but might help if you are network or store bound and have plenty of CPU to spare.

[Vito Botta]

I think the problem is with both CPU and disks due to the fact that several VMs share these resources. Basically we have three hypervisors with a bunch of VMs on each, more or less busy. On each HV we have a node for the Percona MySQL cluster -these are the nodes for which I want to speed up backups. Disks are also not very fast from what I can see... but I need to ask the person who configured the HVs to know more details on the configuration. As for the network, when duplicity is running and volumes are copied I see a transfer rate of ~50MB/sec.