What is the effect of rsync --block-size on large files?

Question

I want to backup a few terabytes of large files (500mb to 1500mb each) to a backup server (BULK storage) via rsync.

For this I use the following command (updated with suggestions from @Gilles):

rsync --stats --progress --ignore-existing --recursive --exclude ".*" /from/ /to

rsync --stats --progress  --archive --update --exclude ".*" /from/ /to 

My transfer speeds are ~40MB/s. I think this speed is limited by the BULK storage system on the other site of the line. (SSD --> 1000mbits/s --> BULK storage)

As I know I will rsync mainly big files, is it useful to specify the block-size for speeding up my transfer? If it does speed up my transfer, will it be reasonably more?

Solution

--block-size is a parameter of rsync's delta transfer algorithm, i.e. its partial transfer algorithm based on block checksum. Since you are doing local copies, the delta transfer algorithm is not used, so --block-size is ignored. The delta transfer algorithm gives worse results on local copies (that's why it's off by default), because rsync then needs to read all the files on the destination side. It's useful when disk access on either side is significantly faster than the network connection between the two sides.

40MB/s of file I/O is about 400Mb/s of raw disk I/O considering the filesystem overhead. Are you sure that both sides can sustain more than that?

You should usually use rsync -a, to preserve all file modes. For a backup, definitely use -a. Add the -u flag so as not to waste time copying files that haven't been modified (in addition to -a: -u isn't useful if you aren't preserving the times).

Comments

[Gilles 'SO- stop being evil']

@CousinCocaine “Local” means using the filesystem, i.e. when you give a local path to rsync (/path/to/file or relative/path), as opposed to a remote path (host:remote/path over rsh or ssh, or rsync://host/remote/path over rsync). Rsync has no way to know whether a remote host is in a LAN. You may want to pass --whole-file on a very fast LAN.

[quimnuss]

would a write to a NFS be local? That is, would --block-size come into action in that case?

[Gilles 'SO- stop being evil']

@quimnuss Yes. “Local” in this context means that rsync is not doing the network transmission. If the filesystem is doing a network transmission, that's transparent from rsync's point of view, rsync is just interacting with the local filesystem.