How to speed up rsync?
Solution 1
For the first sync just use
cp -a /mysourcefolder /mytargetfolder
rsync only adds overhead when the destination is empty.
also.. the -z option is probably killing your performance, you shouldn't be using it if you are not transfering data over a slow link.
Solution 2
If you're using rsync with a fast network or disk to disk in the same machine,
not using compression -z
and using --inplace
speeds it up to the performance of the harddrives or network
compression uses lots of CPU
not using inplace makes the harddrive thrash alot (it uses a temp file before creating the final)
compression & not using inplace is better for doing it over the internet (slow network)
NEW: Be aware of the destination... if there is NTFS "compression" enabled... this severely slows down large files (I'd say 200MB+) rsync almost seems stalled, it's caused by this.
Solution 3
Use the -W
option. This disables delta/diff comparisons. When the file time/sizes differ, rsync copies the whole file.
Also remove the -z
option. This is only useful for compressing network traffic.
Now rsync
should be as fast as cp
.
Solution 4
First - the number of files in this case is going to be a major factor. It's an average size of 3MB each. There's probably an io bottleneck influencing the speed in the OP's case. More here - that's a pretty dry read, but the cover picture is worth it.
So, using rsync to copy to an empty directory? Here are some ways to speed it up:
- No -z - definitely don't use -z as in the OP.
- --no-compress might speed you up. This could have the biggest impact... my test was 13,000 files, total size 200MB, and using rsync 3.1.3. I synced to a different partition on the same internal SSD drive. With --no-compress, I get 18 MBps, and without it I get 15 MBps. cp, by the way, gets 16 MBps. That's a much smaller average file size though. Also - I can't find any documentation for --no-compress. I learned about it from this post on stackexchange.com.
- -W to copy files whole - always use this if you don't want it to compare differences; never mind that the point of rsync is to compare differences and only update the changes.
- -S to handle sparse files well - can't hurt if you don't have sparse files.
- --exclude-from or something similar to exclude files you might not need will cut down the time, but it won't increase your transfer speed.
- It's possible if you send the output to a file like this
rsync -a /source /destination >/somewhere/rsync.out 2>/somewhere/rsync.err
- the first > basically prints a file with all the stuff you would normally see, and the 2> refers to error messages. - Finally, running multiple instances of rsync for different parts of your transfer could be a big help.
My command would be:
rsync -avAXEWSlHh /source /destination --no-compress --info=progress2 --dry-run
If all looked well, I'd delete "--dry-run" and let it go. A, X, and E cover extended attributes and permissions not covered by -a, l is for soft links, H is for hard links, and h is for human readable.
Updating an already synced directory on a USB drive, or the same drive, or over a network, will all require different rsync commands to maximize transfer speed.
Bonus - here's the rsync man page, and if you want to test your hard drive speed, bonnie++ is a good option, and for your network speed, try iperf.
*The post is almost ten years old, but search engines sure like it, and I keep seeing it. It's a good question, and I don't think the top answer to "how to speed up rsync" should be "use cp instead."
Solution 5
You definitely want to give rclone a try. This thing is crazy fast :
$ tree /usr [...] 26105 directories, 293208 files
$ sudo rclone sync /usr /home/fred/temp -P -L --transfers 64
Transferred: 17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ETA 0s Errors: 75 (retrying may help) Checks: 691078 / 691078, 100% Transferred: 345539 / 345539, 100% Elapsed time: 1m50.8s
This is a local copy from and to a LITEONIT LCS-256 (256GB) SSD.
You can add --ignore-checksum on the first run to make it even more faster.
Related videos on Youtube
Jake Wilson
Updated on September 17, 2022Comments
-
Jake Wilson over 1 year
I'm running rsync to sync a directory onto my external USB HDD. It's about 150 gigs of data. 50000+ files I would guess.
It's running it's first sync at the moment, but its copying files at a rate of only 1-5 MB/s. That seems incredibly slow for a USB 2.0 enclosure. There are no other transfers happening on the drive either.
Here are the options I used:
rsync -avz --progress /mysourcefolder /mytargetfolder
I'm running Ubuntu Server 9.10.
-
Admin about 14 yearsare you sure you're getting a USB2 connection? does a (non-rsync) copy or other write operation run at normal speeds? if not, have you tried a copy/other write op with another USB port/cable?
-
Admin about 11 yearsSee also serverfault.com/questions/43014/… - there people also propose using two piped
tar
commands orcpio
. -
Admin over 3 yearsLike @tom-hale pointed out in your case the compression makes no sense, because you copy between local filesystems. Compression only makes sense when you copy between two hosts over a network.
-
-
msanford almost 14 yearsrsync is so called because it's for remote synchronization and is not really appropriate for a locally-connected volume for this very reason.
-
Blaisorblade about 11 yearsIt's supposed to be usable also for local transfers, and it's much more flexible. It's only possibly overkill for the first sync.
-
Speederer almost 11 yearsMinor note:
-z
is only useful for low speed network traffic. If your network is fast enough, it'll slow things down, since you'll be limited by CPU. -
djhworld over 10 yearsThese tips vastly sped up the transfer of my files between two NAS devices, thanks!
-
Jesse the Wind Wanderer over 9 yearsrsync is also a one way sync. Very good for backing up to a server or from a server. However, if you want local TWO-Way sync to a removable drive, you may want to check out csync csync.org/get-it not to be confused with csync2 which is a completely different project.
-
Scott Kramer almost 8 yearsNTFS compression is slowwww on big files
-
GuoLiang Oon almost 7 yearsbut note that according to the man page says for
-W
: "This is the default when both the source and destination are specified as local paths, but only if no batch-writing option is in effect." -
Anthony over 6 yearsI don't see anything about '--inline' on the man page
-
editor over 6 years
rsync -avz --progress /mysourcefolder/ /mytargetfolder
or you'll get a copy ofmysourcefolder
inside ofmytargetfolder
rather than mirroring the contents -
Scott Kramer over 6 yearsIt's '--inplace'
-
oemb1905 over 5 yearsThis answer does not answer the question. The question was about how to optimize rsync - not replace it with the cp command.
-
user2480144 over 4 yearsIn respect of item 7) I could improve a lot more the performance using a traditional HDD as the source, by extending read-ahead buffer using "blockdev --setra 8192 /dev/sdX". So I intend to reduce head seeks.
-
Sammy Guergachi over 4 yearsI've found that S (handling Sparse files) is actually slow. Removing it made it much faster
-
markhahn about 4 years--inplace will only help if you already have a file there, and it's mostly the same as the source. otherwise, rsync defaults to an atomically safe method (destination tmp named, then mv into place.)
-
Chris L. Barnes about 4 years
--no-compress
is not mentioned in the rsync man page. -
Mila Nautikus over 3 years
--transfers 64
is a recipe for file fragmentation (no?) -
logoff over 3 years
-W
was the key in my case (over local network). I went from ~3.5 MB/s to ~35 MB/s. a 10x factor!! -
robertspierre over 3 yearsYou are not explaining anything of what's happening here nor why this is a better solution thatn OP's
-
robertspierre over 3 yearsThe
-W
also worked spectacular for me. 10x factor exactly like logoff. Especially super-useful for big files! -
matson kepson over 3 yearsyou are right @raffaem, i have added some explanation on switches, cheers
-
systemexit almost 3 yearsThe literal text
--no-compress
does not appear in the man page because it falls under the generic--no-OPTION
-
Illya Moskvin over 2 yearsOther answers here already contain good optimizations. This is one I haven't seen mentioned yet. For context, I invoked
rsync
in a "User data" script that runs on AWS EC2 instance startup. The user data script writes to/var/log/cloud-init-output.log
. I was syncing about 25,000 files, so it needed to write 25K lines to the log file. Running rsync silently took about 12 seconds. Running the same command with-v
took 100 seconds. Huge difference! -
Anirudh Ajith about 2 yearsThanks a lot! This gave me a 12x speedup!