Why is rsync 2x faster than sftp?
Or above all, is there an even faster way to transfer (such huge) files over a network?
You can speed up scp with
scp -p -C -o 'CompressionLevel 9' -o 'IPQoS throughput' -c arcfour machine:file .
I saw up to 5 times better speed using scp but of course external factors play a big role here.
Related videos on Youtube
kmario23
Journeyman in Python, NumPy/CuPy/SciPy, C/C++, Fortran, (Multi) Linear Algebra, Computer Vision, Multimodal Learning, Deep Learning, PyTorch, Tensors, TensorFlow, Matplotlib, LaTeX, and Vim. Above all, I see Gaussians everywhere!!! Übung macht den Meister Nice Reads: A delightful NumPy guide for Scientific Computing Python PEP A fast-paced primer on Matplotlib Git Ready Hangouts: Python-Chatroom, C++Chatroom Support: If you're a kind Samaritan and want to support me, please do so, for which I would eternally be thankful! paypal.me/kmario23
Updated on September 18, 2022Comments
-
kmario23 over 1 year
I'm transferring huge files, over a network to a destination server, totaling ~ 67 GB with individual zip files of size ~ 7GB. I tried copying to the destination using both
sftp
andrsync
.rsync
performs (2x) faster copying thansftp
.sftp
was achieving around 700 kbps whilersync
transfers the data at a rate north of 1.4 Mbps. I know that SFTP and SCP uses the same SSH connection for transferring files. So, does this implyrsync
is much (2x) faster than SCP which uses SSH under the hood?Or above all, is there an even faster way to transfer (such huge) files over a network?
P.S.: I'm transferring completely new data to the destination server with the following
rsync
command:rsync --info=progress2 -az source_folder username@destination_server:/location/
-
muru over 6 yearsIs rsync connecting using SSH? Are you just updating data or is this transferring entirely new data? What options did you use for rsync?
-
kmario23 over 6 years@muru please see my updated question!
-
muru over 6 yearsSo you're using compression
-z
. Try without it. -
kmario23 over 6 years@muru On average, the transfer speed is same whether I use compression
-z
or not :( -
Admin over 6 yearsDo you mean
sftp
orscp
?scp
is known to be slower because it requires more overhead.
-
-
kmario23 about 5 yearsI'm getting
command-line line 0: Unsupported option "compressionlevel"
andUnknown cipher type 'arcfour'
errors while trying your suggestion -
kmario23 about 5 yearsAlso, I found exactly same answer here: superuser.com/a/787479 . Not whether this answer was copied from this link