Why is rsync 2x faster than sftp?

7,711

Or above all, is there an even faster way to transfer (such huge) files over a network?

You can speed up scp with

scp -p -C -o 'CompressionLevel 9' -o 'IPQoS throughput'  -c arcfour machine:file .

I saw up to 5 times better speed using scp but of course external factors play a big role here.

Share:
7,711

Related videos on Youtube

kmario23
Author by

kmario23

Journeyman in Python, NumPy/CuPy/SciPy, C/C++, Fortran, (Multi) Linear Algebra, Computer Vision, Multimodal Learning, Deep Learning, PyTorch, Tensors, TensorFlow, Matplotlib, LaTeX, and Vim. Above all, I see Gaussians everywhere!!! Übung macht den Meister                                     Nice Reads: A delightful NumPy guide for Scientific Computing Python PEP A fast-paced primer on Matplotlib Git Ready Hangouts: Python-Chatroom, C++Chatroom Support: If you're a kind Samaritan and want to support me, please do so, for which I would eternally be thankful! paypal.me/kmario23

Updated on September 18, 2022

Comments

  • kmario23
    kmario23 over 1 year

    I'm transferring huge files, over a network to a destination server, totaling ~ 67 GB with individual zip files of size ~ 7GB. I tried copying to the destination using both sftp and rsync. rsync performs (2x) faster copying than sftp. sftp was achieving around 700 kbps while rsync transfers the data at a rate north of 1.4 Mbps. I know that SFTP and SCP uses the same SSH connection for transferring files. So, does this imply rsync is much (2x) faster than SCP which uses SSH under the hood?

    Or above all, is there an even faster way to transfer (such huge) files over a network?

    P.S.: I'm transferring completely new data to the destination server with the following rsync command:

    rsync --info=progress2 -az source_folder username@destination_server:/location/
    
    • muru
      muru over 6 years
      Is rsync connecting using SSH? Are you just updating data or is this transferring entirely new data? What options did you use for rsync?
    • kmario23
      kmario23 over 6 years
      @muru please see my updated question!
    • muru
      muru over 6 years
      So you're using compression -z. Try without it.
    • kmario23
      kmario23 over 6 years
      @muru On average, the transfer speed is same whether I use compression -z or not :(
    • Admin
      Admin over 6 years
      Do you mean sftp or scp? scp is known to be slower because it requires more overhead.
  • kmario23
    kmario23 about 5 years
    I'm getting command-line line 0: Unsupported option "compressionlevel" and Unknown cipher type 'arcfour' errors while trying your suggestion
  • kmario23
    kmario23 about 5 years
    Also, I found exactly same answer here: superuser.com/a/787479 . Not whether this answer was copied from this link