How to make scp go faster?

Question

I use the following command to transfer files across my home network (when I have to transfer a lot of files, I usually archive them with the tar utility):

scp -c blowfish [FILE] [USER]@[ADDRESS]:[PATH]

It takes about 25 minutes to send 500 Mb of data to another computer. The average speed is at about 600 KB/s. I think there absolutely must be a way to make the process go faster because I don't believe that I'm able to pull a 1 Gb file comfortably off the Web in less than 10 minutes, but I can't get a 500 Mb file sent across the network in less that time.

Solution

Two ideas:

1. Experiment with a different cipher
2. Use 'on the fly' compression

1. Experiment with a different cipher:

One idea is to test the various ciphers which are available through scp and try to determine which is the fastest for you. (The idea behind this is that it is the encryption of your data that is taking up the time.) Find the available ciphers by running the following:

andrew@athens:~$ ssh -Q cipher
3des-cbc
blowfish-cbc
cast128-cbc
arcfour
arcfour128
arcfour256
aes128-cbc
aes192-cbc
aes256-cbc
[email protected]
aes128-ctr
aes192-ctr
aes256-ctr
[email protected]
[email protected]
[email protected]
andrew@athens:~$ 

This shows available ciphers on a default Xenial Xerus installation and can be added in using the syntax you have already suggested:

scp -c <cipher> [FILE] [USER]@[ADDRESS]:[PATH]

This option is passed directly to ssh hence testing ssh itself for the available ciphers. Hopefully you can find a cipher that will allow an increase in your transfer speed, when you have found this cipher you can place it in ~/.ssh/config rather than typing it in each time...

2. Use 'on the fly' compression:

scp is able to compress your transfers 'on the fly' meaning that the compressions is only used during transfer and will not be seen at either the host or remote ends. Add this in using the -C option:

scp -c <cipher> -C [FILE] [USER]@[ADDRESS]:[PATH]

Here is an example on my own system:

debug1: compress outgoing: raw data 365865, compressed 150118, factor 0.41
debug1: compress incoming: raw data 150118, compressed 365865, factor 2.44

The compression level is the same as the one used for gzip and if you really wish to experiment you can use the 'CompressionLevel' option which varies from 1-9, 1 being the fastest compression, 9 being the slowest with best compression and 6 being the default. This can be set in ~/.ssh/configas follows:

Host *
Compression yes
CompressionLevel 9

Bear in mind that this sets high compression for all scp / ssh transfers, it can be set for individual hosts if you wish...

Use the -v option to investigate the effects of any changes you make. You will note with experimentation that different data will have different compression ratios.

Comments

[storm]

You can use rsync to transfer data much more fast than scp syntax is quite the same rsync -avP [FILE] [USER]@[ADDRESS]:[PATH]

[psusi]

Or since this is on a home network where you presumably don't need to worry about anyone intercepting the traffic, use no encryption at all.