Why is FileZilla SFTP file transfer max capped at 1.3MiB/sec instead of saturating available bandwidth? rsync and WinSCP are even slower

Question

I'm downloading from a server and downloads are maxing out at 1.3MiB/second with FileZilla but I can start concurrent downloads and they will download at 1.3MiB/second also. So why can't I download just one file at faster than 1.3MB/s and get closer to saturate available bandwidth (~6+MB/s)?

I know that I can use some other SFTP client that supports segmented downloads such as lftp, know of other good ones that are open source?

But I still want to know what is it that limits downloading one file to just 1.3MB/s, is it some technical limitation with TCP and buffers etc or some configuration issue? I checked and for sure there's no traffic throttling enabled at all for FileZilla.

Also I tried rsync and it was worse than FileZilla/SFTP. I also tried WinSCP and it was the slowest regardless of method SCP/SFTP. So at 1.3MB/s constant transfer FileZilla is pretty good compared to the other methods of transfer.

If someone has a good explanation of why transfers peak at 1.3MB/s I'd really like to know, and if its possible to increase this without resorting to using segmented downloading. Server is running OpenSSH 6.7p1 (Debian) client is FileZilla on Windows.

UPDATE: In response to Martin's information (see his answer below) I am adding that ping is 180ms to 190ms pretty constant between server and client that is downloading. Also cpu usage is very low, 2% to 8% max. I tried with latest version winscp 5.73 and with sftp mode I got 555kb/s and about 805kb/s max with scp mode. Whereas if I start a secondary concurrent transfer in Filezilla I get constant 1.3MiB/s for it also.

So could the 180ms delay to the server be a mathematically limiting factor as Martin and Michael touched on a bit? Or could there still be something else to blame such that I can improve the throughput? If not, I'd appreciate if anyone knows any other (like lftp but runs well on Windows) open source downloader that is secure and supports segmented downloading.

Solution 1

There are three common factors that affect a transfer speed:

• Bandwidth – An obvious factor that's apparently not your trouble.

• Network delay/latency – The SFTP is packet oriented-protocol. When downloading, the SFTP client sends a "read" request to the SFTP server, waits for a response, appends returned data to a local file; and repeats, until the end of the file.

  Even if your connection is fast, if the server is far away (or slow), it takes a time for the data to arrive back. If the client spends this time uselessly waiting, your transfer speed will be low.

  Most SFTP clients (including FileZilla and WinSCP) overcome the problem by both requesting a large chunk of the file in each single "read" request and by sending (queuing) multiple "read" requests without waiting for a response to previous. For example WinSCP can request up to 32 chunks for 32 KB each at once, totaling 1 MB (these are defaults). But if there's a big discrepancy between the bandwidth and the network delay, even that 1 MB can be too small to saturate the bandwidth.

  An underlying TCP protocol can suffer a similar problem. So it's not only how the actual SFTP client is efficient, but also how an underlying TCP layer is efficient.

  See also Bandwidth-delay product on Wikipedia.

  I do not think this is your trouble either, at least if you have used the latest version of WinSCP for the tests. There have been some improvements in the recent releases, which allow WinSCP to utilize high-latency connections as efficiently as FileZilla.

• CPU – The SFTP, being encrypted, is CPU intensive. If you have a relatively slow CPU, comparing to a large bandwidth, the transfer can be capped by your CPU being unable to encrypt (or decrypt in case of the download) the data as fast as your network is capable of transferring them.

  Common SFTP clients cannot distribute the encryption/decryption among CPU cores, so it's actually a capacity of a single CPU core that limits the transfer speed.

  Use the Windows Task manager to see, if one of the cores is utilized to its maximum during the transfer.

---

Part of this answer come from WinSCP article File transfer speed is very low. WinSCP does not utilize all available bandwidth. How do I improve the transfer speed?

Solution 2

I had this issue as well.

I Used task manager to set the priority to high.

Now I get up to 5 MiB/s

Comments

[Michael Hampton]

Look up "bandwidth-delay product".

[Dan Pritts]

HPN-SSH + bdp is the answer. Also see fasterdata.es.net

[Håkan Lindqvist]

1.3 MB/s is not a whole lot, you'd have to either already be under load for other reasons or be on the extreme low end for CPU to be a limiting factor already with that little throughput. Without knowing all the details, I think BDP seems like the most likely candidate.

[htfree]

I think it helped me a bit also, I also switched to FTPS mode instead of ssh sftp which helps also seems.