Transfer a file over a unstable SSH connection

Question

I'm trying to upload some big files (around 10GB) with a slow upload speed (200kb/s) on a often disconnected SSH connection (due to poor network conditions).

I'm trying to use scp, but if there is a best way over SSH, I'm ok with it.

What is the best way to do it ?

I've tried to split it up in several parts using split, but it's not really efficient as it require a lot of manual work before and after it is transfered.

Solution

Use rsync with the --partial option

rsync -av --partial sourcedir user@desthost:/destinationdir

The --partial will keep partially transferred files. When you resume the rsync transfer after a ssh broken connection, partially transferred files will start resuming from the point where the ssh connection was lost, and also successfully transferred files will not be transferred again.

Also consider passing in the -z option if you believe the file(s) you are transferring can be compressed significantly; for example, log files comprising of repeated text.

Comments

[Alen Milakovic]

rsync is better. It is restartable - meaning you can continue where you left off if your connection disconnects. You'll probably want to use the --partial flag. And by default it runs over ssh. You could combine with an aggressive compression program like 7z, but you'll need plenty of memory, and it will still take a while.

[blue112]

I've tried with --partial, but it just stars over...

[Alen Milakovic]

"stars over"? You mean starts over? These are the flags I use: rsync -abvz --partial. Can you show an example session?

[blue112]

@FaheemMitha I've tried to reproduce it, but it seems to be working now...

[ImranRazaKhan]

rsync -LvzP remoteserver:path/to/file localfile