sftp - how to only copy files from folder that don't exist in destination folder

Question

I am wondering if it's possible to get files with sftp, but prevent it from re-downloading files that already exist in the destination folder?

Solution 1

sftp has limited capabilities. Nonetheless, the get command has an option which may do the trick: get -a completes partial downloads, so if a file is already present on the client and is at least as large as the file on the server, it won't be downloaded. If the file is present but shorter, the end of the file will be transferred, which makes sense if the local file is the product of an interrupted download.

The easiest way to do complex things over SFTP is to use SSHFS. SSHFS is a filesystem that uses SFTP to make a remote filesystem appear as a local filessytem. On the client, SSHFS requires FUSE, which is available on most modern unices. On the server, SSHFS requires SFTP; if the server allows SFTP then you can use SSHFS with it.

mkdir server
sshfs server.example.com:/ server
rsync -a server/remote/path /local/path/
fusermount -u server

Note that rsync over SSHFS can't take advantage of the delta transfer algorithm, because it's unable to compute partial checksums on the remote side. That's irrelevant for a one-time download but wasteful if you're synchronizing files that have been modified. For efficient synchronization of modified files, use rsync -a server:/remote/path /local/path/, but this requires SSH shell access, not just SFTP access. The shell access can be restricted to the rsync command though.

Solution 2

You can use the -a flag for the get command:

$ sftp example.com
sftp> get -a hello.txt

You can specify * to download the entire directory of course.

However, there are a few caveats.

• This doesn't work on OS X. It may not work on other systems. I've tested it on Ubuntu 15.04.
• The option is meant to resume interrupted downloads. If your file has changed, the result wil be a corrupted download.

A better option would be using rsync:

rsync example.com:hello.txt

This will always give you the file as it is on the server, only transfering the differences between the local and the remote copy if they differ, and it's universally available.