How can I resume a large scp file transfer when using port forwarding?
Solution 1
With some versions of scp (the version on the source computer seems to be the key), just re-executing the scp command will suffice to resume the transfer. But be careful! If your version doesn't support partial transfers, the partial file will simply be overwritten.
The following rsync switches are handy for resuming a broken transfer if scp doesn't support it:
--append append data onto shorter files
--append-verify like --append, but with old data in file checksum
-e, --rsh=COMMAND specify the remote shell to use
--progress show progress during transfer
The command
rsync --append-verify --progress --rsh="ssh -p 4567" me@localhost:/path/to/large/file.gz .
should have the desired effect. Note that the -p
switch must be lowercase for ssh.
Solution 2
Use sftp instead of scp
In your case:
sftp -a -P 4567 me@localhost:/path/to/large/file.gz .
From the sftp man page:
-a Attempt to continue interrupted downloads rather than overwriting existing partial or complete copies of files. If the remote file contents differ from the partial local copy then the resultant file is likely to be corrupt.
Solution 3
rsync uses ssh by default, you may have to specify the exact ssh command using rsync's -e switch. It also has a --partial which should keep the incomplete file around so it can resume the transfer.
chris
Updated on September 18, 2022Comments
-
chris over 1 year
I have a machine a couple of hops away, and I need to set up port forwarding in order to transfer files.
Edit: To be clear, the multiple hops are needed in order to access the remote machine. From my machine, I set up a VPN, where I can access 10.255.x.x - that is the only machine I can connect to via the VPN. Once logged into .x.x, I can then connect to other machines - .y.y being one of those.
From my machine:
ssh -L 4567:localhost:4567 [email protected]
Then from that machine:
ssh -L 4567:localhost:22 [email protected]
I can then
scp -P 4567 me@localhost:/path/to/large/file.gz .
I left this running overnite, only to find that the transfer died at some point.
I've seen a couple of suggestions to use rsync over ssh to resume the transfer, but I'm unclear how to set that up. Is this possible?
-
Dennis about 11 yearsWhat are all these hops for? Wouldn't
scp [email protected]:/path/to/large/file.gz .
achieve exactly the same? Which version of scp (ssh) is installed on client and server? -
chris about 11 years@Dennis: scp does not support resuming a transfer - what you suggest will restart the download.
-
Dennis about 11 yearsSome versions do (my laptop's scp does, my VPS's doesn't). The reason I'm asking is because the hops seems unnecessary to me, and rsyncing without hops will be a lot easier.
-
Alex Allen about 11 yearsThis answer might help you as well: unix.stackexchange.com/a/43097/14084
-
-
chris about 11 yearsThe hops are needed because of the way the network on the destination server is configured. The problem with using rsync over ssh is that I need to use the local port 4567, the file does not exist on the 10.255.x.x server.
-
Dennis about 11 yearsMy mistake. I was assuming that
10.255.y.y
was the client computer. -
Dennis about 11 yearsI've updated my answer.
-
chris about 11 years@ingenous: I added an explanation for the tunnels.
-
dr.dimitru over 8 yearsand what to do if there is no way to use rsync? In my case I have only ftp available on remote host.
-
Dennis over 5 years@Enzo Because, as I said in the first paragraph, not all versions of scp support this. (That may be less of a problem 5 years later.) Also, because I assumed the OP cared more about resuming the download than using a particular tool for it.
-
EnzoR over 5 yearsTheOP is talking explicitly about SCP. You can also switch to HTTP(S) with curl or to FTP(S) and even SFTP with similar results. But the OP is still talking about scp. When the OP will be edited to widen the problem, then your answer will make some sense. At the moment it's offtopic.
-
EnzoR over 5 yearsMoreover, I'd have quoted exactly which scp implemetation supports the resume. I am not aware of any such scp impleemtation. You left your answer in the haze, instead.
-
Dennis over 5 years@Enzo How am I supposed to know which scp implementations support resume? Yours either does or it doesn't. If it does, you can just re-run the command; if it doesn't, you can use rsync instead. Both cases are covered in my answer. If you think it isn't helpful, feel free to downvote it.
-
EnzoR over 5 yearsNo scp implementation, afaik, supports resume as it's a plain "echo" over an SSH channel. SFTP/FTPS/HTTP(S) can support it and most of the opensource implementations support it. You said that "With some versions of scp (the version on the source computer seems to be the key)", so you say you know. While you now say actually don't. The question is about SCP, the answer needs to be about scp. And the only one answers is "you cannot".
-
Dennis over 5 years@Enzo The question is about SCP, the answer needs to be about scp. The question is about an incomplete scp transfer, not about the scp tool. It literally says I've seen a couple of suggestions to use rsync over ssh to resume the transfer, but I'm unclear how to set that up. Is this possible? at the bottom.
-
Dallas Clarke about 2 yearsYou can use
reput
orreget
if you log into the shell.