How to make a local tar file of a remote directory

Question

This link describes how to copy tarred files to minimise the amount of date sent over the network. I am trying to do something slightly different.

I have a number of remote files on different subdirectory levels:

remote:/directory/subdir1/file1.ext
remote:/directory/subdir1/subsubdir11/file11.ext
remote:/directory/subdir2/subsubdir21/file21.ext

And I have a file that lists all of them:

remote:/directory/allfiles.txt

To copy them most efficiently, on the remote site I could just do

tar zcvf allfiles.tgz `cat /directory/allfiles.txt`

but there is not enough space to do that.

I do have enough storage space on my local disc. Is there a way to tar an incoming stream from a remote server (using scp or ssh for the transfer)?

Something like /localdir$ tar zc - | ssh remote `cat /directory/allfiles.txt` I would guess - but that would only list the remote files on the local host.

Solution 1

You got it almost right, just run the tar on the remote host instead of locally. The command should look something like the following:

ssh remote_host tar cvfz - -T /directory/allfiles.txt > remote_files.tar.gz

Solution 2

If you have enough space on your local disk and your goal is to minimise the amount of date sent over the network maybe it is enough to enable the compression with scp or rsync :

scp -avrC remotehost:/path/to/files/file1 /files/file2 ...  local/destination/path

Of course you can do a little script to loop over each file and do an scp compressed transfer, even without the use of tar. All is more cosy with rsync

rsync -avz --files-from=FILE remotehost:/path/to/files  local/destination/path

You can connect via ssh to the remote host and write there

tar cvzf - -T list_of_filenames | ssh Local_Hostname tar xzf -

References:

• from man scp:

  -C      Compression enable.  Passes the -C flag to ssh(1) to enable compression.
  

• from man rsync

  --files-from=FILE       read list of source-file names from FILE
  -z, --compress          compress file data during the transfer
  --compress-level=NUM    explicitly set compression level
  

Comments

[alle_meije]

Excellent, that's what I was hoping to get! I had thought about this but expected the redirection to be interpreted on the remote machine not the local. I guess the hyphen takes care of that?

[alle_meije]

Thanks for mentioning the --files-from option for rsync, that is handy! The compression option of scp is efficient for network traffic -- would the script you mention mean typing the password many times (or storing it in ~ /.ssh)?

[Hastur]

rsync can be even better of scp (check the options -S for sparse files), moreover it requires the authentication only once, and you can skip files just present. For the scp scripted solution you really will want to use the autmate SSH authentication.

[Alexis Wilke]

You may want to fix "date" with "data" in "[...] the amount of dat[e] sent [...]" — and I think that's the main reason to transform that data to a compressed tarball in the first place...

[Alexis Wilke]

I don't think that the -v is a good idea. You're going to use extra bandwidth for that feature. You could instead verify the results with a tar tvf remote_files.tar.gz once the transfer is done.

[dirk]

If you really have to have to most minimal bits transfered over the network, sure. But really how much more bandwidth is '-v' going to take for showing the file progress? Data send over with ssh is probably also compressed too.