resume file copy linux

Question

How do I resume a copy of a large file in linux? I have a huge file (serveral gigabyes) partially copied to a network drive, and it took a long time, and it was mostly done before the copy operation stopped due to a network problem that is now fixed. How do I resume the file copy. I don't want an inefficient script, and ecp didn't work (it doesn't seem to work for large files).

Solution 1

If you know you simply need to append to the local file and do not want to use rsync (which could potentially take a long time calculating checksums), you can use curl. For example, if you have a large file on a slow removable USB stick mounted at /media/CORSAIR/somefile.dat and only half of it is in the current directory, to resume:

curl -C - -O "file:///media/CORSAIR/somefile.dat"

Solution 2

I would try rsync -a /from/file /dest/file.

Solution 3

The command you want is going to be something like

rsync -v --append /path/to/afile /mnt/server/dest/afile

unless you can access the server over ssh and run rsync that way, in that case the command FlyingFish gave is best.

Solution 4

Yes rsync is the way to go. For me, we've transferred 100+GB data over rsync+ssh. If you're looking for a true backup copy, make sure that you use the -a (archive) option to preserve file attributes (times, owners, perms, etc.)

host1> rsync -aP file user@host2:/path/to/new/dir/

It's also useful for copying large files that may change during the course of a migration. You can pre-load the data to the destination and once ready for final copy, do it again but only for a fraction of the time. You can save on actual downtime by using rsync to it's full potential.

P.S. Using v (verbose) can slow down a transfer of many files.

View more solutions

Comments

[Mikeage]

and while we're talking about rsync, note that you can copy directly to any remote machine [supporting SSH] using rsync -az -e ssh /from/file [email protected]:/full/dest/path

[Justin]

What you said about rsync is false. man rsync, read what the --append option does.

[Justin]

wow. running rsync like that without any options will delete /dest/file and retransmit the entire thing. How did this get voted up?

[Malvineous]

How does preserving file attributes (-a) help? Should this be --append?

[Raouf Rahiche]

Thanks for the tip, I had to use this tip because I was copying from a Windows SMB server (so no rsync) when the slow connection was lost, and I didn't want to download the 500MB from the beginning.

[Matt Alexander]

Didn't know about --append. Cool!