Quickest way of moving a large number of files?

Question

I've got to move around 320,000 files, 80,000 folders (only 100 Gb) of data. Some files are > 1GB but most are < 1kB.

I've had a look at Fastest method of copying files but I'm not sure how useful any of these will be, my problem is not about pure transfer rate speeds but really about how quickly Windows can deal with the IO of 320,000 files.

Do you think I will see speed benefits using either xcopy, robocopy, teracopy or fastcopy?

It took us long (12 hours+) to copy them once (using robocopy), and I hate to have to do that again. What can I do to speed it up?

The stuff is on a USB 2.0 External Drive.

Solution 1

Something like robocopy will be your best bet. USB drives can't handle a whole lot of IO to begin with.

I've pushed millions of small files to and from USB drives using robocopy. It takes time, but it gets the job done.

Solution 2

As mrdenny said, Robocopy would be best, partly because of its robustness. The best advise I can offer, provided you're sure about the cleanliness of the files to be moved, is to make sure the antivirus software is disabled while they are being moved. You really don't want the overhead of having all those files being scanned.

Solution 3

You will almost certainly experience better overall performance for the transfer sequence if you first pack up the source files into a single archive (tar, or compressed into zip, etc), then transfer the archive over the network, and then unpack the archive at the destination.

Please don't forget that when you're transferring the archive over the network, you will be better off using ftp (or another file-oriented transfer protocol) than a simple SMB file copy.

Using a process like the above, I've routinely transfer application directories of about 60GB (with about 50,000-75,000 files) between multiple, geographically-separated, datacenters (US, Europe, Asia). The difference between transferring one file at a time, and transferring a compressed archive over FTP is 10-40 times faster.

Rsync can be your friend here too (as it can in many other file transfer scenarios).

If you are open to commercial options, a UDP-based binary streaming solution that can push bits over multiple UDP streams might be of value to you. Take a look at http://www.filecatalyst.com/

Solution 4

Obviously there is a quicker way than all mentioned here. Quicker, but less flexible :-) If you have put the files on a separate partition, you can copy the entire partition to the target disk.

I'm not familiar with any free Windows tool for the job (tool with VSS support would be perfect), but for sure you can boot off Ghost CD, or Partition Magic CD, or boot a Linux standalone CD. In linux you just dd the partition, or ntfsclone if this happens to be an NTFS partition.

View more solutions

Comments

[Michal]

Yes, I know the single tar approach will definitely yield results, but wasn't sure if building the single file would take a significant amount of time.

[Chopper3]

Firewire's good but eSATA can be faster too.

[John Gardeniers]

I don't follow your logic. To create the ISO all those files need to be read and written into the ISO. Then the ISO is read to extract the files and finally write them to the destination. How is that more efficient than reading and writing them once?

[Kiquenet]

any sample code in VBSCript, C#, or another script language ?

[Sun]

@JohnGardeniers Lewis is betting that the time it takes to copy small files across the network have a much greater overhead than the time it takes to put them into an ISO and then transfer that file. It really depends on your particular environment, but as unintuitive as it sounds, it can be faster to transfer one consolidated instead of thousands of tiny files.