What is the most efficient way to move a large number of files that reside in a single directory?

Question

CentOS 5.x

I apologize if this is a repeat question. I've seen a lot of similar questions (regarding deleting files) but not exactly the same scenario.

I have a directory containing hundreds of thousands of files (possibly over a million) and as a short-term fix to a different issue, I need to move these files to another location.

For the purpose of discussion, let's say the these files originally reside in /home/foo/bulk/ and I want to move them to /home/foo2/bulk2/

If I try mv /home/foo/bulk/* /home/foo2/bulk2/ I get a "too many arguments" error.

Mr. Google tells me that an alternative for deleting files in bulk would be to run find. Something like: find . -name "*.pdf" -maxdepth 1 -print0 | xargs -0 rm

That would be fine if I was deleting stuff but in this case I want to move the files... If I type something like find . -name "*" -maxdepth 1 -print0 | xargs -0 mv /home/foo2/bulk2/ bash complains about the file not being a directory.

What's the best command to use here for moving the files in bulk from one directory to another?

Solution 1

Taking advantage of GNU mv's -t option to specify the target directory, instead of relying on the last argument:

find . -name "*" -maxdepth 1 -exec mv -t /home/foo2/bulk2 {} +

If you were on a system without the option, you could use an intermediate shell to get the arguments in the right order (find … -exec … + doesn't support putting extra arguments after the list of files).

find . -name "*" -maxdepth 1 -exec sh -c 'mv "$@" "$0"' /home/foo2/bulk2 {} +

Solution 2

Just for variety, I'm fond of using cpio for some cases like this.

find tmp |cpio -v  -p --make-directories --sparse tmp2

Comments

[pabouk - Ukraine stay strong]

If the source and destination are on the same file system then this approach could be very inefficient (unnecessary copying). Also it is always better to use -print0 and -0 options.

[Batfan]

You can use find ... -exec sh -c 'blah "$@" blah' sh {} +" (the "shell trick") to deal with issues around argument ordering. There are several examples of this in the findutils (info) documentation; search for "sh -c".

[David Beauchemin]

I will add to the solution: do it from the directory /home/foo/bulk/.