Recursively move files of certain type and keep their directory structure

Question

I have a directory which contains multiple sub-directories with mov and jpg files.

/dir/
  /subdir-a/  # contains a-1.jpg, a-2.jpg, a-1.mov
  /subdir-b/  # contains b-1.mov
  /subdir-c/  # contains c-1.jpg
  /subdir-d/  # contains d-1.mov
  ...         # more directories with the same pattern

I need to find a way using command-line tools (on Mac OSX, ideally) to move all the mov files to a new location. However, one requirement is to keep directory structure i.e.:

/dir/
  /subdir-a/  # contains a-1.mov
  /subdir-b/  # contains b-1.mov
              # NOTE: subdir-c isn't copied because it doesn't have mov files 
  /subdir-d/  # contains d-1.mov
  ...

I am familiar with find, grep, and xargs but wasn't sure how to solve this issue. Thank you very much beforehand!

Solution 1

It depends slightly on your O/S and, more particularly, on the facilities in your version of tar and whether you have the command cpio. It also depends a bit on whether you have newlines (in particular) in your file names; most people don't.

Option #1

cd /old-dir
find . -name '*.mov' -print | cpio -pvdumB /new-dir

Option #2

find . -name '*.mov' -print | tar -c -f - -T - |
(cd /new-dir; tar -xf -)

The cpio command has a pass-through (copy) mode which does exactly what you want given a list of file names, one per line, on its standard input.

Some versions of the tar command have an option to read the list of file names, one per line, from standard input; on MacOS X, that option is -T - (where the lone - means 'standard input'). For the first tar command, the option -f - means (in the context of writing an archive with -c, write to standard output); in the second tar command, the -x option means that the -f - means 'read from standard input'.

There may be other options; look at the manual page or help output of tar rather carefully.

This process copies the files rather than moving them. The second half of the operation would be:

find . -name '*.mov' -exec rm -f {} +

Solution 2

ASSERT: No files have newline characters in them. Spaces, however, are AOK.

# TEST FIRST: CREATION OF FOLDERS
find . -type f -iname \*.mov -printf '%h\n' | sort | uniq | xargs -n 1 -d '\n' -I '{}' echo mkdir -vp "/TARGET_FOLDER_ROOT/{}"

# EXECUTE CREATION OF EMPTY TARGET FOLDERS
find . -type f -iname \*.mov -printf '%h\n' | sort | uniq | xargs -n 1 -d '\n' -I '{}' mkdir -vp "/TARGET_FOLDER_ROOT/{}"

# TEST FIRST: REVIEW FILES TO BE MOVED
find . -type f -iname \*.mov -exec echo mv {} /TARGET_FOLDER_ROOT/{} \;

# EXECUTE MOVE FILES
find . -type f -iname \*.mov -exec mv {} /TARGET_FOLDER_ROOT/{} \;

Solution 3

Being large files, if they are on the same file system you don't want to copy them, but just to replicate their directory structure while moving. You can use this function:

# moves a file (or folder) preserving its folder structure (relative to source path)
# usage: move_keep_path source destination
move_keep_path () {
  # create directories up to one level up
  mkdir -p "`dirname "$2"`"
  mv "$1" "$2"
}

Or, adding support to merging existing directories:

# moves a file (or folder) preserving its folder structure (relative to source path)
# usage: move_keep_path source destination
move_keep_path () {
  # create directories up to one level up
  mkdir -p "`dirname "$2"`"
  if [[ -d "$1" && -d "$2" ]]; then
    # merge existing folder
    find "$1" -depth 1 | while read file; do
      # call recursively for all files inside
      mv_merge "$file" "$2/`basename "$file"`"
    done
    # remove after merge
    rmdir "$1"
  else
    # either file or non-existing folder
    mv "$1" "$2"
  fi
}

Solution 4

It is easier to just copy the files like:

cp --parents some/folder/*/*.mov new_folder/

Solution 5

from the parent directory of "dir execute this:

find ./dir -name "*.mov" | xargs tar cif mov.tar

Then cd to the directory you want to move the files to and execute this:

tar xvf /path/to/parent/directory/of"dir"/mov.tar

View more solutions

Comments

[Kevin]

You could, if you have the space, cp the directory and just find newdir ! -name '*.mov' -delete. It ḿay not be the best solution, but as a practical matter it's likely to finish before you get a better answer here.

[jaypal singh]

Can we can use -delete option of find instead of -exec rm?

[Kevin]

You could use -print0 / -0 to take care of the newline problem, and you should use -delete instead of rm -f.

[Jonathan Leffler]

Yes, you probably can use -delete if your variant of find supports it. It wasn't an option in 7th Edition UNIX (and actually still isn't an option in POSIX 2008). I tend to forget about the non-standard options so that what I write has a maximal chance of working on many systems. Similarly with the -print0 / -0 options; I'm not sure whether tar or cpio supports them; specifically, neither cpio nor tar on MacOS X 10.7.2 supports it. That's why I put the 'no newlines in filenames' caveat into the answer. Anything else is fine.

[Jonathan Leffler]

The first command will create files /newlocation/dir/path/under/olddir/file.mov (rather than /newlocation/path/under/olddir/file.mov); you need to start the find command in the top-level directory from which you want to move files.

[jaypal singh]

Thanks Jon, I added couple of tests, didn't really notice any difference. Is it because of the example I am using?

[Jonathan Leffler]

The first command I referred to was the find command, not the various cp commands.

[jaypal singh]

Sorry my bad Jon. I did a test with the find command. It moves the files to new directory. It doesn't move the sub-dir as I had stated in the answer.

[Jonathan Leffler]

Urgh...yes, I got it wrong too. My initial comment is wrong (I was thinking of what happens when names are streamed to cpio, I guess). So, there was a problem; it just wasn't what I diagnosed as the problem. My (half) bad.

[jaypal singh]

No problem Jon. Thanks for all the feedback though. :)

[Arek]

When I add -delete the file is deleted before cpio. How to avoid that?

[Jonathan Leffler]

@arek: I'd separate the copy and delete operations, as in my answer. It's one of those "Doctor, Doctor, it hurts when I …" situations. If you're worried that some .mov files might arrive between the time of the first and second passes, then I'd arrange to run a shell script from the find command, and the script would copy the files from its command line and then delete the originals.

[dgo]

I don't think you've been properly loved for this answer (particularly the cpio part). There are literally 30 or 40 other stackexchange answers that suck compared to this | and show up on Google before this one. Shame on you SEO criteria! Great Answer +1.

[unode]

Can this be expanded to work with wildcards? e.g. move_keep_path /path/to/files/*.txt /dest

[grenix]

For creation of folders wouldnt find . -type d -exec mkdir -vp /TARGET_FOLDER_ROOT/{} \; work also ? Are there generally any traps regardig eg symbolic links?