Move files to multiple folders

Question

I've searched for this function, and there are lots of answers, but I haven't been able to find anything that works for me.

I have a folder with about 30,000 files, and I want a terminal command or bash script that will move these files into different folders. About 3000 files for each folder. Some of the filenames have underscores, spaces and hyphens, so the command should allow me to move those files as well as the files without underscores, spaces, and hyphens. The files are .jpgs, but I'd like a command that I can customize for any file type and file amount, in case I need to use it in the future for different formats and quantities.

UPDATE:

I found a bash script that answers my needs. It will move files in a folder into subfolders. It works with any type of file. The number "3001" represents the number of files you want moved to each subfolder. This number can be changed. So if you have a folder with 20000 files and you want the 20000 files divided into batches of 500 and moved to subfolders then you would replace "3001" with "500". You can also modify the script to only move particular file types, e.g.: to move only .jpg files, change "for file in *" to "for jpg_file in *.jpg". Also change "$file" to "$jpg_file".

#!/bin/bash

c=1; d=1; mkdir -p dir_${d}

for file in *
do
        if [ $c -eq 3001 ]
        then
                d=$(( d + 1 )); c=0; mkdir -p dir_${d}
        fi
        mv "$file" dir_${d}/
        c=$(( c + 1 ))
done

Solution 1

I tried all the scripts but none of them worked for some reason, or only partially worked. I found this bash script and tweaked it to look for .jpg files instead of .xml files:

#!/bin/bash

c=1; d=1; mkdir -p dir_${d}

for jpg_file in *.jpg
do
        if [ $c -eq 501 ]
        then
                d=$(( d + 1 )); c=0; mkdir -p dir_${d}
        fi
        mv "$jpg_file" dir_${d}/
        c=$(( c + 1 ))
done

It worked well and fast. Now my only problem is getting it to move any file into folders, not just ".jpg". It's also case sensitive and will not move ".JPG".Any ideas on how to tweak it to move any file, or at the very least any image file?

Solution 2

How about the following:

for letter in {A..Z}; do
   dir=/path/to/sorted/directories/$letter
   mkdir $dir
   mv "${letter}*.jpg" $dir/
done

And then of course the same loop with the lower case letters {a..z}, except that in that case you don't want to create a separate lower case directory. Then the move line changes to:

mv "${letter}*.jpg" ${dir^^}/

Using ${var^^} to create an upper case version of a variable requires Bash 4.0.

Edit: fixed a missing quote. Thanks G-man for pointing that out.

Solution 3

I think something like:

#!/bin/bash
shopt -s nullglob
for i in <directory>; do
    mv *.jpg "$dir"
done

This will move all jpegs to $dir which you will need to set or you will need to make possibly create an array with the directory name(s). You can easily customize this code by switching *.jpg to any different format. You will need to tweak around with this snippet to customize it to how you want it.

UPDATE For this to be a function try this out:

#!/bin/bash
echo "Enter Directory: ";
read dir
mkdir $dir

moveFiles(){
    arg1=$1    
    counter=0;
    while[$counter -ne 3000]; do
          mv <source> *.jpg "$dir"
          counter++;
          if[$counter -e 3000];
          then
               exit;
          fi
     done
}   
moveFiles "$dir"
exit

Again, this code may need tweaking. This is just an example.

Solution 4

This will list files with ls *.jpg, take first 3000 with head -n 3000, make directory for them with another name defined in $FOLDERLIST and move files into it, this loop repeats 10 times

LISTFILESCMD='ls *.jpg' 
FQUANTITY=3000
FOLDERLIST=`seq -w 1 10`

for FOLDER in $FOLDERLIST; do mkdir $FOLDER; mv `$LISTFILESCMD | head -n $FQUANTITY` $FOLDER; done

u may alter parameters, read man ls for sorting options. this is just an example and not optimal solution (calling ls every time, give so many params to mv is not a good idea, it's better to use xargs instead if total length is too large) but I think it will help you to understand what do you really need.

---

test section

 $ ls|wc -w
41978
 $ LISTFILESCMD='ls *.jpg'
 $ FQUANTITY=3000
 $ FOLDERLIST=`seq -w 1 13`
 $ time(for FOLDER in $FOLDERLIST; do mkdir $FOLDER; mv `$LISTFILESCMD | head -n  $FQUANTITY` $FOLDER; done)
real    0m7.396s
user    0m4.543s
sys     0m2.513s
 $ ls|wc -w 
2991
 $ ls -d */
01/  02/  03/  04/  05/  06/  07/  08/  09/  10/  11/  12/  13/
 $ for fldr in `ls -d */`;do ls $fldr|wc -w;done
3000
3000
...

Solution 5

Assuming you have one directory which you want to move files out of.

You can use:

$ mv <source folder/*.jpg <destination folder/>

Edit:

The script below finds all of the files that match the searching criteria defined by file_screen, then executes a while loop to go move the selected files.

New directories are created based on modulo criteria: loop iteration mod files_in_each. If mod returns zero, new directory is created and files will be moved to it.

Parameters to tweak in your runs:

files_in_each=3000  # controls how many files are placed in each directory
directory_to_move="/home/shadowe/test1/test2" # where are the files located
file_screen="jpg"   # only move files that match this criteria

Please tweak as needed.

#!/bin/bash

# basic definitions and calculations
files_in_each=3000
directory_to_move="/home/shadowe/test1/test2"
file_screen="jpg"
folders_created=0
i=0

# while loop through all of the files that match screening criteria
find $directory_to_move/* -maxdepth 1 -type f -name "*${file_screen}" -print0 | sort -n | while IFS= read -r -d '' file; 
do
    # modulo control for creating directories every files in each completion
    create_dir=`expr $i % $files_in_each`
    if [ $create_dir -eq "0" ]
    then
            new_folder=folder$folders_created
            mkdir $new_folder
            echo "created new folder: " $new_folder
            folders_created=$[$folders_created+1]
    fi
    mv "$file" $new_folder
    i=$[$i+1]
done

Small sample results after running:

$ ls test2/
not a picture.txt
$ ls folder0/
one.jpg      one*two.jpg       picture 1.jpg  two-one.jpg
one-two.jpg  picture 1111.jpg  picture *.jpg  two three.jpg

Large sample results:

$ ls folder0 | wc -l
3000
$ ls folder1 | wc -l
2008
$ ls test2 | wc -l
7501
$ ls test2/ | grep "jpg"
$

Script to generate test files:

#!/bin/bash

mkdir test2
touch test2/one.jpg
touch test2/'one-two.jpg'
touch test2/'one*two.jpg'
touch test2/'two-one.jpg'
touch test2/'two three.jpg'
touch test2/'picture 1.jpg'
touch test2/'picture *.jpg'
touch test2/'picture 1111.jpg'
touch test2/'not a picture.txt'
#for large test sample uncomment below
#for i in `seq 1 7500`; do touch test2/test$i.txt; done
#for j in `seq 1 5000`; do touch test2/picture$j.jpg; done

View more solutions

Comments

[ph0t0nix]

Are there any criteria you'd want to use to determine in which folder the files should go (e.g. alphabetical, year from the EXIF data), or simply 10 folders of 3000 files each?

[Bernhard]

I don't get what you really want. What distincts one file from another. Why do you need this?

[Stubot]

@ph0t0nix: Alphabetical would be best, because not all the files I will want to move have the necessary metadata.

[Stubot]

@Bernhard: I don't like manually going through a folder with 30000 files and sorting them into folders because it's tedious an can be slow due to system limitations.

[Stubot]

@Bernhard: I don't understand why you don't understand my question. You've never had a folder with a huge number of files that you didn't want to have to manually sort into smaller batches?

[Bernhard]

@user8547 It is unclear to me based on what you want to divide them. IF it does not matter at all, but you just want them to be 3000 per subdirectory at random, mention that at least. I also don't see the purpose, but that is a different topic.

[Stubot]

@Bernhard: I did mention in the OP that I want to be able to customize the script for any file. Why you're so bothered by this is weird.

[Stubot]

Tx! Can you add a function to the command to automatically create folders? For example move 3000 files into folder001, 3000 files into folder002, etc.

[Stubot]

Tx! Can you add a function to the command to automatically create folders? For example move 3000 files into folder001, 3000 files into folder002, etc

[Simply_Me]

@ryekayo isn't for loop redundant if there is only one dir and mv * is used?

[Matej Vrzala M4]

Yes you are right, let me correct this.

[Stubot]

Could you modify the script to move any files, regardless of having numerical or alphabetical filenames, uppercase or lowercase, and regardless of whether there are spaces, underscores, and hyphens in the filenames?

[Matej Vrzala M4]

@user8547: this snippet is serving as an example for moving files to a given directory. In the moveFile function, you will need to create a counter that will count files to 3000 and then break out of the function. I will look up what can go in there and update it shortly.

[Stubot]

Tx! But it returns the following error: line 9: syntax error near unexpected token `do'

[Simply_Me]

@user8547 check out the updated answer.

[Stubot]

Tx! But it doesn't work right. It created a bunch of folders, but only moved files into some of them. I changed "find test2/*" to "*jpg". Before I changed that it didn't work at all.

[Simply_Me]

@user8547 no, your change is wrong. test2 should be changed to the directory name where you store the files.

[Simply_Me]

@Leben ls will as used above will not capture file names with special characters as described by the author.

[Leben Gleben]

oh, it's probably because of spaces.

[ph0t0nix]

In order to move any file (regardless of extension), simply change the for loop: for file in * (and then also rename any occurence of $jpg_file with $file.

[G-Man Says 'Reinstate Monica']

You seem to have a problem with quotes. In the first code block, you're quoting the *, which strips it of its meaning as a wildcard. In the second code example, you have only one (unbalanced) quote.

[G-Man Says 'Reinstate Monica']

@user8547: ph0t0nix's answer can trivially be extended to create a directory for each digit. And then you could simply do mkdir /path/to/sorted/directories/special; mv *.jpg /path/to/sorted/directories/special after you've moved all the files whose names begin with a letter (or digit).

[Simply_Me]

@user8547 I've modified the script to find only certain type of files (i.e. "jpg" etc.). Please review updated answer and examples that it works.

[Stubot]

@ph0t0nix: Tx! That worked like a charm!

[Simply_Me]

@user8547 make sure it deals with files that have space in them.

[Stubot]

@Simply_Me: What do you mean? The file names should have spaces in them? I tested it out with files that have no spaces, but do have hyphens and underscores, and it worked well.

[Simply_Me]

@user8547 yes, I just checked it, it works.

[Stubot]

Incidentally the number "501" can probably be changed to deal with any number of files. I just tested the bash script with moving batches of 1001 files and 3001 files and it worked. Only strange thing is that it will move some directories within other directories, but I don't mind because I can just move the directories within directories to where I want them to be.