Solving "mv: Argument list too long"?

Question

I have a folder with more than a million files that needs sorting, but I can't really do anything because mv outputs this message all the time

-bash: /bin/mv: Argument list too long

I'm using this command to move extension-less files:

mv -- !(*.jpg|*.png|*.bmp) targetdir/

Solution 1

xargs is the tool for the job. That, or find with -exec … {} +. These tools run a command several times, with as many arguments as can be passed in one go.

Both methods are easier to carry out when the variable argument list is at the end, which isn't the case here: the final argument to mv is the destination. With GNU utilities (i.e. on non-embedded Linux or Cygwin), the -t option to mv is useful, to pass the destination first.

If the file names have no whitespace nor any of \"' and don't start with -¹, then you can simply provide the file names as input to xargs (the echo command is a bash builtin, so it isn't subject to the command line length limit; if you see !: event not found, you need to enable globbing syntax with shopt -s extglob):

echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir --

You can use the -0 option to xargs to use null-delimited input instead of the default quoted format.

printf '%s\0' !(*.jpg|*.png|*.bmp) | xargs -0 mv -t targetdir --

Alternatively, you can generate the list of file names with find. To avoid recursing into subdirectories, use -type d -prune. Since no action is specified for the listed image files, only the other files are moved.

find . -name . -o -type d -prune -o \
       -name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o \
       -exec mv -t targetdir/ {} +

(This includes dot files, unlike the shell wildcard methods.)

If you don't have GNU utilities, you can use an intermediate shell to get the arguments in the right order. This method works on all POSIX systems.

find . -name . -o -type d -prune -o \
       -name '*.jpg' -o -name '*.png' -o -name '*.bmp' -o \
       -exec sh -c 'mv "$@" "$0"' targetdir/ {} +

---

In zsh, you can load the mv builtin:

setopt extended_glob
zmodload zsh/files
mv -- ^*.(jpg|png|bmp) targetdir/

or if you prefer to let mv and other names keep referring to the external commands:

setopt extended_glob
zmodload -Fm zsh/files b:zf_\*
zf_mv -- ^*.(jpg|png|bmp) targetdir/

or with ksh-style globs:

setopt ksh_glob
zmodload -Fm zsh/files b:zf_\*
zf_mv -- !(*.jpg|*.png|*.bmp) targetdir/

Alternatively, using GNU mv and zargs:

autoload -U zargs
setopt extended_glob
zargs -- ./^*.(jpg|png|bmp) -- mv -t targetdir/ --

---

^{¹ with some xargs implementations, file names must also be valid text in the current locale. Some would also consider a file named _ as indicating the end of input (can be avoided with -E '')}

Solution 2

If working with Linux kernel is enough you can simply do

ulimit -S -s unlimited

That will work because Linux kernel included a patch around 10 years ago that changed argument limit to be based on stack size: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318e43fee84fa7b0b90d68bed92d2ba

If you don't want unlimited stack space, you can say e.g.

ulimit -S -s 100000

to limit the stack to 100MB. Note that you need to set stack space to normal stack usage (usually 8 MB) plus the size of the command line you would want to use.

You can query actual limit as follows:

getconf ARG_MAX

that will output the maximum command line length in bytes. For example, Ubuntu defaults set this to 2097152 which means roughly 2 MB. If I run with unlimited stack I get 4611686018427387903 which is exactly 2^62 or about 46000 TB. If your command line exceeds that, I expect you to be able to workaround the issue by yourself.

Note that if you use sudo as it sudo mv *.dat somewhere/. running ulimit cannot fix that issue because sudo resets the stack size before executing the mv for real. To workaround that, you have to start root shell with sudo -s, then do ulimit -S -s unlimited and finally run the command without sudo in the root shell.

Solution 3

Sometimes it's easiest to just write a little script, e.g. in Python:

import glob, shutil

for i in glob.glob('*.jpg'):
  shutil.move(i, 'new_dir/' + i)

Solution 4

The operating system's argument passing limit does not apply to expansions which happen within the shell interpreter. So in addition to using xargs or find, we can simply use a shell loop to break up the processing into individual mv commands:

for x in *; do case "$x" in *.jpg|*.png|*.bmp) ;; *) mv -- "$x" target ;; esac ; done

This uses only POSIX Shell Command Language features and utilities. This one-liner is clearer with indentation, with unnecessary semicolons removed:

for x in *; do
  case "$x" in
    *.jpg|*.png|*.bmp) 
       ;; # nothing
    *) # catch-all case
       mv -- "$x" target
       ;;
  esac
done

Solution 5

Try this:

find currentdir -name '*.*' -exec mv {} targetdir \;

• find: search a folder
• -name: match a desired criteria
• -exec: run the command that follows
• {}: insert the filename found
• \;: mark the end of the exec command

View more solutions

Comments

[codeforester]

Related: Does "argument list too long" restriction apply to shell builtins?

[Hussein]

A quick hack that did but doesn't count as a general solution is: if you know a way to partition all these files (e.g. A???.txt and B???.doc) in more manageable numbers, you could just move all the A files first and the B files next.

[Dominique]

The first two commands returned "-bash: !: event not found" and the next two did not move any files at all. I'm on CentOS 6.5 if you should know

[Gilles 'SO- stop being evil']

@Dominique I used the same globbing syntax that you used in your question. You'll need shopt -s extglob to enable it. I'd missed a step in the find commands, I've fixed them.

[Dominique]

I'm getting this with the find command "find: invalid expression; you have used a binary operator '-o' with nothing before it." I will now try the other ones.

[Score_Under]

You could use something like the original glob in the for-loop to get a closer solution to what's being asked for.

[Score_Under]

Sorry if that was a little cryptic, I was referring to the glob in the question: !(*.jpg|*.png|*.bmp). You could add that to your for-loop by globbing "$origin"/!(*.jpg|*.png|*.bmp) which would avoid the need for the switch used in Kaz's answer and keep the simple body of the for-loop.

[Whitecat]

Awesome Score. I incorporated your comment and updated my answer.

[CivFan]

With more than a million files, this will in turn spawn more than a million mv processes, instead of just the few needed using the POSIX find solution @Gilles posted. In other words, this way results in lots of unnecessary CPU churn.

[Kaz]

@CivFan Another problem is convincing yourself that the modified version is equivalent to the original. It's easy to see that the case statement on the result of * expansion to filter out several extensions is equivalent to the original !(*.jpg|*.png|*.bmp) expression. The find answer is in fact not equivalent; it descends into subdirectories (I don't see a -maxdepth predicate).

[CivFan]

Gilles, for the find commands, why not use the "not" operator, !? It's more explicit and easier to understand than the odd trailing -o. For example, ! -name '*.jpg' -a ! -name '*.png' -a ! -name '*.bmp'

[CivFan]

-name . -o -type d -prune -o protects from descending into sub-directories. -maxdepth is apparently not POSIX compliant, though that's not mentioned in my find man page.

[Kaz]

Rolled back to revision 1. The question doesn't say anything about source or destination variables, so this adds unnecessary cruft to the answer.

[Gilles 'SO- stop being evil']

@CivFan Now work out what it needs to be with the pruning of directories other than .. I don't think it's clearer than using -o. -o is a very common programming idiom: a list of patterns, where the first match is taken.

[sam]

I still get xargs: mv: Argument list too long

[Gilles 'SO- stop being evil']

@sam Ask a new question that contains the exact command you ran.

[Mithril]

I have 80000 Chinese title files, echo !(*.jpg|*.png|*.bmp) | xargs mv -t targetdir successful move 40000+ files.But left 30000+ files with error mv: invalid option -- '▒'

[Gilles 'SO- stop being evil']

@Mithril You have file names with special characters, possibly spaces. As I state in my answer, this solution does not work with such file names. You have to use one of the other 7 solutions I propose.

[Kusalananda]

That's a hack. How would you know what to set the stack limit to? This also affects other processes started in the same session.

[Mikko Rantalainen]

Yeah, it's a hack. Most of the time this kind of hacks are one-off (how often you manually move huge amount of files anyway?). If you are sure that the process is not going to eat all your RAM, you can set ulimit -s unlimited and it will work for practically unlimited files.

[PesaThe]

@Gilles If I understand correctly, all bash builtins, unlike external tools, are ok with extremely large numbers of arguments? Is there some point to the documentation of this please?

[Mikko Rantalainen]

With ulimit -s unlimited the actual command line limit is 2^31 or 2 GB. (MAX_ARG_STRLEN in kernel source.)

[Alf Eaton]

Adding -maxdepth 1 before -exec helped with moving files to a subdirectory, to avoid matching them again once they'd been moved.

[Mikko Rantalainen]

I have to add that I don't understand why Linux even limits the command line length by default. There's zero technical reason to do because the user process can still waste all the same memory at will so this is not suitable for resource usage limitations either.

[Mikko Rantalainen]

Also note that if you set stack size for your own shell and execute sudo ... the stack limit is not inherited because sudo always defines its own stack size (which seems to be 8 MB in my system).

[Uncle Billy]

@Kusalananda not necessarily. You can set ulimit just for a subshell: (ulimit -S -s unlimited; mv ...). MAX_ARG_STRLEN refers to the maximum length of a single command line argument, and it's much smaller than 2^31 (it's actually 32 pages => 128kb - 2Mb)

[Mikko Rantalainen]

@UncleBilly Note that the problem with sudo is that even if you have ulimit -S -s unlimited in your current shell, executing sudo mv * foobar will fail because sudo will change your limits before executing the command mv. To workaround that you have to do sudo -s instead and then set ulimit. If you don't need to use sudo, the subshell is a good way to proceed.

[Emil]

This solution is super simple (no missing with super long lines, which might cause data loss if you get anything wrong), has no side effects (closing shell discards the changes). So much undervoted.

[Admin]

This answer format is very considerate: the example is followed with one-liner explanations of the find command arguments. This format accelerates the learning process