Maximum number of Bash arguments != max num cp arguments?

Question

I have recently been copying and moving a large number of files (~400,000). I know that there are limitations on the number of arguments that can be expanded on the Bash command line, so I have been using xargs to limit the numbers produced.

Out of curiosity, I wondered what the maximum number of arguments that I could use was, and I found this post saying that it was system-dependant, and that I could run this command to find out:

$ getconf ARG_MAX

To my surprise, the anwser I got back was:

2621440

Just over 2.6 million. As I said, the number of files that I am manipulating is much less than this -- around 400k. I definitely need to use the xargs method of moving and copying these files, because I tried using a normal mv * ... or cp * ... and got a 'Argument list too long' error.

So, do the mv and cp commands have their own fixed limit on the number of arguments that I can use (I couldn't find anything in their man pages), or am I missing something?

Solution 1

As Ignacio said, ARG_MAX is the maximum length of the buffer of arguments passed to exec(), not the maximum number of files (this page has a very in-depth explanation). Specifically, it lists fs/exec.c as checking the following condition:

PAGE_SIZE*MAX_ARG_PAGES-sizeof(void *) / sizeof(void *)

And, it seems, you have some additional limitations:

On a 32-bit Linux, this is ARGMAX/4-1 (32767). This becomes relevant if the average length of arguments is smaller than 4. Since Linux 2.6.23, this function tests if the number exceeds MAX_ARG_STRINGS in <linux/binfmts.h> (2^32-1 = 4294967296-1). And as additional limit, one argument must not be longer than MAX_ARG_STRLEN (131072).

Solution 2

ARG_MAX is the maximum length of the arguments to the exec(3) functions. A shell is not required to support passing this length of arguments from its command line.

Comments

[alexis]

But bash does not enforce any limit, says this; it just tries the exec and passes along an OS error if the argument list is too long.