Argument list too long error for rm, cp, mv commands

Question

I have several hundred PDFs under a directory in UNIX. The names of the PDFs are really long (approx. 60 chars).

When I try to delete all PDFs together using the following command:

rm -f *.pdf

I get the following error:

/bin/rm: cannot execute [Argument list too long]

What is the solution to this error? Does this error occur for mv and cp commands as well? If yes, how to solve for these commands?

Solution 1

The reason this occurs is because bash actually expands the asterisk to every matching file, producing a very long command line.

Try this:

find . -name "*.pdf" -print0 | xargs -0 rm

Warning: this is a recursive search and will find (and delete) files in subdirectories as well. Tack on -f to the rm command only if you are sure you don't want confirmation.

You can do the following to make the command non-recursive:

find . -maxdepth 1 -name "*.pdf" -print0 | xargs -0 rm

Another option is to use find's -delete flag:

find . -name "*.pdf" -delete

Solution 2

tl;dr

It's a kernel limitation on the size of the command line argument. Use a for loop instead.

Origin of problem

This is a system issue, related to execve and ARG_MAX constant. There is plenty of documentation about that (see man execve, debian's wiki, ARG_MAX details).

Basically, the expansion produce a command (with its parameters) that exceeds the ARG_MAX limit. On kernel 2.6.23, the limit was set at 128 kB. This constant has been increased and you can get its value by executing:

getconf ARG_MAX
# 2097152 # on 3.5.0-40-generic

Solution: Using for Loop

Use a for loop as it's recommended on BashFAQ/095 and there is no limit except for RAM/memory space:

Dry run to ascertain it will delete what you expect:

for f in *.pdf; do echo rm "$f"; done

And execute it:

for f in *.pdf; do rm "$f"; done

Also this is a portable approach as glob have strong and consistant behavior among shells (part of POSIX spec).

Note: As noted by several comments, this is indeed slower but more maintainable as it can adapt more complex scenarios, e.g. where one want to do more than just one action.

Solution: Using find

If you insist, you can use find but really don't use xargs as it "is dangerous (broken, exploitable, etc.) when reading non-NUL-delimited input":

find . -maxdepth 1 -name '*.pdf' -delete 

Using -maxdepth 1 ... -delete instead of -exec rm {} + allows find to simply execute the required system calls itself without using an external process, hence faster (thanks to @chepner comment).

References

• I'm getting "Argument list too long". How can I process a large list in chunks? @ wooledge
• execve(2) - Linux man page (search for ARG_MAX) ;
• Error: Argument list too long @ Debian's wiki ;
• Why do I get “/bin/sh: Argument list too long” when passing quoted arguments? @ SuperUser

Solution 3

find has a -delete action:

find . -maxdepth 1 -name '*.pdf' -delete

Solution 4

Another answer is to force xargs to process the commands in batches. For instance to delete the files 100 at a time, cd into the directory and run this:

echo *.pdf | xargs -n 100 rm

Solution 5

If you’re trying to delete a very large number of files at one time (I deleted a directory with 485,000+ today), you will probably run into this error:

/bin/rm: Argument list too long.

The problem is that when you type something like rm -rf *, the * is replaced with a list of every matching file, like “rm -rf file1 file2 file3 file4” and so on. There is a relatively small buffer of memory allocated to storing this list of arguments and if it is filled up, the shell will not execute the program.

To get around this problem, a lot of people will use the find command to find every file and pass them one-by-one to the “rm” command like this:

find . -type f -exec rm -v {} \;

My problem is that I needed to delete 500,000 files and it was taking way too long.

I stumbled upon a much faster way of deleting files – the “find” command has a “-delete” flag built right in! Here’s what I ended up using:

find . -type f -delete

Using this method, I was deleting files at a rate of about 2000 files/second – much faster!

You can also show the filenames as you’re deleting them:

find . -type f -print -delete

…or even show how many files will be deleted, then time how long it takes to delete them:

root@devel# ls -1 | wc -l && time find . -type f -delete
100000
real    0m3.660s
user    0m0.036s
sys     0m0.552s

View more solutions

Comments

[another.anon.coward]

You might find this link useful

[jfs]

related: Solving “mv: Argument list too long”?

[Lorenzo Belli]

Also this can be relevant http://mywiki.wooledge.org/BashFAQ/095

[Vicky]

@jww: And I continued to think for so many years that bash falls under "software tools commonly used by programmers" -- a category whose questions can be asked here!

[jww]

@Nik - Adding "... in a script" is not compelling. When the problem is reduced to a Minimal, Complete, and Verifiable example, its just a question on how to run a command. My apologies if I am missing something obvious.

[Vicky]

@jww: not "how to run a command".. but "how to run this particular command without getting the error I was getting"... no?

[Dhiren Hamal]

Run "ulimit -S -s unlimited" command. Then perform cp/mv/rm operation.

[KriptSkitty]

Doesn't this send exactly the same arguments to rm as rm -f *.pdf does? (Or, if there are subdirectories, even more arguments.)

[tripleee]

No, xargs specifically splits up the list and issues several commands if necessary.

[Bobby Jack]

note that this, and the other 'find' answer will remove PDF files from subdirectories, which is different from the behaviour of 'rm *.pdf' although it's unclear exactly which behaviour the questioner is after.

[DPlusV]

Indeed, this is a recursive search, which is different from the original question. I've added a warning -- thanks for pointing out out this inconsistency.

[Vicky]

@Dennis: What if I replace . with exact directory path -- Will it still remove pdf's from all subdirectories ?

[Vicky]

@BobbyJack: I want to remove pdf's only from the current working directory. Not from all subdirectories. How will above solution get modified for this case ?

[Vicky]

This deletes files from subdirectories as well. How to prevent that ?

[Vicky]

@Dennis: Also could you please give examples for CP and MV commands using your solution ? Where to give destination directory ?

[DPlusV]

@NikunjChauhan I've updated the find command to work in the current directory only (see -maxdepth -1). For cp and mv, I'd use alternative syntax: find . -name "*.pdf" -maxdepth 1 -exec cp {} FOO -- FOO here is the destination. The command will re-run cp each time, replacing {} with the name of the file. mv would work similarly.

[Jon Lin]

@NikunjChauhan Add -maxdepth option: find . -maxdepth 1 -name '*.pdf' -exec rm -f {} \;

[Vicky]

I am not able to insert the maxdepth option

[Barton Chittenden]

@Dennis: -maxdepth 1 needs to be the first argument after the path.

[ThiefMaster]

I think this would do really nice things with a file named e.g. -rf .. .pdf

[BigMike]

yes it would, but generally when used in shell, the issuer of the command "should" give a look at what he's doing :). Actually I prefer to redirect to a file and then inspect every single row.

[Édouard Lopez]

from #bash channel: "xargs(1) is dangerous (broken, exploitable, etc.) when reading non-NUL-delimited input. If you're working with filenames, use find's -exec [command] {} + instead or -print0 | xargs -0 provided you don't care about portability and like doing unnecessary extra work. Othwerwise a nice for loop."

[FooF]

If you have a file named pdf_format_sucks.docx this will be deleted as well... ;-) You should use proper and accurate regular expression when grepping for the pdf files.

[user85155]

correct(er): find . -maxdepth 1 -name "*.pdf" -print0 | xargs -0 rm

[scragar]

Find has a -delete flag to delete the files it finds, and even if it didn't it would still be considered better practice to use -exec to execute rm, rather than invoking xargs(which is now 3 processes and a pipe instead of a single process with -delete or 2 processes with -exec).

[Sergio]

This would still return "Argument list too long". At least for me it does. Using xargs, as per Dennis' answer, works as intended.

[ThiefMaster]

That sounds like a bug in find.

[Reinstate Monica Please]

@ÉdouardLopez ... But this is reading NULL-delimited input. And the whole dangerous (broken, exploitable, etc.), is fairly ridiculous. Undoubtedly you should be careful when using xargs, but it is not quite eval/evil.

[damianostre]

I don't know why someone downvoted this, without even commenting on that (that's policy, folks!). I needed to delete all files inside a folder (the question is not particular about PDFs, mind you), and for that, this trick is working well, all one has to do in the end is to recreate the folder that got deleted along when I used `rm -R /path/to/folder".

[jvriesem]

That option may be a Linux-only option, as per @Dennis's answer, above (the selected answer).

[nukeguy]

What if I need to delete files in sudo mode? When I run this code I get a list of files saying "Permission denied"

[tommed]

Great answer, this is how all SO questions should be answered. Thanks!

[Code Abominator]

on Centos/bash this gave the same error - argument list too long.

[Luxian]

@Sergio had same issue, it was caused by the missing quotes around name pattern.

[Peter Cordes]

This doesn't quote "$f". That's what ThiefMaster was talking about. -rf takes precedence over -i, so your 2nd version is no better (without manual inspection). And is basically useless for mass delete, because of prompting for every file.

[Alvein]

It works because in OP's case, he was using *, which expanded to a huge list of .pdf, giving a directory will cause this to be treated internally, thus, not having to deal with OP's issue. I think it was downvoted for that reason. It might not be usable for OP if he have nested directory or other files (not pdf) in his directory

[Braden Best]

Funny. When this happened to me (cause I had a directory with over 100,000 empty files), I just reflexively changed it to a fancy loop. c=1;l=$(ls | wc -l); for i in *; do rm $i; echo "[$c / $l] $i"; c=$((c + 1)); done

[mathreadler]

argh, why does a tool for finding stuff even have a switch for deleting? is it really just me who find it unnecessary to say the least and also dangerous.

[tripleee]

This is not safe at all, and does not work with file names with newlines in them, to point out one obvious corner case. Parsing ls is a common antipattern which should definitely be avoided, and adds a number of additional bugs here. The grep | grep is just not very elegant.

[tripleee]

Anyway, it's not like this is a new and exotic problem which requires a complex solution. The answers with find are good, and well-documented here and elsewhere. See e.g. the mywiki.wooledge.org for much more on this and related topics.

[James Tocknell]

For a large number of files this seems significantly faster

[ArtificiallyIntelligence]

For deleting command in linux, which can be a disaster if you are an engineer and you typed a mistake, I believe it is the 'safest and I know what's going on' is the best one. Not fancy stuff that if you miss type a dot will let your company crash down in one minute.

[neuralmer]

@scragar With -exec calling rm, the number of processes will be 1 + number of files, although the number of concurrent processes from this may be 2 (maybe find would execute rm processes concurrently). The number of processes using xargs would be reduced dramatically to 2 + n, where n is some number processes less than number of files (say number of files / 10, although likely more depending on the length of the paths). Assuming find does the deletion directly, using -delete should be the only process that would be invoked.

[user1212212]

How can we make this the default expansion for certain commands? There's a good many "standard" linux commands where it's known if they need them all at once or not (like "rm")

[FooF]

Better, but still_pdf_format_sucks.docx will get deleted. The dot . in ".pdf" regular expression matches any character. I would suggest "[.]pdf$" instead of .pdf.

[Toby Speight]

Note that this only works where echo is a shell builtin. If you end up using the command echo, you'll still run into the program arguments limit.

[Toby Speight]

While this code snippet may solve the question, including an explanation of how and why this solves the problem would really help to improve the quality of your post. Remember that you are answering the question for readers in the future, not just the person asking now! Please edit your answer to add explanation, and give an indication of what limitations and assumptions apply.

[Toby Speight]

While this code snippet may solve the question, including an explanation of how and why this solves the problem would really help to improve the quality of your post. Remember that you are answering the question for readers in the future, not just the person asking now! Please edit your answer to add explanation, and give an indication of what limitations and assumptions apply.

[chepner]

@mathreadler It addresses the fact that a common use case for -exec is to remove a bunch of files. -exec rm {} + would do the same thing, but still requires starting at least one external process. -delete allows find to simply execute the required system calls itself without using an external wrapper.

[Robert Dundon]

+1 for mentioning the for loop. I've used find before, but I'm always looking up how to do it as I forget the options, etc. all the time. for seems easier to recall IMHO

[abdul qayyum]

Used it as for f in *; do rm "$f"; done work as a charm

[Freitags]

On a debian system with ext3 filesystem, find was NO solution for me. The for loop solution presented by Edouard Lopez is easy, safe, well explained and worked on my system.

[mmann1123]

Also found this solution inefficient. See for loop options

[AJ H]

The for loop solution actually saved me!

[trey-jones]

The find -exec solution seems to be MUCH faster than the for loop.

[Nigel Alderton]

Thanks. I did sudo find . -type f -delete to delete about 485 thousand files and it worked for me. Took about 20 seconds.

[Toby Speight]

No, rm has no such limit on the number of files it will process (other than that its argc cannot be larger than INT_MAX). It's the kernel's limitation on the maximum size of the entire argument array (that's why the length of the filenames is significant).

[Toby Speight]

In particular, if printf isn't a shell builtin, it will be subject to the same limitation.

[Toby Speight]

Dangerous (or it would be if you used backquotes as evidently intended) - if any filename contains shell metacharacters, including spaces, then the results will not be what you intended.

[Toby Speight]

I don't know, but I'll guess it's because passing the output of ls directly to other commands is a dangerous antipattern - that, and the fact that the expansion of the wildcard will cause the same failure when executing ls as experienced in the original rm command.

[Eaten by a Grue]

the for loop is painfully slow. tried this on a directory with 100,000+ files in it and 30 seconds later it had only deleted 12,000 or so. Tried the find version and it was done in half a second

[Mattwmaster58]

Five years later at 4.15.0 (4.15.0-1019-gcp to be exact) and the limit is still at 2097152. Interestingly enough, searching for ARG_MAX on the linux git repo gives a result showing ARG_MAX to be at 131702.

[jdhao]

The for loop solution is way slower than the find solution.

[Mark G.]

@scragar Using find’s -exec is not always better-practice, particularly in cases like this (“Argument list too long”). While it’s true that using -exec results in 2 subprocesses and no pipe (which certainly sounds a lot better than 3 processes and a pipe) that’s max. processes in parallel which is a very different number from total processes launched to complete the job, as @neuralmer points out. In terms of speed, -delete > xargs > -exec, because sane xargs implementations break args up into batches to reduce subprocessing, whereas -exec == 1 arg per process.

[Mark G.]

…That said, I understand that blind trust of xargs is not without its incredibly painful gotchas (particularly when rm is involved) so if you’re coming at this from a safety rather than a speed perspective, I’m with you all the way.

[Dr_Zaszuś]

Great solution, but is there a way to show deletion progress?

[dr jerry]

not sure about the warning against xargs, from Unix Power Tools (2nd edition, O'Reilly) find -print0 | xargs -0 is fast and will never fail.

[tripleee]

This is pretty vague and requires you to have installed locate back when you still had room on your disk.

[tripleee]

The whole point of -exec is that you don't invoke a shell. The quotes here do absolutely nothing useful. (They prevent any wildcard expansion and token splitting on the string in the shell where you type in this command, but the string {} doesn't contain any whitespace or shell wildcard characters.)

[Charles Duffy]

For context on that, see ParsingLs. And parallel makes some folks who prefer avoiding complexity uncomfortable -- if you look under the hood, it's pretty opaque. See the mailing list thread at lists.gnu.org/archive/html/bug-parallel/2015-05/msg00005.htm‌​l between Stephane (one of the Unix & Linux StackExchange greybeards) and Ole Tange (Parallel's author). xargs -P also paralellizes, but it does it in a simpler, dumber way with fewer moving parts, making its behavior far easier to predict and reason about.

[Prisacari Dmitrii]

The answer is a bit useless as the solution was provided but the genuine reason of the error was not.

[akozi]

If you're ever worried about making sure that you wrote your arguments right and the correct files will be deleted, you can replace the find command with find . -maxdepth 1 -name '*.pdf' -print. Which will show you the file list. Then if it looks fine, replace the print back with the -delete.

[rogerdpack]

Works for multiple globs too for for in *.pdf my_dir/*.pdf; ...

[Michael Roswell]

I actually found this page after running into file limitations for mv so I'm not confident this solves the problem.

[tripleee]

There is also still the inherent problem that echo could produce something else than the literal file names. If a file name contains a newline, it will look to xargs like two separate file names, and you get rm: firsthalfbeforenewline: No such file or directory. On some platform, file names which contain single quotes will also confuse xargs with the default options. (And the -n 100 is probably way too low; just omit the option to let xargs figure out the optimal number of processes it needs.)

[tripleee]

I added the missing quotes, but this still has issues. Manually inspecting thousands of files is error-prone and unnecessary.

[tripleee]

This just reinvents xargs, rather poorly.

[tripleee]

But this still fails on files with irregular names, partly because of the lack of quoting. If you really feel a need to reimplement xargs, you need to understand all the corner cases it handles.

[tripleee]

What's the point of a Python script if you end up shelling out with os.system() anyway? You want os.unlink() instead; then you don't have to solve the quoting problems which this fails to solve properly. But the only reason this is more efficient than find is that it doesn't recurse into subdirectories; you can do the same with printf '%s\0' /tmp/* | xargs -r0 rm -rf or of course by adding a -maxdepth 1 option to the find command.

[tripleee]

(Actually find will probably be quicker because the shell will alphabetize the list of files matched by the wildcard, which could be a significant amount of work when there are a lot of matches.)

[Nishant Kumar]

this is a simple solution: find . -name "*.pdf" -delete

[Daniel Ribeiro]

This is a very nice solution. It doesn't change the original script by much and still makes possible to delete lots of files at once. For me it worked just fine.

[benathon]

I choose -n 100 as a conservatively low number because I don't think there is any reason to increase n to the point of being "optimal" here.

[Lei Zhang]

using find, but still too long