zip: Argument list too long (80.000 files in overall)

Question

I need to compress 80.000 files into multiple zip files. This is the command I use:

zip -s 200M photos_test/*

However I get the following error:

-bash: /usr/bin/zip: Argument list too long

What can I do to solve the issue, beside manually splitting the folder files ?

thanks

Solution 1

If you want the whole directory, you could simply use the -r switch:

zip -r -s 200M myzip photos_test

That will include all subdirectories of photos_test though.

Solution 2

The problem seems to be the expansion of the "*". Use folder name or ".":

If you want to include the root folder within the zip:

zip -r my.zip folder_with_80k_files

If you don´t want to include the root folder inside the zip:

cd folder_with_80k_files
zip -r my.zip .

Solution 3

find photos_test/ -mindepth 1 -maxdepth 1 | zip -@ -s 200M

Solution 4

ls photos_test | zip -s 200M -@ photos

• -@ will cause zip to read a list of files from stdin
• | will pipe an output of ls into the input of zip command

man zip:

USE
⋮
   -@ file lists.  If a file list is specified as -@ [Not on MacOS], zip takes
   the  list  of  input  files from standard input instead of from the command
   line.  For example,

          zip -@ foo

   will store the files listed one per line on stdin in foo.zip.

   Under Unix, this option can be used to powerful effect in conjunction  with
   the  find (1)  command.   For example, to archive all the C source files in
   the current directory and its subdirectories:

          find . -name "*.[ch]" -print | zip source -@

   (note that the pattern must be quoted to keep the shell from expanding it).
⋮

View more solutions

Comments

[shgnInc]

The error -bash: /usr/bin/zip: Argument list too long may cause in to case: 1- because of not using -r switch, 2- there too many files for archiving. So in first case @Mat's answer is true and in the second case the @IgnacioVazquez-Abrams's answer is true.

[deaton.dg]

I've done what you suggest and I have a small myzip.zip (7mb) and the segments (200mb each). However, I cannot unzip the content, I'm running unzip unix myzip.zip but I get "bad zipfile offset (lseek)". Furthermore, I need to extract them in Windows environment as well, and there I only have Windows 7 extractor I guess.

[Mat]

those are different questions, open another question for that (link to this one for reference). make sure you post the names of the generated zip files (not all of them, but first and last at least), and the exact command line you use.

[Daniel Sokolowski]

Use find . -mindepth 1 -maxdepth -name '*.json' | zip {YOURZIPFILENAME}.zip -@ if you don't need to split and want to select files by extension.

[Lucas]

find . -mindepth 1 -maxdepth 1 -name '*.json' | zip {YOURZIPFILENAME}.zip -@ (there was missing 1 after -maxdepth).

[jdhao]

I am seeing errors : zip error: Invalid command arguments (cannot write zip file to terminal).