Free commandline Zip utility for Windows?

Question

I'm trying to create scheduled task for copying a directory tree containing a very large number of small files from one server to another. Copying the tree directly takes a couple of days, so I was thinking it might be faster to zip the directory, copy the zip file, and unzip on the destination server.

I tried 7Zip's command-line utility, 7z.exe. It choked on an "unable to allocate memory" error. So now I'm looking for an alternative.

Does anyone know of a zip-utility that:

1. Runs on all modern flavors of Windows
2. Is free
3. Can be controlled from the command-line
4. Doesn't need to be installed (simply copying the EXE, and perhaps an associated DLL, the way that 7z.exe can be.)
5. Doesn't choke on very large directory trees

???

Solution 1

Have you considered using something like Robocopy with the /MT option? I would guess having a parallel copy is going to beat out compressing, transferring and decompressing. I would start with the default of 8 threads and monitor the disk queue length on both ends to make sure you are saturating, but not thrashing the disk to death. Robocopy can also only copy the files that actually are required to be transferred.

Usage :: ROBOCOPY source destination [file [file]...] [options]

/MT[:n] :: Do multi-threaded copies with n threads (default 8).
           n must be at least 1 and not greater than 128.

If that doesn't work for some reason, I have had good success using the zip.exe executable from the UnxUtils package or for more extreme zip needs, the KZIP.exe executable. Kzip can be quite a bit slower at compressing at the extreme compression settings, but I haven't seen a zip archiver that can consistently beat it for space efficiency with a standard archive. Of course, it will run as a stand-alone executable and is easily command line scriptable. I can't say I have ever used them on extremely large directories.

You should also be aware that a "standard" ZIP archive can be at most 4GB in size, and contain at most 65535 files. If your structure exceeds those limits you will need to use newer versions of the zip standard (ZIP64).

Solution 2

I may suggest PeaZip as possible solution. It makes very easy to create command scripts to batch execute compression tasks without the burden and limitations of GUI, and it is a frontend of Robocopy for file / directory copy operations.

Solution 3

Info-ZIP is the original free ZIP program.

I haven't tried it on very large directories. How big is the directory (number of files, total size, etc.)?

Comments

[Linker3000]

Wow, 7zip's usually pretty darn good - maybe a word in the support forum? 7-zip.org/support.html although I think there can be issues on allocating massive amounts of heap space on 32bit operating systems.

[akira]

What was the Commandline you user for how much of data?

[Rakesh Kumar]

7z.exe a -tzip output.zip dirname

[Rakesh Kumar]

Tens of millions of 2-5k files, in a tree totaling 10-20GB.

[Linker3000]

>>>>> Ouch! <<<<<

[Kez]

+1 for rsync. Copying only the changes to maintain an up to date copy of the entire directory tree on your other server makes much more sense - in terms of both time and bandwidth. I think most programs will likely choke with that many files to be honest.