Unzip from stdin to stdout - funzip, python

Question

The goal is to read a zip file from stdin and uncompress to stdout.

Funzip works and is the solution I am looking for, the zip contains a single file, unfortunately funzip fails when the compressed file size is around 1GB or greater:

funzip error: invalid compressed data--length error

Update: I have discovered the above error may not indicate an actual error. Comparing two uncompressed files, one unzipped traditionally and the other through a pipe using funzip (with the above error written to stderr) the files are identical. I'd like to keep this open, so this can be confirmed or reported.

A related solution using python: Unzipping files that are flying in through a pipe

However this output is directed to a file.

Solution 1

Repost of my answer:

BusyBox's unzip can take stdin and extract all the files to stdout. For example, when you use wget as stdin,

wget -qO- http://downloads.wordpress.org/plugin/akismet.2.5.3.zip | busybox unzip -p -

-p to extract files to pipe. The dash after is to use stdin as input.

You also can, (just like previous answer)

cat file.zip | busybox unzip -p -

But that's just redundant of unzip -p file.zip.

If your distro uses BusyBox by default (e.g. Alpine), just run unzip -p -.

Solution 2

Use bsdtar from libarchive:

> curl 'https://example.com/some.zip' | bsdtar -xOf - | less

Solution 3

Simply use zcat. For example:

cat file.zip | zcat

Please note that in the example above the first part (cat file.zip) is redundant, in the sense that you can simply issue zcat file.zip and have the same results. I included it only to show that zcat is capable to read from stdin

Solution 4

regarding stdout: unzip supports that out of the box with its -c and -p options. regarding stdin: Eric pointed out that the zip format has its directory at the end of the file, so the only way to make it streamable is to copy the input to a temporary storage. The following simple script does that:

#!/bin/bash
# usage: unzipOnTheFly [unzipArgs]
# unzipArgs: additional args to be passed to unzip
tmpFile=/tmp/unzipOnTheFly.$$.zip
cat >$tmpFile
unzip -p $tmpFile "$@"
rm $tmpFile

In the above case the solution would be

<file.zip unzipOnTheFly

If the archive contains more than one file and you want only one you can say

<file.zip unzipOnTheFly pathToOneFile

View more solutions

Comments

[Eric]

zip ang gzip are not the same thing. zip files stores the list of files at the end of the stream which makes it impractical for piping in while downloading. gzip doesn't have that limitation. This answer should be downvoted.

[shodanshok]

@Eric on my CentOS 6 box, I surely can use zcat file.zip to show zipped files. For example: file services.zip returns services.zip: Zip archive data, at least v2.0 to extract, and I can zcat service.zip to show its content

[Eric]

Nope, it doesn't work on CentOS 6.9 when using a zip file that contains more than one file. I get gzip: zip file has more than one entry--rest ignored which is totally understandable since piping creates a stream and a stream is not seekable so zcat can't jump to the end of file to retrieve the files list. So in a nutshell zcat (actually gunzip since zcat calls it) doesn't work with zip files except when they contain only one file. Try it, you'lls see.

[shodanshok]

I was under the impression that OP really want to output to stdout a single zipped file, not an entire zip archive. After all, what is the point of showing multiple, concatenated files on stdout, when you can not tell were is start/end of each file?

[user829755]

@shodanshok, the point is you can add a filename from the list as cmd line argument to unzipper so that the output contains only that file. Yet as Eric pointed out that doesn't work with unzip if the input is a stream.

[Jean-Pierre Matsumoto]

For Cygwin, bsdtar has its own package bsdtar.