Is it possible to pipe from stdin to gzip?

Question

If I run the command s3cmd get s3://bucket/file.gz - I get binary output on the screen. If I try to pipe this to gzip with s3cmd get s3://bucket/file.gz - | zcat I get gzip: stdin: not in gzip format.

How can I get zcat to pick it's input from stdin?

Solution

gzip and its auxilliary commands all read from STDIN by default. We can test this with a really simple test:

$ echo testing | gzip | zcat
testing

Or something more exotic to prove that wasn't a fluke:

$ dd if=/dev/urandom of=bigfile bs=1024 count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 6.42114 s, 16.3 MB/s

$ sha1sum bigfile 
25b4832d3e738e70721d86695ea7a767a3afb229  bigfile

$ cat bigfile | gzip | zcat | sha1sum 
25b4832d3e738e70721d86695ea7a767a3afb229  -

That suggests to me that your s3cmd output is dirty or malformed in some way. Try redirecting to file (rather tha providing a real filename) and then looking at the output in something like head. Or download it properly and compare.

Comments

[KalenGi]

I think you're right about the s3cmd output. I changed my approach and used curl instead.