curl .gz file and pipe it for decompression
Solution 1
A pipe (represented by the |
symbol) sends the standard output of one process to the standard input of another. In your case, you appear to want to use a named file so a pipe is not appropriate - specifically, there is nothing to pipe (hence the gunzip
error) because the remote contents are going to a local file. Instead, you'd need to extract the name of the file - for example, from its URL - something like (using bash's built in string manipulation capabilities)
curl -O "$URL" && gunzip -f "${URL##*/}"
If you want to use a pipe, then the way to do it would be something like
curl "$URL" | gunzip -c
(without the -O
option) so that curl
streams the remote contents to stdout
from where it can be piped into gunzip
, but then you would need to redirect the gunzip
output to overwrite the target uncompressed file as appropriate.
Solution 2
Follow redirects when downloading. Sometimes a web server has hidden redirects for security and/or random reasons. If you don't follow the redirect, the wrong data gets downloaded and your application reading the piped data gets confused. You can follow redirects with curl using the -L flag.
curl -L https://example.com/mygzip.tar.gz | tar zxv
Related videos on Youtube
skyork
Updated on September 18, 2022Comments
-
skyork over 1 year
I am trying to download some
.gz
files (N.B. not.tar.gz
ones) from given URLs and decompress them to overwrite existing files, if any.For each individual download, I tried the following:
curl -O $URL | gunzip -f
However, this did not work as it failed with:
gzip: stdin: unexpected end of file
. I ran a series of this command inside a bash shell script.If I spilt the command into two explicit steps, i.e. first download the file, and then decompress the
.gz
file, it works.Why the piped version does not work?
-
steeldriver over 9 yearsAre you sure that
curl -O
actually streams the file to standard output? Perhaps you are thinking ofwget -O-
? -
Panther over 9 years@steeldriver you should post that as an answer curl | tar xz ...
-
kenorb about 9 yearsThere is also
--compressed
, but this works only for a compressed responses.
-