How do I unzip a compressed file as it's downloading?

Question

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing.

How can I do them simultaneously?

Solution 1

This:

wget -O - -o /dev/null http://download.freebase.com/datadumps/latest/freebase-simple-topic-dump.tsv.bz2 | bunzip2 > freebase-simple-topic-dump.tsv

Where bunzip2 is an unzipping command for your compression format of choice. It must support piped input. And the file must be a single compressed file, not an archive.

It uses wget to pipe the downloaded file to the unzipping application, outputting to the specified filename.

Solution 2

The question is tagged with curl but the answer only uses wget.

With curl it's a little easier than wget because it can request compression and decompress without piping (url truncated for clarity).

curl --compressed http://freebase.com/topic.bz2

Comments

[MaQleod]

Technically, it doesn't do them simultaneously, it just performs the download via wget first and then pipes the results as a whole into bunzip. If you attempted to unzip a file that wasn't completely written you'd get an error indicating that the end of file was reached too soon.

[Max]

@MaQleod Sorry, I'm pretty sure that's incorrect. Try running it without forwarding the output to a file; it starts printing straight away.

[Max]

Looks like it does the trick. Great answer @user23337

[Eneroth3]

I agree with @Alec as it is also faster