Any work-around to pipe zip files?

Question

I use Ubuntu 16.04 with Nginx and Bash. I know that it's not possible to directly pipe data into zip.

For example, if you host websites on Apache/Nginx webserver, this command set would fail after filling in the password:

drt="/var/www/html"
mysqldump -u root -p --all-databases | zip "$drt/db-$date.zip"

What will be your workaround if you really desire the end file to be a zip file?

Solution 1

If you really want to use zip, you can use Jeff Schaller’s trick:

drt="/var/www/html"
mysqldump -u root -p --all-databases | zip "$drt/db-$date.zip" -

This will create a ZIP file containing a file named - whose contents are the database dump. This is mentioned in the zip manpage:

zip also accepts a single dash ("-") as the name of a file to be compressed, in which case it will read the file from standard input, allowing zip to take input from another program. For example:

tar cf - . | zip backup -

You could also use /dev/stdin instead:

mysqldump -u root -p --all-databases | zip -FI "$drt/db-$date.zip" /dev/stdin

This would result in an archive containing a file named dev/stdin which might be harder to handle properly. - is a common short-hand to tell programs to use standard input or output; it’s not something that the shell handles, it has to be supported by each individual program.

In both cases you’d probably want to use funzip to extract the data; it extracts the first member of an archive to its standard output.

Solution 2

If you are open to alternative compression tools, try this almost identical alternative.

drt="/var/www/html"
mysqldump -u root -p --all-databases | gzip >"$drt/db-$date.gz"

If you prefer you can substitute gzip with bzip2 or xz, and the typical extension from gz to bz2 or xz.

Solution 3

If you really prefer a zip file over using gzip or tar:

drt="/var/www/html"
scratch="$(mktemp)"
if mysqldump -u root -p --all-databases > "$scratch"; then
  zip "$drt/db-$date.zip" "$scratch"
fi
rm -f  "$scratch"

Optionally, you can use a temporary directory to have a more useful filename:

drt="/var/www/html"
scratchdir="$(mktemp -d)"
if mysqldump -u root -p --all-databases > "${scratchdir}/db-${date}.sql"; then
  zip -j "$drt/db-$date.zip" "${scratchdir}/db-${date}.sql"
fi
rm -rf "$scratchdir"

Comments

[roaima]

A workaround would be to use gzip, bzip2, or even xz instead of zip, and accept that Unix and Windows are different platforms with different software capabilities.

[user9303970]

hahaha, that's a comment with a funny part. I accept that, but workarounds can be done, sometimes, as you know.

[roaima]

Actually, I'm really quite serious. You're backing up MySQL databases. Why not use native compression tools?

[user9303970]

I know you are. I don't know gzip and I'm not sure I won't have problems uploading these to MySQL and MySQL like databases. It's not likely I'll have any problems but I'm not sure and I would prefer to stay with Zip that I know better, there is a small enough workaround, though I'm open to gzip or anything working slightly with these databases anyway.

[user9303970]

The name scratch is a must here?

[Pankaj Goyal]

No, that's just my default stand-in variable name for a "scratch file", i. e. one written to and discarded after use. mktemp creates a new file with a random name, so you need to capture that name to a variable if you want to actually do anything to it with a script. Helpfully, it outputs the path to the created file, which is why varname="$(mktemp)" works.

[Pankaj Goyal]

mktemp is guaranteed to create a new file which does not already exist. Sanity checking for existing procedural file names are left as an exercise for the OP as they are beyond the scope of the question at hand.

[Jeff Schaller]

My only suggestion here would be to chain the mysqldump, zip, and rm together with &&

[Pankaj Goyal]

The scratch file would need to be removed either way, but I have added a sanity check to only create the archive if mysqldump succeeds. As to the complaint about renaming the scratch file, keep in mind that using a defined name was a retrofit of the earlier example in case the filename actually matters.

[Jeff Schaller]

.. or unzip -p and redirect

[roaima]

@Peter ah yes. I'd mentioned it in my original comment (under the question) but omitted to include it here. Good reminder, thanks.

[user9303970]

I must say I'm still confused about why it must be a named pipe and not a regular pipe, I mean, why would the zip utility developers care if use an anonymous or a named pipe. Maybe there isn't a clear answer to that question.

[user9303970]

So instead having say mysql.sql inside the end zip I'll have - or dev/stdin with the sql data. That's a tiny bit depressing but I might need to eat more non depressing foods like Chocolate, to handle that.

[Stephen Kitt]

It can be a regular pipe, if you use a hyphen.

[user9303970]

I didn't figure it out from the answer. :\ my apologies.

[Stephen Kitt]

I realise for some reason the - in mysqldump -u root -p --all-databases | zip "$drt/db-$date.zip" - is hard to see, but I don’t understand what makes you think you need a named pipe there — “named pipe” isn’t mentioned in the answer.

[user9303970]

I thought that because of the -FI argument which I understood from man zip to represent a FIFO.

[Stephen Kitt]

Notice that -FI is only used for /dev/stdin, not for -. Details matter ;-).

[user9303970]

Indeed details matter. Ii wasn't clear to me at start (likely I misread as I was being hard to concentrate from problems I must get more well from). Now I know this and commented this to myself. When I use just an anonymous pipe and a hyphen, I don't use -FI. Anyway, seems I have trouble running a function containing this command: unix.stackexchange.com/questions/437640/…