How to use 7z to compress via pipe?

Question

I'm using this for creating backups securely (? - is it really secure? with a good password?):

# ENCRYPT
ORIGDIR="DIRECTORYNAMEHERE"; tar cvf - "${ORIGDIR}/" 2>/dev/null | gzip -9 - 2>/dev/null | openssl aes-256-cbc -salt -out "${ORIGDIR}.tar.gz.aes"

# DECRYPT
openssl aes-256-cbc -d -salt -in "ENCDIRECTORYNAMEHERE" | tar -xz -f -

Q: But how can I do this using 7z with max compression rate?

Creating temporary files besides the only OUTPUT file is not good, because if I need to compress ~100 GByte sized files/directories on a 180 GByte FS I wouldn't have enough free space (if ex.: the compressed file would take ~60 GByte).

Solution 1

Is there a reason you want to use 7z specifically, or do you just want better compression than gzip?

The xz utility uses the same compression algorithm as 7z (LZMA), and allows piped compression the same as gzip.

tar cvf ... | xz -9 | openssl ...

Solution 2

This is covered in the man page of 7z:

-si    Read data from StdIn (eg: tar cf - directory | 7z a -si directory.tar.7z)

Solution 3

Some examples of reading from stdin and writing to stdout:

echo "hello world" | 7z a .xz -si -so | xz -dc # hello world
echo "hello world" | 7z a .xz -si -so | wc -b # 64
echo "hello world" | 7z a .xz -si -so | 7z x -txz -si -so # hello world
7z x -so db.sql.xz | less # preview compressed database file

• a create archive. Requires a filename; we're using just .xz to abuse it to set the archive type, since we don't actually care about a name
• -si read from stdin
• -so write to stdout
• x extract; must also specify -t the type since we didn't create an archive w/ filename

Solution 4

For the record as this thread is quite old.

7-zip does not allow streamed write of 7z format, it requires free seeking access as it writes, amongst the other stuff, header at the end of operation. It does allow to pipe standard *nix formats like bz2, gz, xz, though. It also allows to pipe single input without restrictions (-si switch).

So one can compress single file into 7z format but cannot pipe it further.

$ cat archive.tar | 7z a -si archive.tar.7z

Other formats, like mentioned bz2, gz, xz can be written to stdout (-so switch).

$ cat archive.tar | 7z a -tgzip -si -so -an > archive.tar.gz

==

$ 7z a -si -so -tgzip -an < archive.tar > archive.tar.gz

This obviously allows to pipe it further:

$ cat archive.tar | 7z a -tgzip -si -so -an | wc -c

Note -an switch that disables parsing of archive_name. What it actually does, god one knows - this program is bit magical and you have to cope with its quirks. Example of this behaviour can be found here.

Solution 5

To quickly create an remote backup preparation file of all my home scripts.I use the following:

# Scripts backup
ls ~/*.sh | cpio -ov | 7z a -si ~/Documents/SCRIPT_BACKUP_30062017.cpio.7z
# Scripts restore
7z x -so ~/Documents/SCRIPT_BACKUP_30062017.cpio.7z | cpio -iv   

Reason I don't backup my 'home' root is that I specifically sync and send only some directories. Notice how cpio will skip and check for newer existing files on restore. This is powerful. Your work will not be overwritten.

bud@

View more solutions

Comments

[gasko peter]

Can I put a "|" and further process it after the 7z?

[Wieland]

You can use -so to dump the compressed data on stdout instead of a file.

[Unirgy]

The reason I personally needed specifically 7z is ability to split to multiple volumes

[Govind]

xz wasn't multi-threaded at the time (it may not be yet) so on a multi-core computer it could be a lot slower than 7z

[Wieland]

Please open a new question with an example to reproduce it.