Difference between pixz and xz with -T option

Question

I'm trying to compress a 100GB with a good ratio, so I decided to use xz. I don't need it to be very fast, but since I have an 8-threads CPU, I was wondering how to take advantage of them.

I found there is a parallel implementation of xz: pixz. I found too that xz has a multithread option -T. I'm experimenting with both of them but I can't see any difference in their behavior.

What's the difference between them? Why are there 2 parallel implementations for xz?

Edit: I found there is another one: pxz.

Solution

I'm the author of pixz. As speculated above, I did write pixz before threading support was added to xz. I like to think that pixz prodded Lasse Collin to add multi-threading support to xz sooner.

There are still some differences between pixz and xz, of course. In xz's favour:

• It has far more options
• It's been tested by many more people
• It probably has better code quality—pixz is a prototype gone rampant ;)

But there are still some reasons I like to use pixz:

• It's much smaller, and easier to comprehend
• It supports multi-threaded decompression, which xz does not
• It has special behaviour on tarballs: While performing compression, it also creates an index of all the files in the tarball. This allows a single file to be extracted quickly, even if the tarball is huge.

Comments

[Mark K Cowan]

Never heard of pixz before, but intending to use it now just for that last point alone

[osgx]

Is pixz format with index enabled compatible with classic xz tool? How input file is partitioned when compressed in parallel?

[Shane L]

The pixz format is compatible with xz. If pixz adds an index, it will slightly change the output after decompression with xz—but the only change will be after the tar End-Of-Archive marker, and therefore ignored by tar. If you don't want an index, you can disable it with the '-t' option.

[Shane L]

Partitioning the input file into blocks is based on: * The xz dictionary size, which depends on the compression level. For the default level of '-6', it's 8 MiB. See the xz manpage for more details. * A multiplier, provided with the '-f' option. The default is 2.0. The default block size will therefore be 16 MiB. If you want more details about pixz, please check out the docs