Multiprocessor support for `xz`?

xz
16,600

Multiprocessor (multithreading) compression support was added to xz in version 5.2, in December 2014.

To enable the functionality, add the -T option, along with either the number of worker threads to spawn, or -T0 to spawn as many CPU's as the OS reports:

xz -T0 big.tar
xz -T4 bigish.tar

The default single threaded operation is equivalent to -T1.

I have found that running it with a couple of hyper-threads less than the total number of hyperthreads on my CPU provides a good balance of responsiveness and compression speed.

† So -T10 on my 6 core, 12 thread workstation.

As scai and Dzenly said in comments

If you want to use this in combination with tar just call export XZ_DEFAULTS="-T 0" before.

or use smth like: XZ_OPT="-2 -T0"

Share:
16,600
ylluminate
Author by

ylluminate

Updated on June 03, 2022

Comments

  • ylluminate
    ylluminate almost 2 years

    Is there a way to spread xz compression efforts across multiple CPU's? I realize that this doesn't appear possible with xz itself, but are there other utilities that implement the same compression algorithm that would allow more efficient processor utilization? I will be running this in scripts and utility apps on systems with 16+ processors and it would be useful to at least use 4-8 processors to potentially speed up compression rates.

  • scai
    scai about 5 years
    If you want to use this in combination with tar just call export XZ_DEFAULTS="-T 0" before.
  • Dzenly
    Dzenly almost 5 years
    or use smth like: XZ_OPT="-2 -T0"
  • Luciano Andress Martini
    Luciano Andress Martini over 4 years
    I used the option XZ -T4 but I just see only one processor being used.
  • MarkSwears
    MarkSwears over 4 years
    Sorry @LucianoAndressMartini I don't know what to suggest. If you're using version 5.2 or later (check with xz -V) and -T4 doesn't work on large files which aren't trivial to compress, I would suggest asking a question on [superuser.se] detailing your computer, operating system and kind of data you're compressing,
  • jmon12
    jmon12 over 3 years
    It seems that decompressing with multiple cores doesn't work, i.e. xz --decompress -T0 file.tar.xz. Is the decompression fundamentally non-parallelizable with xz?
  • MarkSwears
    MarkSwears over 3 years
    Threading is all about speeding up compression @jmon12 , as decompression is trivial by comparison. From the man page "Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't, even if --block-size=size is used."
  • louigi600
    louigi600 over 2 years
    So supposing I want tar to do multi thread xz compression I could do something like this: tar cpO somename | xz -zT 5 -c > somename.tzx or alternatively set the env variable XZ_DEFAULTS="-T 5" (or whatever number you think is most appropriate for your system .... 0 will probably allow you to do little else until you finish the compression)