Multiprocessor support for `xz`?
Multiprocessor (multithreading) compression support was added to xz
in version 5.2, in December 2014.
To enable the functionality, add the -T
option, along with either the number of worker threads to spawn, or -T0
to spawn as many CPU's as the OS reports:
xz -T0 big.tar
xz -T4 bigish.tar
The default single threaded operation is equivalent to -T1
.
I have found that running it with a couple of hyper-threads less than the total number of hyperthreads on my CPU† provides a good balance of responsiveness and compression speed.
† So -T10
on my 6 core, 12 thread workstation.
As scai and Dzenly said in comments
If you want to use this in combination with
tar
just callexport XZ_DEFAULTS="-T 0"
before.or use smth like:
XZ_OPT="-2 -T0"
ylluminate
Updated on June 03, 2022Comments
-
ylluminate almost 2 years
Is there a way to spread
xz
compression efforts across multiple CPU's? I realize that this doesn't appear possible withxz
itself, but are there other utilities that implement the same compression algorithm that would allow more efficient processor utilization? I will be running this in scripts and utility apps on systems with 16+ processors and it would be useful to at least use 4-8 processors to potentially speed up compression rates. -
scai about 5 yearsIf you want to use this in combination with
tar
just callexport XZ_DEFAULTS="-T 0"
before. -
Dzenly almost 5 yearsor use smth like: XZ_OPT="-2 -T0"
-
Luciano Andress Martini over 4 yearsI used the option XZ -T4 but I just see only one processor being used.
-
MarkSwears over 4 yearsSorry @LucianoAndressMartini I don't know what to suggest. If you're using version 5.2 or later (check with
xz -V
) and-T4
doesn't work on large files which aren't trivial to compress, I would suggest asking a question on [superuser.se] detailing your computer, operating system and kind of data you're compressing, -
jmon12 over 3 yearsIt seems that decompressing with multiple cores doesn't work, i.e.
xz --decompress -T0 file.tar.xz
. Is the decompression fundamentally non-parallelizable with xz? -
MarkSwears over 3 yearsThreading is all about speeding up compression @jmon12 , as decompression is trivial by comparison. From the man page "Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't, even if
--block-size=size
is used." -
louigi600 over 2 yearsSo supposing I want tar to do multi thread xz compression I could do something like this: tar cpO somename | xz -zT 5 -c > somename.tzx or alternatively set the env variable XZ_DEFAULTS="-T 5" (or whatever number you think is most appropriate for your system .... 0 will probably allow you to do little else until you finish the compression)