Faster zlib alternatives

Question

I wonder, if there are any faster builds of zlib around with more advanced optimizations?

If it's possible to optimize it using SSE instructions or Intel C++ compiller, or some trick which were patented earlier (I know patents were a serious limitation during gzip/zlib development), have anyone bothered to implement that?

I am especially interested in compression speed, which have a direct impact on high-performance web-services serving static & dynamic content.

Solution 1

Several years after this question was first asked some faster x86_64 zlib's appeared using the type of optimization BarsMonster suggested:

• Cloudflare zlib
• Intel zlib
• Intel's igzip
• Google's c-zlib
• libdeflate

The authors of samtools (a suite of tools for interacting with high-throughput sequencing data) produced a comparison of zlib speeds.

zlib-ng collects zlib optimizations not in mainline zlib but it may not be as stable as mainline zlib. Its issue tracker is also instructive as a reference to other known zlib speed ups.

Recently a pull request that claims to "speed up inflate_fast by up to 1.5x on x86_64" (by filling and copying 8 bytes at a time) has been made but not accepted into mainline zlib. The patch providing this work is also having a bumpy ride being accepted into Chromium (see this Chromium bug and the Chromium review) but hopefully the submitter will take a break to step away and recharge because from a distance it looks like progress was being very slowly made in a complex situation...

Even more recently, libdeflate is a heavily optimised rewrite that produces/confumes zlib compatible output/input but is does not provide a zlib compatible API. At the time of writing this has the zlib speed crown.

Solution 2

Another alternative is the miniz library (public domain, unlicense.org), which implements most of the zlib API in a single C source file and reads/writes zlib-compatible compressed data streams. At compression level 1 it uses a real-time compressor which is extremely fast (a little slower than minilzo, but with higher compression).

Solution 3

Not a rebuild, but two good alternatives to zlib are quicklz and fastlz. Both are very fast relative to gzip -1, but do not achieve as good of a compression ratio. For my application, size was going up 10-15% but compression speed was 6x, so a very good tradeoff.

Of course neither is compatible with zlib, so it may not work for you.

Comments

[BarsMonster]

Yes, I would need compatible one, for both API and algorithm.