zram vs zswap vs zcache Ultimate guide: when to use which one

kernel performance ram swap

73,414

Solution 1

Regarding 2., zswap does seem to decompress the pages on write-back, confirming @Cbhihe's comment.

/*
 * Attempts to free an entry by adding a page to the swap cache,
 * decompressing the entry data into the page, and issuing a
 * bio write to write the page back to the swap device.
 * ...
 */
static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
{
    ...
    
    case ZSWAP_SWAPCACHE_NEW: /* page is locked */
        /* decompress */
        ...
        
        ret = crypto_comp_decompress(tfm, src, entry->length,
                         dst, &dlen);
        ...
        kunmap_atomic(dst);    


$ git show
commit 1573d2caf713874cfe0d1336c823d0fb548d8bed
Merge: 4cdf8db 0a86248
Author: Linus Torvalds <[email protected]>
Date:   Tue Oct 11 23:59:07 2016 -0700

So zswap is useful for situations where the compressed in-ram cache is likely to be forgotten soon before written back to disk. It is not for applications with large, long living heaps that will eventually need to be backed by the actual swap device.

Solution 2

There is a whole lot of stuff about these three systems but none of it makes simple comparison between them let alone explain them well. I tried to make sense of it but my head exploded. Then I thought I had got it so I tried writing it down and my head exploded again. (see summary of implementations) I thought it will be useful to post this here as there were many stackexchange questions asking about pairwise comparisons between them.

Summary of what to use when:

ZRAM if you have no swap device on HDD/SSD.
ZSWAP if you do have a swap device on HDD/SSD.
ZCACHE: It does what ZSWAP does and ALSO compresses and speeds the filesystem page cache. (It is internally much more complicated and is not in the mainline kernel as it is still under development).

Summary of their implementations:

ZRAM is a compressed RAM based swap device
ZSWAP is a compressed Cache if you already have a swap.
ZCache is a backend for a special type of Virtual RAM thingy (Transcendent memory) that can be used to cache filesystem pages or swap data.

Details:

ZRAM: Makes a swap device in the RAM. Pages sent here are compressed as they are stored. It has a higher priority than other swap devices: pages that are swapped out are preferentially sent to the zram device till it is full, only then are any other swap devices used.
- Benefits: Independent of other (physical) swap devices. It can be used when there is no swap partition to expand the available memory.
- Disadvantages: If other swap devices (HDD/SSD) are present they are not used optimally. As the zram device is an independent swap device, once it is full, any new pages that need to be swapped out are sent to next swap device directly, hence:
  1. There is a real chance of LRU (least recently used) inversion: It will be the most recently swapped data that goes to the slow disk, while inactive pages that were swapped out long ago will remain in the fast ZRAM
  2. The data sent to and read from the disk will consume a lot of bandwidth as it is uncompressed.
- Status: Merged into the mainline kernel 3.14. Once enabled on a system, it requires some userspace configuration to set up the swap devices and use them.
ZSWAP: The frontswap system hooks attempts to swap out pages and uses zswap as write-back-cache for a HDD/SSD swap device: An attempt is made to compress the page and if it contains poorly compressible data it is directly written to the disk. If the data is compressed, it is stored in the pool of zswap memory. If pages are swapped out of memory when the total compressed pages in RAM exceeds a certain size, the Least Recently Used (LRU) compressed page is written to the disk as it is unlikely to be required soon.
- Benefits: Very efficient use RAM and disk based swap. Minimizes Disk I/O by both reducing the number of writes and reads required (data is compressed and held in RAM) and by reducing the bandwidth of these I/O operations as the data is in a compressed form.
- Limitations: It is an enhancement of disk based swap systems and hence depends on a swap partition on the hard disk.
- Status: Merged into the 3.11 mainline linux kernel.
ZCache: It is a backend for the Transcendent memory system. Transcendent memory provides a RAM-like memory that can only be accessed a page at a time by using put and get calls. This is unlike normal memory that can be accessed a byte at a time. The frontswap and cleancache systems hook attempts to swap and reclaim file-system page caches respectively and send them to the transcendent memory backends. When zcache is used as a backend, the data is compressed and stored in the RAM. When it fills up, compressed pages are evicted to the swap. (an alternate backend is RAMster which shares a pool of RAM across networked computers). Using only the frontswap frontend with the zcache backend works just like zswap. (In fact zswap is a simplified subset of zcache)
- Benefits Provides compressed caching both for swap and for filesystem caches.
- Status: Still not mainlined as it is very complicated and is being worked on.

The best resources I found were:

73,414

Author by

staticd

Updated on September 18, 2022

Comments

staticd over 1 year
1. What the hell are they? how are they different (I've written my understanding in an answer below)
2. In the Zswap system, when a page is evicted from the zswap to the actual swap is it stored in a compressed from? (or is it decompressed before storing?, AFAICT it is still compressed but i can't be sure)
3. What is the current state of zcache? it was apparently removed or something in 3.11. What does this mean? (http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=96256460487387d28b8398033928e06eb9e428f7)
- askb almost 10 years
  
  Ans 2. The commit message clearly state the patches (pertaining to zcache) have been removed from 3.11, but will be included into main -mm tree.
- Léo Léopold Hertz 준영 almost 8 years
  
  @staticd Why have you not accepted your own answer? It is very good!
- Cbhihe over 7 years
  
  When a page is evicted from the zswap (compressed swap cache) it is _ decompressed_ and placed in the backing swap device, per one of yr references [lwn.net/Articles/537422/] below...
- user3757405 over 5 years
  
  (lwn.net/Articles/537422 - "During resumed writeback, zswap decompresses the page ..."). @mmin below suggests this can be inefficient or even an exploitable danger for a server!
- thomasrutter over 3 years
  
  I have read the later versions of zram allow it to operate as a writeback cache, which implies it now can do what zswap does. However, I haven't wrapped my head around its configuration yet or if it's really applicable; the traditional behaviour of zram is that it operates very poorly when it fills and disk-based swap starts being used. Also, in my case, the zero-config nature of zswap makes it a no-brainer.
Phlya about 9 years

Is is possible and/or reasonable to use both zram and zswap?
Léo Léopold Hertz 준영 almost 8 years

Can you please summarise here how you activate zswap. Here details askubuntu.com/a/361321/25388
mnish over 7 years

I found a potentially dangerous behaviour of zswap. When an application allocates many pages and writes to them data that compresses very well (say a sequence of zeros), zswap happily stores them in kernel slab memory. However, when something triggers actual disk swapping, then the stored data suddenly 'bursts' -- that many zeros on the pages which took "only" gigabytes in memory now decompress to hundreds of gigabytes on disk.
Marc.2377 almost 7 years

Perhaps consider updating this answer since ZCache appears to be obsolete.
Ken Sharp over 6 years

Did you report this upstream?
Tom Hale over 6 years

Consider adding: With zswap, compressed pages are decompressed before being written to disk
Mihail Malostanidis about 6 years

@TomHale would you mind editing the answer? Currently the benefits of Zswap say the opposite, that the disk IO is additionally minimized by compressed data. Also, please link me to the reason they did that. Seems awful?
Mihail Malostanidis about 6 years

Yet another drawback of writing uncompressed data 🤷
user3757405 over 5 years

Surely it would be better on space and on time to dump the data in a decompressed form! It sounds like something we'd really want it to do. I can only assume that restructuring the swap area to allow this either involves rewriting a lot of existing code, or requires a more complex allocation system.
user3757405 over 5 years

Actually, it looks like it's touched on in lwn.net/Articles/548109. Hugh Dickins says "compression of page cache (file) pages may be appealing, but the filesystem developers do not seem to be that interested in zcache in general. So [...] it might make better sense to start with zswap, perhaps adding zcache features over time ..."
Victor Yarema about 5 years

Every answer that states that zram is a swap is totally wrong. zram IS NOT a swap. The swap only CAN be stored in zram. But this is one of many possible use cases!. Here's an example: "Some of the usecases include /tmp storage, use as swap disks, various caches under /var and maybe many more :)" kernel.org/doc/Documentation/blockdev/zram.txt For example I use it for temporary storage that I format and mount like any other normal block device.
Caesar over 4 years

It seems that the maximum compression ratio that zswap will give you is 2:1 (at least when using zbud). kernel.org/doc/Documentation/vm/zswap.txt While writing uncompressed data is indeed unfortunate, it seems less dangerous with that…
Mikko Rantalainen almost 4 years

@VictorYarema I agree. The zram is technically a compressed ramdrive, not a swap device. One can use that ramdrive exactly like any other block device and format it as e.g. ext4 or run mkswap against it. The most common usage is to format it as swap and use it as a swap device.
RJVB almost 4 years

Re: the "ZCache appears to be obsolete" comment above: that was for the version that was staged in Aug. 2013, almost 4 years before the comment was made. The linked commit message states that Bob Liu has rewritten it and is submitting it for inclusion through the main -mm tree, as it should have been done in the first place..., which is only unclear as to what name the rewritten version used. There are no newer commits referring to zcache so apparently a different name was used, OR the intended submittng never happened.