Can not use Swap file on ZFS: Files with holes

Question

I'm trying to add a new swapfile to increase Swap Space by following this tutorial.

The swapfile creates successfully but, the swapon command won't accept it: says "skipping - it appears to have holes.".

I'm running Xubuntu 19.10 with ZFS on root.

Here is what I've tried so far:

Create Swapfile

Using fallocate - failed

sudo fallocate -l 8G /swapfile
fallocate: fallocate failed: Operation not supported

It seems fallocate currently does not support ZFS.

Alternative way - Using dd - success

sudo dd if=/dev/zero of=/swapfile bs=1MiB count=$((8*1024))
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 2.68284 s, 3.2 GB/s

Prepare the Swap File - success

sudo chmod 600 /swapfile
ls -lah /swapfile 
-rw------- 1 root root 8.0G Dec 27 14:15 /swapfile

sudo mkswap /swapfile
Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=db8aa64c-734c-4eba-a803-72db681ec1a1

Enable the Swap File - failed

sudo swapon /swapfile
swapon: /swapfile: skipping - it appears to have holes.

From swapon manual:

Files with holes

The swap file implementation in the kernel expects to be able to write to the file directly, without the assistance of the filesystem. This is a problem on files with holes or on copy-on-write files on filesystems like Btrfs.

Commands like cp(1) or truncate(1) create files with holes. These files will be rejected by swapon.

Preallocated files created by fallocate(1) may be interpreted as files with holes too depending of the filesystem. Preallocated swap files are supported on XFS since Linux 4.18.

The most portable solution to create a swap file is to use dd(1) and /dev/zero.

How can I create a swapfile on ZFS which the swapon command would accept?

System Info:

sudo parted -l

Model: WDC PC SN520 SDAPNUW-512G-1002 (nvme)
Disk /dev/nvme0n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name                  Flags
 1      1049kB  538MB   537MB   fat32           EFI System Partition  boot, esp
 2      538MB   590MB   52.4MB  ext4
 3      590MB   2738MB  2147MB  linux-swap(v1)
 4      2738MB  4885MB  2147MB  zfs
 5      4885MB  512GB   507GB   zfs

sudo zpool list -v

NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool        1.88G   131M  1.75G        -         -     0%     6%  1.00x    ONLINE  -
  nvme0n1p4  1.88G   131M  1.75G        -         -     0%  6.82%      -  ONLINE  
rpool         472G   112G   360G        -         -     9%    23%  1.00x    ONLINE  -
  nvme0n1p5   472G   112G   360G        -         -     9%  23.8%      -  ONLINE

sudo zfs list

# https://pastebin.ubuntu.com/p/6jDVwbhfCT/

sudo lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.10
Release:    19.10
Codename:   eoan

sudo uname -a

Linux iCyberRoze 5.3.0-24-generic #26-Ubuntu SMP Thu Nov 14 01:33:18 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Solution 1

The tutorial you linked is not under the assumption that your root filesystem is ZFS. The ZFS manpage indicates:

ZFS Volumes as Swap

ZFS volumes may be used as swap devices. After creating the volume with the zfs create -V command set up and enable the swap area using the mkswap(8) and swapon(8) commands. Do not swap to a file on a ZFS file system. A ZFS swap file configuration is not supported.

You may follow the instructions in the other answer from stratus to get swap space working from a zvol which can be used as swap space that is part of your ZFS pool.

However, if you really want to proceed against recommendations to use a swap file stored in a ZFS filesystem:

fallocate does not work within ZFS filesystems, as you already know from the zfsonlinux github bug you already posted. Instead of using dd which will be slower since it has to write every piece of the output file, you may want to quickly create a large sparse file that you can create on a ZFS filesystem, try the truncate command which does the same thing but works on ZFS.

sudo truncate -s 8G /swapfile
sudo chmod 600 /swapfile
sudo swapon /swapfile

It still shows: swapon: /swapfile: skipping - it appears to have holes.

ls -lsh /swapfile

Reports 512 -rw------- 1 root root 8.0G Jan 17 18:27 /swapfile showing that the created file takes up only 512 bytes instead of 8 GB.

Your compression settings for the ZFS filesystem that will contain this file will most likely cause the output of dd to be sparse anyway (taking up much less space on the disk than the reported filesize).

sudo dd if=/dev/zero of=/swapfile bs=1M count=8k status=progress
ls -lsh /swapfile

Reports 512 -rw-rw-r-- 1 root root 8.0G Jan 17 18:39 /swapfile showing that the ZFS filesystem compression settings made the complete 8 GB file fit into 512 bytes.

Instead of using a highly compressible input to dd of /dev/zero you can use the incompressible /dev/urandom so that it will actually take up 8 GB of hard drive space. However from my testing, swapon still rejects it:

sudo dd if=/dev/urandom of=/swapfile bs=1M count=8k status=progress
sudo chmod 600 /swapfile
sudo swapon /swapfile

It still shows: swapon: /swapfile: skipping - it appears to have holes.

One more experiment could be to try it in a ZFS filesystem that has compression turned off.

sudo zfs create rpool/swap -o compression=off -o mountpoint=/swap/
sudo dd if=/dev/zero of=/swap/swapfile bs=1M count=8k status=progress

And it really does write all 0's to the disk this time.

ls -lsh /swap/swapfile

This reports back 8.0G -rw-r--r-- 1 root root 8.0G Jan 17 18:52 /swap/swapfile

sudo chmod 600 /swap/swapfile
sudo swapon /swap/swapfile

It still shows: swapon: /swapfile: skipping - it appears to have holes.

My conclusion is that I can't tell that there's a way to circumvent its refusal of using a file located in a ZFS filesystem as a swap file.

Outside of the scope of this question, alternatives to using swap space on this system would be a swap partition (which I see you have a 590MB swap partition already, or to put a swap file on a non-ZFS filesystem such as on an ext4 partition, or if disk space is unavailable, use Zswap (Compressed blocks of RAM used as swap space) or vramfs (Swap space that lives on your graphics card's RAM).

Solution 2

I think what you want is actually well documented on the ZFSOnlinux page

TLDR;

# To set swap on a zfs drive:
zfs create -V 8G -b $(getconf PAGESIZE) -o logbias=throughput -o sync=always -o primarycache=metadata -o com.sun:auto-snapshot=false VMs/swap

mkswap -f /dev/zvol/VMs/swap
swapon /dev/zvol/VMs/swap

# IN FSTAB
/dev/zvol/VMs/swap none swap discard 0 0

Comments

[user3211705]

Thank you for your response but this is not what the question is about. This solution is for creating a zvol and using it as swap but, the question is about creating a swapfile on a zfs partition.

[stratus]

I understand the question, however, when the ZFS on Linux Project has a documented way to use swap with ZFS, that is the road you should go down. Creating a file on a ZFS volume is going against the way ZFS is intended to be used. Therefore you may run into strange problems doing so. Ultimately, I am not sure why you would not follow the documented approach. I do not see a benefit having a file on a ZVOL that has other attributes that are not needed or may conflict or make swap useless. (Copy on Write for swap alone would be terrible)

[user3211705]

Thank you for your concerns but, I guess then you should comment that doing so is not a good solution in your opinion instead of forcing a solution that is not an answer for this question.

[mohitji]

So how to use encryption on that? One of the reasons that people want to use swapfile as a filesystem file is that it's easy, works, and also is covered by the filesystem encryption. Is it possible to encrypt ZVOL as well? Otherwise what's even the point in having encrypted OS, if all the contents will just be dumped to disk unencrypted through a swap file that attacker can then just read off offline disk?