Ubuntu 20.04 thinks it's out of free space, but it isn't
5,645
Ubuntu 20.04 introduced experimental ZFS support with zsys (ZFS SYStem integration) tool. It creates snapshots on most apt operations for that you will be able to rollback a previous state if something fails.
Ubuntu has a ZFS system tool called Zsys, which provides automated system and user state saving. It also integrates better with GRUB so a user can revert to an earlier system state before booting into the desktop. (via)
Downside is of course that these will also eat up your available space.
Use zfs list -t snapshot to list the snapshots:
$ zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
...
rpool/example@snap1 - - - -
...
and zfs destroy to delete a snapshot, e.g.:
zfs destroy rpool/example@snap1
Read also the official tutorial.
There is also a similar question with not many answers yet on how to disable automatic snapshot creation and another one asking for some official documentation (I couldn't find anything either).
Author by
Thomas
Updated on September 18, 2022Comments
-
Thomas over 1 year
I started to get "not enough free space" errors when trying to copy files, and at some point I had trouble booting and did a ZFS rollback. I still get space errors despite (according to
dfanddubelow) <60% drive space being used.The answer below points to zfs snapshots taking up too much space, and it's a helpful start. But
sudo du /.zfs/snapshot/ -h --max-depth=0returns only60G. Are there 140G of snapshots elsewhere? (or what else could it be?)Also conspicuously
zfs list -t snapshotdoesn't show the (biggest and I assume most defunct?) datasetrpool/USERDATA/thomas_fv2wpzwhich is shown inzfs listoutput below to be278G.Here's a table of contents for outputs:
uname -azfs listzfs list -o space rpool/USERDATA- Should I omit (some/parts of) the following long outputs? this my first time asking a question.
df -hdu -h --max-depth=1lsblk -f
$ uname -a Linux thomas-Latitude-E6530 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux$ zfs list NAME USED AVAIL REFER MOUNTPOINT bpool 272M 1.48G 96K /boot bpool/BOOT 270M 1.48G 96K none bpool/BOOT/ubuntu_nhdx6z 270M 1.48G 180M /boot bpool/BOOT/ubuntu_smztg7 8K 1.48G 180M /boot rpool 441G 4.89G 96K / rpool/ROOT 8.97G 4.89G 96K none rpool/ROOT/ubuntu_nhdx6z 8.51G 4.89G 3.69G / rpool/ROOT/ubuntu_nhdx6z/srv 264K 4.89G 96K /srv rpool/ROOT/ubuntu_nhdx6z/usr 872K 4.89G 96K /usr rpool/ROOT/ubuntu_nhdx6z/usr/local 776K 4.89G 144K /usr/local rpool/ROOT/ubuntu_nhdx6z/var 3.24G 4.89G 96K /var rpool/ROOT/ubuntu_nhdx6z/var/games 152K 4.89G 96K /var/games rpool/ROOT/ubuntu_nhdx6z/var/lib 3.10G 4.89G 2.74G /var/lib rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService 688K 4.89G 104K /var/lib/AccountsService rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager 1.76M 4.89G 164K /var/lib/NetworkManager rpool/ROOT/ubuntu_nhdx6z/var/lib/apt 70.1M 4.89G 53.6M /var/lib/apt rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg 97.8M 4.89G 39.4M /var/lib/dpkg rpool/ROOT/ubuntu_nhdx6z/var/log 138M 4.89G 36.4M /var/log rpool/ROOT/ubuntu_nhdx6z/var/mail 152K 4.89G 96K /var/mail rpool/ROOT/ubuntu_nhdx6z/var/snap 1.31M 4.89G 240K /var/snap rpool/ROOT/ubuntu_nhdx6z/var/spool 848K 4.89G 120K /var/spool rpool/ROOT/ubuntu_nhdx6z/var/www 152K 4.89G 96K /var/www rpool/ROOT/ubuntu_smztg7 471M 4.89G 3.18G / rpool/ROOT/ubuntu_smztg7/srv 56K 4.89G 96K /srv rpool/ROOT/ubuntu_smztg7/usr 144K 4.89G 96K /usr rpool/ROOT/ubuntu_smztg7/usr/local 144K 4.89G 144K /usr/local rpool/ROOT/ubuntu_smztg7/var 203M 4.89G 96K /var rpool/ROOT/ubuntu_smztg7/var/games 112K 4.89G 96K /var/games rpool/ROOT/ubuntu_smztg7/var/lib 177M 4.89G 2.65G /var/lib rpool/ROOT/ubuntu_smztg7/var/lib/AccountsService 144K 4.89G 104K /var/lib/AccountsService rpool/ROOT/ubuntu_smztg7/var/lib/NetworkManager 320K 4.89G 148K /var/lib/NetworkManager rpool/ROOT/ubuntu_smztg7/var/lib/apt 4.08M 4.89G 53.3M /var/lib/apt rpool/ROOT/ubuntu_smztg7/var/lib/dpkg 14.0M 4.89G 39.4M /var/lib/dpkg rpool/ROOT/ubuntu_smztg7/var/log 24.9M 4.89G 15.0M /var/log rpool/ROOT/ubuntu_smztg7/var/mail 112K 4.89G 96K /var/mail rpool/ROOT/ubuntu_smztg7/var/snap 212K 4.89G 188K /var/snap rpool/ROOT/ubuntu_smztg7/var/spool 160K 4.89G 120K /var/spool rpool/ROOT/ubuntu_smztg7/var/www 112K 4.89G 96K /var/www rpool/USERDATA 432G 4.89G 96K / rpool/USERDATA/helper_zthl10 180K 4.89G 124K /home/helper rpool/USERDATA/root_fv2wpz 316K 4.89G 1.12M /root rpool/USERDATA/root_mhe8ya 1.78M 4.89G 992K /root rpool/USERDATA/thomas_fv2wpz 971M 4.89G 278G /home/thomas rpool/USERDATA/thomas_mhe8ya 431G 4.89G 217G /home/thomas$ zfs list -o space rpool/USERDATA NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD rpool/USERDATA 4.88G 432G 0B 96K 0B 432G$ df -h Filesystem Size Used Avail Use% Mounted on udev 5.8G 0 5.8G 0% /dev tmpfs 1.2G 2.1M 1.2G 1% /run rpool/ROOT/ubuntu_nhdx6z 8.8G 3.7G 5.1G 42% / tmpfs 5.8G 253M 5.6G 5% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup rpool/USERDATA/helper_zthl10 5.1G 128K 5.1G 1% /home/helper bpool/BOOT/ubuntu_nhdx6z 1.7G 180M 1.5G 11% /boot rpool/USERDATA/thomas_fv2wpz 284G 279G 5.1G 99% /home/thomas rpool/USERDATA/root_fv2wpz 5.1G 1.0M 5.1G 1% /root rpool/ROOT/ubuntu_nhdx6z/srv 5.1G 128K 5.1G 1% /srv rpool/ROOT/ubuntu_nhdx6z/var/games 5.1G 128K 5.1G 1% /var/games rpool/ROOT/ubuntu_nhdx6z/var/log 5.2G 29M 5.1G 1% /var/log rpool/ROOT/ubuntu_nhdx6z/var/lib 7.9G 2.8G 5.1G 35% /var/lib rpool/ROOT/ubuntu_nhdx6z/usr/local 5.1G 256K 5.1G 1% /usr/local rpool/ROOT/ubuntu_nhdx6z/var/mail 5.1G 128K 5.1G 1% /var/mail rpool/ROOT/ubuntu_nhdx6z/var/www 5.1G 128K 5.1G 1% /var/www /dev/sda1 511M 7.1M 504M 2% /boot/efi rpool/ROOT/ubuntu_nhdx6z/var/snap 5.1G 256K 5.1G 1% /var/snap rpool/ROOT/ubuntu_nhdx6z/var/spool 5.1G 128K 5.1G 1% /var/spool rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService 5.1G 128K 5.1G 1% /var/lib/AccountsService /dev/loop2 62M 62M 0 100% /snap/caprine/36 /dev/loop3 55M 55M 0 100% /snap/core18/1754 /dev/loop4 176M 176M 0 100% /snap/gimp/273 /dev/loop0 9.2M 9.2M 0 100% /snap/canonical-livepatch/95 /dev/loop5 241M 241M 0 100% /snap/gnome-3-34-1804/24 rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg 5.2G 40M 5.1G 1% /var/lib/dpkg rpool/ROOT/ubuntu_nhdx6z/var/lib/apt 5.2G 54M 5.1G 2% /var/lib/apt rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager 5.1G 256K 5.1G 1% /var/lib/NetworkManager /dev/loop7 174M 174M 0 100% /snap/gimp/252 /dev/loop6 94M 94M 0 100% /snap/core/9066 /dev/loop1 55M 55M 0 100% /snap/core18/1705 /dev/loop8 164M 164M 0 100% /snap/spotify/41 /dev/loop9 256K 256K 0 100% /snap/gtk2-common-themes/9 /dev/loop10 143M 143M 0 100% /snap/slack/23 /dev/loop11 256M 256M 0 100% /snap/gnome-3-34-1804/33 /dev/loop13 63M 63M 0 100% /snap/gtk-common-themes/1506 /dev/loop12 28M 28M 0 100% /snap/snapd/7264 /dev/loop14 50M 50M 0 100% /snap/snap-store/454 /dev/loop15 142M 142M 0 100% /snap/zoom-client/83 /dev/loop16 161M 161M 0 100% /snap/gnome-3-28-1804/116 /dev/loop17 87M 87M 0 100% /snap/shotcut/64 /dev/loop18 291M 291M 0 100% /snap/vlc/1620 /dev/loop19 140M 140M 0 100% /snap/slack/24 /dev/loop20 50M 50M 0 100% /snap/snap-store/433 /dev/loop21 145M 145M 0 100% /snap/zoom-client/84 tmpfs 1.2G 52K 1.2G 1% /run/user/1000 /dev/loop22 64M 64M 0 100% /snap/caprine/37$ du -h --max-depth=1 25K ./Music 1.0K ./.ssh 1.0K ./Templates 36K ./.gnome 30M ./.thunderbird 52M ./Documents 368M ./snap 8.4M ./.mozilla 203G ./Pictures 302M ./.config 1021M ./.cache 1.0K ./Public 9.5G ./Downloads 101M ./.local 7.0K ./Steam 41G ./Videos 28K ./.gnupg 7.6G ./Desktop 16G ./.steam 17K ./.pki 11K ./.psensor 279G .$ lsblk -f NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT loop0 squashfs 0 100% /snap/canonical-livepatch/95 loop1 squashfs 0 100% /snap/core18/1705 loop2 squashfs 0 100% /snap/caprine/36 loop3 squashfs 0 100% /snap/core18/1754 loop4 squashfs 0 100% /snap/gimp/273 loop5 squashfs 0 100% /snap/gnome-3-34-1804/24 loop6 squashfs 0 100% /snap/core/9066 loop7 squashfs 0 100% /snap/gimp/252 loop8 squashfs 0 100% /snap/spotify/41 loop9 squashfs 0 100% /snap/gtk2-common-themes/9 loop10 squashfs 0 100% /snap/slack/23 loop11 squashfs 0 100% /snap/gnome-3-34-1804/33 loop12 squashfs 0 100% /snap/snapd/7264 loop13 squashfs 0 100% /snap/gtk-common-themes/1506 loop14 squashfs 0 100% /snap/snap-store/454 loop15 squashfs 0 100% /snap/zoom-client/83 loop16 squashfs 0 100% /snap/gnome-3-28-1804/116 loop17 squashfs 0 100% /snap/shotcut/64 loop18 squashfs 0 100% /snap/vlc/1620 loop19 squashfs 0 100% /snap/slack/24 loop20 squashfs 0 100% /snap/snap-store/433 loop21 squashfs 0 100% /snap/zoom-client/84 loop22 squashfs 0 100% /snap/caprine/37 sda ├─sda1 vfat 105C-066C 503.9M 1% /boot/efi ├─sda2 ├─sda5 swap ca95a35a-589b-4127-91d0-ba74f05c7011 [SWAP] ├─sda6 zfs_member bpool 10888252588400514123 └─sda7 zfs_member rpool 8231086401312639965 sr0-
Admin almost 4 yearsPlease add the output ofdf -h, also maybe add-hto yourducommand ...203Gis more readable than212714827, also your home directory is not the only one of interest..., depending on the programs you have installed there might be a lot of data in/varor/tmp -
Admin almost 4 yearsDo you know
baobab? Its a GUI Tool to analyse the data usage on the whole partition or on a folder. Withdu --max-depth=1you only check the current folder. An other nice script islsblk -fthat gives you an overwiev over all partitions. -
Admin almost 4 yearsRun sudo gparted and make sure there's no exclamation mark shown by the root partition. If you haven't got gparted then install it.
-
Admin almost 4 yearsThanks @pLumo. Question was updated with both of those suggestions. Also: $ sudo du -h --max-depth=1 for /var and /tmp show them to be 3.5G and 70K respectively (can post code if that's helpful).
-
Admin almost 4 yearsThanks @BobaFit. I have used the Disk Usage Analyser. It thinks that / is only 9.4G total, and (similar to df) doesn't show the true space of the drive. not sure if it's relevant, but rpoo and bpool are both unmounted.
-
Admin almost 4 yearsThanks @PaulBenson. gparted shows root to be exclamation point -less (pun?).
-
Admin almost 4 years@Thomas Did you check if gparted also showed the root partition as full (ie > 60% used)?
-
Admin almost 4 years@PaulBenson see gparted output. also added
zpool list, which is telling and I'm investigating further... -
Admin almost 4 yearsAnd the output of
zfs list? -
Admin almost 4 yearssee also reddit.com/r/zfs/comments/4v8wc2/…
-
Admin almost 4 years
zfs list -t snapshotandzfs list -o space rpool/USERDATA -
Admin almost 4 years@pLumo updated.
-
emk2203 almost 4 yearsCorrect answer. The snapshots are hidden with the Ubuntu setup, which leads to this confusion. Had to learn this myself. -
Thomas almost 4 yearsjust destroyed all but a handful of snapshots. disk usage and free space look the same.
-
Michael almost 3 yearsThis seems borderline dangerous... I only accidentally saw that snapshots weren't being made during an update from the GUI because I clicked the thingy to see what what going on, and even then luckily happened to be glancing at the output and managed to remember the error message long enough to search Google for it. If any of that had not happened I would still be in the dark about snapshots no longer working, and as it is, supposing the last update left my system in an unbootable state I would have to go back an additional snapshot to recover since it didn't just make one.
-
Fakhamatia about 2 yearsHow remove only large snapshots?
