Removing files on ZFS does not free disc-space

Question

I'm trying to free space on a ZFS by removing files. They are large (>1GB) so it should show in the numbers. Unfortunately, after unlinking the files, df does not show any increase in available space. It seems as if the removed files have hardlinks somewhere (but ls shows hardlink counters of 1 for all files unless I create a hardlink manually (which I tested)).

The script removing the files opens, reads, and closes these files (I checked that), so there are no open file handles. Afterwards it uses unlink() to remove the files. ls also does not show them anymore. Just the available disk space does not increase.

We are using the ZFS feature of snapshots. Maybe this is a reason why nothing is freed?

Any ideas or hints to documentation concerning my problem are welcome.

Solution

According to Oracle's zfs documentation,

A snapshot is a read-only copy of a file system or volume. Snapshots can be created almost instantly, and they initially consume no additional disk space within the pool. However, as data within the active dataset changes, the snapshot consumes disk space by continuing to reference the old data, thus preventing the disk space from being freed.

If you want the disk space to be freed, you will have to clear out the snapshots referencing that file. You may also want to read up on copy-on-write, which is how zfs works.

I don't actually have any experience with zfs, but I make use of btrfs. btrfs does snapshots in a similar manner to zfs and the exact same 'issue' appears there. Of course, as I do not want a deleted file in the current working set to be removed from a snapshot, I would certainly expect that the available disk space would remain constant when removing a file. After all, the file isn't actually removed until all references (across all snapshots) are removed.

Comments

[ChrisInEdmonton]

If you have a snapshot containing the file, that would certainly explain this issue. Have you tried removing all snapshots?

[Alfe]

No, of course not. I would like to keep those, of course, and just because I think that this might be connected I won't remove them just to check the idea. I also find it weird that the snapshots should be part of the file system. They are on the disk, of course, but why should they be reported as filling the FS space? But if this is so with certainty, then I'd like to know anyway ;-)

[Alfe]

Yeah thanks, that seems to state this clearly. I will still have a test on removing the snapshots. If this also does not help, I will come back ;-)

[ChrisInEdmonton]

Good luck! FYI, a common use of snapshots is to keep them around only long enough to perform a backup, then get rid of them.