Clear unused space with zeros (btrfs)

Question

How to clear unused space with zeros ? (I am asking for tool for btrfs filesystem)

I'm looking for something smarter than

cat /dev/zero > /mnt/X/big_zero ; sync; rm /mnt/X/big_zero

Like FSArchiver is looking for "used space" and ignores unused, but opposite site.

Purpose: I'd like to compress partition images, so filling unused space with zeros is highly recommended.

Btw. For ext3,ext4 : Clear unused space with zeros (ext3,ext4) . Here I ask for btrfs

Clarification of "smarter algorithm": As I see that "smarter algorithm" leads to confusion, here I finally decided to add line about it. Dear reader, please think about filesystem as datastructure. For simplification purposes, let's imagine self balancing red-black tree holding map, that is used as association key->value array, where values are integers. Goal is to clear all negative values with zeros. "not smart" algorithm will iterate over it and add key:value pairs to new map (with eventual altering negative to zero). Smart algorithm will just detect places where negative values are stored and clear them in-place.

Solution 1

The correct and "smart" way to clear the free space is using:

fstrim /mountpoint

Use this on a mounted filesystem.

It's important to note that there is no security advantage to this. If you wish to clear your disk for security, back up your files, zero the entire disk with a command like "shred" and then restore your files.

However the solution works fine if your disk is a thin provision on a sparse filesystem and you wish to recover the unused space.

Solution 2

Here's a modified and much expanded repost of my answer to your previous question:

sfill from the secure-delete package can do what you want.

Unlike zerofree (which works only with ext2, ext3, and ext4 filesystems), sfill will work with any file system.

e.g.

sfill -l -l -z /mnt/X

The sfill/secure-delete home page is now seems to have vanished, but it is packaged for debian and ubuntu. probably other distros too. if you need source code, that can be found in the debian archives if you can't find it anywhere else.

NOTE: sfill only works on a mounted filesystem. If you're zero-filling the image files from the host system rather than from within a VM, you'll need to mount the fs image on the host. The exact method varies depending on what kind of image file it is (e.g. qcow2 or raw).

Mounting 'raw' images is easy and straight-forward. Just use kpartx on the image file to create a loopback device and /dev/mapper/ entries for each partition can then be mounted individually.

here's a partial script modified from a fragment of the script I use to mount net-bootable freedos hard-disk images (used for bios updates on machines where flashrom won't work):

image="myrawimagefile.img"

# use kpartx to build /dev/mapper device nodes
KP=$(kpartx -a -v "$image")

# now mount each partition under ./hd/
for p in $(echo "$KP" | awk '/^add map/ {print $3}') ; do
  dm="/dev/mapper/$p"
  fp=$(echo "$p" | sed -r -e 's/^loop[0-9]+//')
  mkdir -p "./hd/$fp"
  mount "$dm" "./hd/$fp"
  sfill -l -l -v "./hd/$fp"
  umount "$dm"
done

# now remove the loopback device and /dev/mapper entries
kpartx -d "$image"

NOTE: it assumes every partition in the image is mountable. true in my use-case (a freedos hd image with one partition). not true if one of the partitions is, say, swap space. Detecting such partitions and using dd to zero the swap partition is left as an exercise for the reader :)

This method, or a variant of it, should work for LVM volumes too.

If your images are qcow then you can use the qemu-nbd tool from the qemu-utils package, which will present the image file and its partitions as network block devices - e.g. /dev/nbd0 /dev/ndb0p1 - which can be used in a similar fashion to the /dev/mapper devices above.

It's possibly easier and less hassle (but more time-consuming) to use qemu-img to convert from qcow to raw, use the method for raw above, and then convert the modified raw image back to compressed qcow2. This will probably result in slightly smaller images than using qemu-nbd as you'll be compressing the new qcow2 image as it is being created.

Comments

[Kevin]

Did you try the sfill suggestion? It looks like it doesn't require extX.

[Zlatko]

1. I know you can fill existing files with zeros with shred, but this probably isn't it, right? 2.How are you making those images? Maybe you can do the truncation there?

[Alessio]

@kevin: that's why i posted the sfill answer in the original question. i rarely use ext2,3,4 filesystems myself so i find utilities that work only on ext* to be annoyingly useless.

[Grzegorz Wierzowiecki]

I need something "smart". Something what only fills empty space with zeros without touching other places - I mean without affecting internal data-structures, like B-Trees (If your are curious, I've written in question "compression" but I need it also for other purposes, like fs images de-duplication, analysis and research)

[mikeserv]

just use truncate and/or fallocate

[Alessio]

adding error checking would also be a good idea. in particular, don't run sfill -l -l -v "./hd/$fp" if the previous mount failed. that's why i called it only a partial script - it's not complete or ready for use as-is.

[Grzegorz Wierzowiecki]

Could you ensure me if this tool fills free space, until possible, and then deletes file ? (in nutshell, I know it's for "wiping" and supports multi-passes, I want to ensure about basic idea of clearing with zeros)

[Alessio]

To the best of my knowledge, that's what it does. the -l -l make it just zero the empty space without the multiple passes of random data. For quick testing, I suggest making a small (say, 100MB) disk image, mounting it, and running sfill -l -l on it.

[Grzegorz Wierzowiecki]

What I mean is, that it is not considered "smart" way of filling unused space with zeros. It's most obvious, typical way of doing this. I am not asking about things like that.

[Alessio]

Please describe a "smart" algorithm that isn't specific to a particular filesystem family (like zerofill is for ext2/3/4). You wanted a program that could do the job. You didn't say it had to be magical as well.

[Alessio]

and, really, if the filesystem is not in use (because, e.g., it's mounted on the host for zerofilling) what does it matter how smart the algorithm is? Trawling through the fs metadata looking for empty sectors isn't going to be any faster or better than just writing a huge fill full of NULs...the filesystem itself is already plenty smart enough for figuring out where empty space is.

[Grzegorz Wierzowiecki]

On the one hand, for your needs it's enough - so my question already contains code snippet with this solution - so your answer is not providing anything new. On the other hand, my needs require what I specify, because I want to wipe unused space with zeros without modifying internal structures (for example rebalancing "Dancing B-Trees" or other kind of data-structure (e.g. for analyses and de-duplication purposes)

[Fake Name]

@cas - I think you're missing the fact that zero-filling is an essential step before compacting virtual-hard-drives in VMs. If you have a 500 GB expanding disk, with 10 GB of files and 10 GB of "empty" space (i.e. deleted files), running the naive "giant zero file" approach will take FAR longer, and require the VHD to bloat up to the full 500 GB. I'd bet the OP is assuming the utility has to be FS specific too.

[Fake Name]

FWIW, I'm actually currently looking for something similar to what the OP wants. I need something like zerofill that runs on a mounted filesystem. I have a bundle of VMs, and having to reboot them all into a different environment, zerofill the disks, and then compress is not an acceptable solution.

[Nicolas Mommaerts]

@FakeName: That's a great point, but zerofill itself also writes already-zero'd sectors, so both take equally long. I'd also be interested in something that read first, then wrote, but it'd be low-level and anything that writes raw sectors is very dicey to run online while anything else could be writing.

[Fake Name]

@SilverbackNet - If zerofill does that, well, that's extremely annoying. It seems like there's a decent opportunity for a tool that only zeroizes non-zero sectors.

[Fake Name]

Also, online != mounted RW. I seem to remember having issues running zerofill from a partition remounted RO, but it's been a while since I've done any VM hygiene stuff (I gave up and bought some bigger SSDs).

[Alessio]

and that could be optimised to only convert image files where the file size is significantly greater than the filesystem usage (e.g. mount using qemu-nbd, run df, unmount and compare with "disk_size" field from qemu-img info)

[Alessio]

btw, adding -c to the qemu-img convert command above took 1m46s to also compress the output image 768MB (a considerable saving from the original 9.1GB, or the uncompressed 1.7GB). See man qemu-img and search for "convert" for info about interaction between compression and COW.