How to Free Up Space in a Dynamically Growing qcow2 VM image?

Question

This has happened to one of my VM's that's running CentOS under Proxmox.

I copied about 40 GB of data into a VM. I then deleted that 40 GB data a day later. Now taking a backup of the same VM takes over 30 minutes and over 40 GB in size when there is only 9 GB in the VM.

Questions

• So how can I actually free up that space after that file deletion?
• Is there a command that I can run on the VM to free up that space?
• A command on the hypervisor's level to free it up?

Solution

qcow2 works in a grow-only manner, yet it is actually rather understandable. Once the guest OS requests a write to a previously unallocated block (unallocated by the VM in the qcow2 file), the VM allocates some space on the host filesystem and uses it to store information for the guest. When the guest OS removes the file it doesn't tell the VM, that it doesn't need the block any more. And even if it did, you have to remember that for performance reasons, the blocks are allocated in groups - i.e. if the guest OS requests 10 512B blocks usually a much bigger space is allocated; see the cluster_size option in the qemu-img(1) man page. This means, that once an adjacent block is written to, it is already allocated. Now consider the case, when the guest writes two files in the same "cluster". Both would have to be removed to reclaim the space in the VM image file.

That said, there is a way to reclaim the space, although I'm not sure it can be while a VM is running from the image. The procedure is:

In the VM

1. remove what you don't want any more
2. defragment the filesystem

3. fill the free space with zeros, e.g. with

   dd if=/dev/zero of=/tmp/zeros
   

4. make sure the zeros are sent to the (virtual) disk (i,e, not cached by the guest OS) - they must appear in the VM image

5. remove the created file
6. shut down the system (might not be necessary)

Then repack the VM image on the host system using qemu-img convert (from qcow2 to qcow2) on the host system. You can use the -S option to specify how big a zeroed block has to be, to be considered for optimization using sparse file. Also pay attention to -o preallocation=off convert option to prevent full-size file creation at start.

This obviously means that at some point you will need something above (maximum size of the image) + (actual size of the image) free space on the host. On the other hand it is probably the only way to ensure maximum compactification of the final image.

You can also try to zero each file you want to remove prior to actually removing it, for example with shred -n 0 -z. However, you won't be able to reclaim as much space as in the full-disk case.

Additionally, on some file systems this may not have the desired output. For example with btrfs you are doomed anyway, since it has its own copy-on-write layer – unless you disable it in the guest (and I'm not sure it is even possible, since many of this file system's features depend on it), you are out of luck.

Comments

[fcukinyahoo]

now it all makes sense, so clear, thanks. I ll attempt the solution on one of the dummy VM and see where I get.

[hildred]

you can do the same thing with tar

[phoeagon]

@hildred No. Tar doesn't fix MBR for you. (If you are willing to manually fix the partition table and whatsoever you can just mount it on host and rsync)