How to compact VirtualBox's VDI file size?
Solution 1
You have to do the following steps:
-
Run defrag in the guest (Windows only)
-
Nullify free space:
With a Linux Guest run this:
dd if=/dev/zero of=/var/tmp/bigemptyfile bs=4096k ; rm /var/tmp/bigemptyfile
Or:
telinit 1 mount -o remount,ro /dev/sda1 zerofree -v /dev/sda1
With a Windows Guest, download SDelete from Sysinternals and run this:
sdelete.exe c: -z
(replace C: with the drive letter of the VDI)
-
Shutdown the guest VM
-
Now run VBoxManage's
modifymedium
command with the--compact
option:With a Linux Host run this:
vboxmanage modifymedium --compact /path/to/thedisk.vdi
With a Windows Host run this:
VBoxManage.exe modifymedium --compact c:\path\to\thedisk.vdi
With a Mac Host run this:
VBoxManage modifymedium --compact /path/to/thedisk.vdi
VBoxManage is located here: /Applications/VirtualBox.app/Contents/MacOS/VBoxManage
This reduces the vdi size.
Solution 2
I'm on a Windows 7 host with Windows guests, Here is a batch file I wrote to Compact all of the VDIs in a folder tree
echo off
mode con:cols=140 lines=200
cls
:: see https://forums.virtualbox.org/viewtopic.php?p=29272#p29272
:: How can I reduce the size of a dynamic VDI on disk?
:: but that page says to use sdelete -s which is suboptimal.
:: use -z as per http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx
:: First run the sdelete -z c: inside the VMs that zero-out all the free space
:: THEN run this batch file
Title Compacting Free space on Virtual Machine VMs
:: http://ss64.com/nt/for_r.html
:: http://stackoverflow.com/questions/8836368/windows-batch-file-how-to-loop-through-files-in-a-directory/8836401#8836401
Setlocal EnableDelayedExpansion
:: http://ss64.com/nt/delayedexpansion.html ...
:: Notice that within the for loop we use !variable! instead of %variable%.
For /R %CD% %%G IN (*.vdi) DO (
set ohai=%%G
set lastfive=!ohai:~-5!
:: Skip snapshots which are named {guid}.vdi
if NOT !lastfive!==}.vdi (
echo .
echo Compacting %%G
"C:\Program Files\Oracle\VirtualBox\VboxManage.exe" modifyhd "%%G" --compact )
)
pause
exit
I left the links in the comments so you can (sort of) tell how it works.
edit
Well, after all that, I tried the CloneVDI tool and it did a good job in much less time and in one click.
Solution 3
Debian guest on Windows host using discard/TRIM.
This isn't a direct answer per se, as I'm addressing the problem, not the question. Instead of periodically compacting the image, this solution uses discard to automatically remove unused blocks in the host's VM disk image.
This solution requires a guest filesystem that supports continuous TRIM. The Arch Linux wiki has a list of filesystems supporting TRIM operations.
FDE and cryptoroot are specifically not covered, as there are security concerns and none of the other solutions to this question would allow compacting either. The Arch Linux wiki has information about TRIM and dm-crypt devices.
In theory, this will work for all Linux guests on VBox hosts using VDI storage.
Host configuration
With VBox exited and no VMs running, add discard support to your disks by setting both discard
and nonrotational
for each disk in the config file for the VM. At this time discard
is not in the GUI, but nonrotational
is exposed as the "Solid-state Drive" checkbox. (ref: vbox forums, discard support)
<AttachedDevice discard="true" nonrotational="true" type="HardDisk" [..other options..] >
Boot the VM up, and verify that TRIM support is enabled:
sudo hdparm -I /dev/sda | grep TRIM
Guest Configuration
If LVM is in use, change the discard setting in /etc/lvm/lvm.conf
. (ref: debian wiki, lvm.conf example)
devices {
...
issue_discards = 1
}
In fstab, add the discard
option to the filesystems you wish to auto-discard (ref: debian wiki, fstab example)
UUID=8db6787f-1e82-42d8-b39f-8b7491a0523c / ext4 discard,errors=remount-ro 0 1
UUID=70bfca92-8454-4777-9d87-a7face32b7e7 /build ext4 discard,errors=remount-ro,noatime 0 1
Remount the filesystems to have them pick up their new options.
sudo mount -o remount /
sudo mount -o remount /build
Manually trim free blocks now with fstrim
. fstrim
uses the mounted filesystem, not the block device backing it. Instead of setting continuous discard in fstab
, this could be done on a weekly cron. (The weekly cron is recommended for physical SSDs which may have questionable support for TRIM, but this is not relevant here since underlying SSDs are handled by the host OS. see: ssd trim warning).
fstrim /
fstrim /build
At this point, the size of the filesystems inside the VM and the size of the VM images should be pretty close in value.
Tested with:
- Guest1: Debian 8.7, kernel: linux 4.8 grsec from backports, filesystem: ext4
- Guest2: Debian 9 RC2, kernel: linux 4.9, filesystem: ext4
- Host1: VBox 5.1.14, Win7, image fmt: VDI
- Host2: VBox 5.1.14, Win8.1, image fmt: VDI
Solution 4
For MacOS Guest do this:
Nullify free space in guest system:
diskutil secureErase freespace 0 "/Volumes/Macintosh HD"
(replace /Volumes/Macintosh HD with your drive name)
Shutdown the guest VM
Run this command to reduce VDI disk image size
VBoxManage modifyhd /path/to/thedisk.vdi --compact
OR
VBoxManage modifymedium /path/to/thedisk.vdi --compact
Solution 5
IMPORTANT NOTE FOR LEGACY (~1997-2007) OPERATING SYSTEMS
In general, the techniques in the answers previously given are valid; HOWEVER, there is a very important special case.
For a period of some years-- perhaps 1997-2007 or so-- 32-bit operating systems were still the norm, but hard disks larger than 2GB were already in use. As a result, when attempting to consume all free space by writing a file of zeroes (which should always be done as root, to include root's privileged free space, which no one else can touch), you may see:
File too large
instead of what you expect:
No space left on device.
If this occurs, you have most likely hit a 2GB file size limitation. This was common at the time because many file operations returned results in signed 32-bit integers, so that negative values could report error codes. This effectively meant that offset results were limited to 2^31 bytes without special measures.
The workaround is straightforward: keep creating separate, differently-named zeroing files until the disk actually runs out of space.
If you are an instructor wishing to demonstrate this situation for a class, a 4GB disk image with an old copy of Red Hat Linux 7.0 is sufficient.
Related videos on Youtube
Comments
-
DeepNightTwo over 1 year
I've a VirtualBox VM which configured a very large hard disk size (bigger than host). By my mistake, a program on the VM generated lots of log files and the VDI file size keeps growing until there is no space on the host.
Now I've deleted the log files but the VDI file size are not getting smaller after using
VBoxManage.exe modifyhd "C:\Virts\mybox-i386.vdi" compact
Is there a way to really compact the VDI file size? Thanks!
-
Gilles 'SO- stop being evil' over 10 years@user248749 FYI,
dd
with no argument copies one block at a time, not one byte. Using a larger block size can speed things up, but the effect is pretty minor. -
Daniel over 10 yearsFor the next person, my command ended up looking like this: "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "C:\Users\daniel\VirtualBox VMs\....\thedisk.vdi" --compact
-
magicandre1981 about 10 years@Dakatine I never got it working. Add a working command to the post.
-
GiggyLapisar about 10 yearsIt's fun that the the manpage of zerofree states that with dd other concurrent writes will fail, but zerofree needs the filesystem to be mounted read-only! *duh*
-
Frozen Flame almost 10 yearsWhat is
/bigemptyfile
? A special path recognized bydd
? -
magicandre1981 almost 10 yearsen.wikipedia.org/wiki//dev/zero this is simple a large file with zeros
-
Jason Larke almost 10 years@FrozenFlame /bigemptyfile is just an arbitrary name that magicandre1981 used for the temporary file, you can name it anything you want.
sudo dd if=/dev/zero of=/zero.fill bs=4096k
is functionally exactly the same, it will just write to a file namedzero.fill
instead (you'd need to change therm
command to remove this file instead of/bigemptyfile
though) -
jlh almost 10 yearsTip: Put the two commands on one line like so:
dd ...; rm /bigfile
, this will minimize the time with a full disk in case you're not waiting for thedd
to complete. -
jlh almost 10 years@Dakatine Using VirtualBox 4.3.10, the disk image file did not grow to its maximal extent. VirtualBox is clever enough to not bother writing all zero blocks to the physical disk.
-
Reinaldo Gil almost 10 yearsthis dd trick doesn't work for ext3 guest filesystem, but works well for ext4
-
Vincent over 9 yearsIs it normal to have this error? dd: error writing ‘/bigemptyfile’: No space left on device
-
CAD bloke over 9 yearsYou would think on this sort of site there would be some kind of syntax highlighting for DOS but no. It looks much prettier in Notepad++
-
gronostaj almost 9 years@Vincent That's the point of this procedure. You're writing zeros to a new file until it fills entire free space, then you delete that file.
-
Anthony O. almost 9 yearsOn Linux, you can defrag this way: using gparted, shrinking then moving the partition (see the link for more information)
-
Cristian Baciu over 8 years@CAD_bloke that would require a parsing engine and when you consider the number of different languages posted on SE you are looking at a HUGE project. Just think how many versions and dialects of DOS there are for example and that's before you even get to Linux etc.
-
CAD bloke over 8 yearsVery good point. Ironically it is highlighted on the stack exchange iOS app.
-
Robert Achmann over 8 yearsOSX Host - Win8.1 VM - When I try to optimize my C drive, it tells me that optimization is unavailable. - how can I defrag my virtual C drive?
-
magicandre1981 over 8 years@RobertAchmann post a picture of the defrag UI of Win8.1
-
Robert Achmann over 8 yearsI will in the morning, but both the analyze and optimize buttons are greyed out - and beside the drive in the list it says optimization is not available. but it looks as though you can schedule optimization to occur...
-
magicandre1981 over 8 years@RobertAchmann ignore it and run sdelete now.
-
Admin over 8 yearsYeah CloneVDI is much better and faster way for personal use
-
Vasilly.Prokopyev over 8 yearsIf you experience
/ is busy
onmount
command, then reboot your guest in recovery mode. -
Urhixidur about 8 yearsNote that the zerofree man page says the "filesystem has to be unmounted or mounted read-only", so changing the run level with telinit then remounting the device read-only is wasted effort. Just unmount, zerofree, shut down, compact.
-
Katu almost 8 yearsThe
dd
operation will take a while, you can see its progress installingpv
and usingsudo dd if=/dev/zero | pv | sudo dd of=/bigemptyfile bs=4096k
. Source: askubuntu.com/questions/215505/… -
Florin Giurgiuman almost 8 yearsAwesome! Down to 15Gb from 27Gb
-
Francesco Frassinelli almost 8 years
sdelete
requires a target as argument. Examplesdelete.exe c: -z
-
magicandre1981 almost 8 yearsok there was an update, before it used the current drive. I'll add it.
-
Andrea Lazzarotto over 7 yearsI suggested an edit to this post but it seems it has been somehow rejected. Anyway, the command is
VBoxManage
also on Linux, notvboxmanage
. -
PolyTekPatrick over 7 years@Andrea Lazzarotto actually both VBoxManage and vboxmanage exist on my Linux box with VirtualBox 5.0.24 installed.
-
Andrea Lazzarotto over 7 yearsAh! That's weird, I am pretty confident it was not working for me when I first tried to edit the answer. Maybe I was using the Ubuntu version. I am now using the version from their website so I cannot really check.
-
f.ardelian over 7 years
telinit 1
didn't work for me in Ubuntu, so I rebooted holding down the left shift to get the grub menu, I booted in recovery mode and in the recovery menu I selected the root shell. After that themount
and thezerofree
commands worked like a charm. VM reduced from 25 GB to the expected 11 GB. -
Marcus over 7 yearsAs a reference I had to delete my snapshots in order for this to make any difference.
-
magicandre1981 over 7 years@VlastimilBurian thanks for the edit about the new options
-
LinuxSecurityFreak over 7 years@magicandre1981 No problem, as I use VirtualBox daily, I have just contributed to your good answer.
-
Alfabravo over 7 yearsFor a WindowsServer/7 guest on linux host, I used ccleaner to nullify empty space and then used your answer to compact vdi file. Worked great!
-
sbleon over 6 yearsGreat answer! This works well. I just wanted to point out that you don't need
sudo
when writing zeros to disk on a Linux guest. Just write to /tmp/bigemptyfile instead of /bigemptyfile:dd if=/dev/zero | pv | dd of=/tmp/bigemptyfile bs=4096k; rm /tmp/bigemptyfile
-
magicandre1981 over 6 years@sbleon ok, edit the answer and add this change. I don't use linux that much
-
magicandre1981 over 6 years@sbleon thanks, I've approved your edit. thanks for pointing this out
-
Victor Yarema over 6 years@AndreaLazzarotto as PolyTekPatrick already mentioned there are both versions (
vboxmanage
andVBoxManage
) of the tool. Even more, they are just symlinks to a real executable: commandls -o $( which vboxmanage VBoxManage )
produceslrwxrwxrwx 1 root 4 Sep 13 16:37 /usr/bin/VBoxManage -> VBox
andlrwxrwxrwx 1 root 4 Sep 13 16:37 /usr/bin/vboxmanage -> VBox
. -
Andrea Lazzarotto over 6 years@Victor, when I left my previous comment (and we are talking about more than a year ago) there was no lowercase version on my Ubuntu system. Only the uppercase one was present.
-
Victor Yarema over 6 years@AndreaLazzarotto, copy that. I didn't want to say that you are wrong. I know that things change. I just wanted to add some up to date info (including command example) for anyone that may come here for answers. Your comment was useful at that moment. Thanks.
-
Merc over 6 yearsIf you create a file full of zeroes till ALL of the space is taken, won't the file grow to its absolute maximum size? My disk size is 100Gb, whereas the actual size is 15Gb. I know it should be around 9, If I run the command, will I have to wait for
dd
to write 85Gb?!? -
Bosko Mijin over 6 yearsExcellent! From 187GB to 46GB! I am confirming that this procedure works on Ubuntu 17.10 as host OS and Windows 10 as guest OS.
-
atomh33ls over 6 yearsA similar
dd
command suggested here; askubuntu.com/a/903178/516133 -
mgershen almost 6 yearsHad to remove my shared folder first. Worked great after that.
-
vehsakul over 5 yearsCan somebody explain why defrag is needed?
-
Tobias Weibel over 5 yearszerofree: Worked with Win10 host, Ubuntu 18.04 guest and Ubuntu 18.04 live CD/iso. Please note you need plenty of free disk space on the host, otherwise VM stops working!
-
Mooseman about 5 yearsIt's worth noting that Windows 10 guests will not defrag a disk tagged as an SSD. The rest of this worked for me. Thanks.
-
Wang about 5 yearsBe honest, I do not know why this is accepted answer. If I have 500GB VDI, only used about 100GB in win guest. The VDI is about 150GB. The sdelete will actually enlarge the VDI to 500GB. And after compact it only shrink down about 480GB. So it actually make the VDI larger. Just clone the 150GB VDI and run compact on it will shrink down to 120GB. I really want to middle finger the guy who did not test this on recent NTFS at all. Wasting lots of time and energy. This works on linux guest only!
-
Ed Randall almost 5 years
VBoxManage: error: Cannot register the hard disk '/path/to/thedisk.vdi' {eee104d7-cd65-402f-b816-a3be4ac30eb3} because a hard disk '/path/to/thedisk.vdi' with UUID {eee104d7-cd65-402f-b816-a3be4ac30eb3} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox, callee nsISupports
VBoxManage: error: Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, enmAccessMode, fForceNewUuidOnOpen, pMedium.asOutParam())" at line 179 of file VBoxManageDisk.cpp
-
magicandre1981 almost 5 years@EdRandall submit the issue in their bug tracker. I stopped using Virtualbox for some times
-
Dan M. over 4 years@Vasilly.ProkopyevI hadd to
swapoff -a
in addition to booting in recovery mode and disabling systemd services. -
Oleg Mihailik over 4 yearsThat bigemptyfile hung my VM, eating all the space leaving nothing for OS. Horrible advice.
-
Tfb9 over 4 yearsAs what @Marcus inferred, a good addition to this answer would be Step 0: For best results, delete all snapshots of the machine before performing the following steps.
-
DustWolf about 4 yearsIf running the command in the accepted answer produces an unhelpful error message like this VBoxManage.exe: error: Cannot register the hard disk 'thedisk.vdi' {aaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeeee} because a hard disk 'thedisk.vdi' with UUID {aaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeeee} already exists Simply run the command by the UUID instead of the filename: VBoxManage.exe modifyhd {aaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeeee} --compact
-
aswine about 4 yearsShould I use SDelete if the host uses an SSD (if the VDI is on an SSD)?
-
magicandre1981 about 4 years@aswine I always use it, but he older version 1.6.1, sdlete 2 has a bug
-
Andrew Domaszek about 4 yearsAddl. historic info: Linux large file support starts in glibc 2.2 + linux 2.4.0. Windows large file support starts in NTFS, though older versions like Win2k will have other disk limits (48-bit LBA → 128 GiB max disk, etc).
-
Peter Wippermann almost 4 yearsThe download of the CloneVDI tool is attached on page 1 of that forum's thread: forums.virtualbox.org/viewtopic.php?f=6&t=22422
-
StellarVortex over 3 yearsI prefer periodic trim, myself: "sudo systemctrl enable fstrim.timer"
-
Bachsau about 3 yearsDo not run defrag if the virtual machine has snapshots. It might move data that would otherwise not have changed, which results in a bigger differential.
-
Pavel Tankov almost 3 yearsmust be rm -f /var/tmp/bigemptyfile
-
Boris almost 3 yearsFYI: this does not work on encrypted media (probably neither virtualbox's encryption, nor the guest os' encryption). I was confused why it did not work until I realized that it relies on the zeros on disk and with encryption you (often?) don't have zeros on disk.
-
wlf over 2 yearsafter running defrag and sdelete.exe on my Win7 VM, I created a clone of it. The VM clone has the same VDI file size reduction I obtained later using modifymedium on the original VM