esxi vmdk out of disk space

Question

One of our highly utilized vm is giving the error that there is no more space for the the vmdk shown in the following image.

The VM won't start because of this issue, so I'm not able to delete data from the volume at OS level.

The virtual machine has 3 snapshots but the consolidate option is greyed out. Screenshot from the vm's datastore below

Solution 1

You need to either

• Extend DataStore size (can be done online) or
• Storage vMotion some of the other VMs off this Data Store.

The ultimate goal is to free up space on the DataStore so you can unfreeze VM. And by the way - freeing space from the OS level won't help you.
If your vmdk is thin - it will grow in size over time, but will not shrink when you clean underlying OS.
There are some tricks to reduce vmdk size again available, but those require bringing VM down (not online process).

Solution 2

While I've not run in to that particular problem before you might try

• Free up some space and try again, perhaps enough for the consolidate to work by
  • Delete some old log files. I see that you have over 2GB in logs in that one folder alone.
  • You may find that shutting down some VMs will free up some space taken by the *.vswp files, while they are shut down.
  • What Dmitry Zayats suggests about extending the datastore to make it bigger or moving other VMs off that datastore.
  • If all else fails, move the VM to a datastore with enough space, then see if you can get rid of the snapshots. Note that since the machine is not running you can use a few different methods of moving the VM.
• Try the "Delete All" button in the snapshot manager instead of consolidate.

Note: Deleting data at the guest OS level will likely only make the snapshot files bigger, as it makes the snapshots more different to the previous snapshot than before.

Some light reading

• How to consolidate snapshots in vSphere 5.x/6.x (2003638) - basic howto
• Unable to delete the virtual machine snapshots (2017072) - make sure your backup software has let go of the snapshot
• Snapshot consolidate option is grayed out in vSphere Web Client 5.1 and 5.5 (2058138) - try the windows client instead
• Troubleshooting when virtual machine options are grayed out in vSphere Client (2048748) - command line stuff (danger: here be dragons. Instead of deleting snapshots you may end up doing a "go to snapshot" by accident)

Comments

[BeowulfNode42]

Which ESXi version is that server running?

[joebegborg07]

@BeowulfNode42 esxi 5.5

[Chopper3]

added to 'cautionary-tales-thinning-in-esxi-rather-than-on-array.xls' :)

[joebegborg07]

Thanks @joeqwerty. I've tried consolidating the virtual machine, however I got the error that there's not enough disk space. Can anyone help me with what disk space is required ? Is it double to present vmdk disk ? Is it double the combined size of all vmdks ?

[joebegborg07]

@BeofuldNode42. Is there a risk of corruption when deleting older log files ? Sorry for the simple question, I'm new to vmware.

[BeowulfNode42]

The "delete all" option is very much like going through each snapshot and clicking delete on each one, except it does all of them in one go. Though I've never done it with a full datastore before. You need at least some free space. Also any corruption that has been introduced in to the data stored in the vmdk by the guest crashing will be present, and you will no longer have a snapshot to go back to.

[BeowulfNode42]

FYI Joe, from the looks of the picture there is no value written in the provisioned size column for the base disks, so they should be thick provisioned disks. As far as I know snapshots are always thin provisioned, but only the current snapshot file will be growing.

[joebegborg07]

Thanks for your info in reply. I tried consolidation and I get the 'consolidation was successful message' after 2 seconds. The vms still have the same error and the snapshots are still visible in snapshot manager. So I suspect the consolidation did not work. Why would this happen? Could it be because the data store has 18GB left of disk space?

[rackandboneman]

Would vmkfstools -K help/work on a disk in that state?

[BeowulfNode42]

not sure about the K or -k options, but possibly the vmkfstools --fix check disk.vmdk and then vmkfstools --fix repair disk.vmdk and you may want to look in to the --chainConsistent option too. I hope you have a good backup as well...