What creates CPU I/O wait but no disk operations?

Question

I have CPU I/O wait steady around 50%, but when I run iostat 1 it shows little to no disk activity.

What causes wait without iops?

NOTE: There no NFS or FUSE filesystems here, but it is using Xen virtualization.

Solution 1

NFS can do this, and it wouldn't surprise me if other network filesystems (and even FUSE-based devices) had similar effects.

Solution 2

Is there any chance other VMs on the server are thrashing the disk?

I know with virtualisation that you can get some strange results if the host node is overloaded.

Solution 3

If this is the Amazon EC2 Xen environment using instance-based storage, ask Amazon to check the health of the host containing this image.

If this is a Xen environment that you can gain access to the hypervisor, then check the IOwait from without for the disk image (file, network, LVM-slice, whatever) being used for the xvda and xvdb devices. You'll also want to check the I/O system, in general, for the hypervisor since other disk devices might be monopolizing the system's resources.

iostat -txk 5

is usually a good starting diagnostic tool. It takes 5-second summaries of I/O for ALL devices available to it, and thus is useful both with-in and wither-out the VM image.

Solution 4

Check your available file descriptors / inodes. When you hit the limit, they swap and mimic iowait

Edit

I saw you are using xen, have a look at your current interrupts, you might find blkif is higher than normal.

Bit late now, but get munin installed and it will really help future debugging.

Solution 5

sudo sysctl vm.block_dump=1

Then check dmesg to see what is performing block read / writes or dirtying inodes.

Also check nofile limit in limits.conf, a process could be requesting more files than it is permitted to open.

View more solutions

Comments

[RebornGeek]

What distro? What version?

[RebornGeek]

Also: is this a the Xen hyper visor machine or a VM with the iowaits?

[Janne Pikkarainen]

Does iotop show you anything?

[RebornGeek]

In most modern machines, no. Most, if not all recent systems have DMA-capable NICs to prevent precisely this sort of situation.

[lbft]

Steal happens when there's less CPU capacity available than requested by the VMs. If the physical disk is overloaded, your processes are going to spend a lot of time in iowait waiting for their turn at the disk even if they're not hitting the disk much.

[mattdm]

Yeah, this. See another question with the same answer at serverfault.com/a/209031/57468