KVM slow guest i/o

linux debian virtualization kvm-virtualization libvirt

5,439

Solution 1

Ok, this has been finally solved. The problem was in partition misalignment. It looks like Debian's default installer (especially lenny?) can't do the alignment properly, because when I checked the partitions it made with parted, it reported misalignment on all of them.

I solved this by manually partitioning LVM with parted prior to installation and now all guests are enjoying full performance.

Solution 2

Try setting your host's I/O elevator to deadline. Keep the guest at noop.

Solution 3

first of all, try to test the speeds with direct option, to eliminate the possibility of looking at the wrong results. Then, what you describe is something I've seen quite often on Debian and Ubuntu, any chance you can bring up a Fedora or RHEL host and guests to recheck?

5,439

Pyd

Updated on September 18, 2022

Comments

Pyd over 1 year

Host:

Debian 6.0 (squeeze) with qemu-kvm and libvirt from squeeze-backports

ii qemu-kvm 1.0+dfsg-8~bpo60+1

ii libvirt-bin 0.9.8-2~bpo60+2

Has 3TB sata drives with software raid and lvm. It has a sequential write speed of ~140MB/s measured with dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync

Elevator set to cfq

Guest

Debian 6.0 (squeeze)

Uses LVM as storage. Drivers are virtio and cache='none'

Sequential write speed is considerably slower with only 25-50MB/s

Elevator set to noop

I'm kind of running out of ideas for further tweaks but I'm sure that I/O speed should be much faster because many people are reporting almost native performance with lvm.
- mgorven almost 12 years
  
  Are you using the virtio block driver?
- Pyd almost 12 years
  
  Ok, something weird happened. I ran those tests again today when I woke up and the difference is noticeable. All squeeze guest are now up to native performance. But one older lenny guest is still having problems - max. 50MB/s. What could be happening? Is it the older kernel? 2.6.26-2-amd64 ...
- Pyd almost 12 years
  
  I noticed one more thing when looking at 'lspci' output on squeeze and lenny boxes. Squeeze has: SCSI storage controller: Red Hat, Inc Virtio block device | Lenny has: SCSI storage controller: Qumranet, Inc. Device 1001 | ... why is that? Both have virtio enabled..
- Pyd almost 12 years
  
  Ah, I found out that qumranet actually is the virtio controler I want.
Pyd almost 12 years

It's maybe just a tiny bit better but still nowhere near expectations.
Pyd almost 12 years

Fedora is unfortunately out of the question because I can't reinstall anymore. Machine is already serving a few sites that need to be online.
David Corsalini almost 12 years

your recent update kinda proved my point :)
Pyd almost 12 years

What point? That debian is unpredictable? :)
David Corsalini almost 12 years

that performance can depend on the distro. Problem is - most kvm devs work on fedora, and the rest of the distros take code from it. How well they adapt the code is something that's completely out of anyone's control :)
El Yobo almost 12 years

Could you possibly add some more information on what you did here? Misalignment on host or guest? What needed to be done to resolve it?
El Yobo almost 12 years

askubuntu.com/questions/8592/… is useful enough; looks like this isn't my problem, although I have much worse performance problems than you did (125MB/s on the host, <1MB/s on the guest).