Why Windows 10 Virtual Machine show high CPU load on KVM hyprvisor

Question

On KVM (Centos 7) Hypervisors Windows 10 Virtual Machines show High cpu usage. Around 30%. While other Linux, Windows 2k16, Windows 2k12 and Windows 7 Virtual machines just use from 1%to 7% CPU.

By cpu load i am refering to %CPU column in "top" output for qemu-kvm processes. For example in below top output first 2 processes are windows 10 VMs, While this hypervisor also have few linux, Windows 2k12, Windows 7 and Windows 2k16 VMS running.

top - 08:04:31 up 9 days, 20:05,  1 user,  load average: 1.07, 1.04, 0.98
Tasks: 333 total,   2 running, 165 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  1.6 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 39623299+total, 37377676+free, 21391056 used,  1065184 buff/cache
KiB Swap:  8388604 total,  8388604 free,        0 used. 37248361+avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 570162 qemu      20   0 3053036   2.3g  30468 S  29.2  0.6 259:41.40 qemu-kvm
 569759 qemu      20   0 3054316   2.3g  30964 S  28.2  0.6 252:47.93 qemu-kvm
 543245 qemu      20   0 3221396   2.5g  30700 S   7.0  0.7 201:02.59 qemu-kvm
 543511 qemu      20   0 3222416   2.5g  30388 R   6.0  0.7 199:15.69 qemu-kvm
   7803 root      10 -10 1005872 135920   9628 S   4.7  0.0 115:42.79 ovs-vswitchd
 539175 qemu      20   0 3256520   2.5g  30796 S   2.3  0.7 136:58.51 qemu-kvm
 538482 qemu      20   0 3252560   2.5g  30316 S   2.0  0.7 136:27.08 qemu-kvm
 539317 qemu      20   0 3239552   2.5g  30744 S   1.7  0.7  75:38.31 qemu-kvm
 539459 qemu      20   0 3233408   2.5g  30724 S   1.0  0.7  28:17.44 qemu-kvm
   7730 root      10 -10   46072   6632   3780 S   0.7  0.0   9:33.40 ovsdb-server

I looked around and found few suggestions like

1. Turn on hpet (High Precision Event Timer) in VM config
2. Turn of Tablet usb device in VM config

But none of these seems to solve the issue.

I dont see such load issue on XEN Hypervisors.

Hypervisor is Centos 7.6.1810 with kernel 4.20.8-1.el7.elrepo.x86_64 and qemu-kvm-1.5.3-160

Full qemu options VM is running with are below.

/usr/libexec/qemu-kvm -name 7f96031a1afac1394534792b82606e2e -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu IvyBridge-IBRS,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+osxsave,+arat,+stibp,+ssbd,+xsaveopt,+pdpe1gb,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -bios /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd -m 2192 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid c66b5fed-8db2-4b35-984d-8e4c20cea10a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-12-7f96031a1afac1394534/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:cvm2/7f96031a1afac1394534792b82606e2e-1:id=clientuser2:key=AQCWBYtbzlLBABAAuJJjzV6pJuY2GxHvB4Qv8g==:auth_supported=cephx\;none,format=raw,if=none,id=drive-virtio-disk0,throttling.bps-total=500000000,throttling.iops-read=2500000,throttling.iops-write=2500000 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=35 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:ee:61:2d:ba,bus=pci.0,addr=0x3 -netdev tap,fd=36,id=hostnet1,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:27:4e:c4:d5,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 10.1.1.48:4,share=allow-exclusive -vga std -global VGA.vgamem_mb=16 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on

Any help or suggestions will be highly appreciated.

Regards

Solution

I found several hints in a proxmox forum. After testing all of them below is what worked for me

push @$cpuFlags , 'hv_synic';
push @$cpuFlags , 'hv_stimer';

Translating them to libvirt xml format it is

<synic state='on'/>
<stimer state='on'/>

Now windows 10 VMs cpu load droped to 4 to 7%

Comments

[Michael Hampton]

serverfault.com/q/954899/126632

[topmop75]

Although i use qemu VNC and not SPICE but just in case i switched from vnc to SPICE and installed latest spice tools inside VM. but cpu load remain same as before around 30%

[Dominic P]

Thanks for sharing this. I'm running into what I think is the same issue, but when I modify the domain XML the VM fails to start with an error "Property '.hv-synic' not found". It looks from this bug those features aren't supported on CentOS 7. Did you find a work around for your CentOS 7 machine?