Linux GPU Pass-through Virtualization - Verify wich component causes trouble

Question

GPU Passthrough Virtualization with KVM or VirtualBox

For a research project I need to passthrough a PCI GPU from a Ubuntu Host to a Windows 8.1 guest. We need to test a certain setup, where the guest performs GPU intensive tasks. I tried to follow this tutorial with KVM and also VirtualBox. Now before we invest in expensive server grade hardware, we wanted to try the setup with some older hardware that we had available in the lab. I am aware that the setup is very hardware dependent, but I want to learn how I can rule out errors.

I tried KVM and VirtualBox so far, but I think my problem is related to this error in the dmesg log:

~$ dmesg | grep -e IOMMU -e DMAR 
[    0.000000] Intel-IOMMU: enabled
[    0.148515] DMAR: Forcing write-buffer flush capability
[    0.148516] DMAR: Disabling IOMMU for graphics on this chipset
[   24.487950] vboxpci: IOMMU not found (not registered)

Where does this come from? I would like to know which component causes this error. I see many people having this problem but there is no answer available which would apply to several scenarios.

The hardware I use

Motherboard: P5Q-EM, ASUSTeK Computer INC.
BIOS (updated and virtualization enabled)
CPU: Intel(R) Core(TM)2 Quad  CPU   Q9300  @ 2.50GHz
GPU 1: Intel Corporation 4 Series Onboard
GPU 2: GeForce GT 610 (should be passed through)
OS: Ubuntu Server 14.04.2 LTS (with desktop installed)
Grub parameters:  intel_iommu=on. 

As the first dmesg message shows Intel-IOMMU: enabled, I assume that this works.

GPU details from lshw:

  *-display UNCLAIMED
       description: VGA compatible controller
       product: GF119 [GeForce GT 610]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:04:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller cap_list
       configuration: latency=0
       resources: memory:fd000000-fdffffff memory:f0000000-f7ffffff memory:fa000000-fbffffff ioport:ec00(size=128) memory:feb00000-feb7ffff

Now I checked the CPI capabilities with:

ubuntu~$  grep -E "(vmx|svm)" --color=always /proc/cpuinfo
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64      monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm dtherm tpr_shadow vnmi flexpriority

lspci -nn gives:

00:02.0 VGA compatible controller [0300]: Intel Corporation 4 Series Chipset Integrated Graphics Controller [8086:2e22] (rev 03)
00:02.1 Display controller [0380]: Intel Corporation 4 Series Chipset Integrated Graphics Controller [8086:2e23] (rev 03)
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)

KVM says its okay:

~$ sudo kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

I also tried pci-stub.ids=10de:104a, where 10de:104a is the GPU Id in grub. How can I make sure where the DMAR message comes from and what component causes the disableing?

Solution

The "DMAR: Disabling IOMMU for graphics on this chipset" message comes from the kernel, specifically http://lxr.free-electrons.com/source/drivers/iommu/intel-iommu.c?v=3.19#L4634. This quirk was introduced in https://lkml.org/lkml/2013/2/1/327 (the bugs linked from there give useful background information); apparently your chipset has bugs which cause crashes when the IOMMU is used with graphics.

I'm not sure from the discussions whether the bugs only really affect the built-in graphics, or if they would prevent using a separate adapter as you're trying to do. If I'm understanding the source code correctly though, the quirk disables all IOMMUs for graphics devices.

Comments

[Pablo Máximo]

You are right. The issue was solely caused by the combination of the motherboard and the GPU. Unfortunately I do not have any further details. We changed the machine (i.e. cannibalized the machine from a colleague) and retried the procedure. The only thing that we needed to change from the tutorial linked above is that the system needed to be installed locally before we could load it in VirtManager. Obviously it does not like a mounted ISOs. Thanks for your help!