Prevent Radeon driver from attaching to specific PCI devices?

Question

I have two Radeon cards in this machine, a Radeon HD 6570 and a Radeon HD 6950:

lspci | grep VGA

01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks [Radeon HD 6570]
02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman PRO [Radeon HD 6950]

I'm trying to get VGA passthrough to work with KVM on Debian 7 (Wheezy), passing through the 6950 as a secondary video card to a Windows 7 guest. This works fine if I blacklist the radeon kernel module via /etc/modprobe.d/.

If I remove the blacklist to run X11 (or even just a KMS console) on the 6570 the radeon module seems to attach to both cards:

dmesg | egrep "01:00.0|02:00.0|radeon"

pci 0000:01:00.0: [1002:6759] type 0 class 0x000300
pci 0000:01:00.0: reg 10: [mem 0xe0000000-0xefffffff 64bit pref]
pci 0000:01:00.0: reg 18: [mem 0xf7e20000-0xf7e3ffff 64bit]
pci 0000:01:00.0: reg 20: [io 0xe000-0xe0ff]
pci 0000:01:00.0: reg 30: [mem 0xf7e00000-0xf7e1ffff pref]
pci 0000:01:00.0: supports D1 D2
pci 0000:02:00.0: [1002:6719] type 0 class 0x000300
pci 0000:02:00.0: reg 10: [mem 0xd0000000-0xdfffffff 64bit pref]
pci 0000:02:00.0: reg 18: [mem 0xf7d20000-0xf7d3ffff 64bit]
pci 0000:02:00.0: reg 20: [io 0xd000-0xd0ff]
pci 0000:02:00.0: reg 30: [mem 0xf7d00000-0xf7d1ffff pref]
pci 0000:02:00.0: supports D1 D2
vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
vgaarb: bridge control possible 0000:02:00.0
vgaarb: bridge control possible 0000:01:00.0
pci 0000:01:00.0: Boot video device
[drm] radeon kernel modesetting enabled.
radeon 0000:01:00.0: setting latency timer to 64
radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF (1024M used)
radeon 0000:01:00.0: GTT: 512M 0x0000000040000000 - 0x000000005FFFFFFF
[drm] radeon: 1024M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
radeon 0000:01:00.0: irq 46 for MSI/MSI-X
radeon 0000:01:00.0: radeon: using MSI.
[drm] radeon: irq initialized.
radeon 0000:01:00.0: WB enabled
[drm] radeon: ib pool ready.
[drm] radeon: power management initialized
fbcon: radeondrmfb (fb0) is primary device
fb0: radeondrmfb frame buffer device
[drm] Initialized radeon 2.12.0 20080528 for 0000:01:00.0 on minor 0
radeon 0000:02:00.0: enabling device (0000 -> 0003)
radeon 0000:02:00.0: setting latency timer to 64
radeon 0000:02:00.0: VRAM: 2048M 0x0000000000000000 - 0x000000007FFFFFFF (2048M used)
radeon 0000:02:00.0: GTT: 512M 0x0000000080000000 - 0x000000009FFFFFFF
[drm] radeon: 2048M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
radeon 0000:02:00.0: irq 49 for MSI/MSI-X
radeon 0000:02:00.0: radeon: using MSI.
[drm] radeon: irq initialized.
radeon 0000:02:00.0: WB enabled
[drm] radeon: ib pool ready.
[drm] radeon: power management initialized
fb1: radeondrmfb frame buffer device
[drm] Initialized radeon 2.12.0 20080528 for 0000:02:00.0 on minor 1
[drm] radeon: finishing device.
radeon 0000:02:00.0: ffff88041a941800 unpin not necessary
[drm] radeon: ttm finalized
pci-stub 0000:02:00.0: claimed by stub
pci-stub 0000:02:00.0: irq 49 for MSI/MSI-X

This causes the Windows 7 VM to bluescreen on boot.

How can I configure things so that module radeon only attaches to the 6570 and not the 6950?

Solution 1

I had had some pci_stub-related lines in my rc.local but this was apparently too late in the boot process to help, running after radeon started fiddling with things.

I fixed things by moving the pci_stub kernel module stuff to the initramfs:

1. lspci -nn | grep Cayman to find the vendor:device numbers (at the very end of the line) for the 6950 and its HDMI port:

   02:00.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Cayman PRO [Radeon HD 6950] [1002:6719]
   02:00.1 Audio device [0403]: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]
   

2. Pop open /etc/initramfs-tools/modules and add a new line:

   pci_stub ids=1002:6719,1002:aa80
   

3. Rebuild the initramfs:

   sudo update-initramfs -u
   

4. Reboot and observe pci_stub beating out radeon:

   dmesg | egrep "pci-stub|radeon"
   
   pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
   pci-stub 0000:02:00.0: claimed by stub
   pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
   pci-stub 0000:02:00.1: claimed by stub
   [drm] radeon kernel modesetting enabled.
   radeon 0000:01:00.0: setting latency timer to 64
   radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF (1024M used)
   radeon 0000:01:00.0: GTT: 512M 0x0000000040000000 - 0x000000005FFFFFFF
   

5. Enjoy bluescreen-free VGA passthrough!

Solution 2

Using genpfault answer as reference, I created another answer, I think is more simple.

The first step is the same as above:

Then I created a file called /lib/modprobe.d/pci-stub.conf, whith the args I found:

options pci-stub ids=1002:9715,1002:970f

Then I created a file called /lib/modprobe.d/drm.conf, with softdep:

softdep drm pre: pci-stub

This way, when the boot process tries to load drm, first loads pci-stub, which reserves the hardware of one video card, and prevents drm to attach to some device.

Later you can remove the reservation in rc.local, modprobe -r pci-stub.

Optionally you can load another module modprobe another-module for that card.

Comments

[Mansour]

The above didn't work for me. Instead I added pci_stub.ids=1002:6719,1002:aa80 to end of the kernel command line in grub config file. I suspect the reason is that pci-stub module is built into the kernel in my setup.

[Peter Mortensen]

Referenced in Multiheaded NVIDIA Gaming using Ubuntu 14.04 + KVM (step 2).