How do I confirm that VFIO is working in 20.04?

Question

I just installed Ubuntu Server 20.04 and am trying to get GPU Passthrough working, but I can't seem to get VFIO enabled using methods that worked previously:

/etc/initramfs-tools/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net

/etc/default/grub:

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 amd_iommu=on vfio_pci.ids=10de:1b81,10de:10f0 vfio_iommu_type1.allow_unsafe_interrupts=1"
GRUB_CMDLINE_LINUX=

But it seems that VFIO is not enabled in the kernel.

~$  lsmod | grep kvm
kvm_amd                98304  0
kvm                   663552  1 kvm_amd
ccp                    86016  13 kvm_amd
~$ lsmod | grep vfio
~$ 

There is no way I can bind the drivers of my GPU to VFIO unless I can confirm that VFIO is enabled, correct?

Edit: I previously had passthrough and everything working beautifully for about a year under Arch. So this is clearly an Ubuntu thing. I am using a Thredripper 1950X on ASROck x399 Taichi.

Edit2: Some progress I got vfio_mdev loaded. Is that now in place of reguler vfio or do I still need to get vfio loaded?

$ sudo modprobe vfio_mdev
$ lsmod | grep -i vfio
vfio_mdev              16384  0
mdev                   24576  1 vfio_mdev

Edit 3: OK, so it seems that vfio is no longer a module, but is built into the kernel. https://forum.level1techs.com/t/ubuntu-20-04-missing-kernel-modules-for-vfio-pci-and-vfio-iommu-type1/156327/2

that indeed appears to be true:

~$ cat /boot/config-5.4.0-33-generic | grep -i vfio
CONFIG_KVM_VFIO=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO_VIRQFD=y
CONFIG_VFIO=y
CONFIG_VFIO_NOIOMMU=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI_IGD=y
CONFIG_VFIO_MDEV=m
CONFIG_VFIO_MDEV_DEVICE=m
# CONFIG_SAMPLE_VFIO_MDEV_MTTY is not set
# CONFIG_SAMPLE_VFIO_MDEV_MDPY is not set
# CONFIG_SAMPLE_VFIO_MDEV_MDPY_FB is not set
# CONFIG_SAMPLE_VFIO_MDEV_MBOCHS is not set

Ok, so if it's built-in, then it makes sense that lsmod would not show it as it is not a module. Is there some way to confirm that vfio is working. Is this output from dmesg enough to confirm that and move on to my next issue (that pci-alias' aren't working the way they did before). But that is another thread.

dmesg | grep -i -e vfio
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-33-generic root=/dev/mapper/main--VG-lvroot ro quiet splash iommu=1 amd_iommu=on vfio_pci vfio_iommu_type1.allow_unsafe_interrupts=1 vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-33-generic root=/dev/mapper/main--VG-lvroot ro quiet splash iommu=1 amd_iommu=on vfio_pci vfio_iommu_type1.allow_unsafe_interrupts=1 vt.handoff=7
[    0.532762] VFIO - User Level meta-driver version: 0.3

Solution

You couldn't use lsmod to show whether vfio drivers were loaded into linux kernel if the kernel already builded with the vfio drivers. But you still could see whether those drivers were built into kernel with the following commands:

# cat /lib/modules/$(uname -r)/modules.builtin | grep vfio

kernel/drivers/vfio/vfio.ko
kernel/drivers/vfio/vfio_virqfd.ko
kernel/drivers/vfio/vfio_iommu_type1.ko
kernel/drivers/vfio/pci/vfio-pci.ko

This command also could help you check other built-in drivers in kernel with different driver name's pattern after "grep"

Comments

[Adel - MageBinary]

Awesome thanks for that. You can also use sudo lspci -nnv | grep "Kernel driver in use" -C 10 to vertify the card is using the driver.