GPU Passthrough with KVM and Debian Linux

By | 2016/08/28

I have had success with PCI Passthrough and KVM under Debian Linux to pass a dedicated GPU to a Windows VM. Here are some notes!


Required:
  • Debian Stretch or later. Debian Stretch has the latest qemu (2.6) and libvirt (2.1.0) at time of writing. I was unsuccessful with Jessie due to fairly old qemu and libvirt, and no backports of qemu or libvirt are available at this time.
  • Recent motherboard and CPU which supports VT-d or AMD-Vi and IOMMU
  • Two video cards (GPUs). One will be for the Linux desktop and one will be dedicated for the Windows VM.
  • GPU brand: Get a recent AMD GPU. Nvidia disables their device in Windows with ‘Code 43‘ when virtualization is detected though there are some workarounds with recent versions of qemu. Intel passthrough is experimental at this point for Haswell or later and could be worth checking out if so inspired. AMD is the most straightforward for passthrough. Buy card, stick it in, and it works. One can even do some crazy stuff with AMD.
  • Second monitor connected to the passthrough GPU
  • Second USB keyboard and USB mouse to dedicate to the VM to make life easier. One could also use a KVM switch (the other KVM).
  • Windows 8.1 or later. Windows 8.1 has better UEFI support than Windows 7 and will work more care free overall with GPU devices and passthrough.

01-gpupassthru-edit

GPU Passthrough Windows 8.1 VM on the left, Debian desktop on the right!


Reference for more info

Good general reference for PCI Passthrough:

ArchWiki


Performance

How is the performance?

With the aid of Hyper-V Enligtenments in KVM, performance is as native as bare metal.


OK Let’s Do This Bro!

Step 1: Install kvm and associated packages

sudo apt install virt-manager qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin

It is a good idea to add your username to these groups for general kvm usage as a desktop user:

sudo gpasswd -a $(whoami) kvm
sudo gpasswd -a $(whoami) libvirt
sudo gpasswd -a $(whoami) libvirt-qemu


Step 2: Enable IOMMU

Edit /etc/default/grub and add the following to the GRUB_CMDLINE_LINUX_DEFAULT entry:

For Intel cpus, add:

intel_iommu=on

For AMD cpus, add:

amd_iommu=on


Step 3: Create new grub.cfg with that change

sudo grub-mkconfig -o /boot/grub/grub.cfg


Step 4: Blacklist the Linux driver of the passthrough device

Create / edit a file named, say, /etc/modprobe.d/blacklist.conf and add the following contents:

blacklist nouveau
blacklist radeon
blacklist amdgpu

Step 5: Reboot

sudo reboot


Step 6: Locate the PCI IDs of the passthrough GPU device

The following command will locate the iommu groups on your rig:

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done

In my case, the example output shows my AMD device to passthrough:

IOMMU group 11

        05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Venus LE [Radeon HD 8830M] [1002:682b] (rev 87)
        05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]

Note the PCI IDs of the GPU and also the HDMI audio device. The IDs are the last indications in brackets at the end of each device 1002:682b and 1002:aab0 in my case.


Step 7: Add PCI IDs to /etc/modprobe.d/vfio.conf

Create the file /etc/modprobe.d/vfio.conf and add both PCI IDs of the device to passthrough.

In my example, the file looks like:

# pass thru AMD gpu
options vfio-pci ids=1002:682b,1002:aab0


Step 8: Add vifo moudles to initrd

Edit the file /etc/initramfs-tools/modules

Add the following:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Save and then create a new initrd image via:

sudo update-initramfs -u


Step 9: Enable passthrough in qemu’s config

Edit /etc/libvirt/qemu.conf

Uncomment:

nvram = [    "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd"
]


Step 10: Reboot

sudo reboot


Windows VM Setup

From this point, create a VM in virt-manager or select an existing Windows VM one wishes to pass through the PCI device.

For best performance, enable Hyper-V Enlightenments for the Windows VM. A how-to guide can be found here.

To select the passthrough device, from virt-manager click on Add Hardware > PCI Host Device and select the passthrough GPU and then also its HDMI audio device. It is recommended to pass all components of a GPU device, from what I have gathered.

02-gpupassthru

Once inside Windows, be sure to install the GPU’s drivers as one would do with any bare metal Windows install. Then connect a second display to the passthrough GPU and you should be all set.

03-gpupassthru

04-gpupassthru

Note: It is also possible to perhaps use a single monitor which has multiple inputs and switch inputs to the Windows VM and back to Linux. However, I find it very satisfying to have a completely separate monitor to game and then have also my regular Linux desktop available side by side on the desk. The experience is as if there are two computers under one’s desk. I can pause a game, check something online / respond to a chat, and go back to the game. Or watch a video while gaming, etc.

If using an AMD GPU, congrats it just works! For NVIDIA, check out this reddit post for some more info.

05-gpupassthru

NVIDIA: Code 43 🙁


Other general Windows performance tips:

  • Don’t use any anti-virus or ‘security’ software
  • Disable Windows Defender completely
  • Disable System Restore
  • Set Performance Visual Effects to: Adjust for best performance
  • Set Windows Updates to manual check only
  • Disable and remove any of the active start screen news or other widgets, or whatever that mess is called
  • Use virtio drivers for everything (disk, network, etc).


Special shout out to #teamCO

Rock on,