GPU Passthrough with KVM and Debian Linux

By | 2016/08/28

I have had success with PCI Passthrough and KVM under Debian Linux to pass a dedicated GPU to a Windows VM. Here are some notes!


Required:
  • Debian Stretch or later. Debian Stretch has the latest qemu (2.6) and libvirt (2.1.0) at time of writing. I was unsuccessful with Jessie due to fairly old qemu and libvirt, and no backports of qemu or libvirt are available at this time.
  • Recent motherboard and CPU which supports VT-d or AMD-Vi and IOMMU
  • Two video cards (GPUs). One will be for the Linux desktop and one will be dedicated for the Windows VM.
  • GPU brand: Get a recent AMD GPU. Nvidia disables their device in Windows with ‘Code 43‘ when virtualization is detected though there are some workarounds with recent versions of qemu. Intel passthrough is experimental at this point for Haswell or later and could be worth checking out if so inspired. AMD is the most straightforward for passthrough. Buy card, stick it in, and it works. One can even do some crazy stuff with AMD.
  • Second monitor connected to the passthrough GPU
  • Second USB keyboard and USB mouse to dedicate to the VM to make life easier. One could also use a KVM switch (the other KVM).
  • Windows 8.1 or later. Windows 8.1 has better UEFI support than Windows 7 and will work more care free overall with GPU devices and passthrough.

01-gpupassthru-edit

GPU Passthrough Windows 8.1 VM on the left, Debian desktop on the right!


Reference for more info

Good general reference for PCI Passthrough:

ArchWiki


Performance

How is the performance?

With the aid of Hyper-V Enligtenments in KVM, performance is as native as bare metal.


OK Let’s Do This Bro!

Step 1: Install kvm and associated packages

sudo apt install virt-manager qemu-kvm ovmf bridge-utils uml-utilities libvirt-bin

It is a good idea to add your username to these groups for general kvm usage as a desktop user:

sudo gpasswd -a $(whoami) kvm
sudo gpasswd -a $(whoami) libvirt
sudo gpasswd -a $(whoami) libvirt-qemu


Step 2: Enable IOMMU

Edit /etc/default/grub and add the following to the GRUB_CMDLINE_LINUX_DEFAULT entry:

For Intel cpus, add:

intel_iommu=on

For AMD cpus, add:

amd_iommu=on


Step 3: Create new grub.cfg with that change

sudo grub-mkconfig -o /boot/grub/grub.cfg


Step 4: Blacklist the Linux driver of the passthrough device

Create / edit a file named, say, /etc/modprobe.d/blacklist.conf and add the following contents:

blacklist nouveau
blacklist radeon
blacklist amdgpu

Step 5: Reboot

sudo reboot


Step 6: Locate the PCI IDs of the passthrough GPU device

The following command will locate the iommu groups on your rig:

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done

In my case, the example output shows my AMD device to passthrough:

IOMMU group 11

        05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Venus LE [Radeon HD 8830M] [1002:682b] (rev 87)
        05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]

Note the PCI IDs of the GPU and also the HDMI audio device. The IDs are the last indications in brackets at the end of each device 1002:682b and 1002:aab0 in my case.


Step 7: Add PCI IDs to /etc/modprobe.d/vfio.conf

Create the file /etc/modprobe.d/vfio.conf and add both PCI IDs of the device to passthrough.

In my example, the file looks like:

# pass thru AMD gpu
options vfio-pci ids=1002:682b,1002:aab0


Step 8: Add vifo moudles to initrd

Edit the file /etc/initramfs-tools/modules

Add the following:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Save and then create a new initrd image via:

sudo update-initramfs -u


Step 9: Enable passthrough in qemu’s config

Edit /etc/libvirt/qemu.conf

Uncomment:

nvram = [    "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd"
]


Step 10: Reboot

sudo reboot


Windows VM Setup

From this point, create a VM in virt-manager or select an existing Windows VM one wishes to pass through the PCI device.

For best performance, enable Hyper-V Enlightenments for the Windows VM. A how-to guide can be found here.

To select the passthrough device, from virt-manager click on Add Hardware > PCI Host Device and select the passthrough GPU and then also its HDMI audio device. It is recommended to pass all components of a GPU device, from what I have gathered.

02-gpupassthru

Once inside Windows, be sure to install the GPU’s drivers as one would do with any bare metal Windows install. Then connect a second display to the passthrough GPU and you should be all set.

03-gpupassthru

04-gpupassthru

Note: It is also possible to perhaps use a single monitor which has multiple inputs and switch inputs to the Windows VM and back to Linux. However, I find it very satisfying to have a completely separate monitor to game and then have also my regular Linux desktop available side by side on the desk. The experience is as if there are two computers under one’s desk. I can pause a game, check something online / respond to a chat, and go back to the game. Or watch a video while gaming, etc.

If using an AMD GPU, congrats it just works! For NVIDIA, check out this reddit post for some more info.

05-gpupassthru

NVIDIA: Code 43 🙁


Other general Windows performance tips:

  • Don’t use any anti-virus or ‘security’ software
  • Disable Windows Defender completely
  • Disable System Restore
  • Set Performance Visual Effects to: Adjust for best performance
  • Set Windows Updates to manual check only
  • Disable and remove any of the active start screen news or other widgets, or whatever that mess is called
  • Use virtio drivers for everything (disk, network, etc).


Special shout out to #teamCO

Rock on,

13 thoughts on “GPU Passthrough with KVM and Debian Linux

  1. Ignat S

    I have a quick question:
    If I have more than one card to pass through, do I create two vfio.conf files or just add all ids of those cards to a single file?

    Reply
    1. Scott Miller Post author

      Add a second line in the same file (vfio.conf) for an additional device. Cheers,

      Reply
  2. Brandon Bridges

    I’ve been looking at a couple of these KVM gaming builds, but haven’t found anyone who has done what I would like to try. I would actually like to build a system which has both PCI-E and PCI slots. I would then like to build one VM which is modern and have Windows 8.1 or Windows 10. Then I would like to have a Windows XP and a Windows 98 VM and a DOS VM each with an appropriate GPU. Windows 98 would get a older PCI 3DFX card. Has anyone tried this with older GPUs?

    Reply
    1. Abdullah

      I’ve passedthrough Nvidia XFX 8500 GT, and AMD Radeon HD 4870.
      All working great, still I’m not sure were that’s would put me.

      You should try, as it won’t take more than an hour without compiling a kernel.

      Reply
    2. ingo-h

      As I understand you would like to use 5 graphics cards, each for Windows 10, Windows XP, Windows 98, DOS as guests and one for the host and of course then you need 5 monitors/keyboards/mice. In theory that could do (if you have enough slots in your machine), but you need all different cards to blacklist the drivers. If you have cards using the same driver you must use pci_stub to select the right card for the guest https://wiki.debian.org/VGAPassthrough#Driver_association. I have never tried this. Passthrough older graphics cards should do as long as the host knows it. It will give its device to the guest and it’s to the guest to use the right driver for this device.

      Reply
  3. Abdullah

    Hello,
    You could easily fix Nvidia error code 43 by hiding the driver parameters, as both Quadro cards and GeForce cards uses the same driver’s, still Nvidia wish not to allow GPU passthrough with GeForce but Quadro.
    This is the parameter I use to start my VM’s:
    -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX.

    You could also consult the comments here http://vfio.blogspot.com/2014/08/vfiovga-faq.html
    Regards,

    Reply
  4. Ic0

    Hey, really nice article!

    I’ve been meaning to try out something like this to try out on my own rig, but I’m not all too sure about one thing.
    I’m going to try this out with two Nvidia GPU’s in my rig, a GTX 550Ti and a GTX 770. What I want to do is to passthrough the 770 to the guest OS, but keep running the nouveau driver on my GTX 550Ti.

    If I complete this process and load the 770 with the vfio driver I’d prefer to remove the line for blacklisting the nouveau driver as I want to use that with the 550Ti. Will removing the line blacklisting nouveau after having completed this process cause an issue for the passthrough?

    Best Regards,

    Reply
  5. ingo-h

    what is libvirt-bin? Not available in
    ~$ lsb_release -a
    Distributor ID: Debian
    Description: Debian GNU/Linux 9.1 (stretch)
    Release: 9.1
    Codename: stretch
    ~$
    ~$ apt list *libvirt-bin*
    Listing… Done
    ~$

    In stretch user must not be member in group kvm.

    Why using ?
    ~$ sudo grub-mkconfig -o /boot/grub/grub.cfg
    ‘sudo update-grub’ should do.

    Why do you blacklist 3 graphics driver? Do you have 4 different graphics
    cards in your host?

    Reply
  6. Juan AgustĂ­n

    Excellent article!!!
    But one of the steps you’ve taken, in my case, was an error where you say:
    “Step 9: Enable passthrough in qemu’s config
    Edit /etc/libvirt/qemu.conf
    Uncomment:
    nvram = [ “/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd”
    ]”
    If I uncomment only that, the libvirt-daemon refuses to start, and gives error.
    In the qemu.conf file you should not only uncomment the line that you indicate, but also the following two. Finally it would remain like this (without quotes)
    “nvram = [
    “/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd”,
    “/usr/share/OVMF/OVMF_CODE.secboot.fd:/usr/share/OVMF/OVMF_VARS.fd”,
    “/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd”
    ]”
    After performing that step, virt-manager works correctly.
    Regards.
    Juan AgustĂ­n

    Reply
    1. Juan AgustĂ­n

      Without this quotes: “nvram=[…. ]”. The rest of quotes must be in.
      Sorry.
      Juan AgustĂ­n

      Reply
    2. Juan AgustĂ­n

      Bad news…
      I got with VGAPassthrough Radeon R7 on Windows 8.1 and Windows 10, Code:43.
      And there are too many guides for this issue with Nvidia; but nothing with AMD.
      Juan AgustĂ­n

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.