PCI Passthrough (vt-d, vt-x) Multi-OS Workstation with qemu/kvm on AsRock Z87 Exterme 9/ac

Introduction

This article describes how to get passthrough working using the AsRock Z87 Extreme 9/ac. Main hardware passed through is two NVidia Quaddro and two USB-3 Cards.

Overview

This workstation build is running two virtualized 64-bit guest operating systems, each with full GPU support via native drivers The dom-0 is another 64-bit host system which using the i915 graphics.

Guest operating systems are: Windows10 via OVMF and Debian Linux via seabios which demonstrates use of both firmwares. The host operating system is Ubuntu-Gnome variant of the 16.04 LTS for ease of setup.

Patching the kernel is required with this motherboard because of the separation of the iommu groups. Kernel patches are provided via github. Patching the kernel introduces security issues !

Ressources

(best) Archlinux wiki article: PCI Passthrough via OVMF and also posts from the forums at bbs.archlinux.org.
The Ubuntu Kernel guide gives a good overview of kernel related tasks for debian systems.
VFIO, tips and tricks for tricky issues and faq related to errors.

Hardware overview

The system has been successfully built with the setup described below. Other Nvidia PCIe cards than the Quadro can be used but might require more effort to get to work properly. Eventhough the AsRockZ87 board has a great many USB connectors plus USB headers on the motherboard, it actually only has two separatable USB controllers. It gets better: the first has 14x USB 3.0 & 2x USB 2.0 and the second has 4x USB 2.0, so assigning the first to a VM rids you of quite a lot of USB ports. Additionally getting usable USB controllers is tricky, esp. the very common VLA/VIA chipsets will not run stable on the host OS. The chipsets i found usable are mentioned in the list below.

AsRock Z87 Extreme 9/ac Motherboard
Intel i7-4770, hyperthreaded quad core @ 3.4 GHz
32GB/DDR3 RAM (4x8GB)
Samsung SSD 850 EVO 500GB
Samsung SSD 840 EVO 250GB
PNY Nvidia Quadro PNY Quadro 2000K
HP Nvidia Quadro PNY Quadro 4000
Alcor chipset USB 3.0 PCIe controller
PCIe USB 3.0 and 3.1 controllers (
1 PS/2 Keyboard. Had issues accessing BIOS/bootloaders via USB keyboard.
2 USB Keyboards and 3 USB Mouses for the host, guest OSes.
Corsair AX760 modular PSU
Monitor with multiple input channel support, one used to view each system.

Setup Walkthrough

Verify Hardware Support

Make sure your Intel Processor supports vt-d before embarking on this journey. This can be checked via UEFI Setup under Advanced -> Chipset Configuration -> VT-d Capability -> Supported.

AsRock UEFI Setup

Boot into it pressing F2. If you're having trouble with USB, connect a PS/2 keyboard.

OC Tweaker -> Multi Core Enhancement -> Disabled
Advanced -> CPU Configuration -> Intel Hyper Threading Technology -> Enabled
Advanced -> Chipset Configuration -> Primary Graphics Adapter -> Onboard
Advanced -> Chipset Configuration -> Wt-d -> Enabled
Advanced -> Chipset Configuration -> PCIE2 Link Speed -> Auto
Advanced -> ACPI Configuration -> Suspend to RAM -> Auto
Advanced -> ACPI Configuration -> Check Ready Bit -> Enabled
Advanced -> ACPI Configuration -> ACPI HPET Table -> Enabled
Advanced -> ACPI Configuration -> set rest of the list to -> Disabled
Advanced -> USB Configuration -> USB Controller -> Enabled
Advanced -> USB Configuration -> Intel USB3.0 Mode -> Disabled (for now)
Advanced -> USB Configuration -> Legacy USB Support -> Disabled

Install and update host OS

Grab the official 64bit iso for Ubuntu-gnome 16.04 LTS and perform a full install.

Keeping the GPUs display ports disconnected during install/setup should keep video on the monitor you're using. If you have multiple drives, disabling them for the install process is recommended as the Ubuntu installer might automatically install GRUB on a wrong disk (and fail doing so, bricking your install).

From this point on let's asume that the install succeeded, internet is available and all operations are run as root.

Run an update

# Update the fresh install
apt-get update
apt-get upgrade
reboot

Patch and install new kernel

We'll be using the git method of building the kernel. For more info see Ubuntu Kernel Git Guide. The ACS override patch introduces security issues, you can read more about them here. You can find the patches here github.

The Intel i915 driver patch might fail, it looks like this section of code is being actively developed on so the patch becomes out of sync quickly. If that happens, you'll have to complete the patch by hand by checking the content of the .rej files, locate the right spot in the source and inject those changes.

Before proceding uncomment the deb-src entries in /etc/apt/sources.list for xenial and xenial-updates sources.

# add the deb-src sources to apt
apt-get update

# grab dependencies for building the kernel
apt-get build-dep linux-image-`uname -r`

# install git and fakeroot
apt-get install git fakeroot

# pull kernel sources (this will take a while)
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial.git

# patch the kernel using the patches from the repo
patch -p1 < deb-kvm-passthrough/override_for_missing_acs_capabilities.patch
patch -p1 < deb-kvm-passthrough/deb-kvm-passthrough/i915_317.patch

# you might have to do the chmod
chmod a+x debian/rules
chmod a+x debian/scripts/*
chmod a+x debian/scripts/misc/*

# create configs (for all archs), answer all with Y, then Exit
fakeroot debian/rules editconfigs

# modify the kernel tag so we can verify the proper kernel with `uname -a` later
# in the root dir edit the first line of debian.master/changelog, append a tag to
# the version number (something like +qemu). If you don't to this the new kernel
# still should have a + appended at the end of it's name.
#
vim debian.master/changelog
# For example, change
# linux (4.4.0-31.50) xenial; urgency=low
# to
# linux (4.4.0-31.50+qemu) xenial; urgency=low

# clean
fakeroot debian/rules clean

# (this will take a while, but use multiple processor if available))
fakeroot debian/rules binary-headers binary-generic

If everything went fine, install new kernel. Kernel version 4.4.0-31.50 with git commit id 2a1e175b928ae worked for me.

# deb packages will be built into .. from the source tree
cd ..
dpkg -i linux-image-4.4.0-31-generic_4.4.0-31.50_amd64.deb
dpkg -i linux-headers-4.4.0-31-generic_4.4.0-31.50_amd64.deb

Install QEMU, shadow NVIDIA cards and enable kernel patches

apt-get install qemu-kvm libvirt-bin virtinst bridge-utils

Add virtualization modules to your /etc/modules

pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

To blacklist the noveau driver add the following to /etc/modprobe.d/blacklist.conf

# Disable nouveau for passthrough
blacklist nouveau
blacklist lbm-nouveau
alias nouveau off
alias lbm-nouveau off

Update the commandline options in /etc/default/grub to the line below. These GRUB options will enable iommu support, fix i915 VGA graphics issues and apply the acs_override patch options to separate our IOMMU groups into individual chunks. The Z87 board unfortunately groups its PCIe ports so this options are neccesary but unsecure. You can see this if you don't apply the kernel patch.

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream"

After that update grub and reboot to boot into the new kernel with the patches enabled.

update-grub
reboot

Detect hardware ids and assign them to pci_stub

After the reboot create a directory called /vm and copy the following script to /vm/check_iommu.sh.

#!/bin/bash
for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do
        echo "IOMMU group $(basename "$iommu_group")";
        for device in $(ls -1 "$iommu_group"/devices/); do
                echo -n $'\t'; lspci -nns "$device";
        done;
done

Run the script, and check if the GPUs are in separate iommu-groups. Without the pcie_acs_override patch and kernel boot-option the groups would not be separated and look like shown below (not listing full output).

...
IOMMU group 1
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
        01:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
        02:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
        02:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
        02:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
        03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 4000] [10de:06dd] (rev a3)
        03:00.1 Audio device [0403]: NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5] (rev a1)
        04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [Quadro K2000] [10de:0ffe] (rev a1)
        04:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
        05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
IOMMU group 2
        00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
...

If the patch is working, it should instead list the GPUs and other components in their separate iommu groups like shown below:

...
IOMMU group 1
    00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU group 2
    00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
...

IOMMU group 12
    00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)

...
IOMMU group 14
    01:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
IOMMU group 15
    02:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
IOMMU group 16
    02:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
IOMMU group 17
    02:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
IOMMU group 18
    03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF100GL [Quadro 4000] [10de:06dd] (rev a3)
    03:00.1 Audio device [0403]: NVIDIA Corporation GF100 High Definition Audio Controller [10de:0be5] (rev a1)
IOMMU group 19
    04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [Quadro K2000] [10de:0ffe] (rev a1)
    04:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
IOMMU group 20
    05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
...

Edit /etc/initramfs-tools/modules to assign the devices to the pci_stub module, basically locking them from being used by other modules. Note these are the vendor-ids.

pci_stub ids=10de:06dd,10de:0be5,10de:0ffe,10de:0e1b,1b21:1242,8086:8c26

Generate the new ramfs

update-initramfs -u
reboot

If after reboot the iommu groups are not separated, check if the patched kernel has actually been loaded and the boot options hav been used.

This can be done by checking uname -a as well as dmesg | grep acs_override.

Create guest OS FS images, installation ISOs and gather BIOS/UEFI firmware roms.

Install seabios via apt-get install seabios if it's not installed already. OVMF is being developed by Tianocore. However we want the split image version from Gerd Hoffmant's Fedora OVMF builds. See more about this at the OVMF split image support blog. Grab the x86_64 ovmf image, extract it from the rpm archive and copy it to /usr/share/edk2.git/.

Next create linux and windows disk images using the qcow2 format. The options are for performance optimization.

qemu-img create -f qcow2 -o preallocation=metadata,compat=1.1,lazy_refcounts=on linux4.img 120G
qemu-img create -f qcow2 -o preallocation=metadata,compat=1.1,lazy_refcounts=on win10-v1.img 120G

First create two files as config files for the script below with device id's called vfio-pic-win10.cfg and vfio-pic-linux.cfg. The files will contain the ids of the devices to pass through:

vfio-pic-win10.cfg might look something like this:

0000:03:00.0
0000:03:00.1
0000:05:00.0

This will pass through the Quadro 4000 GPU as well as the HDMI Audio as seen by running check_iommu. The 05:00.0 is a USB controller card. Beware of passing through your motherboards usb controller unless you want to, because as mentioned earlier most of the usb ports will be assigned to the VM leaving you stranded on the host OS.

Next create two vm-scripts following snipplet below. Match the device id of PCIe to pass through as well as rom locations and disks for each os. Kudos to all the folks developing various versions of this script.

A boot iso image for each respective system will be needed, as well as the seabios for Linux and ovmf for Windows10.

#!/bin/bash

configfile=vfio-pci-win10.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
          echo "Unbinding dev: $dev"
          echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo "Assigning vendor: $vendor device: $device to vfio-pci driver"
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
    vfiobind $line
done

# QEMU name and PID
O="-name windows-10-pro"
O="$O -pidfile /tmp/windows-10-pro.pid"

# Processor
O="$O -cpu host,hv_time,kvm=off"
O="$O -smp 4,sockets=1,cores=2,threads=2"
O="$O -enable-kvm"
O="$O -M q35 -m 8192"
#
O="$O -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1"

# BIOS / UEFI
# For Linux
# O="$O -bios /usr/share/seabios/bios.bin"
# For Windows, use edk2
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
O="$O -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd"
O="$O -drive if=pflash,format=raw,file=/tmp/my_vars.fd"

## GFX (nvidia quadro 4k)
O="$O -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on"
O="$O -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1"

# USB
O="$O -device vfio-pci,host=76:0.0,bus=pcie.0"

# SCSI drive for Windows10
O="$O -drive id=disk0,if=none,cache=writeback,format=qcow2,file=win10-v1.qcow2"
O="$O -device driver=virtio-scsi-pci,id=scsi0"
O="$O -device scsi-hd,drive=disk0"
# IDE drive for Linux
# O="$O -drive file=linux-deb-v1.qcow2,id=disk,format=qcow2,if=none"
# O="$O -device ide-hd,bus=ide.0,drive=disk"

# VirtIO drivers CD
O="$O -drive id=virtiocd,if=none,format=raw,file=virtio-win.iso"
O="$O -device driver=ide-cd,bus=ide.0,drive=virtiocd"

# OS media
O="$O -drive file=Windows10-64.iso,id=isocd,format=raw,if=none"
O="$O -device ide-cd,bus=ide.1,drive=isocd"
# O="$O -drive file=debian-8.5.0-amd64-xfce-CD-1.iso,id=isocd,format=raw,if=none"
# O="$O -device ide-cd,bus=ide.1,drive=isocd"

# Network
O="$O -netdev user,id=network0"
O="$O -device e1000,netdev=network0"
# Provide local folder as a samba share (install samba on host OS)
# O="$O -smb /usr/vm/drivers"
# Redirect ssh -p 5555 user@localhost to ssh port of VM
# O="$O -redir tcp:5555::22"

# Disable/Enable Console
#O="$O -vga none"
O="$O -boot menu=on"

# Turn on debugging
#O="$O -debugcon file:debug_w10_install.log -global isa-debugcon.iobase=0x402"

sudo qemu-system-x86_64 $O
exit 0

Running the script should yield a black QEMU console window or the boot-screen display of the iso attached. The qemu console can be closed by pressing ALT-2 and typing the quit command.

Debian/Linux guest installation (via seabios)

In general the guest OS installation is performed by disabling the passthrough GPU VGA display and using QEMUs emulation of vga, keyboard and mouse in order to navigate around driver issues, followed by installing native graphics and USB drivers and turning off emulation afterwards. To do this remove x-vga=on from the GPU device line and make sure you don't have the -vga none option in the script. The respective lines for the example script above would be:

# disable x-vga output on card to install OS using stock vga output and drivers
O="$O -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on"

# use seabios
O="$O -bios /usr/share/seabios/bios.bin"

# Commenting this should give you a vga output window. Disable for install.
#O="$O -vga none"

Given the image was empty the bios should boot using the iso image. Perform an installation from the iso and reboot the VM. After reboot and hopefully a system update the native NVIDIA drivers can be installed.

This step depends on the respective linux-distro but any automatic install usually works out of the box. After the native graphics driver has been installed it's time to blacklist the nouveau driver on the guest os. This is done in the same way as described for the host OS by adding the module to blacklist.conf.

Poweroff the system: Connect your HDMI/DVI/DP cable and put the x-vga=on line back to the script. This should enable the graphics adapters output. Upon next boot you should see something on the output of the graphics card.

Check if the graphics driver is being used by running the nvidia-settings tool.

Windows10 guest installation (via OVMF/UEFI)

Windows installation is run in the same way as the linux installation with the difference that we're using the OVMF image. In order for Windows10 to pickup your 'scsi' drive you'll need to provide the virt-io drivers to the installer. Download the virtio drivers for windows by following the link found here and configure the script to point to its proper path as an attached CD-ROM image. During the Windows10 installation procedure you can select to load the virtio driver from cdrom.

Once Windows10 is installed, install Nvidia drivers, and proceed in the same way as for the Linux install.

Providing guests with access to host filesystem

Use the -smb option in the host VM config. You'll also need to install samba: apt-get install samba.

O="$O -smb /point/to/folder"

On the Linux guest install cifs-utils if it's a debian derivate. The following should do

mount.cifs //10.0.2.4/qemu /mnt/data -o user=qemu

On the Windows guest, just use the file explorer and point to //10.0.2.4/qemu.

Changelog

2016-07-31 First Draft Version
2019-04-25 Updated Formatting
2019-07-13 Updated Intro