GPU passthrough with Xen 4.2 on Ubuntu 12.04

[Matthew Dean <mcd40@cam.ac.uk>]

[-- Attachment #1.1: Type: text/plain, Size: 4760 bytes --]

I have been trying to setup GPU passthrough for a couple of days now 
with little luck.  I'm hoping someone can shed some light as to where I 
may be going wrong or at least identify some genuine bugs. Essentially 
pci passthrough works for me but gpu passthrough doesn't.

My system is currently configured as follows (please ask if you need 
further details)

Asrock Z77 e-Itx (stock bios, not sure the version but I can find out.  
vt-d is enabled)
i7 3770S
AMD Radeon HD 7750

Dom0 - Ubuntu server 12.04 with desktop environment and the following 
extra packages installed before the build

build-essential zlib1g-dev python-dev libncurses5-dev libssl-dev 
libx11-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev pkg-config 
bridge-utils iproute udev bison flex gettext bin86 bcc iasl markdown 
ocaml-nox ocaml-findlib git gcc-multilib texinfo pciutils-dev gawk 
libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif 
texlive-latex-base texlive-latex-recommended texlive-fonts-extra 
texlive-fonts-recommended mercurial make gcc libc6-dev python 
python-twisted patch libsdl-dev libjpeg62-dev libbz2-dev e2fslibs-dev 
git-core xz-utils liblzma-dev liblzo2-dev libvncserver-dev

I've used the stock Ubuntu provided kernel which is version 
3.2.0-27-generic.  Dom0 is connected to a display via the intel 
integrated graphics.

xen 4.2 has been build from xen unstable tag 4.2.0-rc1 changeset 25689 
using the following commands

./configure
make world
make install

I had to manually add a line to /etc/fstab to get /proc/xen to mount on 
startup
Modules xen-evtchn, xen-gntdev and xen-pciback were setup to load on 
boot in /etc/modules
I've then setup xencommons to start on boot: "update-rc.d xencommons 
defaults 19 18"

Grub2 has then been setup to automatically boot the xen kernel. I've 
also had to add the option xsave=0 to the xen boot command line to get 
things to boot

On restart everything looks good. "xl list" returns the Dom0 domain 
only.  "cat /proc/xen/capabilities" returns control_d.  I've created a 
windows 7 domU without any pci passthrough and successfully installed 
windows 7 ultimate.  The config file looks like:

builder='hvm'
vcpus = 4
memory = 8192
shadow_memory = 48
name = "xenwin7"
vif = [ 'bridge=br0' ]
acpi = 1
apic = 1
disk = [ 'file:/usr/local/xenImages/xenwin7.img,hda,w']
boot="c"
sdl=0
vnc=1
vncconsole=1
vncpasswd=''
viridian=1
usb=1
serial='pty'
usbdevice='tablet'
on_poweroff="destroy"
on_reboot="restart"
on_crash="destroy"

I would like to be able to pass through the HD 7750 and a USB controller 
to the VM.  I see from lspci there are three devices I need to pass 
through, two for the gpu (the card itseft and the hd audio for hdmi 
device) and one for the USB controller.  The bus/device ids are:

0000:01:00.0
0000:01:00.1
0000:00:14.0

There are other usb controllers but I would like to leave them with 
dom0.  I can successfully configure the devices for passthrough using 
"xl pci-assignable-add".  In the config I then add the line:

  pci=["01:00.0","01:00.1","00:14.0"]

When I boot the VM everything seems to work fine.  Windows picks up all 
three devices and I can install drivers for them.  The USB controller 
works flawlessly and I can use an attached mouse and keyboard.  The 
radeon card requires a restart after which an attached display comes to 
life and I have 3D acceleration. Restarting the VM seems to work OK.  If 
however I shut down the VM I have no end of problems.  On any subsequent 
startup the vm struggles to get past the windows splash screen, waiting 
much longer than usual.  During this period dom0 is sluggish regarding 
mouse and keyboard input even though cpu and memory usage are very low.  
When windows finally loads I have no active display and I have to view 
the VM via VNC.  In device manage I find that windows has disabled the 
GPU saying there are not enough resources to run the card.  From this 
point onwards I can do nothing to get the gpu working again aside from 
removing the device, manually deleting the drivers, and starting again.

Note that at this point I am only trying to do secondary passthrough 
though I would ideally like to get to the point of doing primary 
passthrough.  Adding the line gfx_passthru=1 at any point in all this to 
the machine config however just prevents the VM from booting entirely; 
when I VNC in all I get is a qemu prompt and I never get any ouput to 
the real display.

Any suggestions as to how to get this to work would be greatly 
appreciated as I've hit a bit of a brick wall.  I should also say that I 
have managed to get secondary passthrough working using Debian Wheezy 
and the repository version of xen 4.1.  In that case though Dom0 didn't 
boot reliably.

Sincerely,

Matthew Dean

[-- Attachment #1.2: Type: text/html, Size: 5909 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel