Re: GPU passthrough with Xen 4.2 on Ubuntu 12.04

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Aug 08, 2012 at 02:26:33PM +0100, Matthew Dean wrote:
> I have been trying to setup GPU passthrough for a couple of days now
> with little luck.  I'm hoping someone can shed some light as to
> where I may be going wrong or at least identify some genuine bugs.
> Essentially pci passthrough works for me but gpu passthrough
> doesn't.
> 
> My system is currently configured as follows (please ask if you need
> further details)
> 
> Asrock Z77 e-Itx (stock bios, not sure the version but I can find
> out.  vt-d is enabled)
> i7 3770S
> AMD Radeon HD 7750
> 
> Dom0 - Ubuntu server 12.04 with desktop environment and the
> following extra packages installed before the build
> 
> build-essential zlib1g-dev python-dev libncurses5-dev libssl-dev
> libx11-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev pkg-config
> bridge-utils iproute udev bison flex gettext bin86 bcc iasl markdown
> ocaml-nox ocaml-findlib git gcc-multilib texinfo pciutils-dev gawk
> libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif
> texlive-latex-base texlive-latex-recommended texlive-fonts-extra
> texlive-fonts-recommended mercurial make gcc libc6-dev python
> python-twisted patch libsdl-dev libjpeg62-dev libbz2-dev
> e2fslibs-dev git-core xz-utils liblzma-dev liblzo2-dev
> libvncserver-dev
> 
> I've used the stock Ubuntu provided kernel which is version
> 3.2.0-27-generic.  Dom0 is connected to a display via the intel
> integrated graphics.
> 
> xen 4.2 has been build from xen unstable tag 4.2.0-rc1 changeset
> 25689 using the following commands
> 
> ./configure
> make world
> make install
> 
> I had to manually add a line to /etc/fstab to get /proc/xen to mount
> on startup
> Modules xen-evtchn, xen-gntdev and xen-pciback were setup to load on
> boot in /etc/modules
> I've then setup xencommons to start on boot: "update-rc.d xencommons
> defaults 19 18"
> 
> Grub2 has then been setup to automatically boot the xen kernel. I've
> also had to add the option xsave=0 to the xen boot command line to
> get things to boot
> 
> On restart everything looks good. "xl list" returns the Dom0 domain
> only.  "cat /proc/xen/capabilities" returns control_d.  I've created
> a windows 7 domU without any pci passthrough and successfully
> installed windows 7 ultimate.  The config file looks like:
> 
> builder='hvm'
> vcpus = 4
> memory = 8192
> shadow_memory = 48
> name = "xenwin7"
> vif = [ 'bridge=br0' ]
> acpi = 1
> apic = 1
> disk = [ 'file:/usr/local/xenImages/xenwin7.img,hda,w']
> boot="c"
> sdl=0
> vnc=1
> vncconsole=1
> vncpasswd=''
> viridian=1
> usb=1
> serial='pty'
> usbdevice='tablet'
> on_poweroff="destroy"
> on_reboot="restart"
> on_crash="destroy"
> 
> I would like to be able to pass through the HD 7750 and a USB
> controller to the VM.  I see from lspci there are three devices I
> need to pass through, two for the gpu (the card itseft and the hd
> audio for hdmi device) and one for the USB controller.  The
> bus/device ids are:
> 
> 0000:01:00.0
> 0000:01:00.1
> 0000:00:14.0
> 
> There are other usb controllers but I would like to leave them with
> dom0.  I can successfully configure the devices for passthrough
> using "xl pci-assignable-add".  In the config I then add the line:
> 
>  pci=["01:00.0","01:00.1","00:14.0"]
> 
> When I boot the VM everything seems to work fine.  Windows picks up
> all three devices and I can install drivers for them.  The USB
> controller works flawlessly and I can use an attached mouse and
> keyboard.  The radeon card requires a restart after which an
> attached display comes to life and I have 3D acceleration.
> Restarting the VM seems to work OK.  If however I shut down the VM I
> have no end of problems.  On any subsequent startup the vm struggles
> to get past the windows splash screen, waiting much longer than
> usual.  During this period dom0 is sluggish regarding mouse and
> keyboard input even though cpu and memory usage are very low.  When
> windows finally loads I have no active display and I have to view
> the VM via VNC.  In device manage I find that windows has disabled
> the GPU saying there are not enough resources to run the card.  From
> this point onwards I can do nothing to get the gpu working again
> aside from removing the device, manually deleting the drivers, and
> starting again.
> 
> Note that at this point I am only trying to do secondary passthrough
> though I would ideally like to get to the point of doing primary
> passthrough.  Adding the line gfx_passthru=1 at any point in all
> this to the machine config however just prevents the VM from booting
> entirely; when I VNC in all I get is a qemu prompt and I never get


I am not sure if this the same problem I've been seeing is but every time I
restart a Windows VM with GPU passthrough it crashed. I found that if "Eject"
the GPU before shutting down it would work just great the next time I started
it.

Try that and see if that works. This is btw with an unmodified Xen 4.1 and
Fedora's 3.4.something kernel.

> any ouput to the real display.
> 
> Any suggestions as to how to get this to work would be greatly
> appreciated as I've hit a bit of a brick wall.  I should also say
> that I have managed to get secondary passthrough working using
> Debian Wheezy and the repository version of xen 4.1.  In that case
> though Dom0 didn't boot reliably.
> 
> Sincerely,
> 
> Matthew Dean

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel