Re: AMD GPU passthrough in Xen

[Gordan Bobic <gordan@bobich.net>]

Fair enough, thanks for the info. I guess there isn't much
I can do to help at the moment, then, due to lack of suitable
hardware.

Gordan

On 2014-09-24 14:31, Zytaruk, Kelly wrote:
> AMD has two sets of drivers.  One is our Legacy driver which supports
> HD4xxx and previous GPU families and the other is our current driver
> which supports HD5xxx and up.
> Officially we don't support the Legacy driver anymore.  HD4850 is a
> very old product and there isn't much that I can do with it as the
> driver changed significantly between HD4xxx and HD5xxx.
> 
> Please don't spend too much time on the HD4850.  It uses an entirely
> different driver and there have been a lot of fixes that have gone
> into the latest driver.
> 
> Thanks,
> Kelly
> 
>> -----Original Message-----
>> From: Gordan Bobic [mailto:gordan@bobich.net]
>> Sent: Wednesday, September 24, 2014 8:56 AM
>> To: Zytaruk, Kelly
>> Cc: Sander Eikelenboom; Peter Kay; xen-devel@lists.xen.org
>> Subject: RE: [Xen-devel] AMD GPU passthrough in Xen
>> 
>> On 2014-09-24 13:12, Zytaruk, Kelly wrote:
>> >> Good to know there is interest from AMD into this area :-)
>> >
>> > I am taking a personal interest in this and would like to improve AMD
>> > support and presence within the Xen community.
>> >
>> > Gordan has also reported problems restarting a guest.
>> >
>> > I have been trying to reproduce the problem but have not had any luck.
>> >  As a secondary it restarts for me every time.  I don't know if I
>> > inadvertently made a change that indirectly fixed it in my code base
>> > or what the difference might be.
>> >
>> > What Xen version are you testing with?
>> 
>> I mentioned this before, the behaviour is the same with various 
>> versions of Xen
>> between versions 4.2.0 and 4.3.1 (haven't tried more recent ones). 
>> Also with
>> kernels between 3.8.x and 3.14.x.
>> 
>> I'll try to build another machine for testing and re-attempt with my 
>> old HD4850 (I
>> disposed of all my ATI cards apart from the HD6450 which I'm using in 
>> a
>> different machine and the
>> HD4850 which I kept because it's the latest generation of
>> pre-R9 ATI cards that has dual DL-DVI outputs). Unfortunately, that 
>> may take a
>> little while (days/weeks) to find time to do. :(
>> 
>> Note that the fact it's a HD4xxx series card means I won't be able to 
>> use
>> particularly recent drivers as the support for those has been dropped 
>> a while
>> back, IIRC.
>> 
>> And then there's the issue of the R9 class cards not working properly 
>> behind
>> NF200 bridges (and all 7 of my PCIe slots are behind NF200 bridges), 
>> but I'll
>> worry about that when I get any ATI card working reliably across 
>> reboots.
>> 
>> Gordan
>> 
>> >> -----Original Message-----
>> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> Sent: Tuesday, September 23, 2014 9:45 AM
>> >> To: Zytaruk, Kelly
>> >> Cc: Peter Kay; xen-devel@lists.xen.org
>> >> Subject: Re: [Xen-devel] AMD GPU passthrough in Xen
>> >>
>> >> Good to know there is interest from AMD into this area :-)
>> >>
>> >> I'm experimenting for a while with:
>> >>
>> >> - xen-unstable (and thus xl)
>> >> - latest kernels (both dom0 and domU)
>> >> - qemu-xen
>> >> - Radeon HD 6570
>> >> - secondary passthrough
>> >> - Debian linux (sid) with the opensource (in kernel) radeon driver
>> >>   (i also tried fglrx with succes, but it's a real PITA to build with
>> >> every
>> >>   new kernel, so i ditched that)
>> >>
>> >> It used to work, but something broke at the moment, but that could
>> >> also be the changes to the systemd cruft that Debian jessie/sid is
>> >> currently undergoing (or something else since i regularly update all
>> >> components).
>> >>
>> >> The problems are mostly with restarting the domU, it differs a bit:
>> >> - sometimes screen goes ok, sometimes it's garbage.
>> >> - the radeon powercontrols only seem to work on the first boot and
>> >> give errors on any subsequent one.
>> >>
>> >> But when it works it does:
>> >> - the powercontrol.
>> >> - opengl and opencl benchmarks with (near) native results.
>> >> - hardware video acceleration in xbmc for instance.
>> >>
>> >> So one of the main problems at present seems to be proper resetting
>> >> of the whole device on domain shutdown/start. I did do some
>> >> experiments with the opensource radeon driver, but didn't get
>> >> conclusive results out of that yet.
>> >>
>> >> --
>> >> Sander
>> >>
>> >> Tuesday, September 23, 2014, 3:19:41 PM, you wrote:
>> >>
>> >> > Hi Peter / Sander,
>> >>
>> >> > Yes, I have AMD GPU passthru working as both primary and secondary
>> >> passthru.  Secondary was easy but primary is a bit tricky.
>> >>
>> >> > Getting on to your questions;
>> >>
>> >> >> Is there any specific reason you're using Xen 4.2 rather than 4.4.1?
>> >>
>> >> > I am working on a project that is based on Xen 4.2 (I can't say any more
>> than
>> >> that).  I have looked at some of the more recent versions just to
>> >> check if some
>> >> of the bugs that I have seen have been fixed but I have not studied
>> >> the newer
>> >> versions in detail.  At some point in time in the future I would like
>> >> will make the
>> >> jump to a more recent version but I don't know the scheduling of that.
>> >>
>> >> >> In 4.2, using xl or xm?
>> >>
>> >> > xl
>> >>
>> >> >> qemu-traditional (with rombios) or "upstream"
>> >>
>> >> > qemu-traditional
>> >>
>> >> >> Primary or secondary passthrough?
>> >>
>> >> > Both but I am focusing on secondary right now.
>> >>
>> >> >> Presumably 64 bit versions of Windows?
>> >>
>> >> > 32 bit and 64 bit Win7.  I have tested Win8.1 and it works but my
>> >> > focus is currently Win7
>> >>
>> >> >> I am quite willing to test various scenarios. I've a 6950, 6450 and 5450.
>> >>
>> >> > Awesome.  My goal right now is obtaining stability on Xen 4.2.  Since 4.2 is
>> >> past its feature cutoff I won't be able to submit any open source
>> >> changes for it.
>> >> I would like to eventually work with the community to get passthru
>> >> working with
>> >> a recent version of "upstream".
>> >>
>> >> > Thanks,
>> >> > Kelly
>> >>
>> >>
>> >>
>> >> >> -----Original Message-----
>> >> >> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
>> >> >> Sent: Monday, September 22, 2014 8:38 AM
>> >> >> To: Peter Kay
>> >> >> Cc: xen-devel@lists.xen.org; Zytaruk, Kelly
>> >> >> Subject: Re: [Xen-devel] AMD GPU passthrough in Xen
>> >> >>
>> >> >>
>> >> >> Monday, September 22, 2014, 2:16:58 PM, you wrote:
>> >> >>
>> >> >> > Hi Kelly, list
>> >> >>
>> >> >> > I see you're having AMD GPU success with Xen 4 2 and Linux 3.4.9.
>> >> >> > I've been
>> >> >> less than successful getting passthrough working at all in Xen
>> >> >> (although it's fine in KVM primary passthrough as long as the BIOS is
>> >> >> supplied as a file). Could I confirm the following :
>> >> >>
>> >> >> > Is there any specific reason you're using Xen 4.2 rather than
>> >> >> > 4.4.1? I know in
>> >> >> some ways 4.4 suffers as it's now xl only and some of the xm
>> >> >> functionality has not come across.
>> >> >>
>> >> >> > In 4.2, using xl or xm?
>> >> >>
>> >> >> Another interesting question/aspect would be qemu-traditional (with
>> >> >> rombios) or "upstream" (with seabios) ?
>> >> >>
>> >> >> > Primary or secondary passthrough?
>> >> >>
>> >> >> > Presumably 64 bit versions of Windows?
>> >> >>
>> >> >> > My system is a bit old (Core2Quad) but as mentioned AMD passthrough
>> >> >> > works
>> >> >> in KVM but I've found it tricky in Xen.
>> >> >>
>> >> >> > I am quite willing to test various scenarios. I've a 6950, 6450 and 5450.
>> >> >>
>> >> >> > Thanks
>> >> >>
>> >> >> > Peter
>> >> >>
>> >> >>
>> >>
>> >>