Re: GPU passthrough issue when VM is configured with 4G memory

[George Dunlap <george.dunlap@eu.citrix.com>]

On 06/03/13 11:38, Hanweidong wrote:
>> -----Original Message-----
>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of George
>> Dunlap
>> Sent: 2013年3月5日 20:59
>> To: Gonglei (Arei)
>> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun; Luonengjun;
>> Wangzhenguo; Hanweidong
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei) <arei.gonglei@huawei.com>
>> wrote:
>>> Hi,all
>>>
>>> I have tried to passthrough GPU card(Nvidia quadro 4000) on the
>> latest Xen
>>> unstable version (QEMU is using Qemu-upsteam-unstable, not
>> traditional
>>> Qemu). This issue as below:
>>>
>>>        Windows7 64-bit guest will blue screen when GPU passthrough
>> configure
>>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest will
>> always stay
>>> at the grub screen.  I noticed that it will relocate RAM that
>> overlaps PCI
>>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
>> configured with
>>> 3G, it won't cause relocate RAM that overlaps PCI space in
>> pci_setup(), and
>>> GPU pass-through is no problem. So it seems this issue is related to
>>> "relocate RAM" in pci_setup().
>> So one issue XenServer found with passing through GPUs is that there
>> are bugs in some PCI bridges that completely break VT-d.  The issue
>> was that if the *guest* physical address space overlapped the *host*
>> physical address of a different device, that the PCI bridges would
>> send traffic from the passed-through card intended for the guest to
>> another card instead.  The work-around was to make the hole in the
>> guest MMIO space the same size as the host MMIO hole.  I'm not sure if
>> that made it upstream or not -- let me check...
>>
> Hi George,
>
> Could you post your patch and let us have a try with it? Thanks!

So the patch got checked in, but there still may be some more work if
you want to use it. :-)

The patch modifies xc_hvm_build_args structure to include a field called
"mmio_size". If this is set to zero, it will default to
HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the default
MMIO hole set during the build process. The guest BIOS may modify this
at boot time to make it bigger, but it doesn't make it smaller.

Since this was designed for xapi, however, which calls libxc directly,
we didn't add any options to xend / xl / libxl to set this option.

The easiest way to test it probably is just to hard-code
HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
it to 1GiB should be sufficient).

Then if you want to use it in production, you probably want to either:
1. Try it with the latest version of XCP (which I think has an option
you can set)
2. Implement a config option for xl that allows you to set the MMIO hole
size.

#2 should be a relatively straightforward matter of "plumbing", and
would be a welcome contribution. :-)

If you do implement #2, it might be nice to have an option of
"mmio_hole_size=host", which will set the guest mmio hole to the same
size as the host. That's what we implemented for XenServer, to make sure
there would never be any collisions.

-George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel