Re: GPU passthrough issue when VM is configured with 4G memory

[George Dunlap <george.dunlap@eu.citrix.com>]

On 07/03/13 07:51, Hanweidong wrote:
>> -----Original Message-----
>> From: George Dunlap [mailto:george.dunlap@eu.citrix.com]
>> Sent: 2013年3月6日 20:43
>> To: Hanweidong
>> Cc: Gonglei (Arei); xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
>> Luonengjun; Wangzhenguo
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> On 06/03/13 11:38, Hanweidong wrote:
>>>> -----Original Message-----
>>>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of
>> George
>>>> Dunlap
>>>> Sent: 2013年3月5日 20:59
>>>> To: Gonglei (Arei)
>>>> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun; Luonengjun;
>>>> Wangzhenguo; Hanweidong
>>>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>>>> with 4G memory
>>>>
>>>> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
>> <arei.gonglei@huawei.com>
>>>> wrote:
>>>>> Hi,all
>>>>>
>>>>> I have tried to passthrough GPU card(Nvidia quadro 4000) on the
>>>> latest Xen
>>>>> unstable version (QEMU is using Qemu-upsteam-unstable, not
>>>> traditional
>>>>> Qemu). This issue as below:
>>>>>
>>>>>        Windows7 64-bit guest will blue screen when GPU passthrough
>>>> configure
>>>>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest will
>>>> always stay
>>>>> at the grub screen.  I noticed that it will relocate RAM that
>>>> overlaps PCI
>>>>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
>>>> configured with
>>>>> 3G, it won't cause relocate RAM that overlaps PCI space in
>>>> pci_setup(), and
>>>>> GPU pass-through is no problem. So it seems this issue is related
>> to
>>>>> "relocate RAM" in pci_setup().
>>>> So one issue XenServer found with passing through GPUs is that there
>>>> are bugs in some PCI bridges that completely break VT-d.  The issue
>>>> was that if the *guest* physical address space overlapped the *host*
>>>> physical address of a different device, that the PCI bridges would
>>>> send traffic from the passed-through card intended for the guest to
>>>> another card instead.  The work-around was to make the hole in the
>>>> guest MMIO space the same size as the host MMIO hole.  I'm not sure
>> if
>>>> that made it upstream or not -- let me check...
>>>>
>>> Hi George,
>>>
>>> Could you post your patch and let us have a try with it? Thanks!
>> So the patch got checked in, but there still may be some more work if
>> you want to use it. :-)
>>
>> The patch modifies xc_hvm_build_args structure to include a field
>> called
>> "mmio_size". If this is set to zero, it will default to
>> HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the default
>> MMIO hole set during the build process. The guest BIOS may modify this
>> at boot time to make it bigger, but it doesn't make it smaller.
>>
>> Since this was designed for xapi, however, which calls libxc directly,
>> we didn't add any options to xend / xl / libxl to set this option.
>>
>> The easiest way to test it probably is just to hard-code
>> HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
>> it to 1GiB should be sufficient).
>>
>> Then if you want to use it in production, you probably want to either:
>> 1. Try it with the latest version of XCP (which I think has an option
>> you can set)
>> 2. Implement a config option for xl that allows you to set the MMIO
>> hole
>> size.
>>
>> #2 should be a relatively straightforward matter of "plumbing", and
>> would be a welcome contribution. :-)
>>
>> If you do implement #2, it might be nice to have an option of
>> "mmio_hole_size=host", which will set the guest mmio hole to the same
>> size as the host. That's what we implemented for XenServer, to make
>> sure
>> there would never be any collisions.
>>
> We rooted caused this issue: in pci_setup, it relocates pci_mem_start from 
> 0xF0000000 to 0XE0000000 because GPU has big more MMIO, but in QEMU, 
> xen_ram_init directly uses the macro HVM_BELOW_4G_RAM_END and 
> HVM_BELOW_4G_MMIO_LENGTH, doesn't do corresponding relocation like pci_setup
> does.
>
> We tried to hardcod HVM_BELOW_4G_RAM_END to 0XE0000000, GPU pass-through
> Worked.

I'm not super familiar with the HVM qemu / guest BIOS stuff, but it
sounds like that's a bug in pass-through that needs to be sorted out.
Anthony, can you comment?

-George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel