Re: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0

[George Dunlap <george.dunlap@eu.citrix.com>]

On 31/07/13 20:35, Gordan Bobic wrote:
> On 07/31/2013 06:53 PM, George Dunlap wrote:
>> On Fri, Jul 26, 2013 at 2:11 PM, Gordan Bobic <gordan@bobich.net> wrote:
>>>> Now that is intereting - if this makes the memory holes the same 
>>>> between
>>>> the guest and the host, does it also implicitly vBAR=pBAR?
>>>
>>>
>>> Another thing that occurred to me might be useful to check - it is
>>> pretty easy to modify the BAR size on Nvidia cards. The defaults are
>>> 64MB and 128MB for the two BARs. They can be made much, much larger,
>>> and there is often advantage to enlarging them to at least be equal to
>>> VRAM size. Soooooo... If I boost the BAR from 128MB to 2GB, being a
>>> 64-bit BAR, it might make the BIOS do the sane thing and map it above
>>> 4GB. With the other BAR also suitably enlarged and it being done on
>>> the second GPU as well, there is no obvious option but to map them
>>> above 4GB (unless the BIOS is broken, which it may well be, in
>>> which case all bets are off).
>>>
>>> Which may just alleviate the memory issue if not completely fix
>>> the problem.
>>>
>>> Will try this and see what happens.
>>
>> I believe XenServer has a patch that allows the toolstack (in this
>> case xapi) to set the default size of the MMIO hole.  Andrew, did that
>> ever make it upstream?
>>
>> Unfortunately, it is unlikely to work with upstream qemu until we fix
>> the memory relocation issue...
>
> Interesting you should mention something like this. I've been 
> pondering whether it might be easier (even if it is a bodge) to simply 
> always set the domU E820 map to have 0x80000000 - 0xFFFFFFFF 
> (2GB->4GB) reserved. I have not yet seen a motherboard that maps 
> 32-bit BARs below 2GB.

I'm pretty sure we've seen a memory hole larger than 2GiB, in a box 
loaded up with a boatload of GPUs.

The main problem with doing this unconditionally is that the relocated 
memory isn't available to non-PAE 32-bit guests.  I think we should have 
a work-around in place for 4.4 that will avoid a collision between the 
host MMIO and guest memory addresses; but it will need to be off by 
default, at least for guests that don't have a passed-through device.

  -George