Re: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0

[Gordan Bobic <gordan@bobich.net>]

On 07/31/2013 06:53 PM, George Dunlap wrote:
> On Fri, Jul 26, 2013 at 2:11 PM, Gordan Bobic <gordan@bobich.net> wrote:
>>> Now that is intereting - if this makes the memory holes the same between
>>> the guest and the host, does it also implicitly vBAR=pBAR?
>>
>>
>> Another thing that occurred to me might be useful to check - it is
>> pretty easy to modify the BAR size on Nvidia cards. The defaults are
>> 64MB and 128MB for the two BARs. They can be made much, much larger,
>> and there is often advantage to enlarging them to at least be equal to
>> VRAM size. Soooooo... If I boost the BAR from 128MB to 2GB, being a
>> 64-bit BAR, it might make the BIOS do the sane thing and map it above
>> 4GB. With the other BAR also suitably enlarged and it being done on
>> the second GPU as well, there is no obvious option but to map them
>> above 4GB (unless the BIOS is broken, which it may well be, in
>> which case all bets are off).
>>
>> Which may just alleviate the memory issue if not completely fix
>> the problem.
>>
>> Will try this and see what happens.
>
> I believe XenServer has a patch that allows the toolstack (in this
> case xapi) to set the default size of the MMIO hole.  Andrew, did that
> ever make it upstream?
>
> Unfortunately, it is unlikely to work with upstream qemu until we fix
> the memory relocation issue...

Interesting you should mention something like this. I've been pondering 
whether it might be easier (even if it is a bodge) to simply always set 
the domU E820 map to have 0x80000000 - 0xFFFFFFFF (2GB->4GB) reserved. I 
have not yet seen a motherboard that maps 32-bit BARs below 2GB.

Note: Admittedly, I haven't tested what happens when you have multiple 
Nvidia cards each with a 1GB 32-bit BAR, though, I fully expect 
weirdness. And Nvidia cards have have the 32-bit BAR0 up to 2GB in size! 
But I cannot see a good reason to use such a configuration since it's 
the 64-bit BAR1 (up to 64GB in size) that provides the direct VRAM mapping.

Anyway, if the whole 2GB->4GB area was reserved, then presumably Xen 
would map the 32-bit bars below 2GB, which, provided there's enough 
memory for the OS kernel to load and the BARs, shouldn't be a problem (I 
cannot think of a sane case where this wouldn't hold). 64-bit BARs can 
get re-mapped somewhere sky-high in domU RAM (at the top of the 
addressable range sounds like a reasonable bet, BIOS (for non-broken 
BIOS implementations, of which there seem to be fewer than I'd like to 
believe) would probably set those just above the size of RAM in the 
machine, so to 2^48 minus BAR size would possibly be a safe place to map 
them.

Yes, I know it's a bodge. Yes, I know it wouldn't solve the GeForce 
passthrough problem. Yes, host E820 with vBAR = pBAR (possibly without 
IOMMU involvement) would be an awesome feature to have. But the bodge of 
just punching a 2GB hole at 2GB might just be a lot easier to implement 
as a quick fix.

Gordan