From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gordan Bobic Subject: Re: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0 Date: Wed, 31 Jul 2013 20:35:01 +0100 Message-ID: <51F966E5.9080006@bobich.net> References: <51EF04D8.1090600@bobich.net> <20130724140813.GH2518@phenom.dumpdata.com> <2aa84a31b7b17c2ea6d8483a281ad3f5@mail.shatteredsilicon.net> <20130724160639.GB5804@phenom.dumpdata.com> <8426aecf79e7f55c21bbe259014591a2@mail.shatteredsilicon.net> <20130724163102.GA6308@phenom.dumpdata.com> <51F051F1.5050806@bobich.net> <51F19D11.1090200@bobich.net> <51F1A54D.6070906@bobich.net> <1374798084.10269.2.camel@hastur.hellion.org.uk> <4e207d63809542544ebe45866e746e91@mail.shatteredsilicon.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: George Dunlap Cc: Andrew Bobulsky , Andrew Cooper , Ian Campbell , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 07/31/2013 06:53 PM, George Dunlap wrote: > On Fri, Jul 26, 2013 at 2:11 PM, Gordan Bobic wrote: >>> Now that is intereting - if this makes the memory holes the same between >>> the guest and the host, does it also implicitly vBAR=pBAR? >> >> >> Another thing that occurred to me might be useful to check - it is >> pretty easy to modify the BAR size on Nvidia cards. The defaults are >> 64MB and 128MB for the two BARs. They can be made much, much larger, >> and there is often advantage to enlarging them to at least be equal to >> VRAM size. Soooooo... If I boost the BAR from 128MB to 2GB, being a >> 64-bit BAR, it might make the BIOS do the sane thing and map it above >> 4GB. With the other BAR also suitably enlarged and it being done on >> the second GPU as well, there is no obvious option but to map them >> above 4GB (unless the BIOS is broken, which it may well be, in >> which case all bets are off). >> >> Which may just alleviate the memory issue if not completely fix >> the problem. >> >> Will try this and see what happens. > > I believe XenServer has a patch that allows the toolstack (in this > case xapi) to set the default size of the MMIO hole. Andrew, did that > ever make it upstream? > > Unfortunately, it is unlikely to work with upstream qemu until we fix > the memory relocation issue... Interesting you should mention something like this. I've been pondering whether it might be easier (even if it is a bodge) to simply always set the domU E820 map to have 0x80000000 - 0xFFFFFFFF (2GB->4GB) reserved. I have not yet seen a motherboard that maps 32-bit BARs below 2GB. Note: Admittedly, I haven't tested what happens when you have multiple Nvidia cards each with a 1GB 32-bit BAR, though, I fully expect weirdness. And Nvidia cards have have the 32-bit BAR0 up to 2GB in size! But I cannot see a good reason to use such a configuration since it's the 64-bit BAR1 (up to 64GB in size) that provides the direct VRAM mapping. Anyway, if the whole 2GB->4GB area was reserved, then presumably Xen would map the 32-bit bars below 2GB, which, provided there's enough memory for the OS kernel to load and the BARs, shouldn't be a problem (I cannot think of a sane case where this wouldn't hold). 64-bit BARs can get re-mapped somewhere sky-high in domU RAM (at the top of the addressable range sounds like a reasonable bet, BIOS (for non-broken BIOS implementations, of which there seem to be fewer than I'd like to believe) would probably set those just above the size of RAM in the machine, so to 2^48 minus BAR size would possibly be a safe place to map them. Yes, I know it's a bodge. Yes, I know it wouldn't solve the GeForce passthrough problem. Yes, host E820 with vBAR = pBAR (possibly without IOMMU involvement) would be an awesome feature to have. But the bodge of just punching a 2GB hole at 2GB might just be a lot easier to implement as a quick fix. Gordan