From mboxrd@z Thu Jan 1 00:00:00 1970 From: George Dunlap Subject: Re: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0 Date: Thu, 1 Aug 2013 10:15:00 +0100 Message-ID: <51FA2714.9000405@eu.citrix.com> References: <51EF04D8.1090600@bobich.net> <20130724140813.GH2518@phenom.dumpdata.com> <2aa84a31b7b17c2ea6d8483a281ad3f5@mail.shatteredsilicon.net> <20130724160639.GB5804@phenom.dumpdata.com> <8426aecf79e7f55c21bbe259014591a2@mail.shatteredsilicon.net> <20130724163102.GA6308@phenom.dumpdata.com> <51F051F1.5050806@bobich.net> <51F19D11.1090200@bobich.net> <51F1A54D.6070906@bobich.net> <1374798084.10269.2.camel@hastur.hellion.org.uk> <4e207d63809542544ebe45866e746e91@mail.shatteredsilicon.net> <51F966E5.9080006@bobich.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <51F966E5.9080006@bobich.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Gordan Bobic Cc: Andrew Bobulsky , Andrew Cooper , Ian Campbell , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 31/07/13 20:35, Gordan Bobic wrote: > On 07/31/2013 06:53 PM, George Dunlap wrote: >> On Fri, Jul 26, 2013 at 2:11 PM, Gordan Bobic wrote: >>>> Now that is intereting - if this makes the memory holes the same >>>> between >>>> the guest and the host, does it also implicitly vBAR=pBAR? >>> >>> >>> Another thing that occurred to me might be useful to check - it is >>> pretty easy to modify the BAR size on Nvidia cards. The defaults are >>> 64MB and 128MB for the two BARs. They can be made much, much larger, >>> and there is often advantage to enlarging them to at least be equal to >>> VRAM size. Soooooo... If I boost the BAR from 128MB to 2GB, being a >>> 64-bit BAR, it might make the BIOS do the sane thing and map it above >>> 4GB. With the other BAR also suitably enlarged and it being done on >>> the second GPU as well, there is no obvious option but to map them >>> above 4GB (unless the BIOS is broken, which it may well be, in >>> which case all bets are off). >>> >>> Which may just alleviate the memory issue if not completely fix >>> the problem. >>> >>> Will try this and see what happens. >> >> I believe XenServer has a patch that allows the toolstack (in this >> case xapi) to set the default size of the MMIO hole. Andrew, did that >> ever make it upstream? >> >> Unfortunately, it is unlikely to work with upstream qemu until we fix >> the memory relocation issue... > > Interesting you should mention something like this. I've been > pondering whether it might be easier (even if it is a bodge) to simply > always set the domU E820 map to have 0x80000000 - 0xFFFFFFFF > (2GB->4GB) reserved. I have not yet seen a motherboard that maps > 32-bit BARs below 2GB. I'm pretty sure we've seen a memory hole larger than 2GiB, in a box loaded up with a boatload of GPUs. The main problem with doing this unconditionally is that the relocated memory isn't available to non-PAE 32-bit guests. I think we should have a work-around in place for 4.4 that will avoid a collision between the host MMIO and guest memory addresses; but it will need to be off by default, at least for guests that don't have a passed-through device. -George