From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Scott Parish" Subject: Re: comment request: dom0 dma on large memory systems Date: Sat, 4 Jun 2005 03:59:53 +0000 Message-ID: <20050604035952.GA3182@us.ibm.com> References: <571ACEFD467F7749BC50E0A98C17CDD806E9F77F@pdsmsx403> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <571ACEFD467F7749BC50E0A98C17CDD806E9F77F@pdsmsx403> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Tian, Kevin" Cc: xen-devel@lists.xensource.com, Scott Parish List-Id: xen-devel@lists.xenproject.org On Sat, Jun 04, 2005 at 11:48:16AM +0800, Tian, Kevin wrote: > ----Original Message----- > >From: xen-devel-bounces@lists.xensource.com > >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Scott > Parish > >Sent: Friday, June 03, 2005 3:35 PM > > > >On x86_64 with 6gig ram, dom0's initial allocation is from memory > >above the pci hole (referred to as "high memory" in this email) if > >dom0_mem is set to 2g or higher. The only problem is that most io/dma > >devices (non-dac) can only dma to the first 32bits worth of machine > >addresses--thus for some configurations, dom0 has no memory which is > >dma-able. > > IIRC, 2 or 3 months ago, Keir said that default memory allocation for > Dom0 is all available memory. And then CP has to decrease by balloon > interface before creating other domains. If this still holds true, I'm > not sure whether above problem still exists, since all avail memory > including both <4G and >4G belonging to Dom0 then. (XEN itself only > consumes a small trunk). However after looking at your patch and then > the source, it seems that only the max available order, meaning must be > continuous, is allocated to Dom0 currently. So did I misunderstand this > concept? If it really only means maximum continuous trunk, then you > patch definitely shoots straight on the real problem on all 64bit > platform. ;-) Right, there are several hacks around this problem, a couple i've thought of are: + enforce dom0 take all memory + drop the max order size for MEMZONEs to 18 (in which case alloc_largest should always allocate from the lower memory) + prealloc X amount of low memory (128M for instance) and add it into the dom0 allocation You nailed it when you mentioned driver domains (next email); the long term goal is to make sure we're able to support them and hopefully avoid the hogging of that memory unnecessarily for non-dma uses. Thanks for noticing ;^) (i was also glad Ian brought up numa, i had forgotten about it and this is probably a good time to think about that while i'm tearing up this code) sRp -- Scott Parish Signed-off-by: srparish@us.ibm.com