Re: comment request: dom0 dma on large memory systems

["Scott Parish" <srparish@us.ibm.com>]

On Sat, Jun 04, 2005 at 11:48:16AM +0800, Tian, Kevin wrote:

> ----Original Message-----
> >From: xen-devel-bounces@lists.xensource.com
> >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Scott
> Parish
> >Sent: Friday, June 03, 2005 3:35 PM
> >
> >On x86_64 with 6gig ram, dom0's initial allocation is from memory
> >above the pci hole (referred to as "high memory" in this email) if
> >dom0_mem is set to 2g or higher. The only problem is that most io/dma
> >devices (non-dac) can only dma to the first 32bits worth of machine
> >addresses--thus for some configurations, dom0 has no memory which is
> >dma-able.
> 
> IIRC, 2 or 3 months ago, Keir said that default memory allocation for
> Dom0 is all available memory. And then CP has to decrease by balloon
> interface before creating other domains. If this still holds true, I'm
> not sure whether above problem still exists, since all avail memory
> including both <4G and >4G belonging to Dom0 then. (XEN itself only
> consumes a small trunk). However after looking at your patch and then
> the source, it seems that only the max available order, meaning must be
> continuous, is allocated to Dom0 currently. So did I misunderstand this
> concept? If it really only means maximum continuous trunk, then you
> patch definitely shoots straight on the real problem on all 64bit
> platform. ;-)

Right, there are several hacks around this problem, a couple i've
thought of are:

  + enforce dom0 take all memory
  + drop the max order size for MEMZONEs to 18 (in which case
    alloc_largest should always allocate from the lower memory)
  + prealloc X amount of low memory (128M for instance) and add
    it into the dom0 allocation

You nailed it when you mentioned driver domains (next email); the long
term goal is to make sure we're able to support them and hopefully avoid
the hogging of that memory unnecessarily for non-dma uses. Thanks for
noticing ;^)  

(i was also glad Ian brought up numa, i had forgotten about it and this
is probably a good time to think about that while i'm tearing up this
code)

sRp

-- 
Scott Parish
Signed-off-by: srparish@us.ibm.com