Re: RE: Ballooning up

[Jeremy Fitzhardinge <jeremy@goop.org>]

 On 09/13/2010 05:22 PM, Dan Magenheimer wrote:
>>> I would call this dom0 functionality a bug.  I think both Citrix
>>> and Oracle use dom0_mem as a normal boot option for every
>>> installation and, while I think both employ heuristics to choose
>>> a larger dom0_mem for larger physical memory, I don't think it
>>> grows large enough for, say, >256GB physical memory, to accommodate
>>> the necessarily large number of page tables.
>>>
>>> So, I'd vote for NOT allowing dom0 to balloon up to physical
>>> memory if dom0_mem is specified, and possibly a kernel command
>>> line option that allows it to grow beyond.  Or, possibly, no
>>> option and never allow dom0 memory to grow beyond dom0_mem
>>> unless (possibly) it grows with hot-plug.
>> Yes, its a bit of a problem.  The trouble is that the kernel can't
>> really distinguish the two cases; either way, it sees a Xen-supplied
>> xen_start_info->nr_pages as the amount of initial memory available, and
>> an E820 table referring to more RAM beyond that.
>>
>> I guess there are three options:
>>
>>    1. add a "xen_maxmem" (or something) kernel parameter to override
>>       space specified in the E820 table
>>    2. ignore E820 if its a privileged domain
>>    3. only allow extra memory up to a certain ratio of the base memory
>>       (8x? 16x? 32x?)
>>
>> I think the third is probably the simplest and least hacky, as it
>> directly addresses the underlying issue (and prevents domU mishaps as
>> well).
> I like 2': ignore E820 if it is dom0 and dom0_mem has been specified.
> This most closely conforms to current behavior in shipping systems
> and I don't really see a use model for allowing dom0 memory to
> grow beyond dom0_mem (if dom0_mem is specified).

The kernel never gets to see dom0_mem, since that's a Xen parameter.

Only doing the limit doesn't help with domUs, potentially have the same
problem.

> (1) will most likely result in vendors specifying dom0_mem AND
>     xen_maxmem to the same value, so IMHO will just be confusing

Who for?   Dom0 won't be able to balloon up, but if dom0 is being
managed by a vendor software stack, that doesn't matter much.

> (2) for non-dom0 privileged domains, I can't offhand come up with
>     a scenario where memory<>maxmem would be valuable, so this
>     would be my second choice (after 2').

What doe you mean by "memory<>maxmem"?

> (3) seems like policy enforcement with insufficient information
>     as the "correct" ratio might change in future kernels and
>     we don't even know what it should be now (and it may be
>     very kernel dependent?)

Not really.  The main structure that scales with memory size is the
struct page array, which is about 64 bytes per page (so, putting an
upper limit of ~64x if you don't mind all kernel memory being filled
with struct pages).  For systems with highmem it must be in lowmem, so
the scaling would be on base low memory, not all base memory.  But,
true, if you don't intend to balloon up, there's no point wasting memory
on unused page structures.

    J