Re: Proposed new "memory capacity claim" hypercall/feature

[Keir Fraser <keir@xen.org>]

On 29/10/2012 21:08, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> The core issue is that, in the hypervisor, every current method of
> "allocating RAM" is slow enough that if you want to allocate millions
> of pages (e.g. for a large domain), the total RAM can't be allocated
> atomically.  In fact, it may even take minutes, so currently a large
> allocation is explicitly preemptible, not atomic.
>
> The problems the proposal solves are (1) some toolstacks (including
> Oracle's "cloud orchestration layer") want to launch domains in parallel;
> currently xl/xapi require launches to be serialized which isn't very
> scalable in a large data center;

Well it does depend how scalable domain creation actually is as an
operation. If it is spending most of its time allocating memory then it is
quite likely that parallel creations will spend a lot of time competing for
the heap spinlock, and actually there will be little/no speedup compared
with serialising the creations. Further, if domain creation can take
minutes, it may be that we simply need to go optimise that -- we already
found one stupid thing in the heap allocator recently that was burining
loads of time during large-memory domain creations, and fixed it for a
massive speedup in that particular case.

> and (2) tmem and/or other dynamic
> memory mechanisms may be asynchronously absorbing small-but-significant
> portions of RAM for other purposes during an attempted domain launch.

This is an argument against allocate-rather-than-reserve? I don't think that
makes sense -- so is this instead an argument against
reservation-as-a-toolstack-only-mechanism? I'm not actually convinced yet we
need reservations *at all*, before we get down to where it should be
implemented.

 -- Keir

> In either case, this is a classic race, and a large allocation may
> unexpectedly fail, possibly even after several minutes, which is
> unacceptable for a data center operator or for automated tools trying
> to launch any very large domain.
> 
> Does that make sense?  I'm very open to other solutions, but the
> only one I've heard so far was essentially "disallow independent
> dynamic memory allocations" plus keep track of all "claiming" in the
> toolstack.