Re: domain creation vs querying free memory (xend and xl)

[Dan Magenheimer <dan.magenheimer@oracle.com>]

> From: Konrad Rzeszutek Wilk
> Subject: Re: domain creation vs querying free memory (xend and xl)
> 
> On Wed, Sep 26, 2012 at 02:17:06PM -0700, Dan Magenheimer wrote:
> > I was asked a question that seems like it should be obvious
> > but it doesn't seem to be, at least in xm-land.  I'll look
> > into it further, as well as for xl, but I thought I'd ask
> > first to see if there is a known answer or if this is a known
> > problem:
> >
> > Suppose that xm/xl create is issued on a large-memory
> > domain (PV or HVM or, future, PVH).  It takes awhile
> > for this domain to launch and during at least part of this
> > time, the toolset hasn't yet requested all of the
> > required memory from the hypervisor to complete the
> > launch of the domain...  or perhaps the toolset has,
> > but the hypervisor is slow about calling the long sequence
> > of page allocations (e.g. maybe because it is zeroing
> > each page?).
> >
> > Then it is desired to launch a second large-memory domain.
> > The tools can query Xen to see if there is sufficient RAM
> > and there is, because the first launch has not yet
> > allocated all the RAM assigned to it.
> >
> > But the second domain launch fails, possibly after
> > several minutes because, actually, there isn't enough
> > physical RAM for both.
> >
> > Does this make sense?  Should the tools "reserve"
> > maxmem as a "transaction" and/or ensure that "xm/xl
> > free" calls account for the entire requested amount
> > of RAM?  Or maybe xl _does_ work this way?
> 
> So say "freeze" the amount of free memory. Lets CC the XCP folks

Hmmm... the problem is the opposite (I think, since I don't
have hardware at hand to reproduce it).

Assume a machine has 2TB of physical RAM and a "xm create"
is started to launch a 1TB guest called "X".  While X is
being launched, another thread watches "xm free" and sees
that it slowly goes down from 1.995TB.  That thread does not
know what the eventual "floor" will be.  Now a third thread
does a "xm create" to launch a second 1TB guest "Y".
The "xm create" asks the hypervisor and sees, yep, there
is, at this moment, 1.376TB of free memory so it commences
launching the guest.

Because the hypervisor and dom0 consume some RAM, both
of these "xm create" will eventually fail, possibly
after several minutes.

Seems like a "xm unreserved" is needed, similar to "xm free"
but takes into account the tools' knowledge of what RAM is
in the process of being reserved for launching domains,
not just the allocation requests the hypervisor has already
processed.