Re: domain creation vs querying free memory (xend and xl)

[George Dunlap <george.dunlap@eu.citrix.com>]

On 04/10/12 17:54, Dan Magenheimer wrote:
>>
> Scanning through the archived message I am under the impression
> that the focus is on a single server... i.e. "punt if actor is
> not xl", i.e. it addressed "balloon-to-fit" and only tries to avoid
> stepping on other memory overcommit technologies.  That makes it
> almost orthogonal, I think, to the problem I originally raised.
No, the idea was to allow the flexibility of different actors in 
different situations.  The plan was to start with a simple actor, but to 
add new ones as necessary.  But on reflection, it seems like the whole 
"actor" thing was actually something completely separate to what we're 
talking about here.  The idea behind the actor (IIRC) was that you could 
tell the toolstack, "Make VM A use X amount of host memory"; and the 
actor would determine the best way to do that -- either by only 
ballooning, or ballooning first and then swapping.  But it doesn't 
decide how to get the value X.

This thread has been very hard to follow for some reason, so let me see 
if I can understand everything:
* You are concerned about being able to predictably start VMs in the 
face of:
  - concurrent requests, and
  - dynamic memory technologies (including PoD, ballooning, paging, page 
sharing, and tmem)
Any of which may change the amount of free memory between the time a 
decision is made and the time memory is actually allocated.
* You have proposed a hypervisor-based solution that allows the 
toolstack to "reserve" a specific amount of memory to a VM that will not 
be used for something else; this allocation is transactional -- it will 
either completely succeed, or completely fail, and do it quickly.

Is that correct?

The problem with that solution, it seems to me, is that the hypervisor 
does not (and I think probably should not) have any insight into the 
policy for allocating or freeing memory as a result of other activities, 
such as ballooning or page sharing.  Suppose someone were ballooning 
down domain M to get 8GiB in order to start domain A; and at some point 
, another process looks and says, "Oh look, there's 4GiB free, that's 
enough to start domain B" and asks Xen to reserve that memory.  Xen has 
no way of knowing that the memory freed by domain M was "earmarked" for 
domain A, and so will happily give it to domain B, causing domain A's 
creation to fail (potentially).

So it seems like we need to have the idea of a memory controller -- one 
central process (per host, as you say) that would know about all of the 
knobs -- ballooning, paging, page sharing, tmem, whatever -- that could 
be in charge of knowing where all the memory was coming from and where 
it was going.  So if xl wanted to start a new VM, it can ask the memory 
controller for 3GiB, and the controller could decide, "I'll take 1GiB 
from domain M and 2 from domain N, and give it to the new domain", and 
respond when it has the memory that it needs.  Similarly, it can know 
that it should try to keep X megabytes for un-sharing of pages, and it 
can be responsible for freeing up more memory if that memory becomes 
exhausted.

At the moment, the administrator himself (or the cloud orchestration 
layer) needs to be his own memory controller; that is, he needs to 
manually decide if there's enough free memory to start a VM; if there's 
not, he needs to figure out how to get that memory (either by ballooning 
or swapping).  Ballooning and swapping are both totally under his 
control; the only thing he doesn't control is the unsharing of pages.  
But as long as there was a way to tell the page sharing daemon not to 
allocate an amount of free memory, then this 
"administrator-as-memory-controller" should work just fine.

Does that make sense?  Or am I still confused? :-)

  -George