Re: Proposed new "memory capacity claim" hypercall/feature

[Dan Magenheimer <dan.magenheimer@oracle.com>]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, October 30, 2012 2:29 AM
> To: Dan Magenheimer
> Cc: Olaf Hering; IanCampbell; GeorgeDunlap; IanJackson; George Shuklin; DarioFaggioli; xen-
> devel@lists.xen.org; Konrad Wilk; Kurt Hackel; Mukesh Rathor; Zhigang Wang; Keir (Xen.org); Tim Deegan
> Subject: RE: Proposed new "memory capacity claim" hypercall/feature
> 
> >>> On 30.10.12 at 00:21, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> >>  From: Tim Deegan [mailto:tim@xen.org]
> >> As I said, I'm not opposed to this, though even after reading through
> >> the other thread I'm not convinced that it's necessary (except in cases
> >> where guest-controlled operations are allowed to consume unbounded
> >> memory, which frankly gives me the heebie-jeebies).
> >
> > A really detailed discussion of tmem would probably be good but,
> > yes, with tmem, guest-controlled* operations can and frequently will
> > absorb ALL physical RAM.  However, this is "freeable" (ephemeral)
> > memory used by the hypervisor on behalf of domains, not domain-owned
> > memory.
> >
> > * "guest-controlled" I suspect is the heebie-jeebie word... in
> >   tmem, a better description might be "guest-controls-which-data-
> >   and-hypervisor-controls-how-many-pages"
> 
> But isn't tmem use supposed to be transparent in this respect, i.e.
> if a "normal" allocation cannot be satisfied, tmem would jump in
> and free sufficient space? In which case there's no need to do
> any accounting outside of the control tools (leaving aside the
> smaller hypervisor internal allocations, which the tool stack needs
> to provide room for anyway).

Hi Jan --

Tmem can only "free sufficient space" up to the total amount
of ephemeral space of which it has control (ie. all "freeable"
memory).

Let me explain further:  Let's oversimplify a bit and say that
there are three types of pages:

a) Truly free memory (each free page is on the hypervisor free list)
b) Freeable memory ("ephmeral" memory managed by tmem)
c) Owned memory (pages allocated by the hypervisor or for a domain)

The sum of these three is always a constant: The total number of
RAM pages in the system.  However, when tmem is active, the values
of all _three_ of these change constantly.  So if at the start of a
domain launch, the sum of free+freeable exceeds the intended size
of the domain, the domain allocation/launch can start.  But then
if "owned" increases enough, there may no longer be enough memory
and the domain launch will fail.

With tmem, memory "owned" by domain (d.tot_pages) increases dynamically
in two ways: selfballooning and persistent puts (aka frontswap),
but is always capped by d.max_pages.  Neither of these communicate
to the toolstack.

Similarly, tmem (or selfballooning) may be dynamically freeing up lots
of memory without communicating to the toolstack, which could result in
the toolstack rejecting a domain launch believing there is insufficient
memory.

I am thinking the "claim" hypercall/subop eliminates these problems
and hope you agree!

Thanks,
Dan