Re: [PATCH] RFC: initial libxl support for xenpaging

[Ian Campbell <Ian.Campbell@citrix.com>]

On Mon, 2012-02-20 at 14:48 +0000, George Dunlap wrote:
> On Mon, Feb 20, 2012 at 11:12 AM, Olaf Hering <olaf@aepfle.de> wrote:
> > On Mon, Feb 20, George Dunlap wrote:
> >
> >> Start-of-day:
> >>  xenpaging off: Set balloon target to M, use PoD
> >>  xenpaging on: ??
> >>  xenpaging delay: Set balloon target to M, use PoD.  Wait
> >> $pagingdelayboot seconds, if target not reached, set paging?
> >
> > Is the delay required?
> > If paging and PoD target is M, xenpaging will do nothing because the
> > guest can not exceed M (it will crash with OOM).
> 
> Ah, of course -- you don't need paging because it already has M
> memory.  Forgot about that.
> 
> It would be nice, of course, if the pager could act as a back-fill for
> these PoD pages; but that's another project, I think.
> 
> So that leaves us with:
> 
> maxmem=X
> memory=M
> xenpaging=[off|on|delay]
> pagingdelay=60

FWIW these two can be expressed as:
xenpaging=[off|on]
pagingdelay=[0|60]

(and lets drop the "xen" prefix)

[...]
> xl mem-set domain M
>  xenpaging off: Set balloon target to M
>  xenpaging on: Set paging target to M
>  xenpaging delay: Set balloon target to M, and wait for actual memory
> to reach M.  If it hasn't reached it by $paging_delay seconds, set
> balloon target to M.

Did you mean "paging target" the second time you said "balloon target"
in this one? I'll assume so. I would also suggest 
	s/If it hasn't reached it by/After/
since I think that will simplify things somewhat and setting page target
to M makes no odds if the guest has ballooned to M.

I don't really like mem-set having such completely different behaviour
depending on whether paging is on or off.

As you described before having paging on == set paging and balloon
target to M results in fairly suboptimal behaviour and the name would
also lead to people thinking it is the one they should use.

So why not make the "on" case the same as your "delay" case and do away
with the distinction? If advanced users really want what you describe as
"on" then they can set the delay to 0.

If the paging daemon could be start/stopped on demand (rather than being
a domain build time choice) we could even consider making paging the
default.

> xl mem-balloon-set domain M
>  Set balloon target to M
> xl mem-paging-set domain M
>  Set paging target to M

How do these interact with mem-set. Especially in the delay case?

e.g. would mem-paging-set disable the after delay behaviour of mem-set?
Should we have "mem-paging-set domain auto" to turn that back on?

We also need to consider the behaviour of mem-set to increase things.
Obviously you don't want to leave paging target set to the smaller value
for a minute after setting the balloon target. I think we want to set it
straight away in that case, if not before setting the balloon.

How about the following? I've tried to include the "user facing"
description as well as the actual implementation. I think the "user
facing" portion is actually where we disagree but I also suspect that we
may not actually disagree -- it's just that we are talking in terms of
implementation so we don't see that the user facing interface is the
same in what we are each thinking of ;-)

maxmem=X			# maximum RAM the domain can ever see
memory=M			# current amount of RAM seen by the domain
paging=[off|on]			# allow the amount of memory a guest 
				# thinks it has to differ from the
				# amount actually available to it (its
				# "footprint")
pagingauto=[off|on] (dflt=on)   # enable automatic enforcement of 
				# "footprint" for guests which do not
				# voluntarily obey changes to memory=M 
pagingdelay=60			# amount of time to give a guest to 
				# voluntarily comply before enforcing a 
				# footprint

xl mem-set domain M
        Sets the amount of RAM which the guest believes it has available
        to M. The guest should arrange to use only that much RAM and
        return the rest to the hypervisor (e.g. by using a balloon
        driver). If the guest does not do so then the host may use
        technical means to enforce the guest's footprint of M. The guest
        may suffer a performance penalty for this enforcement.

	paging off:	set balloon target to M.
	paging on: 	set balloon target to M.
			if pagingauto:
				wait delay IFF new target < old
				set paging target to M
				support -t <delay> to override default?

xl mem-paging-set domain N
        Overrides the amount of RAM which the guest actually has
        available (its "footprint") to N. The host will use technical
        means to continue to provide the illusion to the guest that it
        has memory=M (as adjusted by mem-set). There may be a
        performance penalty for this.
        
	paging off:	error
	paging on:	set paging target
			set pagingauto=off

xl mem-paging-set domain auto
        Automatically manage paging. Request that the guest uses
        memory=M (current value of memory, as adjusted by mem-set)
        enforced when the guest is uncooperative (as described in
        "mem-set")
        
	paging off:	error
	paging on:	set paging target to M
			set pagingauto=on

No need for a separate balloon-set since that == mem-set with
pagingauto=off.

Perhaps a separate "mem-paging-set domain manual" would be handy to
enable that mode without having to remember M so you can use it as N 

We could consider making "mem-paging-set domain N" fail with an error
unless you previously set manual, to prevent users accidentally
disabling the recommended automatic behaviour e.g. by typing
mem-paging-set when they mean mem-set.

I liked Andres' suggestions of footprint as a term here BTW so I would
prefer "mem-footprint-set" to "mem-paging-set" (at least I think so, I'm
not 100% on that). If we don't have balloon-set then avoiding the name
paging seems like a good idea too. Other possible names might be
"mem-override-set" or something.

I don't really like the extra configuration option for pagingauto but I
think pagingauto and mem-{paging,footprint}-set should be considered
advanced options and by default we would recommend that folks just set
"paging=on" and use mem-set. It should be reasonably clear to users that
if they disable auto mode then they are expected to understand what is
happening sufficiently to make their own choices about paging targets
etc.

We can probably think of more useful algorithms than raw pagingdelay
(i.e. based on rate of progress or something) which might be useful for
larger domains making large changes to the balloon -- lets leave that
aside for now though. Likewise "auto" mode allows scope for us to
implement improved algorithms in the future.

Ian.