Re: RFC: Doing a superpage zero-sweep on decrease_reservation

[George Dunlap <george.dunlap@citrix.com>]

On Wed, 2011-03-09 at 14:23 +0000, Tim Deegan wrote:
> Hi, 
> 
> At 13:47 +0000 on 09 Mar (1299678468), George Dunlap wrote:
> > It occured to me, it might be simpler if we check for a zero superpage
> > when doing a decrease_reservation call instead.  We'd only do a sweep
> > if the p2m entry is currently a superpage.  If that's the case, we
> > sweep the superpage containing that mfn and nothing else.  If the
> > whole superpage isn't zero, we'll shatter the superpage in the p2m
> > tables, so that even if we get an mfn from the same page again, we
> > won't scan it.
> 
> Do you think ballooned-out pages are likely to be in 2MB chunks of
> zeroed memory?  Apart from avoiding shattering and re-gathering a 2MB
> p2m range (which is nice, but proabbly not the perf bottleneck for PoD
> right now), this is a change from scanning arbitrary memory when we run
> out to scanning memory around balloon sites as we go.

On the contrary, the function of the hypercall was to do a full memory
sweep regardless of whether the cache was running low or not -- that is,
the purpose of the hypercall (as used by the Citrix PV drivers) has
nothing to do with out-of-memory situations; it only has to do with
superpage consolidation.

(Avoiding confusion in the future over what the hypercall is actually
for is another good reason not to re-use the emergency sweep code if we
don't need to!)

> If this is likely to have a high hit rate, great.  If not, it might be
> (overall) worse because we waste time scanning as we balloon and have to
> do the full scan later anyway.

This shouldn't have a significant impact on the number of emergency
sweeps we do -- in theory we shouldn't need to do any sweeps once the
balloon driver starts handing memory back to Xen.  But I will run some
tests, to see what impact it has on boot time, and number of intact
superpages returned to Xen as a part of the initial balloon process.

> > I was going to say, "scan only if the number of p2m entries is higher
> > than the number of entries in the cache".  But it occurred to me, it
> > might not be a bad idea to scan for zero pages in any case  -- so that
> > we can consolidate even after the guest has been running for a while.
> 
> I'm not so keen on that - AFAICS once we've passed #pod-entries ==
> #pod-pages, we shouldn't be doing any expensive work for PoD.  We might
> want to think about defragging HVM guests independently of the PoD
> stuff, though. 

That sounds reasonable.  

OK, hopefully I'll have a patch (with some experiments to back it up) in
a week or so.

 -George