RFC: Doing a superpage zero-sweep on decrease_reservation

RFC: Doing a superpage zero-sweep on decrease_reservation

[George Dunlap]

I'm going through the populate-on-demand patches that we've had in our
XenServer patchqueue and trying to figure out what he best thing is to
do.

One of the things we implemented was a hypercall that would allow the
tools to initiate a zero-page sweep.  The idea for this actually was
to try to help keep contiguous superpages if possible.  Asking Windows
for 2meg pages couldn't be made to work properly (Paul said, you can
ask Windows, but it just wouldn't give any). When ballooning down, the
balloon driver first calls for a zero-page sweep, hoping Xen will find
big contiguous zero superpages it can reclaim; then as the balloon
driver returns individual pages, eventually it will end up filling in
a lot of whole superpages anyway.

The problem with this is that for machines with very large memory,
this hypercall can take a long time; and if the VM has already been
running for a while, it's likely that a significant portion of the
space *won't* be zero.

It occured to me, it might be simpler if we check for a zero superpage
when doing a decrease_reservation call instead.  We'd only do a sweep
if the p2m entry is currently a superpage.  If that's the case, we
sweep the superpage containing that mfn and nothing else.  If the
whole superpage isn't zero, we'll shatter the superpage in the p2m
tables, so that even if we get an mfn from the same page again, we
won't scan it.

I was going to say, "scan only if the number of p2m entries is higher
than the number of entries in the cache".  But it occurred to me, it
might not be a bad idea to scan for zero pages in any case  -- so that
we can consolidate even after the guest has been running for a while.

Thoughts?
 -George

Re: RFC: Doing a superpage zero-sweep on decrease_reservation

[Tim Deegan]

Hi, 

At 13:47 +0000 on 09 Mar (1299678468), George Dunlap wrote:
> It occured to me, it might be simpler if we check for a zero superpage
> when doing a decrease_reservation call instead.  We'd only do a sweep
> if the p2m entry is currently a superpage.  If that's the case, we
> sweep the superpage containing that mfn and nothing else.  If the
> whole superpage isn't zero, we'll shatter the superpage in the p2m
> tables, so that even if we get an mfn from the same page again, we
> won't scan it.

Do you think ballooned-out pages are likely to be in 2MB chunks of
zeroed memory?  Apart from avoiding shattering and re-gathering a 2MB
p2m range (which is nice, but proabbly not the perf bottleneck for PoD
right now), this is a change from scanning arbitrary memory when we run
out to scanning memory around balloon sites as we go.

If this is likely to have a high hit rate, great.  If not, it might be
(overall) worse because we waste time scanning as we balloon and have to
do the full scan later anyway.

> I was going to say, "scan only if the number of p2m entries is higher
> than the number of entries in the cache".  But it occurred to me, it
> might not be a bad idea to scan for zero pages in any case  -- so that
> we can consolidate even after the guest has been running for a while.

I'm not so keen on that - AFAICS once we've passed #pod-entries ==
#pod-pages, we shouldn't be doing any expensive work for PoD.  We might
want to think about defragging HVM guests independently of the PoD
stuff, though. 

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: RFC: Doing a superpage zero-sweep on decrease_reservation

[George Dunlap]

On Wed, 2011-03-09 at 14:23 +0000, Tim Deegan wrote:
> Hi, 
> 
> At 13:47 +0000 on 09 Mar (1299678468), George Dunlap wrote:
> > It occured to me, it might be simpler if we check for a zero superpage
> > when doing a decrease_reservation call instead.  We'd only do a sweep
> > if the p2m entry is currently a superpage.  If that's the case, we
> > sweep the superpage containing that mfn and nothing else.  If the
> > whole superpage isn't zero, we'll shatter the superpage in the p2m
> > tables, so that even if we get an mfn from the same page again, we
> > won't scan it.
> 
> Do you think ballooned-out pages are likely to be in 2MB chunks of
> zeroed memory?  Apart from avoiding shattering and re-gathering a 2MB
> p2m range (which is nice, but proabbly not the perf bottleneck for PoD
> right now), this is a change from scanning arbitrary memory when we run
> out to scanning memory around balloon sites as we go.

On the contrary, the function of the hypercall was to do a full memory
sweep regardless of whether the cache was running low or not -- that is,
the purpose of the hypercall (as used by the Citrix PV drivers) has
nothing to do with out-of-memory situations; it only has to do with
superpage consolidation.

(Avoiding confusion in the future over what the hypercall is actually
for is another good reason not to re-use the emergency sweep code if we
don't need to!)

> If this is likely to have a high hit rate, great.  If not, it might be
> (overall) worse because we waste time scanning as we balloon and have to
> do the full scan later anyway.

This shouldn't have a significant impact on the number of emergency
sweeps we do -- in theory we shouldn't need to do any sweeps once the
balloon driver starts handing memory back to Xen.  But I will run some
tests, to see what impact it has on boot time, and number of intact
superpages returned to Xen as a part of the initial balloon process.

> > I was going to say, "scan only if the number of p2m entries is higher
> > than the number of entries in the cache".  But it occurred to me, it
> > might not be a bad idea to scan for zero pages in any case  -- so that
> > we can consolidate even after the guest has been running for a while.
> 
> I'm not so keen on that - AFAICS once we've passed #pod-entries ==
> #pod-pages, we shouldn't be doing any expensive work for PoD.  We might
> want to think about defragging HVM guests independently of the PoD
> stuff, though. 

That sounds reasonable.  

OK, hopefully I'll have a patch (with some experiments to back it up) in
a week or so.

 -George

Re: RFC: Doing a superpage zero-sweep on decrease_reservation

[George Dunlap]

On Wed, 2011-03-09 at 14:23 +0000, Tim Deegan wrote:
> Do you think ballooned-out pages are likely to be in 2MB chunks of
> zeroed memory?

Sorry, didn't answer this.  I think that for a large guest, the initial
balloon is very likely that for the first mfn freed in a superpage, that
mfn will turn out to have been in a 2MB chunks of zeroed memory, all of
which will eventually passed back to Xen anyway.  (e.g., imagine a VM
with 32GiB static_max but a 16GiB target -- of the 16GiB of individual
4k pages passed back by the balloon driver, the vast majority of them
will be in a page which (1) is zeroed already and (2) will end up being
passed back entirely anyway.)

One could imagine the PV drivers allocating the full balloon size, then
going through and sorting the pages by pfn, consolidating the
superpages, and passing them back as one big chunk -- but that would be
a big waste of time compared to just having Xen optimistically reclaim
the whole superpage.

 -George