Re: what happens when a PoD page is touched?

[George Dunlap <George.Dunlap@eu.citrix.com>]

On Tue, May 17, 2011 at 10:37 AM, James Harper
<james.harper@bendigoit.com.au> wrote:
> I still can't quite get my head around why this happens at all... I
> thought it would go like this:
>
> 1. Allocate 1MB of memory
> 2. Still under our limit so xen populates the pages when Windows clears
> them
> 3. Hand them back to xen
> 4. Repeat
>
> I'm doing that in a tight loop very early in boot. If I keep handing
> back pages (and thus reducing my populated page count) why am I hitting
> any PoD limit at all and invoking the page scavenging code? Windows
> isn't doing anything else at this point, and even if it was, I'm the
> boot driver so it has to wait for me before the boot can progress so
> it's not like it would be consuming gigabytes of memory.

Suppose for simplicity we're using 1G pages and not splintering them.

Suppose we boot a VM with 8G maxmem, 4G memory.

* Domain built
pages 0-7 are PoD, 4 pages in PoD pool.

* HVMLoader loaded into memory
Page 0 populated; 1-7 PoD, 3 pages in PoD pool

* Windows boots, starts scrubbing (from top of memory)
 + page 7 populated, 2 pages in PoD pool.
 + Page 6 populated, 1 page in PoD pool
 + Page 5 populated, 0 pages in PoD pool
 + Page 4 touched
  - emergency sweep marks page 7 PoD; page 4 populated
 + Page 3 touched
  - emergency sweep marks page 6 PoD; page 3 populated
 etc
 + page 1 touched
  - emergency sweep marks page 5 PoD; page 1 populated
 + Result at the end: pages 0-4 populated, 5-7 PoD, no pages in PoD pool


* Balloon driver starts.
 Let's suppose Windows happens to do all allocation from the PoD region.
 + Asks windows for 1 page
  - Windows chooses page 6, scrubs it.
  - emergency sweep marks page 4 PoD, populates page 6
  - Balloon driver hands page 6 back to Xen
  - page 5 marked empty; page put in PoD pool
 + Asks Windows for 1 more page
  - Windows chooses page 5, scrubs it.
  - Xen fills page 5 with the page in the PoD pool; PoD now empty
  - Balloon driver hands page 5 back to Xen
  - page 5 marked empty; page put in PoD pool
 + Repeat previous

So in theory, once you get your initial allocation from Windows, you
shouldn't have to do any sweeps.

The exception to this would be the case where you're allocating N
pages, where N > (used_pages - total allocation).

I guess try allocating fewer pages before returning them to Xen, and
seeing if that helps at all.

Hmm -- looks like I didn't upstream my PoD tracing patch, otherwise we
could see what was going on.  If you're interested in taking a look at
what's happening at a PoD level, I can try to dig it up and send it to
you.  (Probably need some porting because of the p2m restructuring Tim
Deegan has been doing.)

Peace,
 -George