Re: PoD, 4.2, and current/maximum reservation

[George Dunlap <george.dunlap@eu.citrix.com>]

On 11/02/13 11:39, James Harper wrote:
>> On Mon, Feb 11, 2013 at 12:03 AM, James Harper
>> <james.harper@bendigoit.com.au> wrote:
>>
>> 	A user has pointed out a problem with GPLPV under Xen 4.2 when
>> using PoD. I'm using the difference between
>> XENMEM_maximum_reservation and XENMEM_current_reservation to tell
>> me how much I should balloon down to account for PoD, but when the user
>> has ballooned down to 1G (from 4Gb or 8GB), GPLPV logs the following:
>>
>> 	13005008825593: XenPCI     XENMEM_maximum_reservation =
>> 262400
>> 	13005008825593: XenPCI     XENMEM_current_reservation = 262136
>> 	13005008825609: XenPCI     Trying to give 1056 KB (1 MB) to Xen
>>
>> 	What is the correct way to tell how much PoD memory there is under
>> 4.2? Am I doing it wrong?
>>
>> 	I balloon down as early as possible (before xenbus starts) to avoid
>> windows going over its limit so I'm hoping I can determine the size of PoD
>> memory just via hypercalls.
>>
>>
>>
>> You shouldn't have to know anything specifically about PoD -- you should just
>> look at the balloon target for the guest written in xenstore.  The idea was as
>> much as possible for the toolstack and Xen to work together to make it
>> transparent to the balloon driver, in part because we expected to be running
>> legacy drivers.  The Citrix PV drivers don't do anything special wrt PoD
>> memory.  (Paul, please correct me if I'm wrong.)
> So I should just balloon down to the current_reservation figure right?

...I don't think so -- it looks like you're getting that from a 
hypercall, not from xenstore.  You want the normal balloon target value 
from xenstore.  (But I might be confused -- I'm not super familiar with 
the technical details of the ballooning codepath, just the overall 
principles.)

>> WRT timing and xenbus, a couple of things:
>>
>> * Windows does a scrub of all memory at start-of-day.  Especially on multiple-
>> vcpu systems, we were unable to start the balloon process early enough to
>> win the race against the scrubber, so we had to have ways of "reclaiming"
>> zeroed pages for the PoD pool.  What this means is that it's not a matter of
>> Windows *touching* memory, but of windows *dirtying* memory that will
>> lead to a problem.
>>
>> * So there is a minimum amount of memory Windows needs to be able to
>> make it to the stage where the balloon driver can run.  When XenServer first
>> implemented DMC, the team did extensive testing to determine minimum
>> values above which Windows never crashed or hung, and (as I understand it)
>> baked those into the xapi toolstack as a "seatbelt" to prevent users from
>> setting the value too low.
>>
>> Not sure if that helps in your particular case -- I think 1G was within the limit,
>> but I could be wrong.  Dave, any comments?
>>
> I think I could go from 4GB down to 128MB reliably without crashing, although the resulting OS wasn't particularly usable :)

I think what we found was that Windows required a certain amount of 
memory to *boot*.  If you gave it less than that, then either one of two 
things would happen:
* The processes dirtying RAM beat the balloon driver, and the guest 
crashed with a PoD-cache-empty error
* The balloon driver beat the processes dirtying ram, and the guest 
hung; presumably some service somewhere waiting for some memory to "free 
up".

In theory after boot you could reduce the memory below that threshold.  
But then what would you do if you needed to reboot the guest?  You'd 
have to go grab the memory from somewhere else, which is the situation 
we were trying to avoid with having PoD in the first place.  So the guy 
architecting DMC just made it a hard floor (again, as I understand it).

  -George