PoD in other (not GPLPV) drivers

PoD in other (not GPLPV) drivers

[James Harper]

I'm looking at PoD support in GPLPV. There was a bug preventing the
initial balloon down of DomU which I've now fixed, but it's still a bit
of a race with Windows to try and balloon down the pages before Windows
tries to use them. Under Windows 2008 x32, memory=512 + maxmem=1024 is
enough to crash the system just after my DriverEntry is called.

Do other Windows PV drivers support PoD? Under what scenarios? I'm
trying to figure out how achievable this is.

Thanks

James

Re: PoD in other (not GPLPV) drivers

[George Dunlap]

We tested a wide range of memory parameters for the Citrix PV drivers
on XenServer.  I just asked the guy in charge of that, and he said
that they were able to reliably boot w2k8x32 with memory=128MiB and
maxmem=1024MiB.

I know we had to do make some changes to move the balloon driver
allocation further back in the boot process to make that happen; but
I'm not familiar with the details.  Paul would be the person to ask
about that.

 -George

On Sun, Feb 27, 2011 at 1:47 AM, James Harper
<james.harper@bendigoit.com.au> wrote:
> I'm looking at PoD support in GPLPV. There was a bug preventing the
> initial balloon down of DomU which I've now fixed, but it's still a bit
> of a race with Windows to try and balloon down the pages before Windows
> tries to use them. Under Windows 2008 x32, memory=512 + maxmem=1024 is
> enough to crash the system just after my DriverEntry is called.
>
> Do other Windows PV drivers support PoD? Under what scenarios? I'm
> trying to figure out how achievable this is.
>
> Thanks
>
> James
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: PoD in other (not GPLPV) drivers

[Paul Durrant]

I actually have plans to push it earlier because we balloon down quite late at the moment (off the back of the START IRP in the top level bus driver). We are reliant upon zero-page sweeping code in Xen to save us from guest crashing up to that point.

  Paul

> -----Original Message-----
> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of
> George Dunlap
> Sent: 28 February 2011 11:59
> To: James Harper
> Cc: xen devel; Paul Durrant
> Subject: Re: [Xen-devel] PoD in other (not GPLPV) drivers
> 
> We tested a wide range of memory parameters for the Citrix PV
> drivers on XenServer.  I just asked the guy in charge of that, and
> he said that they were able to reliably boot w2k8x32 with
> memory=128MiB and maxmem=1024MiB.
> 
> I know we had to do make some changes to move the balloon driver
> allocation further back in the boot process to make that happen; but
> I'm not familiar with the details.  Paul would be the person to ask
> about that.
> 
>  -George
> 
> On Sun, Feb 27, 2011 at 1:47 AM, James Harper
> <james.harper@bendigoit.com.au> wrote:
> > I'm looking at PoD support in GPLPV. There was a bug preventing
> the
> > initial balloon down of DomU which I've now fixed, but it's still
> a
> > bit of a race with Windows to try and balloon down the pages
> before
> > Windows tries to use them. Under Windows 2008 x32, memory=512 +
> > maxmem=1024 is enough to crash the system just after my
> DriverEntry is called.
> >
> > Do other Windows PV drivers support PoD? Under what scenarios? I'm
> > trying to figure out how achievable this is.
> >
> > Thanks
> >
> > James
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >

RE: PoD in other (not GPLPV) drivers

[James Harper]

> 
> I actually have plans to push it earlier because we balloon down quite
late at
> the moment (off the back of the START IRP in the top level bus
driver). We are
> reliant upon zero-page sweeping code in Xen to save us from guest
crashing up
> to that point.
> 

I've modified GPLPV to balloon down at DriverEntry time, which seems to
be early enough. Prior to that, memory=128 and maxmem=1024 was enough to
cause a crash under 2008, basically as soon as I tried to access the
registry in DriverEntry. My drivers are using WDF and are therefore
loading after the KMDF framework which is going to use additional
resources. My backup plan is to write a WDM driver that loads even
earlier than that and does the allocation, passing it to the real PV
drivers later on, although my concern there is that Windows may not like
memory allocated by one driver being freed by another...

I've never heard of 'zero-page sweeping code' before... is that a way of
xen reallocating a previously touched page if it contains all 0's if we
want a page beyond our allocation limit? That might explain why my
initial balloon down is so slow! I can tell windows to not zero pages
before it gives them to me when I do the initial balloon down... what
are your thoughts on that? Although it's unlikely at boot time, in
theory they could contain sensitive information and I'm supposed to zero
them before handing them back to xen according to the docs.

James

RE: PoD in other (not GPLPV) drivers

[Paul Durrant]

James,

 Indeed, zero page sweeping is code inside xen that runs if the PoD cache is exhausted. It refills the cache with pages that are zeroed out (replacing them with pod entries in the p2m). Your initial balloon down may well be slow if you (or the windows kernel) touches the page before you hand it back to xen, since the pod code will have to populate each page in the p2m as it is touched.

  Paul

> -----Original Message-----
> From: James Harper [mailto:james.harper@bendigoit.com.au]
> Sent: 28 February 2011 12:29
> To: Paul Durrant; George Dunlap
> Cc: xen devel
> Subject: RE: [Xen-devel] PoD in other (not GPLPV) drivers
> 
> >
> > I actually have plans to push it earlier because we balloon down
> quite
> late at
> > the moment (off the back of the START IRP in the top level bus
> driver). We are
> > reliant upon zero-page sweeping code in Xen to save us from guest
> crashing up
> > to that point.
> >
> 
> I've modified GPLPV to balloon down at DriverEntry time, which seems
> to be early enough. Prior to that, memory=128 and maxmem=1024 was
> enough to cause a crash under 2008, basically as soon as I tried to
> access the registry in DriverEntry. My drivers are using WDF and are
> therefore loading after the KMDF framework which is going to use
> additional resources. My backup plan is to write a WDM driver that
> loads even earlier than that and does the allocation, passing it to
> the real PV drivers later on, although my concern there is that
> Windows may not like memory allocated by one driver being freed by
> another...
> 
> I've never heard of 'zero-page sweeping code' before... is that a
> way of xen reallocating a previously touched page if it contains all
> 0's if we want a page beyond our allocation limit? That might
> explain why my initial balloon down is so slow! I can tell windows
> to not zero pages before it gives them to me when I do the initial
> balloon down... what are your thoughts on that? Although it's
> unlikely at boot time, in theory they could contain sensitive
> information and I'm supposed to zero them before handing them back
> to xen according to the docs.
> 
> James

Re: PoD in other (not GPLPV) drivers

[George Dunlap]

Just for the public record: I believe that I have upstreamed all of
the PoD fixes and improvements that we had in the XenServer tree with
one exception: There are a series of patches to improve the "emergency
zero-page sweep" behavior in Xen, particularly for pretty large
(>24GiB) guests.  This is because:
* They were really ugly; hacks upon hacks, not suitable for upstreaming
* As far as I know, they're performance enhancements only, not correctness ones.

It doesn't sound to me like the zero-page sweeps should be involved in
your issue, James.  But if you (or anyone) would like the patches, I'd
be happy to send them along for you to try.  (And for the shy, or
archaeologists looking at this long after I'm gone from Citrix, they
can be found on the XenServer source ISOs.)  I'm not sure they'll
apply cleanly to 4.1, but that code hasn't had a lot of changes, so
you should be able to work out how they apply.

As part of my work to port XenServer to 4.1, I'm going to rewrite the
patches, and I'll upstream them at that time.

 -George

On Mon, Feb 28, 2011 at 1:41 PM, Paul Durrant <Paul.Durrant@citrix.com> wrote:
> James,
>
>  Indeed, zero page sweeping is code inside xen that runs if the PoD cache is exhausted. It refills the cache with pages that are zeroed out (replacing them with pod entries in the p2m). Your initial balloon down may well be slow if you (or the windows kernel) touches the page before you hand it back to xen, since the pod code will have to populate each page in the p2m as it is touched.
>
>  Paul
>
>> -----Original Message-----
>> From: James Harper [mailto:james.harper@bendigoit.com.au]
>> Sent: 28 February 2011 12:29
>> To: Paul Durrant; George Dunlap
>> Cc: xen devel
>> Subject: RE: [Xen-devel] PoD in other (not GPLPV) drivers
>>
>> >
>> > I actually have plans to push it earlier because we balloon down
>> quite
>> late at
>> > the moment (off the back of the START IRP in the top level bus
>> driver). We are
>> > reliant upon zero-page sweeping code in Xen to save us from guest
>> crashing up
>> > to that point.
>> >
>>
>> I've modified GPLPV to balloon down at DriverEntry time, which seems
>> to be early enough. Prior to that, memory=128 and maxmem=1024 was
>> enough to cause a crash under 2008, basically as soon as I tried to
>> access the registry in DriverEntry. My drivers are using WDF and are
>> therefore loading after the KMDF framework which is going to use
>> additional resources. My backup plan is to write a WDM driver that
>> loads even earlier than that and does the allocation, passing it to
>> the real PV drivers later on, although my concern there is that
>> Windows may not like memory allocated by one driver being freed by
>> another...
>>
>> I've never heard of 'zero-page sweeping code' before... is that a
>> way of xen reallocating a previously touched page if it contains all
>> 0's if we want a page beyond our allocation limit? That might
>> explain why my initial balloon down is so slow! I can tell windows
>> to not zero pages before it gives them to me when I do the initial
>> balloon down... what are your thoughts on that? Although it's
>> unlikely at boot time, in theory they could contain sensitive
>> information and I'm supposed to zero them before handing them back
>> to xen according to the docs.
>>
>> James
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: PoD in other (not GPLPV) drivers

[James Harper]

> 
> Just for the public record: I believe that I have upstreamed all of
> the PoD fixes and improvements that we had in the XenServer tree with
> one exception: There are a series of patches to improve the "emergency
> zero-page sweep" behavior in Xen, particularly for pretty large
> (>24GiB) guests.  This is because:
> * They were really ugly; hacks upon hacks, not suitable for
upstreaming
> * As far as I know, they're performance enhancements only, not
correctness
> ones.
> 
> It doesn't sound to me like the zero-page sweeps should be involved in
> your issue, James.  But if you (or anyone) would like the patches, I'd
> be happy to send them along for you to try.  (And for the shy, or
> archaeologists looking at this long after I'm gone from Citrix, they
> can be found on the XenServer source ISOs.)  I'm not sure they'll
> apply cleanly to 4.1, but that code hasn't had a lot of changes, so
> you should be able to work out how they apply.
> 
> As part of my work to port XenServer to 4.1, I'm going to rewrite the
> patches, and I'll upstream them at that time.
> 

Even without pre-zeroing the memory under Windows, it is still pretty
slow doing the initial balloon down. Even though I ask windows not to
zero fill the page (therefore populating it), windows may still be
touching it though. But even then I give it straight back to xen so in
hindsight I think the zero page sweep shouldn't get invoked.

For testing I'm trying some pretty extreme numbers though, eg
maxmem=16384, memory=512, so ballooning down 15.5GB of memory is never
going to be that fast. I now suspect that the delay is in the Windows
API. When I balloon down during normal runtime it is about 10x slower
than ballooning up.

James

RE: PoD in other (not GPLPV) drivers

[James Harper]

> 
> We tested a wide range of memory parameters for the Citrix PV drivers
> on XenServer.  I just asked the guy in charge of that, and he said
> that they were able to reliably boot w2k8x32 with memory=128MiB and
> maxmem=1024MiB.
> 
> I know we had to do make some changes to move the balloon driver
> allocation further back in the boot process to make that happen; but
> I'm not familiar with the details.  Paul would be the person to ask
> about that.
> 

Is there a way for DomU to get an interrupt when an unpopulated page is
hit when the DomU has gone past its initial limit? That would allow me
to at least write a message to the log and crash usefully if I can't do
anything else about it.

James

RE: PoD in other (not GPLPV) drivers

[George Dunlap]

On Mon, 2011-02-28 at 22:20 +0000, James Harper wrote:
> Is there a way for DomU to get an interrupt when an unpopulated page is
> hit when the DomU has gone past its initial limit? That would allow me
> to at least write a message to the log and crash usefully if I can't do
> anything else about it.

At the moment there's no mechanism for this.  Doing so would be a bit
tricky, as most of the places which do gfn_to_mfn() translations aren't
designed to handle failures gracefully (although that may have changed
since some of the guest paging stuff was introduced).

If we were to do this at some point in the future, what do you think
you'd like the interface to look like?  (i.e., deliver NMI? MCE? a
different but well-known interrupt?  &c)

 -George

RE: PoD in other (not GPLPV) drivers

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

> 
> On Mon, 2011-02-28 at 22:20 +0000, James Harper wrote:
> > Is there a way for DomU to get an interrupt when an unpopulated page is
> > hit when the DomU has gone past its initial limit? That would allow me
> > to at least write a message to the log and crash usefully if I can't do
> > anything else about it.
> 
> At the moment there's no mechanism for this.  Doing so would be a bit
> tricky, as most of the places which do gfn_to_mfn() translations aren't
> designed to handle failures gracefully (although that may have changed
> since some of the guest paging stuff was introduced).
> 
> If we were to do this at some point in the future, what do you think
> you'd like the interface to look like?  (i.e., deliver NMI? MCE? a
> different but well-known interrupt?  &c)
> 

I don't know that there is a sactioned way to intercept an NMI under windows. An MCE would do the trick probably. Basically I want the user to know that their VM crashed because it hit the PoD limit, rather than have no idea why it crashed. Most of the time if a user gets a crash I ask for a copy of /var/log/xen/qemu-dm-<domu>.log, so if the message appeared there then that would be fine with me too.

I was under the impression that xen would blow the DomU up if it overstepped its PoD limit, but now I'm not so sure as I've seen windows BSoD with lots of different bug check codes when it happens. What should happen? I think that instant death is the only sensible outcome.

James



[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: PoD in other (not GPLPV) drivers

[Paul Durrant]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]



> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of James Harper
> Sent: 03 March 2011 10:22
> To: George Dunlap
> Cc: George Dunlap; xen devel
> Subject: RE: [Xen-devel] PoD in other (not GPLPV) drivers
> 
> >
> > On Mon, 2011-02-28 at 22:20 +0000, James Harper wrote:
> > > Is there a way for DomU to get an interrupt when an unpopulated
> page
> > > is hit when the DomU has gone past its initial limit? That would
> > > allow me to at least write a message to the log and crash
> usefully
> > > if I can't do anything else about it.
> >
> > At the moment there's no mechanism for this.  Doing so would be a
> bit
> > tricky, as most of the places which do gfn_to_mfn() translations
> > aren't designed to handle failures gracefully (although that may
> have
> > changed since some of the guest paging stuff was introduced).
> >
> > If we were to do this at some point in the future, what do you
> think
> > you'd like the interface to look like?  (i.e., deliver NMI? MCE? a
> > different but well-known interrupt?  &c)
> >
> 
> I don't know that there is a sactioned way to intercept an NMI under
> windows.

KeRegisterNmiCallback()

See http://msdn.microsoft.com/en-us/library/ff553116%28v=vs.85%29.aspx

> An MCE would do the trick probably. Basically I want the
> user to know that their VM crashed because it hit the PoD limit,
> rather than have no idea why it crashed. Most of the time if a user
> gets a crash I ask for a copy of /var/log/xen/qemu-dm-<domu>.log, so
> if the message appeared there then that would be fine with me too.
> 
> I was under the impression that xen would blow the DomU up if it
> overstepped its PoD limit, but now I'm not so sure as I've seen
> windows BSoD with lots of different bug check codes when it happens.
> What should happen? I think that instant death is the only sensible
> outcome.
> 
> James
> 


[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel