Re: [Fwd: Re: [RFC] XI Shadow Page Table Mechanism]

Re: [Fwd: Re: [RFC] XI Shadow Page Table Mechanism]

[Robert Phillips]

[-- Attachment #1.1: Type: text/plain, Size: 4405 bytes --]

I am replying for Ben.
-- rsp

On 6/22/06, Ben Thomas <bthomas@virtualiron.com> wrote:
>
>
>
> -------- Original Message --------
> Subject: Re: [Xen-devel] [RFC] XI Shadow Page Table Mechanism
> Date: Wed, 21 Jun 2006 15:31:59 -0500
> From: Anthony Liguori <aliguori@us.ibm.com >
> To: Ben Thomas <bthomas@virtualiron.com>
> CC: xen-devel@lists.xensource.com
> References: < 44998B2C.2000109@virtualiron.com>
>
> Ben Thomas wrote:
> > This post contains the design document for what is currently known
> > as the "XI Shadow Mechanism".  This is a design for shadow
> > page table code for fully virtualized HVM domains running on a 64-bit
> > Xen hypervisor.
> >
> > This work was undertaken to address a number of goals.  These are
> > enumerated in the document and include:
> >
> > - ability to run fully virtualized 32, 32PAE and 64 bit guest
> >   domains concurrently on a 64-bit hypervisor
>
> This isn't supported currently?  Since an HVM must go through 16 bit, 32
> bit, and 64 bit mode to boot up, how can we start more than one guest at
> a time currently if this doesn't already work?


Ed Smith's daily test results show there have been problems with SMP.  We
haven't diagnosed these problems but, from reading the pre-XI shadow code,
it's not clear how it copes with multiple VCPUs in the same domain running
in different modes.

> - support live migration of fully virtualized domains
>
> Sweet.  What about the current shadow page code limited this?


Nothing intrinsic.  Just a SMOP.

> - provide good performance and robustness
> >
> > This design has been implemented and is currently being tested.
> > It has been supporting the variety of memory models as noted above,
> > and using widely used Windows and Linux distributions (SuSe,
> > RedHat and others). At a point in the near future, a patch
> > will be available.
> >
> > This design center is the x86-64 architecture. It is not our
> > intent to completely replace all shadow page management, and
> > we've attempted to limit the scope of change.
> >
> > A preliminary version of this design concept has undergone
> > brief review with some members of the Xen community. We hope
> > that this is of value to the Xen community and welcome your
> > feedback and comments.
>
> Can't really comment as it's not quotable.  Some questions that
> immediately come to mind are:
>
> - how do you deal with large pages within the hypervisor?  do you
> coalesce or just hope there is contiguous pages available?


We just hope that contiguous pages are available.  However we allocate pages
for the guest in large extents to maximize this likelihood.  In practice
this is very effective for guests created soon after boot time.  There may
be fragmentation problems later.

- what is the performance benefit in saving the shadow pages for each
> domain?  there's clearly a memory trade-off here so understanding the
> performance gain seems important.


The performance benefit appears to be substantial but we have not done a
thorough study yet.

- OOM can be dealt with in the existing code by just invalidating
> existing mappings to free up pages.  what advantages do your approach
> have to this?  (i realize we don't do this today but in theory, we could).


Ultimately XI deals with OOM  by tearing down cached shadow pages, just as
you say.  But it uses LRU to pick the victims.

Interesting stuff.  I'm eager to see the code.
>
> Regards,
>
> Anthony Liguori
>
> > Thanks,
> > -b
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
>
> --
> ------------------------------------------------------------------------
> Ben Thomas                                         Virtual Iron Software
> bthomas@virtualiron.com                             Tower 1, Floor 2
> 978-849-1214                                       900 Chelmsford Street
>                                                     Lowell, MA 01851
>
------------------------------------------------------------------------
Robert S. Phillips                          Virtual Iron Software
rsp.vi.xen@virtualiron.com             Tower 1, Floor 2
978-849-1220                                 900 Chelmsford Street
                                                    Lowell, MA 01851

[-- Attachment #1.2: Type: text/html, Size: 8245 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Fwd: Re: [RFC] XI Shadow Page Table Mechanism]

[Anthony Liguori]

Robert Phillips wrote:
>
>     This isn't supported currently?  Since an HVM must go through 16
>     bit, 32
>     bit, and 64 bit mode to boot up, how can we start more than one
>     guest at
>     a time currently if this doesn't already work?
>
>
> Ed Smith's daily test results show there have been problems with SMP.  
> We haven't diagnosed these problems but, from reading the pre-XI 
> shadow code, it's not clear how it copes with multiple VCPUs in the 
> same domain running in different modes.

There's only a short period of time when this would be happening right?  
During boot up?

>     - how do you deal with large pages within the hypervisor?  do you
>     coalesce or just hope there is contiguous pages available? 
>
>
> We just hope that contiguous pages are available.  However we allocate 
> pages for the guest in large extents to maximize this likelihood.  In 
> practice this is very effective for guests created soon after boot 
> time.  There may be fragmentation problems later.

Any ideas about how to deal with this long term?  Large page support 
would also be useful for PV domains.  There was a lot of people at the 
last summit that were interested in this...

Thanks for the responses,

Anthony Liguori

>     - what is the performance benefit in saving the shadow pages for each
>     domain?  there's clearly a memory trade-off here so understanding the
>     performance gain seems important.
>
>
> The performance benefit appears to be substantial but we have not done 
> a thorough study yet.
>
>     - OOM can be dealt with in the existing code by just invalidating
>     existing mappings to free up pages.  what advantages do your approach
>     have to this?  (i realize we don't do this today but in theory, we
>     could).
>
>
> Ultimately XI deals with OOM  by tearing down cached shadow pages, 
> just as you say.  But it uses LRU to pick the victims.
>
>     Interesting stuff.  I'm eager to see the code.
>
>     Regards,
>
>     Anthony Liguori
>
>     > Thanks,
>     > -b
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     > _______________________________________________
>     > Xen-devel mailing list
>     > Xen-devel@lists.xensource.com <mailto:Xen-devel@lists.xensource.com>
>     > http://lists.xensource.com/xen-devel
>
>
>     --
>     ------------------------------------------------------------------------
>     Ben Thomas                                         Virtual Iron
>     Software
>     bthomas@virtualiron.com
>     <mailto:bthomas@virtualiron.com>                            Tower
>     1, Floor 2
>     978-849-1214                                       900 Chelmsford
>     Street
>                                                         Lowell, MA 01851
>
> ------------------------------------------------------------------------
> Robert S. Phillips                          Virtual Iron Software
> rsp.vi.xen@virtualiron.com 
> <mailto:rsp.vi.xen@virtualiron.com>             Tower 1, Floor 2
> 978-849-1220                                 900 Chelmsford Street
>                                                     Lowell, MA 01851