Seamlessly sharing identical memory pages among domains

Seamlessly sharing identical memory pages among domains

[Scheer, Roque]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="us-ascii", Size: 651 bytes --]

I'd like to know if XEN is or will be implementing any mechanism for
transparently sharing pages with identical content among domains, to
minimize memory consumption, similar to what is implemented in VMWare's
ESX server (chapter 4 of the paper whose link is below).

http://www.waldspurger.org/carl/papers/esx-mem-osdi02.pdf

Thanks,
- Roque


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_id66&op=click

Re: Seamlessly sharing identical memory pages among domains

[Ian Pratt]

> I'd like to know if XEN is or will be implementing any mechanism for
> transparently sharing pages with identical content among domains, to
> minimize memory consumption, similar to what is implemented in VMWare's
> ESX server (chapter 4 of the paper whose link is below).

I'm not convinced there's that much sharing to be had between
VMs: One side effect of implementing some test code for live
migration was that I collected some traces containing crc
fingerprints of the memory pages of several domains. There
weren't many hashes in common.  Perhaps this shouldn't be
surprising -- application text is typically small compared to the
heap data they operate on. The actual datasets operated on by
domains are typically distinct.

However, there might be some value in having a shared read-only
buffer cache, if only for the saved IO ops.  Implementing a cache
that replicates pages in memory (rather than sharing) is trivial.

Doing a proper shared cache is slightly trickier given the
paravirtualised memory interface -- we'd have to introduce guests
to a new kind of write fault: "A write fault has occurred, and
you'll have to copy the page because this machine page is
immutable as it is already shared with other domains". Modifying
Linux to handle this wouldn't be hard.

Of course, if you're running a domain on Xen's shadow page tables
then all of this manipulation is easy and can be done
transparently to the domain. Of course, you pay a performance
penalty...

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click

Re: Seamlessly sharing identical memory pages among domains

[Kip Macy]

>
> Doing a proper shared cache is slightly trickier given the
> paravirtualised memory interface -- we'd have to introduce guests
> to a new kind of write fault: "A write fault has occurred, and
> you'll have to copy the page because this machine page is
> immutable as it is already shared with other domains". Modifying
> Linux to handle this wouldn't be hard.

Would this just involve adding an extra error code to what x86 already
uses and then modifying the guest to understand that that error code
means that the page has to be COWed?


				-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click

Re: Seamlessly sharing identical memory pages among domains

[Ian Pratt]

> >
> > Doing a proper shared cache is slightly trickier given the
> > paravirtualised memory interface -- we'd have to introduce guests
> > to a new kind of write fault: "A write fault has occurred, and
> > you'll have to copy the page because this machine page is
> > immutable as it is already shared with other domains". Modifying
> > Linux to handle this wouldn't be hard.
> 
> Would this just involve adding an extra error code to what x86 already
> uses and then modifying the guest to understand that that error code
> means that the page has to be COWed?

Yep. The only slight subtlety is that that the page in question
may appear in multiple page tables and at multiple VAs: if the
page is truly read-write shared in the guest (e.g. sysv shared
memory [*]) then it it will be necessary to find all the ptes and
patch them to the newly allocated machine frame. I think such
pages are rare, and tracking them down by scanning the list of
vma's that are mapping the object (e.g. a file) in question is
pretty easy and quick. However, this would most sensibly be done
by making minor modifications to architecture independent code,
which is something we try to avoid. I guess we could just have
arch xen grub its way through the higher-level data structures...

Ian

[*] Hmm, do memory mapped files behave like this too? I can't
remember. 


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click

Re: Seamlessly sharing identical memory pages among domains

[Kip Macy]

>
> Yep. The only slight subtlety is that that the page in question
> may appear in multiple page tables and at multiple VAs: if the
> page is truly read-write shared in the guest (e.g. sysv shared
> memory [*]) then it it will be necessary to find all the ptes and
> patch them to the newly allocated machine frame. I think such
> pages are rare, and tracking them down by scanning the list of
> vma's that are mapping the object (e.g. a file) in question is
> pretty easy and quick. However, this would most sensibly be done
> by making minor modifications to architecture independent code,

Just adding an upcall wouldn't be too terrible. Iterating through the
list of vtop mappings for a page should be fairly light weight.
Politically, the most realistic thing to do is put the arch-independent
code changes in the xen arch-dependent tree wherever possible. I don't
think anyone wants to see xen specific changes going on outside of the
arch subtree.

> [*] Hmm, do memory mapped files behave like this too? I can't
> remember.

If multiple processes share the page read-write, the OS has to have some
way to know about it.



				-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click

Re: Seamlessly sharing identical memory pages among domains

[Niraj Tolia]

While not exactly related, this paper 
<http://www.cs.rice.edu/~ssiyer/r/superpages/osdi02superpages/> (Section 
4.7) presented at OSDI'02 showed that there was significant overhead 
when trying to detect dirty pages using SHA-1 hash digests. It would be 
interesting to see if this applied to Xen.

Niraj
-- 
http://www.andrew.cmu.edu/~ntolia

Ian Pratt wrote:
>>I'd like to know if XEN is or will be implementing any mechanism for
>>transparently sharing pages with identical content among domains, to
>>minimize memory consumption, similar to what is implemented in VMWare's
>>ESX server (chapter 4 of the paper whose link is below).
> 
> 
> I'm not convinced there's that much sharing to be had between
> VMs: One side effect of implementing some test code for live
> migration was that I collected some traces containing crc
> fingerprints of the memory pages of several domains. There
> weren't many hashes in common.  Perhaps this shouldn't be
> surprising -- application text is typically small compared to the
> heap data they operate on. The actual datasets operated on by
> domains are typically distinct.
> 
> However, there might be some value in having a shared read-only
> buffer cache, if only for the saved IO ops.  Implementing a cache
> that replicates pages in memory (rather than sharing) is trivial.
> 
> Doing a proper shared cache is slightly trickier given the
> paravirtualised memory interface -- we'd have to introduce guests
> to a new kind of write fault: "A write fault has occurred, and
> you'll have to copy the page because this machine page is
> immutable as it is already shared with other domains". Modifying
> Linux to handle this wouldn't be hard.
> 
> Of course, if you're running a domain on Xen's shadow page tables
> then all of this manipulation is easy and can be done
> transparently to the domain. Of course, you pay a performance
> penalty...
> 
> Ian
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle 10g. 
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click

RE: Seamlessly sharing identical memory pages among domains

[Scheer, Roque]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="us-ascii", Size: 1042 bytes --]

>> I'm not convinced there's that much sharing to be had between
>> VMs

In the paper Carl Waldspurger reports some worthwhile sharing under
certain real world workloads:

           | Total| Shared   |Reclaimed   |
Guest Types|  MB  | MB |  %  | MB  |   %  |
-----------+------+----+-----+-----+------+
A)10 WinNT | 2048 | 880| 42.9| 673 | 32.9 |
B) 9 Linux | 1846 | 539| 29.2| 345 | 18.7 |
C) 5 Linux | 1658 | 165| 10.0| 120 | 7.2  |

>> Doing a proper shared cache is slightly trickier given the
paravirtualised 
>> memory interface -- we'd have to introduce guests to a new kind of
write 
>> fault

Wouldn't it be possible for XEN to share the pages tranparently, without
the guest OSes needing to know it?

- Roque


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g.
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id149&alloc_id66&op=click

Re: Seamlessly sharing identical memory pages among domains

[Ian Pratt]

> In the paper Carl Waldspurger reports some worthwhile sharing under
> certain real world workloads:
> 
>            | Total| Shared   |Reclaimed   |
> Guest Types|  MB  | MB |  %  | MB  |   %  |
> -----------+------+----+-----+-----+------+
> A)10 WinNT | 2048 | 880| 42.9| 673 | 32.9 |
> B) 9 Linux | 1846 | 539| 29.2| 345 | 18.7 |
> C) 5 Linux | 1658 | 165| 10.0| 120 | 7.2  |

I haven't got any figures for WinXP. The amount of sharing for
Linux instances will be very work load dependent. I'd bet that
almost all of it could be had through a shared buffer cache (from
a CoW file system).

> Wouldn't it be possible for XEN to share the pages tranparently, without
> the guest OSes needing to know it?

Only for guests running in "shadow page table" mode, otherwise
the guest knows about real machine frame numbers.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click

Re: Seamlessly sharing identical memory pages among domains

[Steven Hand]

> >> I'm not convinced there's that much sharing to be had between
> >> VMs
> 
> In the paper Carl Waldspurger reports some worthwhile sharing under
> certain real world workloads:
> 
>            | Total| Shared   |Reclaimed   |
> Guest Types|  MB  | MB |  %  | MB  |   %  |
> -----------+------+----+-----+-----+------+
> A)10 WinNT | 2048 | 880| 42.9| 673 | 32.9 |
> B) 9 Linux | 1846 | 539| 29.2| 345 | 18.7 |
> C) 5 Linux | 1658 | 165| 10.0| 120 | 7.2  |

Yes; and we may do some shared text support at some point. Not clear
how much additional benefit comes from the "scanning" they did; a 
simpler approach to detect read-only sharing is to explicitly check 
for resident versions of a certain text segment across VMs. Having
some "look-aside" in the block device interface (partic if also 
doing sharing at that level as per our current CoW implementation) 
might get considerable benefit for little cost. 


> >> Doing a proper shared cache is slightly trickier given the
> >> paravirtualised memory interface -- we'd have to introduce guests 
> >>to a new kind of write fault
> 
> Wouldn't it be possible for XEN to share the pages tranparently, without
> the guest OSes needing to know it?

Possibly. One challenge here is to maintain partitioning (hard guarantees) 
while providing sharing. Depending on how important isolation is to you, 
this is relatively less or more difficult. 

Something we'll be considering post OSDI anyway..


cheers,

S.

 


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click