Shadow page tables?

Shadow page tables?

[Chengyuan Li]

Hi,

I'm using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn't
enable the shadow page tables in Xen though there is code there.

If the shadow page tables are enabled, does it mean that the guest
Linux' memory management module could be fully virtualized?

Thanks,
Chengyuan


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Shadow page tables?

[Keir Fraser]

Currently, shadow p.t.'s are only enabled when migrating OSes between
machines. This allows us to copy the memory image while the OS is
still running, and track which pages are updated after being copied
(and so need to be recopied).

We intend to flesh out the shadow p.t. code a little mroe to support
full memory virtualisation. It's not there yet, but it won't require
an enormous amount of code to get it going.

 -- Keir

> Hi,
> 
> I'm using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn't
> enable the shadow page tables in Xen though there is code there.
> 
> If the shadow page tables are enabled, does it mean that the guest
> Linux' memory management module could be fully virtualized?
> 
> Thanks,
> Chengyuan
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Shadow page tables?

[Chengyuan Li]

If the shadow pagetable is enabled, Xen will maintain pagetables for
every guest process. If 1000 processes are running in the guest, xen
will at least allocate 1000 pages as shadow pagetables. Will it
consume too many memory frames?


On Fri, 08 Oct 2004 10:11:18 +0100, Keir Fraser
<keir.fraser@cl.cam.ac.uk> wrote:
> 
> Currently, shadow p.t.'s are only enabled when migrating OSes between
> machines. This allows us to copy the memory image while the OS is
> still running, and track which pages are updated after being copied
> (and so need to be recopied).
> 
> We intend to flesh out the shadow p.t. code a little mroe to support
> full memory virtualisation. It's not there yet, but it won't require
> an enormous amount of code to get it going.
> 
> -- Keir
> 
> 
> 
> > Hi,
> >
> > I'm using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn't
> > enable the shadow page tables in Xen though there is code there.
> >
> > If the shadow page tables are enabled, does it mean that the guest
> > Linux' memory management module could be fully virtualized?
> >
> > Thanks,
> > Chengyuan
> >
> > 
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > Use IT products in your business? Tell us what you think of them. Give us
> > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
>


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

RE: Shadow page tables?

[Ian Pratt]

> If the shadow pagetable is enabled, Xen will maintain pagetables for
every guest 
> process. If 1000 processes are running in the guest, xen will at least
allocate 1000 
> pages as shadow pagetables. Will it consume too many memory frames?

That's the downside of shadow pagetables, and hence why we prefer to run
guests in 'direct mode' unless we're doing something 'weird' to them -
e.g. when we migrate a domain we switch to shadow page tables for the
duration of the migration.

Apache 1.3 is a good example of an application that requires large
amounts of memory for pagetables (many thousands of pages), and with
shadow pagetables this requiment effectively doubles. The shadow
pagetables are just a cache, so its simple to evict pages if you run low
on memory, but then performance will suck big time.

Ian

On Fri, 08 Oct 2004 10:11:18 +0100, Keir Fraser
<keir.fraser@cl.cam.ac.uk> wrote:
> 
> Currently, shadow p.t.'s are only enabled when migrating OSes between 
> machines. This allows us to copy the memory image while the OS is 
> still running, and track which pages are updated after being copied 
> (and so need to be recopied).
> 
> We intend to flesh out the shadow p.t. code a little mroe to support 
> full memory virtualisation. It's not there yet, but it won't require 
> an enormous amount of code to get it going.
> 
> -- Keir
> 
> 
> 
> > Hi,
> >
> > I'm using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn't

> > enable the shadow page tables in Xen though there is code there.
> >
> > If the shadow page tables are enabled, does it mean that the guest 
> > Linux' memory management module could be fully virtualized?
> >
> > Thanks,
> > Chengyuan
> >
> > 
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on 
> > ITManagersJournal Use IT products in your business? Tell us what you

> > think of them. Give us Your Opinions, Get Free ThinkGeek Gift 
> > Certificates! Click to find out more 
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
>


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give
us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find
out more http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Shadow page tables?

[Michael Vrable]

On Fri, Oct 08, 2004 at 10:11:18AM +0100, Keir Fraser wrote:
> We intend to flesh out the shadow p.t. code a little mroe to support
> full memory virtualisation. It's not there yet, but it won't require
> an enormous amount of code to get it going.

Any idea what the time frame for this is?  I was considering taking a
stab at it myself, but don't want to duplicate work.

My longer term goal is to try to get copy-on-write sharing of memory
pages between domains and to see how far Xen can scale in running many
nearly-identical virtual machines.  To implement this, however, will
require (at least as I've thought it through) memory virtualization.  So
that was the first thing I was planning to work on.

If no one else was actively working on this, I'm happy to discuss the
design and contribute what I end up with.  (Both for the memory
virtualization and the copy-on-write work.)

(This is also the reason for my interest in copy-on-write disks, though
it's looking like LVM may be too heavyweight for supporting large
numbers of VMs.)

--Michael Vrable


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Shadow page tables?

[Andrew Warfield]

> (This is also the reason for my interest in copy-on-write disks, though
> it's looking like LVM may be too heavyweight for supporting large
> numbers of VMs.)

Hi Michael,

  I'd be interested to know what sort of overhead results you've seen
on lvm -- I've been looking at block devices myself a bit lately and
if lvm doesn't look like it can be made to scale as a solution for
cow, I might be able to arrange some sort of alternative.  Any sort of
summary of your lvm thoughts/experiences would be great.

best,
andy.


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Shadow page tables?

[Michael Vrable]

On Tue, Oct 12, 2004 at 09:09:24PM +0100, Andrew Warfield wrote:
>   I'd be interested to know what sort of overhead results you've seen
> on lvm -- I've been looking at block devices myself a bit lately and
> if lvm doesn't look like it can be made to scale as a solution for
> cow, I might be able to arrange some sort of alternative.  Any sort of
> summary of your lvm thoughts/experiences would be great.

My primary concern with LVM snapshot scalability is kernel memory
consumption.  (I haven't done any performance benchmarking.)  For each
LVM snapshot that is created, the kernel allocates:
  - 1 MB of pages for a kcopyd_client.
  - Up to a 2 MB hash table for tracking exceptions (areas of the disk
    that have been changed).
  - The exceptions themselves.  (Not very large individually, but it all
    has to be in kernel memory--on the order of 16 bytes per area of
    disk [say ~16 K] that has been remapped, so that could grow to 1 MB
    of memory per 1 GB COW disk.)
(These numbers are gathered from looking through the snapshot code.)

The kcopyd_client can (I think) be safely shared between snapshots--I
patched the kernel to do this, and it worked fine under the limited
testing I gave it.  So this issue could be worked around.  But the other
two are more troublesome, since the assumption that all the COW mapping
tables available in kernel memory can't be changed without a lot of
work.

The problem is currently worse than it has to be since the LVM code
won't sleep if the kernel memory isn't immediately available; it errors
out.  Even with that fixed (which may happen eventually:
http://www.redhat.com/archives/dm-devel/2004-January/msg00079.html), LVM
is still pinning on the order of megabytes of kernel memory per
snapshot.  That is fine for some workloads, but I'd like to see if I can
get to hundreds or thousands of COW disks--in which case this is too
much memory per disk.

I'm currently wondering if a solution implemented partially in userspace
is viable--put the COW driver in userspace, let the on-disk data
structures get cached in the page cache, but any COW disks not being
actively utilized would have data swapped out of memory.  Perhaps
communicate to the kernel via the nbd interface or something similar?  I
haven't put much thought into this yet.  I expect performance would be
worse than LVM, but that's a tradeoff I'm willing to make.

--Michael Vrable


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Shadow page tables?

[Peri Hankey]

Hello

Some time ago you talked about copy-on-write memory to enable large 
numbers of nearly identical machines to run on the same physical 
hardware. I have also thought about that - after all equivalent  
mechansisms already exist to provide copy-on-write semantics for process 
forking (and also I think in the /proc/mm mechanisms used in some 
versions of UML separate kernel address space (SKAS) handling).

Have you advanced along this path, or is anyone actively doing that? The 
problem you found the other day suggests that you were looking pretty 
closely at code that relates to this question. My very crude 
understanding is that one would want to share the physical memory 
allocation that is currently given to a single xenU domain between a 
group of xenU domains each of which has its own  copy-on-write mapping 
that starts from a read-only image that is shared between all of them. 
The table grant mehanism would need to understand the relationship 
between members of such a group of domains. But as I say, this is still 
very hazy to me.

On the other point you raised, about copy-on-write filesystem images, I 
looked at evms2 (http://evms.sourceforge.net/), which sits above lvm2 
and other systems and offers its own snapshot feature. It is only usable 
on disks that it controls, so you either have to use a separate disk 
from the one you boot from, or you have to use an initrd to be able to 
boot from an evms volume. The snapshot feature seemed 'snappier' (I was 
able to create a number of snapshot objects and activate them without 
memory problems of the kind that I encountered with lvm2 snapshots).

However it didn't seem possible to make it provide multiple persistent 
writable snapshots of the kind that would support multiple xenU domains 
where each uses its own copy-on-write image of an underlying readonly 
root filesystem. Did I miss something? Has anyone else tried using evms 
for these purposes?

I saw Ian's comments about gnbd and csnap 
(http://sources.redhat.com/cluster/), but it seems that the csnap 
mechanism is in a pretty early state of development.

Regards
Peri

Michael Vrable wrote:


Michael Vrable wrote:

>On Fri, Oct 08, 2004 at 10:11:18AM +0100, Keir Fraser wrote:
>  
>
>>We intend to flesh out the shadow p.t. code a little mroe to support
>>full memory virtualisation. It's not there yet, but it won't require
>>an enormous amount of code to get it going.
>>    
>>
>
>Any idea what the time frame for this is?  I was considering taking a
>stab at it myself, but don't want to duplicate work.
>
>My longer term goal is to try to get copy-on-write sharing of memory
>pages between domains and to see how far Xen can scale in running many
>nearly-identical virtual machines.  To implement this, however, will
>require (at least as I've thought it through) memory virtualization.  So
>that was the first thing I was planning to work on.
>
>If no one else was actively working on this, I'm happy to discuss the
>design and contribute what I end up with.  (Both for the memory
>virtualization and the copy-on-write work.)
>
>(This is also the reason for my interest in copy-on-write disks, though
>it's looking like LVM may be too heavyweight for supporting large
>numbers of VMs.)
>
>--Michael Vrable
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>Use IT products in your business? Tell us what you think of them. Give us
>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
>  
>



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Copy-on-write memory to allow many more xenU domains per machine

[Peri Hankey]

Hello

Some time ago Michael Vrable talked about copy-on-write memory to enable 
large numbers of nearly identical machines to run on the same physical
hardware. Is this a feasible proposition? I asked a few days ago in the 
original thread, but no-one seems to have noticed. It seems to me that 
it would be a very significant feature to offer.

Regards
Peri



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Copy-on-write memory to allow many more xenU domains per machine

[Ian Pratt]

> Hello
> 
> Some time ago Michael Vrable talked about copy-on-write memory to enable 
> large numbers of nearly identical machines to run on the same physical
> hardware. Is this a feasible proposition? I asked a few days ago in the 
> original thread, but no-one seems to have noticed. It seems to me that 
> it would be a very significant feature to offer.

It's useful for honeypots and other situations where you want
very large numbers of VMs, but isn't generally a huge win.

When I was doing the live migration work I recorded fingerprints
of all VM pages that were on several systems, and didn't find a
whole lot of commonality between VMs. CoW sharing of memory is on
the list of features we'd like to add to Xen, but implementing
the shared buffer cache is rather higher on the todo list.

Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Copy-on-write memory to allow many more xenU domains per machine

[David Hopwood]

Ian Pratt wrote:
>>Hello
>>
>>Some time ago Michael Vrable talked about copy-on-write memory to enable 
>>large numbers of nearly identical machines to run on the same physical
>>hardware. Is this a feasible proposition? I asked a few days ago in the 
>>original thread, but no-one seems to have noticed. It seems to me that 
>>it would be a very significant feature to offer.
> 
> It's useful for honeypots and other situations where you want
> very large numbers of VMs, but isn't generally a huge win.
> 
> When I was doing the live migration work I recorded fingerprints
> of all VM pages that were on several systems, and didn't find a
> whole lot of commonality between VMs.

I thought the idea would be to have a 'clone' primitive that would work
in a similar way to Unix fork? The clone would start off sharing all
pages, although a lot of them would get copied fairly quickly. Does
anyone have an idea of what proportion or number of pages in a Linux
system stay constant (and are not paged out) after boot?

-- 
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Copy-on-write memory to allow many more xenU domains per machine

[Ronald G. Minnich]

On Mon, 25 Oct 2004, David Hopwood wrote:

> I thought the idea would be to have a 'clone' primitive that would work
> in a similar way to Unix fork? The clone would start off sharing all
> pages, although a lot of them would get copied fairly quickly. Does


yes but ... if most or almost most pages get copied, COW is a loss. Try to 
find that old paper somebody wrote about "bovaphobic" behaviour -- 
interesting numbers in there. 

COW is not always a good thing.

ron


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Copy-on-write memory to allow many more xenU domains per machine

[Ian Pratt]

> I thought the idea would be to have a 'clone' primitive that would work
> in a similar way to Unix fork? The clone would start off sharing all
> pages, although a lot of them would get copied fairly quickly. Does
> anyone have an idea of what proportion or number of pages in a Linux
> system stay constant (and are not paged out) after boot?

We've already done some work on implementing very lightweight VM
checkpoints, for purposes of debugging and s/w fault
tolerance. 'VM clone' is on the todo list, but I think it'll be a
while before this makes it into a stable release.

Ian



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Copy-on-write memory to allow many more xenU domains per machine

[Michael Vrable]

On Mon, Oct 25, 2004 at 01:12:22PM +0100, Peri Hankey wrote:
> Some time ago Michael Vrable talked about copy-on-write memory to enable 
> large numbers of nearly identical machines to run on the same physical
> hardware. Is this a feasible proposition? I asked a few days ago in the 
> original thread, but no-one seems to have noticed. It seems to me that 
> it would be a very significant feature to offer.

I'm still around, though a bit slow to respond.

Ian mentioned that honeypots are one the scenarios where copy-on-write
sharing of memory makes a lot of sense; that's the reason that I'm
trying implement it.  I hope, of course, that if I do manage to get
something working it would be useful in other scenarios as well.

My current status: I've been working first on using shadow page tables
to hide the actual machine addresses of pages from domains.  I'm only
partway through doing that, and may yet go back and change how I'm
approaching that (I'm still learning bits about how Xen works as I'm
going through to modify the code; I'm also thinking now of changes that
would be good to make copy-on-write easier later).  I'm trying to write
up some of my design thoughts; if anyone here is interested or would
like to offer comments, I can post that here.

David mentioned implementing a clone() for VMs; that is something that
I'd also like to try out.  (It should be possible to test that after
only the first step of hiding machine addresses, before copy-on-write is
implemented, though copy-on-write will make it efficient.)

If someone else beats me to getting this implemented, I suppose I can't
complain.  But if this is an area that isn't getting much current work,
and I'm able to contribute (a hopefully clean implementation that others
wouldn't mind using), that would be great.

--Michael Vrable


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Copy-on-write memory to allow many more xenU domains per machine

[Peri Hankey]

I hadn't been thinking of the honeypot scenario myself, but of the plain 
old 'more virtual computers than you would have thought possible' scene. 
>From Ian's account of it there's less scope for that than I had imagined.

I can be heavy slow myself ('heavy' is apparently Glaswgian slang for 
'very' - think 'heavy light') - especially when immersed in the 
technical undergrowth.

Regards
Peri




Michael Vrable wrote:

>On Mon, Oct 25, 2004 at 01:12:22PM +0100, Peri Hankey wrote:
>  
>
>>Some time ago Michael Vrable talked about copy-on-write memory to enable 
>>large numbers of nearly identical machines to run on the same physical
>>hardware. Is this a feasible proposition? I asked a few days ago in the 
>>original thread, but no-one seems to have noticed. It seems to me that 
>>it would be a very significant feature to offer.
>>    
>>
>
>I'm still around, though a bit slow to respond.
>
>Ian mentioned that honeypots are one the scenarios where copy-on-write
>sharing of memory makes a lot of sense; that's the reason that I'm
>trying implement it.  I hope, of course, that if I do manage to get
>something working it would be useful in other scenarios as well.
>
>My current status: I've been working first on using shadow page tables
>to hide the actual machine addresses of pages from domains.  I'm only
>partway through doing that, and may yet go back and change how I'm
>approaching that (I'm still learning bits about how Xen works as I'm
>going through to modify the code; I'm also thinking now of changes that
>would be good to make copy-on-write easier later).  I'm trying to write
>up some of my design thoughts; if anyone here is interested or would
>like to offer comments, I can post that here.
>
>David mentioned implementing a clone() for VMs; that is something that
>I'd also like to try out.  (It should be possible to test that after
>only the first step of hiding machine addresses, before copy-on-write is
>implemented, though copy-on-write will make it efficient.)
>
>If someone else beats me to getting this implemented, I suppose I can't
>complain.  But if this is an area that isn't getting much current work,
>and I'm able to contribute (a hopefully clean implementation that others
>wouldn't mind using), that would be great.
>
>--Michael Vrable
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
>Use IT products in your business? Tell us what you think of them. Give us
>Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
>http://productguide.itmanagersjournal.com/guidepromo.tmpl
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
>  
>



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Copy-on-write memory to allow many more xenU domains per machine

[Michael Vrable]

On Tue, Oct 26, 2004 at 09:15:05AM +0100, Peri Hankey wrote:
> I hadn't been thinking of the honeypot scenario myself, but of the plain 
> old 'more virtual computers than you would have thought possible' scene. 
> From Ian's account of it there's less scope for that than I had imagined.
> 
> I can be heavy slow myself ('heavy' is apparently Glaswgian slang for 
> 'very' - think 'heavy light') - especially when immersed in the 
> technical undergrowth.

"Memory Resource Management in VMware ESX Server" by Carl Waldspurger
has some numbers on memory sharing between VMs:
    http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf
(This paper is also cited in the Xen SOSP 2003 paper.)

Sharing between identical machines is good, though sharing between
dissimilar machines is not as high as I remembered (copy-on-write
sharing is able to save around 7-30% of memory in real-world testing).

--Michael Vrable


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Copy-on-write memory to allow many more xenU domains per machine

[Ian Pratt]

> "Memory Resource Management in VMware ESX Server" by Carl Waldspurger
> has some numbers on memory sharing between VMs:
>     http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf
> (This paper is also cited in the Xen SOSP 2003 paper.)
> 
> Sharing between identical machines is good, though sharing between
> dissimilar machines is not as high as I remembered (copy-on-write
> sharing is able to save around 7-30% of memory in real-world testing).

As I recall, Carl's results were on Windows, which might explain
why I was seeing rather less. Linux resident set sizes tend to be
smaller, and there's generally rather more diversity in the
install base (at least around here).

Ian


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Copy-on-write memory to allow many more xenU domains per machine

[Michael Vrable]

On Tue, Oct 26, 2004 at 08:43:30PM +0100, Ian Pratt wrote:
> > "Memory Resource Management in VMware ESX Server" by Carl Waldspurger
> > has some numbers on memory sharing between VMs:
> >     http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf
> > (This paper is also cited in the Xen SOSP 2003 paper.)
> > 
> > Sharing between identical machines is good, though sharing between
> > dissimilar machines is not as high as I remembered (copy-on-write
> > sharing is able to save around 7-30% of memory in real-world testing).
> 
> As I recall, Carl's results were on Windows, which might explain
> why I was seeing rather less. Linux resident set sizes tend to be
> smaller, and there's generally rather more diversity in the
> install base (at least around here).

Both were tested in that paper; Windows was at the high end of the scale
and Linux at the low end:

Reproducing Figure 5 from the paper in slightly abbreviated form:

          Guests        % Shared        % Reclaimed
        10 x WinNT        42.9              32.9
         9 x Linux        29.2              18.7
         5 x Linux        10.0               7.2

"% Shared" is the fraction of pages which are shared; "% Reclaimed"
measures how much memory sharing the pages is able to save.

Three systems is still a limited test set.

--Michael Vrable


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Copy-on-write memory to allow many more xenU domains per machine

[Jacob Gorm Hansen]

  Michael Vrable wrote:

 > My current status: I've been working first on using shadow page tables
 > to hide the actual machine addresses of pages from domains.  I'm only
 > partway through doing that, and may yet go back and change how I'm
 > approaching that (I'm still learning bits about how Xen works as I'm
 > going through to modify the code; I'm also thinking now of changes that
 > would be good to make copy-on-write easier later).  I'm trying to write
 > up some of my design thoughts; if anyone here is interested or would
 > like to offer comments, I can post that here.
 >
 > David mentioned implementing a clone() for VMs; that is something that
 > I'd also like to try out.  (It should be possible to test that after
 > only the first step of hiding machine addresses, before copy-on-write is
 > implemented, though copy-on-write will make it efficient.

What about having a special variant of the pte-update operation, which 
takes as input a pointer to a pte-entry, and will change that entry into 
a read-only mapping of the same page contents, though perhaps with a 
different frame number in the pte.

Then Xen can lookup the page contents in a global hashtable or binary 
tree (which might be faster, because you don't need to scan the entire 
page contents unless you have a match), and see if there is a matching 
copy somewhere. If there is, it can decrease a refcount for the page 
provided by the user, change the mapping to point to the matching page, 
and free the user's page if the refcount has reached zero.

In this way, domains become responsible for sharing memory across the 
system, but only pages that are likely candidates for sharing (kernel 
text, user process text) are actually scanned, and you don't get the 
overhead of shadow page tables.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Copy-on-write memory to allow many more xenU domains per machine

[Ian Pratt]

 
> What about having a special variant of the pte-update operation, which 
> takes as input a pointer to a pte-entry, and will change that entry into 
> a read-only mapping of the same page contents, though perhaps with a 
> different frame number in the pte.

I'm still not convinced we wouldn't get most of the benefit by
just sharing pages read from the block/file system, and hence
avoiding all the hashing. 

Collecting this data is actually very easy -- its just a case of
coming up with a few "realistic usage scenarios" in which to
collect the data. We'd be happy to work with someone to do
this...

Ian


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Copy-on-write memory to allow many more xenU domains per machine

[urmk]

Check the s/390 port of the kernel for the XIP2 filesystem.  It exploits
the z/VM shared memory system to make a read-only ext2 like filesystem
that is shared amonst all the guests - and uses execute-in-place.  Any
binaries on the xip2 filesystem aren't cached into local shared memory,
they're just mapped across.

This lets an admin build explicit shared systems, does shared ro-memory
nicely, etc.  Might be worth looking into implementing similar in Xen.

Apologies if this was already discussed/noticed,

-m

On Tue, Nov 02, 2004 at 09:33:00PM +0000, Ian Pratt wrote:
>  
> > What about having a special variant of the pte-update operation, which 
> > takes as input a pointer to a pte-entry, and will change that entry into 
> > a read-only mapping of the same page contents, though perhaps with a 
> > different frame number in the pte.
> 
> I'm still not convinced we wouldn't get most of the benefit by
> just sharing pages read from the block/file system, and hence
> avoiding all the hashing. 
> 
> Collecting this data is actually very easy -- its just a case of
> coming up with a few "realistic usage scenarios" in which to
> collect the data. We'd be happy to work with someone to do
> this...
> 
> Ian
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click