yanked share, round 2

yanked share, round 2

[King, Steven R]

[-- Attachment #1.1: Type: text/plain, Size: 1368 bytes --]

Hi folks,
A previous thread discussed complications around DomU's sharing memory
pages with each other:
http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.html
 
To summarize, DomU's get into trouble, e.g. unable to shutdown, unless
the remote DomU's play nice.  Since DomU's do not trust each other, that
is problematic.  I'd like to discuss how to clean away this dependency.
 
Here's one idea.  The goal is to robustly decouple the sharing and
remote domains.
 
Grant tables add a new GTF_safe flag, settable by the sharing DomU.
In order to map a GTF_safe page, a remote domain must provide a page of
its own, which I'll call an "under page".
Xen holds the under-page on behalf of the remote DomU and maps the
shared page into the remote DomU's machine.
At any time, the sharing DomU can unshare the page, crash, etc, which
ends ALL foreign access to that page, not just new mappings.
For each remote domain that still maps the unshared page, Xen maps the
remote's under-page in place of the unshared page.
The remote domain can unmap at any time and recover its under-page.
 
The purpose of the under-page is to plug the memory hole in the remote
DomU created by a surprise unsharing.  A nervous remote DomU could check
that a share is GTF_safe before proceeding to map the page.
 
Good, bad or ugly?
-steve
 
 
 

[-- Attachment #1.2: Type: text/html, Size: 4139 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: yanked share, round 2

[Keir Fraser]

On 13 Jan 2006, at 19:07, King, Steven R wrote:
 
> The purpose of the under-page is to plug the memory hole in the remote 
> DomU created by a surprise unsharing.  A nervous remote DomU could 
> check that a share is GTF_safe before proceeding to map the page.
>  
> Good, bad or ugly?

What's wrong with the current scheme (sharing domU sticks around as a 
zombie until foreign mappings disappear)? This needn't stop control 
tools from restarting the domain (they can see that it has shut down, 
for example, and is simply awaiting reaping when its refcnt falls to 
zero).

It's arguable the zombies needn't even be kept on the domain list, so 
they become invisible to the control tools (since they're really just a 
resource container for foreign page mappings). OTOH hiding things from 
control tools is probably a bad idea. :-)

  -- Keir

Re: yanked share, round 2

[Anthony Liguori]

An ideal solution to this problem would be to keep a separate pool of 
shared memory that neither domain owned.  That removes any concerns 
about ownership.

Regards,

Anthony Liguori

King, Steven R wrote:

> Hi folks,
> A previous thread discussed complications around DomU's sharing memory 
> pages with each other:
> http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.html
>  
> To summarize, DomU's get into trouble, e.g. unable to shutdown, 
> unless the remote DomU's play nice.  Since DomU's do not trust each 
> other, that is problematic.  I'd like to discuss how to clean away 
> this dependency.
>  
> Here's one idea.  The goal is to robustly decouple the sharing and 
> remote domains.
>  
> Grant tables add a new GTF_safe flag, settable by the sharing DomU.
> In order to map a GTF_safe page, a remote domain must provide a page 
> of its own, which I'll call an "under page".
> Xen holds the under-page on behalf of the remote DomU and maps 
> the shared page into the remote DomU's machine.
> At any time, the sharing DomU can unshare the page, crash, etc, 
> which ends ALL foreign access to that page, not just new mappings.
> For each remote domain that still maps the unshared page, Xen maps the 
> remote's under-page in place of the unshared page.
> The remote domain can unmap at any time and recover its under-page.
>  
> The purpose of the under-page is to plug the memory hole in the remote 
> DomU created by a surprise unsharing.  A nervous remote DomU could 
> check that a share is GTF_safe before proceeding to map the page.
>  
> Good, bad or ugly?
> -steve
>  
>  
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>

RE: yanked share, round 2

[King, Steven R]

Hi Anthony -- Can you explain why this is ideal?  I prefer that sharers
and mappers have their own skin the game--that way, Xen doesn't have to
manage a pool and nobody has to worry about the pool being depleted.

-----Original Message-----
From: Anthony Liguori [mailto:aliguori@us.ibm.com] 
Sent: Friday, January 13, 2006 11:23 AM
To: King, Steven R
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] yanked share, round 2

An ideal solution to this problem would be to keep a separate pool of
shared memory that neither domain owned.  That removes any concerns
about ownership.

Regards,

Anthony Liguori

King, Steven R wrote:

> Hi folks,
> A previous thread discussed complications around DomU's sharing memory

> pages with each other:
> http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht
> ml
>  
> To summarize, DomU's get into trouble, e.g. unable to shutdown, unless

> the remote DomU's play nice.  Since DomU's do not trust each other, 
> that is problematic.  I'd like to discuss how to clean away this 
> dependency.
>  
> Here's one idea.  The goal is to robustly decouple the sharing and 
> remote domains.
>  
> Grant tables add a new GTF_safe flag, settable by the sharing DomU.
> In order to map a GTF_safe page, a remote domain must provide a page 
> of its own, which I'll call an "under page".
> Xen holds the under-page on behalf of the remote DomU and maps the 
> shared page into the remote DomU's machine.
> At any time, the sharing DomU can unshare the page, crash, etc, which 
> ends ALL foreign access to that page, not just new mappings.
> For each remote domain that still maps the unshared page, Xen maps the

> remote's under-page in place of the unshared page.
> The remote domain can unmap at any time and recover its under-page.
>  
> The purpose of the under-page is to plug the memory hole in the remote

> DomU created by a surprise unsharing.  A nervous remote DomU could 
> check that a share is GTF_safe before proceeding to map the page.
>  
> Good, bad or ugly?
> -steve
>  
>  
>  
>
>-----------------------------------------------------------------------
>-
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>

Re: yanked share, round 2

[Anthony Liguori]

You avoid zombies and either side can break the sharing without causing 
harm to the other side.  Domains restarting are transparent to either 
end (the restarting domain just reattachs and keeps going).  It avoids 
the general ownership problem altogether.

Regards,

Anthony Liguori

King, Steven R wrote:

>Hi Anthony -- Can you explain why this is ideal?  I prefer that sharers
>and mappers have their own skin the game--that way, Xen doesn't have to
>manage a pool and nobody has to worry about the pool being depleted.
>
>-----Original Message-----
>From: Anthony Liguori [mailto:aliguori@us.ibm.com] 
>Sent: Friday, January 13, 2006 11:23 AM
>To: King, Steven R
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] yanked share, round 2
>
>An ideal solution to this problem would be to keep a separate pool of
>shared memory that neither domain owned.  That removes any concerns
>about ownership.
>
>Regards,
>
>Anthony Liguori
>
>King, Steven R wrote:
>
>  
>
>>Hi folks,
>>A previous thread discussed complications around DomU's sharing memory
>>    
>>
>
>  
>
>>pages with each other:
>>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht
>>ml
>> 
>>To summarize, DomU's get into trouble, e.g. unable to shutdown, unless
>>    
>>
>
>  
>
>>the remote DomU's play nice.  Since DomU's do not trust each other, 
>>that is problematic.  I'd like to discuss how to clean away this 
>>dependency.
>> 
>>Here's one idea.  The goal is to robustly decouple the sharing and 
>>remote domains.
>> 
>>Grant tables add a new GTF_safe flag, settable by the sharing DomU.
>>In order to map a GTF_safe page, a remote domain must provide a page 
>>of its own, which I'll call an "under page".
>>Xen holds the under-page on behalf of the remote DomU and maps the 
>>shared page into the remote DomU's machine.
>>At any time, the sharing DomU can unshare the page, crash, etc, which 
>>ends ALL foreign access to that page, not just new mappings.
>>For each remote domain that still maps the unshared page, Xen maps the
>>    
>>
>
>  
>
>>remote's under-page in place of the unshared page.
>>The remote domain can unmap at any time and recover its under-page.
>> 
>>The purpose of the under-page is to plug the memory hole in the remote
>>    
>>
>
>  
>
>>DomU created by a surprise unsharing.  A nervous remote DomU could 
>>check that a share is GTF_safe before proceeding to map the page.
>> 
>>Good, bad or ugly?
>>-steve
>> 
>> 
>> 
>>
>>-----------------------------------------------------------------------
>>-
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>> 
>>
>>    
>>
>
>  
>

RE: yanked share, round 2

[King, Steven R]

Hi Keir, I'm not familiar enough with zombie anatomy to help there, so let me try this reasoning: today's Xen architecture cannot promise that a shared page can ever be returned to normal, non-shared service.  Thus, any DomU that routinely creates shared pages *must* eventually run out of memory.  Of course if all DomU's try to play nice, it would take a long while.  We still face a snag in which Xen allows DomU bugs, DomU crashes and DomU evil to accumulate over time.  By analogy to the hardware world, a PCI device that could not promise to let go of pages would be unacceptable.

-steve



-----Original Message-----
From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
Sent: Friday, January 13, 2006 11:21 AM
To: King, Steven R
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] yanked share, round 2


On 13 Jan 2006, at 19:07, King, Steven R wrote:
 
> The purpose of the under-page is to plug the memory hole in the remote 
> DomU created by a surprise unsharing.  A nervous remote DomU could 
> check that a share is GTF_safe before proceeding to map the page.
>  
> Good, bad or ugly?

What's wrong with the current scheme (sharing domU sticks around as a zombie until foreign mappings disappear)? This needn't stop control tools from restarting the domain (they can see that it has shut down, for example, and is simply awaiting reaping when its refcnt falls to zero).

It's arguable the zombies needn't even be kept on the domain list, so they become invisible to the control tools (since they're really just a resource container for foreign page mappings). OTOH hiding things from control tools is probably a bad idea. :-)

  -- Keir

Re: yanked share, round 2

[Keir Fraser]

On 13 Jan 2006, at 19:55, King, Steven R wrote:

> Hi Keir, I'm not familiar enough with zombie anatomy to help there, so 
> let me try this reasoning: today's Xen architecture cannot promise 
> that a shared page can ever be returned to normal, non-shared service. 
>  Thus, any DomU that routinely creates shared pages *must* eventually 
> run out of memory.  Of course if all DomU's try to play nice, it would 
> take a long while.  We still face a snag in which Xen allows DomU 
> bugs, DomU crashes and DomU evil to accumulate over time.  By analogy 
> to the hardware world, a PCI device that could not promise to let go 
> of pages would be unacceptable.

Just as a PCI device can be reset, we can kill an offensive device 
driver or other service domain and restart it. I think these problems 
need addressing in the tools / control plane, not with extra mechanism 
in Xen. At the end of the day, the memory backing these bogus/buggy 
mappings has to come from somewhere. Even if you add mechanism such 
that the mapping domain is made accountable, what should its clients do 
when it runs out of memory and finally hits the brick wall?

  -- Keir

RE: yanked share, round 2

[King, Steven R]

On 13 Jan 2006, at 20:09, Keir wrote:

>Just as a PCI device can be reset, we can kill an offensive device
>driver or other service domain and restart it. I think these problems
>need addressing in the tools / control plane, not with extra mechanism
in Xen.
>[snip]

Are you suggesting that operator intervention is the durable solution?
Surely, that doesn't wash. ;^)

>Even if you add mechanism such that the mapping domain is
>made accountable, what should its clients do when it runs
>out of memory and finally hits the brick wall?

Why should the mapping domain be in special danger of running out of
memory?  It is 'accountable' 1 for 1 with the number of pages it maps.
If a surprise unshare occurs, the mapping domain can unmap and robustly
recover the associated under-page.

Without advocating any particular approach, the architectural goal is
that either DomU can independently decouple itself from the other,
something not possible now.

-steve


-----Original Message-----
From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
Sent: Friday, January 13, 2006 12:09 PM
To: King, Steven R
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] yanked share, round 2


On 13 Jan 2006, at 19:55, King, Steven R wrote:

> Hi Keir, I'm not familiar enough with zombie anatomy to help there, so

> let me try this reasoning: today's Xen architecture cannot promise 
> that a shared page can ever be returned to normal, non-shared service.
>  Thus, any DomU that routinely creates shared pages *must* eventually 
> run out of memory.  Of course if all DomU's try to play nice, it would

> take a long while.  We still face a snag in which Xen allows DomU 
> bugs, DomU crashes and DomU evil to accumulate over time.  By analogy 
> to the hardware world, a PCI device that could not promise to let go 
> of pages would be unacceptable.

Just as a PCI device can be reset, we can kill an offensive device
driver or other service domain and restart it. I think these problems
need addressing in the tools / control plane, not with extra mechanism
in Xen. At the end of the day, the memory backing these bogus/buggy
mappings has to come from somewhere. Even if you add mechanism such that
the mapping domain is made accountable, what should its clients do when
it runs out of memory and finally hits the brick wall?

  -- Keir

RE: yanked share, round 2

[King, Steven R]

I would like hear your ideas for who manages the pool and how the pool
avoids becoming depleted.

In addition to avoiding the ownership problem, I see another nice
advantage:
The third party (Xen? a DomP?) can hand up to the DomU's a nice tidy key
value representing the shared pages, which is very similar to the way
SysV IPC memory sharing works.

-steve

-----Original Message-----
From: Anthony Liguori [mailto:aliguori@us.ibm.com] 
Sent: Friday, January 13, 2006 11:35 AM
To: King, Steven R
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] yanked share, round 2

You avoid zombies and either side can break the sharing without causing
harm to the other side.  Domains restarting are transparent to either
end (the restarting domain just reattachs and keeps going).  It avoids
the general ownership problem altogether.

Regards,

Anthony Liguori

King, Steven R wrote:

>Hi Anthony -- Can you explain why this is ideal?  I prefer that sharers

>and mappers have their own skin the game--that way, Xen doesn't have to

>manage a pool and nobody has to worry about the pool being depleted.
>
>-----Original Message-----
>From: Anthony Liguori [mailto:aliguori@us.ibm.com]
>Sent: Friday, January 13, 2006 11:23 AM
>To: King, Steven R
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] yanked share, round 2
>
>An ideal solution to this problem would be to keep a separate pool of 
>shared memory that neither domain owned.  That removes any concerns 
>about ownership.
>
>Regards,
>
>Anthony Liguori
>
>King, Steven R wrote:
>
>  
>
>>Hi folks,
>>A previous thread discussed complications around DomU's sharing memory
>>    
>>
>
>  
>
>>pages with each other:
>>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht
>>ml
>> 
>>To summarize, DomU's get into trouble, e.g. unable to shutdown, unless
>>    
>>
>
>  
>
>>the remote DomU's play nice.  Since DomU's do not trust each other, 
>>that is problematic.  I'd like to discuss how to clean away this 
>>dependency.
>> 
>>Here's one idea.  The goal is to robustly decouple the sharing and 
>>remote domains.
>> 
>>Grant tables add a new GTF_safe flag, settable by the sharing DomU.
>>In order to map a GTF_safe page, a remote domain must provide a page 
>>of its own, which I'll call an "under page".
>>Xen holds the under-page on behalf of the remote DomU and maps the 
>>shared page into the remote DomU's machine.
>>At any time, the sharing DomU can unshare the page, crash, etc, which 
>>ends ALL foreign access to that page, not just new mappings.
>>For each remote domain that still maps the unshared page, Xen maps the
>>    
>>
>
>  
>
>>remote's under-page in place of the unshared page.
>>The remote domain can unmap at any time and recover its under-page.
>> 
>>The purpose of the under-page is to plug the memory hole in the remote
>>    
>>
>
>  
>
>>DomU created by a surprise unsharing.  A nervous remote DomU could 
>>check that a share is GTF_safe before proceeding to map the page.
>> 
>>Good, bad or ugly?
>>-steve
>> 
>> 
>> 
>>
>>----------------------------------------------------------------------
>>-
>>-
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>> 
>>
>>    
>>
>
>  
>

Re: yanked share, round 2

[Anthony Liguori]

King, Steven R wrote:

>I would like hear your ideas for who manages the pool and how the pool
>avoids becoming depleted.
>  
>
I won't take credit for this as it is Rusty's idea actually :-)

Depletion can be handled by setting a maximum amount of shared memory 
per-domain (2MB for instance).  Then as long as there is enough free 
memory to satisfy the per-domain sharing requirement, you're fine.

The memory can be allocated straight from the xen heap and referenced 
counted such that it is returned back to the heap when the last user 
stops sharing it.  The 2MB limit would be somewhat virtual since the 
same page would be factored into every domain's actual limit who had a 
reference to the page.

This is my understand at least, perhaps Rusty can clarify with what he 
was thinking.

>In addition to avoiding the ownership problem, I see another nice
>advantage:
>The third party (Xen? a DomP?) can hand up to the DomU's a nice tidy key
>value representing the shared pages, which is very similar to the way
>SysV IPC memory sharing works.
>  
>
Precisely :-)

Regards,

Anthony Liguori

>-steve
>
>-----Original Message-----
>From: Anthony Liguori [mailto:aliguori@us.ibm.com] 
>Sent: Friday, January 13, 2006 11:35 AM
>To: King, Steven R
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] yanked share, round 2
>
>You avoid zombies and either side can break the sharing without causing
>harm to the other side.  Domains restarting are transparent to either
>end (the restarting domain just reattachs and keeps going).  It avoids
>the general ownership problem altogether.
>
>Regards,
>
>Anthony Liguori
>
>King, Steven R wrote:
>
>  
>
>>Hi Anthony -- Can you explain why this is ideal?  I prefer that sharers
>>    
>>
>
>  
>
>>and mappers have their own skin the game--that way, Xen doesn't have to
>>    
>>
>
>  
>
>>manage a pool and nobody has to worry about the pool being depleted.
>>
>>-----Original Message-----
>>From: Anthony Liguori [mailto:aliguori@us.ibm.com]
>>Sent: Friday, January 13, 2006 11:23 AM
>>To: King, Steven R
>>Cc: xen-devel@lists.xensource.com
>>Subject: Re: [Xen-devel] yanked share, round 2
>>
>>An ideal solution to this problem would be to keep a separate pool of 
>>shared memory that neither domain owned.  That removes any concerns 
>>about ownership.
>>
>>Regards,
>>
>>Anthony Liguori
>>
>>King, Steven R wrote:
>>
>> 
>>
>>    
>>
>>>Hi folks,
>>>A previous thread discussed complications around DomU's sharing memory
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>pages with each other:
>>>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht
>>>ml
>>>
>>>To summarize, DomU's get into trouble, e.g. unable to shutdown, unless
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>the remote DomU's play nice.  Since DomU's do not trust each other, 
>>>that is problematic.  I'd like to discuss how to clean away this 
>>>dependency.
>>>
>>>Here's one idea.  The goal is to robustly decouple the sharing and 
>>>remote domains.
>>>
>>>Grant tables add a new GTF_safe flag, settable by the sharing DomU.
>>>In order to map a GTF_safe page, a remote domain must provide a page 
>>>of its own, which I'll call an "under page".
>>>Xen holds the under-page on behalf of the remote DomU and maps the 
>>>shared page into the remote DomU's machine.
>>>At any time, the sharing DomU can unshare the page, crash, etc, which 
>>>ends ALL foreign access to that page, not just new mappings.
>>>For each remote domain that still maps the unshared page, Xen maps the
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>remote's under-page in place of the unshared page.
>>>The remote domain can unmap at any time and recover its under-page.
>>>
>>>The purpose of the under-page is to plug the memory hole in the remote
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>DomU created by a surprise unsharing.  A nervous remote DomU could 
>>>check that a share is GTF_safe before proceeding to map the page.
>>>
>>>Good, bad or ugly?
>>>-steve
>>>
>>>
>>>
>>>
>>>----------------------------------------------------------------------
>>>-
>>>-
>>>
>>>_______________________________________________
>>>Xen-devel mailing list
>>>Xen-devel@lists.xensource.com
>>>http://lists.xensource.com/xen-devel
>>>
>>>
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>
>  
>

RE: yanked share, round 2

[King, Steven R]

How about combining both ideas:  In order to map N pages from Xen's
share pool, a DomU must provide N pages back to the pool.  When a DomU
unmaps (or crashes), it gets N pages back.

We get these features:
1) A domU can never cause the Xen share pool to shrink.
2) The number of pages mappable by a DomU is bounded only by the DomU
itself.
3) No page ownership problems.
4) We can have a nice share key ala SysV IPC semantics.
5) When no shares exist, the memory pool consumes no pages.
6) We leave Xen's heap alone.

The first benefit is critical since DomU's are untrusted.  A downside is
that a platform with many DomU mappings will have many idle pages
sitting in the pool.  Given all the benefits above, a very reasonable
price to pay.

-steve


-----Original Message-----
From: Anthony Liguori [mailto:aliguori@us.ibm.com] 
Sent: Friday, January 13, 2006 1:32 PM
To: King, Steven R
Cc: Rusty Russell; xen-devel
Subject: Re: [Xen-devel] yanked share, round 2

King, Steven R wrote:

>I would like hear your ideas for who manages the pool and how the pool 
>avoids becoming depleted.
>  
>
I won't take credit for this as it is Rusty's idea actually :-)

Depletion can be handled by setting a maximum amount of shared memory
per-domain (2MB for instance).  Then as long as there is enough free
memory to satisfy the per-domain sharing requirement, you're fine.

The memory can be allocated straight from the xen heap and referenced
counted such that it is returned back to the heap when the last user
stops sharing it.  The 2MB limit would be somewhat virtual since the
same page would be factored into every domain's actual limit who had a
reference to the page.

This is my understand at least, perhaps Rusty can clarify with what he
was thinking.

>In addition to avoiding the ownership problem, I see another nice
>advantage:
>The third party (Xen? a DomP?) can hand up to the DomU's a nice tidy 
>key value representing the shared pages, which is very similar to the 
>way SysV IPC memory sharing works.
>  
>
Precisely :-)

Regards,

Anthony Liguori

>-steve
>
>-----Original Message-----
>From: Anthony Liguori [mailto:aliguori@us.ibm.com]
>Sent: Friday, January 13, 2006 11:35 AM
>To: King, Steven R
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] yanked share, round 2
>
>You avoid zombies and either side can break the sharing without causing

>harm to the other side.  Domains restarting are transparent to either 
>end (the restarting domain just reattachs and keeps going).  It avoids 
>the general ownership problem altogether.
>
>Regards,
>
>Anthony Liguori
>
>King, Steven R wrote:
>
>  
>
>>Hi Anthony -- Can you explain why this is ideal?  I prefer that 
>>sharers
>>    
>>
>
>  
>
>>and mappers have their own skin the game--that way, Xen doesn't have 
>>to
>>    
>>
>
>  
>
>>manage a pool and nobody has to worry about the pool being depleted.
>>
>>-----Original Message-----
>>From: Anthony Liguori [mailto:aliguori@us.ibm.com]
>>Sent: Friday, January 13, 2006 11:23 AM
>>To: King, Steven R
>>Cc: xen-devel@lists.xensource.com
>>Subject: Re: [Xen-devel] yanked share, round 2
>>
>>An ideal solution to this problem would be to keep a separate pool of 
>>shared memory that neither domain owned.  That removes any concerns 
>>about ownership.
>>
>>Regards,
>>
>>Anthony Liguori
>>
>>King, Steven R wrote:
>>
>> 
>>
>>    
>>
>>>Hi folks,
>>>A previous thread discussed complications around DomU's sharing 
>>>memory
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>pages with each other:
>>>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.h
>>>t
>>>ml
>>>
>>>To summarize, DomU's get into trouble, e.g. unable to shutdown, 
>>>unless
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>the remote DomU's play nice.  Since DomU's do not trust each other, 
>>>that is problematic.  I'd like to discuss how to clean away this 
>>>dependency.
>>>
>>>Here's one idea.  The goal is to robustly decouple the sharing and 
>>>remote domains.
>>>
>>>Grant tables add a new GTF_safe flag, settable by the sharing DomU.
>>>In order to map a GTF_safe page, a remote domain must provide a page 
>>>of its own, which I'll call an "under page".
>>>Xen holds the under-page on behalf of the remote DomU and maps the 
>>>shared page into the remote DomU's machine.
>>>At any time, the sharing DomU can unshare the page, crash, etc, which

>>>ends ALL foreign access to that page, not just new mappings.
>>>For each remote domain that still maps the unshared page, Xen maps 
>>>the
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>remote's under-page in place of the unshared page.
>>>The remote domain can unmap at any time and recover its under-page.
>>>
>>>The purpose of the under-page is to plug the memory hole in the 
>>>remote
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>DomU created by a surprise unsharing.  A nervous remote DomU could 
>>>check that a share is GTF_safe before proceeding to map the page.
>>>
>>>Good, bad or ugly?
>>>-steve
>>>
>>>
>>>
>>>
>>>---------------------------------------------------------------------
>>>-
>>>-
>>>-
>>>
>>>_______________________________________________
>>>Xen-devel mailing list
>>>Xen-devel@lists.xensource.com
>>>http://lists.xensource.com/xen-devel
>>>
>>>
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>
>  
>

Re: yanked share, round 2

[Keir Fraser]

On 13 Jan 2006, at 23:02, King, Steven R wrote:

> We get these features:
> 1) A domU can never cause the Xen share pool to shrink.
> 2) The number of pages mappable by a DomU is bounded only by the DomU
> itself.
> 3) No page ownership problems.
> 4) We can have a nice share key ala SysV IPC semantics.
> 5) When no shares exist, the memory pool consumes no pages.
> 6) We leave Xen's heap alone.
>
> The first benefit is critical since DomU's are untrusted.  A downside 
> is
> that a platform with many DomU mappings will have many idle pages
> sitting in the pool.  Given all the benefits above, a very reasonable
> price to pay.

This is similar to ideas people have had for shared buffer cache 
without memory overcommitment. Seems to me that the Xen-level mechanism 
for doing the sharing could easily be identical, and the differences in 
interface, access control and other policy could be hidden in a share 
manager in a suitably-privileged domain.

  -- Keir

RE: yanked share, round 2

[Rusty Russell]

On Fri, 2006-01-13 at 15:02 -0800, King, Steven R wrote:
> How about combining both ideas:  In order to map N pages from Xen's
> share pool, a DomU must provide N pages back to the pool.  When a DomU
> unmaps (or crashes), it gets N pages back.

OK, early last year I hacked up a prototype implementation, which I'm
now bringing up-to-date with the modern Xen code, and some new
simplifications I'm toying with.

It's certainly a simple model to never have domains share their own
pages, and the hypervisor provide a DMA-like mechanism between domains
to allow bulk transfer.  This would then page-swap if appropriate.

It makes drivers nice and simple, and avoids problems with starvation
et. al.  Earlier results suggested it is competitive on benchmarks, too,
but it needs real testing under various loads.

Noone here should be surprised though, that I regard the simplicity of
the scheme as the most important factor.

Cheers!
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: yanked share, round 2

[Mark Williamson]

> The third party (Xen? a DomP?) can hand up to the DomU's a nice tidy key
> value representing the shared pages, which is very similar to the way
> SysV IPC memory sharing works.

This is starting to sound rather like Saved Segments on IBM zSeries to me...

Also sounds like named shared regions could be managed by dom0, with control 
plane communication (naming, grant refs, etc) occurring through XenStore - 
i.e. using a share back/front driver.

Cheers,
Mark

> -steve
>
> -----Original Message-----
> From: Anthony Liguori [mailto:aliguori@us.ibm.com]
> Sent: Friday, January 13, 2006 11:35 AM
> To: King, Steven R
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] yanked share, round 2
>
> You avoid zombies and either side can break the sharing without causing
> harm to the other side.  Domains restarting are transparent to either
> end (the restarting domain just reattachs and keeps going).  It avoids
> the general ownership problem altogether.
>
> Regards,
>
> Anthony Liguori
>
> King, Steven R wrote:
> >Hi Anthony -- Can you explain why this is ideal?  I prefer that sharers
> >
> >and mappers have their own skin the game--that way, Xen doesn't have to
> >
> >manage a pool and nobody has to worry about the pool being depleted.
> >
> >-----Original Message-----
> >From: Anthony Liguori [mailto:aliguori@us.ibm.com]
> >Sent: Friday, January 13, 2006 11:23 AM
> >To: King, Steven R
> >Cc: xen-devel@lists.xensource.com
> >Subject: Re: [Xen-devel] yanked share, round 2
> >
> >An ideal solution to this problem would be to keep a separate pool of
> >shared memory that neither domain owned.  That removes any concerns
> >about ownership.
> >
> >Regards,
> >
> >Anthony Liguori
> >
> >King, Steven R wrote:
> >>Hi folks,
> >>A previous thread discussed complications around DomU's sharing memory
> >>
> >>
> >>
> >>
> >>
> >>pages with each other:
> >>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht
> >>ml
> >>
> >>To summarize, DomU's get into trouble, e.g. unable to shutdown, unless
> >>
> >>
> >>
> >>
> >>
> >>the remote DomU's play nice.  Since DomU's do not trust each other,
> >>that is problematic.  I'd like to discuss how to clean away this
> >>dependency.
> >>
> >>Here's one idea.  The goal is to robustly decouple the sharing and
> >>remote domains.
> >>
> >>Grant tables add a new GTF_safe flag, settable by the sharing DomU.
> >>In order to map a GTF_safe page, a remote domain must provide a page
> >>of its own, which I'll call an "under page".
> >>Xen holds the under-page on behalf of the remote DomU and maps the
> >>shared page into the remote DomU's machine.
> >>At any time, the sharing DomU can unshare the page, crash, etc, which
> >>ends ALL foreign access to that page, not just new mappings.
> >>For each remote domain that still maps the unshared page, Xen maps the
> >>
> >>
> >>
> >>
> >>
> >>remote's under-page in place of the unshared page.
> >>The remote domain can unmap at any time and recover its under-page.
> >>
> >>The purpose of the under-page is to plug the memory hole in the remote
> >>
> >>
> >>
> >>
> >>
> >>DomU created by a surprise unsharing.  A nervous remote DomU could
> >>check that a share is GTF_safe before proceeding to map the page.
> >>
> >>Good, bad or ugly?
> >>-steve
> >>
> >>
> >>
> >>
> >>----------------------------------------------------------------------
> >>-
> >>-
> >>
> >>_______________________________________________
> >>Xen-devel mailing list
> >>Xen-devel@lists.xensource.com
> >>http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: yanked share, round 2

[Mark Williamson]

> This is similar to ideas people have had for shared buffer cache
> without memory overcommitment.

Quite similar to the mechanisms I'm incorporating into XenFS, for one thing.

> Seems to me that the Xen-level mechanism 
> for doing the sharing could easily be identical, and the differences in
> interface, access control and other policy could be hidden in a share
> manager in a suitably-privileged domain.

Agreed, doesn't seem like it needs to be an in-Xen service.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: yanked share, round 2

[Mark Williamson]

> Hi Keir, I'm not familiar enough with zombie anatomy to help there, so let
> me try this reasoning: today's Xen architecture cannot promise that a
> shared page can ever be returned to normal, non-shared service.  Thus, any
> DomU that routinely creates shared pages *must* eventually run out of
> memory.

Eveni if it could reboot, causing the control tools to allocate it new 
memory... the more worrying problem is that a domain that does *not* return 
shared pages could eventually end up holding all the memory in the machine, 
with it all being credited to other (zombie) domains.

> Of course if all DomU's try to play nice, it would take a long 
> while.  We still face a snag in which Xen allows DomU bugs, DomU crashes
> and DomU evil to accumulate over time.  By analogy to the hardware world, a
> PCI device that could not promise to let go of pages would be unacceptable.

A centralised store (managed by a trusted entity) would be rather good for 
domUs that really don't trust each other.  It'd be worth having a driver that 
could set these up - presumably with shares advertised using XenStore.

Cheers,
Mark

> -steve
>
>
>
> -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> Sent: Friday, January 13, 2006 11:21 AM
> To: King, Steven R
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] yanked share, round 2
>
>
> On 13 Jan 2006, at 19:07, King, Steven R wrote:
>  
>
> > The purpose of the under-page is to plug the memory hole in the remote
> > DomU created by a surprise unsharing.  A nervous remote DomU could
> > check that a share is GTF_safe before proceeding to map the page.
> >  
> > Good, bad or ugly?
>
> What's wrong with the current scheme (sharing domU sticks around as a
> zombie until foreign mappings disappear)? This needn't stop control tools
> from restarting the domain (they can see that it has shut down, for
> example, and is simply awaiting reaping when its refcnt falls to zero).
>
> It's arguable the zombies needn't even be kept on the domain list, so they
> become invisible to the control tools (since they're really just a resource
> container for foreign page mappings). OTOH hiding things from control tools
> is probably a bad idea. :-)
>
>   -- Keir
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!