Ownership of machine pages: Was: Essay on an important Xen decision (long)

Ownership of machine pages: Was: Essay on an important Xen decision (long)

[Magenheimer, Dan (HP Labs Fort Collins)]

> Date: Wed, 11 Jan 2006 15:56:43 +0800
> From: "Tian, Kevin" <kevin.tian@intel.com>
> Subject: RE: [Xen-devel] Essay on an important Xen decision (long)
> To: "Magenheimer, Dan \(HP Labs Fort Collins\)"
> 	<dan.magenheimer@hp.com>,	"xen-devel" 
> <xen-devel@lists.xensource.com>,
> 	<xen-ia64-devel@lists.xensource.com>
> 
> Hi, Dan,
> 	Good background for discussion.

For some reason, I only got this through xen-devel (which I
only receive in digest form) so I couldn't respond directly.
But this subtopic -- though highly related -- might deserve
a new thread anyway.

> I don't think it's a good design choice by complete takeover 
> to dom0. Moving ownership to dom0 doesn¡¯t mean a simple 
> move, since memory sub-system is the core/base of Xen

Very true.  But I expect Xen's memory subsystem will need
to evolve considerably to handle things like:

- support of different page sizes
- cacheable vs uncacheable memory
- sparsely populated memory
- NUMA
- hotplug memory
- oversubscription
- memory reserved for firmware

Now might be a very good time to consider alternative
approaches rather than try to hack these one at a time
into the existing memory subsystem of core/base Xen.

> Extra 
> context switches are added for any page related operation.

Hmmm... not really.  Page operations (such as alloc and free)
are relatively rare and can be batched (e.g. at domain startup).
And domU's are not going to be making policy decisions...
If dom0 is making policy decisions (e.g. this stick of RAM is
about to die, who do I steal memory from?), dom0 ownership
of the pages may even reduce context switches.

> Also by P==M model, how do you ensure a scalable allocation 
> environment after a long run? Any activities within dom0 
> which consumes Physical frames, thus actually eats Machine 
> frames.

Just like today's balloon driver, a "machine memory policy
driver" would steal all but some fixed number of dom0's pages
such that the dom0 kernel cannot use them for other purposes.

Dan

Re: Ownership of machine pages: Was: Essay on an important Xen decision (long)

[Keir Fraser]

On 11 Jan 2006, at 22:49, Magenheimer, Dan (HP Labs Fort Collins) wrote:

>> Extra
>> context switches are added for any page related operation.
>
> Hmmm... not really.  Page operations (such as alloc and free)
> are relatively rare and can be batched (e.g. at domain startup).
> And domU's are not going to be making policy decisions...
> If dom0 is making policy decisions (e.g. this stick of RAM is
> about to die, who do I steal memory from?), dom0 ownership
> of the pages may even reduce context switches.

In a literal/extreme form of your model, any address space 
access/update would need to be translated and/or validated by domain0. 
That's certainly going to be a significant overhead. I believe that 
there is a balance to be struck between sufficient mechanism in Xen to 
ensure good performance while allowing flexible policy expression in 
the control plane. I think talking about this at the summit would be 
very interesting -- are your ideas solidified enough perhaps even for a 
short presentation?

  -- Keir

Re: Ownership of machine pages: Was: Essay on an important Xen decision (long)

[Magenheimer, Dan (HP Labs Fort Collins)]

(Sorry again for the late and off-thread reply... even though
your message is addressed to me personally, to xen-devel,
and to xen-ia64-devel, I didn't get a copy and just saw it
in xen-devel which I only receive digested.  Some strangeness
in the mailing list nodupe feature maybe?  James cc'ed....)

Yes, I think I could do a short presentation about what
I perceive are the issues and my (admittedly half-baked)
solution, but this would probably be better as a discussion
or brainstorming session than a "classroom" session
with 50 people listening.

> In a literal/extreme form of your model, any address space 
> access/update would need to be translated and/or validated by 
> domain0.

The literal/extreme form of just about *any* model generally
sucks ;-)  I'm thinking that Xen retains lookup tables so
all accesses are low cost, but (most) updates require domain0.
I think (without enough expertise in Xen/x86 memory management)
that the sticky overhead part will be page-flipping for netif:
Is there any "page affinity" for skbuffs or are they random pages?

Dan

> From: Keir Fraser <Keir.Fraser@cl.cam.ac.uk>
> Subject: Re: Ownership of machine pages: Was: [Xen-devel] Essay on an
> 	important Xen decision (long)
> 
> On 11 Jan 2006, at 22:49, Magenheimer, Dan (HP Labs Fort 
> Collins) wrote:
> 
> >> Extra
> >> context switches are added for any page related operation.
> >
> > Hmmm... not really.  Page operations (such as alloc and free)
> > are relatively rare and can be batched (e.g. at domain startup).
> > And domU's are not going to be making policy decisions...
> > If dom0 is making policy decisions (e.g. this stick of RAM is
> > about to die, who do I steal memory from?), dom0 ownership
> > of the pages may even reduce context switches.
> 
> In a literal/extreme form of your model, any address space 
> access/update would need to be translated and/or validated by 
> domain0. 
> That's certainly going to be a significant overhead. I believe that 
> there is a balance to be struck between sufficient mechanism 
> in Xen to 
> ensure good performance while allowing flexible policy expression in 
> the control plane. I think talking about this at the summit would be 
> very interesting -- are your ideas solidified enough perhaps 
> even for a 
> short presentation?

Re: Ownership of machine pages: Was: Essay on an important Xen decision (long)

[James Bulpin]

Magenheimer, Dan (HP Labs Fort Collins) wrote:
> (Sorry again for the late and off-thread reply... even though
> your message is addressed to me personally, to xen-devel,
> and to xen-ia64-devel, I didn't get a copy and just saw it
> in xen-devel which I only receive digested.  Some strangeness
> in the mailing list nodupe feature maybe?  James cc'ed....)

If you were on the recipient list of the original mail from Keir then 
that should have gone directly to you without passing anywhere near the 
list server, if you didn't get that then it's a problem with mail 
servers at your or Keir's end. If you've got nodupe set for 
xen-ia64-devel then your address being on the message means that you 
won't get the message via the list. You still get the message on the 
xen-devel digest because nodupe doesn't apply (or make sense) to digests.

Cheers,
James

Re: Ownership of machine pages: Was: Essay on an important Xen decision (long)

[Jayesh Salvi]

[-- Attachment #1.1: Type: text/plain, Size: 1740 bytes --]

Sorry, if I missed something. But I did not read any discussion on Harry
Butterworth's idea of generic memory management API (xenidc and USB driver).

My experience is not enough to comment on this topic, but I would like to
understand these design decisions. As far as I know, generic API's that
provide proper abstractions prove useful (e.g. Linux VirtualFS). There are
counter arguments to it (performance degradation mainly / wrong
abstractions). But I didn't find any comments on the xenidc API approach.

Apart from that, thanks for elaborate description of some key concepts.

Jayesh

On 1/12/06, James Bulpin <james@xensource.com> wrote:
>
> Magenheimer, Dan (HP Labs Fort Collins) wrote:
> > (Sorry again for the late and off-thread reply... even though
> > your message is addressed to me personally, to xen-devel,
> > and to xen-ia64-devel, I didn't get a copy and just saw it
> > in xen-devel which I only receive digested.  Some strangeness
> > in the mailing list nodupe feature maybe?  James cc'ed....)
>
> If you were on the recipient list of the original mail from Keir then
> that should have gone directly to you without passing anywhere near the
> list server, if you didn't get that then it's a problem with mail
> servers at your or Keir's end. If you've got nodupe set for
> xen-ia64-devel then your address being on the message means that you
> won't get the message via the list. You still get the message on the
> xen-devel digest because nodupe doesn't apply (or make sense) to digests.
>
> Cheers,
> James
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>



--
Jayesh

[-- Attachment #1.2: Type: text/html, Size: 2206 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Ownership of machine pages: Was: Essay on an important Xen decision (long)

[Magenheimer, Dan (HP Labs Fort Collins)]

[-- Attachment #1.1: Type: text/plain, Size: 2442 bytes --]

If the Xen drivers are rewritten to be more generic (e.g. to handle more
than just p2m), I agree that xenidc should be seriously considered.
I think the most serious impediment to acceptance of xenidc is
fear of breaking drivers (blk and net) that are already stable.
 
My two cents...
Dan


________________________________

	From: Jayesh Salvi [mailto:jayeshsalvi@gmail.com] 
	Sent: Thursday, January 12, 2006 9:04 PM
	To: James Bulpin
	Cc: Magenheimer, Dan (HP Labs Fort Collins); xen-devel;
xen-ia64-devel@lists.xensource.com
	Subject: Re: Ownership of machine pages: Was: [Xen-devel] Essay
on an important Xen decision (long)
	
	
	Sorry, if I missed something. But I did not read any discussion
on Harry Butterworth's idea of generic memory management API (xenidc and
USB driver).
	
	My experience is not enough to comment on this topic, but I
would like to understand these design decisions. As far as I know,
generic API's that provide proper abstractions prove useful (e.g. Linux
VirtualFS). There are counter arguments to it (performance degradation
mainly / wrong abstractions). But I didn't find any comments on the
xenidc API approach.
	
	Apart from that, thanks for elaborate description of some key
concepts.
	
	Jayesh
	
	
	On 1/12/06, James Bulpin <james@xensource.com> wrote: 

		Magenheimer, Dan (HP Labs Fort Collins) wrote:
		> (Sorry again for the late and off-thread reply... even
though
		> your message is addressed to me personally, to
xen-devel,
		> and to xen-ia64-devel, I didn't get a copy and just
saw it 
		> in xen-devel which I only receive digested.  Some
strangeness
		> in the mailing list nodupe feature maybe?  James
cc'ed....)
		
		If you were on the recipient list of the original mail
from Keir then
		that should have gone directly to you without passing
anywhere near the 
		list server, if you didn't get that then it's a problem
with mail
		servers at your or Keir's end. If you've got nodupe set
for
		xen-ia64-devel then your address being on the message
means that you
		won't get the message via the list. You still get the
message on the 
		xen-devel digest because nodupe doesn't apply (or make
sense) to digests.
		
		Cheers,
		James
		
		_______________________________________________
		Xen-devel mailing list
		Xen-devel@lists.xensource.com
		http://lists.xensource.com/xen-devel
		




	-- 
	Jayesh
	
	


[-- Attachment #1.2: Type: text/html, Size: 4321 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel