[BUNDLE] Testing a simpler inter-domain transport

[BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

Hi all,

	I've finally found time to resurrect my "share" code, update it for Xen
3.0 and (with Tony Breeds' help) created a simple LAN driver.  You can
find the bundle here:

	http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg

It's unoptimized, but shows some promise.  Here are the benchmarks for
tcpblast and tbench, on a uniproc 3GHz Pentium 4.  I'd appreciate SMP
numbers if someone has hardware on hand:

	UDP blast: tcpblast -u -s50000 dom0 9999
	  Current Xen = 254961 KB/s
	  Simple share = 233952 KB/s
	TCP blast: tcpblast -t -s50000 dom0 9999
	  Current Xen = 86566.4 KB/s
	  Simple share = 135415 KB/s
	Bidir tcp load: tbench 10
	  Current Xen = 31.9551 MB/sec
	  Simple share = 64.2113 MB/sec

It's not plumbed into xenbus, so creating LANs is a manual process,
using the dmesg output from the initial creation:

	dom0# modprobe ohlan create
	ohlan: created lan eth1 at address 0x1b6000
	domU# modprobe ohlan address=0x1b6000

Feedback welcome!
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: [BUNDLE] Testing a simpler inter-domain transport

[NAHieu]

On 2/5/06, Rusty Russell <rusty@rustcorp.com.au> wrote:
> Hi all,
>
>         I've finally found time to resurrect my "share" code, update it for Xen
> 3.0 and (with Tony Breeds' help) created a simple LAN driver.  You can
> find the bundle here:
>
>         http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg

Rusty, could you tell us what this code does???

Thanks.
Hieu

Re: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Sun, 2006-02-05 at 19:26 +0900, NAHieu wrote:
> On 2/5/06, Rusty Russell <rusty@rustcorp.com.au> wrote:
> > Hi all,
> >
> >         I've finally found time to resurrect my "share" code, update it for Xen
> > 3.0 and (with Tony Breeds' help) created a simple LAN driver.  You can
> > find the bundle here:
> >
> >         http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg
> 
> Rusty, could you tell us what this code does???

Strange, there's a README.xen-share file in that dir, but Apache doesn't
seem to show it.  I've put an explicit index in there now, but here's
the README.xen-share file:

Introduction to the Xen Share Code

Goal
----
To produce a simple interdomain transport which has device-like
characteristics.

Method
------
We need mapped I/O, interrupts and DMA.  For mapped I/O, dom0
explicitly creates "sharable" pages for the devices to use, using the
dom0 ops:

	DOM0_CREATESHAREDPAGES(num-pages)
	- Returns physically contiguous pages as a sharable region.
	DOM0_DESTROYSHAREDPAGES(pfn)
	- Free the shared pages once no domain is referencing them (see below)
	DOM0_GRANTSHAREDPAGES(domain, pfn)
	- Allow this domain access to these shared pages.

These are mapped by the domain using the multiplexed "share_op"
hypercall:
	XEN_SHARE_get(pfn,evtchn)
	- Returns a non-negative lowest-possible "peer id".  The
	  evtchn is used to notify of all events which occur on this
	  region.  The pages can then be mapped.
	XEN_SHARE_drop(pfn)
	- Drops the shared pages.

We have a method of receiving and sending notifications, based on
addresses within the shared region:

	XEN_SHARE_watch(address, u32 *decaddr)
	- When someone triggers this address (see below), atomically
	  decrement decaddr: if it hits 0, raise the eventchannel.
	XEN_SHARE_trigger(address)
	- Trigger any watches on this address (see above).

Finally, there is a method for registering scatter-gather lists for
input or output:

	XEN_SHARE_sg_register(pfn, read/write, #sgs, sg[], u32 *lenaddr)
	- Register an array of machine address/length pairs associated
	  with this share.  When it's used (see below), the length will be
	  written at lenaddr, the sg unregistered, and the event
	  channel raised.
	XEN_SHARE_sg_unregister(pfn, sg_addr)
	- Unregister the scatter-gather list with this first address.
	XEN_SHARE_sg_xfer(pfn, read/write, peerid, #sgs, sg[])
	- Copy from these virtual address/length pairs to this peer
	  associated with this share.  Returns the number of bytes
	  actually transferred.

These mechanisms create an efficient and simple way of writing virtual
drivers which behave like normal device drivers.

Future optimizations / improvements:

(1) The hypervisor code uses dumb linked lists, which could be arrays
    and hashes.

(2) The hypervisor currently always copies, but could page-flip.  This
    needs involve guest awareness in the non-shadow-pagetable model
    (the hypervisor *could* reach in and update the pfn array).

(3) The hypervisor should probably use the smallest possible sg entry.

(4) The entire protocol can be transparently remoted by the hypervisor.

(5) A trusted partition could ask the hypervisor for the sg list
    details for a domain, which it could then program directly into a
    device.

(6) The code isn't PAE friendly, and actually hands addresses in some
    places.  These should be fixed.

(7) The decaddr trick is a cute idea for interrupt mitigation, but
    currently unused and probably unnecessary.  It could be turned
    into a simple "set this to 1".

(8) Perhaps the hypervisor should wake at offset 0 whenever someone
    drops the share.


-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: [BUNDLE] Testing a simpler inter-domain transport

[Anthony Liguori]

Rusty Russell wrote:

>It's unoptimized, but shows some promise.  Here are the benchmarks for
>tcpblast and tbench, on a uniproc 3GHz Pentium 4.  I'd appreciate SMP
>numbers if someone has hardware on hand:
>
>	UDP blast: tcpblast -u -s50000 dom0 9999
>	  Current Xen = 254961 KB/s
>	  Simple share = 233952 KB/s
>	TCP blast: tcpblast -t -s50000 dom0 9999
>	  Current Xen = 86566.4 KB/s
>	  Simple share = 135415 KB/s
>	Bidir tcp load: tbench 10
>	  Current Xen = 31.9551 MB/sec
>	  Simple share = 64.2113 MB/sec
>  
>
I imagine the numbers for the Simple share should be pretty similiar for 
domU to domU right?  I also imagine that domU to domU under Current xen 
should be considerably worse right?

Any idea why there's UDP degradation?

This stuff looks awesome :-)

Regards,

Anthony Liguori

>It's not plumbed into xenbus, so creating LANs is a manual process,
>using the dmesg output from the initial creation:
>
>	dom0# modprobe ohlan create
>	ohlan: created lan eth1 at address 0x1b6000
>	domU# modprobe ohlan address=0x1b6000
>
>Feedback welcome!
>Rusty.
>  
>

Re: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Sun, 2006-02-05 at 10:56 -0600, Anthony Liguori wrote:
> Rusty Russell wrote:
> 
> >It's unoptimized, but shows some promise.  Here are the benchmarks for
> >tcpblast and tbench, on a uniproc 3GHz Pentium 4.  I'd appreciate SMP
> >numbers if someone has hardware on hand:

> I imagine the numbers for the Simple share should be pretty similiar for 
> domU to domU right?  I also imagine that domU to domU under Current xen 
> should be considerably worse right?

Yes.  The current driver is a hack, which creates the shared page when
invoked (in dom0) with "create" as a module param.  However, you can
attach as many domUs as you want to that same LAN.

> Any idea why there's UDP degradation?

No 8(.  I'm going to write a block driver and see what that's like.

I was surprised at how aggressively Xen is switching between domains
when an event channel was activated: so much so that we rarely got 32
packets off before the switch.   Less aggressive scheduling will
definitely help us, and possibly help the existing drivers.

> This stuff looks awesome :-)

Thanks, just fiddling.  It'll be interesting to see what changes the PPC
guys will need for this.

Cheers!
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: [BUNDLE] Testing a simpler inter-domain transport

[Magenheimer, Dan (HP Labs Fort Collins)]

> Thanks, just fiddling.  It'll be interesting to see what 
> changes the PPC
> guys will need for this.
> 
> Cheers!
> Rusty.

(I'm sure you meant PPC *and* ia64 ;*)

On just a quick skim, one thing to note:

IIRC from the summit, domain0 and driver domains for
neither PPC nor ia64 will have a p2m lookup table so
a p2m translation will require a hypercall. So
while virt_to_machine is cheap for domains on x86,
it is not on PPC and ia64.  If HYPERVISOR_share can
take physical addresses instead of machine addresses
(with Xen doing the phys_to_machine part of the
translation), I think the code would work better
for PPC and ia64, as well as better hide the
virtual->physical->machine memory abstraction.

Dan

RE: [BUNDLE] Testing a simpler inter-domain transport

[Yang, Fred]

Dan,

>From Xen summit, isn't it to be more P2M liked approach due to
consideration on driver domain and domain0 needs to get P2M for
VBD/VNIF?   

Don't remember there is decision on taking Hypercall only approach and
dropped P2M table lookup.  Any justification here?

-Fred

Magenheimer, Dan (HP Labs Fort Collins) wrote:
> (I'm sure you meant PPC *and* ia64 ;*)
> 
> On just a quick skim, one thing to note:
> 
> IIRC from the summit, domain0 and driver domains for
> neither PPC nor ia64 will have a p2m lookup table so
> a p2m translation will require a hypercall. So
> while virt_to_machine is cheap for domains on x86,
> it is not on PPC and ia64.  If HYPERVISOR_share can
> take physical addresses instead of machine addresses
> (with Xen doing the phys_to_machine part of the
> translation), I think the code would work better
> for PPC and ia64, as well as better hide the
> virtual->physical->machine memory abstraction.
> 
> Dan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: [BUNDLE] Testing a simpler inter-domain transport

[Magenheimer, Dan (HP Labs Fort Collins)]

Hi Fred --

My recollection is that we agreed (with the IBM PPC guys also)
that dom0 should use virtual-physical (VP), except for DMA
which would be forced to use a software IOMMU.

Note that Rusty's idt code replaces the existing VBD/VNIF
and grant table mechanism, so it would be nice to ensure
it is more portable before it becomes part of the Xen core.

Dan

> -----Original Message-----
> From: Yang, Fred [mailto:fred.yang@intel.com] 
> Sent: Monday, February 06, 2006 1:48 PM
> To: Magenheimer, Dan (HP Labs Fort Collins); 
> xen-devel@lists.xensource.com
> Cc: Rusty Russell; xen-ia64-devel
> Subject: RE: [Xen-devel] [BUNDLE] Testing a simpler 
> inter-domain transport
> 
> Dan,
> 
> From Xen summit, isn't it to be more P2M liked approach due to
> consideration on driver domain and domain0 needs to get P2M for
> VBD/VNIF?   
> 
> Don't remember there is decision on taking Hypercall only approach and
> dropped P2M table lookup.  Any justification here?
> 
> -Fred
> 
> Magenheimer, Dan (HP Labs Fort Collins) wrote:
> > (I'm sure you meant PPC *and* ia64 ;*)
> > 
> > On just a quick skim, one thing to note:
> > 
> > IIRC from the summit, domain0 and driver domains for
> > neither PPC nor ia64 will have a p2m lookup table so
> > a p2m translation will require a hypercall. So
> > while virt_to_machine is cheap for domains on x86,
> > it is not on PPC and ia64.  If HYPERVISOR_share can
> > take physical addresses instead of machine addresses
> > (with Xen doing the phys_to_machine part of the
> > translation), I think the code would work better
> > for PPC and ia64, as well as better hide the
> > virtual->physical->machine memory abstraction.
> > 
> > Dan
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 

Re: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Mon, 2006-02-06 at 12:01 -0800, Magenheimer, Dan (HP Labs Fort
Collins) wrote:
> > Thanks, just fiddling.  It'll be interesting to see what 
> > changes the PPC
> > guys will need for this.
> > 
> > Cheers!
> > Rusty.
> 
> (I'm sure you meant PPC *and* ia64 ;*)

Of course!

> On just a quick skim, one thing to note:
> 
> IIRC from the summit, domain0 and driver domains for
> neither PPC nor ia64 will have a p2m lookup table so
> a p2m translation will require a hypercall. So
> while virt_to_machine is cheap for domains on x86,
> it is not on PPC and ia64.  If HYPERVISOR_share can
> take physical addresses instead of machine addresses
> (with Xen doing the phys_to_machine part of the
> translation), I think the code would work better
> for PPC and ia64, as well as better hide the
> virtual->physical->machine memory abstraction.

First person to implement this on a different arch wins.  I'm currently
sewing in feedback and cleanups from Hollis for PPC requirements.

Now, I think that the virt_to_machine is fugly too.  I'd prefer to keep
all archs the same though.  Ideally, the hypercall would pass virtual
addresses and the hypervisor would figure out the machine address.  If
you want to figure out how to do that on x86, I'll gladly use it...

Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

RE: [BUNDLE] Testing a simpler inter-domain transport

[Magenheimer, Dan (HP Labs Fort Collins)]

 > > IIRC from the summit, domain0 and driver domains for
> > neither PPC nor ia64 will have a p2m lookup table so
> > a p2m translation will require a hypercall. So
> > while virt_to_machine is cheap for domains on x86,
> > it is not on PPC and ia64.  If HYPERVISOR_share can
> > take physical addresses instead of machine addresses
> > (with Xen doing the phys_to_machine part of the
> > translation), I think the code would work better
> > for PPC and ia64, as well as better hide the
> > virtual->physical->machine memory abstraction.
> 
> First person to implement this on a different arch wins.  I'm 
> currently
> sewing in feedback and cleanups from Hollis for PPC requirements.
> 
> Now, I think that the virt_to_machine is fugly too.  I'd 
> prefer to keep
> all archs the same though.  Ideally, the hypercall would pass virtual
> addresses and the hypervisor would figure out the machine address.  If
> you want to figure out how to do that on x86, I'll gladly use it...

On ia64 and I believe also on PPC, a guest can translate
from virtual to (pseudo)physical but only on x86 can
a guest translate from virtual to machine -- at least
without an extra hypercall.  On all three,
Xen can translate from (pseudo)physical to machine but
only on x86 can Xen translate from virtual to
(pseudo)physical. 

So it seems to me that if you "prefer to keep all archs the
same", the proper way to pass the parameters are as
(pseudo)physical addresses: the guest translates the
virtual address to a (pseudo)physical address and
Xen translates from the (pseudo)physical address to
the machine address and everybody is happy.

Hollis, is this correct for PPC?

Thanks,
Dan

RE: [BUNDLE] Testing a simpler inter-domain transport

[Hollis Blanchard]

On Tue, 2006-02-07 at 17:31 -0800, Magenheimer, Dan (HP Labs Fort
Collins) wrote:
> 
> On ia64 and I believe also on PPC, a guest can translate
> from virtual to (pseudo)physical but only on x86 can
> a guest translate from virtual to machine -- at least
> without an extra hypercall.  On all three,
> Xen can translate from (pseudo)physical to machine but
> only on x86 can Xen translate from virtual to
> (pseudo)physical. 

Yup.

> So it seems to me that if you "prefer to keep all archs the
> same", the proper way to pass the parameters are as
> (pseudo)physical addresses: the guest translates the
> virtual address to a (pseudo)physical address and
> Xen translates from the (pseudo)physical address to
> the machine address and everybody is happy.

Agreed.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: [BUNDLE] Testing a simpler inter-domain transport

[Daniel Veillard]

On Sun, Feb 05, 2006 at 02:38:51PM +1100, Rusty Russell wrote:
> Hi all,
> 
> 	I've finally found time to resurrect my "share" code, update it for Xen
> 3.0 and (with Tony Breeds' help) created a simple LAN driver.  You can
> find the bundle here:
> 
> 	http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg

  Hum, I tried
    hg clone http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg
and my version fails with a zlib compression error raising an incorrect header
check, 

> It's unoptimized, but shows some promise.  Here are the benchmarks for
> tcpblast and tbench, on a uniproc 3GHz Pentium 4.  I'd appreciate SMP
> numbers if someone has hardware on hand:
> 
> 	UDP blast: tcpblast -u -s50000 dom0 9999
> 	  Current Xen = 254961 KB/s
> 	  Simple share = 233952 KB/s
> 	TCP blast: tcpblast -t -s50000 dom0 9999
> 	  Current Xen = 86566.4 KB/s
> 	  Simple share = 135415 KB/s
> 	Bidir tcp load: tbench 10
> 	  Current Xen = 31.9551 MB/sec
> 	  Simple share = 64.2113 MB/sec
> 
> It's not plumbed into xenbus, so creating LANs is a manual process,
> using the dmesg output from the initial creation:
> 
> 	dom0# modprobe ohlan create
> 	ohlan: created lan eth1 at address 0x1b6000
> 	domU# modprobe ohlan address=0x1b6000
> 
> Feedback welcome!

  point to point performances looks good, but I'm starting to worry about
thing like group communication in a large Xen machine, assuming you can run
a few dozens domains consurrently, it may be useful to get some efficient
muticast based communication mechanism, the shared pages should help in some
ways (usuall memory/speed tradeoff though). Did you look at this ?

Daniel

-- 
Daniel Veillard      | Red Hat http://redhat.com/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Wed, 2006-02-08 at 05:12 -0500, Daniel Veillard wrote:
> On Sun, Feb 05, 2006 at 02:38:51PM +1100, Rusty Russell wrote:
> > Hi all,
> > 
> > 	I've finally found time to resurrect my "share" code, update it for Xen
> > 3.0 and (with Tony Breeds' help) created a simple LAN driver.  You can
> > find the bundle here:
> > 
> > 	http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg
> 
>   Hum, I tried
>     hg clone http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg
> and my version fails with a zlib compression error raising an incorrect header
> check, 

It's a bundle, so you should use:

	hg unbundle http://ozlabs.org/~rusty/xen/xen-share-2006-02-07.hg

(Note the updated version).

It'd be cool if clone did the Right Thing here, though...
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: [BUNDLE] Testing a simpler inter-domain transport

[Muli Ben-Yehuda]

On Wed, Feb 08, 2006 at 09:23:58PM +1100, Rusty Russell wrote:

> It's a bundle, so you should use:
> 
> 	hg unbundle http://ozlabs.org/~rusty/xen/xen-share-2006-02-07.hg
> 
> (Note the updated version).

FWIW, you also need -u here, or at least 'hg update' after
unbundling to actually see the changes, analogue to 'hg pull' vs. 'hg
pull -u'.

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

Re: [BUNDLE] Testing a simpler inter-domain transport

[Daniel Veillard]

On Wed, Feb 08, 2006 at 12:34:41PM +0200, Muli Ben-Yehuda wrote:
> On Wed, Feb 08, 2006 at 09:23:58PM +1100, Rusty Russell wrote:
> 
> > It's a bundle, so you should use:
> > 
> > 	hg unbundle http://ozlabs.org/~rusty/xen/xen-share-2006-02-07.hg
> > 
> > (Note the updated version).
> 
> FWIW, you also need -u here, or at least 'hg update' after
> unbundling to actually see the changes, analogue to 'hg pull' vs. 'hg
> pull -u'.

  Without -u it fails complaining about no repo available, creating a fake
.hg directory it tries to add changesets but aborts with 
   'unknown base 6b1d39a5!'
Trying to add -u as suggested result in 
   'hg unbundle: option -u not recognized'
using mercurial-0.7

  The bundle doesn't seems able to fully provide a repo state, and without
a repo state it seems unable to unbundle the data. I think mercurial hates
me, probably because I still don't really understand how it works internally...

Daniel

-- 
Daniel Veillard      | Red Hat http://redhat.com/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [BUNDLE] Testing a simpler inter-domain transport

[Muli Ben-Yehuda]

On Wed, Feb 08, 2006 at 05:46:38AM -0500, Daniel Veillard wrote:

>   Without -u it fails complaining about no repo available, creating a fake
> .hg directory it tries to add changesets but aborts with 
>    'unknown base 6b1d39a5!'

Ok, you need to unbundle into an existing Xen mercurial tree (just
clone unstable from xenbits). Think of unbundling as applying a set of
patches - you need something to apply them to...

> Trying to add -u as suggested result in 
>    'hg unbundle: option -u not recognized'
> using mercurial-0.7

Sorry, I think it's new in 0.8. You can always update in a seperate
step.

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

Re: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Wed, 2006-02-08 at 05:12 -0500, Daniel Veillard wrote:
> On Sun, Feb 05, 2006 at 02:38:51PM +1100, Rusty Russell wrote:
> > 	dom0# modprobe ohlan create
> > 	ohlan: created lan eth1 at address 0x1b6000
> > 	domU# modprobe ohlan address=0x1b6000
> > 
> > Feedback welcome!
> 
>   point to point performances looks good, but I'm starting to worry about
> thing like group communication in a large Xen machine, assuming you can run
> a few dozens domains consurrently, it may be useful to get some efficient
> muticast based communication mechanism, the shared pages should help in some
> ways (usuall memory/speed tradeoff though). Did you look at this ?

Yes, you can add other domains to the same lan:

	domU2# modprobe ohlan address=0x1b6000

Currently limited to 32 partitions for no particularly good reason.
Note that like a real LAN, one badly behaved partition can block
communication for the others they share the lan with...

Sorry for the confusion!
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: [BUNDLE] Testing a simpler inter-domain transport

[Matt Mackall]

On Wed, Feb 08, 2006 at 09:23:58PM +1100, Rusty Russell wrote:
> On Wed, 2006-02-08 at 05:12 -0500, Daniel Veillard wrote:
> > On Sun, Feb 05, 2006 at 02:38:51PM +1100, Rusty Russell wrote:
> > > Hi all,
> > > 
> > > 	I've finally found time to resurrect my "share" code, update it for Xen
> > > 3.0 and (with Tony Breeds' help) created a simple LAN driver.  You can
> > > find the bundle here:
> > > 
> > > 	http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg
> > 
> >   Hum, I tried
> >     hg clone http://ozlabs.org/~rusty/xen/xen-share-2006-02-05.hg
> > and my version fails with a zlib compression error raising an incorrect header
> > check, 

Hmm, never tried that. Surprised it got anywhere at all.

> It's a bundle, so you should use:
> 
> 	hg unbundle http://ozlabs.org/~rusty/xen/xen-share-2006-02-07.hg
> 
> (Note the updated version).
> 
> It'd be cool if clone did the Right Thing here, though...

Well clone can't in this case, as this isn't a full repo history. But
pull could..

We'd have to teach pull/clone to do a preliminary query on a URL to
see if it was a bundle.

-- 
Mathematics is the supreme nostalgia of our time.

RE: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Tue, 2006-02-07 at 17:31 -0800, Magenheimer, Dan (HP Labs Fort
Collins) wrote:
> On ia64 and I believe also on PPC, a guest can translate
> from virtual to (pseudo)physical but only on x86 can
> a guest translate from virtual to machine -- at least
> without an extra hypercall.  On all three,
> Xen can translate from (pseudo)physical to machine but
> only on x86 can Xen translate from virtual to
> (pseudo)physical. 

I don't think x86 Xen can translate physical to machine, only machine to
physical.  So I think we're going to need arch-specific wrappers for
these translations anyway, since everyone wants different things.

This shouldn't be too bad: if you want to take a crack at it I'd be
happy to apply your patches, or you can wait until I finish the block
device (probably next week).

Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

RE: [BUNDLE] Testing a simpler inter-domain transport

[Magenheimer, Dan (HP Labs Fort Collins)]

> > On ia64 and I believe also on PPC, a guest can translate
> > from virtual to (pseudo)physical but only on x86 can
> > a guest translate from virtual to machine -- at least
> > without an extra hypercall.  On all three,
> > Xen can translate from (pseudo)physical to machine but
> > only on x86 can Xen translate from virtual to
> > (pseudo)physical. 
> 
> I don't think x86 Xen can translate physical to machine, only 
> machine to
> physical.  So I think we're going to need arch-specific wrappers for
> these translations anyway, since everyone wants different things.

Ah, I see you are (mostly) correct.  The phys_to_machine_mapping
table is apparently only used on x86 by HVM and shadow mode.

> This shouldn't be too bad: if you want to take a crack at it I'd be
> happy to apply your patches, or you can wait until I finish the block
> device (probably next week).

Given the above, I think I will wait to see what you come up with.

Thanks,
Dan

RE: [BUNDLE] Testing a simpler inter-domain transport

[King, Steven R]

> Note that like a real LAN, one badly behaved partition
> can block communication for the others they share the lan with... 

Shared page LAN is much less secure than a real LAN.  Any domain
attached to the shared page, i.e. in the LAN, can modify any frame "in
flight" on the page.  Recipients have no confidence that the received
frame is actually what the sender sent.

-steve



-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Rusty
Russell
Sent: Wednesday, February 08, 2006 3:05 AM
To: veillard@redhat.com
Cc: xen-devel; Tony Breeds
Subject: Re: [Xen-devel] [BUNDLE] Testing a simpler inter-domain
transport

On Wed, 2006-02-08 at 05:12 -0500, Daniel Veillard wrote:
> On Sun, Feb 05, 2006 at 02:38:51PM +1100, Rusty Russell wrote:
> > 	dom0# modprobe ohlan create
> > 	ohlan: created lan eth1 at address 0x1b6000
> > 	domU# modprobe ohlan address=0x1b6000
> > 
> > Feedback welcome!
> 
>   point to point performances looks good, but I'm starting to worry 
> about thing like group communication in a large Xen machine, assuming 
> you can run a few dozens domains consurrently, it may be useful to get

> some efficient muticast based communication mechanism, the shared 
> pages should help in some ways (usuall memory/speed tradeoff though).
Did you look at this ?

Yes, you can add other domains to the same lan:

	domU2# modprobe ohlan address=0x1b6000

Currently limited to 32 partitions for no particularly good reason.
Note that like a real LAN, one badly behaved partition can block
communication for the others they share the lan with...

Sorry for the confusion!
Rusty.
--
 ccontrol: http://ozlabs.org/~rusty/ccontrol


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [BUNDLE] Testing a simpler inter-domain transport

[Ronald G Minnich]

King, Steven R wrote:

> Shared page LAN is much less secure than a real LAN.  Any domain
> attached to the shared page, i.e. in the LAN, can modify any frame "in
> flight" on the page.  Recipients have no confidence that the received
> frame is actually what the sender sent.

yes, this discussion came up some time ago (I noticed it when i was 
doing the Plan 9 port) and I think the conclusion was "yep, it's an 
issue". I am not sure where it all went -- I have not yet started the 
xen 3.0 port for plan 9.

ron

RE: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Sun, 2006-02-12 at 15:39 -0800, King, Steven R wrote:
> > Note that like a real LAN, one badly behaved partition
> > can block communication for the others they share the lan with... 
> 
> Shared page LAN is much less secure than a real LAN.  Any domain
> attached to the shared page, i.e. in the LAN, can modify any frame "in
> flight" on the page.  Recipients have no confidence that the received
> frame is actually what the sender sent.

Hi Steve,

	I don't quite know how to respond to this!  Your statement is true
given some assumptions, but not relevent to my implementation, hence the
presence of your assertion in this thread is quixotic.

	In my implementation, you can't tell which domain on the LAN a packet
came from, nor do I try to prevent malicious domains on the LAN from
effectively stopping all useful traffic.  I believe that multi-domain
access is useful in some scenarious, nonetheless.

Hope that clarifies?
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

RE: [BUNDLE] Testing a simpler inter-domain transport

[King, Steven R]

Sorry, quixotic as charged.  :^)  Your patch is one thing, multi-domain
shared page LAN's are another.

If multi-domain shared page LAN's become more than a proof-of-concept
for your patch, we can worry about it then.  You mention the DOS attack,
but there are other problems that have no wired-LAN analog.  From Mr.
Minnich, it sounds such a thread already ran its course.  I looked
briefly but could not find it in the xen-devel archives.

-steve

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Rusty
Russell
Sent: Sunday, February 12, 2006 6:33 PM
To: King, Steven R
Cc: xen-devel
Subject: RE: [Xen-devel] [BUNDLE] Testing a simpler inter-domain
transport

On Sun, 2006-02-12 at 15:39 -0800, King, Steven R wrote:
> > Note that like a real LAN, one badly behaved partition can block 
> > communication for the others they share the lan with...
> 
> Shared page LAN is much less secure than a real LAN.  Any domain 
> attached to the shared page, i.e. in the LAN, can modify any frame "in

> flight" on the page.  Recipients have no confidence that the received 
> frame is actually what the sender sent.

Hi Steve,

	I don't quite know how to respond to this!  Your statement is
true given some assumptions, but not relevent to my implementation,
hence the presence of your assertion in this thread is quixotic.

	In my implementation, you can't tell which domain on the LAN a
packet came from, nor do I try to prevent malicious domains on the LAN
from effectively stopping all useful traffic.  I believe that
multi-domain access is useful in some scenarious, nonetheless.

Hope that clarifies?
Rusty.
--
 ccontrol: http://ozlabs.org/~rusty/ccontrol


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [BUNDLE] Testing a simpler inter-domain transport

[Rusty Russell]

On Sun, 2006-02-12 at 19:24 -0800, King, Steven R wrote:
> Sorry, quixotic as charged.  :^)  Your patch is one thing, multi-domain
> shared page LAN's are another.

That's OK, I was a little confused, that's all.

> If multi-domain shared page LAN's become more than a proof-of-concept
> for your patch, we can worry about it then.  You mention the DOS attack,
> but there are other problems that have no wired-LAN analog.

Agreed.  Although it's not a problem I'm losing sleep on.  At least with
virtual networks you can cheaply set up one LAN per pair.

Cheers!
Rusty.
-- 
 ccontrol: http://ozlabs.org/~rusty/ccontrol

Re: [BUNDLE] Testing a simpler inter-domain transport

[Daniel Veillard]

On Sun, Feb 12, 2006 at 03:39:01PM -0800, King, Steven R wrote:
> > Note that like a real LAN, one badly behaved partition
> > can block communication for the others they share the lan with... 
> 
> Shared page LAN is much less secure than a real LAN.  Any domain
> attached to the shared page, i.e. in the LAN, can modify any frame "in
> flight" on the page.  Recipients have no confidence that the received
> frame is actually what the sender sent.

  I don't see why this would have to be the case, it should be
possible to allow only read access to pages containing packets
sent from other domains (at the expense of a bit more physical
memory being used for the service).
  Concerning host blocking communications:
    - blocking on burst write could be detected, the sender
      set of pages are full, it's a FIFO mechanism with counters
    - blocking on non-read is more problematic, but it's IMHO
      very similar to the already very well analyzed problem
      of detecting failing nodes (and partitions) in distributed
      systems. However being on the same physical machine should
      simplify handling of those case quite a bit.
(note that with only 2 domains you can't distinguish the two cases
 but if you have a garanteed well behaved node on Domain-0 you 
 should be able to always distinguish them).

  It would probably require significant code changes from an
optimistic implementation of the inter-domain transport but I don't
see why this could not be done. Sounds like it would target a relatively
different set of use case than the optimistic one, and maybe the 
cost isn't worth the protection this would bring in that context.

  Or did I missed something ?

Daniel

-- 
Daniel Veillard      | Red Hat http://redhat.com/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/