RE: RFC: "local" communications

RE: RFC: "local" communications

[Ian Pratt]

> In order for this to work better, it would be good if there 
> was a way for software in a domU to communicate with software 
> on the local dom0 - so after being migrated elsewhere, a 
> guest would start talking to the new dom0 automatically.
> 
> Now what I'm not sure about is, what would be the best way to do this?
> 
> Should it be some virtual device analogous to the virtual 
> network driver, or should we have some other kind of socket 
> for node local communications ?
> 
> What kind of approach would the Xen community (that's you, if 
> you read this far) prefer ?

For relatively static stuff (e.g. textual configuration data), it would
be reasonable to use xenbus/xenstore. We could give each domain a
subtree in the store that it can read/write via a /proc/sys interface.

For other low bandwidth stuff we could have a simple protocol for
multiplexing datagrams over the virtual console channel. 

For high bandwidth stuff we'd want to set up separate shared memory
rings. 

Rik: Does this mean you could be pursuaded to look at splitting the
console handling out of xend? :-)

Ian

Re: RFC: "local" communications

[Nivedita Singhvi]

Ian Pratt wrote:

> For relatively static stuff (e.g. textual configuration data), it would
> be reasonable to use xenbus/xenstore. We could give each domain a
> subtree in the store that it can read/write via a /proc/sys interface.

Ian, how flexible is the format/layout of the xenstore going to
be? Does it allow for more of a general purpose use? That would
be neat..

thanks,
Nivedita

RE: RFC: "local" communications

[Rik van Riel]

On Wed, 15 Jun 2005, Ian Pratt wrote:

> > What kind of approach would the Xen community (that's you, if 
> > you read this far) prefer ?
> 
> For relatively static stuff (e.g. textual configuration data), it would
> be reasonable to use xenbus/xenstore. We could give each domain a
> subtree in the store that it can read/write via a /proc/sys interface.

I guess xenbus should work fine...

The main uses are for doing things like querying "which physical
system am I running on", fencing off a virtual cluster member, etc.

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

RE: RFC: "local" communications

[Lon Hohberger]

On Wed, 2005-06-15 at 18:43 -0400, Rik van Riel wrote:

> > For relatively static stuff (e.g. textual configuration data), it would
> > be reasonable to use xenbus/xenstore. We could give each domain a
> > subtree in the store that it can read/write via a /proc/sys interface.
> 
> I guess xenbus should work fine...
> 
> The main uses are for doing things like querying "which physical
> system am I running on", fencing off a virtual cluster member, etc.

Agreed, that's the primary thing we need right now.  There are other
things, but one thing at a time.  Xenbus sounds like a good start, but
we'll need something for more dynamic information too.

-- Lon

RE: RFC: "local" communications

[Ian Pratt]

 

> -----Original Message-----
> From: Nivedita Singhvi [mailto:niv@us.ibm.com] 
> Sent: 15 June 2005 23:11
> To: Ian Pratt
> Cc: Rik van Riel; xen-devel@lists.xensource.com; Lon Hohberger
> Subject: Re: [Xen-devel] RFC: "local" communications
> 
> Ian Pratt wrote:
> 
> > For relatively static stuff (e.g. textual configuration data), it 
> > would be reasonable to use xenbus/xenstore. We could give 
> each domain 
> > a subtree in the store that it can read/write via a 
> /proc/sys interface.
> 
> Ian, how flexible is the format/layout of the xenstore going 
> to be? Does it allow for more of a general purpose use? That 
> would be neat..

It enables something akin to /proc. i.e. you can have an arbitrary
hierarchy of files. To retain the same spirit as everything else in
xenstore it would be preferable that both file names and contents were
text (rather than binary).

I think I quite like the idea of having a /proc/sys interface onto
xenstore. We'll still need the console multiplexing approach for
stream/datagram data.

Ian



  

RE: RFC: "local" communications

[Rusty Russell]

On Wed, 2005-06-15 at 23:32 +0100, Ian Pratt wrote:
> I think I quite like the idea of having a /proc/sys interface onto
> xenstore.

It's one of those tempting things that doesn't *quite* work.  Creation
of new directories and nodes is kinda important if userspace is going to
be doing anything non-trivial. 8(

Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman

RFC: "local" communications

[Rik van Riel]

Hi,

When combining clustering with cluster storage, the situation
arises where virtual machines are not only managed by the cluster
software, but also need to participate in the cluster software
themselves...

In order for this to work better, it would be good if there
was a way for software in a domU to communicate with software
on the local dom0 - so after being migrated elsewhere, a guest
would start talking to the new dom0 automatically.

Now what I'm not sure about is, what would be the best way to
do this?

Should it be some virtual device analogous to the virtual
network driver, or should we have some other kind of socket
for node local communications ?

What kind of approach would the Xen community (that's you, if
you read this far) prefer ?

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: RFC: "local" communications

[Anthony Liguori]

Rik van Riel wrote:

>Hi,
>
>When combining clustering with cluster storage, the situation
>arises where virtual machines are not only managed by the cluster
>software, but also need to participate in the cluster software
>themselves...
>
>In order for this to work better, it would be good if there
>was a way for software in a domU to communicate with software
>on the local dom0 - so after being migrated elsewhere, a guest
>would start talking to the new dom0 automatically.
>
>Now what I'm not sure about is, what would be the best way to
>do this?
>
>Should it be some virtual device analogous to the virtual
>network driver, or should we have some other kind of socket
>for node local communications ?
>
>What kind of approach would the Xen community (that's you, if
>you read this far) prefer ?
>  
>
I can only speak for myself, but I'd personally like to see a small 
userspace library that established a datagram connection between two 
domains (using an event channel and a shared page) that could be reused 
for many applications.

If it could be used with normal read/write calls that would be even 
better (although first thought is that that would require a daemon).

Regards,

Anthony Liguori

Re: RFC: "local" communications

[Nivedita Singhvi]

Anthony Liguori wrote:
> Rik van Riel wrote:
> 
>> Hi,
>>
>> When combining clustering with cluster storage, the situation
>> arises where virtual machines are not only managed by the cluster
>> software, but also need to participate in the cluster software
>> themselves...
>>
>> In order for this to work better, it would be good if there
>> was a way for software in a domU to communicate with software
>> on the local dom0 - so after being migrated elsewhere, a guest
>> would start talking to the new dom0 automatically.

Rik, is this a heartbeat kind of protocol or some such thing?
Is there value to making it be a generic solution that would
work when communicating in a non-xen environment?

>> Should it be some virtual device analogous to the virtual
>> network driver, or should we have some other kind of socket
>> for node local communications ?

Using protocols like TCP for node-local comms is undesirable
as TCP is engineered for the heavy machinery of the Internet.
UDP or raw sockets will be somewhat better, although AF_UNIX
would be best of all (albeit with its own constraints).

>> What kind of approach would the Xen community (that's you, if
>> you read this far) prefer ?
>>  
>>
> I can only speak for myself, but I'd personally like to see a small 
> userspace library that established a datagram connection between two 
> domains (using an event channel and a shared page) that could be reused 
> for many applications.

Which datagram service, Anthony?  Having a small lib like this
would be a good idea, although the benefit would be in the details,
and of course, depending on how many apps we need to support that
we don't want to modify...

> If it could be used with normal read/write calls that would be even 
> better (although first thought is that that would require a daemon).

Probably don't want to do that...

thanks,
Nivedita

Re: RFC: "local" communications

[Anthony Liguori]

Nivedita Singhvi wrote:

> UDP or raw sockets will be somewhat better, although AF_UNIX
> would be best of all (albeit with its own constraints).

I was thinking of a raw interface although a UDP-like protocol would be 
kind of neat (so that you could do multiplexing).

> Which datagram service, Anthony?  Having a small lib like this
> would be a good idea, although the benefit would be in the details,
> and of course, depending on how many apps we need to support that
> we don't want to modify...

I must admit I've not put much thought into it.  What are your thoughts?

Regards,

Anthony Liguori

Re: RFC: "local" communications

[Nivedita Singhvi]

Anthony Liguori wrote:
> Nivedita Singhvi wrote:
> 
>> UDP or raw sockets will be somewhat better, although AF_UNIX
>> would be best of all (albeit with its own constraints).
> 
> 
> I was thinking of a raw interface although a UDP-like protocol would be 
> kind of neat (so that you could do multiplexing).

It would be less work and easier on the apps to use UDP than raw,
but that's just my experience (and, potentially more bugs). This
could be solved by having a user space library wrap the details
and call a simple low level protocol over raw sockets.

>> Which datagram service, Anthony?  Having a small lib like this
>> would be a good idea, although the benefit would be in the details,
>> and of course, depending on how many apps we need to support that
>> we don't want to modify...
> 
> 
> I must admit I've not put much thought into it.  What are your thoughts?

Me neither, much, yet, :). But, it would be nice to make things easy
for the large number of apps which need intra-node communication. At
the moment, they fall into the two categories of strictly intra-node
versus common-over-intra_off_node. Much easier designing things for
the former.

thanks,
Nivedita

Re: RFC: "local" communications

[Lon Hohberger]

On Wed, 2005-06-15 at 14:31 -0700, Nivedita Singhvi wrote:
> >> Should it be some virtual device analogous to the virtual
> >> network driver, or should we have some other kind of socket
> >> for node local communications ?
> 
> Using protocols like TCP for node-local comms is undesirable
> as TCP is engineered for the heavy machinery of the Internet.
> UDP or raw sockets will be somewhat better, although AF_UNIX
> would be best of all (albeit with its own constraints).

For domU->dom0 within a node, I agree.

For domU->domU communications, I kind of disagree: If we migrate a domU
to another node, we shouldn't have to change how we're talking to that
domain.  Then again, in this case, we might not care for the type of
information we're sending/receiving.

Also, I think there's only a very limited set of information which
should be available over whatever communications model is employed (at
least, initially).  If a domU is compromised, we certainly don't want
the intruder to be able to find out anything significant about other
domains.

(Sorry for sounding like Captain Obvious, I'm sure this has been
considered)

-- Lon

Re: RFC: "local" communications

[Rusty Russell]

On Wed, 2005-06-15 at 15:24 -0500, Anthony Liguori wrote:
> Rik van Riel wrote:
> >In order for this to work better, it would be good if there
> >was a way for software in a domU to communicate with software
> >on the local dom0 - so after being migrated elsewhere, a guest
> >would start talking to the new dom0 automatically.
> 
> I can only speak for myself, but I'd personally like to see a small 
> userspace library that established a datagram connection between two 
> domains (using an event channel and a shared page) that could be reused 
> for many applications.

We have such a thing in the xenstore code.  It could be extracted fairly
easily.  Naturally, you need a setup mechanism, but the store could do
that.

Cheers,
Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman