Communicating with the domU from dom0 without Network

Communicating with the domU from dom0 without Network

[Ligesh]

 Hi Folks,

  I would like to have a discussion on whether if it isn't desirable to have some mechanism to directly communicate with the domU from the dom0. Wouldn't that make managing this entire setup much easier? In the openVZ, the vzctl tool can actually run anything inside a virtual machine. For instance, we can say 'vzctl exec $vpsid ps ax', and it will return the output of the 'ps' executed inside the vps. I realize that the virtualization concept is radically different in Xen, and thus this type of communication would be non-trivial.

  Anyway, I would like to know the pros and cons of having a direct mechanism to communicate with the domUs. As far as I have seen, the inability to manage the domUs directly has been the ONLY drawback of xen. 

  Thanks in advance.

Re: Communicating with the domU from dom0 without Network

[Muli Ben-Yehuda]

On Mon, Aug 07, 2006 at 06:10:15PM +0530, Ligesh wrote:
>  Hi Folks,
> 
>   I would like to have a discussion on whether if it isn't desirable
>   to have some mechanism to directly communicate with the domU from
>   the dom0. Wouldn't that make managing this entire setup much
>   easier? In the openVZ, the vzctl tool can actually run anything
>   inside a virtual machine. For instance, we can say 'vzctl exec
>   $vpsid ps ax', and it will return the output of the 'ps' executed
>   inside the vps. I realize that the virtualization concept is
>   radically different in Xen, and thus this type of communication
>   would be non-trivial.
> 
>   Anyway, I would like to know the pros and cons of having a direct
>   mechanism to communicate with the domUs. As far as I have seen,
>   the inability to manage the domUs directly has been the ONLY
>   drawback of xen. 

I agree this could be useful. If you have networking setup between
dom0 and the domU's (which I think is set up by default), you can do
this without any Xen involvement by just using ssh:

ssh domu.ip.add.ress command args args args

To avoid having ssh ask for a password, use passwordless SSH as
explained in the SSH documentation.

Cheers,
Muli

Re: Communicating with the domU from dom0 without Network

[Ligesh]

On Mon, Aug 07, 2006 at 03:57:54PM +0300, Muli Ben-Yehuda wrote:
> On Mon, Aug 07, 2006 at 06:10:15PM +0530, Ligesh wrote:
> >  Hi Folks,
> > 
> >   I would like to have a discussion on whether if it isn't desirable
> >   to have some mechanism to directly communicate with the domU from
> >   the dom0. Wouldn't that make managing this entire setup much
> >   easier? In the openVZ, the vzctl tool can actually run anything
> >   inside a virtual machine. For instance, we can say 'vzctl exec
> >   $vpsid ps ax', and it will return the output of the 'ps' executed
> >   inside the vps. I realize that the virtualization concept is
> >   radically different in Xen, and thus this type of communication
> >   would be non-trivial.
> > 
> >   Anyway, I would like to know the pros and cons of having a direct
> >   mechanism to communicate with the domUs. As far as I have seen,
> >   the inability to manage the domUs directly has been the ONLY
> >   drawback of xen. 
> 
> I agree this could be useful. If you have networking setup between
> dom0 and the domU's (which I think is set up by default), you can do
> this without any Xen involvement by just using ssh:
> 
> ssh domu.ip.add.ress command args args args

 How do we do this for windows? The ideal would be an internal mechanism to spawn and execute a process inside the DomU, and return its output/error/status back. Or at least we should be able to send some message to the domU using its NAME, and not ipaddress. This would make it easy to write scripts to automate the management, which would be difficult with the ipaddress, since ipaddress can change at some time.

 xen exec 'myvm.xm' command args

 or at least 

 xen sendinfo 'myvm.xm' command args.

 Inside the vm there can be a process that can access these messages, and respond accordingly. Anyway, the point is to have a mechanism to manipulate a domU using its name, rather than the ip.

 Thanks.



--
:: Ligesh :: http://ligesh.com 

Re: Communicating with the domU from dom0 without Network

[Muli Ben-Yehuda]

On Mon, Aug 07, 2006 at 07:01:30PM +0530, Ligesh wrote:

>  How do we do this for windows? The ideal would be an internal
>  mechanism to spawn and execute a process inside the DomU, and
>  return its output/error/status back.

Unlike openvz where all containers can run the same executables and
largely share the same environment, in Xen domU's have nothing in
common except being able to run on the same CPU. "Running an
executable" is inherently domain-specific and thus not something that
Xen should know about.

>  Or at least we should be able to send some message to the domU
>  using its NAME, and not ipaddress. This would make it easy to write
>  scripts to automate the management, which would be difficult with
>  the ipaddress, since ipaddress can change at some time.

How about a simple script that given a domain's name or domain ID will
return its IP?

Cheers,
Muli

Re: Communicating with the domU from dom0 without Network

[Ligesh]

On Mon, Aug 07, 2006 at 04:32:03PM +0300, Muli Ben-Yehuda wrote:
> On Mon, Aug 07, 2006 at 07:01:30PM +0530, Ligesh wrote:
> Unlike openvz where all containers can run the same executables and
> largely share the same environment, in Xen domU's have nothing in
> common except being able to run on the same CPU. "Running an
> executable" is inherently domain-specific and thus not something that
> Xen should know about.
> 

  It is evident that it is a bad design choice to let Xen be aware of the internals of domU, but I would like to know if it can be implemented as some sort of extra or something. Anyway, that is what would be the _ideal_ system management scenario. What would be practical is some form a message passing system, where a process running inside the domU can get messages directly from the dom0 without the help of network.


> >  Or at least we should be able to send some message to the domU
> >  using its NAME, and not ipaddress. This would make it easy to write
> >  scripts to automate the management, which would be difficult with
> >  the ipaddress, since ipaddress can change at some time.
> 
> How about a simple script that given a domain's name or domain ID will
> return its IP?

 Is there any fool proof way to get an ip of the domU using its name? It is possible that the domU has been configured internally with a totally different IP from the one that has been assigned to it via the dhcp.

 Thanks for your time.

--
:: Ligesh :: http://ligesh.com 

Re: Re: Communicating with the domU from dom0 without Network

[Michael LeMay]

This is precisely the sort of problem that the Keir's proposal seems to
address.  I've copied my comments on the proposal below; perhaps we can
discuss them further now since nobody was interested when I originally
posted them. :-)

---

Here's another general comment for discussion...

The bottom of page 18 in the Xen Roadmap proposal recommends considering
how to "export byte stream
(TCP) data between domains in a high performance fashion."  For
communications that occur between domains on a single physical machine,
it would seem logical to setup a new address and protocol family within
Linux that could be used to create and manipulate stream sockets via the
standard interfaces (I'm focusing on Linux at this point, although
similar adaptations could be made to other kernels).  Then, behind the
scenes, the Xen grant tables could be used to efficiently transfer
socket buffers between the domains.  This should involve much less
overhead than directly connecting two network frontends or performing
other optimizations at lower layers, since it would truncate the
protocol stack and avoid unnecessary TCP-style flow control protocols.

An enhancement such as this could help to eliminate the network
dependence of some Xen management applications, particularly those that
rely on XML-RPC to communicate.  For example, xm currently uses a UNIX
domain socket to communicate with Xend, which introduces an artificial
requirement that xend and xm be running in the same domain.  Once XenSE
gains traction and management utilities are scattered across multiple
domains, UNIX domain sockets will no longer be adequate.  Under this
scheme, stream sockets to specific domains could easily be constructed,
without regard for the network configuration on the system.

One important detail that I haven't yet resolved is how to address
inter-domain sockets.  Of course, the most important component in the
address for each socket would be the domain ID.  However, some sort of
port specification or pathname would also be necessary.  I'm not sure
which of those options would be appropriate in this case.  Port numbers
would be consistent with TCP and would probably ease the task of porting
applications based on TCP, but pathnames are more consistent with the
UNIX domain sockets used by xm and xend.  Perhaps we could provide both,
using two address families associated with the same protocol family?

What other ideas have been floating around on how to accomplish
byte-stream transport between domains?  Are any concrete efforts to
provide this functionality currently underway?  Thanks!




Ligesh wrote:
> On Mon, Aug 07, 2006 at 04:32:03PM +0300, Muli Ben-Yehuda wrote:
>   
>> On Mon, Aug 07, 2006 at 07:01:30PM +0530, Ligesh wrote:
>> Unlike openvz where all containers can run the same executables and
>> largely share the same environment, in Xen domU's have nothing in
>> common except being able to run on the same CPU. "Running an
>> executable" is inherently domain-specific and thus not something that
>> Xen should know about.
>>
>>     
>
>   It is evident that it is a bad design choice to let Xen be aware of the internals of domU, but I would like to know if it can be implemented as some sort of extra or something. Anyway, that is what would be the _ideal_ system management scenario. What would be practical is some form a message passing system, where a process running inside the domU can get messages directly from the dom0 without the help of network.
>
>
>   
>>>  Or at least we should be able to send some message to the domU
>>>  using its NAME, and not ipaddress. This would make it easy to write
>>>  scripts to automate the management, which would be difficult with
>>>  the ipaddress, since ipaddress can change at some time.
>>>       
>> How about a simple script that given a domain's name or domain ID will
>> return its IP?
>>     
>
>  Is there any fool proof way to get an ip of the domU using its name? It is possible that the domU has been configured internally with a totally different IP from the one that has been assigned to it via the dhcp.
>
>  Thanks for your time.
>
> --
> :: Ligesh :: http://ligesh.com 
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

Re: Communicating with the domU from dom0 without Network

[Ligesh]

 Hey, that's a handful as far as I am concerned. :-)  I was more interested in the 'manageability' area, where I was looking for a fool proof method to communicate with the VM, since the networking identification is not a reliable mechanism to access one. But, now I realize maybe I have been a bit remiss, and actually the serial port is a unique identifier for a domU, and can be used to communicate with it.  Inside the domU, you can have a long running server waiting at one of the serial ports, and from the Dom0, you can communicate with a domU using the serial port. Since everything is in software, I am sure the communication is going to be fast--i am not privy enough with serial communication to know if there are some real bottlenecks with the protocol or something, but from my informal approach, I think the serial port should actually work fine. Even the ip address can be configured r
 emotely in this fashion.

 I see that you are more interested in the performance, and I think that is a cool idea, though implementing the protocols for different Operating systems would be non-trivial. You will need to make kernel changes, and that would mean that you have to have drivers, which should be specifically loaded into the domU Operating system, and then you will have to maintain all the drivers for each OS. Secondly, the question is how ubiquitous Xen is going to be. If every single computer is going to have Xen, then such a method would actually yeild a lot of practical benefits. I guess we have to compare the overheads of writing and maintaining a new protocol for MANY operating systems to the efficiency derived in the long term.

 Anyway, I am just curious. Won't an extra serial port actually solve all the problems that you are intending to solve using tcp like communication? Is there any limitation to serial port communication, especially since everything is happening in software?

 Thanks.
 
--
:: Ligesh :: http://ligesh.com 




On Mon, Aug 07, 2006 at 12:18:55PM -0400, Michael LeMay wrote:
> This is precisely the sort of problem that the Keir's proposal seems to
> address.  I've copied my comments on the proposal below; perhaps we can
> discuss them further now since nobody was interested when I originally
> posted them. :-)
> 
> ---
> 
> Here's another general comment for discussion...
> 
> The bottom of page 18 in the Xen Roadmap proposal recommends considering
> how to "export byte stream
> (TCP) data between domains in a high performance fashion."  For
> communications that occur between domains on a single physical machine,
> it would seem logical to setup a new address and protocol family within
> Linux that could be used to create and manipulate stream sockets via the
> standard interfaces (I'm focusing on Linux at this point, although
> similar adaptations could be made to other kernels).  Then, behind the
> scenes, the Xen grant tables could be used to efficiently transfer
> socket buffers between the domains.  This should involve much less
> overhead than directly connecting two network frontends or performing
> other optimizations at lower layers, since it would truncate the
> protocol stack and avoid unnecessary TCP-style flow control protocols.
> 
> An enhancement such as this could help to eliminate the network
> dependence of some Xen management applications, particularly those that
> rely on XML-RPC to communicate.  For example, xm currently uses a UNIX
> domain socket to communicate with Xend, which introduces an artificial
> requirement that xend and xm be running in the same domain.  Once XenSE
> gains traction and management utilities are scattered across multiple
> domains, UNIX domain sockets will no longer be adequate.  Under this
> scheme, stream sockets to specific domains could easily be constructed,
> without regard for the network configuration on the system.
> 
> One important detail that I haven't yet resolved is how to address
> inter-domain sockets.  Of course, the most important component in the
> address for each socket would be the domain ID.  However, some sort of
> port specification or pathname would also be necessary.  I'm not sure
> which of those options would be appropriate in this case.  Port numbers
> would be consistent with TCP and would probably ease the task of porting
> applications based on TCP, but pathnames are more consistent with the
> UNIX domain sockets used by xm and xend.  Perhaps we could provide both,
> using two address families associated with the same protocol family?
> 
> What other ideas have been floating around on how to accomplish
> byte-stream transport between domains?  Are any concrete efforts to
> provide this functionality currently underway?  Thanks!
> 
> 
> 
> 
> Ligesh wrote:
> >On Mon, Aug 07, 2006 at 04:32:03PM +0300, Muli Ben-Yehuda wrote:
> >  
> >>On Mon, Aug 07, 2006 at 07:01:30PM +0530, Ligesh wrote:
> >>Unlike openvz where all containers can run the same executables and
> >>largely share the same environment, in Xen domU's have nothing in
> >>common except being able to run on the same CPU. "Running an
> >>executable" is inherently domain-specific and thus not something that
> >>Xen should know about.
> >>
> >>    
> >
> >  It is evident that it is a bad design choice to let Xen be aware of the 
> >  internals of domU, but I would like to know if it can be implemented as 
> >  some sort of extra or something. Anyway, that is what would be the 
> >  _ideal_ system management scenario. What would be practical is some form 
> >  a message passing system, where a process running inside the domU can 
> >  get messages directly from the dom0 without the help of network.
> >
> >
> >  
> >>> Or at least we should be able to send some message to the domU
> >>> using its NAME, and not ipaddress. This would make it easy to write
> >>> scripts to automate the management, which would be difficult with
> >>> the ipaddress, since ipaddress can change at some time.
> >>>      
> >>How about a simple script that given a domain's name or domain ID will
> >>return its IP?
> >>    
> >
> > Is there any fool proof way to get an ip of the domU using its name? It 
> > is possible that the domU has been configured internally with a totally 
> > different IP from the one that has been assigned to it via the dhcp.
> >
> > Thanks for your time.
> >
> >--
> >:: Ligesh :: http://ligesh.com 
> >
> >
> >
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel
> >  
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Communicating with the domU from dom0 without Network

[Michael LeMay]

On Tue, 2006-08-08 at 16:19 +0530, Ligesh wrote:
>  Hey, that's a handful as far as I am concerned. :-)  I was more
> interested in the 'manageability' area, where I was looking for a fool
> proof method to communicate with the VM, since the networking
> identification is not a reliable mechanism to access one.

You're right, my solution is more general and complicated than what
you're talking about.  It will permit communications between multiple
DomU's, not just between Dom0 and a DomU.  Keir's proposal calls for
just such a general solution, and it will certainly handle your
scenario.  However, I understand your desire for something simple that
will work now; I just want to open up this more general discussion in
parallel to help keep the long-term picture in mind.

>  But, now I realize maybe I have been a bit remiss, and actually the
> serial port is a unique identifier for a domU, and can be used to
> communicate with it.  Inside the domU, you can have a long running
> server waiting at one of the serial ports, and from the Dom0, you can
> communicate with a domU using the serial port. Since everything is in
> software, I am sure the communication is going to be fast--i am not
> privy enough with serial communication to know if there are some real
> bottlenecks with the protocol or something, but from my informal
> approach, I think the serial port should actually work fine. Even the
> ip address can be configured remotely in this fashion.
> 

There certainly are severe limitations on serial port performance.  The
RS232 spec only goes to 20000 bps, but most PC serial ports can handle
up to 115,200 bps.  Perhaps you could modify the drivers to handle
higher data rates, but it doesn't seem like an ideal solution.  The
sockets approach is at least as familiar to Linux programmers, and
natively supports unlimited data rates.

>  I see that you are more interested in the performance, and I think
> that is a cool idea, though implementing the protocols for different
> Operating systems would be non-trivial. You will need to make kernel
> changes, and that would mean that you have to have drivers, which
> should be specifically loaded into the domU Operating system, and then
> you will have to maintain all the drivers for each OS.

Very true.  However, that's how the rest of para-virtualized Xen is
implemented.  It's a little trickier for HVM guests.  My initial
approach might involve assigning private IP addresses to each DomU that
can be addressed over the inter-domain sockets and provide an adapter
that converts between IP traffic and inter-domain traffic.  Otherwise,
it might be possible to export a special network device with its own
addressing modes to the HVM guest and provide drivers to support it.

>  Secondly, the question is how ubiquitous Xen is going to be. If every
> single computer is going to have Xen, then such a method would
> actually yeild a lot of practical benefits. I guess we have to compare
> the overheads of writing and maintaining a new protocol for MANY
> operating systems to the efficiency derived in the long term.
> 

Xen already includes many specialized drivers for use within
para-virtualized guests, and this would simply be one more for the
collection.  For those OS's that don't have the appropriate driver it
may be possible to use an IP->interdomain adapter like that discussed
above, or they could forced to rely on regular TCP networking.

>  Anyway, I am just curious. Won't an extra serial port actually solve
> all the problems that you are intending to solve using tcp like
> communication? Is there any limitation to serial port communication,
> especially since everything is happening in software?
> 

I still think that a lightweight sockets interface with specialized
addressing modes would be sufficiently useful to include within Xen, but
that's what these discussions will determine. :-)  Thanks for the
interesting counter-proposal, it does sound like it might satisfy your
short-term needs.

>  Thanks.
>  
> --
> :: Ligesh :: http://ligesh.com 
> 
> 
> 
> 
> On Mon, Aug 07, 2006 at 12:18:55PM -0400, Michael LeMay wrote:
> > This is precisely the sort of problem that the Keir's proposal seems
> to
> > address.  I've copied my comments on the proposal below; perhaps we
> can
> > discuss them further now since nobody was interested when I
> originally
> > posted them. :-)
> > 
> > ---
> > 
> > Here's another general comment for discussion...
> > 
> > The bottom of page 18 in the Xen Roadmap proposal recommends
> considering
> > how to "export byte stream
> > (TCP) data between domains in a high performance fashion."  For
> > communications that occur between domains on a single physical
> machine,
> > it would seem logical to setup a new address and protocol family
> within
> > Linux that could be used to create and manipulate stream sockets via
> the
> > standard interfaces (I'm focusing on Linux at this point, although
> > similar adaptations could be made to other kernels).  Then, behind
> the
> > scenes, the Xen grant tables could be used to efficiently transfer
> > socket buffers between the domains.  This should involve much less
> > overhead than directly connecting two network frontends or
> performing
> > other optimizations at lower layers, since it would truncate the
> > protocol stack and avoid unnecessary TCP-style flow control
> protocols.
> > 
> > An enhancement such as this could help to eliminate the network
> > dependence of some Xen management applications, particularly those
> that
> > rely on XML-RPC to communicate.  For example, xm currently uses a
> UNIX
> > domain socket to communicate with Xend, which introduces an
> artificial
> > requirement that xend and xm be running in the same domain.  Once
> XenSE
> > gains traction and management utilities are scattered across
> multiple
> > domains, UNIX domain sockets will no longer be adequate.  Under this
> > scheme, stream sockets to specific domains could easily be
> constructed,
> > without regard for the network configuration on the system.
> > 
> > One important detail that I haven't yet resolved is how to address
> > inter-domain sockets.  Of course, the most important component in
> the
> > address for each socket would be the domain ID.  However, some sort
> of
> > port specification or pathname would also be necessary.  I'm not
> sure
> > which of those options would be appropriate in this case.  Port
> numbers
> > would be consistent with TCP and would probably ease the task of
> porting
> > applications based on TCP, but pathnames are more consistent with
> the
> > UNIX domain sockets used by xm and xend.  Perhaps we could provide
> both,
> > using two address families associated with the same protocol family?
> > 
> > What other ideas have been floating around on how to accomplish
> > byte-stream transport between domains?  Are any concrete efforts to
> > provide this functionality currently underway?  Thanks!
> > 
> > 
> > 
> > 
> > Ligesh wrote:
> > >On Mon, Aug 07, 2006 at 04:32:03PM +0300, Muli Ben-Yehuda wrote:
> > >  
> > >>On Mon, Aug 07, 2006 at 07:01:30PM +0530, Ligesh wrote:
> > >>Unlike openvz where all containers can run the same executables
> and
> > >>largely share the same environment, in Xen domU's have nothing in
> > >>common except being able to run on the same CPU. "Running an
> > >>executable" is inherently domain-specific and thus not something
> that
> > >>Xen should know about.
> > >>
> > >>    
> > >
> > >  It is evident that it is a bad design choice to let Xen be aware
> of the 
> > >  internals of domU, but I would like to know if it can be
> implemented as 
> > >  some sort of extra or something. Anyway, that is what would be
> the 
> > >  _ideal_ system management scenario. What would be practical is
> some form 
> > >  a message passing system, where a process running inside the domU
> can 
> > >  get messages directly from the dom0 without the help of network.
> > >
> > >
> > >  
> > >>> Or at least we should be able to send some message to the domU
> > >>> using its NAME, and not ipaddress. This would make it easy to
> write
> > >>> scripts to automate the management, which would be difficult
> with
> > >>> the ipaddress, since ipaddress can change at some time.
> > >>>      
> > >>How about a simple script that given a domain's name or domain ID
> will
> > >>return its IP?
> > >>    
> > >
> > > Is there any fool proof way to get an ip of the domU using its
> name? It 
> > > is possible that the domU has been configured internally with a
> totally 
> > > different IP from the one that has been assigned to it via the
> dhcp.
> > >
> > > Thanks for your time.
> > >
> > >--
> > >:: Ligesh :: http://ligesh.com 
> > >
> > >
> > >
> > >
> > >_______________________________________________
> > >Xen-devel mailing list
> > >Xen-devel@lists.xensource.com
> > >http://lists.xensource.com/xen-devel
> > >  
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

Re: Communicating with the domU from dom0 without Network

[Ligesh]

On Tue, Aug 08, 2006 at 08:17:31AM -0400, Michael LeMay wrote:
> >  Anyway, I am just curious. Won't an extra serial port actually solve
> > all the problems that you are intending to solve using tcp like
> > communication? Is there any limitation to serial port communication,
> > especially since everything is happening in software?
> > 

 Actually I think serial port won't be sufficient actually. I am looking at it from purely manageability perspective, but even then, serial ports does not allow multiple simultaneious streams. I think in tcp/ip, the server gets a connection at a particular master port, say 80, and then it creates a slave socket at another port and spawns a child, and child actually carries out the communication through this slave port. I am not really an expert in networking, and that's my generic idea of how networking happens. Will such a system be possible with serial ports. I don't think so. I think I will I have to come up with some protocol to make multiple streams through serial port possible. Anyway, a generalized tcp-like communication stream independent of the networking stack looks like a good idea to me.

 Or maybe you can build a networking stack kind of thing on top of the serial port. I am very new to Xen, and I will have to first get a proper system running on it test it further.

 Thanks for your comments.

--
:: Ligesh :: http://ligesh.com 

Re: Re: Communicating with the domU from dom0 without Network

[Keir Fraser]

On 8/8/06 3:45 pm, "Ligesh" <xen@lxlabs.com> wrote:

>  Actually I think serial port won't be sufficient actually. I am looking at it
> from purely manageability perspective, but even then, serial ports does not
> allow multiple simultaneious streams. I think in tcp/ip, the server gets a
> connection at a particular master port, say 80, and then it creates a slave
> socket at another port and spawns a child, and child actually carries out the
> communication through this slave port. I am not really an expert in
> networking, and that's my generic idea of how networking happens. Will such a
> system be possible with serial ports. I don't think so. I think I will I have
> to come up with some protocol to make multiple streams through serial port
> possible. Anyway, a generalized tcp-like communication stream independent of
> the networking stack looks like a good idea to me.

That's not how TCP/IP works and a statement like 'serial ports do not allow
multiple simultaneous streams' is not true since a serial port is part of a
physical link layer so you can implement whatever protocol you like over the
top (just as you can with e.g. an Ethernet port).

 -- Keir

Re: Communicating with the domU from dom0 without Network

[Ligesh]

On Tue, Aug 08, 2006 at 03:56:56PM +0100, Keir Fraser wrote:
> 
> That's not how TCP/IP works and a statement like 'serial ports do not allow
> multiple simultaneous streams' is not true since a serial port is part of a
> physical link layer so you can implement whatever protocol you like over the
> top (just as you can with e.g. an Ethernet port).
> 

 You have to build a network stack over serial port where every packet has a header with the destination port.  You can simulate the entire tcp/ip stack over the serial port, but this is again a non-trivial effort. What I meant was out-of-box, as in, you just do an open("/dev/ttyS0", "r"), and read and write, you won't get multiple connections; you will have to ensure the co-operation of the kernel, where it will have to create virtual network ports that will allow simultaneous connections to exist.

 Anyway, ultimately my proposal is that multi-stream message passing system that is not depended on the ip address would be highly desirable. You can actually implement it over the serial port or even the virtual network card itself--by using the mac addresses to identify the machines. This would make writing system management softwares much easier. Since the future seems to point at virtualization getting ubiquitous, this almost becomes a necessary feature.

--
:: Ligesh :: http://ligesh.com 

Re: Re: Communicating with the domU from dom0 without Network

[Ian Campbell]

On Tue, 2006-08-08 at 23:45 +0530, Ligesh wrote:
>  You have to build a network stack over serial port where every packet
>  has a header with the destination port.  You can simulate the entire 
> tcp/ip stack over the serial port, but this is again a non-trivial effort.

SLIP and PPP both do this perfectly well and have done for many years.

Ian.

Re: Communicating with the domU from dom0 without Network

[Mike D. Day]

On 07/08/06 12:18 -0400, Michael LeMay wrote:
>This is precisely the sort of problem that the Keir's proposal seems to
>address.  I've copied my comments on the proposal below; perhaps we can
>discuss them further now since nobody was interested when I originally
>posted them. :-)

Sounds like RDMA, with some xen optimization. Would be good to see
this happen.

Mike

>The bottom of page 18 in the Xen Roadmap proposal recommends considering
>how to "export byte stream
>(TCP) data between domains in a high performance fashion."  For
>communications that occur between domains on a single physical machine,
>it would seem logical to setup a new address and protocol family within
>Linux that could be used to create and manipulate stream sockets via the
>standard interfaces (I'm focusing on Linux at this point, although
>similar adaptations could be made to other kernels).  Then, behind the
>scenes, the Xen grant tables could be used to efficiently transfer
>socket buffers between the domains.  This should involve much less
>overhead than directly connecting two network frontends or performing
>other optimizations at lower layers, since it would truncate the
>protocol stack and avoid unnecessary TCP-style flow control protocols.

Re: Re: Communicating with the domU from dom0 without Network

[Daniel Stodden]

[-- Attachment #1.1: Type: text/plain, Size: 8600 bytes --]

On Mon, 2006-08-07 at 12:18 -0400, Michael LeMay wrote:
> This is precisely the sort of problem that the Keir's proposal seems to
> address.  I've copied my comments on the proposal below; perhaps we can
> discuss them further now since nobody was interested when I originally
> posted them. :-)
> 
> ---
> 
> Here's another general comment for discussion...
> 
> The bottom of page 18 in the Xen Roadmap proposal recommends considering
> how to "export byte stream
> (TCP) data between domains in a high performance fashion."  For
> communications that occur between domains on a single physical machine,
> it would seem logical to setup a new address and protocol family within
> Linux that could be used to create and manipulate stream sockets via the
> standard interfaces (I'm focusing on Linux at this point, although
> similar adaptations could be made to other kernels).  Then, behind the
> scenes, the Xen grant tables could be used to efficiently transfer
> socket buffers between the domains.  This should involve much less
> overhead than directly connecting two network frontends or performing
> other optimizations at lower layers, since it would truncate the
> protocol stack and avoid unnecessary TCP-style flow control protocols.
> 
> An enhancement such as this could help to eliminate the network
> dependence of some Xen management applications, particularly those that
> rely on XML-RPC to communicate.  For example, xm currently uses a UNIX
> domain socket to communicate with Xend, which introduces an artificial
> requirement that xend and xm be running in the same domain.  Once XenSE
> gains traction and management utilities are scattered across multiple
> domains, UNIX domain sockets will no longer be adequate.  Under this
> scheme, stream sockets to specific domains could easily be constructed,
> without regard for the network configuration on the system.
> 
> One important detail that I haven't yet resolved is how to address
> inter-domain sockets.  Of course, the most important component in the
> address for each socket would be the domain ID.  However, some sort of
> port specification or pathname would also be necessary.  I'm not sure
> which of those options would be appropriate in this case.  Port numbers
> would be consistent with TCP and would probably ease the task of porting
> applications based on TCP, but pathnames are more consistent with the
> UNIX domain sockets used by xm and xend.  Perhaps we could provide both,
> using two address families associated with the same protocol family?
> 
> What other ideas have been floating around on how to accomplish
> byte-stream transport between domains?  Are any concrete efforts to
> provide this functionality currently underway?  Thanks!


hi all.

since you've explicitly asked for comments, here's mine.

from a performance point of view, it is all obviously correct. get rid
of the tcp congestion/flow/reliability overhead. in a synchronous,
reliable environment like host-local domain intercommunication
infrastructure, as you propose, it is nothing but overhead, and should
speed up things a lot. plus it saves a whole bunch of memory.

but there's a different point of view, which i would like to point out.

if you think about the whole 'virtualization' thing, some of the
relevant literature is correct to point out that simple unix process is
nothing but a virtual machine. a 'process vm', in many respects quite
different from a system vm on top of a hypervisor, like xen, though it
already has a number of features which make up a virtual machine.
resource control and abstraction, as an example, being the most
prominent ones. such comparisons are especially daunting if you look at
a paravirtualizing, microkernel-style hypervisor design, like xen is
one.

so, if operating systems and hypervisors are already so similar, where's
the merit? one of the major features which make many system VMMs,
carrying whole systems, so interesting and different from a unix-style
operating system, carrying a number simple processes, is proper
isolation. 'isolation' here means separation of an entity, here the
guest os instance, from its environment. currently, there is only a few
communication primitives connecting a guest from its supporting
processing environment. apart from vcpu-state, it's block I/O, network
I/O, memory aquisition. a small number of interfaces, each of them on a
sufficiently abstract level to enable one of the most distinguishing
features (compared with conventional os-level processing models) a
system vm has to offer: migrateablity. if the communication primitives
remain simple, and more important, location-indepent enough, you can
just freeze the whole thing on one place, move it around, and thaw
processing state at whatever different location you see fit. for xen as
of today, complexity of implementation varies somewhere between
'trivial' and 'easy enough'.

now try the same thing with your regular unix process. let's see what we
need to carry: system v ipc. shared memory. ip sockets, ok, but then
unix domain sockets, netlink sockets. pipes. named pipes. device special
files. for starters, just migrate open files terminating in block
storage. then try maintaining original process identifiers, your
application may have inquired about them and computational state
therefore depends on their consistency as well. save all that, migrate,
now try to restore elsewhere. the bottom line is: unix processes are
anything but isolated. for good reason, a lot of useful applications
depend on on inter-process communication. but that lack of isolation has
its cost.

what the proposal above means is basically addition of dedicated ipc to
the domain model. good for performance, but also a good step towards
breaking isolation. dom3 may call connect(socket(PF_XEN), "dom7") in
future. does that mean if i move dom3 to a backup node, dom7 has to move
as well? no not desirable, ok, let's write a proxy service redirecting
the once so efficient channel over tcp back to maintain transparency,
then. that's just a few additional lines of code. then don't forget
those few additional additional code lines telling the domain controller
to automatically reroot that proxy as well, in case either domain needs
to remigrate at a later point because the machine maintainer cleary
doesn't want to have to care.
yes, still it's all 'virtually' possible. after all, computers are state
machines, and state could always be captured. it just turns out to be a
whole lot of work if the number of state machines connecting your vm to
its environment keeps morphing and multiplying. operating systems are a
moving target, and probably always stay so. that's why hypervisors make
sense, as long as they stay simple and reasonably nonintrusive to the
guest. that is why os virtualization like openvz may be doomed. if they
don't make it into the stock kernel, so others help to maintain their
code, they'll keep maintaining on a pretty regular basis, until
infinity. xen does as well due to paravitualization, but not as much as
the vmm/os-integrated approach. i even suggest a term for this class of
proposal: "overparavirtualization". the point where you modified the
guest so deeply that you need someone else to maintain the patches.

apart from that, i'm all for performance.
there's a compromise: add those features, but take good care to separate
them from the isolated, network-transparent, preferably IP-based,
regular standard guest state. never make such a thing a dependency of
anything. most users of vm technology are better off rejecting it, if
they wish to keep the features distinguishing the result from a standard
operating system environment.

make it absolutely clear to users, that if they wish to configure fast
host-local inter-domain-communications, they get what they desire: fast,
but host-local domain-interdependent communications. if your customer
asks for light-weight optimized inter-domain-communication, ask her if
that specific application would not rather demand for regular
inter-process-communications on a standard operating system, because
that's what they get then. 

kind regards,
daniel

-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Communicating with the domU from dom0 without Network

[Harry Butterworth]

On Tue, 2006-08-08 at 22:54 +0200, Daniel Stodden wrote:

> make it absolutely clear to users, that if they wish to configure fast
> host-local inter-domain-communications, they get what they desire: fast,
> but host-local domain-interdependent communications. if your customer
> asks for light-weight optimized inter-domain-communication, ask her if
> that specific application would not rather demand for regular
> inter-process-communications on a standard operating system, because
> that's what they get then. 

The xenidc code that I wrote was an efficient interdomain transport
which was compatible with a network transparent implementation.

Harry.

Re: Re: Communicating with the domU from dom0 without Network

[Daniel Stodden]

[-- Attachment #1.1: Type: text/plain, Size: 1288 bytes --]

On Wed, 2006-08-09 at 00:04 +0100, Harry Butterworth wrote:
> On Tue, 2006-08-08 at 22:54 +0200, Daniel Stodden wrote:
> 
> > make it absolutely clear to users, that if they wish to configure fast
> > host-local inter-domain-communications, they get what they desire: fast,
> > but host-local domain-interdependent communications. if your customer
> > asks for light-weight optimized inter-domain-communication, ask her if
> > that specific application would not rather demand for regular
> > inter-process-communications on a standard operating system, because
> > that's what they get then. 
> 
> The xenidc code that I wrote was an efficient interdomain transport
> which was compatible with a network transparent implementation.

i would like to see the code. is it available somewhere? i haven't been
following the list too closely in the past.

still not sure whether we are we talking about the same thing when
saying "network transparent".

thanks,
daniel

-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Communicating with the domU from dom0 without Network

[Ligesh]

 Your arguments are valid in the case of domU-domU communication system, but not for dom0-domU. The dom0 will be, and should be aware of migrations and also there should be non-networking mechanism to manage and control every aspect of domU from dom0. This in my opinion, is a basic requirement.  But I think ppp over serial port might be the exact thing what I was looking for. PPP doesn't provide domU-domU, but yes domU-domU actually defeats the entire purpose of Xen which is isolation, and also that Xen should be agnostic about the type of the OS running inside the domU.

--
:: Ligesh :: http://ligesh.com 



On Tue, Aug 08, 2006 at 10:54:32PM +0200, Daniel Stodden wrote:
> hi all.
> 
> since you've explicitly asked for comments, here's mine.
> 
> from a performance point of view, it is all obviously correct. get rid
> of the tcp congestion/flow/reliability overhead. in a synchronous,
> reliable environment like host-local domain intercommunication
> infrastructure, as you propose, it is nothing but overhead, and should
> speed up things a lot. plus it saves a whole bunch of memory.
> 
> but there's a different point of view, which i would like to point out.
> 
> if you think about the whole 'virtualization' thing, some of the
> relevant literature is correct to point out that simple unix process is
> nothing but a virtual machine. a 'process vm', in many respects quite
> different from a system vm on top of a hypervisor, like xen, though it
> already has a number of features which make up a virtual machine.
> resource control and abstraction, as an example, being the most
> prominent ones. such comparisons are especially daunting if you look at
> a paravirtualizing, microkernel-style hypervisor design, like xen is
> one.
> 

Re: Communicating with the domU from dom0 without Network

[Daniel Stodden]

[-- Attachment #1.1: Type: text/plain, Size: 2279 bytes --]

On Wed, 2006-08-09 at 05:17 +0530, Ligesh wrote:
>  Your arguments are valid in the case of domU-domU communication system, but not for dom0-domU. The dom0 will be, and should be aware of migrations and also there should be non-networking mechanism to manage and control every aspect of domU from dom0. This in my opinion, is a basic requirement.  But I think ppp over serial port might be the exact thing what I was looking for. PPP doesn't provide domU-domU, but yes domU-domU actually defeats the entire purpose of Xen which is isolation, and also that Xen should be agnostic about the type of the OS running inside the domU.

hi ligesh.

i agree that dom0 may be considered special. regarding transparent
migration: flattening the network stack remains at the expense of
migrating or proxying the dom0-internal channel endpoint together with
the domain, right? together with either migrating or proxying whatever
entity/ies were connected to that endpoint. remember, if you want the
relocation to remain transparent, you'll need to guarantee that a
presumably stateful connection to whatever connected to domU remains
persistent. simple, if it's running on top of an IP layer hosted by a
virtual LAN, the former located _inside_ domU. that state gets
transferred as well, and IP routing of the LAN makes the relocation
transparent. if you lose that, you'll need to rebuild this feature
externally from domU, and that's a lot of complicated work and makes
your management infrastructure a whole more complex.

again, i totally agree that there's a lot to be gained from lightweight
host-local packet streams. but it will remain coming at a cost whenever
applied to domU, and the fun part is that what you originally wanted to
achieve, simplyfing domain management by short-circuiting the domU
access, will ultimately turn into something way more adventurous than
the IP-based approach as soon as migration comes into play.

kind regard,
daniel


-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Communicating with the domU from dom0 without Network

[Daniel Stodden]

[-- Attachment #1.1: Type: text/plain, Size: 1950 bytes --]

On Wed, 2006-08-09 at 14:09 +0200, Daniel Stodden wrote:
> On Wed, 2006-08-09 at 05:17 +0530, Ligesh wrote:
> >  Your arguments are valid in the case of domU-domU communication system, but not for dom0-domU. The dom0 will be, and should be aware of migrations and also there should be non-networking mechanism to manage and control every aspect of domU from dom0. This in my opinion, is a basic requirement.  But I think ppp over serial port might be the exact thing what I was looking for. PPP doesn't provide domU-domU, but yes domU-domU actually defeats the entire purpose of Xen which is isolation, and also that Xen should be agnostic about the type of the OS running inside the domU.

> again, i totally agree that there's a lot to be gained from lightweight
> host-local packet streams. 

addendum: the problem may just be that different people expect different
things from xen. once you're running non-server systems, e.g.
integrating a windows guest with your otherwise xen/linux-driven host
for desktop applications, migration gets less interesting. and that is
probably where i'm all for a flat alternative network stack as well.

the whole posting was just meant to make people keeping the stateful
connection migration thing in mind when discussing inter-domain
communications.

> but it will remain coming at a cost whenever
> applied to domU, and the fun part is that what you originally wanted to
> achieve, simplyfing domain management by short-circuiting the domU
> access, will ultimately turn into something way more adventurous than
> the IP-based approach as soon as migration comes into play.

daniel

-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel