Guest to Host communication

Guest to Host communication

[Jose A. Lopes]

Hi,

My name is Jose and I work at Google on the Ganeti project.
I am using Xen 4.0.

I would like to ask your advice on what you think is the best way to
send data between the guest and the host, for example, the guest OS
tells Ganeti that it wants to be reinstalled at reboot time.  Even a
simple integer suffices for our purposes.

We have considered adding a new device, such as, a char/block device.
For example, we could add an SSD device that is backed by a file.  We
have also considered a separate console.

What do you think is the preferred approach to this problem?

Thanks,
Jose

-- 
Jose Antonio Lopes
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370

Re: Guest to Host communication

[Paul Durrant]

> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> bounces@lists.xen.org] On Behalf Of Jose A. Lopes
> Sent: 22 October 2013 09:00
> To: xen-devel@lists.xenproject.org
> Subject: [Xen-devel] Guest to Host communication
> 
> Hi,
> 
> My name is Jose and I work at Google on the Ganeti project.
> I am using Xen 4.0.
> 
> I would like to ask your advice on what you think is the best way to
> send data between the guest and the host, for example, the guest OS
> tells Ganeti that it wants to be reinstalled at reboot time.  Even a
> simple integer suffices for our purposes.
> 
> We have considered adding a new device, such as, a char/block device.
> For example, we could add an SSD device that is backed by a file.  We
> have also considered a separate console.
> 
> What do you think is the preferred approach to this problem?
> 

If you're talking about a small amount of data and only need a low bandwidth channel then xenstore is probably what you want.

  Paul

Re: Guest to Host communication

[Jose A. Lopes]

Hi,

If I understood correctly, Xenstore requires writing a driver and
loading it inside the VM.  Is this correct?

If so, then it seems this approach would require writing different
drivers for different OSes, such as, Linux and Windows.  Currently, we
are just exploring different design options and we would like to aim
at a uniform approach across different OSes. Is there an option like that ?

Thank you,
Jose

On Tue, Oct 22, 2013 at 08:48:24AM +0000, Paul Durrant wrote:
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> > bounces@lists.xen.org] On Behalf Of Jose A. Lopes
> > Sent: 22 October 2013 09:00
> > To: xen-devel@lists.xenproject.org
> > Subject: [Xen-devel] Guest to Host communication
> > 
> > Hi,
> > 
> > My name is Jose and I work at Google on the Ganeti project.
> > I am using Xen 4.0.
> > 
> > I would like to ask your advice on what you think is the best way to
> > send data between the guest and the host, for example, the guest OS
> > tells Ganeti that it wants to be reinstalled at reboot time.  Even a
> > simple integer suffices for our purposes.
> > 
> > We have considered adding a new device, such as, a char/block device.
> > For example, we could add an SSD device that is backed by a file.  We
> > have also considered a separate console.
> > 
> > What do you think is the preferred approach to this problem?
> > 
> 
> If you're talking about a small amount of data and only need a low bandwidth channel then xenstore is probably what you want.
> 
>   Paul

-- 
Jose Antonio Lopes
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370

Re: Guest to Host communication

[Paul Durrant]

> -----Original Message-----
> From: Jose A. Lopes [mailto:jabolopes@google.com]
> Sent: 22 October 2013 13:49
> To: Paul Durrant
> Cc: xen-devel@lists.xenproject.org
> Subject: Re: [Xen-devel] Guest to Host communication
> 
> Hi,
> 
> If I understood correctly, Xenstore requires writing a driver and
> loading it inside the VM.  Is this correct?
> 

Well, yes you'll need some code in the guest, but drivers already exist for linux and windows so you could just use them.

> If so, then it seems this approach would require writing different
> drivers for different OSes, such as, Linux and Windows.  Currently, we
> are just exploring different design options and we would like to aim
> at a uniform approach across different OSes. Is there an option like that ?
> 

What option do you expect that doesn't involve writing at least some code for each OS you want to use?

  Paul

Re: Guest to Host communication

[Jose A. Lopes]

On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:
> > -----Original Message-----
> > From: Jose A. Lopes [mailto:jabolopes@google.com]
> > Sent: 22 October 2013 13:49
> > To: Paul Durrant
> > Cc: xen-devel@lists.xenproject.org
> > Subject: Re: [Xen-devel] Guest to Host communication
> > 
> > Hi,
> > 
> > If I understood correctly, Xenstore requires writing a driver and
> > loading it inside the VM.  Is this correct?
> > 
> 
> Well, yes you'll need some code in the guest, but drivers already exist for linux and windows so you could just use them.
> 
> > If so, then it seems this approach would require writing different
> > drivers for different OSes, such as, Linux and Windows.  Currently, we
> > are just exploring different design options and we would like to aim
> > at a uniform approach across different OSes. Is there an option like that ?
> > 
> 
> What option do you expect that doesn't involve writing at least some code for each OS you want to use?

We were thinking of simply attaching a device.
For example, add a SSD disk in DomU which is backed in Dom0 by a file.
Is this possible?

> 
>   Paul

-- 
Jose Antonio Lopes
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370

Re: Guest to Host communication

[George Dunlap]

On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com> wrote:
> On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:
>> > -----Original Message-----
>> > From: Jose A. Lopes [mailto:jabolopes@google.com]
>> > Sent: 22 October 2013 13:49
>> > To: Paul Durrant
>> > Cc: xen-devel@lists.xenproject.org
>> > Subject: Re: [Xen-devel] Guest to Host communication
>> >
>> > Hi,
>> >
>> > If I understood correctly, Xenstore requires writing a driver and
>> > loading it inside the VM.  Is this correct?
>> >
>>
>> Well, yes you'll need some code in the guest, but drivers already exist for linux and windows so you could just use them.
>>
>> > If so, then it seems this approach would require writing different
>> > drivers for different OSes, such as, Linux and Windows.  Currently, we
>> > are just exploring different design options and we would like to aim
>> > at a uniform approach across different OSes. Is there an option like that ?
>> >
>>
>> What option do you expect that doesn't involve writing at least some code for each OS you want to use?
>
> We were thinking of simply attaching a device.
> For example, add a SSD disk in DomU which is backed in Dom0 by a file.
> Is this possible?

Of course you can attach as many disks as you want; or you could just
have dom0 look for a file in the primary filesystem.  But whether you
use the primary filesystem or a secondary one, you still have the
basic problem of introspection: if this is going to be a filesystem,
then there may be a difference at any given time between what domU
thinks the state of the filesystem is, and what dom0 can see on the
disk (because there may be things in domU's buffer cache which haven't
"hit the disk" yet).

I suppose one option would be to make a block device with "magic" --
i.e., "write to block 0 for this bit of info, block 2 for this bit".
But the way that you write specific blocks is different for each OS
anyway -- you're essentially writing a driver for this "magic block
device" anyway; *and* you have to write new code in dom0 to look at
this device and notice changes to it, which will be non-trivial.

Fortunately, there already is a well-defined way of doing this which
already has drivers and libraries for all major operating systems
(Linux, Windows, NetBSD, FreeBSD, probably even Gnu Hurd): xenstore.
It already has "watch" functionality, so that a program in dom0 can
say, "Tell me when this key {....} is created or changed" and then
contentedly go to sleep, secure in the knowledge that the well-tested
data path will wake it up when the guest writes to it.  It's almost
certainly your best solution.

 -George

Re: Guest to Host communication

[Jose A. Lopes]

On Thu, Oct 31, 2013 at 02:21:38PM +0000, George Dunlap wrote:
> On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com> wrote:
> > On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:
> >> > -----Original Message-----
> >> > From: Jose A. Lopes [mailto:jabolopes@google.com]
> >> > Sent: 22 October 2013 13:49
> >> > To: Paul Durrant
> >> > Cc: xen-devel@lists.xenproject.org
> >> > Subject: Re: [Xen-devel] Guest to Host communication
> >> >
> >> > Hi,
> >> >
> >> > If I understood correctly, Xenstore requires writing a driver and
> >> > loading it inside the VM.  Is this correct?
> >> >
> >>
> >> Well, yes you'll need some code in the guest, but drivers already exist for linux and windows so you could just use them.
> >>
> >> > If so, then it seems this approach would require writing different
> >> > drivers for different OSes, such as, Linux and Windows.  Currently, we
> >> > are just exploring different design options and we would like to aim
> >> > at a uniform approach across different OSes. Is there an option like that ?
> >> >
> >>
> >> What option do you expect that doesn't involve writing at least some code for each OS you want to use?
> >
> > We were thinking of simply attaching a device.
> > For example, add a SSD disk in DomU which is backed in Dom0 by a file.
> > Is this possible?
> 
> Of course you can attach as many disks as you want; or you could just
> have dom0 look for a file in the primary filesystem.  But whether you
> use the primary filesystem or a secondary one, you still have the
> basic problem of introspection: if this is going to be a filesystem,
> then there may be a difference at any given time between what domU
> thinks the state of the filesystem is, and what dom0 can see on the
> disk (because there may be things in domU's buffer cache which haven't
> "hit the disk" yet).

We have changed our use case to the situation where the instance
writes a few bytes to this device.  However, Ganeti only needs this
information once the instance has stopped.  Therefore, while the
instance is running it can write and overwrite the information because
it is not important.

I wonder if it is possible to change the caching policy of the device
in order to immediately flush the writes.  This is not so important
because I imagine that by the time the instance has stopped (if it
didn't crash), the cache should have been flushed.

We were however thinking about a usb serial device to make it easier
to do successive writes within the instance.

Jose

> 
> I suppose one option would be to make a block device with "magic" --
> i.e., "write to block 0 for this bit of info, block 2 for this bit".
> But the way that you write specific blocks is different for each OS
> anyway -- you're essentially writing a driver for this "magic block
> device" anyway; *and* you have to write new code in dom0 to look at
> this device and notice changes to it, which will be non-trivial.
> 
> Fortunately, there already is a well-defined way of doing this which
> already has drivers and libraries for all major operating systems
> (Linux, Windows, NetBSD, FreeBSD, probably even Gnu Hurd): xenstore.
> It already has "watch" functionality, so that a program in dom0 can
> say, "Tell me when this key {....} is created or changed" and then
> contentedly go to sleep, secure in the knowledge that the well-tested
> data path will wake it up when the guest writes to it.  It's almost
> certainly your best solution.
> 
>  -George

-- 
Jose Antonio Lopes
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370

Re: Guest to Host communication

[George Dunlap]

On 31/10/13 16:08, Jose A. Lopes wrote:
> On Thu, Oct 31, 2013 at 02:21:38PM +0000, George Dunlap wrote:
>> On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com> wrote:
>>> On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:
>>>>> -----Original Message-----
>>>>> From: Jose A. Lopes [mailto:jabolopes@google.com]
>>>>> Sent: 22 October 2013 13:49
>>>>> To: Paul Durrant
>>>>> Cc: xen-devel@lists.xenproject.org
>>>>> Subject: Re: [Xen-devel] Guest to Host communication
>>>>>
>>>>> Hi,
>>>>>
>>>>> If I understood correctly, Xenstore requires writing a driver and
>>>>> loading it inside the VM.  Is this correct?
>>>>>
>>>> Well, yes you'll need some code in the guest, but drivers already exist for linux and windows so you could just use them.
>>>>
>>>>> If so, then it seems this approach would require writing different
>>>>> drivers for different OSes, such as, Linux and Windows.  Currently, we
>>>>> are just exploring different design options and we would like to aim
>>>>> at a uniform approach across different OSes. Is there an option like that ?
>>>>>
>>>> What option do you expect that doesn't involve writing at least some code for each OS you want to use?
>>> We were thinking of simply attaching a device.
>>> For example, add a SSD disk in DomU which is backed in Dom0 by a file.
>>> Is this possible?
>> Of course you can attach as many disks as you want; or you could just
>> have dom0 look for a file in the primary filesystem.  But whether you
>> use the primary filesystem or a secondary one, you still have the
>> basic problem of introspection: if this is going to be a filesystem,
>> then there may be a difference at any given time between what domU
>> thinks the state of the filesystem is, and what dom0 can see on the
>> disk (because there may be things in domU's buffer cache which haven't
>> "hit the disk" yet).
> We have changed our use case to the situation where the instance
> writes a few bytes to this device.  However, Ganeti only needs this
> information once the instance has stopped.  Therefore, while the
> instance is running it can write and overwrite the information because
> it is not important.
>
> I wonder if it is possible to change the caching policy of the device
> in order to immediately flush the writes.  This is not so important
> because I imagine that by the time the instance has stopped (if it
> didn't crash), the cache should have been flushed.
>
> We were however thinking about a usb serial device to make it easier
> to do successive writes within the instance.

Oh, right -- in that case, probably the easiest thing to do would be to 
just create a file in the guest filesystem and have your tools look 
inside to see if it's there.  You could have a secondary disk just for 
that purpose, if that makes it easier to make it the same across instances.

  -George

Re: Guest to Host communication

[Jose A. Lopes]

[-- Attachment #1.1: Type: text/plain, Size: 3260 bytes --]

On Thu, Oct 31, 2013 at 5:13 PM, George Dunlap
<george.dunlap@eu.citrix.com>wrote:

> On 31/10/13 16:08, Jose A. Lopes wrote:
>
>> On Thu, Oct 31, 2013 at 02:21:38PM +0000, George Dunlap wrote:
>>
>>> On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com>
>>> wrote:
>>>
>>>> On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:
>>>>
>>>>> -----Original Message-----
>>>>>> From: Jose A. Lopes [mailto:jabolopes@google.com]
>>>>>> Sent: 22 October 2013 13:49
>>>>>> To: Paul Durrant
>>>>>> Cc: xen-devel@lists.xenproject.org
>>>>>> Subject: Re: [Xen-devel] Guest to Host communication
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> If I understood correctly, Xenstore requires writing a driver and
>>>>>> loading it inside the VM.  Is this correct?
>>>>>>
>>>>>>  Well, yes you'll need some code in the guest, but drivers already
>>>>> exist for linux and windows so you could just use them.
>>>>>
>>>>>  If so, then it seems this approach would require writing different
>>>>>> drivers for different OSes, such as, Linux and Windows.  Currently, we
>>>>>> are just exploring different design options and we would like to aim
>>>>>> at a uniform approach across different OSes. Is there an option like
>>>>>> that ?
>>>>>>
>>>>>>  What option do you expect that doesn't involve writing at least some
>>>>> code for each OS you want to use?
>>>>>
>>>> We were thinking of simply attaching a device.
>>>> For example, add a SSD disk in DomU which is backed in Dom0 by a file.
>>>> Is this possible?
>>>>
>>> Of course you can attach as many disks as you want; or you could just
>>> have dom0 look for a file in the primary filesystem.  But whether you
>>> use the primary filesystem or a secondary one, you still have the
>>> basic problem of introspection: if this is going to be a filesystem,
>>> then there may be a difference at any given time between what domU
>>> thinks the state of the filesystem is, and what dom0 can see on the
>>> disk (because there may be things in domU's buffer cache which haven't
>>> "hit the disk" yet).
>>>
>> We have changed our use case to the situation where the instance
>> writes a few bytes to this device.  However, Ganeti only needs this
>> information once the instance has stopped.  Therefore, while the
>> instance is running it can write and overwrite the information because
>> it is not important.
>>
>> I wonder if it is possible to change the caching policy of the device
>> in order to immediately flush the writes.  This is not so important
>> because I imagine that by the time the instance has stopped (if it
>> didn't crash), the cache should have been flushed.
>>
>> We were however thinking about a usb serial device to make it easier
>> to do successive writes within the instance.
>>
>
> Oh, right -- in that case, probably the easiest thing to do would be to
> just create a file in the guest filesystem and have your tools look inside
> to see if it's there.  You could have a secondary disk just for that
> purpose, if that makes it easier to make it the same across instances.
>
>
When you say "look inside" the filesystem do you mean to mount that
filesystem
in the host OS? If so, it seems that it is very dangerous to mount guest
filesystems
due to a number of exploits.


>  -George
>

[-- Attachment #1.2: Type: text/html, Size: 4813 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Guest to Host communication

[Ian Campbell]

On Thu, 2013-10-31 at 19:42 +0100, Jose A. Lopes wrote:

> When you say "look inside" the filesystem do you mean to mount that
> filesystem
> in the host OS? If so, it seems that it is very dangerous to mount
> guest filesystems
> due to a number of exploits.

If you only need r/o access from the host then you could use
tools/libfsimage (which has python binding, used by pygrub) which at
least constrains things to a userspace process and not a kernel mode
exploit. With suitable privilege dropping this can be made reasonably
safe...

Ian.

Re: Guest to Host communication

[Jose A. Lopes]

Hi,

Thanks, we'll have a look at that.

Regards,
Jose

On Thu, Oct 31, 2013 at 08:02:29PM +0000, Ian Campbell wrote:
> On Thu, 2013-10-31 at 19:42 +0100, Jose A. Lopes wrote:
> 
> > When you say "look inside" the filesystem do you mean to mount that
> > filesystem
> > in the host OS? If so, it seems that it is very dangerous to mount
> > guest filesystems
> > due to a number of exploits.
> 
> If you only need r/o access from the host then you could use
> tools/libfsimage (which has python binding, used by pygrub) which at
> least constrains things to a userspace process and not a kernel mode
> exploit. With suitable privilege dropping this can be made reasonably
> safe...
> 
> Ian.
> 
> 

-- 
Jose Antonio Lopes
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370

Re: Guest to Host communication

[Konrad Rzeszutek Wilk]

On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:
> > -----Original Message-----
> > From: Jose A. Lopes [mailto:jabolopes@google.com]
> > Sent: 22 October 2013 13:49
> > To: Paul Durrant
> > Cc: xen-devel@lists.xenproject.org
> > Subject: Re: [Xen-devel] Guest to Host communication
> > 
> > Hi,
> > 
> > If I understood correctly, Xenstore requires writing a driver and
> > loading it inside the VM.  Is this correct?
> > 
> 
> Well, yes you'll need some code in the guest, but drivers already exist for linux and windows so you could just use them.

So, Microsoft came up with this kvp (key value) driver that can
communicate with the host. The guest only needs to use a socket
interface to talk - and best of all it is in the upstream kernel.

The driver (user-space) is this one: http://lxr.free-electrons.com/source/tools/hv/hv_kvp_daemon.c
and it listens to the socket. The kernel driver gets the
key/values from the host and it ends up pushing them to the
daemon. The daemon does whatever it needs to do with them
(say, alter the IP of the guest, return the OS info back and
so on).

It should be possible to make the kernel driver use a
XenBus instead of a VMBus and piggyback on this.

> 
> > If so, then it seems this approach would require writing different
> > drivers for different OSes, such as, Linux and Windows.  Currently, we

The neat thing is that it would also work out of the box on Windows (I hope).

> > are just exploring different design options and we would like to aim
> > at a uniform approach across different OSes. Is there an option like that ?
> > 
> 
> What option do you expect that doesn't involve writing at least some code for each OS you want to use?
> 
>   Paul
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel