questions regarding interdomain communication and xen-guest communication

questions regarding interdomain communication and xen-guest communication

[Aastha Mehta]

Hello,

I would like to clarify certain things about the communication
mechanisms between guests and between xen and guest domains.

If there is an interdomain event channel set up between two guests, do
they communicate directly with each other, bypassing the hypervisor
completely? In case of an asynchronous I/O does one guest have to
register its callbacks with the other guest directly?
Or does one guest hypercall into xen, which then upcalls into the
other guest and then transfer reply in the reverse using another pair
of hypercall and upcall? Can event channels be set up ad hoc between
two, for example, dom0 and a guest domain? Or can they only be used by
drivers, e.g. between network frontend and backend drivers?

Regarding xen -- domain communication: how can xen send some request
to dom0 for some operation that needs to be performed in the dom0 and
get back results from the dom0? Does such a communication have to be
synchronous?

Thanks,
Aastha.

-- 
Aastha Mehta
MPI-SWS, Germany
E-mail: aasthakm@mpi-sws.org

Re: questions regarding interdomain communication and xen-guest communication

[Jan Beulich]

>>> Aastha Mehta <aasthakm@gmail.com> 10/23/13 7:44 PM >>>
>If there is an interdomain event channel set up between two guests, do
>they communicate directly with each other, bypassing the hypervisor
>completely?

No, event channel delivery works via the hypervisor.

> In case of an asynchronous I/O does one guest have to
>register its callbacks with the other guest directly?
>Or does one guest hypercall into xen, which then upcalls into the
>other guest and then transfer reply in the reverse using another pair
>of hypercall and upcall?

Again, the latter.

> Can event channels be set up ad hoc between
>two, for example, dom0 and a guest domain? Or can they only be used by
>drivers, e.g. between network frontend and backend drivers?

The former.

>Regarding xen -- domain communication: how can xen send some request
>to dom0 for some operation that needs to be performed in the dom0 and
>get back results from the dom0?

Look at the MCE polling Dom0 performs for example, or the memory pressure
threshold handling: One model would be for Dom0 to set up a (dedicated)
vIRQ for that purpose, and Xen trigger that vIRQ when it needs Dom0's help.

> Does such a communication have to be synchronous?

I would say no, unless your use case has such a requirement.

Jan

Re: questions regarding interdomain communication and xen-guest communication

[Aastha Mehta]

On 24 October 2013 09:09, Jan Beulich <jbeulich@suse.com> wrote:
>>>> Aastha Mehta <aasthakm@gmail.com> 10/23/13 7:44 PM >>>
>>If there is an interdomain event channel set up between two guests, do
>>they communicate directly with each other, bypassing the hypervisor
>>completely?
>
> No, event channel delivery works via the hypervisor.
>
>> In case of an asynchronous I/O does one guest have to
>>register its callbacks with the other guest directly?
>>Or does one guest hypercall into xen, which then upcalls into the
>>other guest and then transfer reply in the reverse using another pair
>>of hypercall and upcall?
>
> Again, the latter.

So when one guest (G1) shares pages with second (G2), for example for
block I/O, G1 will inform second via xen about a shared page. G1 will
also register a callback with Xen. When Xen upcalls into G2 to forward
G1's request, does it block on G2's reponse? How does G2 send back the
result of I/O to G1?

>
>> Can event channels be set up ad hoc between
>>two, for example, dom0 and a guest domain? Or can they only be used by
>>drivers, e.g. between network frontend and backend drivers?
>
> The former.
>
>>Regarding xen -- domain communication: how can xen send some request
>>to dom0 for some operation that needs to be performed in the dom0 and
>>get back results from the dom0?
>
> Look at the MCE polling Dom0 performs for example, or the memory pressure
> threshold handling: One model would be for Dom0 to set up a (dedicated)
> vIRQ for that purpose, and Xen trigger that vIRQ when it needs Dom0's help.
>

Just to clarify, will guest return the result of the operation using a
hypercall?

>> Does such a communication have to be synchronous?
>
> I would say no, unless your use case has such a requirement.
>
> Jan
>

Thanks for the response.


-- 
Aastha Mehta
MPI-SWS, Germany
E-mail: aasthakm@mpi-sws.org

Re: questions regarding interdomain communication and xen-guest communication

[Jan Beulich]

>>> Aastha Mehta <aasthakm@gmail.com> 10/24/13 7:32 PM >>>
>On 24 October 2013 09:09, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> Aastha Mehta <aasthakm@gmail.com> 10/23/13 7:44 PM >>>
>>> In case of an asynchronous I/O does one guest have to
>>>register its callbacks with the other guest directly?
>>>Or does one guest hypercall into xen, which then upcalls into the
>>>other guest and then transfer reply in the reverse using another pair
>>>of hypercall and upcall?
>>
>> Again, the latter.
>
>So when one guest (G1) shares pages with second (G2), for example for
>block I/O, G1 will inform second via xen about a shared page.

How this information is propagated is up to you; commonly this is being
done via xenstore.

> G1 will also register a callback with Xen.

That callback will need to be registered in any case - the (general)
event upcall.

> When Xen upcalls into G2 to forward G1's request, does it block on G2's reponse?

Why would it - it acts as a passive agent here only.

> How does G2 send back the result of I/O to G1?

Generally as contents of the shared page.

But really - please take a look at the various existing backend/frontend
drivers.

>>>Regarding xen -- domain communication: how can xen send some request
>>>to dom0 for some operation that needs to be performed in the dom0 and
>>>get back results from the dom0?
>>
>> Look at the MCE polling Dom0 performs for example, or the memory pressure
>> threshold handling: One model would be for Dom0 to set up a (dedicated)
>> vIRQ for that purpose, and Xen trigger that vIRQ when it needs Dom0's help.
>
>Just to clarify, will guest return the result of the operation using a
>hypercall?

First of all there's generally no "result" in the examples I gave. Instead,
Xen simply requests a certain kind of action to be taken. And Xen in fact
must not depend on whether or when that action is being taken. But in
general it might well be that the action requested consists of one or more
hypercalls.

Jan

Re: questions regarding interdomain communication and xen-guest communication

[Aastha Mehta]

On 25 October 2013 14:18, Jan Beulich <jbeulich@suse.com> wrote:
>>>> Aastha Mehta <aasthakm@gmail.com> 10/24/13 7:32 PM >>>
>>On 24 October 2013 09:09, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>> Aastha Mehta <aasthakm@gmail.com> 10/23/13 7:44 PM >>>
>>>> In case of an asynchronous I/O does one guest have to
>>>>register its callbacks with the other guest directly?
>>>>Or does one guest hypercall into xen, which then upcalls into the
>>>>other guest and then transfer reply in the reverse using another pair
>>>>of hypercall and upcall?
>>>
>>> Again, the latter.
>>
>>So when one guest (G1) shares pages with second (G2), for example for
>>block I/O, G1 will inform second via xen about a shared page.
>
> How this information is propagated is up to you; commonly this is being
> done via xenstore.
>
>> G1 will also register a callback with Xen.
>
> That callback will need to be registered in any case - the (general)
> event upcall.
>
Could you please clarify which general event upcall are you referring to here?

>> When Xen upcalls into G2 to forward G1's request, does it block on G2's reponse?
>
> Why would it - it acts as a passive agent here only.
>
>> How does G2 send back the result of I/O to G1?
>
> Generally as contents of the shared page.
>
> But really - please take a look at the various existing backend/frontend
> drivers.
>
>>>>Regarding xen -- domain communication: how can xen send some request
>>>>to dom0 for some operation that needs to be performed in the dom0 and
>>>>get back results from the dom0?
>>>
>>> Look at the MCE polling Dom0 performs for example, or the memory pressure
>>> threshold handling: One model would be for Dom0 to set up a (dedicated)
>>> vIRQ for that purpose, and Xen trigger that vIRQ when it needs Dom0's help.
>>
>>Just to clarify, will guest return the result of the operation using a
>>hypercall?
>
> First of all there's generally no "result" in the examples I gave. Instead,
> Xen simply requests a certain kind of action to be taken. And Xen in fact
> must not depend on whether or when that action is being taken. But in
> general it might well be that the action requested consists of one or more
> hypercalls.
>
> Jan
>

Let me clarify my question a bit more. My requirement is that I need
to perform some computations on an application data, but I cannot
allow the application to do so directly. I would like to provide the
computation as a service from, say dom0. Here are two approaches that
come to my mind -
Approach 1
1. Application makes a syscall into DomU kernel requesting for the computation.
2. DomU copies the application data into a kernel buffer and makes a
hypercall into Xen forwarding application's request.
3. Xen upcalls into dom0 kernel for the service (a new vIRQ?).
4. Dom0 performs the computation and sends the results back to Xen via
a hypercall.
5. Xen returns the results to DomU kernel through another upcall (another vIRQ).
6. DomU kernel forwards the result to the application and returns the syscall.

Approach 2
Replace steps 2-5 in the above approach with interdomain event channel
notification between dom0 and domU and sharing the data pages through
grant table mechanism. So domU will notify dom0 that it has sent a
request for the computation, after placing the data to be used in the
shared memory. dom0's handler will get the request and the shared
data, compute the result, place it in another shared memory region and
notify domU of the completion of the request.

Do the two approaches even make sense?
I looked at the blkfront-blkback drivers, and how they use event
channels, grant tables and xenbus. If I understand correctly, the
event channels have to be associated with the xenbus, but it seems
that xenbus can only be used by drivers. I do not understand how the
event channels can be setup ad hoc between two domains. How do I know
what xenbus_device to connect to?

Another question I have is, if a domU makes a hypercall into Xen and
if the hypercall is long running, does it block the entire domU until
the hypercall returns? What if there are multiple vcpus?


Thanks and regards,
Aastha.

Re: questions regarding interdomain communication and xen-guest communication

[Andrew Cooper]

On 28/10/13 15:21, Aastha Mehta wrote:
> On 25 October 2013 14:18, Jan Beulich <jbeulich@suse.com> wrote:
>>>>> Aastha Mehta <aasthakm@gmail.com> 10/24/13 7:32 PM >>>
>>> On 24 October 2013 09:09, Jan Beulich <jbeulich@suse.com> wrote:
>>>>>>> Aastha Mehta <aasthakm@gmail.com> 10/23/13 7:44 PM >>>
>>>>> In case of an asynchronous I/O does one guest have to
>>>>> register its callbacks with the other guest directly?
>>>>> Or does one guest hypercall into xen, which then upcalls into the
>>>>> other guest and then transfer reply in the reverse using another pair
>>>>> of hypercall and upcall?
>>>> Again, the latter.
>>> So when one guest (G1) shares pages with second (G2), for example for
>>> block I/O, G1 will inform second via xen about a shared page.
>> How this information is propagated is up to you; commonly this is being
>> done via xenstore.
>>
>>> G1 will also register a callback with Xen.
>> That callback will need to be registered in any case - the (general)
>> event upcall.
>>
> Could you please clarify which general event upcall are you referring to here?
>
>>> When Xen upcalls into G2 to forward G1's request, does it block on G2's reponse?
>> Why would it - it acts as a passive agent here only.
>>
>>> How does G2 send back the result of I/O to G1?
>> Generally as contents of the shared page.
>>
>> But really - please take a look at the various existing backend/frontend
>> drivers.
>>
>>>>> Regarding xen -- domain communication: how can xen send some request
>>>>> to dom0 for some operation that needs to be performed in the dom0 and
>>>>> get back results from the dom0?
>>>> Look at the MCE polling Dom0 performs for example, or the memory pressure
>>>> threshold handling: One model would be for Dom0 to set up a (dedicated)
>>>> vIRQ for that purpose, and Xen trigger that vIRQ when it needs Dom0's help.
>>> Just to clarify, will guest return the result of the operation using a
>>> hypercall?
>> First of all there's generally no "result" in the examples I gave. Instead,
>> Xen simply requests a certain kind of action to be taken. And Xen in fact
>> must not depend on whether or when that action is being taken. But in
>> general it might well be that the action requested consists of one or more
>> hypercalls.
>>
>> Jan
>>
> Let me clarify my question a bit more. My requirement is that I need
> to perform some computations on an application data, but I cannot
> allow the application to do so directly. I would like to provide the
> computation as a service from, say dom0. Here are two approaches that
> come to my mind -
> Approach 1
> 1. Application makes a syscall into DomU kernel requesting for the computation.
> 2. DomU copies the application data into a kernel buffer and makes a
> hypercall into Xen forwarding application's request.
> 3. Xen upcalls into dom0 kernel for the service (a new vIRQ?).
> 4. Dom0 performs the computation and sends the results back to Xen via
> a hypercall.
> 5. Xen returns the results to DomU kernel through another upcall (another vIRQ).
> 6. DomU kernel forwards the result to the application and returns the syscall.
>
> Approach 2
> Replace steps 2-5 in the above approach with interdomain event channel
> notification between dom0 and domU and sharing the data pages through
> grant table mechanism. So domU will notify dom0 that it has sent a
> request for the computation, after placing the data to be used in the
> shared memory. dom0's handler will get the request and the shared
> data, compute the result, place it in another shared memory region and
> notify domU of the completion of the request.
>
> Do the two approaches even make sense?
> I looked at the blkfront-blkback drivers, and how they use event
> channels, grant tables and xenbus. If I understand correctly, the
> event channels have to be associated with the xenbus, but it seems
> that xenbus can only be used by drivers. I do not understand how the
> event channels can be setup ad hoc between two domains. How do I know
> what xenbus_device to connect to?
>
> Another question I have is, if a domU makes a hypercall into Xen and
> if the hypercall is long running, does it block the entire domU until
> the hypercall returns? What if there are multiple vcpus?
>
>
> Thanks and regards,
> Aastha.

This description sounds very much like you want to be using libchan,
which provides a communication mechanism between two domains.  It will
take care of the shared memory and event channels, leaving you with an
API for sending and receiving data.

With hypercalls, they will typically only block the vcpu which makes the
hypercall.

~Andrew

Re: questions regarding interdomain communication and xen-guest communication

[Jan Beulich]

>>> On 28.10.13 at 16:21, Aastha Mehta <aasthakm@gmail.com> wrote:
> On 25 October 2013 14:18, Jan Beulich <jbeulich@suse.com> wrote:
>> That callback will need to be registered in any case - the (general)
>> event upcall.
>>
> Could you please clarify which general event upcall are you referring to 
> here?

The one you would e.g. register with CALLBACKTYPE_event.

> Let me clarify my question a bit more. My requirement is that I need
> to perform some computations on an application data, but I cannot
> allow the application to do so directly. I would like to provide the
> computation as a service from, say dom0. Here are two approaches that
> come to my mind -
> Approach 1
> 1. Application makes a syscall into DomU kernel requesting for the 
> computation.
> 2. DomU copies the application data into a kernel buffer and makes a
> hypercall into Xen forwarding application's request.
> 3. Xen upcalls into dom0 kernel for the service (a new vIRQ?).

Definitely not. This would be a normal event channel.

> 4. Dom0 performs the computation and sends the results back to Xen via
> a hypercall.
> 5. Xen returns the results to DomU kernel through another upcall (another 
> vIRQ).

Again an event channel.

> 6. DomU kernel forwards the result to the application and returns the 
> syscall.
> 
> Approach 2
> Replace steps 2-5 in the above approach with interdomain event channel

Ah, okay - so just forget about #1 then.

> notification between dom0 and domU and sharing the data pages through
> grant table mechanism. So domU will notify dom0 that it has sent a
> request for the computation, after placing the data to be used in the
> shared memory. dom0's handler will get the request and the shared
> data, compute the result, place it in another shared memory region and
> notify domU of the completion of the request.
> 
> Do the two approaches even make sense?
> I looked at the blkfront-blkback drivers, and how they use event
> channels, grant tables and xenbus. If I understand correctly, the
> event channels have to be associated with the xenbus, but it seems
> that xenbus can only be used by drivers. I do not understand how the
> event channels can be setup ad hoc between two domains. How do I know
> what xenbus_device to connect to?

There's examples of how to use the event channel device to use
them from user space. But as pointed out in someone else's
response - perhaps all you need to do is use an already existing
library.

> Another question I have is, if a domU makes a hypercall into Xen and
> if the hypercall is long running, does it block the entire domU until
> the hypercall returns? What if there are multiple vcpus?

No, just the particular vCPU would be blocked.

Jan

Re: questions regarding interdomain communication and xen-guest communication

[Aastha Mehta]

On 28 October 2013 17:10, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 28.10.13 at 16:21, Aastha Mehta <aasthakm@gmail.com> wrote:
>> On 25 October 2013 14:18, Jan Beulich <jbeulich@suse.com> wrote:
>>> That callback will need to be registered in any case - the (general)
>>> event upcall.
>>>
>> Could you please clarify which general event upcall are you referring to
>> here?
>
> The one you would e.g. register with CALLBACKTYPE_event.
>
>> Let me clarify my question a bit more. My requirement is that I need
>> to perform some computations on an application data, but I cannot
>> allow the application to do so directly. I would like to provide the
>> computation as a service from, say dom0. Here are two approaches that
>> come to my mind -
>> Approach 1
>> 1. Application makes a syscall into DomU kernel requesting for the
>> computation.
>> 2. DomU copies the application data into a kernel buffer and makes a
>> hypercall into Xen forwarding application's request.
>> 3. Xen upcalls into dom0 kernel for the service (a new vIRQ?).
>
> Definitely not. This would be a normal event channel.
>
>> 4. Dom0 performs the computation and sends the results back to Xen via
>> a hypercall.
>> 5. Xen returns the results to DomU kernel through another upcall (another
>> vIRQ).
>
> Again an event channel.
>
>> 6. DomU kernel forwards the result to the application and returns the
>> syscall.
>>
>> Approach 2
>> Replace steps 2-5 in the above approach with interdomain event channel
>
> Ah, okay - so just forget about #1 then.
>
>> notification between dom0 and domU and sharing the data pages through
>> grant table mechanism. So domU will notify dom0 that it has sent a
>> request for the computation, after placing the data to be used in the
>> shared memory. dom0's handler will get the request and the shared
>> data, compute the result, place it in another shared memory region and
>> notify domU of the completion of the request.
>>
>> Do the two approaches even make sense?
>> I looked at the blkfront-blkback drivers, and how they use event
>> channels, grant tables and xenbus. If I understand correctly, the
>> event channels have to be associated with the xenbus, but it seems
>> that xenbus can only be used by drivers. I do not understand how the
>> event channels can be setup ad hoc between two domains. How do I know
>> what xenbus_device to connect to?
>
> There's examples of how to use the event channel device to use
> them from user space. But as pointed out in someone else's
> response - perhaps all you need to do is use an already existing
> library.
>
>> Another question I have is, if a domU makes a hypercall into Xen and
>> if the hypercall is long running, does it block the entire domU until
>> the hypercall returns? What if there are multiple vcpus?
>
> No, just the particular vCPU would be blocked.
>
> Jan
>

Thanks for the quick responses. libvchan seems to be useful for what I
would like to do. A few more questions on that -
I tried running the vchan-node1 and vchan-node2 test executables. I
provide the nodepath as data/vchan/0/ports or even data/vchan/4/ports
but xs_read/write do not seem to work. Is there something I need to do
to the xenstore?

I looked at qubes OS and linpicker, which use this library, but I
don't completely follow how it is to be used. I need to have the
libraries in both the domains that I want to communicate between, and
the applications running in these domains use it.

Could you please point me to the example on using the event channel
device from user space?

Regards


-- 
Aastha Mehta
MPI-SWS, Germany
E-mail: aasthakm@mpi-sws.org

Re: questions regarding interdomain communication and xen-guest communication

[Jan Beulich]

>>> On 29.10.13 at 14:22, Aastha Mehta <aasthakm@gmail.com> wrote:
> Thanks for the quick responses. libvchan seems to be useful for what I
> would like to do. A few more questions on that -
> I tried running the vchan-node1 and vchan-node2 test executables. I
> provide the nodepath as data/vchan/0/ports or even data/vchan/4/ports
> but xs_read/write do not seem to work. Is there something I need to do
> to the xenstore?

Sorry, I'll have to defer to people having actually used libvchan to
help you further here.

> I looked at qubes OS and linpicker, which use this library, but I
> don't completely follow how it is to be used. I need to have the
> libraries in both the domains that I want to communicate between, and
> the applications running in these domains use it.
> 
> Could you please point me to the example on using the event channel
> device from user space?

I'm sorry to say that, but - could you please invest a little bit of
effort yourself in finding those? (And if that's really too much to
ask for, grep-ing the libxc sources for "evtchn" would be a good
start.)

Jan