RE: Event channels in KVM?

RE: Event channels in KVM?

[Kapadia, Vivek]

I came across this thread looking for an efficient event channel mechanism between two guests (running on different cpu cores).

While I can use available emulated IO mechanism (guest1->host kernel driver->Qemu1->Qemu2) in conjunction with interrupt mechanism (Qemu2->host kernel driver->guest2) in KVM, this involves several context switches. Xen handles notifications in hypervisor via hypercall and hence is likely more efficient.

Is there a way I can perform direct notification (guest1->host kernel driver->guest2) in kvm?

Thanks,
Vivek.


Matt Anger wrote:
> I was referring to the bounce from host kernel to qemu
> and then back
> to the host kernel for my BE driver.
> Xen:
> guest -> guest kernel driver-> host kernel driver
>
> For both situations I need a FE and BE driver, but for
> KVM I need to modify QEMU and teach it how to pass the
> virtio calls to my Host
> driver. In Xen BE driver and Host driver are the same and
> I don't have
> to recompile any part of Xen.
>
You can implement (host) kernel level device model too which will have
same result with Xen, i.e. skip Qemu.
I don't think Xen & KVM has fundamental difference here, but different
implementation style. Xen implement BE driver in driver domain kernel,
but KVM like to implement in user level host process. Different people
has different favorite, I am neutral actually :)

Thx, eddie
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html <http://vger.kernel.org/majordomo-info.html>

Re: Event channels in KVM?

[Anthony Liguori]

Kapadia, Vivek wrote:
> I came across this thread looking for an efficient event channel mechanism between two guests (running on different cpu cores).
>
> While I can use available emulated IO mechanism (guest1->host kernel driver->Qemu1->Qemu2) in conjunction with interrupt mechanism (Qemu2->host kernel driver->guest2) in KVM, this involves several context switches. Xen handles notifications in hypervisor via hypercall and hence is likely more efficient.
>   

They almost certainly aren't more efficient.

An event channel notification involves a hypercall to the hypervisor.  
When using VT, the performance difference between a vmcall exit vs. a 
pio exit is quite small (especially compared to the overhead of the 
exit).  We're talking in the order of nanoseconds compared to microseconds.

What makes KVM particularly different from Xen is that in KVM, the PIO 
operation results in a direct transition to QEMU.  In Xen, typically 
event channel notifications result in a bit being set in a bitmap which 
then results in an interrupt injection depending on the next opportunity 
the hypervisor has to schedule/run the receiving domain.  This is not 
deterministic and can potentially be a very long period of time.

Event channels are inherently asynchronous whereas PIO notifications in 
KVM are synchronous.  Since the scheduler isn't involved and control 
never leaves the CPU, the KVM PIO notifications are actually extremely 
efficient.  IMHO, it's one of KVM's best design features.

It used to be in HVM that since things like PIO operations are 
inherently synchronous, and there's not point in a VM waiting around for 
the asynchronous event channel notification to result in qemu-dm 
invocation,, there was a very special code path in the hypervisor to 
ensure that Domain-0 was scheduled immediately when receiving an event 
channel notification from an HVM domain.  This was an important 
optimization because event channel notification latency was 
prohibitively high.

Now in the context of the stub domain, I'm not sure what changes they've 
made.  In the earliest prototypes of stub domain, the same short cutting 
logic was maintained but the stub domain was executed instead of domain-0.

> Is there a way I can perform direct notification (guest1->host kernel driver->guest2) in kvm?
>   

Between guests, we don't have a notification framework today.  You can 
use IPC from two QEMU processes and I'd expect that to perform pretty 
well.  I'm not sure you can get much advantages from doing things in the 
kernel because you cannot avoid the heavy weight exit.

Regards,

Anthony Liguori

Re: Event channels in KVM?

[Avi Kivity]

Anthony Liguori wrote:
> Kapadia, Vivek wrote:
>> I came across this thread looking for an efficient event channel 
>> mechanism between two guests (running on different cpu cores).
>>
>> While I can use available emulated IO mechanism (guest1->host kernel 
>> driver->Qemu1->Qemu2) in conjunction with interrupt mechanism 
>> (Qemu2->host kernel driver->guest2) in KVM, this involves several 
>> context switches. Xen handles notifications in hypervisor via 
>> hypercall and hence is likely more efficient.
>>   
>
> They almost certainly aren't more efficient.
>
> An event channel notification involves a hypercall to the hypervisor.  
> When using VT, the performance difference between a vmcall exit vs. a 
> pio exit is quite small (especially compared to the overhead of the 
> exit).  We're talking in the order of nanoseconds compared to 
> microseconds.
>
> What makes KVM particularly different from Xen is that in KVM, the PIO 
> operation results in a direct transition to QEMU.  In Xen, typically 
> event channel notifications result in a bit being set in a bitmap 
> which then results in an interrupt injection depending on the next 
> opportunity the hypervisor has to schedule/run the receiving domain.  
> This is not deterministic and can potentially be a very long period of 
> time.
>
> Event channels are inherently asynchronous whereas PIO notifications 
> in KVM are synchronous.  Since the scheduler isn't involved and 
> control never leaves the CPU, the KVM PIO notifications are actually 
> extremely efficient.  IMHO, it's one of KVM's best design features.
>

If you make the pio operation wake up another guest, then the operation 
becomes asynchronous.  There's really no fundamental different between 
Xen and kvm here, and both will require the same number of context 
switches (one) to transfer control.

Handling a pio that is completely internal to the guest is different 
(Xen has to schedule dom0 or the stub domain), but that's not related to 
interguest communications.



-- 
error compiling committee.c: too many arguments to function

Event channels in KVM?

[Matt Anger]

Does KVM have any interface similar to event-channels like Xen does?
Basically a way to send notifications between the host and guest.

Thanks,
-Matt

Re: Event channels in KVM?

[Anthony Liguori]

Matt Anger wrote:
> Does KVM have any interface similar to event-channels like Xen does?
> Basically a way to send notifications between the host and guest.
>   

virtio is the abstraction we use.

But virtio is based on the standard hardware interfaces of the PC--PIO, 
MMIO, and interrupts.

Regards,

Anthony Liguori

> Thanks,
> -Matt
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: Event channels in KVM?

[Javier Guerra]

On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Matt Anger wrote:
>>
>> Does KVM have any interface similar to event-channels like Xen does?
>> Basically a way to send notifications between the host and guest.
>>
>
> virtio is the abstraction we use.
>
> But virtio is based on the standard hardware interfaces of the PC--PIO,
> MMIO, and interrupts.

this is rather low-level, it would be nice to have a multiplatform
interface to this abstraction.

just for kicks, i've found and printed Rusty's paper about it. hope
it's current :-)

-- 
Javier

Re: Event channels in KVM?

[Anthony Liguori]

Javier Guerra wrote:
> On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
>   
>> Matt Anger wrote:
>>     
>>> Does KVM have any interface similar to event-channels like Xen does?
>>> Basically a way to send notifications between the host and guest.
>>>
>>>       
>> virtio is the abstraction we use.
>>
>> But virtio is based on the standard hardware interfaces of the PC--PIO,
>> MMIO, and interrupts.
>>     
>
> this is rather low-level, it would be nice to have a multiplatform
> interface to this abstraction.
>   

That's exactly the purpose of virtio.  virtio is a high-level, cross 
platform interface.  It's been tested on x86, PPC, s390, and I believe 
ia64.  It also works in lguest.

It happens to use PIO, MMIO, and interrupts on x86 under KVM but other 
virtio implementations exist for other platforms.

> just for kicks, i've found and printed Rusty's paper about it. hope
> it's current :-)
>   

The other good thing to look at is the lguest documentation.  You can 
skip to just the virtio bits if you're so inclined.  It's really quite 
thoroughly documented.

Regards,

Anthony Liguori

Re: Event channels in KVM?

[Matt Anger]

Thanks for the info, I've been looking into it by trying to look
around kvm source code.
Apparently I have to write a kernel driver for the guest os and then
also write backend driver and modify qemu to use it? Is that correct?
That seems ugly, especially since now my io goes guest->guest kernel
driver->host kernel kvm->qemu->host kernel driver....
With Xen event-channels my driver gets the event notification directly
with no user space intervention removing the middle 2 steps and I
don't have to touch any of Xen's codebase. My driver just registers
for an event-channel.

Thanks
-Matt

On Fri, Sep 19, 2008 at 12:14 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Javier Guerra wrote:
>>
>> On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori <anthony@codemonkey.ws>
>> wrote:
>>
>>>
>>> Matt Anger wrote:
>>>
>>>>
>>>> Does KVM have any interface similar to event-channels like Xen does?
>>>> Basically a way to send notifications between the host and guest.
>>>>
>>>>
>>>
>>> virtio is the abstraction we use.
>>>
>>> But virtio is based on the standard hardware interfaces of the PC--PIO,
>>> MMIO, and interrupts.
>>>
>>
>> this is rather low-level, it would be nice to have a multiplatform
>> interface to this abstraction.
>>
>
> That's exactly the purpose of virtio.  virtio is a high-level, cross
> platform interface.  It's been tested on x86, PPC, s390, and I believe ia64.
>  It also works in lguest.
>
> It happens to use PIO, MMIO, and interrupts on x86 under KVM but other
> virtio implementations exist for other platforms.
>
>> just for kicks, i've found and printed Rusty's paper about it. hope
>> it's current :-)
>>
>
> The other good thing to look at is the lguest documentation.  You can skip
> to just the virtio bits if you're so inclined.  It's really quite thoroughly
> documented.
>
> Regards,
>
> Anthony Liguori
>
>

RE: Event channels in KVM?

[Dong, Eddie]

Matt Anger wrote:
> Thanks for the info, I've been looking into it by trying
> to look 
> around kvm source code.
> Apparently I have to write a kernel driver for the guest
> os and then 
> also write backend driver and modify qemu to use it? Is
> that correct? 
> That seems ugly, especially since now my io goes
> guest->guest kernel driver->host kernel kvm->qemu->host
> kernel driver.... 
> With Xen event-channels my driver gets the event
> notification directly 
> with no user space intervention removing the middle 2
> steps and I 

Why virtio needs start from guest user space while event channel not? 
It is exactly same, event channel can start from user space or kernel
space, same for virtio.
The only difference is guest<->host communication needs to go thru Qemu
(host process) in your description model, while Xen handle event channel
in hypervisor. But this can be enahnced also IMO by employing a kernel
level device model which does inter-communication for CPU if performance
is critical, but mostly probably not. 

> don't have to touch any of Xen's codebase. My driver just
> registers 
> for an event-channel.

But you still need the counter part side to handle it.
I think you may want to reuse Xen BE side code which is a wrong
assumption.

> 
> Thanks
> -Matt
> 
> On Fri, Sep 19, 2008 at 12:14 PM, Anthony Liguori
> <anthony@codemonkey.ws> wrote: 
>> Javier Guerra wrote:
>>> 
>>> On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori
>>> <anthony@codemonkey.ws> wrote: 
>>> 
>>>> 
>>>> Matt Anger wrote:
>>>> 
>>>>> 
>>>>> Does KVM have any interface similar to event-channels
>>>>> like Xen does? Basically a way to send notifications
>>>>> between the host and guest. 
>>>>> 
>>>>> 
>>>> 
>>>> virtio is the abstraction we use.
>>>> 
>>>> But virtio is based on the standard hardware
>>>> interfaces of the PC--PIO, MMIO, and interrupts. 
>>>> 
>>> 
>>> this is rather low-level, it would be nice to have a
>>> multiplatform interface to this abstraction.
>>> 
>> 
>> That's exactly the purpose of virtio.  virtio is a
>> high-level, cross platform interface.  It's been tested
>> on x86, PPC, s390, and I believe ia64.  It also works in
>> lguest.  
>> 
>> It happens to use PIO, MMIO, and interrupts on x86 under
>> KVM but other virtio implementations exist for other
>> platforms. 
>> 
>>> just for kicks, i've found and printed Rusty's paper
>>> about it. hope it's current :-) 
>>> 
>> 
>> The other good thing to look at is the lguest
>> documentation.  You can skip to just the virtio bits if
>> you're so inclined.  It's really quite thoroughly
>> documented.  
>> 
>> Regards,
>> 
>> Anthony Liguori

Re: Event channels in KVM?

[Matt Anger]

I was referring to the bounce from host kernel to qemu and then back
to the host kernel for my BE driver.
Xen:
guest -> guest kernel driver-> host kernel driver

For both situations I need a FE and BE driver, but for KVM I need to
modify QEMU and teach it how to pass the virtio calls to my Host
driver. In Xen BE driver and Host driver are the same and I don't have
to recompile any part of Xen.

-Matt

On Mon, Sep 22, 2008 at 5:30 PM, Dong, Eddie <eddie.dong@intel.com> wrote:
> Matt Anger wrote:
>> Thanks for the info, I've been looking into it by trying
>> to look
>> around kvm source code.
>> Apparently I have to write a kernel driver for the guest
>> os and then
>> also write backend driver and modify qemu to use it? Is
>> that correct?
>> That seems ugly, especially since now my io goes
>> guest->guest kernel driver->host kernel kvm->qemu->host
>> kernel driver....
>> With Xen event-channels my driver gets the event
>> notification directly
>> with no user space intervention removing the middle 2
>> steps and I
>
> Why virtio needs start from guest user space while event channel not?
> It is exactly same, event channel can start from user space or kernel
> space, same for virtio.
> The only difference is guest<->host communication needs to go thru Qemu
> (host process) in your description model, while Xen handle event channel
> in hypervisor. But this can be enahnced also IMO by employing a kernel
> level device model which does inter-communication for CPU if performance
> is critical, but mostly probably not.
>
>> don't have to touch any of Xen's codebase. My driver just
>> registers
>> for an event-channel.
>
> But you still need the counter part side to handle it.
> I think you may want to reuse Xen BE side code which is a wrong
> assumption.
>
>>
>> Thanks
>> -Matt
>>
>> On Fri, Sep 19, 2008 at 12:14 PM, Anthony Liguori
>> <anthony@codemonkey.ws> wrote:
>>> Javier Guerra wrote:
>>>>
>>>> On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori
>>>> <anthony@codemonkey.ws> wrote:
>>>>
>>>>>
>>>>> Matt Anger wrote:
>>>>>
>>>>>>
>>>>>> Does KVM have any interface similar to event-channels
>>>>>> like Xen does? Basically a way to send notifications
>>>>>> between the host and guest.
>>>>>>
>>>>>>
>>>>>
>>>>> virtio is the abstraction we use.
>>>>>
>>>>> But virtio is based on the standard hardware
>>>>> interfaces of the PC--PIO, MMIO, and interrupts.
>>>>>
>>>>
>>>> this is rather low-level, it would be nice to have a
>>>> multiplatform interface to this abstraction.
>>>>
>>>
>>> That's exactly the purpose of virtio.  virtio is a
>>> high-level, cross platform interface.  It's been tested
>>> on x86, PPC, s390, and I believe ia64.  It also works in
>>> lguest.
>>>
>>> It happens to use PIO, MMIO, and interrupts on x86 under
>>> KVM but other virtio implementations exist for other
>>> platforms.
>>>
>>>> just for kicks, i've found and printed Rusty's paper
>>>> about it. hope it's current :-)
>>>>
>>>
>>> The other good thing to look at is the lguest
>>> documentation.  You can skip to just the virtio bits if
>>> you're so inclined.  It's really quite thoroughly
>>> documented.
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>
>

RE: Event channels in KVM?

[Dong, Eddie]

Matt Anger wrote:
> I was referring to the bounce from host kernel to qemu
> and then back 
> to the host kernel for my BE driver.
> Xen:
> guest -> guest kernel driver-> host kernel driver
> 
> For both situations I need a FE and BE driver, but for
> KVM I need to modify QEMU and teach it how to pass the
> virtio calls to my Host 
> driver. In Xen BE driver and Host driver are the same and
> I don't have 
> to recompile any part of Xen.
> 
You can implement (host) kernel level device model too which will have
same result with Xen, i.e. skip Qemu.
I don't think Xen & KVM has fundamental difference here, but different
implementation style. Xen implement BE driver in driver domain kernel,
but KVM like to implement in user level host process. Different people
has different favorite, I am neutral actually :)

Thx, eddie

Re: Event channels in KVM?

[Avi Kivity]

Matt Anger wrote:
> Thanks for the info, I've been looking into it by trying to look
> around kvm source code.
> Apparently I have to write a kernel driver for the guest os and then
> also write backend driver and modify qemu to use it? Is that correct?
> That seems ugly, especially since now my io goes guest->guest kernel
> driver->host kernel kvm->qemu->host kernel driver....
> With Xen event-channels my driver gets the event notification directly
> with no user space intervention removing the middle 2 steps and I
> don't have to touch any of Xen's codebase. My driver just registers
> for an event-channel.
>   

The qemu device in kvm is equivalent to the dom0 backend driver.
kvm's qemu is equivalent to Xen's dom0.
The guest kernel driver in kvm is equivalent to the domU kernel driver.
Event channels map to interrupts (host->guest) and mmio/pio 
(guest/host); virtio provides an abstraction on top of that.

-- 
error compiling committee.c: too many arguments to function