Inter VM Communication

Inter VM Communication

[Kumar, Venkat]

Just like how Xen has Xenbus, Emulated Platform-PCI device and Events for Inter VM communication, Does KVM has any mechanism for Inter VM communication? 
How to share a page between two virtual machines running on KVM?

Thx,
Venkat

Re: Inter VM Communication

[Avi Kivity]

Kumar, Venkat wrote:
> Just like how Xen has Xenbus, Emulated Platform-PCI device and Events for Inter VM communication, Does KVM has any mechanism for Inter VM communication? 
> How to share a page between two virtual machines running on KVM?
>   

If you just want to share a page (or a bunch of memory), write a qemu 
PCI device model that exposes that page through a BAR.  The guests can 
then map the BAR and access the page.

To share the page, use normal Linux memory sharing, such as shared 
memory segments or mapped files (possibly on /dev/shm).

Note that sharing will break as soon as one of the guests is migrated away.

To send events you can utilize pci interrupts.

An alternative approach is to use virtio, but this is somewhat more 
complicated.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Inter VM Communication

[Cam Macdonell]

Avi Kivity wrote:
> Kumar, Venkat wrote:
>> Just like how Xen has Xenbus, Emulated Platform-PCI device and Events 
>> for Inter VM communication, Does KVM has any mechanism for Inter VM 
>> communication? How to share a page between two virtual machines 
>> running on KVM?
>>   
> 
> If you just want to share a page (or a bunch of memory), write a qemu 
> PCI device model that exposes that page through a BAR.  The guests can 
> then map the BAR and access the page.
> 
> To share the page, use normal Linux memory sharing, such as shared 
> memory segments or mapped files (possibly on /dev/shm).

I've actually created a patch (and corresponding device driver) that 
works this way based on Avi's suggestion of this approach before.  I'm 
willing to release it of course, but it's in pretty rough form being my 
first Qemu/KVM PCI device.  Avi (or anyone else), would you mind having 
a quick look first before I release it to the list?

FYI, I'm using this for my PhD research.

> Note that sharing will break as soon as one of the guests is migrated away.
> 
> To send events you can utilize pci interrupts.

I haven't implemented PCI interrupts yet, but would appreciate some 
pointers on how to go about this.

Thanks,
Cam

Re: Inter VM Communication

[Avi Kivity]

Cam Macdonell wrote:
>> If you just want to share a page (or a bunch of memory), write a qemu 
>> PCI device model that exposes that page through a BAR.  The guests 
>> can then map the BAR and access the page.
>>
>> To share the page, use normal Linux memory sharing, such as shared 
>> memory segments or mapped files (possibly on /dev/shm).
>
> I've actually created a patch (and corresponding device driver) that 
> works this way based on Avi's suggestion of this approach before.  I'm 
> willing to release it of course, but it's in pretty rough form being 
> my first Qemu/KVM PCI device.  Avi (or anyone else), would you mind 
> having a quick look first before I release it to the list?

I'm of course willing to take a look, but I think you should post it 
directly to the list.  More eyes mean more comments, and on this list we 
usually see constructive criticism.

>
> I haven't implemented PCI interrupts yet, but would appreciate some 
> pointers on how to go about this.

Call qemu_set_irq(pci_dev->irq[0], 1) when you want your guest to take 
notice, and qemu_set_irq(pci_dev->irq[0], 0) when service is no longer 
necessary.

You'll also need to set pci_dev->config[PCI_INTERRUPT_PIN] to 1 to let 
the guest know you're using irq[0].

See virtio or rtl8139.c as examples.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Inter VM Communication

[Brian Jackson]

On Tuesday 24 March 2009 15:41:43 Cam Macdonell wrote:
> Avi Kivity wrote:
> > Kumar, Venkat wrote:
> >> Just like how Xen has Xenbus, Emulated Platform-PCI device and Events
> >> for Inter VM communication, Does KVM has any mechanism for Inter VM
> >> communication? How to share a page between two virtual machines
> >> running on KVM?
> >
> > If you just want to share a page (or a bunch of memory), write a qemu
> > PCI device model that exposes that page through a BAR.  The guests can
> > then map the BAR and access the page.
> >
> > To share the page, use normal Linux memory sharing, such as shared
> > memory segments or mapped files (possibly on /dev/shm).
>
> I've actually created a patch (and corresponding device driver) that
> works this way based on Avi's suggestion of this approach before.  I'm
> willing to release it of course, but it's in pretty rough form being my
> first Qemu/KVM PCI device.  Avi (or anyone else), would you mind having
> a quick look first before I release it to the list?

I agree with Avi on just going ahead and sending it. I'd like to take a look 
too.

>
> FYI, I'm using this for my PhD research.
>
> > Note that sharing will break as soon as one of the guests is migrated
> > away.
> >
> > To send events you can utilize pci interrupts.
>
> I haven't implemented PCI interrupts yet, but would appreciate some
> pointers on how to go about this.
>
> Thanks,
> Cam
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Inter VM Communication

[Anthony Liguori]

Avi Kivity wrote:
> Kumar, Venkat wrote:
>> Just like how Xen has Xenbus, Emulated Platform-PCI device and Events 
>> for Inter VM communication, Does KVM has any mechanism for Inter VM 
>> communication? How to share a page between two virtual machines 
>> running on KVM?
>>   
>
> If you just want to share a page (or a bunch of memory), write a qemu 
> PCI device model that exposes that page through a BAR.  The guests can 
> then map the BAR and access the page.

So one gotcha about using a BAR is that we emulate a 32-bit PCI device 
so that limits where the BAR can live in memory and how large it can be.

I think bars also have to be powers of two in size.

Regards,

Anthony Liguori

> To share the page, use normal Linux memory sharing, such as shared 
> memory segments or mapped files (possibly on /dev/shm).
>
> Note that sharing will break as soon as one of the guests is migrated 
> away.
>
> To send events you can utilize pci interrupts.
>
> An alternative approach is to use virtio, but this is somewhat more 
> complicated.
>

Re: Inter VM Communication

[Avi Kivity]

Anthony Liguori wrote:
> So one gotcha about using a BAR is that we emulate a 32-bit PCI device 
> so that limits where the BAR can live in memory and how large it can be.

We can overcome that by emulating a 64-bit device...

> I think bars also have to be powers of two in size.

I think that's right; but the device can define the the range [0, X) as 
shared RAM and the range [X, 2^n) as undefined.  So long as it makes X 
available in the config space.

Besides, computer people have trouble imagining numbers which are not 
powers of two.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.