ioeventfd usage in KVM

ioeventfd usage in KVM

[Cam Macdonell]

Hi,

I'm trying to use ioeventfd/irqfds for my shared memory patch.  I
followed the usage in the vhost-net patches to see how it's setup for
virtio-pci and tried to follow it as closely as I could. Despite the
call to kvm_vm_ioctl() returning 0, any writes to the assigned 4-byte
memory area do not seem to trigger a write to the corresponding fd.
At this point, I'm just trying to get the ioeventfd happening.

I notice that virtio-pci allocates it's BAR as
PCI_BASE_ADDRESS_SPACE_IO and then uses register_ioport_{read,write}
whereas I use cpu_register_io_memory and the
PCI_BASE_ADDRESS_SPACE_MEMORY type as shown below.

+static void ivshmem_mmio_map(PCIDevice *pci_dev, int region_num,
+                       pcibus_t addr, pcibus_t size, int type)
+{
+    PCI_IVShmemState *d = (PCI_IVShmemState *)pci_dev;
+    IVShmemState *s = &d->ivshmem_state;
+
+    s->otheraddr = addr  /* this address will be used for the ioeventfd*/
+    cpu_register_physical_memory(addr + 0, 0x100, s->ivshmem_mmio_io_addr);
+}


+    s->ivshmem_mmio_io_addr = cpu_register_io_memory(ivshmem_mmio_read,
+                                    ivshmem_mmio_write, s);
+    /* region for registers*/
+    pci_register_bar(&d->dev, 0, 0x100,
+                           PCI_BASE_ADDRESS_SPACE_MEMORY, ivshmem_mmio_map);

my basic attempt looks like this:

    struct kvm_ioeventfd ked;

    ked.addr = s->otheraddr + Doorbell;
    ked.len = 4;
    ked.flags = KVM_IOEVENTFD_FLAG_PIO;
    ked.fd = an_eventfd;
    ret = kvm_vm_ioctl(kvm_state, KVM_IOEVENTFD, &ked);

but when the guest writes to the offset of Doorbell, I cannot see any
action (via a select on the fd).  Is there something obviously wrong
that I'm doing?

When I get this working, I'd be happy to write up a page for the KVM site.

Thanks,
Cam

Re: ioeventfd usage in KVM

[Avi Kivity]

On 03/12/2010 07:08 AM, Cam Macdonell wrote:
>
> +    s->ivshmem_mmio_io_addr = cpu_register_io_memory(ivshmem_mmio_read,
> +                                    ivshmem_mmio_write, s);
> +    /* region for registers*/
> +    pci_register_bar(&d->dev, 0, 0x100,
> +                           PCI_BASE_ADDRESS_SPACE_MEMORY, ivshmem_mmio_map);
>    

You've selected the memory address space here.

> my basic attempt looks like this:
>
>      struct kvm_ioeventfd ked;
>
>      ked.addr = s->otheraddr + Doorbell;
>      ked.len = 4;
>      ked.flags = KVM_IOEVENTFD_FLAG_PIO;
>      ked.fd = an_eventfd;
>      ret = kvm_vm_ioctl(kvm_state, KVM_IOEVENTFD,&ked);
>    

But the PIO address space here.

> but when the guest writes to the offset of Doorbell, I cannot see any
> action (via a select on the fd).  Is there something obviously wrong
> that I'm doing?
>    

Yes - they must match.  Not PIO is faster on x86 but nonexistant elsewhere.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: ioeventfd usage in KVM

[Cam Macdonell]

On Fri, Mar 12, 2010 at 12:43 AM, Avi Kivity <avi@redhat.com> wrote:
> On 03/12/2010 07:08 AM, Cam Macdonell wrote:
>>
>> +    s->ivshmem_mmio_io_addr = cpu_register_io_memory(ivshmem_mmio_read,
>> +                                    ivshmem_mmio_write, s);
>> +    /* region for registers*/
>> +    pci_register_bar(&d->dev, 0, 0x100,
>> +                           PCI_BASE_ADDRESS_SPACE_MEMORY,
>> ivshmem_mmio_map);
>>
>
> You've selected the memory address space here.
>
>> my basic attempt looks like this:
>>
>>     struct kvm_ioeventfd ked;
>>
>>     ked.addr = s->otheraddr + Doorbell;
>>     ked.len = 4;
>>     ked.flags = KVM_IOEVENTFD_FLAG_PIO;
>>     ked.fd = an_eventfd;
>>     ret = kvm_vm_ioctl(kvm_state, KVM_IOEVENTFD,&ked);
>>
>
> But the PIO address space here.
>
>> but when the guest writes to the offset of Doorbell, I cannot see any
>> action (via a select on the fd).  Is there something obviously wrong
>> that I'm doing?
>>
>
> Yes - they must match.  Not PIO is faster on x86 but nonexistant elsewhere.

Does it matter that PIO is slower since with ioeventfd Qemu is being
bypassed all together?  Can I use the memory space with ioeventfds?
There doesn't seem to be a flag (KVM_IOEVENTFD_FLAG_MMIO) for it.

Cam

>
> --
> Do not meddle in the internals of kernels, for they are subtle and quick to
> panic.
>
>

Re: ioeventfd usage in KVM

[Avi Kivity]

On 03/12/2010 05:50 PM, Cam Macdonell wrote:
>
>> Yes - they must match.  Not PIO is faster on x86 but nonexistant elsewhere.
>>      
>    

s/Not/Note/.

> Does it matter that PIO is slower since with ioeventfd Qemu is being
> bypassed all together?

mmio is slower in kvm, not qemu; mmio must go through the emulator, 
fetch the instruction, decode and execute it, while pio decoding is done 
by the processor microcode.

> Can I use the memory space with ioeventfds?
>    

Yes.

> There doesn't seem to be a flag (KVM_IOEVENTFD_FLAG_MMIO) for it.
>    

If you don't specify _PIO, _MMIO is the default, as 
Documentation/kvm/api.txt doesn't say.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.