Re: KVM devices assignment; PCIe AER?

Alex Williamson
2010-10-27T19:16:44Z

On Wed, 2010-10-27 at 11:23 -0700, Etienne Martineau wrote:
> On Wed, 27 Oct 2010, Alex Williamson wrote:
> > I've actually been working on porting the qemu vfio driver over to
> > qemu-kvm recently, and nearly have it working. For MSI-X interrupts
> > I've ported to the common msix.c code, which abstracts how the interrupt
> > is actually sent to the guest. I've also added irqfd support in the
> > msix mask notifier so MSI-X interrupts avoid bouncing through qemu. MSI
> > support should work the same once Michael's msi.c is upstream.
> >
> > For INTx interrupts, qemu_set_irq will also work with KVM (it has to or
> > all of the emulated drivers would break). The problem is getting an EOI
> > back from the KVM kernel apic. I'm currently working on code that adds
> > a new KVM ioctl to register an eventfd for the EOI, which then triggers
> > qemu-kvm to re-enable the interrupt. My hope is that we can add irqfd
> > support to both of these paths, so INTx is injected directly from VFIO
> > into KVM, and VFIO can directly consume the KVM EOI.
>
> OK let me try to understand what you've done (please correct me if I'm
> wrong). Emulated devices relies on 'kvm_irqfd' for interrupts delivery.

No, emulated devices trigger interrupts directly with qemu_set_irq.
irqfds are currently only used by vhost afaik, since it's being
interrupted externally, much like pass through devices are.

> Somehow you've modify VFIO to understand 'kvm_irqfd' so that whenever the
> assigned devices receive an IRQ it pass it directly to kvm without
> bouncing to userspace?

Sort of. When the VFIO device triggers an interrupt, we get notified
via the eventfd we've registered for that interrupt. We can then call
qemu_set_irq directly to raise that interrupt in the KVM kernel APIC.
That much works today. The irqfd mechanism is simply a way for KVM to
directly consume the eventfd and raise an interrupt via a pre-setup
vector. That's yet to be implemented for INTx on VFIO, but should
mostly be a matter of connecting existing pieces together. It's working
for MSI-X.

> I'm not sure to understand the part where VFIO signal back the EIO to KVM?

When VFIO sends an interrupt, it disables the physical device from
generating more interrupts (this is where VFIO requires PCI 2.3
compliant devices for the INTx disable bit int he status register).
When the guest services the interrupt, we can detect this by catching
the EOI of the IOAPIC. At that point, we can re-eanble interrupts on
the device. Wash, rinse, repeat.

To do this in qemu, I created a callback on the ioapic where drivers can
register for the interrupt they care about. Since KVM moves the ioapic
into the kernel, we need to extend this into KVM and have yet another
eventfd mechanism. It's possible that we could have the VFIO kernel
module also receive this eventfd, re-enabling interrupts on the device,
in much the same way as above. I haven't tried this yet, but it should
just be a matter of creating another VFIO ioctl and stealing code from
the KVM irqfd setup.

> Also, with your change do you think that VFIO can be keept generic?
> Reason I'm asking is because we are plannig to use VFIO for some userspace
> drivers...

Yes, none of this requires KVM specific modifications to VFIO. VFIO is
still just triggering eventfds, and hopefully receiving one via an
irqfd-like mechanism for EOI.

> >> Because qemu device assignment is working on VFIO I'm making the
> >> assumption that kvm iommu code can be entirely deprecated. Maybe I'm
> >> totally wrong here?
> >
> > Yes, VFIO makes no use of it.
>
> Yes I'm wrong?

VFIO does not make any use of the KVM iommu code, or any of the KVM
device assignment ioctls for that matter. Thanks,

Alex