Re: KVM devices assignment; PCIe AER?

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 2010-10-27 at 14:43 -0700, Etienne Martineau wrote:
> On Wed, 27 Oct 2010, Alex Williamson wrote:
> > No, emulated devices trigger interrupts directly with qemu_set_irq.
> > irqfds are currently only used by vhost afaik, since it's being
> > interrupted externally, much like pass through devices are.
> 
> Fair enough. Thanks for the clarification.
> 
> > Sort of.  When the VFIO device triggers an interrupt, we get notified
> > via the eventfd we've registered for that interrupt.  We can then call
> > qemu_set_irq directly to raise that interrupt in the KVM kernel APIC.
> > That much works today.
> 
> Understood but performance wise this is no good for KVM right?

Right, bouncing interrupts and EOIs through qemu via eventfds is going
to add latency.  On the interrupt path we already have irqfds, which
will avoid the bounce through userspace, we just need to use them.
Doing something similar with EOIs could avoid that path, giving us
something comparable to current device assignment.

> > The irqfd mechanism is simply a way for KVM to
> > directly consume the eventfd and raise an interrupt via a pre-setup
> > vector.  That's yet to be implemented for INTx on VFIO, but should
> > mostly be a matter of connecting existing pieces together.  It's working
> > for MSI-X.
> 
> OK, I was on the impression you already had irqfd 'connected' to KVM from 
> VFIO... This is why I was asking about the nature of the changed in VFIO.
> 
> > When VFIO sends an interrupt, it disables the physical device from
> > generating more interrupts (this is where VFIO requires PCI 2.3
> > compliant devices for the INTx disable bit int he status register).
> > When the guest services the interrupt, we can detect this by catching
> > the EOI of the IOAPIC.  At that point, we can re-eanble interrupts on
> > the device.  Wash, rinse, repeat.
> >
> > To do this in qemu, I created a callback on the ioapic where drivers can
> > register for the interrupt they care about.  Since KVM moves the ioapic
> > into the kernel, we need to extend this into KVM and have yet another
> > eventfd mechanism.  It's possible that we could have the VFIO kernel
> > module also receive this eventfd, re-enabling interrupts on the device,
> > in much the same way as above.
> 
> In the cases of KVM where are you going to catch the EIO? For some 
> reason I'm on the impression that this is part of KVM. If so then how are 
> you going to 'signal' to VFIO? Cannot use eventfd here right?

KVM already has an internal IRQ ACK notifier (which is what current
device assignment uses to do the same thing), it's just a matter of
adding a callback that does a kvm_register_irq_ack_notifier that sends
off the eventfd signal.  I've got this working and will probably send
out the KVM patch this week.  For now the eventfd goes to userspace, but
this is where I imagine we could steal some of the irqfd code to make
VFIO consume the irqfd signal directly.  Thanks,

Alex