Re: [Qemu-devel] [PATCH 2/3] KVM: add kvm_arch_irqchip_add_msi_route

[Alex Williamson <alex.williamson@redhat.com>]

On Sat, 2013-06-22 at 08:21 +1000, Benjamin Herrenschmidt wrote:
> On Sat, 2013-06-22 at 00:12 +0200, Alexander Graf wrote:
> > On 21.06.2013, at 23:54, Benjamin Herrenschmidt wrote:
> > 
> > > On Fri, 2013-06-21 at 15:46 +0200, Alexander Graf wrote:
> > >> Not sure. We could just declare a "direct virq==irq" mode in which
> > >> msi.data == virq == irq. No need for any translation then.
> > > 
> > > Maybe. Beware that MSI data is only 16-bit on the wire but we may
> > not
> > > care here.
> > > 
> > > One thing I'm not 100% certain of is how Alexey makes all that work
> > with
> > > VFIO since the MSI address/data in the device shall not be the qemu
> > > "cooked up" ones, but the real HW ones (corresponding to a different
> > > host interrupt).
> > > 
> > > How do that work ?
> > 
> > The real device address/data go to a normal host interrupt vector.
> 
> Right, I know that :-)
> 
> >  Once we hit such a vector, we need to find out that it's destined for
> > the guest in real mode - no idea how you planned to do that - and then
> > reinject it back into the guest with the virtual irq vector that you
> > can find out by asking the irqfd hopefully.
> > 
> > It might make sense to implement it the easy way without real mode
> > first, and then take it from there ;).
> 
> We have that already. That isn't my question.
> 
> How is the "configuration" done ?
> 
> On x86, I assume, the guest kernel directly whacks the MSI config space
> and MSI-X BAR space using some address/data it cooks up which are *not*
> the real ones needed in the HW right ?
> 
> So qemu seems to play tricks to intercept the MSI-X BAR, I swa that, but
> how does it know what real value to put in the HW instead ?
> 
> In our case, the guest calls RTAS, which needs to configure "something"
> in the device. What does it do ?
> 
> Does it call into VFIO which then does a pci_enable_msi[x] ? In that
> case how do you deal with a guest for example prodding a single MSI-X
> in the middle of the array ? This is not an interface supported by
> pci_enable_msix ....

MSI-X is rather ugly.  As you suggest, we trap accesses to the MSI-X
table.  We don't know how many vectors the guest is going to use, so we
incrementally add them by disabling and re-enabling with a new vector
count.  The host decides what to put in the table, we don't care.  All
interrupts bounce through the host and get to the guest via eventfd,
either through qemu or directly through KVM irqfd.

If an in-use vector is modified, we write the new "MSI route" to KVM,
the host doesn't need to care.  If a vector is masked, we free the host
irq handler w/o modifying the vector configuration.  There's a comment
in hw/misc/vfio.c that we could also just bounce masked vectors through
qemu and let it drop it if we wanted to completely avoid toggling the
host.  Linux currently does not have a usable interface for masking
vectors at the device.  Thanks,

Alex