Re: [PATCH 0/8][v2] MSI-X mask emulation support for assigned device

Michael S. Tsirkin
2010-10-20T13:50:02Z

On Wed, Oct 20, 2010 at 12:59:42PM +0200, Avi Kivity wrote:
> On 10/20/2010 12:44 PM, Michael S. Tsirkin wrote:
> >>
> >> Any performance numbers? What are the affected guests? just RHEL
> >> 4, or any others?
> >
> >Likely any old linux.
>
> I meant that people are likely to virtualize and expect high
> performance from.
>
> What about RHEL 3? Does it support msi?

Yes. I think it's the same.

> How about RHEL 5 - has it
> fixed this problem?

Yes, it has reduced the problem. Instead of masking immediately it
records the masked state and masks the first time it gets an interrupt.
Most devices mask and unmask immediately so the chance of this happening
is small.

> >> Alex, Michael, how would you do this with vfio?
> >
> >With current VFIO we would catch mask writes in qemu and
> >call a KVM ioctl.
>
> Doing what? Updating the irq routing to include/exclude the
> interrupt? Disconnecting the irqfd?

No. I mean call the new mask ioctl.

> Note you could disconnect the irqfd from either vfio or kvm.

This is what current code does, implementing mask in userspace.
But it is an rcu write side so it is a slow path operation.

> >We would also need an ioctl to retrieve
> >pending bits long term.
>
> Suppose you disconnect the irqfd. Isn't the value of the eventfd
> equivalent to the pending bit?

Yes. This is what current code does.

> >I think that it is unfortunate that we need to do this in userspace
> >while rest of configuration is done in kernel.
> >I would be much happier with userspace simply forwarding
> >everything to VFIO, so emulation does not have to
> >be split. That would be a clean interface: just mmap
> >MSIX BAR and forget about it.
>
> Agree.
>
> >If instead of eventfd we had a file descriptor that can pass vector
> >information from vfio to kvm and back, that would fix it,
> >as we would not need to set us GSIs at all,
> >and not need for userspace to handle MSIX specially.
> >
>
> But if we emulate the entire msix bar in vfio, that's not needed, right?

Yes, I think it is. How does kvm know which interrupt to inject?
Either vfio needs to pass that info to qemu and qemu would pass it
to kvm, or vfio would have some way to pass that info to kvm
directly.

> How far away is vfio? If it's merged soon, we might avoid making
> changes to the old assigned device infrastructure and instead update
> vfio.

Hard to be sure, hopefully 2.6.38 material.

Some issues off the top of my head are
- readonly/virtualized table correctness
hopefully will start converging now that
we are switching to standard registers from pci_regs.h
- work out some capability negotiation mechanism so userspace/kernel
can detect bug fixes/missing features and recover or fail gracefully
- with multiple assigned devices in a guest:
I don't think we have figured out how do they share an iommu context
- maybe: reset handling (flr support) - need to look a it

> On the other hand, changes to the old infrastructure are much
> more amenable to backporting for long term support distro kernels,
> so we may need to actively develop both for a while.

Right.

> --
> error compiling committee.c: too many arguments to function