Re: [RFC][PATCH] KVM: Introduce direct MSI message injection for in-kernel irqchips

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2011-10-24 15:11, Jan Kiszka wrote:
> On 2011-10-24 14:43, Michael S. Tsirkin wrote:
>> On Mon, Oct 24, 2011 at 02:06:08PM +0200, Jan Kiszka wrote:
>>> On 2011-10-24 13:09, Avi Kivity wrote:
>>>> On 10/24/2011 12:19 PM, Jan Kiszka wrote:
>>>>>>
>>>>>> With the new feature it may be worthwhile, but I'd like to see the whole
>>>>>> thing, with numbers attached.
>>>>>
>>>>> It's not a performance issue, it's a resource limitation issue: With the
>>>>> new API we can stop worrying about user space device models consuming
>>>>> limited IRQ routes of the KVM subsystem.
>>>>>
>>>>
>>>> Only if those devices are in the same process (or have access to the
>>>> vmfd).  Interrupt routing together with irqfd allows you to disaggregate
>>>> the device model.  Instead of providing a competing implementation with
>>>> new limitations, we need to remove the limitations of the old
>>>> implementation.
>>>
>>> That depends on where we do the cut. Currently we let the IRQ source
>>> signal an abstract edge on a pre-allocated pseudo IRQ line. But we
>>> cannot build correct MSI-X on top of the current irqfd model as we lack
>>> the level information (for PBA emulation). *)
>>
>>
>> I don't agree here. IMO PBA emulation would need to
>> clear pending bits on interrupt status register read.
>> So clearing pending bits could be done by ioctl from qemu
>> while setting them would be done from irqfd.
> 
> How should QEMU know if the reason for "pending" has been cleared at
> device level if the device is outside the scope of QEMU? This model only
> works for PV devices when you agree that spurious IRQs are OK.
> 
>>
>>> So we either need to
>>> extend the existing model anyway -- or push per-vector masking back to
>>> the IRQ source. In the latter case, it would be a very good chance to
>>> give up on limited pseudo GSIs with static routes and do MSI messaging
>>> from external IRQ sources to KVM directly.
>>> But all those considerations affect different APIs than what I'm
>>> proposing here. We will always need a way to inject MSIs in the context
>>> of the VM as there will always be scenarios where devices are better run
>>> in that very same context, for performance or simplicity or whatever
>>> reasons. E.g., I could imagine that one would like to execute an
>>> emulated IRQ remapper rather in the hypervisor context than
>>> "over-microkernelized" in a separate process.
>>>
>>> Jan
>>>
>>> *) Realized this while trying to generalize the proposed MSI-X MMIO
>>> acceleration for assigned devices to arbitrary device models, vhost-net,
>>
>> I'm actually working on a qemu patch to get pba emulation working correctly.
>> I think it's doable with existing irqfd.
> 
> irqfd has no notion of level. You can only communicate a rising edge and
> then need a side channel for the state of the edge reason.
> 
>>
>>> and specifically vfio.
>>
>> Interesting. How would you clear the pseudo interrupt level?
> 
> Ideally: not at all (for MSI). If we manage the mask at device level, we
> only need to send the message if there is actually something to deliver
> to the interrupt controller and masked input events would be lost on
> real HW as well.

This wouldn't work out nicely as well. We rather need a combined model:

Devices need to maintain the PBA actively, i.e. set & clear them
themselves and do not rely on the core here (with the core being either
QEMU user space or an in-kernel MSI-X MMIO accelerator). The core only
checks the PBA if it is about to deliver some message and refrains from
doing so if the bit became 0 in the meantime (specifically during the
masked period). For QEMU device models, that means no additional IOCTLs,
just memory sharing of the PBA which is required anyway.

But that means QEMU-external device models need to gain at least basic
MSI-X knowledge. And if they gain this awareness, they could also use it
to send full-blown messages directly (e.g. device-id/vector tuples)
instead of encoding them into finite GSI numbers. But that's an add-on
topic.

Moreover, we still need a corresponding side channel for line-base
interrupts.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux