Re: [RFC v4 0/7] KVM: arm/arm64: gsi routing support

[Eric Auger <eric.auger@linaro.org>]

Hi Christoffer,
On 04/14/2016 02:04 PM, Christoffer Dall wrote:
> On Mon, Apr 04, 2016 at 10:47:30AM +0200, Eric Auger wrote:
>> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
>> becomes mandated for proper VIRTIO-PCI vhost integration.
>>
>> In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
>> we previously used direct_mapping trick: this consists in extracting the
>> SPI ID found in the MSI message and associating an irqfd to that SPI ID.
>> When vhost worker thread gets a new buffer it signals the irqfd and kvm
>> then injects this SPI ID on guest. That way although the guest uses MSIs,
>> no MSI emulation is used.
>>
>> This worked fine with GICv2m but does not work anymore with GICV3 ITS.
>> Indeed this latter implements IRQ translation: what is found in the MSI
>> message no more is the target SPI ID but is an intermediate event ID used
>> in the translation process.
>>
>> Hence true MSI routing is needed so that the vhost back channel irqfd is
>> associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
> 
> Doesn't the guest have to program the device with some ID?  So how is
> this dummy GSI ID assigned?
Please apologize for the delay.

We have this path:
irqfd -> dummy GSI -> MSI -> LPI ID

the MSI route does the link between the dummy GSI and the MSI.
the device id is embedded in the route entry definition (devid field of
kvm_irq_routing_msi)
The eventid is put in the MSI data

see https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg06178.html
(Pavel's vITS RFC)

in QEMU, kvm-all.c/kvm_irqchip_add_msi_route allocates the "dummy" gsi
(kvm_irqchip_get_virq) and builds the routing entry.


> 
>> injects the MSI through the in-kernel ITS emulation, the MSI is properly
>> translated and eventually the SPI ID associated to the event ID is injected
>> on guest.
> 
> Isn't it an LPI that is injected to the guest when you have an emulated
> ITS then?
It is definitively an LPI ID.
> 
>>
>> irqchip routing does not sound to be really useful on arm but usage of
> 
> Is this also true if you had multiple emulated ITS devices?
> 
> (I just don't want to block our options of doing this in the future if
> ever required.)

yes irqchip routing may be useful to handle multiple guest irqchips. I
meant: currently irqchip routing comes with this series although this is
not the series' primary goal. There is no special limitation except it
does not apply to KVM_IRQ_LINE ioctl because of the gsi semantic chosen
in the past, for ARM:

bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
field: | irq_type  | vcpu_index |     irq_id     |

Best Regards

Eric

> 
>> MSI routing also mandates to integrate irqchip routing. The initial
>> implementation of irqfd on arm must be upgraded with the integration
>> of kvm irqchip.c code and the implementation of its standard hooks
>> in the architecture specific part.
>>
>> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
>> table with flat irqchip routing entries is built enabling to inject gsi
>> corresponding to the SPI indexes seen by the guest.
>>
>> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
>> default routing table and is responsible for building the whole routing
>> table.
>>
>> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
>> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
>>
>> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>>   with and without explicit routing
>> - MSI routing without GICv3 ITS was tested using APM Xgene-I
>>   (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
>> - MSI routing with GICv3 ITS is *NOT* tested.
>>
>> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.5-rc6-its-emul-v4-gsi-routing-v4
>>
>> The series applies on top of Andre's branch featuring NEW-VGIC and in-kernel
>> ITS emulation series:
>> http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4
>>
>> [1]: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
>>      http://www.spinics.net/lists/arm-kernel/msg492770.html
>> [2]: [RFC PATCH 00/45] KVM: arm/arm64: Rework virtual GIC emulation
>>      http://www.spinics.net/lists/arm-kernel/msg492639.html
>>
>> GSI flat routing setup on QEMU can be found at:
>> https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html
>>
>> History:
>> v3 -> v4:
>> - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
>>   a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
>>   reflesh/reminder.
>> - rewrite the cover letter
>>
>> v2 -> v3:
>> - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
>>   use KVM_MSI_VALID_DEVID flag instead
>> - propagate user flags downto the kernel to make sure the userspace
>>   correctly set devid in GICv3 ITS case (still under discussion)
>>
>> v1 -> v2:
>> - user API changed:
>>   x devid id passed in kvm_irq_routing_msi
>>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
>> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
>> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>>   in eventfd.c
>> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
>> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>>   in kvm_send_userspace_msi
>> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
>>
>> RFC -> PATCH:
>> - clearly state limited support on arm/arm64:
>>   KVM_IRQ_LINE not impacted by GSI routing
>> - add default routing table feature (new patch file)
>> - changed uapi to use padding field area
>> - reword api.txt
>>
>>
>>
>> Eric Auger (7):
>>   KVM: api: pass the devid in the msi routing entry
>>   KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
>>   KVM: irqchip: convey devid to kvm_set_msi
>>   KVM: arm/arm64: enable irqchip routing
>>   KVM: arm/arm64: build a default routing table
>>   KVM: arm/arm64: enable MSI routing
>>   KVM: arm: enable KVM_SIGNAL_MSI and MSI routing
>>
>>  Documentation/virtual/kvm/api.txt | 35 ++++++++++++---
>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>  arch/arm/kvm/Kconfig              |  3 ++
>>  arch/arm/kvm/Makefile             |  1 +
>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>  arch/arm64/kvm/Kconfig            |  3 ++
>>  arch/arm64/kvm/Makefile           |  1 +
>>  include/kvm/vgic/vgic.h           |  2 -
>>  include/linux/kvm_host.h          |  8 +++-
>>  include/uapi/linux/kvm.h          |  5 ++-
>>  virt/kvm/arm/vgic/vgic-v2.c       | 15 +++++++
>>  virt/kvm/arm/vgic/vgic.c          |  7 ---
>>  virt/kvm/arm/vgic/vgic.h          |  1 +
>>  virt/kvm/arm/vgic/vgic_init.c     | 24 +++++++++++
>>  virt/kvm/arm/vgic/vgic_irqfd.c    | 91 +++++++++++++++++++++++++++++++--------
>>  virt/kvm/irqchip.c                |  8 +++-
>>  16 files changed, 170 insertions(+), 37 deletions(-)
>>
>> -- 
>> 1.9.1
>>