[RFC v4 0/7] KVM: arm/arm64: gsi routing support

[RFC v4 0/7] KVM: arm/arm64: gsi routing support

[Eric Auger]

With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
becomes mandated for proper VIRTIO-PCI vhost integration.

In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
we previously used direct_mapping trick: this consists in extracting the
SPI ID found in the MSI message and associating an irqfd to that SPI ID.
When vhost worker thread gets a new buffer it signals the irqfd and kvm
then injects this SPI ID on guest. That way although the guest uses MSIs,
no MSI emulation is used.

This worked fine with GICv2m but does not work anymore with GICV3 ITS.
Indeed this latter implements IRQ translation: what is found in the MSI
message no more is the target SPI ID but is an intermediate event ID used
in the translation process.

Hence true MSI routing is needed so that the vhost back channel irqfd is
associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
injects the MSI through the in-kernel ITS emulation, the MSI is properly
translated and eventually the SPI ID associated to the event ID is injected
on guest.

irqchip routing does not sound to be really useful on arm but usage of
MSI routing also mandates to integrate irqchip routing. The initial
implementation of irqfd on arm must be upgraded with the integration
of kvm irqchip.c code and the implementation of its standard hooks
in the architecture specific part.

In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
table with flat irqchip routing entries is built enabling to inject gsi
corresponding to the SPI indexes seen by the guest.

As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
default routing table and is responsible for building the whole routing
table.

for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
- only applies to KVM_IRQFD and not to KVM_IRQ_LINE

- irqchip routing was tested on Calxeda midway (VFIO with irqfd)
  with and without explicit routing
- MSI routing without GICv3 ITS was tested using APM Xgene-I
  (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
- MSI routing with GICv3 ITS is *NOT* tested.

Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.5-rc6-its-emul-v4-gsi-routing-v4

The series applies on top of Andre's branch featuring NEW-VGIC and in-kernel
ITS emulation series:
http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4

[1]: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
     http://www.spinics.net/lists/arm-kernel/msg492770.html
[2]: [RFC PATCH 00/45] KVM: arm/arm64: Rework virtual GIC emulation
     http://www.spinics.net/lists/arm-kernel/msg492639.html

GSI flat routing setup on QEMU can be found at:
https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html

History:
v3 -> v4:
- rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
  a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
  reflesh/reminder.
- rewrite the cover letter

v2 -> v3:
- don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
  use KVM_MSI_VALID_DEVID flag instead
- propagate user flags downto the kernel to make sure the userspace
  correctly set devid in GICv3 ITS case (still under discussion)

v1 -> v2:
- user API changed:
  x devid id passed in kvm_irq_routing_msi
  x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
- kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
  composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
- Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
  in eventfd.c
- added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
- fix bug reported by Andre: bad setting of msi.flags and msi.devid
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH:
- clearly state limited support on arm/arm64:
  KVM_IRQ_LINE not impacted by GSI routing
- add default routing table feature (new patch file)
- changed uapi to use padding field area
- reword api.txt



Eric Auger (7):
  KVM: api: pass the devid in the msi routing entry
  KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
  KVM: irqchip: convey devid to kvm_set_msi
  KVM: arm/arm64: enable irqchip routing
  KVM: arm/arm64: build a default routing table
  KVM: arm/arm64: enable MSI routing
  KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

 Documentation/virtual/kvm/api.txt | 35 ++++++++++++---
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  3 ++
 arch/arm/kvm/Makefile             |  1 +
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  3 ++
 arch/arm64/kvm/Makefile           |  1 +
 include/kvm/vgic/vgic.h           |  2 -
 include/linux/kvm_host.h          |  8 +++-
 include/uapi/linux/kvm.h          |  5 ++-
 virt/kvm/arm/vgic/vgic-v2.c       | 15 +++++++
 virt/kvm/arm/vgic/vgic.c          |  7 ---
 virt/kvm/arm/vgic/vgic.h          |  1 +
 virt/kvm/arm/vgic/vgic_init.c     | 24 +++++++++++
 virt/kvm/arm/vgic/vgic_irqfd.c    | 91 +++++++++++++++++++++++++++++++--------
 virt/kvm/irqchip.c                |  8 +++-
 16 files changed, 170 insertions(+), 37 deletions(-)

-- 
1.9.1

[PATCH v4 1/7] KVM: api: pass the devid in the msi routing entry

[Eric Auger]

On ARM, the MSI msg (address and data) comes along with
out-of-band device ID information. The device ID encodes the
device that writes the MSI msg. Let's convey the device id in
kvm_irq_routing_msi and use KVM_MSI_VALID_DEVID flag value in
kvm_irq_routing_entry to indicate the msi devid is populated.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v2 -> v3:
- replace usage of KVM_IRQ_ROUTING_EXTENDED_MSI type by
  usage of KVM_MSI_VALID_DEVID flag
- add note about KVM_CAP_MSI_DEVID capability

v1 -> v2:
- devid id passed in kvm_irq_routing_msi instead of in
  kvm_irq_routing_entry

RFC -> PATCH
- remove kvm_irq_routing_extended_msi and use union instead
---
 Documentation/virtual/kvm/api.txt | 18 ++++++++++++++++--
 include/uapi/linux/kvm.h          |  5 ++++-
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 536f19b..c436bb6 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1462,7 +1462,10 @@ struct kvm_irq_routing_entry {
 #define KVM_IRQ_ROUTING_S390_ADAPTER 3
 #define KVM_IRQ_ROUTING_HV_SINT 4
 
-No flags are specified so far, the corresponding field must be set to zero.
+flags:
+- KVM_MSI_VALID_DEVID: used along with KVM_IRQ_ROUTING_MSI
+  routing entry type, it tells the msi devid contains a valid value.
+- zero otherwise
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1473,9 +1476,20 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
+       for the device that wrote the MSI message.
+       For PCI, this is usually a BFD identifier in the lower 16 bits.
+
+The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
+the device ID. If this capability is not set, userland cannot rely on
+the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
+
 struct kvm_irq_routing_s390_adapter {
 	__u64 ind_addr;
 	__u64 summary_addr;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6a02871..52a973f 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -865,7 +865,10 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
 struct kvm_irq_routing_s390_adapter {
-- 
1.9.1

Re: [PATCH v4 1/7] KVM: api: pass the devid in the msi routing entry

[Christoffer Dall]

On Mon, Apr 04, 2016 at 10:47:31AM +0200, Eric Auger wrote:
> On ARM, the MSI msg (address and data) comes along with
> out-of-band device ID information. The device ID encodes the
> device that writes the MSI msg. Let's convey the device id in
> kvm_irq_routing_msi and use KVM_MSI_VALID_DEVID flag value in
> kvm_irq_routing_entry to indicate the msi devid is populated.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 
> ---
> v2 -> v3:
> - replace usage of KVM_IRQ_ROUTING_EXTENDED_MSI type by
>   usage of KVM_MSI_VALID_DEVID flag
> - add note about KVM_CAP_MSI_DEVID capability
> 
> v1 -> v2:
> - devid id passed in kvm_irq_routing_msi instead of in
>   kvm_irq_routing_entry
> 
> RFC -> PATCH
> - remove kvm_irq_routing_extended_msi and use union instead
> ---
>  Documentation/virtual/kvm/api.txt | 18 ++++++++++++++++--
>  include/uapi/linux/kvm.h          |  5 ++++-
>  2 files changed, 20 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 536f19b..c436bb6 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1462,7 +1462,10 @@ struct kvm_irq_routing_entry {
>  #define KVM_IRQ_ROUTING_S390_ADAPTER 3
>  #define KVM_IRQ_ROUTING_HV_SINT 4
>  
> -No flags are specified so far, the corresponding field must be set to zero.
> +flags:
> +- KVM_MSI_VALID_DEVID: used along with KVM_IRQ_ROUTING_MSI
> +  routing entry type, it tells the msi devid contains a valid value.

s/it tells the msi devid/specifies that the devid field/

> +- zero otherwise
>  
>  struct kvm_irq_routing_irqchip {
>  	__u32 irqchip;
> @@ -1473,9 +1476,20 @@ struct kvm_irq_routing_msi {
>  	__u32 address_lo;
>  	__u32 address_hi;
>  	__u32 data;
> -	__u32 pad;
> +	union {
> +		__u32 pad;
> +		__u32 devid;
> +	};
>  };
>  
> +devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
> +       for the device that wrote the MSI message.
> +       For PCI, this is usually a BFD identifier in the lower 16 bits.
> +
> +The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide

"the need to" or "the possibility to"?  Is there ever a situation where
KVM_CAP_MSI_DEVID is set but you don't need to provide a device ID and
set the flag?

> +the device ID. If this capability is not set, userland cannot rely on
> +the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
> +
>  struct kvm_irq_routing_s390_adapter {
>  	__u64 ind_addr;
>  	__u64 summary_addr;
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 6a02871..52a973f 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -865,7 +865,10 @@ struct kvm_irq_routing_msi {
>  	__u32 address_lo;
>  	__u32 address_hi;
>  	__u32 data;
> -	__u32 pad;
> +	union {
> +		__u32 pad;
> +		__u32 devid;
> +	};
>  };
>  
>  struct kvm_irq_routing_s390_adapter {
> -- 
> 1.9.1
> 

Re: [PATCH v4 1/7] KVM: api: pass the devid in the msi routing entry

[Eric Auger]

Christoffer,
On 04/14/2016 02:04 PM, Christoffer Dall wrote:
> On Mon, Apr 04, 2016 at 10:47:31AM +0200, Eric Auger wrote:
>> On ARM, the MSI msg (address and data) comes along with
>> out-of-band device ID information. The device ID encodes the
>> device that writes the MSI msg. Let's convey the device id in
>> kvm_irq_routing_msi and use KVM_MSI_VALID_DEVID flag value in
>> kvm_irq_routing_entry to indicate the msi devid is populated.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
>> ---
>> v2 -> v3:
>> - replace usage of KVM_IRQ_ROUTING_EXTENDED_MSI type by
>>   usage of KVM_MSI_VALID_DEVID flag
>> - add note about KVM_CAP_MSI_DEVID capability
>>
>> v1 -> v2:
>> - devid id passed in kvm_irq_routing_msi instead of in
>>   kvm_irq_routing_entry
>>
>> RFC -> PATCH
>> - remove kvm_irq_routing_extended_msi and use union instead
>> ---
>>  Documentation/virtual/kvm/api.txt | 18 ++++++++++++++++--
>>  include/uapi/linux/kvm.h          |  5 ++++-
>>  2 files changed, 20 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index 536f19b..c436bb6 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -1462,7 +1462,10 @@ struct kvm_irq_routing_entry {
>>  #define KVM_IRQ_ROUTING_S390_ADAPTER 3
>>  #define KVM_IRQ_ROUTING_HV_SINT 4
>>  
>> -No flags are specified so far, the corresponding field must be set to zero.
>> +flags:
>> +- KVM_MSI_VALID_DEVID: used along with KVM_IRQ_ROUTING_MSI
>> +  routing entry type, it tells the msi devid contains a valid value.
> 
> s/it tells the msi devid/specifies that the devid field/
ok
> 
>> +- zero otherwise
>>  
>>  struct kvm_irq_routing_irqchip {
>>  	__u32 irqchip;
>> @@ -1473,9 +1476,20 @@ struct kvm_irq_routing_msi {
>>  	__u32 address_lo;
>>  	__u32 address_hi;
>>  	__u32 data;
>> -	__u32 pad;
>> +	union {
>> +		__u32 pad;
>> +		__u32 devid;
>> +	};
>>  };
>>  
>> +devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
>> +       for the device that wrote the MSI message.
>> +       For PCI, this is usually a BFD identifier in the lower 16 bits.
>> +
>> +The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
> 
> "the need to" or "the possibility to"?  Is there ever a situation where
> KVM_CAP_MSI_DEVID is set but you don't need to provide a device ID and
> set the flag?
I could replace by "requirement":
if it is not requested to pass a devid, we don't set the flag. If it is
we pass it.

if the user fails passing the devid and this ends up in the in-kernel
ITS emul, injection will fail.

Thanks

Eric
> 
>> +the device ID. If this capability is not set, userland cannot rely on
>> +the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
>> +
>>  struct kvm_irq_routing_s390_adapter {
>>  	__u64 ind_addr;
>>  	__u64 summary_addr;
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 6a02871..52a973f 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -865,7 +865,10 @@ struct kvm_irq_routing_msi {
>>  	__u32 address_lo;
>>  	__u32 address_hi;
>>  	__u32 data;
>> -	__u32 pad;
>> +	union {
>> +		__u32 pad;
>> +		__u32 devid;
>> +	};
>>  };
>>  
>>  struct kvm_irq_routing_s390_adapter {
>> -- 
>> 1.9.1
>>

[PATCH v4 2/7] KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry

[Eric Auger]

Extend kvm_kernel_irq_routing_entry to transport the device id
field, devid. A new flags field makes possible to indicate the
devid is valid. Those additions are used for ARM GICv3 ITS MSI
injection.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v2 -> v3:
- add flags

v1 -> v2:
- replace msi_msg field by a struct composed of msi_msg and devid

RFC -> PATCH:
- reword the commit message after change in first patch (uapi)
---
 include/linux/kvm_host.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 861f690..461e480 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -309,7 +309,11 @@ struct kvm_kernel_irq_routing_entry {
 			unsigned irqchip;
 			unsigned pin;
 		} irqchip;
-		struct msi_msg msi;
+		struct {
+			struct msi_msg msi;
+			u32 flags;
+			u32 devid;
+		};
 		struct kvm_s390_adapter_int adapter;
 		struct kvm_hv_sint hv_sint;
 	};
-- 
1.9.1

Re: [PATCH v4 2/7] KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry

[Christoffer Dall]

On Mon, Apr 04, 2016 at 10:47:32AM +0200, Eric Auger wrote:
> Extend kvm_kernel_irq_routing_entry to transport the device id
> field, devid. A new flags field makes possible to indicate the
> devid is valid. Those additions are used for ARM GICv3 ITS MSI
> injection.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

[PATCH v4 3/7] KVM: irqchip: convey devid to kvm_set_msi

[Eric Auger]

on ARM, a devid field is populated in kvm_msi struct in case the
flag is set to KVM_MSI_VALID_DEVID. Let's propagate both flags and
devid field in kvm_kernel_irq_routing_entry.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

v2 -> v3:
- do not set the type to KVM_IRQ_ROUTING_EXTENDED_MSI anymore as
  suggested by Andre
- correct msi->flags check
- propagate the flags
---
 virt/kvm/irqchip.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index fe84e1a..1c556cb 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -62,12 +62,14 @@ int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
 {
 	struct kvm_kernel_irq_routing_entry route;
 
-	if (!irqchip_in_kernel(kvm) || msi->flags != 0)
+	if (!irqchip_in_kernel(kvm) || (msi->flags & ~KVM_MSI_VALID_DEVID))
 		return -EINVAL;
 
 	route.msi.address_lo = msi->address_lo;
 	route.msi.address_hi = msi->address_hi;
 	route.msi.data = msi->data;
+	route.flags = msi->flags;
+	route.devid = msi->devid;
 
 	return kvm_set_msi(&route, kvm, KVM_USERSPACE_IRQ_SOURCE_ID, 1, false);
 }
-- 
1.9.1

Re: [PATCH v4 3/7] KVM: irqchip: convey devid to kvm_set_msi

[Christoffer Dall]

On Mon, Apr 04, 2016 at 10:47:33AM +0200, Eric Auger wrote:
> on ARM, a devid field is populated in kvm_msi struct in case the
> flag is set to KVM_MSI_VALID_DEVID. Let's propagate both flags and
> devid field in kvm_kernel_irq_routing_entry.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

[PATCH v4 4/7] KVM: arm/arm64: enable irqchip routing

[Eric Auger]

This patch adds compilation and link against irqchip.

On ARM, irqchip routing is not really useful since there is
a single irqchip. However main motivation behind using irqchip
code is to enable MSI routing code.

Routing standard callbacks now are implemented in vgic_irqfd:
- kvm_set_routing_entry
- kvm_set_irq
- kvm_set_msi

They only are supported with new_vgic code.

Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.

MSI routing setup is not yet allowed.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v3 -> v4:
- provide support only for new-vgic
- code previously in vgic.c now in vgic_irqfd.c

v2 -> v3:
- unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
  by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
- vgic_irqfd_set_irq now is static
- propagate flags
- add comments

v1 -> v2:
- fix bug reported by Andre related to msi.flags and msi.devid setting
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH
- reword api.txt:
  x move MSI routing comments in a subsequent patch,
  x clearly state GSI routing does not apply to KVM_IRQ_LINE

Conflicts:
	arch/arm/include/asm/kvm_host.h
	arch/arm/kvm/Kconfig
	arch/arm64/include/asm/kvm_host.h
	arch/arm64/kvm/Kconfig
---
 Documentation/virtual/kvm/api.txt | 12 ++++--
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  2 +
 arch/arm/kvm/Makefile             |  1 +
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  3 ++
 arch/arm64/kvm/Makefile           |  1 +
 include/kvm/vgic/vgic.h           |  2 -
 virt/kvm/arm/vgic/vgic.c          |  7 ----
 virt/kvm/arm/vgic/vgic_irqfd.c    | 83 ++++++++++++++++++++++++++++++---------
 virt/kvm/irqchip.c                |  2 +
 11 files changed, 85 insertions(+), 31 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index c436bb6..61f8f27 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1427,13 +1427,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
 4.52 KVM_SET_GSI_ROUTING
 
 Capability: KVM_CAP_IRQ_ROUTING
-Architectures: x86 s390
+Architectures: x86 s390 arm arm64
 Type: vm ioctl
 Parameters: struct kvm_irq_routing (in)
 Returns: 0 on success, -1 on error
 
 Sets the GSI routing table entries, overwriting any previously set entries.
 
+On arm/arm64, GSI routing has the following limitation:
+- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
+
 struct kvm_irq_routing {
 	__u32 nr;
 	__u32 flags;
@@ -2361,9 +2364,10 @@ Note that closing the resamplefd is not sufficient to disable the
 irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
 and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
 
-On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
-Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
-given by gsi + 32.
+On arm/arm64, gsi routing being supported, the following can happen:
+- in case no routing entry is associated to this gsi, injection fails
+- in case the gsi is associated to an irqchip routing entry,
+  irqchip.pin + 32 corresponds to the injected SPI ID.
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 494b004..67dc11d 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -37,6 +37,8 @@
 
 #define KVM_VCPU_MAX_FEATURES 2
 
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */
+
 #include <kvm/arm_vgic.h>
 
 #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 02abfff..92c3aec 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -50,6 +50,8 @@ config KVM_NEW_VGIC
 	bool "New VGIC implementation"
 	depends on KVM
 	default y
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	---help---
 	  uses the new VGIC implementation
 
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index aa7d724..b8aa5ef 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -29,6 +29,7 @@ obj-y += $(KVM)/arm/vgic/vgic_irqfd.o
 obj-y += $(KVM)/arm/vgic/vgic-v2.o
 obj-y += $(KVM)/arm/vgic/vgic_mmio.o
 obj-y += $(KVM)/arm/vgic/vgic_kvm_device.o
+obj-y += $(KVM)//irqchip.o
 else
 obj-y += $(KVM)/arm/vgic.o
 obj-y += $(KVM)/arm/vgic-v2.o
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 2cdd7ae..95e1779 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -35,6 +35,7 @@
 #define KVM_PRIVATE_MEM_SLOTS 4
 #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
 #define KVM_HALT_POLL_NS_DEFAULT 500000
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */
 
 #include <kvm/arm_vgic.h>
 #include <kvm/arm_arch_timer.h>
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 71c9ebc..bd597dc9 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -60,6 +60,9 @@ config KVM_NEW_VGIC
 	bool "New VGIC implementation"
 	depends on KVM
 	default y
+	select HAVE_KVM_MSI
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
         ---help---
           uses the new VGIC implementation
 
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 3bec10e..37f2a47 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -29,6 +29,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
 else
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index c50890f..625a777 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -284,6 +284,4 @@ static inline int kvm_vgic_get_max_vcpus(void)
 
 bool vgic_has_its(struct kvm *kvm);
 
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
-
 #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 82bfb33..a175d93 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -623,10 +623,3 @@ bool vgic_has_its(struct kvm *kvm)
 	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
 }
 
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
-{
-	if (vgic_has_its(kvm))
-		return vits_inject_msi(kvm, msi);
-	else
-		return -ENODEV;
-}
diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
index 3eee1bd..a76994f 100644
--- a/virt/kvm/arm/vgic/vgic_irqfd.c
+++ b/virt/kvm/arm/vgic/vgic_irqfd.c
@@ -17,35 +17,82 @@
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <trace/events/kvm.h>
+#include <kvm/vgic/vgic.h>
+#include "vgic.h"
 
-int kvm_irq_map_gsi(struct kvm *kvm,
-		    struct kvm_kernel_irq_routing_entry *entries,
-		    int gsi)
+/**
+ * vgic_irqfd_set_irq: inject the IRQ corresponding to the
+ * irqchip routing entry
+ *
+ * This is the entry point for irqfd IRQ injection
+ */
+static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
+			struct kvm *kvm, int irq_source_id,
+			int level, bool line_status)
 {
-	return 0;
-}
+	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
+	struct vgic_dist *dist = &kvm->arch.vgic;
 
-int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
-{
-	return pin;
-}
+	trace_kvm_set_irq(spi_id, level, irq_source_id);
 
-int kvm_set_irq(struct kvm *kvm, int irq_source_id,
-		u32 irq, int level, bool line_status)
-{
-	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
+	BUG_ON(!vgic_initialized(kvm));
 
-	trace_kvm_set_irq(irq, level, irq_source_id);
+	if (spi_id > min(dist->nr_spis, 1020))
+		return -EINVAL;
+	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
+}
 
-	BUG_ON(!vgic_initialized(kvm));
+/**
+ * kvm_set_routing_entry: populate a kvm routing entry
+ * from a user routing entry
+ *
+ * @e: kvm kernel routing entry handle
+ * @ue: user api routing entry handle
+ * return 0 on success, -EINVAL on errors.
+ */
+int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
+			  const struct kvm_irq_routing_entry *ue)
+{
+	int r = -EINVAL;
 
-	return kvm_vgic_inject_irq(kvm, 0, spi, level);
+	switch (ue->type) {
+	case KVM_IRQ_ROUTING_IRQCHIP:
+		e->set = vgic_irqfd_set_irq;
+		e->irqchip.irqchip = ue->u.irqchip.irqchip;
+		e->irqchip.pin = ue->u.irqchip.pin;
+		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
+		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
+			goto out;
+		break;
+	default:
+		goto out;
+	}
+	r = 0;
+out:
+	return r;
 }
 
-/* MSI not implemented yet */
+/**
+ * kvm_set_msi: inject the MSI corresponding to the
+ * MSI routing entry
+ *
+ * This is the entry point for irqfd MSI injection
+ * and userspace MSI injection.
+ */
 int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 		struct kvm *kvm, int irq_source_id,
 		int level, bool line_status)
 {
-	return 0;
+	struct kvm_msi msi;
+
+	msi.address_lo = e->msi.address_lo;
+	msi.address_hi = e->msi.address_hi;
+	msi.data = e->msi.data;
+	msi.flags = e->flags;
+	msi.devid = e->devid;
+
+	if (!vgic_has_its(kvm))
+		return -ENODEV;
+
+	return vits_inject_msi(kvm, &msi);
 }
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 1c556cb..b4222d6 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -29,7 +29,9 @@
 #include <linux/srcu.h>
 #include <linux/export.h>
 #include <trace/events/kvm.h>
+#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
 #include "irq.h"
+#endif
 
 int kvm_irq_map_gsi(struct kvm *kvm,
 		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
-- 
1.9.1

Re: [PATCH v4 4/7] KVM: arm/arm64: enable irqchip routing

[Christoffer Dall]

REVIEW INCOMPLETE

On Mon, Apr 04, 2016 at 10:47:34AM +0200, Eric Auger wrote:
> This patch adds compilation and link against irqchip.
> 
> On ARM, irqchip routing is not really useful since there is
> a single irqchip. However main motivation behind using irqchip
> code is to enable MSI routing code.

As commented on the cover letter, could we not have multiple ITS devices
in the guest at some point?  (I think Marc suggeste this may be useful
for a combination of passthrough and emulated devices).

> 
> Routing standard callbacks now are implemented in vgic_irqfd:
> - kvm_set_routing_entry
> - kvm_set_irq
> - kvm_set_msi
> 
> They only are supported with new_vgic code.
> 
> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
> 
> MSI routing setup is not yet allowed.

Then why are we selecting CONFIG_HAVE_KVM_MSI here?

> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 
> ---
> v3 -> v4:
> - provide support only for new-vgic
> - code previously in vgic.c now in vgic_irqfd.c
> 
> v2 -> v3:
> - unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
>   by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
> - vgic_irqfd_set_irq now is static
> - propagate flags
> - add comments
> 
> v1 -> v2:
> - fix bug reported by Andre related to msi.flags and msi.devid setting
>   in kvm_send_userspace_msi
> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> 
> RFC -> PATCH
> - reword api.txt:
>   x move MSI routing comments in a subsequent patch,
>   x clearly state GSI routing does not apply to KVM_IRQ_LINE
> 
> Conflicts:
> 	arch/arm/include/asm/kvm_host.h
> 	arch/arm/kvm/Kconfig
> 	arch/arm64/include/asm/kvm_host.h
> 	arch/arm64/kvm/Kconfig
> ---
>  Documentation/virtual/kvm/api.txt | 12 ++++--
>  arch/arm/include/asm/kvm_host.h   |  2 +
>  arch/arm/kvm/Kconfig              |  2 +
>  arch/arm/kvm/Makefile             |  1 +
>  arch/arm64/include/asm/kvm_host.h |  1 +
>  arch/arm64/kvm/Kconfig            |  3 ++
>  arch/arm64/kvm/Makefile           |  1 +
>  include/kvm/vgic/vgic.h           |  2 -
>  virt/kvm/arm/vgic/vgic.c          |  7 ----
>  virt/kvm/arm/vgic/vgic_irqfd.c    | 83 ++++++++++++++++++++++++++++++---------
>  virt/kvm/irqchip.c                |  2 +
>  11 files changed, 85 insertions(+), 31 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index c436bb6..61f8f27 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1427,13 +1427,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>  4.52 KVM_SET_GSI_ROUTING
>  
>  Capability: KVM_CAP_IRQ_ROUTING
> -Architectures: x86 s390
> +Architectures: x86 s390 arm arm64
>  Type: vm ioctl
>  Parameters: struct kvm_irq_routing (in)
>  Returns: 0 on success, -1 on error
>  
>  Sets the GSI routing table entries, overwriting any previously set entries.
>  
> +On arm/arm64, GSI routing has the following limitation:
> +- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
> +
>  struct kvm_irq_routing {
>  	__u32 nr;
>  	__u32 flags;
> @@ -2361,9 +2364,10 @@ Note that closing the resamplefd is not sufficient to disable the
>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>  
> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
> -given by gsi + 32.
> +On arm/arm64, gsi routing being supported, the following can happen:
> +- in case no routing entry is associated to this gsi, injection fails
> +- in case the gsi is associated to an irqchip routing entry,
> +  irqchip.pin + 32 corresponds to the injected SPI ID.
>  
>  4.76 KVM_PPC_ALLOCATE_HTAB
>  
> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
> index 494b004..67dc11d 100644
> --- a/arch/arm/include/asm/kvm_host.h
> +++ b/arch/arm/include/asm/kvm_host.h
> @@ -37,6 +37,8 @@
>  
>  #define KVM_VCPU_MAX_FEATURES 2
>  
> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */

nit: s/SPI/SPIs/

> +
>  #include <kvm/arm_vgic.h>
>  
>  #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> index 02abfff..92c3aec 100644
> --- a/arch/arm/kvm/Kconfig
> +++ b/arch/arm/kvm/Kconfig
> @@ -50,6 +50,8 @@ config KVM_NEW_VGIC
>  	bool "New VGIC implementation"
>  	depends on KVM
>  	default y
> +	select HAVE_KVM_IRQCHIP
> +	select HAVE_KVM_IRQ_ROUTING
>  	---help---
>  	  uses the new VGIC implementation
>  
> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
> index aa7d724..b8aa5ef 100644
> --- a/arch/arm/kvm/Makefile
> +++ b/arch/arm/kvm/Makefile
> @@ -29,6 +29,7 @@ obj-y += $(KVM)/arm/vgic/vgic_irqfd.o
>  obj-y += $(KVM)/arm/vgic/vgic-v2.o
>  obj-y += $(KVM)/arm/vgic/vgic_mmio.o
>  obj-y += $(KVM)/arm/vgic/vgic_kvm_device.o
> +obj-y += $(KVM)//irqchip.o
>  else
>  obj-y += $(KVM)/arm/vgic.o
>  obj-y += $(KVM)/arm/vgic-v2.o
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 2cdd7ae..95e1779 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -35,6 +35,7 @@
>  #define KVM_PRIVATE_MEM_SLOTS 4
>  #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
>  #define KVM_HALT_POLL_NS_DEFAULT 500000
> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */
>  
>  #include <kvm/arm_vgic.h>
>  #include <kvm/arm_arch_timer.h>
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 71c9ebc..bd597dc9 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -60,6 +60,9 @@ config KVM_NEW_VGIC
>  	bool "New VGIC implementation"
>  	depends on KVM
>  	default y
> +	select HAVE_KVM_MSI
> +	select HAVE_KVM_IRQCHIP
> +	select HAVE_KVM_IRQ_ROUTING
>          ---help---
>            uses the new VGIC implementation
>  
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index 3bec10e..37f2a47 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -29,6 +29,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
>  else
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index c50890f..625a777 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -284,6 +284,4 @@ static inline int kvm_vgic_get_max_vcpus(void)
>  
>  bool vgic_has_its(struct kvm *kvm);
>  
> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
> -
>  #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
> index 82bfb33..a175d93 100644
> --- a/virt/kvm/arm/vgic/vgic.c
> +++ b/virt/kvm/arm/vgic/vgic.c
> @@ -623,10 +623,3 @@ bool vgic_has_its(struct kvm *kvm)
>  	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>  }
>  
> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
> -{
> -	if (vgic_has_its(kvm))
> -		return vits_inject_msi(kvm, msi);
> -	else
> -		return -ENODEV;
> -}

I don't understand why we're removing these two entries here, or rather,
why we had something here already, given that we don't select HAVE_KVM_MSI
before this patch as well?

> diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
> index 3eee1bd..a76994f 100644
> --- a/virt/kvm/arm/vgic/vgic_irqfd.c
> +++ b/virt/kvm/arm/vgic/vgic_irqfd.c
> @@ -17,35 +17,82 @@
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
>  #include <trace/events/kvm.h>
> +#include <kvm/vgic/vgic.h>
> +#include "vgic.h"
>  
> -int kvm_irq_map_gsi(struct kvm *kvm,
> -		    struct kvm_kernel_irq_routing_entry *entries,
> -		    int gsi)
> +/**
> + * vgic_irqfd_set_irq: inject the IRQ corresponding to the
> + * irqchip routing entry
> + *
> + * This is the entry point for irqfd IRQ injection
> + */
> +static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
> +			struct kvm *kvm, int irq_source_id,
> +			int level, bool line_status)
>  {
> -	return 0;
> -}
> +	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
> +	struct vgic_dist *dist = &kvm->arch.vgic;
>  
> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
> -{
> -	return pin;
> -}
> +	trace_kvm_set_irq(spi_id, level, irq_source_id);

but this is not kvm_set_irq ?  Perhaps it doesn't matter because the
functionality is the same, but in that case, rename it to something more
generic.

>  
> -int kvm_set_irq(struct kvm *kvm, int irq_source_id,
> -		u32 irq, int level, bool line_status)
> -{
> -	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
> +	BUG_ON(!vgic_initialized(kvm));
>  
> -	trace_kvm_set_irq(irq, level, irq_source_id);
> +	if (spi_id > min(dist->nr_spis, 1020))
> +		return -EINVAL;
> +	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
> +}
>  
> -	BUG_ON(!vgic_initialized(kvm));
> +/**
> + * kvm_set_routing_entry: populate a kvm routing entry
> + * from a user routing entry
> + *
> + * @e: kvm kernel routing entry handle
> + * @ue: user api routing entry handle
> + * return 0 on success, -EINVAL on errors.
> + */
> +int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
> +			  const struct kvm_irq_routing_entry *ue)
> +{
> +	int r = -EINVAL;
>  
> -	return kvm_vgic_inject_irq(kvm, 0, spi, level);
> +	switch (ue->type) {
> +	case KVM_IRQ_ROUTING_IRQCHIP:
> +		e->set = vgic_irqfd_set_irq;
> +		e->irqchip.irqchip = ue->u.irqchip.irqchip;
> +		e->irqchip.pin = ue->u.irqchip.pin;
> +		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
> +		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
> +			goto out;
> +		break;
> +	default:
> +		goto out;
> +	}
> +	r = 0;
> +out:
> +	return r;
>  }
>  
> -/* MSI not implemented yet */
> +/**
> + * kvm_set_msi: inject the MSI corresponding to the
> + * MSI routing entry
> + *
> + * This is the entry point for irqfd MSI injection
> + * and userspace MSI injection.
> + */
>  int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>  		struct kvm *kvm, int irq_source_id,
>  		int level, bool line_status)
>  {
> -	return 0;
> +	struct kvm_msi msi;
> +
> +	msi.address_lo = e->msi.address_lo;
> +	msi.address_hi = e->msi.address_hi;
> +	msi.data = e->msi.data;
> +	msi.flags = e->flags;
> +	msi.devid = e->devid;

why do we need to copy the data to a new struct?

> +
> +	if (!vgic_has_its(kvm))
> +		return -ENODEV;
> +
> +	return vits_inject_msi(kvm, &msi);
>  }
> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
> index 1c556cb..b4222d6 100644
> --- a/virt/kvm/irqchip.c
> +++ b/virt/kvm/irqchip.c
> @@ -29,7 +29,9 @@
>  #include <linux/srcu.h>
>  #include <linux/export.h>
>  #include <trace/events/kvm.h>
> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>  #include "irq.h"
> +#endif
>  
>  int kvm_irq_map_gsi(struct kvm *kvm,
>  		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
> -- 
> 1.9.1
> 

Re: [PATCH v4 4/7] KVM: arm/arm64: enable irqchip routing

[Christoffer Dall]

On Thu, Apr 14, 2016 at 02:04:58PM +0200, Christoffer Dall wrote:
> REVIEW INCOMPLETE
> 
this was a note to myself, please ignore.

Re: [PATCH v4 4/7] KVM: arm/arm64: enable irqchip routing

[Eric Auger]

Hi Christoffer,
On 04/14/2016 02:04 PM, Christoffer Dall wrote:
> REVIEW INCOMPLETE
> 
> On Mon, Apr 04, 2016 at 10:47:34AM +0200, Eric Auger wrote:
>> This patch adds compilation and link against irqchip.
>>
>> On ARM, irqchip routing is not really useful since there is
>> a single irqchip. However main motivation behind using irqchip
>> code is to enable MSI routing code.
> 
> As commented on the cover letter, could we not have multiple ITS devices
> in the guest at some point?  (I think Marc suggeste this may be useful
> for a combination of passthrough and emulated devices).

So yes we can and irqchip routing can be used along with irqfd injection.

> 
>>
>> Routing standard callbacks now are implemented in vgic_irqfd:
>> - kvm_set_routing_entry
>> - kvm_set_irq
>> - kvm_set_msi
>>
>> They only are supported with new_vgic code.
>>
>> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
>> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
>>
>> MSI routing setup is not yet allowed.
> 
> Then why are we selecting CONFIG_HAVE_KVM_MSI here?

CONFIG_HAVE_KVM_MSI does not relate to MSI routing but enables the
capability to inject an MSI using KVM_SIGNAL_MSI ioctl
(KVM_CAP_SIGNAL_MSI capability).

The config was set by Andre when he enabled KVM_SIGNAL_MSI in GICv3 ITS:
"KVM: arm64: enable ITS emulation as a virtual MSI controller". This was
relevant since it enabled the modality.

However what I missed is that the previous "select HAVE_KVM_MSI" was in
config KVM and I think it is wrong since it only works with NEW_VGIC and
ITS emulation. So in practice you're right, HAVE_KVM_MSI should already
be in NEW_VGIC after last Andre's patch.
> 
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
>> ---
>> v3 -> v4:
>> - provide support only for new-vgic
>> - code previously in vgic.c now in vgic_irqfd.c
>>
>> v2 -> v3:
>> - unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
>>   by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
>> - vgic_irqfd_set_irq now is static
>> - propagate flags
>> - add comments
>>
>> v1 -> v2:
>> - fix bug reported by Andre related to msi.flags and msi.devid setting
>>   in kvm_send_userspace_msi
>> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
>>
>> RFC -> PATCH
>> - reword api.txt:
>>   x move MSI routing comments in a subsequent patch,
>>   x clearly state GSI routing does not apply to KVM_IRQ_LINE
>>
>> Conflicts:
>> 	arch/arm/include/asm/kvm_host.h
>> 	arch/arm/kvm/Kconfig
>> 	arch/arm64/include/asm/kvm_host.h
>> 	arch/arm64/kvm/Kconfig
>> ---
>>  Documentation/virtual/kvm/api.txt | 12 ++++--
>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>  arch/arm/kvm/Kconfig              |  2 +
>>  arch/arm/kvm/Makefile             |  1 +
>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>  arch/arm64/kvm/Kconfig            |  3 ++
>>  arch/arm64/kvm/Makefile           |  1 +
>>  include/kvm/vgic/vgic.h           |  2 -
>>  virt/kvm/arm/vgic/vgic.c          |  7 ----
>>  virt/kvm/arm/vgic/vgic_irqfd.c    | 83 ++++++++++++++++++++++++++++++---------
>>  virt/kvm/irqchip.c                |  2 +
>>  11 files changed, 85 insertions(+), 31 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index c436bb6..61f8f27 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -1427,13 +1427,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>>  4.52 KVM_SET_GSI_ROUTING
>>  
>>  Capability: KVM_CAP_IRQ_ROUTING
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm arm64
>>  Type: vm ioctl
>>  Parameters: struct kvm_irq_routing (in)
>>  Returns: 0 on success, -1 on error
>>  
>>  Sets the GSI routing table entries, overwriting any previously set entries.
>>  
>> +On arm/arm64, GSI routing has the following limitation:
>> +- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
>> +
>>  struct kvm_irq_routing {
>>  	__u32 nr;
>>  	__u32 flags;
>> @@ -2361,9 +2364,10 @@ Note that closing the resamplefd is not sufficient to disable the
>>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>  
>> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
>> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
>> -given by gsi + 32.
>> +On arm/arm64, gsi routing being supported, the following can happen:
>> +- in case no routing entry is associated to this gsi, injection fails
>> +- in case the gsi is associated to an irqchip routing entry,
>> +  irqchip.pin + 32 corresponds to the injected SPI ID.
>>  
>>  4.76 KVM_PPC_ALLOCATE_HTAB
>>  
>> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
>> index 494b004..67dc11d 100644
>> --- a/arch/arm/include/asm/kvm_host.h
>> +++ b/arch/arm/include/asm/kvm_host.h
>> @@ -37,6 +37,8 @@
>>  
>>  #define KVM_VCPU_MAX_FEATURES 2
>>  
>> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */
> 
> nit: s/SPI/SPIs/
sure
> 
>> +
>>  #include <kvm/arm_vgic.h>
>>  
>>  #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 02abfff..92c3aec 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -50,6 +50,8 @@ config KVM_NEW_VGIC
>>  	bool "New VGIC implementation"
>>  	depends on KVM
>>  	default y
>> +	select HAVE_KVM_IRQCHIP
>> +	select HAVE_KVM_IRQ_ROUTING
>>  	---help---
>>  	  uses the new VGIC implementation
>>  
>> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
>> index aa7d724..b8aa5ef 100644
>> --- a/arch/arm/kvm/Makefile
>> +++ b/arch/arm/kvm/Makefile
>> @@ -29,6 +29,7 @@ obj-y += $(KVM)/arm/vgic/vgic_irqfd.o
>>  obj-y += $(KVM)/arm/vgic/vgic-v2.o
>>  obj-y += $(KVM)/arm/vgic/vgic_mmio.o
>>  obj-y += $(KVM)/arm/vgic/vgic_kvm_device.o
>> +obj-y += $(KVM)//irqchip.o
>>  else
>>  obj-y += $(KVM)/arm/vgic.o
>>  obj-y += $(KVM)/arm/vgic-v2.o
>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>> index 2cdd7ae..95e1779 100644
>> --- a/arch/arm64/include/asm/kvm_host.h
>> +++ b/arch/arm64/include/asm/kvm_host.h
>> @@ -35,6 +35,7 @@
>>  #define KVM_PRIVATE_MEM_SLOTS 4
>>  #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
>>  #define KVM_HALT_POLL_NS_DEFAULT 500000
>> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */
>>  
>>  #include <kvm/arm_vgic.h>
>>  #include <kvm/arm_arch_timer.h>
>> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
>> index 71c9ebc..bd597dc9 100644
>> --- a/arch/arm64/kvm/Kconfig
>> +++ b/arch/arm64/kvm/Kconfig
>> @@ -60,6 +60,9 @@ config KVM_NEW_VGIC
>>  	bool "New VGIC implementation"
>>  	depends on KVM
>>  	default y
>> +	select HAVE_KVM_MSI
>> +	select HAVE_KVM_IRQCHIP
>> +	select HAVE_KVM_IRQ_ROUTING
>>          ---help---
>>            uses the new VGIC implementation
>>  
>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>> index 3bec10e..37f2a47 100644
>> --- a/arch/arm64/kvm/Makefile
>> +++ b/arch/arm64/kvm/Makefile
>> @@ -29,6 +29,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
>>  else
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index c50890f..625a777 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -284,6 +284,4 @@ static inline int kvm_vgic_get_max_vcpus(void)
>>  
>>  bool vgic_has_its(struct kvm *kvm);
>>  
>> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
>> -
>>  #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>> index 82bfb33..a175d93 100644
>> --- a/virt/kvm/arm/vgic/vgic.c
>> +++ b/virt/kvm/arm/vgic/vgic.c
>> @@ -623,10 +623,3 @@ bool vgic_has_its(struct kvm *kvm)
>>  	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>>  }
>>  
>> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
>> -{
>> -	if (vgic_has_its(kvm))
>> -		return vits_inject_msi(kvm, msi);
>> -	else
>> -		return -ENODEV;
>> -}
> 
> I don't understand why we're removing these two entries here, or rather,
> why we had something here already, given that we don't select HAVE_KVM_MSI
> before this patch as well?

we are not removing the functionality, we move its implementation.

Before this patch we did not compile irqchip.c at all. So we implemented
kvm_send_userspace_msi directly in the vgic.c code. Now irqchip is
compiled,  kvm_send_userspace_msi is natively implemented in the
irqchip.c framework and calls kvm_set_msi. This latter now is
implemented in kvm_irqfd, in this patch. I know, its difficult to follow ;-)
> 
>> diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
>> index 3eee1bd..a76994f 100644
>> --- a/virt/kvm/arm/vgic/vgic_irqfd.c
>> +++ b/virt/kvm/arm/vgic/vgic_irqfd.c
>> @@ -17,35 +17,82 @@
>>  #include <linux/kvm.h>
>>  #include <linux/kvm_host.h>
>>  #include <trace/events/kvm.h>
>> +#include <kvm/vgic/vgic.h>
>> +#include "vgic.h"
>>  
>> -int kvm_irq_map_gsi(struct kvm *kvm,
>> -		    struct kvm_kernel_irq_routing_entry *entries,
>> -		    int gsi)
>> +/**
>> + * vgic_irqfd_set_irq: inject the IRQ corresponding to the
>> + * irqchip routing entry
>> + *
>> + * This is the entry point for irqfd IRQ injection
>> + */
>> +static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
>> +			struct kvm *kvm, int irq_source_id,
>> +			int level, bool line_status)
>>  {
>> -	return 0;
>> -}
>> +	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>>  
>> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
>> -{
>> -	return pin;
>> -}
>> +	trace_kvm_set_irq(spi_id, level, irq_source_id);
> 
> but this is not kvm_set_irq ?  Perhaps it doesn't matter because the
> functionality is the same, but in that case, rename it to something more
> generic.
Sure
> 
>>  
>> -int kvm_set_irq(struct kvm *kvm, int irq_source_id,
>> -		u32 irq, int level, bool line_status)
>> -{
>> -	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
>> +	BUG_ON(!vgic_initialized(kvm));
>>  
>> -	trace_kvm_set_irq(irq, level, irq_source_id);
>> +	if (spi_id > min(dist->nr_spis, 1020))
>> +		return -EINVAL;
>> +	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
>> +}
>>  
>> -	BUG_ON(!vgic_initialized(kvm));
>> +/**
>> + * kvm_set_routing_entry: populate a kvm routing entry
>> + * from a user routing entry
>> + *
>> + * @e: kvm kernel routing entry handle
>> + * @ue: user api routing entry handle
>> + * return 0 on success, -EINVAL on errors.
>> + */
>> +int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
>> +			  const struct kvm_irq_routing_entry *ue)
>> +{
>> +	int r = -EINVAL;
>>  
>> -	return kvm_vgic_inject_irq(kvm, 0, spi, level);
>> +	switch (ue->type) {
>> +	case KVM_IRQ_ROUTING_IRQCHIP:
>> +		e->set = vgic_irqfd_set_irq;
>> +		e->irqchip.irqchip = ue->u.irqchip.irqchip;
>> +		e->irqchip.pin = ue->u.irqchip.pin;
>> +		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
>> +		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
>> +			goto out;
>> +		break;
>> +	default:
>> +		goto out;
>> +	}
>> +	r = 0;
>> +out:
>> +	return r;
>>  }
>>  
>> -/* MSI not implemented yet */
>> +/**
>> + * kvm_set_msi: inject the MSI corresponding to the
>> + * MSI routing entry
>> + *
>> + * This is the entry point for irqfd MSI injection
>> + * and userspace MSI injection.
>> + */
>>  int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>>  		struct kvm *kvm, int irq_source_id,
>>  		int level, bool line_status)
>>  {
>> -	return 0;
>> +	struct kvm_msi msi;
>> +
>> +	msi.address_lo = e->msi.address_lo;
>> +	msi.address_hi = e->msi.address_hi;
>> +	msi.data = e->msi.data;
>> +	msi.flags = e->flags;
>> +	msi.devid = e->devid;
> 
> why do we need to copy the data to a new struct?
kvm_set_msi is the callback called by irqchip framework. It takes as
parameter a kvm_kernel_irq_routing_entry pointer. This prototype is imposed.

in vgic we currently us kvm_msi * instead. Maybe I should now consider
changing the vits_inject_msi proto to take a struct
kvm_kernel_irq_routing_entry * as a parameter.

Thanks

Eric

> 
>> +
>> +	if (!vgic_has_its(kvm))
>> +		return -ENODEV;
>> +
>> +	return vits_inject_msi(kvm, &msi);
>>  }
>> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
>> index 1c556cb..b4222d6 100644
>> --- a/virt/kvm/irqchip.c
>> +++ b/virt/kvm/irqchip.c
>> @@ -29,7 +29,9 @@
>>  #include <linux/srcu.h>
>>  #include <linux/export.h>
>>  #include <trace/events/kvm.h>
>> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>>  #include "irq.h"
>> +#endif
>>  
>>  int kvm_irq_map_gsi(struct kvm *kvm,
>>  		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
>> -- 
>> 1.9.1
>>

[PATCH v4 5/7] KVM: arm/arm64: build a default routing table

[Eric Auger]

Implement a default routing table made of flat irqchip routing
entries (gsi = irqchip.pin) covering the VGIC SPI indexes.
This routing table is overwritten by the first user-space call
to KVM_SET_GSI_ROUTING ioctl.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
---
 virt/kvm/arm/vgic/vgic_init.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
index e4459e3..041443b 100644
--- a/virt/kvm/arm/vgic/vgic_init.c
+++ b/virt/kvm/arm/vgic/vgic_init.c
@@ -261,6 +261,10 @@ int vgic_init(struct kvm *kvm)
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_vgic_vcpu_init(vcpu);
 
+	ret = kvm_setup_default_irq_routing(kvm);
+	if (ret)
+		goto out;
+
 	dist->initialized = true;
 out:
 	return ret;
@@ -454,3 +458,23 @@ out_free_irq:
 			kvm_get_running_vcpus());
 	return ret;
 }
+
+int kvm_setup_default_irq_routing(struct kvm *kvm)
+{
+	struct kvm_irq_routing_entry *entries;
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u32 nr = dist->nr_spis;
+	int i, ret;
+
+	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
+			  GFP_KERNEL);
+	for (i = 0; i < nr; i++) {
+		entries[i].gsi = i;
+		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
+		entries[i].u.irqchip.irqchip = 0;
+		entries[i].u.irqchip.pin = i;
+	}
+	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
+	kfree(entries);
+	return ret;
+}
-- 
1.9.1

Re: [PATCH v4 5/7] KVM: arm/arm64: build a default routing table

[Christoffer Dall]

On Mon, Apr 04, 2016 at 10:47:35AM +0200, Eric Auger wrote:
> Implement a default routing table made of flat irqchip routing
> entries (gsi = irqchip.pin) covering the VGIC SPI indexes.
> This routing table is overwritten by the first user-space call
> to KVM_SET_GSI_ROUTING ioctl.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 

so before applying this patch, but after applying the previous patch,
will our current GICv2m-based MSI injection work?

If not, is there a better split of these patches (e.g. squash this one
into the previous one)?

> ---
> ---
>  virt/kvm/arm/vgic/vgic_init.c | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index e4459e3..041443b 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -261,6 +261,10 @@ int vgic_init(struct kvm *kvm)
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_init(vcpu);
>  
> +	ret = kvm_setup_default_irq_routing(kvm);
> +	if (ret)
> +		goto out;
> +
>  	dist->initialized = true;
>  out:
>  	return ret;
> @@ -454,3 +458,23 @@ out_free_irq:
>  			kvm_get_running_vcpus());
>  	return ret;
>  }
> +
> +int kvm_setup_default_irq_routing(struct kvm *kvm)
> +{
> +	struct kvm_irq_routing_entry *entries;
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	u32 nr = dist->nr_spis;
> +	int i, ret;
> +
> +	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
> +			  GFP_KERNEL);

I think you need to check if the allocation succeeded here or not...

> +	for (i = 0; i < nr; i++) {
> +		entries[i].gsi = i;
> +		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
> +		entries[i].u.irqchip.irqchip = 0;
> +		entries[i].u.irqchip.pin = i;
> +	}
> +	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
> +	kfree(entries);
> +	return ret;
> +}
> -- 
> 1.9.1
> 

Re: [PATCH v4 5/7] KVM: arm/arm64: build a default routing table

[Eric Auger]

On 04/14/2016 02:05 PM, Christoffer Dall wrote:
> On Mon, Apr 04, 2016 at 10:47:35AM +0200, Eric Auger wrote:
>> Implement a default routing table made of flat irqchip routing
>> entries (gsi = irqchip.pin) covering the VGIC SPI indexes.
>> This routing table is overwritten by the first user-space call
>> to KVM_SET_GSI_ROUTING ioctl.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
> 
> so before applying this patch, but after applying the previous patch,
> will our current GICv2m-based MSI injection work?

If you don't apply that patch and kernel does not build a default table,
userspace is obliged to do the job. if it doesn't the irqfd-> virtual
gsi forwarding will fail. So effectively it makes sense to squash both
patches.

> 
> If not, is there a better split of these patches (e.g. squash this one
> into the previous one)?
> 
>> ---
>> ---
>>  virt/kvm/arm/vgic/vgic_init.c | 24 ++++++++++++++++++++++++
>>  1 file changed, 24 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>> index e4459e3..041443b 100644
>> --- a/virt/kvm/arm/vgic/vgic_init.c
>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>> @@ -261,6 +261,10 @@ int vgic_init(struct kvm *kvm)
>>  	kvm_for_each_vcpu(i, vcpu, kvm)
>>  		kvm_vgic_vcpu_init(vcpu);
>>  
>> +	ret = kvm_setup_default_irq_routing(kvm);
>> +	if (ret)
>> +		goto out;
>> +
>>  	dist->initialized = true;
>>  out:
>>  	return ret;
>> @@ -454,3 +458,23 @@ out_free_irq:
>>  			kvm_get_running_vcpus());
>>  	return ret;
>>  }
>> +
>> +int kvm_setup_default_irq_routing(struct kvm *kvm)
>> +{
>> +	struct kvm_irq_routing_entry *entries;
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	u32 nr = dist->nr_spis;
>> +	int i, ret;
>> +
>> +	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
>> +			  GFP_KERNEL);
> 
> I think you need to check if the allocation succeeded here or not...
sure

thanks

Eric
> 
>> +	for (i = 0; i < nr; i++) {
>> +		entries[i].gsi = i;
>> +		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
>> +		entries[i].u.irqchip.irqchip = 0;
>> +		entries[i].u.irqchip.pin = i;
>> +	}
>> +	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
>> +	kfree(entries);
>> +	return ret;
>> +}
>> -- 
>> 1.9.1
>>

[PATCH v4 6/7] KVM: arm/arm64: enable MSI routing

[Eric Auger]

Up to now, only irqchip routing entries could be set. This patch
adds the capability to insert MSI routing entries.

For ARM64, let's also increase KVM_MAX_IRQ_ROUTES to 4096: this
include SPI irqchip routes plus MSI routes. In the future this
might be extended.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v2 -> v3:
- remove any reference to KVM_IRQ_ROUTING_EXTENDED_MSI type
- unconditionnaly uapi flags and devid downto the kernel
  routing entry struct
- handle KVM_MSI_VALID_DEVID flag in kvm_set_irq_routing
- note about KVM_CAP_MSI_DEVID moved in the first patch file
  of the series

v1 -> v2:
- adapt to new routing entry types

RFC -> PATCH:
- move api MSI routing updates into that patch file
- use new devid field of user api struct
---
 Documentation/virtual/kvm/api.txt | 5 +++++
 include/linux/kvm_host.h          | 2 ++
 virt/kvm/arm/vgic/vgic_irqfd.c    | 8 ++++++++
 virt/kvm/irqchip.c                | 2 +-
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 61f8f27..217b460 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2368,6 +2368,11 @@ On arm/arm64, gsi routing being supported, the following can happen:
 - in case no routing entry is associated to this gsi, injection fails
 - in case the gsi is associated to an irqchip routing entry,
   irqchip.pin + 32 corresponds to the injected SPI ID.
+- in case the gsi is associated to an MSI routing entry,
+  * without GICv3 ITS in-kernel emulation, MSI data matches the SPI ID
+    of the injected SPI
+  * with GICv3 ITS in-kernel emulation, the MSI message and device ID
+    are translated into an LPI.
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 461e480..f25613d 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1021,6 +1021,8 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
 
 #ifdef CONFIG_S390
 #define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that...
+#elif defined(CONFIG_ARM64)
+#define KVM_MAX_IRQ_ROUTES 4096
 #else
 #define KVM_MAX_IRQ_ROUTES 1024
 #endif
diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
index a76994f..a3a7f02 100644
--- a/virt/kvm/arm/vgic/vgic_irqfd.c
+++ b/virt/kvm/arm/vgic/vgic_irqfd.c
@@ -64,6 +64,14 @@ int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
 		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
 			goto out;
 		break;
+	case KVM_IRQ_ROUTING_MSI:
+		e->set = kvm_set_msi;
+		e->msi.address_lo = ue->u.msi.address_lo;
+		e->msi.address_hi = ue->u.msi.address_hi;
+		e->msi.data = ue->u.msi.data;
+		e->flags = ue->flags;
+		e->devid = ue->u.msi.devid;
+		break;
 	default:
 		goto out;
 	}
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index b4222d6..269d949 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -211,7 +211,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
 			goto out;
 
 		r = -EINVAL;
-		if (ue->flags) {
+		if (ue->flags & ~KVM_MSI_VALID_DEVID) {
 			kfree(e);
 			goto out;
 		}
-- 
1.9.1

Re: [PATCH v4 6/7] KVM: arm/arm64: enable MSI routing

[Christoffer Dall]

On Mon, Apr 04, 2016 at 10:47:36AM +0200, Eric Auger wrote:
> Up to now, only irqchip routing entries could be set. This patch
> adds the capability to insert MSI routing entries.
> 
> For ARM64, let's also increase KVM_MAX_IRQ_ROUTES to 4096: this
> include SPI irqchip routes plus MSI routes. In the future this
> might be extended.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 
> ---
> v2 -> v3:
> - remove any reference to KVM_IRQ_ROUTING_EXTENDED_MSI type
> - unconditionnaly uapi flags and devid downto the kernel
>   routing entry struct
> - handle KVM_MSI_VALID_DEVID flag in kvm_set_irq_routing
> - note about KVM_CAP_MSI_DEVID moved in the first patch file
>   of the series
> 
> v1 -> v2:
> - adapt to new routing entry types
> 
> RFC -> PATCH:
> - move api MSI routing updates into that patch file
> - use new devid field of user api struct
> ---
>  Documentation/virtual/kvm/api.txt | 5 +++++
>  include/linux/kvm_host.h          | 2 ++
>  virt/kvm/arm/vgic/vgic_irqfd.c    | 8 ++++++++
>  virt/kvm/irqchip.c                | 2 +-
>  4 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 61f8f27..217b460 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -2368,6 +2368,11 @@ On arm/arm64, gsi routing being supported, the following can happen:
>  - in case no routing entry is associated to this gsi, injection fails
>  - in case the gsi is associated to an irqchip routing entry,
>    irqchip.pin + 32 corresponds to the injected SPI ID.
> +- in case the gsi is associated to an MSI routing entry,
> +  * without GICv3 ITS in-kernel emulation, MSI data matches the SPI ID
> +    of the injected SPI

can we be more specific here and perhaps express this in the positive?
For example, if the user creates a KVM_DEV_TYPE_ARM_VGIC_V2, then...

> +  * with GICv3 ITS in-kernel emulation, the MSI message and device ID
> +    are translated into an LPI.
>  
>  4.76 KVM_PPC_ALLOCATE_HTAB
>  
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index 461e480..f25613d 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1021,6 +1021,8 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
>  
>  #ifdef CONFIG_S390
>  #define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that...
> +#elif defined(CONFIG_ARM64)
> +#define KVM_MAX_IRQ_ROUTES 4096
>  #else
>  #define KVM_MAX_IRQ_ROUTES 1024
>  #endif
> diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
> index a76994f..a3a7f02 100644
> --- a/virt/kvm/arm/vgic/vgic_irqfd.c
> +++ b/virt/kvm/arm/vgic/vgic_irqfd.c
> @@ -64,6 +64,14 @@ int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
>  		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
>  			goto out;
>  		break;
> +	case KVM_IRQ_ROUTING_MSI:
> +		e->set = kvm_set_msi;
> +		e->msi.address_lo = ue->u.msi.address_lo;
> +		e->msi.address_hi = ue->u.msi.address_hi;
> +		e->msi.data = ue->u.msi.data;
> +		e->flags = ue->flags;
> +		e->devid = ue->u.msi.devid;
> +		break;
>  	default:
>  		goto out;
>  	}
> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
> index b4222d6..269d949 100644
> --- a/virt/kvm/irqchip.c
> +++ b/virt/kvm/irqchip.c
> @@ -211,7 +211,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
>  			goto out;
>  
>  		r = -EINVAL;
> -		if (ue->flags) {
> +		if (ue->flags & ~KVM_MSI_VALID_DEVID) {
>  			kfree(e);
>  			goto out;
>  		}
> -- 
> 1.9.1
> 

[PATCH v4 7/7] KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

[Eric Auger]

If the ITS modality is not available, let's simply support MSI
injection by transforming the MSI.data into an SPI ID.

This becomes possible to use KVM_SIGNAL_MSI ioctl and MSI
routing for arm too.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

v2 -> v3:
- reword the commit message
- add sanity check about devid provision

v1 -> v2:
- introduce vgic_v2m_inject_msi in vgic-v2-emul.c following Andre's
  advice

Conflicts:
	arch/arm/kvm/Kconfig

Conflicts:
	arch/arm/kvm/Kconfig
---
 arch/arm/kvm/Kconfig           |  1 +
 virt/kvm/arm/vgic/vgic-v2.c    | 15 +++++++++++++++
 virt/kvm/arm/vgic/vgic.h       |  1 +
 virt/kvm/arm/vgic/vgic_irqfd.c |  2 +-
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 92c3aec..67019e5 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -50,6 +50,7 @@ config KVM_NEW_VGIC
 	bool "New VGIC implementation"
 	depends on KVM
 	default y
+	select HAVE_KVM_MSI
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_IRQ_ROUTING
 	---help---
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index 5f7c289..7400af0 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -248,6 +248,21 @@ void vgic_v2_enable(struct kvm_vcpu *vcpu)
 	vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr = GICH_HCR_EN;
 }
 
+/**
+ * vgic_v2m_inject_msi: emulates GICv2M MSI injection by injecting
+ * the SPI ID matching the msi data
+ *
+ * @kvm: pointer to the kvm struct
+ * @msi: the msi struct handle
+ */
+int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	if (msi->flags & KVM_MSI_VALID_DEVID)
+		return -EINVAL;
+
+	return kvm_vgic_inject_irq(kvm, 0, msi->data, 1);
+}
+
 int vgic_v2_map_resources(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 223c778..354a865 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -44,6 +44,7 @@ int vgic_v2_probe(struct device_node *vgic_node);
 int vgic_v2_map_resources(struct kvm *kvm);
 int vgic_register_dist_regions(struct kvm *kvm, gpa_t dist_base_address,
 			       enum vgic_type);
+int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
 
 #ifdef CONFIG_KVM_ARM_VGIC_V3
 void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid, u64 mpidr);
diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
index a3a7f02..588cdd6 100644
--- a/virt/kvm/arm/vgic/vgic_irqfd.c
+++ b/virt/kvm/arm/vgic/vgic_irqfd.c
@@ -100,7 +100,7 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 	msi.devid = e->devid;
 
 	if (!vgic_has_its(kvm))
-		return -ENODEV;
+		return vgic_v2m_inject_msi(kvm, &msi);
 
 	return vits_inject_msi(kvm, &msi);
 }
-- 
1.9.1

Re: [PATCH v4 7/7] KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

[Christoffer Dall]

On Mon, Apr 04, 2016 at 10:47:37AM +0200, Eric Auger wrote:
> If the ITS modality is not available, let's simply support MSI
> injection by transforming the MSI.data into an SPI ID.
> 
> This becomes possible to use KVM_SIGNAL_MSI ioctl and MSI
> routing for arm too.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> 
> ---
> 
> v2 -> v3:
> - reword the commit message
> - add sanity check about devid provision
> 
> v1 -> v2:
> - introduce vgic_v2m_inject_msi in vgic-v2-emul.c following Andre's
>   advice
> 
> Conflicts:
> 	arch/arm/kvm/Kconfig
> 
> Conflicts:
> 	arch/arm/kvm/Kconfig
> ---
>  arch/arm/kvm/Kconfig           |  1 +
>  virt/kvm/arm/vgic/vgic-v2.c    | 15 +++++++++++++++
>  virt/kvm/arm/vgic/vgic.h       |  1 +
>  virt/kvm/arm/vgic/vgic_irqfd.c |  2 +-
>  4 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> index 92c3aec..67019e5 100644
> --- a/arch/arm/kvm/Kconfig
> +++ b/arch/arm/kvm/Kconfig
> @@ -50,6 +50,7 @@ config KVM_NEW_VGIC
>  	bool "New VGIC implementation"
>  	depends on KVM
>  	default y
> +	select HAVE_KVM_MSI
>  	select HAVE_KVM_IRQCHIP
>  	select HAVE_KVM_IRQ_ROUTING
>  	---help---
> diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
> index 5f7c289..7400af0 100644
> --- a/virt/kvm/arm/vgic/vgic-v2.c
> +++ b/virt/kvm/arm/vgic/vgic-v2.c
> @@ -248,6 +248,21 @@ void vgic_v2_enable(struct kvm_vcpu *vcpu)
>  	vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr = GICH_HCR_EN;
>  }
>  
> +/**
> + * vgic_v2m_inject_msi: emulates GICv2M MSI injection by injecting
> + * the SPI ID matching the msi data
> + *
> + * @kvm: pointer to the kvm struct
> + * @msi: the msi struct handle
> + */
> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
> +{
> +	if (msi->flags & KVM_MSI_VALID_DEVID)
> +		return -EINVAL;
> +
> +	return kvm_vgic_inject_irq(kvm, 0, msi->data, 1);

I think you need to validate the msi->data here, otherwise
vgic_get_irq() is going to raise warnings etc.

> +}
> +
>  int vgic_v2_map_resources(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 223c778..354a865 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -44,6 +44,7 @@ int vgic_v2_probe(struct device_node *vgic_node);
>  int vgic_v2_map_resources(struct kvm *kvm);
>  int vgic_register_dist_regions(struct kvm *kvm, gpa_t dist_base_address,
>  			       enum vgic_type);
> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
>  
>  #ifdef CONFIG_KVM_ARM_VGIC_V3
>  void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid, u64 mpidr);
> diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
> index a3a7f02..588cdd6 100644
> --- a/virt/kvm/arm/vgic/vgic_irqfd.c
> +++ b/virt/kvm/arm/vgic/vgic_irqfd.c
> @@ -100,7 +100,7 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>  	msi.devid = e->devid;
>  
>  	if (!vgic_has_its(kvm))
> -		return -ENODEV;
> +		return vgic_v2m_inject_msi(kvm, &msi);
>  
>  	return vits_inject_msi(kvm, &msi);
>  }
> -- 
> 1.9.1
> 

Re: [PATCH v4 7/7] KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

[Eric Auger]

On 04/14/2016 02:12 PM, Christoffer Dall wrote:
> On Mon, Apr 04, 2016 at 10:47:37AM +0200, Eric Auger wrote:
>> If the ITS modality is not available, let's simply support MSI
>> injection by transforming the MSI.data into an SPI ID.
>>
>> This becomes possible to use KVM_SIGNAL_MSI ioctl and MSI
>> routing for arm too.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>
>> ---
>>
>> v2 -> v3:
>> - reword the commit message
>> - add sanity check about devid provision
>>
>> v1 -> v2:
>> - introduce vgic_v2m_inject_msi in vgic-v2-emul.c following Andre's
>>   advice
>>
>> Conflicts:
>> 	arch/arm/kvm/Kconfig
>>
>> Conflicts:
>> 	arch/arm/kvm/Kconfig
>> ---
>>  arch/arm/kvm/Kconfig           |  1 +
>>  virt/kvm/arm/vgic/vgic-v2.c    | 15 +++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h       |  1 +
>>  virt/kvm/arm/vgic/vgic_irqfd.c |  2 +-
>>  4 files changed, 18 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 92c3aec..67019e5 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -50,6 +50,7 @@ config KVM_NEW_VGIC
>>  	bool "New VGIC implementation"
>>  	depends on KVM
>>  	default y
>> +	select HAVE_KVM_MSI
>>  	select HAVE_KVM_IRQCHIP
>>  	select HAVE_KVM_IRQ_ROUTING
>>  	---help---
>> diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
>> index 5f7c289..7400af0 100644
>> --- a/virt/kvm/arm/vgic/vgic-v2.c
>> +++ b/virt/kvm/arm/vgic/vgic-v2.c
>> @@ -248,6 +248,21 @@ void vgic_v2_enable(struct kvm_vcpu *vcpu)
>>  	vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr = GICH_HCR_EN;
>>  }
>>  
>> +/**
>> + * vgic_v2m_inject_msi: emulates GICv2M MSI injection by injecting
>> + * the SPI ID matching the msi data
>> + *
>> + * @kvm: pointer to the kvm struct
>> + * @msi: the msi struct handle
>> + */
>> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>> +{
>> +	if (msi->flags & KVM_MSI_VALID_DEVID)
>> +		return -EINVAL;
>> +
>> +	return kvm_vgic_inject_irq(kvm, 0, msi->data, 1);
> 
> I think you need to validate the msi->data here, otherwise
> vgic_get_irq() is going to raise warnings etc.
OK will check it is within the SPI ID range.

Thank you for your time!

Best Regards

Eric


> 
>> +}
>> +
>>  int vgic_v2_map_resources(struct kvm *kvm)
>>  {
>>  	struct vgic_dist *dist = &kvm->arch.vgic;
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index 223c778..354a865 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -44,6 +44,7 @@ int vgic_v2_probe(struct device_node *vgic_node);
>>  int vgic_v2_map_resources(struct kvm *kvm);
>>  int vgic_register_dist_regions(struct kvm *kvm, gpa_t dist_base_address,
>>  			       enum vgic_type);
>> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
>>  
>>  #ifdef CONFIG_KVM_ARM_VGIC_V3
>>  void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid, u64 mpidr);
>> diff --git a/virt/kvm/arm/vgic/vgic_irqfd.c b/virt/kvm/arm/vgic/vgic_irqfd.c
>> index a3a7f02..588cdd6 100644
>> --- a/virt/kvm/arm/vgic/vgic_irqfd.c
>> +++ b/virt/kvm/arm/vgic/vgic_irqfd.c
>> @@ -100,7 +100,7 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>>  	msi.devid = e->devid;
>>  
>>  	if (!vgic_has_its(kvm))
>> -		return -ENODEV;
>> +		return vgic_v2m_inject_msi(kvm, &msi);
>>  
>>  	return vits_inject_msi(kvm, &msi);
>>  }
>> -- 
>> 1.9.1
>>

RE: [RFC v4 0/7] KVM: arm/arm64: gsi routing support

[Pavel Fedin]

 Hello!

 I'd like to remind that this will break with current vITS code and VFIO. 
 The problem is short path for raising MSIs via routed GSI. There is a code
which directly calls IRQ injection instead of triggering irqfd. With VFIO
this code is called from within interrupt context. This will cause random
lockups in vits_inject_msi(), which does not use irq-save spin-locking.
 I hit this several times and I had to rework ITS emulation in my local
tree.

Kind regards,
Pavel Fedin
Senior Engineer
Samsung Electronics Research center Russia


-----Original Message-----
From: Eric Auger [mailto:eric.auger@linaro.org] 
Sent: Monday, April 04, 2016 11:48 AM
To: eric.auger@st.com; eric.auger@linaro.org; christoffer.dall@linaro.org;
marc.zyngier@arm.com; andre.przywara@arm.com; kvmarm@lists.cs.columbia.edu;
kvm@vger.kernel.org
Cc: patches@linaro.org; pbonzini@redhat.com; p.fedin@samsung.com;
Manish.Jaggi@caviumnetworks.com
Subject: [RFC v4 0/7] KVM: arm/arm64: gsi routing support

With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
becomes mandated for proper VIRTIO-PCI vhost integration.

In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
we previously used direct_mapping trick: this consists in extracting the
SPI ID found in the MSI message and associating an irqfd to that SPI ID.
When vhost worker thread gets a new buffer it signals the irqfd and kvm
then injects this SPI ID on guest. That way although the guest uses MSIs,
no MSI emulation is used.

This worked fine with GICv2m but does not work anymore with GICV3 ITS.
Indeed this latter implements IRQ translation: what is found in the MSI
message no more is the target SPI ID but is an intermediate event ID used
in the translation process.

Hence true MSI routing is needed so that the vhost back channel irqfd is
associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
injects the MSI through the in-kernel ITS emulation, the MSI is properly
translated and eventually the SPI ID associated to the event ID is injected
on guest.

irqchip routing does not sound to be really useful on arm but usage of
MSI routing also mandates to integrate irqchip routing. The initial
implementation of irqfd on arm must be upgraded with the integration
of kvm irqchip.c code and the implementation of its standard hooks
in the architecture specific part.

In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
table with flat irqchip routing entries is built enabling to inject gsi
corresponding to the SPI indexes seen by the guest.

As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
default routing table and is responsible for building the whole routing
table.

for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
- only applies to KVM_IRQFD and not to KVM_IRQ_LINE

- irqchip routing was tested on Calxeda midway (VFIO with irqfd)
  with and without explicit routing
- MSI routing without GICv3 ITS was tested using APM Xgene-I
  (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
- MSI routing with GICv3 ITS is *NOT* tested.

Code can be found at
https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.5-
rc6-its-emul-v4-gsi-routing-v4

The series applies on top of Andre's branch featuring NEW-VGIC and in-kernel
ITS emulation series:
http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4

[1]: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
     http://www.spinics.net/lists/arm-kernel/msg492770.html
[2]: [RFC PATCH 00/45] KVM: arm/arm64: Rework virtual GIC emulation
     http://www.spinics.net/lists/arm-kernel/msg492639.html

GSI flat routing setup on QEMU can be found at:
https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html

History:
v3 -> v4:
- rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
  a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
  reflesh/reminder.
- rewrite the cover letter

v2 -> v3:
- don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level
anymore;
  use KVM_MSI_VALID_DEVID flag instead
- propagate user flags downto the kernel to make sure the userspace
  correctly set devid in GICv3 ITS case (still under discussion)

v1 -> v2:
- user API changed:
  x devid id passed in kvm_irq_routing_msi
  x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
- kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
  composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
- Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
  in eventfd.c
- added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
- fix bug reported by Andre: bad setting of msi.flags and msi.devid
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH:
- clearly state limited support on arm/arm64:
  KVM_IRQ_LINE not impacted by GSI routing
- add default routing table feature (new patch file)
- changed uapi to use padding field area
- reword api.txt



Eric Auger (7):
  KVM: api: pass the devid in the msi routing entry
  KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
  KVM: irqchip: convey devid to kvm_set_msi
  KVM: arm/arm64: enable irqchip routing
  KVM: arm/arm64: build a default routing table
  KVM: arm/arm64: enable MSI routing
  KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

 Documentation/virtual/kvm/api.txt | 35 ++++++++++++---
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  3 ++
 arch/arm/kvm/Makefile             |  1 +
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  3 ++
 arch/arm64/kvm/Makefile           |  1 +
 include/kvm/vgic/vgic.h           |  2 -
 include/linux/kvm_host.h          |  8 +++-
 include/uapi/linux/kvm.h          |  5 ++-
 virt/kvm/arm/vgic/vgic-v2.c       | 15 +++++++
 virt/kvm/arm/vgic/vgic.c          |  7 ---
 virt/kvm/arm/vgic/vgic.h          |  1 +
 virt/kvm/arm/vgic/vgic_init.c     | 24 +++++++++++
 virt/kvm/arm/vgic/vgic_irqfd.c    | 91
+++++++++++++++++++++++++++++++--------
 virt/kvm/irqchip.c                |  8 +++-
 16 files changed, 170 insertions(+), 37 deletions(-)

-- 
1.9.1

Re: [RFC v4 0/7] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Pavel,
On 04/04/2016 12:15 PM, Pavel Fedin wrote:
>  Hello!
> 
>  I'd like to remind that this will break with current vITS code and VFIO. 
>  The problem is short path for raising MSIs via routed GSI. There is a code
> which directly calls IRQ injection instead of triggering irqfd. With VFIO
> this code is called from within interrupt context. This will cause random
> lockups in vits_inject_msi(), which does not use irq-save spin-locking.
>  I hit this several times and I had to rework ITS emulation in my local
> tree.
Thank you for the reminder. We will sync with Andre and fix that in the
next ITS emulation version.

Best Regards

Eric
> 
> Kind regards,
> Pavel Fedin
> Senior Engineer
> Samsung Electronics Research center Russia
> 
> 
> -----Original Message-----
> From: Eric Auger [mailto:eric.auger@linaro.org] 
> Sent: Monday, April 04, 2016 11:48 AM
> To: eric.auger@st.com; eric.auger@linaro.org; christoffer.dall@linaro.org;
> marc.zyngier@arm.com; andre.przywara@arm.com; kvmarm@lists.cs.columbia.edu;
> kvm@vger.kernel.org
> Cc: patches@linaro.org; pbonzini@redhat.com; p.fedin@samsung.com;
> Manish.Jaggi@caviumnetworks.com
> Subject: [RFC v4 0/7] KVM: arm/arm64: gsi routing support
> 
> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
> becomes mandated for proper VIRTIO-PCI vhost integration.
> 
> In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
> we previously used direct_mapping trick: this consists in extracting the
> SPI ID found in the MSI message and associating an irqfd to that SPI ID.
> When vhost worker thread gets a new buffer it signals the irqfd and kvm
> then injects this SPI ID on guest. That way although the guest uses MSIs,
> no MSI emulation is used.
> 
> This worked fine with GICv2m but does not work anymore with GICV3 ITS.
> Indeed this latter implements IRQ translation: what is found in the MSI
> message no more is the target SPI ID but is an intermediate event ID used
> in the translation process.
> 
> Hence true MSI routing is needed so that the vhost back channel irqfd is
> associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
> injects the MSI through the in-kernel ITS emulation, the MSI is properly
> translated and eventually the SPI ID associated to the event ID is injected
> on guest.
> 
> irqchip routing does not sound to be really useful on arm but usage of
> MSI routing also mandates to integrate irqchip routing. The initial
> implementation of irqfd on arm must be upgraded with the integration
> of kvm irqchip.c code and the implementation of its standard hooks
> in the architecture specific part.
> 
> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
> table with flat irqchip routing entries is built enabling to inject gsi
> corresponding to the SPI indexes seen by the guest.
> 
> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
> default routing table and is responsible for building the whole routing
> table.
> 
> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
> 
> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>   with and without explicit routing
> - MSI routing without GICv3 ITS was tested using APM Xgene-I
>   (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
> - MSI routing with GICv3 ITS is *NOT* tested.
> 
> Code can be found at
> https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.5-
> rc6-its-emul-v4-gsi-routing-v4
> 
> The series applies on top of Andre's branch featuring NEW-VGIC and in-kernel
> ITS emulation series:
> http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4
> 
> [1]: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
>      http://www.spinics.net/lists/arm-kernel/msg492770.html
> [2]: [RFC PATCH 00/45] KVM: arm/arm64: Rework virtual GIC emulation
>      http://www.spinics.net/lists/arm-kernel/msg492639.html
> 
> GSI flat routing setup on QEMU can be found at:
> https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html
> 
> History:
> v3 -> v4:
> - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
>   a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
>   reflesh/reminder.
> - rewrite the cover letter
> 
> v2 -> v3:
> - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level
> anymore;
>   use KVM_MSI_VALID_DEVID flag instead
> - propagate user flags downto the kernel to make sure the userspace
>   correctly set devid in GICv3 ITS case (still under discussion)
> 
> v1 -> v2:
> - user API changed:
>   x devid id passed in kvm_irq_routing_msi
>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>   in eventfd.c
> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>   in kvm_send_userspace_msi
> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> 
> RFC -> PATCH:
> - clearly state limited support on arm/arm64:
>   KVM_IRQ_LINE not impacted by GSI routing
> - add default routing table feature (new patch file)
> - changed uapi to use padding field area
> - reword api.txt
> 
> 
> 
> Eric Auger (7):
>   KVM: api: pass the devid in the msi routing entry
>   KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
>   KVM: irqchip: convey devid to kvm_set_msi
>   KVM: arm/arm64: enable irqchip routing
>   KVM: arm/arm64: build a default routing table
>   KVM: arm/arm64: enable MSI routing
>   KVM: arm: enable KVM_SIGNAL_MSI and MSI routing
> 
>  Documentation/virtual/kvm/api.txt | 35 ++++++++++++---
>  arch/arm/include/asm/kvm_host.h   |  2 +
>  arch/arm/kvm/Kconfig              |  3 ++
>  arch/arm/kvm/Makefile             |  1 +
>  arch/arm64/include/asm/kvm_host.h |  1 +
>  arch/arm64/kvm/Kconfig            |  3 ++
>  arch/arm64/kvm/Makefile           |  1 +
>  include/kvm/vgic/vgic.h           |  2 -
>  include/linux/kvm_host.h          |  8 +++-
>  include/uapi/linux/kvm.h          |  5 ++-
>  virt/kvm/arm/vgic/vgic-v2.c       | 15 +++++++
>  virt/kvm/arm/vgic/vgic.c          |  7 ---
>  virt/kvm/arm/vgic/vgic.h          |  1 +
>  virt/kvm/arm/vgic/vgic_init.c     | 24 +++++++++++
>  virt/kvm/arm/vgic/vgic_irqfd.c    | 91
> +++++++++++++++++++++++++++++++--------
>  virt/kvm/irqchip.c                |  8 +++-
>  16 files changed, 170 insertions(+), 37 deletions(-)
> 

Re: [RFC v4 0/7] KVM: arm/arm64: gsi routing support

[Christoffer Dall]

On Mon, Apr 04, 2016 at 10:47:30AM +0200, Eric Auger wrote:
> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
> becomes mandated for proper VIRTIO-PCI vhost integration.
> 
> In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
> we previously used direct_mapping trick: this consists in extracting the
> SPI ID found in the MSI message and associating an irqfd to that SPI ID.
> When vhost worker thread gets a new buffer it signals the irqfd and kvm
> then injects this SPI ID on guest. That way although the guest uses MSIs,
> no MSI emulation is used.
> 
> This worked fine with GICv2m but does not work anymore with GICV3 ITS.
> Indeed this latter implements IRQ translation: what is found in the MSI
> message no more is the target SPI ID but is an intermediate event ID used
> in the translation process.
> 
> Hence true MSI routing is needed so that the vhost back channel irqfd is
> associated to a dummy gsi ID, routed towards the programmed MSI. When KVM

Doesn't the guest have to program the device with some ID?  So how is
this dummy GSI ID assigned?

> injects the MSI through the in-kernel ITS emulation, the MSI is properly
> translated and eventually the SPI ID associated to the event ID is injected
> on guest.

Isn't it an LPI that is injected to the guest when you have an emulated
ITS then?

> 
> irqchip routing does not sound to be really useful on arm but usage of

Is this also true if you had multiple emulated ITS devices?

(I just don't want to block our options of doing this in the future if
ever required.)

> MSI routing also mandates to integrate irqchip routing. The initial
> implementation of irqfd on arm must be upgraded with the integration
> of kvm irqchip.c code and the implementation of its standard hooks
> in the architecture specific part.
> 
> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
> table with flat irqchip routing entries is built enabling to inject gsi
> corresponding to the SPI indexes seen by the guest.
> 
> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
> default routing table and is responsible for building the whole routing
> table.
> 
> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
> 
> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>   with and without explicit routing
> - MSI routing without GICv3 ITS was tested using APM Xgene-I
>   (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
> - MSI routing with GICv3 ITS is *NOT* tested.
> 
> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.5-rc6-its-emul-v4-gsi-routing-v4
> 
> The series applies on top of Andre's branch featuring NEW-VGIC and in-kernel
> ITS emulation series:
> http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4
> 
> [1]: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
>      http://www.spinics.net/lists/arm-kernel/msg492770.html
> [2]: [RFC PATCH 00/45] KVM: arm/arm64: Rework virtual GIC emulation
>      http://www.spinics.net/lists/arm-kernel/msg492639.html
> 
> GSI flat routing setup on QEMU can be found at:
> https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html
> 
> History:
> v3 -> v4:
> - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
>   a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
>   reflesh/reminder.
> - rewrite the cover letter
> 
> v2 -> v3:
> - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
>   use KVM_MSI_VALID_DEVID flag instead
> - propagate user flags downto the kernel to make sure the userspace
>   correctly set devid in GICv3 ITS case (still under discussion)
> 
> v1 -> v2:
> - user API changed:
>   x devid id passed in kvm_irq_routing_msi
>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>   in eventfd.c
> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>   in kvm_send_userspace_msi
> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> 
> RFC -> PATCH:
> - clearly state limited support on arm/arm64:
>   KVM_IRQ_LINE not impacted by GSI routing
> - add default routing table feature (new patch file)
> - changed uapi to use padding field area
> - reword api.txt
> 
> 
> 
> Eric Auger (7):
>   KVM: api: pass the devid in the msi routing entry
>   KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
>   KVM: irqchip: convey devid to kvm_set_msi
>   KVM: arm/arm64: enable irqchip routing
>   KVM: arm/arm64: build a default routing table
>   KVM: arm/arm64: enable MSI routing
>   KVM: arm: enable KVM_SIGNAL_MSI and MSI routing
> 
>  Documentation/virtual/kvm/api.txt | 35 ++++++++++++---
>  arch/arm/include/asm/kvm_host.h   |  2 +
>  arch/arm/kvm/Kconfig              |  3 ++
>  arch/arm/kvm/Makefile             |  1 +
>  arch/arm64/include/asm/kvm_host.h |  1 +
>  arch/arm64/kvm/Kconfig            |  3 ++
>  arch/arm64/kvm/Makefile           |  1 +
>  include/kvm/vgic/vgic.h           |  2 -
>  include/linux/kvm_host.h          |  8 +++-
>  include/uapi/linux/kvm.h          |  5 ++-
>  virt/kvm/arm/vgic/vgic-v2.c       | 15 +++++++
>  virt/kvm/arm/vgic/vgic.c          |  7 ---
>  virt/kvm/arm/vgic/vgic.h          |  1 +
>  virt/kvm/arm/vgic/vgic_init.c     | 24 +++++++++++
>  virt/kvm/arm/vgic/vgic_irqfd.c    | 91 +++++++++++++++++++++++++++++++--------
>  virt/kvm/irqchip.c                |  8 +++-
>  16 files changed, 170 insertions(+), 37 deletions(-)
> 
> -- 
> 1.9.1
> 

Re: [RFC v4 0/7] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Christoffer,
On 04/14/2016 02:04 PM, Christoffer Dall wrote:
> On Mon, Apr 04, 2016 at 10:47:30AM +0200, Eric Auger wrote:
>> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
>> becomes mandated for proper VIRTIO-PCI vhost integration.
>>
>> In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
>> we previously used direct_mapping trick: this consists in extracting the
>> SPI ID found in the MSI message and associating an irqfd to that SPI ID.
>> When vhost worker thread gets a new buffer it signals the irqfd and kvm
>> then injects this SPI ID on guest. That way although the guest uses MSIs,
>> no MSI emulation is used.
>>
>> This worked fine with GICv2m but does not work anymore with GICV3 ITS.
>> Indeed this latter implements IRQ translation: what is found in the MSI
>> message no more is the target SPI ID but is an intermediate event ID used
>> in the translation process.
>>
>> Hence true MSI routing is needed so that the vhost back channel irqfd is
>> associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
> 
> Doesn't the guest have to program the device with some ID?  So how is
> this dummy GSI ID assigned?
Please apologize for the delay.

We have this path:
irqfd -> dummy GSI -> MSI -> LPI ID

the MSI route does the link between the dummy GSI and the MSI.
the device id is embedded in the route entry definition (devid field of
kvm_irq_routing_msi)
The eventid is put in the MSI data

see https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg06178.html
(Pavel's vITS RFC)

in QEMU, kvm-all.c/kvm_irqchip_add_msi_route allocates the "dummy" gsi
(kvm_irqchip_get_virq) and builds the routing entry.


> 
>> injects the MSI through the in-kernel ITS emulation, the MSI is properly
>> translated and eventually the SPI ID associated to the event ID is injected
>> on guest.
> 
> Isn't it an LPI that is injected to the guest when you have an emulated
> ITS then?
It is definitively an LPI ID.
> 
>>
>> irqchip routing does not sound to be really useful on arm but usage of
> 
> Is this also true if you had multiple emulated ITS devices?
> 
> (I just don't want to block our options of doing this in the future if
> ever required.)

yes irqchip routing may be useful to handle multiple guest irqchips. I
meant: currently irqchip routing comes with this series although this is
not the series' primary goal. There is no special limitation except it
does not apply to KVM_IRQ_LINE ioctl because of the gsi semantic chosen
in the past, for ARM:

bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
field: | irq_type  | vcpu_index |     irq_id     |

Best Regards

Eric

> 
>> MSI routing also mandates to integrate irqchip routing. The initial
>> implementation of irqfd on arm must be upgraded with the integration
>> of kvm irqchip.c code and the implementation of its standard hooks
>> in the architecture specific part.
>>
>> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
>> table with flat irqchip routing entries is built enabling to inject gsi
>> corresponding to the SPI indexes seen by the guest.
>>
>> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
>> default routing table and is responsible for building the whole routing
>> table.
>>
>> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
>> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
>>
>> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>>   with and without explicit routing
>> - MSI routing without GICv3 ITS was tested using APM Xgene-I
>>   (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
>> - MSI routing with GICv3 ITS is *NOT* tested.
>>
>> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.5-rc6-its-emul-v4-gsi-routing-v4
>>
>> The series applies on top of Andre's branch featuring NEW-VGIC and in-kernel
>> ITS emulation series:
>> http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4
>>
>> [1]: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
>>      http://www.spinics.net/lists/arm-kernel/msg492770.html
>> [2]: [RFC PATCH 00/45] KVM: arm/arm64: Rework virtual GIC emulation
>>      http://www.spinics.net/lists/arm-kernel/msg492639.html
>>
>> GSI flat routing setup on QEMU can be found at:
>> https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html
>>
>> History:
>> v3 -> v4:
>> - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
>>   a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
>>   reflesh/reminder.
>> - rewrite the cover letter
>>
>> v2 -> v3:
>> - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
>>   use KVM_MSI_VALID_DEVID flag instead
>> - propagate user flags downto the kernel to make sure the userspace
>>   correctly set devid in GICv3 ITS case (still under discussion)
>>
>> v1 -> v2:
>> - user API changed:
>>   x devid id passed in kvm_irq_routing_msi
>>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
>> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
>> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>>   in eventfd.c
>> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
>> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>>   in kvm_send_userspace_msi
>> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
>>
>> RFC -> PATCH:
>> - clearly state limited support on arm/arm64:
>>   KVM_IRQ_LINE not impacted by GSI routing
>> - add default routing table feature (new patch file)
>> - changed uapi to use padding field area
>> - reword api.txt
>>
>>
>>
>> Eric Auger (7):
>>   KVM: api: pass the devid in the msi routing entry
>>   KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
>>   KVM: irqchip: convey devid to kvm_set_msi
>>   KVM: arm/arm64: enable irqchip routing
>>   KVM: arm/arm64: build a default routing table
>>   KVM: arm/arm64: enable MSI routing
>>   KVM: arm: enable KVM_SIGNAL_MSI and MSI routing
>>
>>  Documentation/virtual/kvm/api.txt | 35 ++++++++++++---
>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>  arch/arm/kvm/Kconfig              |  3 ++
>>  arch/arm/kvm/Makefile             |  1 +
>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>  arch/arm64/kvm/Kconfig            |  3 ++
>>  arch/arm64/kvm/Makefile           |  1 +
>>  include/kvm/vgic/vgic.h           |  2 -
>>  include/linux/kvm_host.h          |  8 +++-
>>  include/uapi/linux/kvm.h          |  5 ++-
>>  virt/kvm/arm/vgic/vgic-v2.c       | 15 +++++++
>>  virt/kvm/arm/vgic/vgic.c          |  7 ---
>>  virt/kvm/arm/vgic/vgic.h          |  1 +
>>  virt/kvm/arm/vgic/vgic_init.c     | 24 +++++++++++
>>  virt/kvm/arm/vgic/vgic_irqfd.c    | 91 +++++++++++++++++++++++++++++++--------
>>  virt/kvm/irqchip.c                |  8 +++-
>>  16 files changed, 170 insertions(+), 37 deletions(-)
>>
>> -- 
>> 1.9.1
>>