* [RFC PATCH 01/19] arm/gicv4 add management structure definitions
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-11 8:28 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 02/19] arm/gicv4-its: Add GICv4 ITS command definitions Mykyta Poturai
` (18 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Add GICv4 specific structures to be used in later patches.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/include/asm/vgic.h | 43 +++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 6f9ab1c98c..a874a02d70 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -127,6 +127,42 @@ struct vgic_irq_rank {
uint8_t vcpu[32];
};
+#ifdef CONFIG_GICV4
+struct its_vm {
+ struct its_vpe **vpes;
+ /* Number of VPE. */
+ unsigned int nr_vpes;
+ uint32_t *db_lpi_bases;
+ unsigned int nr_db_lpis;
+ /* Property table per VM. */
+ void *vproptable;
+};
+
+struct its_vpe {
+ rwlock_t lock;
+ uint32_t vpe_id;
+ /* Pending table per VCPU. */
+ void *vpendtable;
+ uint32_t vpe_db_lpi;
+ struct its_vm *its_vm;
+ unsigned int col_idx;
+ bool resident;
+ /* Pending VLPIs on schedule out? */
+ bool pending_last;
+ struct {
+ /* Implementation Defined Area Invalid */
+ bool idai;
+ /* VPE proxy mapping */
+ int vpe_proxy_event;
+ };
+ /*
+ * Ensure mutual exclusion between affinity setting of the vPE
+ * and vLPI operations using vpe->col_idx.
+ */
+ spinlock_t vpe_lock;
+};
+#endif
+
struct vgic_dist {
/* Version of the vGIC */
enum gic_version version;
@@ -193,6 +229,10 @@ struct vgic_dist {
*/
bool rdists_enabled; /* Is any redistributor enabled? */
bool has_its;
+#ifdef CONFIG_GICV4
+ struct its_vm *its_vm;
+#endif
+ bool nassgireq;
#endif
};
@@ -227,6 +267,9 @@ struct vgic_cpu {
#define VGIC_V3_RDIST_LAST (1 << 0) /* last vCPU of the rdist */
#define VGIC_V3_LPIS_ENABLED (1 << 1)
uint8_t flags;
+#ifdef CONFIG_GICV4
+ struct its_vpe *its_vpe;
+#endif
};
struct sgi_target {
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 00/19] GICv4 Support for Xen
@ 2026-02-02 16:14 Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 01/19] arm/gicv4 add management structure definitions Mykyta Poturai
` (19 more replies)
0 siblings, 20 replies; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
This series introduces GICv4 direct LPI injection for Xen.
Direct LPI injection relies on the GIC tracking the mapping between physical and
virtual CPUs. Each VCPU requires a VPE that is created and registered with the
GIC via the `VMAPP` ITS command. The GIC is then informed of the current
VPE-to-PCPU placement by programming `VPENDBASER` and `VPROPBASER` in the
appropriate redistributor. LPIs are associated with VPEs through the `VMAPTI`
ITS command, after which the GIC handles delivery without trapping into the
hypervisor for each interrupt.
When a VPE is not scheduled but has pending interrupts, the GIC raises a per-VPE
doorbell LPI. Doorbells are owned by the hypervisor and prompt rescheduling so
the VPE can drain its pending LPIs.
Because GICv4 lacks a native doorbell invalidation mechanism, this series
includes a helper that invalidates doorbell LPIs via synthetic “proxy” devices,
following the approach used until GICv4.1.
All of this work is mostly based on the work of Penny Zheng
<penny.zheng@arm.com> and Luca Fancellu <luca.fancellu@arm.com>. And also from
Linux patches by Mark Zyngier.
Some patches are still a little rough and need some styling fixes and more
testing, as all of them needed to be carved line by line from a giant ~4000 line
patch. This RFC is directed mostly to get a general idea if the proposed
approach is suitable and OK with everyone. And there is still an open question
of how to handle Signed-off-by lines for Penny and Luca, since they have not
indicated their preference yet.
Mykyta Poturai (19):
arm/gicv4 add management structure definitions
arm/gicv4-its: Add GICv4 ITS command definitions
arm/its: Export struct its_device
arm/its: Add vlpi configuration
arm/irq: Add hw flag to pending_irq
arm/gicv4-its: Add VLPI map/unmap operations
xen/domain: Alloc enough pages for VCPU struct
arm/gic: Keep track of GIC features
arm/its: Implement LPI invalidation
arm/its: Keep track of BASER regs
arm/its: Add ITS VM and VPE allocation/teardown
arm/gic: Add VPENDBASER/VPROPBASER accessors
arm/gic: VPE scheduling
arm/its: VPE affinity changes
arm: Add gicv4 to domain creation
arm/gic: Fix LR group handling for GICv4
arm/gicv4: Handle doorbells
arm/gic: Add VPE proxy support
arm/gicv4: Add GICv4 to the build system
xen/arch/arm/Kconfig | 6 +
xen/arch/arm/Makefile | 1 +
xen/arch/arm/dom0less-build.c | 1 +
xen/arch/arm/domain.c | 16 +
xen/arch/arm/gic-v2.c | 2 +-
xen/arch/arm/gic-v3-its.c | 339 +++++--
xen/arch/arm/gic-v3-lpi.c | 169 +++-
xen/arch/arm/gic-v3.c | 215 ++++-
xen/arch/arm/gic-v4-its.c | 1136 ++++++++++++++++++++++++
xen/arch/arm/gic-vgic.c | 6 +
xen/arch/arm/include/asm/gic.h | 4 +-
xen/arch/arm/include/asm/gic_v3_defs.h | 22 +
xen/arch/arm/include/asm/gic_v3_its.h | 139 ++-
xen/arch/arm/include/asm/gic_v4_its.h | 114 +++
xen/arch/arm/include/asm/vgic.h | 79 +-
xen/arch/arm/vgic-v3-its.c | 60 +-
xen/arch/arm/vgic.c | 37 +-
xen/common/domain.c | 14 +-
xen/include/public/arch-arm.h | 2 +
19 files changed, 2174 insertions(+), 188 deletions(-)
create mode 100644 xen/arch/arm/gic-v4-its.c
create mode 100644 xen/arch/arm/include/asm/gic_v4_its.h
--
2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* [RFC PATCH 02/19] arm/gicv4-its: Add GICv4 ITS command definitions
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 01/19] arm/gicv4 add management structure definitions Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-11 8:28 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 03/19] arm/its: Export struct its_device Mykyta Poturai
` (17 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Add definitions for GICv4 specific ITS commands.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/include/asm/gic_v4_its.h | 41 +++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
create mode 100644 xen/arch/arm/include/asm/gic_v4_its.h
diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
new file mode 100644
index 0000000000..f48eae60ad
--- /dev/null
+++ b/xen/arch/arm/include/asm/gic_v4_its.h
@@ -0,0 +1,41 @@
+/*
+ * ARM GICv4 ITS support
+ *
+ * Penny Zheng <penny.zheng@arm.com>
+ * Copyright (c) 2023 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; under version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __ASM_ARM_GICV4_ITS_H__
+#define __ASM_ARM_GICV4_ITS_H__
+
+#define GITS_CMD_VMOVI 0x21
+#define GITS_CMD_VMOVP 0x22
+#define GITS_CMD_VSGI 0x23
+#define GITS_CMD_VSYNC 0x25
+#define GITS_CMD_VMAPP 0x29
+#define GITS_CMD_VMAPTI 0x2a
+#define GITS_CMD_VINVALL 0x2d
+#define GITS_CMD_INVDB 0x2e
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 04/19] arm/its: Add vlpi configuration
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (2 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 03/19] arm/its: Export struct its_device Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 06/19] arm/gicv4-its: Add VLPI map/unmap operations Mykyta Poturai
` (15 subsequent siblings)
19 siblings, 0 replies; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Add a helper to write LPI configuration entries. Expose lpi_data
structure in the header to be used in said helper.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-lpi.c | 39 +++++++++++----------------
xen/arch/arm/include/asm/gic_v3_its.h | 27 +++++++++++++++++++
2 files changed, 43 insertions(+), 23 deletions(-)
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index de5052e5cf..c029d5d7a4 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -46,29 +46,7 @@ union host_lpi {
#define LPI_PROPTABLE_NEEDS_FLUSHING (1U << 0)
/* Global state */
-static struct {
- /* The global LPI property table, shared by all redistributors. */
- uint8_t *lpi_property;
- /*
- * A two-level table to lookup LPIs firing on the host and look up the
- * VCPU and virtual LPI number to inject into.
- */
- union host_lpi **host_lpis;
- /*
- * Number of physical LPIs the host supports. This is a property of
- * the GIC hardware. We depart from the habit of naming these things
- * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
- * in a different context to differentiate them from "virtual LPIs".
- */
- unsigned long int max_host_lpi_ids;
- /*
- * Protects allocation and deallocation of host LPIs and next_free_lpi,
- * but not the actual data stored in the host_lpi entry.
- */
- spinlock_t host_lpis_lock;
- uint32_t next_free_lpi;
- unsigned int flags;
-} lpi_data;
+struct __lpi_data lpi_data;
struct lpi_redist_data {
paddr_t redist_addr;
@@ -492,6 +470,21 @@ static int find_unused_host_lpi(uint32_t start, uint32_t *index)
return -1;
}
+void lpi_write_config(uint8_t *prop_table, uint32_t lpi, uint8_t clr,
+ uint8_t set)
+{
+ u8 *cfg;
+
+ cfg = prop_table + lpi - LPI_OFFSET;
+ *cfg &= ~clr;
+ *cfg |= set | LPI_PROP_RES1;
+
+ /* Make the above write visible to the redistributors. */
+ if ( lpi_data.flags & LPI_PROPTABLE_NEEDS_FLUSHING ) {
+ clean_and_invalidate_dcache_va_range(cfg, sizeof(*cfg));
+ }
+}
+
/*
* Allocate a block of 32 LPIs on the given host ITS for device "devid",
* starting with "eventid". Put them into the respective ITT by issuing a
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index aa54e9a364..9f0ea9ccb1 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -157,6 +157,31 @@ int gicv3_its_setup_collection(unsigned int cpu);
#ifdef CONFIG_HAS_ITS
+struct __lpi_data {
+ /* The global LPI property table, shared by all redistributors. */
+ uint8_t *lpi_property;
+ /*
+ * A two-level table to lookup LPIs firing on the host and look up the
+ * VCPU and virtual LPI number to inject into.
+ */
+ union host_lpi **host_lpis;
+ /*
+ * Number of physical LPIs the host supports. This is a property of
+ * the GIC hardware. We depart from the habit of naming these things
+ * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
+ * in a different context to differentiate them from "virtual LPIs".
+ */
+ unsigned long int max_host_lpi_ids;
+ /*
+ * Protects allocation and deallocation of host LPIs and next_free_lpi,
+ * but not the actual data stored in the host_lpi entry.
+ */
+ spinlock_t host_lpis_lock;
+ uint32_t next_free_lpi;
+ unsigned int flags;
+};
+extern struct __lpi_data lpi_data;
+
extern struct list_head host_its_list;
int its_send_cmd_inv(struct host_its *its, uint32_t deviceid, uint32_t eventid);
@@ -223,6 +248,8 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
uint32_t virt_lpi);
void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
uint32_t virt_lpi);
+void lpi_write_config(uint8_t *prop_table, uint32_t lpi, uint8_t clr,
+ uint8_t set);
int its_send_command(struct host_its *hw_its, const void *its_cmd);
struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 03/19] arm/its: Export struct its_device
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 01/19] arm/gicv4 add management structure definitions Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 02/19] arm/gicv4-its: Add GICv4 ITS command definitions Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 04/19] arm/its: Add vlpi configuration Mykyta Poturai
` (16 subsequent siblings)
19 siblings, 0 replies; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Make struct its_device and its commands sending methods publicly
available so they can be used by GICv4 implementation.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 30 +++++----------------------
xen/arch/arm/include/asm/gic_v3_its.h | 29 ++++++++++++++++++++++++++
2 files changed, 34 insertions(+), 25 deletions(-)
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 34833166ad..25c07eb861 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -30,26 +30,6 @@
*/
LIST_HEAD(host_its_list);
-/*
- * Describes a device which is using the ITS and is used by a guest.
- * Since device IDs are per ITS (in contrast to vLPIs, which are per
- * guest), we have to differentiate between different virtual ITSes.
- * We use the doorbell address here, since this is a nice architectural
- * property of MSIs in general and we can easily get to the base address
- * of the ITS and look that up.
- */
-struct its_device {
- struct rb_node rbnode;
- struct host_its *hw_its;
- void *itt_addr;
- unsigned int itt_order;
- paddr_t guest_doorbell; /* Identifies the virtual ITS */
- uint32_t host_devid;
- uint32_t guest_devid;
- uint32_t eventids; /* Number of event IDs (MSIs) */
- uint32_t *host_lpi_blocks; /* Which LPIs are used on the host */
- struct pending_irq *pend_irqs; /* One struct per event */
-};
/*
* It is unlikely that a platform implements ITSes with different quirks,
@@ -155,7 +135,7 @@ bool gicv3_its_host_has_its(void)
}
#define BUFPTR_MASK GENMASK(19, 5)
-static int its_send_command(struct host_its *hw_its, const void *its_cmd)
+int its_send_command(struct host_its *hw_its, const void *its_cmd)
{
/*
* The command queue should actually never become full, if it does anyway
@@ -258,7 +238,7 @@ static uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
return reg;
}
-static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
+int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
{
uint64_t cmd[4];
@@ -270,7 +250,7 @@ static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
return its_send_command(its, cmd);
}
-static int its_send_cmd_mapti(struct host_its *its,
+int its_send_cmd_mapti(struct host_its *its,
uint32_t deviceid, uint32_t eventid,
uint32_t pintid, uint16_t icid)
{
@@ -322,7 +302,7 @@ static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
return its_send_command(its, cmd);
}
-static int its_send_cmd_inv(struct host_its *its,
+int its_send_cmd_inv(struct host_its *its,
uint32_t deviceid, uint32_t eventid)
{
uint64_t cmd[4];
@@ -897,7 +877,7 @@ out:
}
/* Must be called with the its_device_lock held. */
-static struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
+struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
uint32_t vdevid)
{
struct rb_node *node = d->arch.vgic.its_devices.rb_node;
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index fc5a84892c..aa54e9a364 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -116,6 +116,27 @@
/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
#define LPI_BLOCK 32U
+/*
+ * Describes a device which is using the ITS and is used by a guest.
+ * Since device IDs are per ITS (in contrast to vLPIs, which are per
+ * guest), we have to differentiate between different virtual ITSes.
+ * We use the doorbell address here, since this is a nice architectural
+ * property of MSIs in general and we can easily get to the base address
+ * of the ITS and look that up.
+ */
+struct its_device {
+ struct rb_node rbnode;
+ struct host_its *hw_its;
+ unsigned int itt_order;
+ void *itt_addr;
+ paddr_t guest_doorbell; /* Identifies the virtual ITS */
+ uint32_t host_devid;
+ uint32_t guest_devid;
+ uint32_t eventids; /* Number of event IDs (MSIs) */
+ uint32_t *host_lpi_blocks; /* Which LPIs are used on the host */
+ struct pending_irq *pend_irqs; /* One struct per event */
+};
+
/* data structure for each hardware ITS */
struct host_its {
struct list_head entry;
@@ -138,6 +159,10 @@ int gicv3_its_setup_collection(unsigned int cpu);
extern struct list_head host_its_list;
+int its_send_cmd_inv(struct host_its *its, uint32_t deviceid, uint32_t eventid);
+int its_send_cmd_clear(struct host_its *its, uint32_t deviceid, uint32_t eventid);
+int its_send_cmd_mapti(struct host_its *its, uint32_t deviceid,
+ uint32_t eventid, uint32_t pintid, uint16_t icid);
#ifdef CONFIG_ACPI
unsigned long gicv3_its_make_hwdom_madt(const struct domain *d,
void *base_ptr);
@@ -198,6 +223,10 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
uint32_t virt_lpi);
void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
uint32_t virt_lpi);
+int its_send_command(struct host_its *hw_its, const void *its_cmd);
+
+struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
+ uint32_t vdevid);
/* ITS quirks handling. */
uint64_t gicv3_its_get_cacheability(void);
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 07/19] xen/domain: Alloc enough pages for VCPU struct
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (4 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 06/19] arm/gicv4-its: Add VLPI map/unmap operations Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-02 16:45 ` Jan Beulich
2026-02-02 16:14 ` [RFC PATCH 05/19] arm/irq: Add hw flag to pending_irq Mykyta Poturai
` (13 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Andrew Cooper,
Anthony PERARD, Michal Orzel, Jan Beulich, Julien Grall,
Roger Pau Monné, Stefano Stabellini
With introduction of GICv4 the size of struct vcpu can again be more
than one page. Modify struct vcpu allocation to request enough pages
again.
Don't reintroduce the MAX_PAGES_PER_VCPU check.
As per commit b77d774d8274183c2252f5fbc9fa3b3b7022ba06
> It turns out that beyond efficiency, maybe, there is no real technical
> reason this struct has to fit in one page
Since there is no technical reason to limit struct vcpu size to one page,
there also seems to be little reason the fiddle with 1 or 2 page limits.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/common/domain.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 376351b528..c791fb5033 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -342,18 +342,24 @@ static struct vcpu *alloc_vcpu_struct(const struct domain *d)
# define arch_vcpu_struct_memflags(d) ((void)(d), 0)
#endif
struct vcpu *v;
+ unsigned int order = get_order_from_bytes(sizeof(*v));
- BUILD_BUG_ON(sizeof(*v) > PAGE_SIZE);
- v = alloc_xenheap_pages(0, arch_vcpu_struct_memflags(d));
+ v = alloc_xenheap_pages(order, arch_vcpu_struct_memflags(d));
if ( v )
- clear_page(v);
+ {
+ unsigned int i;
+
+ for ( i = 0; i < DIV_ROUND_UP(sizeof(*v), PAGE_SIZE); i++ )
+ clear_page((void *)v + i * PAGE_SIZE);
+ }
return v;
}
static void free_vcpu_struct(struct vcpu *v)
{
- free_xenheap_page(v);
+ unsigned int order = get_order_from_bytes(sizeof(*v));
+ free_xenheap_pages(v, order);
}
static void vmtrace_free_buffer(struct vcpu *v)
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 06/19] arm/gicv4-its: Add VLPI map/unmap operations
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (3 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 04/19] arm/its: Add vlpi configuration Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-11 8:31 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 07/19] xen/domain: Alloc enough pages for VCPU struct Mykyta Poturai
` (14 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
For VLPI to be injected into a guest, it needs to be mapped or moved to
a corresponding VPE first. Add a struct to handle the info about the
VLPI mapping and a flag indicating whether the IRQ is tied to a HW one.
Implement mapping/unmapping of VLPIs to VPEs, also handle moving. Tie
them to emulated MAPTI/MOVI/DISCARD commands.
Add GIC_IRQ_GUEST_FORWARDED IRQ status flag to keep track of which LPIs
are mapped to virtual ones.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 14 ++
xen/arch/arm/gic-v4-its.c | 292 ++++++++++++++++++++++++++
xen/arch/arm/include/asm/gic_v3_its.h | 20 ++
xen/arch/arm/include/asm/gic_v4_its.h | 20 ++
xen/arch/arm/include/asm/vgic.h | 5 +
xen/arch/arm/vgic-v3-its.c | 42 +++-
6 files changed, 387 insertions(+), 6 deletions(-)
create mode 100644 xen/arch/arm/gic-v4-its.c
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 25c07eb861..25889445f5 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -315,6 +315,20 @@ int its_send_cmd_inv(struct host_its *its,
return its_send_command(its, cmd);
}
+int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
+ uint32_t eventid)
+{
+ uint64_t cmd[4];
+ uint32_t deviceid = dev->host_devid;
+
+ cmd[0] = GITS_CMD_DISCARD | ((uint64_t)deviceid << 32);
+ cmd[1] = (uint64_t)eventid;
+ cmd[2] = 0x00;
+ cmd[3] = 0x00;
+
+ return its_send_command(its, cmd);
+}
+
/* Set up the (1:1) collection mapping for the given host CPU. */
int gicv3_its_setup_collection(unsigned int cpu)
{
diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
new file mode 100644
index 0000000000..9bbd0d96b7
--- /dev/null
+++ b/xen/arch/arm/gic-v4-its.c
@@ -0,0 +1,292 @@
+/*
+ * xen/arch/arm/gic-v4-its.c
+ *
+ * ARM Generic Interrupt Controller support v4 version
+ * based on xen/arch/arm/gic-v3-its.c and kernel GICv4 driver
+ *
+ * Copyright (C) 2023 - ARM Ltd
+ * Penny Zheng <penny.zheng@arm.com>, ARM Ltd ported to Xen
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/errno.h>
+#include <xen/sched.h>
+#include <xen/spinlock.h>
+#include <asm/gic_v3_defs.h>
+#include <asm/gic_v3_its.h>
+#include <asm/gic_v4_its.h>
+#include <asm/vgic.h>
+
+
+static int its_send_cmd_vsync(struct host_its *its, uint16_t vpeid)
+{
+ uint64_t cmd[4];
+
+ cmd[0] = GITS_CMD_VSYNC;
+ cmd[1] = (uint64_t)vpeid << 32;
+ cmd[2] = 0x00;
+ cmd[3] = 0x00;
+
+ return its_send_command(its, cmd);
+}
+
+static int its_send_cmd_vmapti(struct host_its *its, struct its_device *dev,
+ uint32_t eventid)
+{
+ uint64_t cmd[4];
+ uint32_t deviceid = dev->host_devid;
+ struct its_vlpi_map *map = &dev->event_map.vlpi_maps[eventid];
+ uint16_t vpeid = map->vm->vpes[map->vpe_idx]->vpe_id;
+ uint32_t vintid = map->vintid;
+ uint32_t db_pintid;
+
+ if ( map->db_enabled )
+ db_pintid = map->vm->vpes[map->vpe_idx]->vpe_db_lpi;
+ else
+ db_pintid = INVALID_LPI;
+
+ cmd[0] = GITS_CMD_VMAPTI | ((uint64_t)deviceid << 32);
+ cmd[1] = eventid | ((uint64_t)vpeid << 32);
+ cmd[2] = vintid | ((uint64_t)db_pintid << 32);
+ cmd[3] = 0x00;
+
+ return its_send_command(its, cmd);
+}
+
+static bool pirq_is_forwarded_to_vcpu(struct pending_irq *pirq)
+{
+ ASSERT(pirq);
+ return test_bit(GIC_IRQ_GUEST_FORWARDED, &pirq->status);
+}
+
+bool event_is_forwarded_to_vcpu(struct its_device *dev, uint32_t eventid)
+{
+ struct pending_irq *pirq;
+
+ /* No vlpi maps at all ? */
+ if ( !dev->event_map.vlpi_maps)
+ return false;
+
+ pirq = dev->event_map.vlpi_maps[eventid].pirq;
+ return pirq_is_forwarded_to_vcpu(pirq);
+}
+
+static int its_send_cmd_vmovi(struct host_its *its, struct its_vlpi_map *map)
+{
+ uint64_t cmd[4];
+ struct its_device *dev = map->dev;
+ uint32_t eventid = map->eventid;
+ uint32_t deviceid = dev->host_devid;
+ uint16_t vpeid = map->vm->vpes[map->vpe_idx]->vpe_id;
+ uint32_t db_pintid;
+
+ if ( map->db_enabled )
+ db_pintid = map->vm->vpes[map->vpe_idx]->vpe_db_lpi;
+ else
+ db_pintid = INVALID_IRQ;
+
+ cmd[0] = GITS_CMD_VMOVI | ((uint64_t)deviceid << 32);
+ cmd[1] = eventid | ((uint64_t)vpeid << 32);
+ cmd[2] = (map->db_enabled ? 1UL : 0UL) | ((uint64_t)db_pintid << 32);
+ cmd[3] = 0x00;
+
+ return its_send_command(its, cmd);
+}
+
+static int gicv4_its_vlpi_map(struct its_vlpi_map *map)
+{
+ struct its_device *dev;
+ struct host_its *its;
+ uint32_t eventid;
+ int ret;
+
+ if ( !map )
+ return -EINVAL;
+ dev = map->dev;
+ its = map->dev->hw_its;
+ eventid = map->eventid;
+
+ spin_lock(&dev->event_map.vlpi_lock);
+
+ if ( !dev->event_map.vm )
+ {
+ struct its_vlpi_map *maps;
+
+ maps = xzalloc_array(struct its_vlpi_map, dev->event_map.nr_lpis);
+ if ( !maps )
+ {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dev->event_map.vm = map->vm;
+ dev->event_map.vlpi_maps = maps;
+ }
+ else if ( dev->event_map.vm != map->vm )
+ {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ /* Get our private copy of the mapping information */
+ dev->event_map.vlpi_maps[eventid] = *map;
+
+ if ( pirq_is_forwarded_to_vcpu(map->pirq) )
+ {
+ struct its_vlpi_map *old = &dev->event_map.vlpi_maps[eventid];
+ uint32_t old_vpeid = old->vm->vpes[old->vpe_idx]->vpe_id;
+
+ /* Already mapped, move it around */
+ ret = its_send_cmd_vmovi(dev->hw_its, map);
+ if ( ret )
+ goto err;
+
+ /*
+ * ARM spec says that If, after using VMOVI to move an interrupt from
+ * vPE A to vPE B, software moves the same interrupt again, a VSYNC
+ * command must be issued to vPE A between the moves to ensure correct
+ * behavior.
+ * So each time we issue VMOVI, we VSYNC the old VPE for good measure.
+ */
+ ret = its_send_cmd_vsync(dev->hw_its, old_vpeid);
+ }
+ else
+ {
+ /* Drop the original physical mapping firstly */
+ ret = its_send_cmd_discard(its, dev, eventid);
+ if ( ret )
+ goto err;
+
+ /* Then install the virtual one */
+ ret = its_send_cmd_vmapti(its, dev, eventid);
+ if ( ret )
+ goto err;
+
+ /* Increment the number of VLPIs */
+ dev->event_map.nr_vlpis++;
+ }
+
+ goto out;
+
+ err:
+ xfree(dev->event_map.vlpi_maps);
+ out:
+ spin_unlock(&dev->event_map.vlpi_lock);
+ return ret;
+}
+int gicv4_its_vlpi_unmap(struct pending_irq *pirq)
+{
+ struct its_vlpi_map *map = pirq->vlpi_map;
+ struct its_device *dev = map->dev;
+ int ret;
+ uint32_t host_lpi;
+
+ spin_lock(&dev->event_map.vlpi_lock);
+
+ if ( !dev->event_map.vm || !pirq_is_tied_to_hw(pirq) )
+ {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /* Drop the virtual mapping */
+ ret = its_send_cmd_discard(dev->hw_its, dev, map->eventid);
+ if ( ret )
+ goto out;
+
+ /* Restore the physical one */
+ clear_bit(GIC_IRQ_GUEST_FORWARDED, &pirq->status);
+ host_lpi = dev->host_lpi_blocks[map->eventid / LPI_BLOCK] +
+ (map->eventid % LPI_BLOCK);
+ /* Map every host LPI to host CPU 0 */
+ ret = its_send_cmd_mapti(dev->hw_its, dev->host_devid, map->eventid,
+ host_lpi, 0);
+ if ( ret )
+ goto out;
+
+ lpi_write_config(lpi_data.lpi_property, host_lpi, 0xff, LPI_PROP_ENABLED);
+
+ ret = its_inv_lpi(dev->hw_its, dev, map->eventid, 0);
+ if ( ret )
+ goto out;
+
+ xfree(map);
+ /*
+ * Drop the refcount and make the device available again if
+ * this was the last VLPI.
+ */
+ if ( !--dev->event_map.nr_vlpis )
+ {
+ dev->event_map.vm = NULL;
+ xfree(dev->event_map.vlpi_maps);
+ }
+
+out:
+ spin_unlock(&dev->event_map.vlpi_lock);
+ return ret;
+}
+
+int gicv4_assign_guest_event(struct domain *d, paddr_t vdoorbell_address,
+ uint32_t vdevid, uint32_t eventid,
+ struct pending_irq *pirq)
+
+{
+ int ret = ENODEV;
+ struct its_vm *vm = d->arch.vgic.its_vm;
+ struct its_vlpi_map *map;
+ struct its_device *dev;
+
+ spin_lock(&d->arch.vgic.its_devices_lock);
+ dev = get_its_device(d, vdoorbell_address, vdevid);
+ if ( dev && eventid < dev->eventids )
+ {
+ /* Prepare the vlpi mapping info */
+ map = xzalloc(struct its_vlpi_map);
+ if ( !map )
+ goto out;
+ map->vm = vm;
+ map->vintid = pirq->irq;
+ map->db_enabled = true;
+ map->vpe_idx = pirq->lpi_vcpu_id;
+ map->properties = pirq->lpi_priority |
+ (test_bit(GIC_IRQ_GUEST_ENABLED, &pirq->status) ?
+ LPI_PROP_ENABLED : 0);
+ map->pirq = pirq;
+ map->dev = dev;
+ map->eventid = eventid;
+
+ ret = gicv4_its_vlpi_map(map);
+ if ( ret )
+ {
+ xfree(map);
+ goto out;
+ }
+
+ pirq->vlpi_map = map;
+ }
+
+ out:
+ spin_unlock(&d->arch.vgic.its_devices_lock);
+ return ret;
+}
+
+int gicv4_its_vlpi_move(struct pending_irq *pirq, struct vcpu *vcpu)
+{
+ struct its_vlpi_map *map = pirq->vlpi_map;
+ struct its_device *dev = map->dev;
+
+ if ( !dev->event_map.vm || !map )
+ return -EINVAL;
+
+ map->vpe_idx = vcpu->vcpu_id;
+ return gicv4_its_vlpi_map(map);
+}
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index 9f0ea9ccb1..75c91c0426 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -116,6 +116,9 @@
/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
#define LPI_BLOCK 32U
+#ifdef CONFIG_GICV4
+#include <asm/gic_v4_its.h>
+#endif
/*
* Describes a device which is using the ITS and is used by a guest.
* Since device IDs are per ITS (in contrast to vLPIs, which are per
@@ -135,6 +138,9 @@ struct its_device {
uint32_t eventids; /* Number of event IDs (MSIs) */
uint32_t *host_lpi_blocks; /* Which LPIs are used on the host */
struct pending_irq *pend_irqs; /* One struct per event */
+#ifdef CONFIG_GICV4
+ struct event_vlpi_map event_map;
+#endif
};
/* data structure for each hardware ITS */
@@ -184,6 +190,8 @@ extern struct __lpi_data lpi_data;
extern struct list_head host_its_list;
+int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
+ uint32_t eventid);
int its_send_cmd_inv(struct host_its *its, uint32_t deviceid, uint32_t eventid);
int its_send_cmd_clear(struct host_its *its, uint32_t deviceid, uint32_t eventid);
int its_send_cmd_mapti(struct host_its *its, uint32_t deviceid,
@@ -254,6 +262,18 @@ int its_send_command(struct host_its *hw_its, const void *its_cmd);
struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
uint32_t vdevid);
+/* GICv4 functions */
+int gicv4_assign_guest_event(struct domain *d, paddr_t vdoorbell_address,
+ uint32_t vdevid, uint32_t eventid,
+ struct pending_irq *pirq);
+int gicv4_its_vlpi_move(struct pending_irq *pirq, struct vcpu *vcpu);
+#ifndef CONFIG_GICV4
+#define event_is_forwarded_to_vcpu(dev, eventid) ((void)dev, (void)eventid, false)
+#else
+bool event_is_forwarded_to_vcpu(struct its_device *dev, uint32_t eventid);
+void its_vpe_mask_db(struct its_vpe *vpe);
+#endif
+int gicv4_its_vlpi_unmap(struct pending_irq *pirq);
/* ITS quirks handling. */
uint64_t gicv3_its_get_cacheability(void);
diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
index f48eae60ad..722247ec60 100644
--- a/xen/arch/arm/include/asm/gic_v4_its.h
+++ b/xen/arch/arm/include/asm/gic_v4_its.h
@@ -29,6 +29,26 @@
#define GITS_CMD_VINVALL 0x2d
#define GITS_CMD_INVDB 0x2e
+/* Describes the mapping of a VLPI */
+struct its_vlpi_map {
+ struct its_vm *vm;
+ unsigned int vpe_idx; /* Index of the VPE */
+ uint32_t vintid; /* Virtual LPI number */
+ bool db_enabled; /* Is the VPE doorbell to be generated? */
+ uint8_t properties;
+ struct pending_irq *pirq;
+ struct its_device *dev;
+ uint32_t eventid;
+};
+
+struct event_vlpi_map {
+ unsigned int nr_lpis;
+ spinlock_t vlpi_lock;
+ struct its_vm *vm;
+ struct its_vlpi_map *vlpi_maps;
+ unsigned int nr_vlpis;
+};
+
#endif
/*
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 77323b2584..360f8a968e 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -70,6 +70,7 @@ struct pending_irq
* LPI with the same number in an LR must be from an older LPI, which
* has been unmapped before.
*
+ * GIC_IRQ_GUEST_FORWARDED: the IRQ is forwarded to a VCPU(GICv4 only)
*/
#define GIC_IRQ_GUEST_QUEUED 0
#define GIC_IRQ_GUEST_ACTIVE 1
@@ -77,6 +78,7 @@ struct pending_irq
#define GIC_IRQ_GUEST_ENABLED 3
#define GIC_IRQ_GUEST_MIGRATING 4
#define GIC_IRQ_GUEST_PRISTINE_LPI 5
+#define GIC_IRQ_GUEST_FORWARDED 6
unsigned long status;
struct irq_desc *desc; /* only set if the irq corresponds to a physical irq */
unsigned int irq;
@@ -95,6 +97,9 @@ struct pending_irq
* vgic lock is not going to be enough. */
struct list_head lr_queue;
bool hw; /* Tied to HW IRQ */
+#ifdef CONFIG_GICV4
+ struct its_vlpi_map *vlpi_map;
+#endif
};
#define NR_INTERRUPT_PER_RANK 32
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 576e7fd4b0..94f7dd7d90 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -589,6 +589,14 @@ static int its_discard_event(struct virt_its *its,
if ( vlpi == INVALID_LPI )
return -ENOENT;
+ p = gicv3_its_get_event_pending_irq(its->d, its->doorbell_address,
+ vdevid, vevid);
+ if ( unlikely(!p) )
+ return -EINVAL;
+
+ if ( pirq_is_tied_to_hw(p) )
+ if ( gicv4_its_vlpi_unmap(p) )
+ return -EINVAL;
/*
* TODO: This relies on the VCPU being correct in the ITS tables.
* This can be fixed by either using a per-IRQ lock or by using
@@ -751,6 +759,27 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
vgic_init_pending_irq(pirq, intid, gic_is_gicv4());
+ pirq->lpi_vcpu_id = vcpu->vcpu_id;
+
+ if ( pirq_is_tied_to_hw(pirq) )
+ /*
+ * If on GICv4, we could let the VLPI being directly injected
+ * to the guest. To achieve that, the VLPI must be mapped using
+ * the VMAPTI command.
+ */
+ if ( gicv4_assign_guest_event(its->d, its->doorbell_address, devid,
+ eventid, pirq) )
+ goto out_remove_mapping;
+
+ if ( pirq_is_tied_to_hw(pirq) )
+ set_bit(GIC_IRQ_GUEST_FORWARDED, &pirq->status);
+ else
+ /*
+ * Mark this LPI as new, so any older (now unmapped) LPI in any LR
+ * can be easily recognised as such.
+ */
+ set_bit(GIC_IRQ_GUEST_PRISTINE_LPI, &pirq->status);
+
/*
* Now read the guest's property table to initialize our cached state.
* We don't need the VGIC VCPU lock here, because the pending_irq isn't
@@ -761,12 +790,6 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
goto out_remove_host_entry;
pirq->lpi_vcpu_id = vcpu->vcpu_id;
- /*
- * Mark this LPI as new, so any older (now unmapped) LPI in any LR
- * can be easily recognised as such.
- */
- set_bit(GIC_IRQ_GUEST_PRISTINE_LPI, &pirq->status);
-
/*
* Now insert the pending_irq into the domain's LPI tree, so that
* it becomes live.
@@ -824,6 +847,13 @@ static int its_handle_movi(struct virt_its *its, uint64_t *cmdptr)
if ( unlikely(!p) )
goto out_unlock;
+ if ( pirq_is_tied_to_hw(p) )
+ {
+ ret = gicv4_its_vlpi_move(p, nvcpu);
+ if ( ret )
+ goto out_unlock;
+ }
+
/*
* TODO: This relies on the VCPU being correct in the ITS tables.
* This can be fixed by either using a per-IRQ lock or by using
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 05/19] arm/irq: Add hw flag to pending_irq
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (5 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 07/19] xen/domain: Alloc enough pages for VCPU struct Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 09/19] arm/its: Implement LPI invalidation Mykyta Poturai
` (12 subsequent siblings)
19 siblings, 0 replies; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Add a new flag to vgic_init_pending_irq() so that we can record
whether a virtual interrupt is tied to a hardware one. The ITS code sets
the flag when mapping an LPI on GICv4 systems and the cleanup path skips
removing such IRQs from the virtual queues to avoid interfering with
hardware-backed injections.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/include/asm/vgic.h | 15 ++++++++++++++-
xen/arch/arm/vgic-v3-its.c | 4 ++--
xen/arch/arm/vgic.c | 10 ++++++----
3 files changed, 22 insertions(+), 7 deletions(-)
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index a874a02d70..77323b2584 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -94,11 +94,23 @@ struct pending_irq
* TODO: when implementing irq migration, taking only the current
* vgic lock is not going to be enough. */
struct list_head lr_queue;
+ bool hw; /* Tied to HW IRQ */
};
#define NR_INTERRUPT_PER_RANK 32
#define INTERRUPT_RANK_MASK (NR_INTERRUPT_PER_RANK - 1)
+#ifdef CONFIG_GICV4
+static inline bool pirq_is_tied_to_hw(struct pending_irq *pirq)
+{
+ ASSERT(pirq);
+ return pirq->hw;
+}
+
+#else
+#define pirq_is_tied_to_hw(pirq) ((void)pirq, false)
+#endif
+
/* Represents state corresponding to a block of 32 interrupts */
struct vgic_irq_rank {
spinlock_t lock; /* Covers access to all other members of this struct */
@@ -360,7 +372,8 @@ static inline paddr_t vgic_dist_base(const struct vgic_dist *vgic)
extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
-extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
+extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq,
+ bool hw);
extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
extern struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq);
extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v,
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index bc738614bb..576e7fd4b0 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -610,7 +610,7 @@ static int its_discard_event(struct virt_its *its,
/* Cleanup the pending_irq and disconnect it from the LPI. */
vgic_remove_irq_from_queues(vcpu, p);
- vgic_init_pending_irq(p, INVALID_LPI);
+ vgic_init_pending_irq(p, INVALID_LPI, false);
spin_unlock_irqrestore(&vcpu->arch.vgic.lock, flags);
@@ -749,7 +749,7 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
if ( !pirq )
goto out_remove_mapping;
- vgic_init_pending_irq(pirq, intid);
+ vgic_init_pending_irq(pirq, intid, gic_is_gicv4());
/*
* Now read the guest's property table to initialize our cached state.
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 6647071ad4..0da8c1a425 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -112,7 +112,7 @@ struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq)
return vgic_get_rank(v, rank);
}
-void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
+void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq, bool hw)
{
/* The lpi_vcpu_id field must be big enough to hold a VCPU ID. */
BUILD_BUG_ON(BIT(sizeof(p->lpi_vcpu_id) * 8, UL) < MAX_VIRT_CPUS);
@@ -122,6 +122,8 @@ void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
INIT_LIST_HEAD(&p->lr_queue);
p->irq = virq;
p->lpi_vcpu_id = INVALID_VCPU_ID;
+ /* Whether virtual irq is tied to a HW one. */
+ p->hw = hw;
}
static void vgic_rank_init(struct vgic_irq_rank *rank, uint8_t index,
@@ -202,7 +204,7 @@ static int init_vgic_espi(struct domain *d)
for ( i = d->arch.vgic.nr_spis, idx = 0;
i < vgic_num_spi_lines(d); i++, idx++ )
vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
- espi_idx_to_intid(idx));
+ espi_idx_to_intid(idx), false);
for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i],
@@ -304,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
return -ENOMEM;
for (i=0; i<d->arch.vgic.nr_spis; i++)
- vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i], i + 32);
+ vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i], i + 32, false);
/* SPIs are routed to VCPU0 by default */
for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
@@ -381,7 +383,7 @@ int vcpu_vgic_init(struct vcpu *v)
memset(&v->arch.vgic.pending_irqs, 0, sizeof(v->arch.vgic.pending_irqs));
for (i = 0; i < 32; i++)
- vgic_init_pending_irq(&v->arch.vgic.pending_irqs[i], i);
+ vgic_init_pending_irq(&v->arch.vgic.pending_irqs[i], i, false);
INIT_LIST_HEAD(&v->arch.vgic.inflight_irqs);
INIT_LIST_HEAD(&v->arch.vgic.lr_pending);
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 09/19] arm/its: Implement LPI invalidation
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (6 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 05/19] arm/irq: Add hw flag to pending_irq Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 08/19] arm/gic: Keep track of GIC features Mykyta Poturai
` (11 subsequent siblings)
19 siblings, 0 replies; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Add helpers for LPI invalidation.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 21 ++++++++++++++++++++-
xen/arch/arm/gic-v4-its.c | 22 ++++++++++++++++++++++
xen/arch/arm/include/asm/gic_v3_its.h | 5 +++++
3 files changed, 47 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 25889445f5..c628959f42 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -201,7 +201,7 @@ int its_send_command(struct host_its *hw_its, const void *its_cmd)
}
/* Wait for an ITS to finish processing all commands. */
-static int gicv3_its_wait_commands(struct host_its *hw_its)
+int gicv3_its_wait_commands(struct host_its *hw_its)
{
/*
* As there could be quite a number of commands in a queue, we will
@@ -672,6 +672,25 @@ static int compare_its_guest_devices(struct its_device *dev,
return 0;
}
+int its_inv_lpi(struct host_its *its, struct its_device *dev,
+ uint32_t eventid, unsigned int cpu)
+{
+ int ret;
+
+ if ( event_is_forwarded_to_vcpu(dev, eventid) )
+ return its_send_cmd_vinv(its, dev, eventid);
+
+ ret = its_send_cmd_inv(its, dev->host_devid, eventid);
+ if ( ret )
+ return ret;
+
+ ret = its_send_cmd_sync(its, cpu);
+ if ( ret )
+ return ret;
+
+ return gicv3_its_wait_commands(its);
+}
+
/*
* On the host ITS @its, map @nr_events consecutive LPIs.
* The mapping connects a device @devid and event @eventid pair to LPI @lpi,
diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
index 9bbd0d96b7..358d0bffb9 100644
--- a/xen/arch/arm/gic-v4-its.c
+++ b/xen/arch/arm/gic-v4-its.c
@@ -290,3 +290,25 @@ int gicv4_its_vlpi_move(struct pending_irq *pirq, struct vcpu *vcpu)
map->vpe_idx = vcpu->vcpu_id;
return gicv4_its_vlpi_map(map);
}
+
+/*
+ * There is no real VINV command.
+ * We do a normal INV, with a VSYNC instead of a SYNC.
+ */
+int its_send_cmd_vinv(struct host_its *its, struct its_device *dev,
+ uint32_t eventid)
+{
+ int ret;
+ struct its_vlpi_map *map = &dev->event_map.vlpi_maps[eventid];
+ uint16_t vpeid = map->vm->vpes[map->vpe_idx]->vpe_id;
+
+ ret = its_send_cmd_inv(its, dev->host_devid, eventid);
+ if ( ret )
+ return ret;
+
+ ret = its_send_cmd_vsync(its, vpeid);
+ if ( ret )
+ return ret;
+
+ return gicv3_its_wait_commands(its);
+}
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index 75c91c0426..973ca6acdd 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -194,6 +194,9 @@ int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
uint32_t eventid);
int its_send_cmd_inv(struct host_its *its, uint32_t deviceid, uint32_t eventid);
int its_send_cmd_clear(struct host_its *its, uint32_t deviceid, uint32_t eventid);
+int gicv3_its_wait_commands(struct host_its *hw_its);
+int its_inv_lpi(struct host_its *its, struct its_device *dev,
+ uint32_t eventid, unsigned int cpu);
int its_send_cmd_mapti(struct host_its *its, uint32_t deviceid,
uint32_t eventid, uint32_t pintid, uint16_t icid);
#ifdef CONFIG_ACPI
@@ -267,6 +270,8 @@ int gicv4_assign_guest_event(struct domain *d, paddr_t vdoorbell_address,
uint32_t vdevid, uint32_t eventid,
struct pending_irq *pirq);
int gicv4_its_vlpi_move(struct pending_irq *pirq, struct vcpu *vcpu);
+int its_send_cmd_vinv(struct host_its *its, struct its_device *dev,
+ uint32_t eventid);
#ifndef CONFIG_GICV4
#define event_is_forwarded_to_vcpu(dev, eventid) ((void)dev, (void)eventid, false)
#else
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 08/19] arm/gic: Keep track of GIC features
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (7 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 09/19] arm/its: Implement LPI invalidation Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-12 7:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 10/19] arm/its: Keep track of BASER regs Mykyta Poturai
` (10 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Different versions of GICv4 may support different features. Record them
and provide functions to check for their availability.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3.c | 175 ++++++++++++++++++++-----
xen/arch/arm/include/asm/gic.h | 2 +
xen/arch/arm/include/asm/gic_v3_defs.h | 9 ++
xen/arch/arm/include/asm/vgic.h | 9 ++
4 files changed, 162 insertions(+), 33 deletions(-)
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 9b8b87078b..14852d18c2 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -100,6 +100,38 @@ static struct {
static struct gic_info gicv3_info;
+#ifdef CONFIG_GICV4
+/* Global state */
+static struct {
+ bool has_vlpis;
+ bool has_direct_lpi;
+ bool has_vpend_valid_dirty;
+ bool has_rvpeid;
+} gicv4 = { .has_vlpis = true, .has_direct_lpi = true,
+ .has_vpend_valid_dirty = true, .has_rvpeid = true, };
+
+
+bool gic_support_directLPI(void)
+{
+ return gicv4.has_direct_lpi;
+}
+
+bool gic_support_vptValidDirty(void)
+{
+ return gicv4.has_vpend_valid_dirty;
+}
+
+bool gic_has_v4_1_extension(void)
+{
+ return gicv4.has_rvpeid;
+}
+
+bool gic_is_gicv4(void)
+{
+ return gicv4.has_vlpis;
+}
+#endif
+
/* per-cpu re-distributor base */
static DEFINE_PER_CPU(void __iomem*, rbase);
@@ -914,7 +946,8 @@ static bool gicv3_enable_lpis(void)
return true;
}
-static int __init gicv3_populate_rdist(void)
+static int __init gic_iterate_rdists(int (*fn)(struct rdist_region *,
+ void __iomem *))
{
int i;
uint32_t aff;
@@ -958,40 +991,16 @@ static int __init gicv3_populate_rdist(void)
if ( (typer >> 32) == aff )
{
+ int ret;
+
this_cpu(rbase) = ptr;
- if ( typer & GICR_TYPER_PLPIS )
- {
- paddr_t rdist_addr;
- unsigned int procnum;
- int ret;
-
- /*
- * The ITS refers to redistributors either by their physical
- * address or by their ID. Which one to use is an ITS
- * choice. So determine those two values here (which we
- * can do only here in GICv3 code) and tell the
- * ITS code about it, so it can use them later to be able
- * to address those redistributors accordingly.
- */
- rdist_addr = gicv3.rdist_regions[i].base;
- rdist_addr += ptr - gicv3.rdist_regions[i].map_base;
- procnum = (typer & GICR_TYPER_PROC_NUM_MASK);
- procnum >>= GICR_TYPER_PROC_NUM_SHIFT;
-
- gicv3_set_redist_address(rdist_addr, procnum);
-
- ret = gicv3_lpi_init_rdist(ptr);
- if ( ret && ret != -ENODEV )
- {
- printk("GICv3: CPU%d: Cannot initialize LPIs: %u\n",
- smp_processor_id(), ret);
- break;
- }
- }
-
- printk("GICv3: CPU%d: Found redistributor in region %d @%p\n",
- smp_processor_id(), i, ptr);
+ ret = fn(gicv3.rdist_regions + i, ptr);
+ if ( ret )
+ return ret;
+
+ printk("GICv3: CPU%d: Found redistributor @%p\n",
+ smp_processor_id(), ptr);
return 0;
}
@@ -1010,11 +1019,107 @@ static int __init gicv3_populate_rdist(void)
} while ( !(typer & GICR_TYPER_LAST) );
}
+ return -ENODEV;
+}
+
+static int __init __gicv3_populate_rdist(struct rdist_region *region,
+ void __iomem *ptr)
+{
+ uint64_t typer;
+
+ typer = readq_relaxed(ptr + GICR_TYPER);
+ if ( typer & GICR_TYPER_PLPIS )
+ {
+ paddr_t rdist_addr;
+ unsigned int procnum;
+ int ret;
+
+ /*
+ * The ITS refers to redistributors either by their physical
+ * address or by their ID. Which one to use is an ITS
+ * choice. So determine those two values here (which we
+ * can do only here in GICv3 code) and tell the
+ * ITS code about it, so it can use them later to be able
+ * to address those redistributors accordingly.
+ */
+ rdist_addr = region->base;
+ rdist_addr += ptr - region->map_base;
+ procnum = (typer & GICR_TYPER_PROC_NUM_MASK);
+ procnum >>= GICR_TYPER_PROC_NUM_SHIFT;
+
+ gicv3_set_redist_address(rdist_addr, procnum);
+
+ ret = gicv3_lpi_init_rdist(ptr);
+ if ( ret && ret != -ENODEV )
+ {
+ printk("GICv3: CPU%d: Cannot initialize LPIs: %d\n",
+ smp_processor_id(), ret);
+ printk("%s %d\n", __func__, __LINE__);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int __init gicv3_populate_rdist(void)
+{
+ int ret = gic_iterate_rdists(__gicv3_populate_rdist);
+ if ( ret == 0)
+ return 0;
+
dprintk(XENLOG_ERR, "GICv3: CPU%d: mpidr 0x%"PRIregister" has no re-distributor!\n",
smp_processor_id(), cpu_logical_map(smp_processor_id()));
+ return -ENODEV;
+}
+
+#ifdef CONFIG_GICV4
+static int __init __gicv4_update_vlpi_properties(struct rdist_region *region,
+ void __iomem *ptr)
+{
+ uint64_t typer;
+
+ typer = readq_relaxed(ptr + GICR_TYPER);
+ gicv4.has_vlpis &= !!(typer & GICR_TYPER_VLPIS);
+ gicv4.has_rvpeid &= !!(typer & GICR_TYPER_RVPEID);
+ /* RVPEID implies some form of DirectLPI. */
+ gicv4.has_direct_lpi &= (!!(typer & GICR_TYPER_DirectLPIS) ||
+ !!(typer & GICR_TYPER_RVPEID));
+ gicv4.has_vpend_valid_dirty &= !!(typer & GICR_TYPER_DIRTY);
+
+ /* Detect non-sensical configurations */
+ if ( gicv4.has_rvpeid && !gicv4.has_vlpis )
+ {
+ gicv4.has_direct_lpi = false;
+ gicv4.has_vlpis = false;
+ gicv4.has_rvpeid = false;
+ }
+
+ printk("GICv4: CPU%d: %sVLPI support, %sdirect LPI support, %sValid+Dirty support, %sRVPEID support\n",
+ smp_processor_id(), !!(typer & GICR_TYPER_VLPIS) ? "" : "no ",
+ (!!(typer & GICR_TYPER_DirectLPIS) ||
+ !!(typer & GICR_TYPER_RVPEID)) ? "" : "no ",
+ !!(typer & GICR_TYPER_DIRTY) ? "" : "no ",
+ !!(typer & GICR_TYPER_RVPEID) ? "" : "no ");
+
+ return 0;
+}
+
+static int __init gicv4_update_vlpi_properties(void)
+{
+ int ret = gic_iterate_rdists(__gicv4_update_vlpi_properties);
+
+ if ( ret == 0 )
+ return 0;
return -ENODEV;
}
+#else
+static int __init gicv4_update_vlpi_properties(void)
+{
+ return 0;
+}
+#endif
static int gicv3_cpu_init(void)
{
@@ -1024,6 +1129,10 @@ static int gicv3_cpu_init(void)
if ( gicv3_populate_rdist() )
return -ENODEV;
+ ret = gicv4_update_vlpi_properties();
+ if ( ret )
+ return ret;
+
if ( gicv3_enable_redist() )
return -ENODEV;
diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
index 8e713aa477..afb1cc3751 100644
--- a/xen/arch/arm/include/asm/gic.h
+++ b/xen/arch/arm/include/asm/gic.h
@@ -235,6 +235,8 @@ enum gic_version {
GIC_INVALID = 0, /* the default until explicitly set up */
GIC_V2,
GIC_V3,
+ GIC_V4,
+ GIC_V4_1,
};
DECLARE_PER_CPU(uint64_t, lr_mask);
diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
index c373b94d19..3a7d18ef59 100644
--- a/xen/arch/arm/include/asm/gic_v3_defs.h
+++ b/xen/arch/arm/include/asm/gic_v3_defs.h
@@ -93,6 +93,12 @@
#define GICD_TYPE_LPIS (1U << 17)
+#define GICD_TYPER2 0x000c
+
+#define GICD_TYPER2_VIL (1U << 7)
+#define GICD_TYPER2_VID GENMASK(4, 0)
+#define GICD_TYPER2_nASSGIcap (1U << 8)
+
#define GICD_CTLR_RWP (1UL << 31)
#define GICD_CTLR_ARE_NS (1U << 4)
#define GICD_CTLR_ENABLE_G1A (1U << 1)
@@ -149,7 +155,10 @@
#define GICR_TYPER_PLPIS (1U << 0)
#define GICR_TYPER_VLPIS (1U << 1)
+#define GICR_TYPER_DIRTY (1U << 2)
+#define GICR_TYPER_DirectLPIS (1U << 3)
#define GICR_TYPER_LAST (1U << 4)
+#define GICR_TYPER_RVPEID (1U << 7)
#define GICR_TYPER_PROC_NUM_SHIFT 8
#define GICR_TYPER_PROC_NUM_MASK (0xffff << GICR_TYPER_PROC_NUM_SHIFT)
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 360f8a968e..f12d736808 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -405,6 +405,15 @@ extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int ir
extern void vgic_check_inflight_irqs_pending(struct vcpu *v,
unsigned int rank, uint32_t r);
+/* GICV4 functions */
+#ifdef CONFIG_GICV4
+bool gic_support_vptValidDirty(void);
+bool gic_is_gicv4(void);
+#else
+#define gic_support_vptValidDirty() (false)
+#define gic_is_gicv4() (false)
+#endif
+
#endif /* !CONFIG_NEW_VGIC */
/*** Common VGIC functions used by Xen arch code ****/
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 10/19] arm/its: Keep track of BASER regs
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (8 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 08/19] arm/gic: Keep track of GIC features Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-12 7:52 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 11/19] arm/its: Add ITS VM and VPE allocation/teardown Mykyta Poturai
` (9 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Keep all the info about BASER regs in host_its structure to be able to easily
access it later.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 36 +++++++++++++++++++++------
xen/arch/arm/include/asm/gic_v3_its.h | 14 +++++++++++
2 files changed, 43 insertions(+), 7 deletions(-)
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index c628959f42..2328595a85 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -430,16 +430,32 @@ static void *its_map_cbaser(struct host_its *its)
/* The ITS BASE registers work with page sizes of 4K, 16K or 64K. */
#define BASER_PAGE_BITS(sz) ((sz) * 2 + 12)
+struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type)
+{
+ unsigned int i;
+
+ for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
+ {
+ if (GITS_BASER_TYPE(hw_its->tables[i].val) == type)
+ return &hw_its->tables[i];
+ }
+
+ return NULL;
+}
+
static int its_map_baser(void __iomem *basereg, uint64_t regc,
- unsigned int nr_items)
+ unsigned int nr_items, struct its_baser *baser)
{
uint64_t attr, reg;
unsigned int entry_size = GITS_BASER_ENTRY_SIZE(regc);
- unsigned int pagesz = 2; /* try 64K pages first, then go down. */
+ unsigned int page_size[4] = {SZ_4K, SZ_16K, SZ_64K, SZ_64K};
+ unsigned int pagesz = 0; /* try 64K pages first, then go down. */
unsigned int table_size;
unsigned int order;
void *buffer;
+ uint32_t type;
+ type = GITS_BASER_TYPE(regc);
attr = gicv3_its_get_shareability() << GITS_BASER_SHAREABILITY_SHIFT;
attr |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
attr |= gicv3_its_get_cacheability() << GITS_BASER_INNER_CACHEABILITY_SHIFT;
@@ -479,6 +495,11 @@ retry:
writeq_relaxed(reg, basereg);
regc = readq_relaxed(basereg);
+ baser->val = regc;
+ baser->base = buffer;
+ baser->table_size = table_size;
+ baser->pagesz = page_size[pagesz];
+
/* The host didn't like our attributes, just use what it returned. */
if ( (regc & BASER_ATTR_MASK) != attr )
{
@@ -490,7 +511,7 @@ retry:
}
attr = regc & BASER_ATTR_MASK;
}
- if ( (regc & GITS_BASER_INNER_CACHEABILITY_MASK) <= GIC_BASER_CACHE_nC )
+ if ( gicv3_its_get_cacheability() <= GIC_BASER_CACHE_nC )
clean_and_invalidate_dcache_va_range(buffer, table_size);
/* If the host accepted our page size, we are done. */
@@ -568,26 +589,27 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
{
void __iomem *basereg = hw_its->its_base + GITS_BASER0 + i * 8;
unsigned int type;
+ struct its_baser *baser = hw_its->tables + i;
reg = readq_relaxed(basereg);
- type = (reg & GITS_BASER_TYPE_MASK) >> GITS_BASER_TYPE_SHIFT;
+ type = GITS_BASER_TYPE(reg);
switch ( type )
{
case GITS_BASER_TYPE_NONE:
continue;
case GITS_BASER_TYPE_DEVICE:
- ret = its_map_baser(basereg, reg, BIT(hw_its->devid_bits, UL));
+ ret = its_map_baser(basereg, reg, BIT(hw_its->devid_bits, UL), baser);
if ( ret )
return ret;
break;
case GITS_BASER_TYPE_COLLECTION:
- ret = its_map_baser(basereg, reg, num_possible_cpus());
+ ret = its_map_baser(basereg, reg, num_possible_cpus(), baser);
if ( ret )
return ret;
break;
/* In case this is a GICv4, provide a (dummy) vPE table as well. */
case GITS_BASER_TYPE_VCPU:
- ret = its_map_baser(basereg, reg, 1);
+ ret = its_map_baser(basereg, reg, 32, baser);
if ( ret )
return ret;
break;
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index 973ca6acdd..bd2696f354 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -64,6 +64,7 @@
#define GITS_BASER_INNER_CACHEABILITY_SHIFT 59
#define GITS_BASER_TYPE_SHIFT 56
#define GITS_BASER_TYPE_MASK (7ULL << GITS_BASER_TYPE_SHIFT)
+#define GITS_BASER_TYPE(reg) ((reg & GITS_BASER_TYPE_MASK) >> GITS_BASER_TYPE_SHIFT)
#define GITS_BASER_OUTER_CACHEABILITY_SHIFT 53
#define GITS_BASER_TYPE_NONE 0UL
#define GITS_BASER_TYPE_DEVICE 1UL
@@ -143,6 +144,17 @@ struct its_device {
#endif
};
+/*
+ * The ITS_BASER structure - contains memory information, cached
+ * value of BASER register configuration.
+ */
+struct its_baser {
+ void *base;
+ uint64_t val;
+ unsigned int table_size;
+ unsigned int pagesz;
+};
+
/* data structure for each hardware ITS */
struct host_its {
struct list_head entry;
@@ -156,6 +168,7 @@ struct host_its {
spinlock_t cmd_lock;
void *cmd_buf;
unsigned int flags;
+ struct its_baser tables[GITS_BASER_NR_REGS];
};
/* Map a collection for this host CPU to each host ITS. */
@@ -259,6 +272,7 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
uint32_t virt_lpi);
void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
uint32_t virt_lpi);
+struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type);
void lpi_write_config(uint8_t *prop_table, uint32_t lpi, uint8_t clr,
uint8_t set);
int its_send_command(struct host_its *hw_its, const void *its_cmd);
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 11/19] arm/its: Add ITS VM and VPE allocation/teardown
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (9 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 10/19] arm/its: Keep track of BASER regs Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-18 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 12/19] arm/gic: Add VPENDBASER/VPROPBASER accessors Mykyta Poturai
` (8 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Do necessary allocations for GICv4 VLPI injection.
When creating a domain allocate its_vm and property tables.
For each VCPU allocate a VPe with a unique vpe id and separate pending table.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 157 ++++++++++++----
xen/arch/arm/gic-v3-lpi.c | 61 +++++-
xen/arch/arm/gic-v3.c | 18 ++
xen/arch/arm/gic-v4-its.c | 259 ++++++++++++++++++++++++++
xen/arch/arm/include/asm/gic_v3_its.h | 17 ++
xen/arch/arm/include/asm/gic_v4_its.h | 1 +
xen/arch/arm/include/asm/vgic.h | 3 +
xen/arch/arm/vgic.c | 25 ++-
8 files changed, 496 insertions(+), 45 deletions(-)
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 2328595a85..fb1d2709be 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -31,6 +31,8 @@
LIST_HEAD(host_its_list);
+unsigned int nvpeid = 16;
+
/*
* It is unlikely that a platform implements ITSes with different quirks,
* so assume they all share the same.
@@ -228,7 +230,7 @@ int gicv3_its_wait_commands(struct host_its *hw_its)
return -ETIMEDOUT;
}
-static uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
+uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
uint64_t reg)
{
reg &= ~GENMASK(51, 16);
@@ -443,6 +445,54 @@ struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type)
return NULL;
}
+bool its_alloc_table_entry(struct its_baser *baser, uint32_t id)
+{
+ uint64_t reg = baser->val;
+ bool indirect = reg & GITS_BASER_INDIRECT;
+ unsigned int idx;
+ __le64 *table;
+ unsigned int entry_size = GITS_BASER_ENTRY_SIZE(reg);
+
+ /* Don't allow id that exceeds single, flat table limit */
+ if ( !indirect )
+ return (id < (baser->table_size / entry_size));
+
+ /* Compute 1st level table index & check if that exceeds table limit */
+ idx = id / (baser->pagesz / entry_size);
+ if ( idx >= (baser->pagesz / GITS_LVL1_ENTRY_SIZE) )
+ return false;
+
+ table = baser->base;
+
+ /* Allocate memory for 2nd level table */
+ if (!table[idx])
+ {
+ unsigned int page_size = baser->pagesz;
+ void *buffer;
+
+ buffer = alloc_xenheap_pages(get_order_from_bytes(page_size),
+ gicv3_its_get_memflags());
+ if ( !buffer )
+ return -ENOMEM;
+
+ /* Flush Lvl2 table to PoC if hw doesn't support coherency */
+ if ( gicv3_its_get_cacheability() <= GIC_BASER_CACHE_nC )
+ clean_and_invalidate_dcache_va_range(buffer, page_size);
+
+ table[idx] = cpu_to_le64(virt_to_maddr(buffer) | GITS_VALID_BIT);
+
+ /* Flush Lvl1 entry to PoC if hw doesn't support coherency */
+ if ( gicv3_its_get_cacheability() <= GIC_BASER_CACHE_nC )
+ clean_and_invalidate_dcache_va_range(table + idx,
+ GITS_LVL1_ENTRY_SIZE);
+
+ /* Ensure updated table contents are visible to ITS hardware */
+ dsb(sy);
+ }
+
+ return true;
+}
+
static int its_map_baser(void __iomem *basereg, uint64_t regc,
unsigned int nr_items, struct its_baser *baser)
{
@@ -737,13 +787,75 @@ static int gicv3_its_map_host_events(struct host_its *its,
return ret;
}
- /* TODO: Consider using INVALL here. Didn't work on the model, though. */
+ return 0;
+}
+
+static bool its_alloc_device_table(struct host_its *hw_its, uint32_t dev_id)
+{
+ struct its_baser *baser;
+
+ baser = its_get_baser(hw_its, GITS_BASER_TYPE_DEVICE);
+ if ( !baser )
+ return false;
+
+ return its_alloc_table_entry(baser, dev_id);
+}
+
+struct its_device *its_create_device(struct host_its *hw_its,
+ uint32_t host_devid, uint64_t nr_events)
+{
+ void *itt_addr = NULL;
+ struct its_device *dev = NULL;
+ int ret;
+
+ /* Sanitise the provided hardware values against the host ITS. */
+ if ( host_devid >= BIT(hw_its->devid_bits, UL) )
+ return NULL;
+
+ dev = xzalloc(struct its_device);
+ if ( !dev )
+ return NULL;
+
+ /* An Interrupt Translation Table needs to be 256-byte aligned. */
+ dev->itt_order = get_order_from_bytes(nr_events * hw_its->itte_size);
+ itt_addr = alloc_xenheap_pages(dev->itt_order, gicv3_its_get_memflags());
+ if ( !itt_addr )
+ goto fail_dev;
+
+ clean_and_invalidate_dcache_va_range(itt_addr,
+ nr_events * hw_its->itte_size);
+
- ret = its_send_cmd_sync(its, 0);
+ if ( !its_alloc_device_table(hw_its, host_devid) )
+ goto fail_itt;
+
+ ret = its_send_cmd_mapd(hw_its, host_devid, max(fls(nr_events - 1), 1U),
+ virt_to_maddr(itt_addr), true);
if ( ret )
- return ret;
+ goto fail_itt;
- return gicv3_its_wait_commands(its);
+ dev->itt_addr = itt_addr;
+ dev->hw_its = hw_its;
+ dev->host_devid = host_devid;
+ dev->eventids = nr_events;
+
+ return dev;
+
+fail_itt:
+ free_xenheap_pages(itt_addr, dev->itt_order);
+fail_dev:
+ xfree(dev);
+
+ return NULL;
+}
+
+static void its_free_device(struct its_device *dev)
+{
+ xfree(dev->host_lpi_blocks);
+ xfree(dev->itt_addr);
+ if ( dev->pend_irqs )
+ xfree(dev->pend_irqs);
+ xfree(dev);
}
/*
@@ -758,12 +870,10 @@ int gicv3_its_map_guest_device(struct domain *d,
paddr_t guest_doorbell, uint32_t guest_devid,
uint64_t nr_events, bool valid)
{
- void *itt_addr = NULL;
struct host_its *hw_its;
struct its_device *dev = NULL;
struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
int i, ret = -ENOENT; /* "i" must be signed to check for >= 0 below. */
- unsigned int order;
hw_its = gicv3_its_find_by_doorbell(host_doorbell);
if ( !hw_its )
@@ -823,23 +933,12 @@ int gicv3_its_map_guest_device(struct domain *d,
if ( !valid )
goto out_unlock;
- ret = -ENOMEM;
-
- /* An Interrupt Translation Table needs to be 256-byte aligned. */
- order = get_order_from_bytes(max(nr_events * hw_its->itte_size, 256UL));
- itt_addr = alloc_xenheap_pages(order, gicv3_its_get_memflags());
- if ( !itt_addr )
- goto out_unlock;
-
- memset(itt_addr, 0, PAGE_SIZE << order);
-
- clean_and_invalidate_dcache_va_range(itt_addr,
- nr_events * hw_its->itte_size);
-
- dev = xzalloc(struct its_device);
+ dev = its_create_device(hw_its, host_devid, nr_events);
if ( !dev )
goto out_unlock;
+ ret = -ENOMEM;
+
/*
* Allocate the pending_irqs for each virtual LPI. They will be put
* into the domain's radix tree upon the guest's MAPTI command.
@@ -860,14 +959,6 @@ int gicv3_its_map_guest_device(struct domain *d,
if ( !dev->host_lpi_blocks )
goto out_unlock;
- ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1),
- virt_to_maddr(itt_addr), true);
- if ( ret )
- goto out_unlock;
-
- dev->itt_addr = itt_addr;
- dev->itt_order = order;
- dev->hw_its = hw_its;
dev->guest_doorbell = guest_doorbell;
dev->guest_devid = guest_devid;
dev->host_devid = host_devid;
@@ -920,13 +1011,7 @@ out_unlock:
out:
if ( dev )
- {
- xfree(dev->pend_irqs);
- xfree(dev->host_lpi_blocks);
- }
- if ( itt_addr )
- free_xenheap_pages(itt_addr, order);
- xfree(dev);
+ its_free_device(dev);
return ret;
}
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index c029d5d7a4..3c2649b695 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -58,6 +58,7 @@ static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
#define MAX_NR_HOST_LPIS (lpi_data.max_host_lpi_ids - LPI_OFFSET)
#define HOST_LPIS_PER_PAGE (PAGE_SIZE / sizeof(union host_lpi))
+uint32_t lpi_id_bits;
static union host_lpi *gic_get_host_lpi(uint32_t plpi)
{
@@ -202,14 +203,11 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
write_u64_atomic(&hlpip->data, hlpi.data);
}
-static int gicv3_lpi_allocate_pendtable(unsigned int cpu)
+struct page_info *lpi_allocate_pendtable(void)
{
void *pendtable;
unsigned int order;
- if ( per_cpu(lpi_redist, cpu).pending_table )
- return -EBUSY;
-
/*
* The pending table holds one bit per LPI and even covers bits for
* interrupt IDs below 8192, so we allocate the full range.
@@ -219,20 +217,34 @@ static int gicv3_lpi_allocate_pendtable(unsigned int cpu)
order = get_order_from_bytes(max(lpi_data.max_host_lpi_ids / 8, (unsigned long)SZ_64K));
pendtable = alloc_xenheap_pages(order, gicv3_its_get_memflags());
if ( !pendtable )
- return -ENOMEM;
+ return NULL;
memset(pendtable, 0, PAGE_SIZE << order);
/* Make sure the physical address can be encoded in the register. */
if ( virt_to_maddr(pendtable) & ~GENMASK(51, 16) )
{
free_xenheap_pages(pendtable, order);
- return -ERANGE;
+ return NULL;
}
clean_and_invalidate_dcache_va_range(pendtable,
lpi_data.max_host_lpi_ids / 8);
- per_cpu(lpi_redist, cpu).pending_table = pendtable;
+ return virt_to_page(pendtable);
+}
+
+static int gicv3_lpi_allocate_pendtable(unsigned int cpu)
+{
+ struct page_info *pendtable;
+
+ if ( per_cpu(lpi_redist, cpu).pending_table )
+ return -EBUSY;
+
+ pendtable = lpi_allocate_pendtable();
+ if ( !pendtable )
+ return -EINVAL;
+ per_cpu(lpi_redist, cpu).pending_table = page_to_virt(pendtable);
+
return 0;
}
@@ -274,6 +286,38 @@ static int gicv3_lpi_set_pendtable(void __iomem *rdist_base)
return 0;
}
+void *lpi_allocate_proptable(void)
+{
+ void *table;
+ int order;
+
+ /* The property table holds one byte per LPI. */
+ order = get_order_from_bytes(lpi_data.max_host_lpi_ids);
+ table = alloc_xenheap_pages(order, gicv3_its_get_memflags());
+ if ( !table )
+ return NULL;
+
+ /* Make sure the physical address can be encoded in the register. */
+ if ( (virt_to_maddr(table) & ~GENMASK(51, 12)) )
+ {
+ free_xenheap_pages(table, order);
+ return NULL;
+ }
+ memset(table, GIC_PRI_IRQ | LPI_PROP_RES1, MAX_NR_HOST_LPIS);
+ clean_and_invalidate_dcache_va_range(table, MAX_NR_HOST_LPIS);
+
+ return table;
+}
+
+void lpi_free_proptable(void *vproptable)
+{
+ int order;
+
+ /* The property table holds one byte per LPI. */
+ order = get_order_from_bytes(lpi_data.max_host_lpi_ids);
+ free_xenheap_pages(vproptable, order);
+}
+
/*
* Tell a redistributor about the (shared) property table, allocating one
* if not already done.
@@ -314,7 +358,8 @@ static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
}
/* Encode the number of bits needed, minus one */
- reg |= fls(lpi_data.max_host_lpi_ids - 1) - 1;
+ lpi_id_bits = fls(lpi_data.max_host_lpi_ids - 1);
+ reg |= lpi_id_bits - 1;
reg |= virt_to_maddr(lpi_data.lpi_property);
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 14852d18c2..d4af332b0e 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -2083,6 +2083,22 @@ static bool gic_dist_supports_lpis(void)
return (readl_relaxed(GICD + GICD_TYPER) & GICD_TYPE_LPIS);
}
+#ifdef CONFIG_GICV4
+static void __init gicv4_init(void)
+{
+ gicv3_info.hw_version = GIC_V4;
+
+
+ gicv4_its_vpeid_allocator_init();
+
+}
+#else
+static void __init gicv4_init(void)
+{
+ ASSERT_UNREACHABLE();
+}
+#endif
+
/* Set up the GIC */
static int __init gicv3_init(void)
{
@@ -2157,6 +2173,8 @@ static int __init gicv3_init(void)
gicv3_hyp_init();
+ if ( gic_is_gicv4() )
+ gicv4_init();
out:
spin_unlock(&gicv3.lock);
diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
index 358d0bffb9..fac3b44a94 100644
--- a/xen/arch/arm/gic-v4-its.c
+++ b/xen/arch/arm/gic-v4-its.c
@@ -27,6 +27,83 @@
#include <asm/vgic.h>
+/*
+ * VPE ID is at most 16 bits.
+ * Using a bitmap here limits us to 65536 concurrent VPEs.
+ */
+static unsigned long *vpeid_mask;
+
+static spinlock_t vpeid_alloc_lock = SPIN_LOCK_UNLOCKED;
+
+void __init gicv4_its_vpeid_allocator_init(void)
+{
+ /* Allocate space for vpeid_mask based on MAX_VPEID */
+ vpeid_mask = xzalloc_array(unsigned long, BITS_TO_LONGS(MAX_VPEID));
+
+ if ( !vpeid_mask )
+ panic("Could not allocate VPEID bitmap space\n");
+}
+
+static int __init its_alloc_vpeid(struct its_vpe *vpe)
+{
+ int id;
+
+ spin_lock(&vpeid_alloc_lock);
+
+ id = find_first_zero_bit(vpeid_mask, MAX_VPEID);
+
+ if ( id == MAX_VPEID )
+ {
+ id = -EBUSY;
+ printk(XENLOG_ERR "VPEID pool exhausted\n");
+ goto out;
+ }
+
+ set_bit(id, vpeid_mask);
+
+out:
+ spin_unlock(&vpeid_alloc_lock);
+
+ return id;
+}
+
+static void __init its_free_vpeid(uint32_t vpe_id)
+{
+ spin_lock(&vpeid_alloc_lock);
+
+ clear_bit(vpe_id, vpeid_mask);
+
+ spin_unlock(&vpeid_alloc_lock);
+}
+
+static bool __init its_alloc_vpe_entry(uint32_t vpe_id)
+{
+ struct host_its *hw_its;
+
+ /*
+ * Make sure the L2 tables are allocated on *all* v4 ITSs. We
+ * could try and only do it on ITSs corresponding to devices
+ * that have interrupts targeted at this VPE, but the
+ * complexity becomes crazy.
+ */
+ list_for_each_entry(hw_its, &host_its_list, entry)
+ {
+ struct its_baser *baser;
+
+ if ( !hw_its->is_v4 )
+ continue;
+
+ baser = its_get_baser(hw_its, GITS_BASER_TYPE_VCPU);
+ if ( !baser )
+ return false;
+
+ if ( !its_alloc_table_entry(baser, vpe_id) )
+ return false;
+ }
+
+ return true;
+}
+
static int its_send_cmd_vsync(struct host_its *its, uint16_t vpeid)
{
uint64_t cmd[4];
@@ -39,6 +116,188 @@ static int its_send_cmd_vsync(struct host_its *its, uint16_t vpeid)
return its_send_command(its, cmd);
}
+static int its_send_cmd_vmapp(struct host_its *its, struct its_vpe *vpe,
+ bool valid)
+{
+ uint64_t cmd[4];
+ uint16_t vpeid = vpe->vpe_id;
+ uint64_t vpt_addr;
+ int ret;
+
+ cmd[0] = GITS_CMD_VMAPP;
+ cmd[1] = (uint64_t)vpeid << 32;
+ cmd[2] = valid ? GITS_VALID_BIT : 0;
+
+ /* Unmap command */
+ if ( !valid )
+ goto out;
+
+ /* Target redistributor */
+ cmd[2] |= encode_rdbase(its, vpe->col_idx, 0x0);
+ vpt_addr = virt_to_maddr(vpe->vpendtable);
+ cmd[3] = (vpt_addr & GENMASK(51, 16)) |
+ ((HOST_LPIS_NRBITS - 1) & GENMASK(4, 0));
+
+ out:
+ ret = its_send_command(its, cmd);
+
+ return ret;
+}
+
+static int its_send_cmd_vinvall(struct host_its *its, struct its_vpe *vpe)
+{
+ uint64_t cmd[4];
+ uint16_t vpeid = vpe->vpe_id;
+
+ cmd[0] = GITS_CMD_VINVALL;
+ cmd[1] = (uint64_t)vpeid << 32;
+ cmd[2] = 0x00;
+ cmd[3] = 0x00;
+
+ return its_send_command(its, cmd);
+}
+
+static int its_map_vpe(struct host_its *its, struct its_vpe *vpe)
+{
+ int ret;
+
+ /*
+ * VMAPP command maps the vPE to the target RDbase, including an
+ * associated virtual LPI Pending table.
+ */
+ ret = its_send_cmd_vmapp(its, vpe, true);
+ if ( ret )
+ return ret;
+
+ ret = its_send_cmd_vinvall(its, vpe);
+ if ( ret )
+ return ret;
+
+ ret = its_send_cmd_vsync(its, vpe->vpe_id);
+ if ( ret )
+ return ret;
+
+ return 0;
+}
+static int __init its_vpe_init(struct its_vpe *vpe)
+{
+ int vpe_id, rc = -ENOMEM;
+ struct page_info *vpendtable;
+ struct host_its *hw_its;
+
+ /* Allocate vpe id */
+ vpe_id = its_alloc_vpeid(vpe);
+ if ( vpe_id < 0 )
+ return rc;
+
+ /* Allocate VPT */
+ vpendtable = lpi_allocate_pendtable();
+
+ if ( !vpendtable )
+ goto fail_vpt;
+
+ if ( !its_alloc_vpe_entry(vpe_id) )
+ goto fail_entry;
+
+ rwlock_init(&vpe->lock);
+ vpe->vpe_id = vpe_id;
+ vpe->vpendtable = page_to_virt(vpendtable);
+ /*
+ * We eagerly inform all the v4 ITS and map vPE to the first
+ * possible CPU
+ */
+ vpe->col_idx = cpumask_first(&cpu_online_map);
+ list_for_each_entry(hw_its, &host_its_list, entry)
+ {
+ if ( !hw_its->is_v4 )
+ continue;
+
+ if ( its_map_vpe(hw_its, vpe) )
+ goto fail_entry;
+ }
+
+ return 0;
+
+ fail_entry:
+ xfree(page_to_virt(vpendtable));
+ fail_vpt:
+ its_free_vpeid(vpe_id);
+
+ return rc;
+}
+
+static void __init its_vpe_teardown(struct its_vpe *vpe)
+{
+ unsigned int order;
+
+ order = get_order_from_bytes(max(lpi_data.max_host_lpi_ids / 8, (unsigned long)SZ_64K));
+ its_free_vpeid(vpe->vpe_id);
+ free_xenheap_pages(vpe->vpendtable, order);
+ xfree(vpe);
+}
+
+int vgic_v4_its_vm_init(struct domain *d)
+{
+ unsigned int nr_vcpus = d->max_vcpus;
+ int ret = -ENOMEM;
+
+ if ( !gicv3_its_host_has_its() )
+ return 0;
+
+ d->arch.vgic.its_vm = xzalloc(struct its_vm);
+ if ( !d->arch.vgic.its_vm )
+ return ret;
+
+ d->arch.vgic.its_vm->vpes = xzalloc_array(struct its_vpe *, nr_vcpus);
+ if ( !d->arch.vgic.its_vm->vpes )
+ goto fail_vpes;
+ d->arch.vgic.its_vm->nr_vpes = nr_vcpus;
+
+ d->arch.vgic.its_vm->vproptable = lpi_allocate_proptable();
+ if ( !d->arch.vgic.its_vm->vproptable )
+ goto fail_vprop;
+
+ return 0;
+
+fail_vprop:
+ xfree(d->arch.vgic.its_vm->vpes);
+ fail_vpes:
+ xfree(d->arch.vgic.its_vm);
+
+ return ret;
+}
+
+void vgic_v4_free_its_vm(struct domain *d)
+{
+ struct its_vm *its_vm = d->arch.vgic.its_vm;
+ if ( its_vm->vpes )
+ xfree(its_vm->vpes);
+ if ( its_vm->vproptable )
+ lpi_free_proptable(its_vm);
+}
+
+int vgic_v4_its_vpe_init(struct vcpu *vcpu)
+{
+ int ret;
+ struct its_vm *its_vm = vcpu->domain->arch.vgic.its_vm;
+ unsigned int vcpuid = vcpu->vcpu_id;
+
+ vcpu->arch.vgic.its_vpe = xzalloc(struct its_vpe);
+ if ( !vcpu->arch.vgic.its_vpe )
+ return -ENOMEM;
+
+ its_vm->vpes[vcpuid] = vcpu->arch.vgic.its_vpe;
+ vcpu->arch.vgic.its_vpe->its_vm = its_vm;
+
+ ret = its_vpe_init(vcpu->arch.vgic.its_vpe);
+ if ( ret )
+ {
+ its_vpe_teardown(vcpu->arch.vgic.its_vpe);
+ return ret;
+ }
+ return 0;
+}
+
static int its_send_cmd_vmapti(struct host_its *its, struct its_device *dev,
uint32_t eventid)
{
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index bd2696f354..411beb81c8 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -77,6 +77,7 @@
#define GITS_BASER_ENTRY_SIZE_SHIFT 48
#define GITS_BASER_ENTRY_SIZE(reg) \
((((reg) >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
+#define GITS_LVL1_ENTRY_SIZE 8UL
#define GITS_BASER_SHAREABILITY_SHIFT 10
#define GITS_BASER_PAGE_SIZE_SHIFT 8
#define GITS_BASER_SIZE_MASK 0xff
@@ -117,9 +118,19 @@
/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
#define LPI_BLOCK 32U
+extern unsigned int nvpeid;
+/* The maximum number of VPEID bits supported by VLPI commands */
+#define ITS_MAX_VPEID_BITS nvpeid
+#define MAX_VPEID (1UL << ITS_MAX_VPEID_BITS)
+
#ifdef CONFIG_GICV4
#include <asm/gic_v4_its.h>
#endif
+
+extern uint32_t lpi_id_bits;
+#define HOST_LPIS_NRBITS lpi_id_bits
+#define MAX_HOST_LPIS BIT(lpi_id_bits, UL)
+
/*
* Describes a device which is using the ITS and is used by a guest.
* Since device IDs are per ITS (in contrast to vLPIs, which are per
@@ -169,6 +180,7 @@ struct host_its {
void *cmd_buf;
unsigned int flags;
struct its_baser tables[GITS_BASER_NR_REGS];
+ bool is_v4;
};
/* Map a collection for this host CPU to each host ITS. */
@@ -273,8 +285,13 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
uint32_t virt_lpi);
struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type);
+bool its_alloc_table_entry(struct its_baser *baser, uint32_t id);
+struct page_info *lpi_allocate_pendtable(void);
+void *lpi_allocate_proptable(void);
+void lpi_free_proptable(void *vproptable);
void lpi_write_config(uint8_t *prop_table, uint32_t lpi, uint8_t clr,
uint8_t set);
+uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu, uint64_t reg);
int its_send_command(struct host_its *hw_its, const void *its_cmd);
struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
index 722247ec60..fb0ef37bbe 100644
--- a/xen/arch/arm/include/asm/gic_v4_its.h
+++ b/xen/arch/arm/include/asm/gic_v4_its.h
@@ -49,6 +49,7 @@ struct event_vlpi_map {
unsigned int nr_vlpis;
};
+void gicv4_its_vpeid_allocator_init(void);
#endif
/*
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index f12d736808..580310fec4 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -414,6 +414,9 @@ bool gic_is_gicv4(void);
#define gic_is_gicv4() (false)
#endif
+int vgic_v4_its_vm_init(struct domain *d);
+void vgic_v4_free_its_vm(struct domain *d);
+int vgic_v4_its_vpe_init(struct vcpu *vcpu);
#endif /* !CONFIG_NEW_VGIC */
/*** Common VGIC functions used by Xen arch code ****/
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 0da8c1a425..6baf870ad5 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -22,6 +22,7 @@
#include <asm/mmio.h>
#include <asm/gic.h>
+#include <asm/gic_v3_its.h>
#include <asm/vgic.h>
@@ -329,6 +330,15 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
for ( i = 0; i < NR_GIC_SGI; i++ )
set_bit(i, d->arch.vgic.allocated_irqs);
+ if ( gic_is_gicv4() )
+ {
+ ret = vgic_v4_its_vm_init(d);
+ if ( ret )
+ {
+ printk(XENLOG_ERR "GICv4 its vm allocation failed\n");
+ return ret;
+ }
+ }
return 0;
}
@@ -366,11 +376,14 @@ void domain_vgic_free(struct domain *d)
#endif
xfree(d->arch.vgic.pending_irqs);
xfree(d->arch.vgic.allocated_irqs);
+
+ if ( gic_is_gicv4() )
+ vgic_v4_free_its_vm(d);
}
int vcpu_vgic_init(struct vcpu *v)
{
- int i;
+ int i, ret;
v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
if ( v->arch.vgic.private_irqs == NULL )
@@ -389,6 +402,16 @@ int vcpu_vgic_init(struct vcpu *v)
INIT_LIST_HEAD(&v->arch.vgic.lr_pending);
spin_lock_init(&v->arch.vgic.lock);
+ if ( gic_is_gicv4() && gicv3_its_host_has_its())
+ {
+ ret = vgic_v4_its_vpe_init(v);
+ if ( ret )
+ {
+ printk(XENLOG_ERR "GICv4 its vpe allocation failed\n");
+ return ret;
+ }
+ }
+
return 0;
}
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 13/19] arm/gic: VPE scheduling
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (11 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 12/19] arm/gic: Add VPENDBASER/VPROPBASER accessors Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-18 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 16/19] arm/gic: Fix LR group handling for GICv4 Mykyta Poturai
` (6 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
GICv4 needs to know which VCPU is currently scheduled to be able to
deliver VLPIs. Implement switching of VPEs on VCPU context switch by
extending the existing save/restore mechanism used for GICv2 and GICv3.
Scheduling a VPE is done by setting up the VPENDBASER and VPROPBASER
registers to the appropriate tables for the currently running VCPU. When
scheduling out, preserve the IDAI and PendingLast bits from VPENDBASER.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v2.c | 2 +-
xen/arch/arm/gic-v3.c | 9 +-
xen/arch/arm/gic-v4-its.c | 138 +++++++++++++++++++++++++
xen/arch/arm/gic-vgic.c | 6 ++
xen/arch/arm/include/asm/gic.h | 2 +-
xen/arch/arm/include/asm/gic_v3_defs.h | 6 ++
xen/arch/arm/include/asm/gic_v4_its.h | 9 ++
xen/arch/arm/include/asm/vgic.h | 2 +
8 files changed, 170 insertions(+), 4 deletions(-)
diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 0cd41eac12..c16fa5d67e 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -181,7 +181,7 @@ static void gicv2_save_state(struct vcpu *v)
writel_gich(0, GICH_HCR);
}
-static void gicv2_restore_state(const struct vcpu *v)
+static void gicv2_restore_state(struct vcpu *v)
{
int i;
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index d4af332b0e..07736179db 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -133,7 +133,7 @@ bool gic_is_gicv4(void)
#endif
/* per-cpu re-distributor base */
-static DEFINE_PER_CPU(void __iomem*, rbase);
+DEFINE_PER_CPU(void __iomem*, rbase);
#define GICD (gicv3.map_dbase)
#define GICD_RDIST_BASE (this_cpu(rbase))
@@ -475,13 +475,15 @@ static void gicv3_save_state(struct vcpu *v)
* are now visible to the system register interface
*/
dsb(sy);
+ if ( gic_is_gicv4() )
+ vgic_v4_put(v, false);
gicv3_save_lrs(v);
save_aprn_regs(&v->arch.gic);
v->arch.gic.v3.vmcr = READ_SYSREG(ICH_VMCR_EL2);
v->arch.gic.v3.sre_el1 = READ_SYSREG(ICC_SRE_EL1);
}
-static void gicv3_restore_state(const struct vcpu *v)
+static void gicv3_restore_state(struct vcpu *v)
{
register_t val;
@@ -510,6 +512,9 @@ static void gicv3_restore_state(const struct vcpu *v)
restore_aprn_regs(&v->arch.gic);
gicv3_restore_lrs(v);
+ if ( gic_is_gicv4() )
+ vgic_v4_load(v);
+
/*
* Make sure all stores are visible the GIC
*/
diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
index fac3b44a94..6a550a65b2 100644
--- a/xen/arch/arm/gic-v4-its.c
+++ b/xen/arch/arm/gic-v4-its.c
@@ -18,6 +18,7 @@
* GNU General Public License for more details.
*/
+#include <xen/delay.h>
#include <xen/errno.h>
#include <xen/sched.h>
#include <xen/spinlock.h>
@@ -44,6 +45,21 @@ void __init gicv4_its_vpeid_allocator_init(void)
panic("Could not allocate VPEID bitmap space\n");
}
+static void __iomem *gic_data_rdist_vlpi_base(unsigned int cpu)
+{
+ /*
+ * Each Redistributor defines two 64KB frames in the physical address map.
+ * In GICv4, there are two additional 64KB frames.
+ * The frames for each Redistributor must be contiguous and must be
+ * ordered as follows:
+ * 1. RD_base
+ * 2. SGI_base
+ * 3. VLPI_base
+ * 4. Reserved
+ */
+ return GICD_RDIST_BASE_CPU(cpu) + SZ_128K;
+}
+
static int __init its_alloc_vpeid(struct its_vpe *vpe)
{
int id;
@@ -571,3 +587,125 @@ int its_send_cmd_vinv(struct host_its *its, struct its_device *dev,
return gicv3_its_wait_commands(its);
}
+
+static uint64_t read_vpend_dirty_clean(void __iomem *vlpi_base,
+ unsigned int count)
+{
+ uint64_t val;
+ bool clean;
+
+ do {
+ val = gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER);
+ /* Poll GICR_VPENDBASER.Dirty until it reads 0. */
+ clean = !(val & GICR_VPENDBASER_Dirty);
+ if ( !clean )
+ {
+ count--;
+ cpu_relax();
+ udelay(1);
+ }
+ } while ( !clean && count );
+
+ if ( !clean )
+ {
+ printk(XENLOG_WARNING "ITS virtual pending table not totally parsed\n");
+ val |= GICR_VPENDBASER_PendingLast;
+ }
+
+ return val;
+}
+
+/*
+ * When a vPE is made resident, the GIC starts parsing the virtual pending
+ * table to deliver pending interrupts. This takes place asynchronously,
+ * and can at times take a long while.
+ */
+static void its_wait_vpt_parse_complete(void __iomem *vlpi_base)
+{
+ if ( !gic_support_vptValidDirty() )
+ return;
+
+ read_vpend_dirty_clean(vlpi_base, 500);
+}
+
+static uint64_t its_clear_vpend_valid(void __iomem *vlpi_base, uint64_t clr,
+ uint64_t set)
+{
+ unsigned int count = 1000000; /* 1s! */
+ uint64_t val;
+
+ /*
+ * Clearing the Valid bit informs the Redistributor that a context
+ * switch is taking place.
+ */
+ val = gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER);
+ val &= ~GICR_VPENDBASER_Valid;
+ val &= ~clr;
+ val |= set;
+ gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
+
+ return read_vpend_dirty_clean(vlpi_base, count);
+}
+
+static void its_make_vpe_resident(struct its_vpe *vpe, unsigned int cpu)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base(cpu);
+ uint64_t val;
+
+ /* Switch in this VM's virtual property table. */
+ val = virt_to_maddr(vpe->its_vm->vproptable) & GENMASK(51, 12);
+ val |= gicv3_its_get_cacheability() << GICR_VPROPBASER_INNER_CACHEABILITY_SHIFT;
+ val |= gicv3_its_get_shareability() << GICR_VPROPBASER_SHAREABILITY_SHIFT;
+ val |= GIC_BASER_CACHE_SameAsInner << GICR_VPROPBASER_OUTER_CACHEABILITY_SHIFT;
+ val |= (HOST_LPIS_NRBITS - 1) & GICR_VPROPBASER_IDBITS_MASK;
+ gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
+
+ /* Switch in this VCPU's VPT. */
+ val = virt_to_maddr(vpe->vpendtable) & GENMASK(51, 16);
+ val |= gicv3_its_get_cacheability() << GICR_VPENDBASER_INNER_CACHEABILITY_SHIFT;
+ val |= gicv3_its_get_shareability() << GICR_VPENDBASER_SHAREABILITY_SHIFT;
+ val |= GIC_BASER_CACHE_SameAsInner << GICR_VPENDBASER_OUTER_CACHEABILITY_SHIFT;
+ /*
+ * When the GICR_VPENDBASER.Valid bit is written from 0 to 1,
+ * this bit is RES1.
+ */
+ val |= GICR_VPENDBASER_PendingLast;
+ val |= vpe->idai ? GICR_VPENDBASER_IDAI : 0;
+ val |= GICR_VPENDBASER_Valid;
+ gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
+
+ its_wait_vpt_parse_complete(vlpi_base);
+}
+
+static void its_make_vpe_non_resident(struct its_vpe *vpe, unsigned int cpu)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base(cpu);
+ uint64_t val;
+
+ val = its_clear_vpend_valid(vlpi_base, 0, 0);
+ vpe->idai = val & GICR_VPENDBASER_IDAI;
+ vpe->pending_last = val & GICR_VPENDBASER_PendingLast;
+}
+
+void vgic_v4_load(struct vcpu *vcpu)
+{
+ struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
+
+
+ if ( vpe->resident )
+ return;
+
+ its_make_vpe_resident(vpe, vcpu->processor);
+ vpe->resident = true;
+}
+
+void vgic_v4_put(struct vcpu *vcpu, bool need_db)
+{
+ struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
+
+ if ( !vpe->resident )
+ return;
+
+ its_make_vpe_non_resident(vpe, vcpu->processor);
+ vpe->resident = false;
+}
diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
index ea48c5375a..44db142dbd 100644
--- a/xen/arch/arm/gic-vgic.c
+++ b/xen/arch/arm/gic-vgic.c
@@ -377,6 +377,12 @@ int vgic_vcpu_pending_irq(struct vcpu *v)
}
}
+#ifdef CONFIG_GICV4
+ if ( gic_is_gicv4() )
+ if ( v->arch.vgic.its_vpe->pending_last )
+ rc = 1;
+#endif
+
out:
spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
return rc;
diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
index afb1cc3751..04a20bdca5 100644
--- a/xen/arch/arm/include/asm/gic.h
+++ b/xen/arch/arm/include/asm/gic.h
@@ -362,7 +362,7 @@ struct gic_hw_operations {
/* Save GIC registers */
void (*save_state)(struct vcpu *v);
/* Restore GIC registers */
- void (*restore_state)(const struct vcpu *v);
+ void (*restore_state)(struct vcpu *v);
/* Dump GIC LR register information */
void (*dump_state)(const struct vcpu *v);
diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
index 3a7d18ef59..0db75309cf 100644
--- a/xen/arch/arm/include/asm/gic_v3_defs.h
+++ b/xen/arch/arm/include/asm/gic_v3_defs.h
@@ -257,6 +257,12 @@ struct rdist_region {
bool single_rdist;
};
+/* per-cpu re-distributor base */
+DECLARE_PER_CPU(void __iomem*, rbase);
+
+#define GICD_RDIST_BASE (this_cpu(rbase))
+#define GICD_RDIST_BASE_CPU(cpu) (per_cpu(rbase, cpu))
+
#endif /* __ASM_ARM_GIC_V3_DEFS_H__ */
/*
diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
index ba81b25bde..37b6b92f0c 100644
--- a/xen/arch/arm/include/asm/gic_v4_its.h
+++ b/xen/arch/arm/include/asm/gic_v4_its.h
@@ -56,6 +56,15 @@ void gicv4_its_vpeid_allocator_init(void);
#define GICR_VPROPBASER 0x0070
#define GICR_VPENDBASER 0x0078
+#define GICR_VPROPBASER_OUTER_CACHEABILITY_SHIFT 56
+#define GICR_VPROPBASER_SHAREABILITY_SHIFT 10
+#define GICR_VPROPBASER_SHAREABILITY_MASK \
+ (3UL << GICR_VPROPBASER_SHAREABILITY_SHIFT)
+#define GICR_VPROPBASER_INNER_CACHEABILITY_SHIFT 7
+#define GICR_VPROPBASER_INNER_CACHEABILITY_MASK \
+ (7UL << GICR_VPROPBASER_INNER_CACHEABILITY_SHIFT)
+#define GICR_VPROPBASER_IDBITS_MASK 0x1f
+
#define GICR_VPENDBASER_Dirty (1UL << 60)
#define GICR_VPENDBASER_PendingLast (1UL << 61)
#define GICR_VPENDBASER_IDAI (1UL << 62)
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 580310fec4..9ef667decb 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -417,6 +417,8 @@ bool gic_is_gicv4(void);
int vgic_v4_its_vm_init(struct domain *d);
void vgic_v4_free_its_vm(struct domain *d);
int vgic_v4_its_vpe_init(struct vcpu *vcpu);
+void vgic_v4_load(struct vcpu *vcpu);
+void vgic_v4_put(struct vcpu *vcpu, bool need_db);
#endif /* !CONFIG_NEW_VGIC */
/*** Common VGIC functions used by Xen arch code ****/
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 12/19] arm/gic: Add VPENDBASER/VPROPBASER accessors
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (10 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 11/19] arm/its: Add ITS VM and VPE allocation/teardown Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-18 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 13/19] arm/gic: VPE scheduling Mykyta Poturai
` (7 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Implement accessors for GICv4 registers VPENDBASER and VPROPBASER.
VPENDBASER access needs special handling to clear the Valid bit before
writing a new value.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/include/asm/gic_v4_its.h | 38 +++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
index fb0ef37bbe..ba81b25bde 100644
--- a/xen/arch/arm/include/asm/gic_v4_its.h
+++ b/xen/arch/arm/include/asm/gic_v4_its.h
@@ -17,6 +17,8 @@
* along with this program; If not, see <http://www.gnu.org/licenses/>.
*/
+#include <asm/arm64/io.h>
+
#ifndef __ASM_ARM_GICV4_ITS_H__
#define __ASM_ARM_GICV4_ITS_H__
@@ -50,6 +52,42 @@ struct event_vlpi_map {
};
void gicv4_its_vpeid_allocator_init(void);
+
+#define GICR_VPROPBASER 0x0070
+#define GICR_VPENDBASER 0x0078
+
+#define GICR_VPENDBASER_Dirty (1UL << 60)
+#define GICR_VPENDBASER_PendingLast (1UL << 61)
+#define GICR_VPENDBASER_IDAI (1UL << 62)
+#define GICR_VPENDBASER_Valid (1UL << 63)
+
+#define GICR_VPENDBASER_OUTER_CACHEABILITY_SHIFT 56
+#define GICR_VPENDBASER_SHAREABILITY_SHIFT 10
+#define GICR_VPENDBASER_INNER_CACHEABILITY_SHIFT 7
+
+#define gits_read_vpropbaser(c) readq_relaxed(c)
+#define gits_write_vpropbaser(v, c) {writeq_relaxed(v, c);}
+
+/*
+ * GICR_VPENDBASER - the Valid bit must be cleared before changing
+ * anything else.
+ */
+static inline void gits_write_vpendbaser(uint64_t val, void __iomem *addr)
+{
+ uint64_t tmp;
+
+ tmp = readq_relaxed(addr);
+ while ( tmp & GICR_VPENDBASER_Valid )
+ {
+ tmp &= ~GICR_VPENDBASER_Valid;
+ writeq_relaxed(tmp, addr);
+ tmp = readq_relaxed(addr);
+ }
+
+ writeq_relaxed(val, addr);
+}
+#define gits_read_vpendbaser(c) readq_relaxed(c)
+
#endif
/*
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 14/19] arm/its: VPE affinity changes
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (14 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 15/19] arm: Add gicv4 to domain creation Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-18 21:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 17/19] arm/gicv4: Handle doorbells Mykyta Poturai
` (3 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
When a VCPU is migrated to another PCPU, its VPE affinity must be
updated. Hook into VPE scheduling to ensure that the VPE to be scheduled
is located on the correct PCPU, if not, move it with VMOVP command.
VMOVP needs to be issued on all ITSes in the system, and in the same
order, unlsess single VMOVP capable ITS is used.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 55 ++++++++++++++
xen/arch/arm/gic-v4-its.c | 105 ++++++++++++++++++++++++++
xen/arch/arm/include/asm/gic_v3_its.h | 12 +++
3 files changed, 172 insertions(+)
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index fb1d2709be..be840fbc8f 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -31,6 +31,8 @@
LIST_HEAD(host_its_list);
+unsigned long its_list_map;
+
unsigned int nvpeid = 16;
/*
@@ -612,10 +614,47 @@ static int gicv3_disable_its(struct host_its *hw_its)
return -ETIMEDOUT;
}
+static int __init its_compute_its_list_map(struct host_its *hw_its)
+{
+ int its_number;
+ uint32_t ctlr;
+
+ its_number = find_first_zero_bit(&its_list_map, GICv4_ITS_LIST_MAX);
+ if ( its_number >= GICv4_ITS_LIST_MAX )
+ {
+ printk(XENLOG_ERR
+ "ITS@%lx: No ITSList entry available!\n", hw_its->addr);
+ return -EINVAL;
+ }
+
+ ctlr = readl_relaxed(hw_its->its_base + GITS_CTLR);
+ ctlr &= ~GITS_CTLR_ITS_NUMBER;
+ ctlr |= its_number << GITS_CTLR_ITS_NUMBER_SHIFT;
+ writel_relaxed(ctlr, hw_its->its_base + GITS_CTLR);
+ ctlr = readl_relaxed(hw_its->its_base + GITS_CTLR);
+ if ( (ctlr & GITS_CTLR_ITS_NUMBER) !=
+ (its_number << GITS_CTLR_ITS_NUMBER_SHIFT) )
+ {
+ its_number = ctlr & GITS_CTLR_ITS_NUMBER;
+ its_number >>= GITS_CTLR_ITS_NUMBER_SHIFT;
+ }
+
+ if ( test_and_set_bit(its_number, &its_list_map) )
+ {
+ printk(XENLOG_ERR
+ "ITS@%lx: Duplicate ITSList entry %d\n",
+ hw_its->addr, its_number);
+ return -EINVAL;
+ }
+
+ return its_number;
+}
+
static int gicv3_its_init_single_its(struct host_its *hw_its)
{
uint64_t reg;
int i, ret;
+ int its_number;
hw_its->its_base = ioremap_nocache(hw_its->addr, hw_its->size);
if ( !hw_its->its_base )
@@ -633,6 +672,22 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
if ( reg & GITS_TYPER_PTA )
hw_its->flags |= HOST_ITS_USES_PTA;
+ hw_its->is_v4 = reg & GITS_TYPER_VLPIS;
+ if ( hw_its->is_v4 )
+ {
+ if ( !(reg & GITS_TYPER_VMOVP) )
+ {
+ its_number = its_compute_its_list_map(hw_its);
+ if ( its_number < 0 )
+ return its_number;
+ dprintk(XENLOG_INFO,
+ "ITS@%lx: Using ITS number %d\n",
+ hw_its->addr, its_number);
+ }
+ else
+ dprintk(XENLOG_INFO,
+ "ITS@%lx: Single VMOVP capable\n", hw_its->addr);
+ }
spin_lock_init(&hw_its->cmd_lock);
for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
index 6a550a65b2..175fda7acb 100644
--- a/xen/arch/arm/gic-v4-its.c
+++ b/xen/arch/arm/gic-v4-its.c
@@ -36,6 +36,9 @@ static unsigned long *vpeid_mask;
static spinlock_t vpeid_alloc_lock = SPIN_LOCK_UNLOCKED;
+static uint16_t vmovp_seq_num;
+static spinlock_t vmovp_lock = SPIN_LOCK_UNLOCKED;
+
void __init gicv4_its_vpeid_allocator_init(void)
{
/* Allocate space for vpeid_mask based on MAX_VPEID */
@@ -242,6 +245,57 @@ static int __init its_vpe_init(struct its_vpe *vpe)
return rc;
}
+static int its_send_cmd_vmovp(struct its_vpe *vpe)
+{
+ uint16_t vpeid = vpe->vpe_id;
+ int ret;
+ struct host_its *hw_its;
+
+ if ( !its_list_map )
+ {
+ uint64_t cmd[4];
+
+ hw_its = list_first_entry(&host_its_list, struct host_its, entry);
+ cmd[0] = GITS_CMD_VMOVP;
+ cmd[1] = (uint64_t)vpeid << 32;
+ cmd[2] = encode_rdbase(hw_its, vpe->col_idx, 0x0);
+ cmd[3] = 0x00;
+
+ return its_send_command(hw_its, cmd);
+ }
+
+ /*
+ * If using the its_list "feature", we need to make sure that all ITSs
+ * receive all VMOVP commands in the same order. The only way
+ * to guarantee this is to make vmovp a serialization point.
+ */
+ spin_lock(&vmovp_lock);
+
+ vmovp_seq_num++;
+
+ /* Emit VMOVPs */
+ list_for_each_entry(hw_its, &host_its_list, entry)
+ {
+ uint64_t cmd[4];
+
+ cmd[0] = GITS_CMD_VMOVP | ((uint64_t)vmovp_seq_num << 32);
+ cmd[1] = its_list_map | ((uint64_t)vpeid << 32);
+ cmd[2] = encode_rdbase(hw_its, vpe->col_idx, 0x0);
+ cmd[3] = 0x00;
+
+ ret = its_send_command(hw_its, cmd);
+ if ( ret )
+ {
+ spin_unlock(&vmovp_lock);
+ return ret;
+ }
+ }
+
+ spin_unlock(&vmovp_lock);
+
+ return 0;
+}
+
static void __init its_vpe_teardown(struct its_vpe *vpe)
{
unsigned int order;
@@ -687,6 +741,52 @@ static void its_make_vpe_non_resident(struct its_vpe *vpe, unsigned int cpu)
vpe->pending_last = val & GICR_VPENDBASER_PendingLast;
}
+static int vpe_to_cpuid_lock(struct its_vpe *vpe, unsigned long *flags)
+{
+ spin_lock_irqsave(&vpe->vpe_lock, *flags);
+ return vpe->col_idx;
+}
+
+static void vpe_to_cpuid_unlock(struct its_vpe *vpe, unsigned long *flags)
+{
+ spin_unlock_irqrestore(&vpe->vpe_lock, *flags);
+}
+
+static int gicv4_vpe_set_affinity(struct vcpu *vcpu)
+{
+ struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
+ unsigned int from, to = vcpu->processor;
+ unsigned long flags;
+ int ret = 0;
+
+ /*
+ * Changing affinity is mega expensive, so let's be as lazy as
+ * we can and only do it if we really have to. Also, if mapped
+ * into the proxy device, we need to move the doorbell interrupt
+ * to its new location.
+ *
+ * Another thing is that changing the affinity of a vPE affects
+ * *other interrupts* such as all the vLPIs that are routed to
+ * this vPE. This means that we must ensure nobody samples
+ * vpe->col_idx during the update, hence the lock below which
+ * must also be taken on any vLPI handling path that evaluates
+ * vpe->col_idx, such as reg-based vLPI invalidation.
+ */
+ from = vpe_to_cpuid_lock(vpe, &flags);
+ if ( from == to )
+ goto out;
+
+ vpe->col_idx = to;
+
+ ret = its_send_cmd_vmovp(vpe);
+ if ( ret )
+ goto out;
+
+ out:
+ vpe_to_cpuid_unlock(vpe, &flags);
+ return ret;
+}
+
void vgic_v4_load(struct vcpu *vcpu)
{
struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
@@ -695,6 +795,11 @@ void vgic_v4_load(struct vcpu *vcpu)
if ( vpe->resident )
return;
+ /*
+ * Before making the VPE resident, make sure the redistributor
+ * corresponding to our current CPU expects us here
+ */
+ WARN_ON(gicv4_vpe_set_affinity(vcpu));
its_make_vpe_resident(vpe, vcpu->processor);
vpe->resident = true;
}
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index 411beb81c8..f03a8fad47 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -43,6 +43,9 @@
#define GITS_CTLR_QUIESCENT BIT(31, UL)
#define GITS_CTLR_ENABLE BIT(0, UL)
+#define GITS_CTLR_ITS_NUMBER_SHIFT 4
+#define GITS_CTLR_ITS_NUMBER (0xfUL << GITS_CTLR_ITS_NUMBER_SHIFT)
+
#define GITS_TYPER_PTA BIT(19, UL)
#define GITS_TYPER_DEVIDS_SHIFT 13
#define GITS_TYPER_DEVIDS_MASK (0x1fUL << GITS_TYPER_DEVIDS_SHIFT)
@@ -60,6 +63,8 @@
GITS_TYPER_ITT_SIZE_SHIFT) + 1)
#define GITS_TYPER_PHYSICAL (1U << 0)
+#define GITS_TYPER_VLPIS (1UL << 1)
+#define GITS_TYPER_VMOVP (1UL << 37)
#define GITS_BASER_INDIRECT BIT(62, UL)
#define GITS_BASER_INNER_CACHEABILITY_SHIFT 59
#define GITS_BASER_TYPE_SHIFT 56
@@ -118,6 +123,12 @@
/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
#define LPI_BLOCK 32U
+/*
+ * Maximum number of ITSs when GITS_TYPER.VMOVP == 0, using the
+ * ITSList mechanism to perform inter-ITS synchronization.
+ */
+#define GICv4_ITS_LIST_MAX 16
+
extern unsigned int nvpeid;
/* The maximum number of VPEID bits supported by VLPI commands */
#define ITS_MAX_VPEID_BITS nvpeid
@@ -214,6 +225,7 @@ struct __lpi_data {
extern struct __lpi_data lpi_data;
extern struct list_head host_its_list;
+extern unsigned long its_list_map;
int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
uint32_t eventid);
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 16/19] arm/gic: Fix LR group handling for GICv4
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (12 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 13/19] arm/gic: VPE scheduling Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-18 21:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 15/19] arm: Add gicv4 to domain creation Mykyta Poturai
` (5 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Extend the check to mark interrupts as Group1 for all GIC versions >= 3
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 07736179db..1cb3169b72 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1366,10 +1366,10 @@ static void gicv3_update_lr(int lr, unsigned int virq, uint8_t priority,
val = (((uint64_t)state & 0x3) << ICH_LR_STATE_SHIFT);
/*
- * When the guest is GICv3, all guest IRQs are Group 1, as Group0
- * would result in a FIQ in the guest, which it wouldn't expect
+ * When the guest is GICv3/GICv4/GICv4.1, all guest IRQs are Group 1, as
+ * Group0 would result in a FIQ in the guest, which it wouldn't expect
*/
- if ( current->domain->arch.vgic.version == GIC_V3 )
+ if ( current->domain->arch.vgic.version >= GIC_V3 )
val |= ICH_LR_GRP1;
val |= (uint64_t)priority << ICH_LR_PRIORITY_SHIFT;
@@ -1455,10 +1455,10 @@ static void gicv3_write_lr(int lr, const struct gic_lr *lr_reg)
}
/*
- * When the guest is using vGICv3, all the IRQs are Group 1. Group 0
- * would result in a FIQ, which will not be expected by the guest OS.
+ * When the guest is using vGICv3/vGICv4/vGICv4.1, all the IRQs are Group 1.
+ * Group 0 would result in a FIQ, which will not be expected by the guest OS.
*/
- if ( vgic_version == GIC_V3 )
+ if ( vgic_version >= GIC_V3 )
lrv |= ICH_LR_GRP1;
gicv3_ich_write_lr(lr, lrv);
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 15/19] arm: Add gicv4 to domain creation
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (13 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 16/19] arm/gic: Fix LR group handling for GICv4 Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-18 21:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 14/19] arm/its: VPE affinity changes Mykyta Poturai
` (4 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Allow creating domains with GIC_V4 version.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/dom0less-build.c | 1 +
xen/arch/arm/domain.c | 16 ++++++++++++++++
xen/arch/arm/vgic.c | 2 ++
xen/include/public/arch-arm.h | 2 ++
4 files changed, 21 insertions(+)
diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
index 4181c10538..3c4b074b05 100644
--- a/xen/arch/arm/dom0less-build.c
+++ b/xen/arch/arm/dom0less-build.c
@@ -157,6 +157,7 @@ int __init make_intc_domU_node(struct kernel_info *kinfo)
{
#ifdef CONFIG_GICV3
case GIC_V3:
+ case GIC_V4:
return make_gicv3_domU_node(kinfo);
#endif
#ifdef CONFIG_VGICV2
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 507df807ed..864e5511c9 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -604,6 +604,14 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
config->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_V3;
break;
+ case GIC_V4:
+ config->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_V4;
+ break;
+
+ case GIC_V4_1:
+ config->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_V4_1;
+ break;
+
default:
ASSERT_UNREACHABLE();
return -EINVAL;
@@ -679,6 +687,14 @@ int arch_domain_create(struct domain *d,
d->arch.vgic.version = GIC_V3;
break;
+ case XEN_DOMCTL_CONFIG_GIC_V4:
+ d->arch.vgic.version = GIC_V4;
+ break;
+
+ case XEN_DOMCTL_CONFIG_GIC_V4_1:
+ d->arch.vgic.version = GIC_V4_1;
+ break;
+
default:
BUG();
}
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 6baf870ad5..e5752faeb7 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -153,6 +153,7 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
{
#ifdef CONFIG_GICV3
case GIC_V3:
+ case GIC_V4:
if ( vgic_v3_init(d, mmio_count) )
return -ENODEV;
break;
@@ -929,6 +930,7 @@ unsigned int vgic_max_vcpus(unsigned int domctl_vgic_version)
#ifdef CONFIG_GICV3
case XEN_DOMCTL_CONFIG_GIC_V3:
+ case XEN_DOMCTL_CONFIG_GIC_V4:
return 4096;
#endif
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 96365e5603..12690cf3ac 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -322,6 +322,8 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
#define XEN_DOMCTL_CONFIG_GIC_NATIVE 0
#define XEN_DOMCTL_CONFIG_GIC_V2 1
#define XEN_DOMCTL_CONFIG_GIC_V3 2
+#define XEN_DOMCTL_CONFIG_GIC_V4 3
+#define XEN_DOMCTL_CONFIG_GIC_V4_1 4
#define XEN_DOMCTL_CONFIG_TEE_NONE 0
#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 17/19] arm/gicv4: Handle doorbells
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (15 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 14/19] arm/its: VPE affinity changes Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-19 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 19/19] arm/gicv4: Add GICv4 to the build system Mykyta Poturai
` (2 subsequent siblings)
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
When GIC wants to inject a virtual interrupt to a VCPU that is not
currently scheduled, it sends a kick for a hypervisor to schedule it. To
receive such kicks, we need to set up a doorbell interrupt for each VPE.
Add changes necessary to allocate, mask/unmask and handle doorbell
interrupts for each VPE. When a doorbell interrupt is received, set the
pending_last flag for the corresponding VPE and kick it, so that the
hypervisor schedules the VCPU to handle pending VLPIs.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 13 ++-
xen/arch/arm/gic-v3-lpi.c | 69 ++++++++++----
xen/arch/arm/gic-v4-its.c | 127 ++++++++++++++++++++++++++
xen/arch/arm/include/asm/gic_v3_its.h | 6 +-
xen/arch/arm/vgic-v3-its.c | 14 ++-
5 files changed, 203 insertions(+), 26 deletions(-)
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index be840fbc8f..fa5c1eb6d1 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -1016,8 +1016,11 @@ int gicv3_its_map_guest_device(struct domain *d,
dev->guest_doorbell = guest_doorbell;
dev->guest_devid = guest_devid;
- dev->host_devid = host_devid;
- dev->eventids = nr_events;
+
+ #ifdef CONFIG_GICV4
+ spin_lock_init(&dev->event_map.vlpi_lock);
+ dev->event_map.nr_lpis = nr_events;
+ #endif
rb_link_node(&dev->rbnode, parent, new);
rb_insert_color(&dev->rbnode, &d->arch.vgic.its_devices);
@@ -1142,7 +1145,8 @@ int gicv3_remove_guest_event(struct domain *d, paddr_t vdoorbell_address,
if ( host_lpi == INVALID_LPI )
return -EINVAL;
- gicv3_lpi_update_host_entry(host_lpi, d->domain_id, INVALID_LPI);
+ gicv3_lpi_update_host_entry(host_lpi, d->domain_id, INVALID_LPI,
+ false, INVALID_VCPU_ID);
return 0;
}
@@ -1169,7 +1173,8 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
if ( !pirq )
return NULL;
- gicv3_lpi_update_host_entry(host_lpi, d->domain_id, virt_lpi);
+ gicv3_lpi_update_host_entry(host_lpi, d->domain_id, virt_lpi,
+ false, INVALID_VCPU_ID);
return pirq;
}
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 3c2649b695..37f1aa1064 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -39,7 +39,7 @@ union host_lpi {
struct {
uint32_t virt_lpi;
uint16_t dom_id;
- uint16_t pad;
+ uint16_t db_vcpu_id;
};
};
@@ -161,24 +161,48 @@ void gicv3_do_LPI(unsigned int lpi)
* ignore them, as they have no further state and no-one can expect
* to see them if they have not been mapped.
*/
- if ( hlpi.virt_lpi == INVALID_LPI )
+ if ( hlpi.virt_lpi == INVALID_LPI && hlpi.db_vcpu_id == INVALID_VCPU_ID )
goto out;
d = rcu_lock_domain_by_id(hlpi.dom_id);
if ( !d )
goto out;
- /*
- * TODO: Investigate what to do here for potential interrupt storms.
- * As we keep all host LPIs enabled, for disabling LPIs we would need
- * to queue a ITS host command, which we avoid so far during a guest's
- * runtime. Also re-enabling would trigger a host command upon the
- * guest sending a command, which could be an attack vector for
- * hogging the host command queue.
- * See the thread around here for some background:
- * https://lists.xen.org/archives/html/xen-devel/2016-12/msg00003.html
- */
- vgic_vcpu_inject_lpi(d, hlpi.virt_lpi);
+ /* It is a doorbell interrupt. */
+ if ( hlpi.db_vcpu_id != INVALID_VCPU_ID )
+ {
+#ifdef CONFIG_GICV4
+ struct vcpu *v = d->vcpu[hlpi.db_vcpu_id];
+
+ /* We got the message, no need to fire again */
+ its_vpe_mask_db(v->arch.vgic.its_vpe);
+
+ /*
+ * Update the pending_last flag that indicates that VLPIs are pending.
+ * And the corresponding vcpu is also kicked into action.
+ */
+ v->arch.vgic.its_vpe->pending_last = true;
+
+ vcpu_kick(v);
+#else
+ printk(XENLOG_WARNING
+ "Doorbell LPI is only suooprted on GICV4\n");
+#endif
+ }
+ else
+ {
+ /*
+ * TODO: Investigate what to do here for potential interrupt storms.
+ * As we keep all host LPIs enabled, for disabling LPIs we would need
+ * to queue a ITS host command, which we avoid so far during a guest's
+ * runtime. Also re-enabling would trigger a host command upon the
+ * guest sending a command, which could be an attack vector for
+ * hogging the host command queue.
+ * See the thread around here for some background:
+ * https://lists.xen.org/archives/html/xen-devel/2016-12/msg00003.html
+ */
+ vgic_vcpu_inject_lpi(d, hlpi.virt_lpi);
+ }
rcu_unlock_domain(d);
@@ -187,7 +211,8 @@ out:
}
void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
- uint32_t virt_lpi)
+ uint32_t virt_lpi, bool is_db,
+ uint16_t db_vcpu_id)
{
union host_lpi *hlpip, hlpi;
@@ -197,8 +222,16 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
hlpip = &lpi_data.host_lpis[host_lpi / HOST_LPIS_PER_PAGE][host_lpi % HOST_LPIS_PER_PAGE];
- hlpi.virt_lpi = virt_lpi;
- hlpi.dom_id = domain_id;
+ if ( !is_db )
+ {
+ hlpi.virt_lpi = virt_lpi;
+ hlpi.dom_id = domain_id;
+ }
+ else
+ {
+ hlpi.dom_id = domain_id;
+ hlpi.db_vcpu_id = db_vcpu_id;
+ }
write_u64_atomic(&hlpip->data, hlpi.data);
}
@@ -595,6 +628,7 @@ int gicv3_allocate_host_lpi_block(struct domain *d, uint32_t *first_lpi)
*/
hlpi.virt_lpi = INVALID_LPI;
hlpi.dom_id = d->domain_id;
+ hlpi.db_vcpu_id = INVALID_VCPU_ID;
write_u64_atomic(&lpi_data.host_lpis[chunk][lpi_idx + i].data,
hlpi.data);
@@ -602,7 +636,8 @@ int gicv3_allocate_host_lpi_block(struct domain *d, uint32_t *first_lpi)
* Enable this host LPI, so we don't have to do this during the
* guest's runtime.
*/
- lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
+ lpi_write_config(lpi_data.lpi_property, lpi + i + LPI_OFFSET, 0xff,
+ LPI_PROP_ENABLED);
}
lpi_data.next_free_lpi = lpi + LPI_BLOCK;
diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
index 175fda7acb..0462976b93 100644
--- a/xen/arch/arm/gic-v4-its.c
+++ b/xen/arch/arm/gic-v4-its.c
@@ -157,6 +157,9 @@ static int its_send_cmd_vmapp(struct host_its *its, struct its_vpe *vpe,
cmd[3] = (vpt_addr & GENMASK(51, 16)) |
((HOST_LPIS_NRBITS - 1) & GENMASK(4, 0));
+ /* Default doorbell interrupt */
+ cmd[1] |= (uint64_t)vpe->vpe_db_lpi;
+
out:
ret = its_send_command(its, cmd);
@@ -296,6 +299,37 @@ static int its_send_cmd_vmovp(struct its_vpe *vpe)
return 0;
}
+
+static void its_vpe_send_inv_db(struct its_vpe *vpe)
+{
+ // struct its_device *dev = vpe_proxy.dev;
+ // unsigned long flags;
+
+ // spin_lock_irqsave(&vpe_proxy.lock, flags);
+ // gicv4_vpe_db_proxy_map_locked(vpe);
+ // its_send_cmd_inv(dev->hw_its, dev->host_devid, vpe->vpe_proxy_event);
+ // spin_unlock_irqrestore(&vpe_proxy.lock, flags);
+}
+
+static void its_vpe_inv_db(struct its_vpe *vpe)
+{
+ its_vpe_send_inv_db(vpe);
+}
+
+void its_vpe_mask_db(struct its_vpe *vpe)
+{
+ /* Only clear enable bit. */
+ lpi_write_config(lpi_data.lpi_property, vpe->vpe_db_lpi, LPI_PROP_ENABLED, 0);
+ its_vpe_inv_db(vpe);
+}
+
+static void its_vpe_unmask_db(struct its_vpe *vpe)
+{
+ /* Only set enable bit. */
+ lpi_write_config(lpi_data.lpi_property, vpe->vpe_db_lpi, 0, LPI_PROP_ENABLED);
+ its_vpe_inv_db(vpe);
+}
+
static void __init its_vpe_teardown(struct its_vpe *vpe)
{
unsigned int order;
@@ -309,6 +343,8 @@ static void __init its_vpe_teardown(struct its_vpe *vpe)
int vgic_v4_its_vm_init(struct domain *d)
{
unsigned int nr_vcpus = d->max_vcpus;
+ unsigned int nr_db_lpis, nr_chunks, i = 0;
+ uint32_t *db_lpi_bases;
int ret = -ENOMEM;
if ( !gicv3_its_host_has_its() )
@@ -326,9 +362,31 @@ int vgic_v4_its_vm_init(struct domain *d)
d->arch.vgic.its_vm->vproptable = lpi_allocate_proptable();
if ( !d->arch.vgic.its_vm->vproptable )
goto fail_vprop;
+ /* Allocate a doorbell interrupt for each VPE. */
+ nr_db_lpis = d->arch.vgic.its_vm->nr_vpes;
+ nr_chunks = DIV_ROUND_UP(nr_db_lpis, LPI_BLOCK);
+ db_lpi_bases = xzalloc_array(uint32_t, nr_chunks);
+ if ( !db_lpi_bases )
+ goto fail_db_bases;
+
+ do {
+ /* Allocate doorbell interrupts in chunks of LPI_BLOCK (=32). */
+ ret = gicv3_allocate_host_lpi_block(d, &db_lpi_bases[i]);
+ if ( ret )
+ goto fail_db;
+ } while ( ++i < nr_chunks );
+
+ d->arch.vgic.its_vm->db_lpi_bases = db_lpi_bases;
+ d->arch.vgic.its_vm->nr_db_lpis = nr_db_lpis;
return 0;
+fail_db:
+ while ( --i >= 0 )
+ gicv3_free_host_lpi_block(d->arch.vgic.its_vm->db_lpi_bases[i]);
+ xfree(db_lpi_bases);
+fail_db_bases:
+ lpi_free_proptable(d->arch.vgic.its_vm->vproptable);
fail_vprop:
xfree(d->arch.vgic.its_vm->vpes);
fail_vpes:
@@ -340,8 +398,13 @@ fail_vprop:
void vgic_v4_free_its_vm(struct domain *d)
{
struct its_vm *its_vm = d->arch.vgic.its_vm;
+ int nr_chunks = DIV_ROUND_UP(its_vm->nr_db_lpis, LPI_BLOCK);
if ( its_vm->vpes )
xfree(its_vm->vpes);
+ while ( --nr_chunks >= 0 )
+ gicv3_free_host_lpi_block(its_vm->db_lpi_bases[nr_chunks]);
+ if ( its_vm->db_lpi_bases )
+ xfree(its_vm->db_lpi_bases);
if ( its_vm->vproptable )
lpi_free_proptable(its_vm);
}
@@ -357,14 +420,29 @@ int vgic_v4_its_vpe_init(struct vcpu *vcpu)
return -ENOMEM;
its_vm->vpes[vcpuid] = vcpu->arch.vgic.its_vpe;
+ vcpu->arch.vgic.its_vpe = vcpu->arch.vgic.its_vpe;
+ vcpu->arch.vgic.its_vpe->vpe_db_lpi = its_vm->db_lpi_bases[vcpuid/32] + (vcpuid % 32);
+ /*
+ * Sometimes vlpi gets firstly mapped before associated vpe
+ * becoming resident, so in case missing the interrupt, we intend to
+ * enable doorbell at the initialization stage
+ */
+
vcpu->arch.vgic.its_vpe->its_vm = its_vm;
+ gicv3_lpi_update_host_entry(vcpu->arch.vgic.its_vpe->vpe_db_lpi,
+ vcpu->domain->domain_id, INVALID_LPI, true,
+ vcpu->vcpu_id);
+
+
ret = its_vpe_init(vcpu->arch.vgic.its_vpe);
if ( ret )
{
its_vpe_teardown(vcpu->arch.vgic.its_vpe);
return ret;
}
+ its_vpe_unmask_db(vcpu->arch.vgic.its_vpe);
+
return 0;
}
@@ -800,6 +878,7 @@ void vgic_v4_load(struct vcpu *vcpu)
* corresponding to our current CPU expects us here
*/
WARN_ON(gicv4_vpe_set_affinity(vcpu));
+ its_vpe_mask_db(vpe);
its_make_vpe_resident(vpe, vcpu->processor);
vpe->resident = true;
}
@@ -812,5 +891,53 @@ void vgic_v4_put(struct vcpu *vcpu, bool need_db)
return;
its_make_vpe_non_resident(vpe, vcpu->processor);
+ if ( need_db )
+ /* Enable the doorbell, as the guest is going to block */
+ its_vpe_unmask_db(vpe);
vpe->resident = false;
}
+
+static int its_vlpi_set_doorbell(struct its_vlpi_map *map, bool enable)
+{
+ if (map->db_enabled == enable)
+ return 0;
+
+ map->db_enabled = enable;
+
+ /*
+ * Ideally, we'd issue a VMAPTI to set the doorbell to its LPI
+ * value or to 1023, depending on the enable bit. But that
+ * would be issuing a mapping for an /existing/ DevID+EventID
+ * pair, which is UNPREDICTABLE. Instead, let's issue a VMOVI
+ * to the /same/ vPE, using this opportunity to adjust the doorbell.
+ */
+ return its_send_cmd_vmovi(map->dev->hw_its, map);
+}
+
+int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
+ bool needs_inv)
+{
+ struct its_vlpi_map *map;
+ unsigned int cpu;
+ int ret;
+
+ if ( !pirq->vlpi_map )
+ return -EINVAL;
+
+ map = pirq->vlpi_map;
+
+ /* Cache the updated property and update the vproptable. */
+ map->properties = property;
+ lpi_write_config(map->vm->vproptable, pirq->irq, 0xff, property);
+
+ if ( needs_inv )
+ {
+ cpu = map->vm->vpes[map->vpe_idx]->col_idx;
+ ret = its_inv_lpi(map->dev->hw_its, map->dev, map->eventid, cpu);
+ if ( ret )
+ return ret;
+ }
+
+ return its_vlpi_set_doorbell(map, property & LPI_PROP_ENABLED);
+}
+
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index f03a8fad47..dababe97cd 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -295,7 +295,9 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
uint32_t vdevid, uint32_t eventid,
uint32_t virt_lpi);
void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
- uint32_t virt_lpi);
+ uint32_t virt_lpi, bool is_db,
+ uint16_t db_vcpu_id);
+
struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type);
bool its_alloc_table_entry(struct its_baser *baser, uint32_t id);
struct page_info *lpi_allocate_pendtable(void);
@@ -322,6 +324,8 @@ bool event_is_forwarded_to_vcpu(struct its_device *dev, uint32_t eventid);
void its_vpe_mask_db(struct its_vpe *vpe);
#endif
int gicv4_its_vlpi_unmap(struct pending_irq *pirq);
+int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
+ bool needs_inv);
/* ITS quirks handling. */
uint64_t gicv3_its_get_cacheability(void);
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 94f7dd7d90..0a740ad68f 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -387,7 +387,7 @@ out_unlock:
* property table and update the virtual IRQ's state in the given pending_irq.
* Must be called with the respective VGIC VCPU lock held.
*/
-static int update_lpi_property(struct domain *d, struct pending_irq *p)
+int update_lpi_property(struct domain *d, struct pending_irq *p, bool needs_inv)
{
paddr_t addr;
uint8_t property;
@@ -417,6 +417,9 @@ static int update_lpi_property(struct domain *d, struct pending_irq *p)
else
clear_bit(GIC_IRQ_GUEST_ENABLED, &p->status);
+ if ( pirq_is_tied_to_hw(p) )
+ return its_vlpi_prop_update(p, property, needs_inv);
+
return 0;
}
@@ -430,6 +433,9 @@ static int update_lpi_property(struct domain *d, struct pending_irq *p)
*/
static void update_lpi_vgic_status(struct vcpu *v, struct pending_irq *p)
{
+ if ( pirq_is_tied_to_hw(p) )
+ return;
+
ASSERT(spin_is_locked(&v->arch.vgic.lock));
if ( test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) )
@@ -479,7 +485,7 @@ static int its_handle_inv(struct virt_its *its, uint64_t *cmdptr)
spin_lock_irqsave(&vcpu->arch.vgic.lock, flags);
/* Read the property table and update our cached status. */
- if ( update_lpi_property(d, p) )
+ if ( update_lpi_property(d, p, true) )
goto out_unlock;
/* Check whether the LPI needs to go on a VCPU. */
@@ -552,7 +558,7 @@ static int its_handle_invall(struct virt_its *its, uint64_t *cmdptr)
vlpi = pirqs[i]->irq;
/* If that fails for a single LPI, carry on to handle the rest. */
- err = update_lpi_property(its->d, pirqs[i]);
+ err = update_lpi_property(its->d, pirqs[i], false);
if ( !err )
update_lpi_vgic_status(vcpu, pirqs[i]);
else
@@ -785,7 +791,7 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
* We don't need the VGIC VCPU lock here, because the pending_irq isn't
* in the radix tree yet.
*/
- ret = update_lpi_property(its->d, pirq);
+ ret = update_lpi_property(its->d, pirq, true);
if ( ret )
goto out_remove_host_entry;
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 19/19] arm/gicv4: Add GICv4 to the build system
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (16 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 17/19] arm/gicv4: Handle doorbells Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-19 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 18/19] arm/gic: Add VPE proxy support Mykyta Poturai
2026-02-03 10:01 ` [RFC PATCH 00/19] GICv4 Support for Xen Bertrand Marquis
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Add a config option to enable GICv4 support and include the
corresponding source file in the build system.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/Kconfig | 6 ++++++
xen/arch/arm/Makefile | 1 +
2 files changed, 7 insertions(+)
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 442d353b43..c18eca9f9a 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -255,6 +255,12 @@ config ARM64_BTI
Branch Target Identification support.
This feature is not supported in Xen.
+config GICV4
+ bool "GICv4 driver"
+ depends on GICV3 && HAS_ITS
+ help
+ Driver for ARM Generic Interrupt Controller v4 extension.
+
source "arch/arm/tee/Kconfig"
config PARTIAL_EMULATION
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 95bc7ad25e..8569df571b 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -27,6 +27,7 @@ obj-y += gic.o
obj-$(CONFIG_GICV2) += gic-v2.o
obj-$(CONFIG_GICV3) += gic-v3.o
obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
+obj-$(CONFIG_GICV4) += gic-v4-its.o
obj-$(CONFIG_HAS_ITS) += gic-v3-lpi.o
obj-y += guestcopy.o
obj-y += guest_atomics.o
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [RFC PATCH 18/19] arm/gic: Add VPE proxy support
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (17 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 19/19] arm/gicv4: Add GICv4 to the build system Mykyta Poturai
@ 2026-02-02 16:14 ` Mykyta Poturai
2026-02-19 16:00 ` Mykola Kvach
2026-02-03 10:01 ` [RFC PATCH 00/19] GICv4 Support for Xen Bertrand Marquis
19 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-02 16:14 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: xakep.amatop@gmail.com, Mykyta Poturai, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
xen/arch/arm/gic-v3-its.c | 13 ++
xen/arch/arm/gic-v3.c | 1 +
xen/arch/arm/gic-v4-its.c | 207 ++++++++++++++++++++++++-
xen/arch/arm/include/asm/gic_v3_defs.h | 7 +
xen/arch/arm/include/asm/gic_v3_its.h | 7 +
xen/arch/arm/include/asm/gic_v4_its.h | 5 +
xen/arch/arm/include/asm/vgic.h | 2 +
7 files changed, 235 insertions(+), 7 deletions(-)
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index fa5c1eb6d1..5979a82526 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -333,6 +333,19 @@ int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
return its_send_command(its, cmd);
}
+int its_send_cmd_movi(struct host_its *its, uint32_t deviceid, uint32_t eventid,
+ uint16_t icid)
+{
+ uint64_t cmd[4];
+
+ cmd[0] = GITS_CMD_MOVI | ((uint64_t)deviceid << 32);
+ cmd[1] = eventid;
+ cmd[2] = icid;
+ cmd[3] = 0x00;
+
+ return its_send_command(its, cmd);
+}
+
/* Set up the (1:1) collection mapping for the given host CPU. */
int gicv3_its_setup_collection(unsigned int cpu)
{
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 1cb3169b72..fb80038f17 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -2096,6 +2096,7 @@ static void __init gicv4_init(void)
gicv4_its_vpeid_allocator_init();
+ gicv4_init_vpe_proxy();
}
#else
static void __init gicv4_init(void)
diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
index 0462976b93..83ee0510ac 100644
--- a/xen/arch/arm/gic-v4-its.c
+++ b/xen/arch/arm/gic-v4-its.c
@@ -39,6 +39,13 @@ static spinlock_t vpeid_alloc_lock = SPIN_LOCK_UNLOCKED;
static uint16_t vmovp_seq_num;
static spinlock_t vmovp_lock = SPIN_LOCK_UNLOCKED;
+static struct {
+ spinlock_t lock;
+ struct its_device *dev;
+ struct its_vpe **vpes;
+ int next_victim;
+} vpe_proxy;
+
void __init gicv4_its_vpeid_allocator_init(void)
{
/* Allocate space for vpeid_mask based on MAX_VPEID */
@@ -201,6 +208,124 @@ static int its_map_vpe(struct host_its *its, struct its_vpe *vpe)
return 0;
}
+static int gicv4_vpe_db_proxy_unmap_locked(struct its_vpe *vpe)
+{
+ int ret;
+
+ /* Already unmapped? */
+ if ( vpe->vpe_proxy_event == -1 )
+ return 0;
+
+ ret = its_send_cmd_discard(vpe_proxy.dev->hw_its, vpe_proxy.dev,
+ vpe->vpe_proxy_event);
+ if ( ret )
+ return ret;
+ vpe_proxy.vpes[vpe->vpe_proxy_event] = NULL;
+
+ /*
+ * We don't track empty slots at all, so let's move the
+ * next_victim pointer to quickly reuse the unmapped slot
+ */
+ if ( vpe_proxy.vpes[vpe_proxy.next_victim] )
+ vpe_proxy.next_victim = vpe->vpe_proxy_event;
+
+ vpe->vpe_proxy_event = -1;
+
+ return 0;
+}
+
+static void gicv4_vpe_db_proxy_unmap(struct its_vpe *vpe)
+{
+ if ( !gic_support_directLPI() )
+ {
+ unsigned long flags;
+
+ spin_lock_irqsave(&vpe_proxy.lock, flags);
+ gicv4_vpe_db_proxy_unmap_locked(vpe);
+ spin_unlock_irqrestore(&vpe_proxy.lock, flags);
+ }
+}
+
+/*
+ * If a GICv4.0 doesn't implement Direct LPIs (which is extremely
+ * likely), the only way to perform an invalidate is to use a fake
+ * device to issue an INV command, implying that the LPI has first
+ * been mapped to some event on that device. Since this is not exactly
+ * cheap, we try to keep that mapping around as long as possible, and
+ * only issue an UNMAP if we're short on available slots.
+ *
+ * GICv4.1 mandates that we're able to invalidate by writing to a
+ * MMIO register. And most of the time, we don't even have to invalidate
+ * vPE doorbell, as the redistributor can be told whether to generate a
+ * doorbell or not.
+ */
+static int gicv4_vpe_db_proxy_map_locked(struct its_vpe *vpe)
+{
+ int ret;
+
+ /* Already mapped? */
+ if ( vpe->vpe_proxy_event != -1 )
+ return 0;
+
+ /* This slot was already allocated. Kick the other VPE out. */
+ if ( vpe_proxy.vpes[vpe_proxy.next_victim] )
+ {
+ struct its_vpe *old_vpe = vpe_proxy.vpes[vpe_proxy.next_victim];
+
+ ret = gicv4_vpe_db_proxy_unmap_locked(old_vpe);
+ if ( ret )
+ return ret;
+ }
+
+ /* Map the new VPE instead */
+ vpe_proxy.vpes[vpe_proxy.next_victim] = vpe;
+ vpe->vpe_proxy_event = vpe_proxy.next_victim;
+ vpe_proxy.next_victim = (vpe_proxy.next_victim + 1) %
+ vpe_proxy.dev->eventids;
+
+ return its_send_cmd_mapti(vpe_proxy.dev->hw_its, vpe_proxy.dev->host_devid,
+ vpe->vpe_proxy_event, vpe->vpe_db_lpi,
+ vpe->col_idx);
+}
+
+int __init gicv4_init_vpe_proxy(void)
+{
+ struct host_its *hw_its;
+ uint32_t devid;
+
+ if ( gic_support_directLPI() )
+ {
+ printk("ITS: Using DirectLPI for GICv4 VPE invalidation\n");
+ return 0;
+ }
+
+ /* Any ITS will do, even if not v4 */
+ hw_its = list_first_entry(&host_its_list, struct host_its, entry);
+
+ vpe_proxy.vpes = xzalloc_array(struct its_vpe *, nr_cpu_ids);
+ if ( !vpe_proxy.vpes )
+ {
+ printk(XENLOG_ERR "ITS: Can't allocate GICv4 VPE proxy device array\n");
+ return -ENOMEM;
+ }
+
+ /* Use the last possible DevID */
+ devid = BIT(hw_its->devid_bits, UL) - 1;
+ vpe_proxy.dev = its_create_device(hw_its, devid, nr_cpu_ids);
+ if ( !vpe_proxy.dev )
+ {
+ printk(XENLOG_ERR "ITS: Can't allocate GICv4 VPE proxy device\n");
+ return -ENOMEM;
+ }
+
+ spin_lock_init(&vpe_proxy.lock);
+ vpe_proxy.next_victim = 0;
+ printk(XENLOG_INFO
+ "ITS: Allocated DevID %u as GICv4 VPE proxy device\n", devid);
+
+ return 0;
+}
+
static int __init its_vpe_init(struct its_vpe *vpe)
{
int vpe_id, rc = -ENOMEM;
@@ -224,6 +349,7 @@ static int __init its_vpe_init(struct its_vpe *vpe)
rwlock_init(&vpe->lock);
vpe->vpe_id = vpe_id;
vpe->vpendtable = page_to_virt(vpendtable);
+ vpe->vpe_proxy_event = -1;
/*
* We eagerly inform all the v4 ITS and map vPE to the first
* possible CPU
@@ -299,16 +425,45 @@ static int its_send_cmd_vmovp(struct its_vpe *vpe)
return 0;
}
+/* GICR_SYNCR.Busy == 1 until the invalidation completes. */
+static void wait_for_syncr(void __iomem *rdbase)
+{
+ while ( readl_relaxed(rdbase + GICR_SYNCR) & 1 )
+ cpu_relax();
+}
+
+void direct_lpi_inv(struct its_device *dev, uint32_t eventid,
+ uint32_t db_lpi, unsigned int cpu)
+{
+ void __iomem *rdbase;
+ uint64_t val;
+ /* Register-based LPI invalidation for DB on GICv4.0 */
+ val = FIELD_PREP(GICR_INVLPIR_INTID, db_lpi);
+
+ rdbase = per_cpu(rbase, cpu);
+ writeq_relaxed(val, rdbase + GICR_INVLPIR);
+ wait_for_syncr(rdbase);
+}
static void its_vpe_send_inv_db(struct its_vpe *vpe)
{
- // struct its_device *dev = vpe_proxy.dev;
- // unsigned long flags;
+ if ( gic_support_directLPI() )
+ {
+ unsigned int cpu = vpe->col_idx;
- // spin_lock_irqsave(&vpe_proxy.lock, flags);
- // gicv4_vpe_db_proxy_map_locked(vpe);
- // its_send_cmd_inv(dev->hw_its, dev->host_devid, vpe->vpe_proxy_event);
- // spin_unlock_irqrestore(&vpe_proxy.lock, flags);
+ /* Target the redistributor this VPE is currently known on */
+ direct_lpi_inv(NULL, 0, vpe->vpe_db_lpi, cpu);
+ }
+ else
+ {
+ struct its_device *dev = vpe_proxy.dev;
+ unsigned long flags;
+
+ spin_lock_irqsave(&vpe_proxy.lock, flags);
+ gicv4_vpe_db_proxy_map_locked(vpe);
+ its_send_cmd_inv(dev->hw_its, dev->host_devid, vpe->vpe_proxy_event);
+ spin_unlock_irqrestore(&vpe_proxy.lock, flags);
+ }
}
static void its_vpe_inv_db(struct its_vpe *vpe)
@@ -335,6 +490,7 @@ static void __init its_vpe_teardown(struct its_vpe *vpe)
unsigned int order;
order = get_order_from_bytes(max(lpi_data.max_host_lpi_ids / 8, (unsigned long)SZ_64K));
+ gicv4_vpe_db_proxy_unmap(vpe);
its_free_vpeid(vpe->vpe_id);
free_xenheap_pages(vpe->vpendtable, order);
xfree(vpe);
@@ -830,6 +986,43 @@ static void vpe_to_cpuid_unlock(struct its_vpe *vpe, unsigned long *flags)
spin_unlock_irqrestore(&vpe->vpe_lock, *flags);
}
+static void gicv4_vpe_db_proxy_move(struct its_vpe *vpe, unsigned int from,
+ unsigned int to)
+{
+ unsigned long flags;
+
+ if ( gic_support_directLPI() )
+ {
+ void __iomem *rdbase;
+
+ rdbase = per_cpu(rbase, from);
+ /* Clear potential pending state on the old redistributor */
+ writeq_relaxed(vpe->vpe_db_lpi, rdbase + GICR_CLRLPIR);
+ wait_for_syncr(rdbase);
+ return;
+ }
+
+ spin_lock_irqsave(&vpe_proxy.lock, flags);
+
+ gicv4_vpe_db_proxy_map_locked(vpe);
+
+ /* MOVI instructs the appropriate Redistributor to move the pending state */
+ its_send_cmd_movi(vpe_proxy.dev->hw_its, vpe_proxy.dev->host_devid,
+ vpe->vpe_proxy_event, to);
+
+ /*
+ * ARM spec says that If, after using MOVI to move an interrupt from
+ * collection A to collection B, software moves the same interrupt again
+ * from collection B to collection C, a SYNC command must be used before
+ * the second MOVI for the Redistributor associated with collection A to
+ * ensure correct behavior.
+ * So each time we issue VMOVI, we VSYNC the old VPE for good measure.
+ */
+ WARN_ON(its_send_cmd_sync(vpe_proxy.dev->hw_its, from));
+
+ spin_unlock_irqrestore(&vpe_proxy.lock, flags);
+}
+
static int gicv4_vpe_set_affinity(struct vcpu *vcpu)
{
struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
@@ -859,6 +1052,7 @@ static int gicv4_vpe_set_affinity(struct vcpu *vcpu)
ret = its_send_cmd_vmovp(vpe);
if ( ret )
goto out;
+ gicv4_vpe_db_proxy_move(vpe, from, to);
out:
vpe_to_cpuid_unlock(vpe, &flags);
@@ -940,4 +1134,3 @@ int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
return its_vlpi_set_doorbell(map, property & LPI_PROP_ENABLED);
}
-
diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
index 0db75309cf..b4d50516ef 100644
--- a/xen/arch/arm/include/asm/gic_v3_defs.h
+++ b/xen/arch/arm/include/asm/gic_v3_defs.h
@@ -20,6 +20,13 @@
#include <xen/sizes.h>
+#ifndef FIELD_GET
+#define FIELD_GET(_mask, _reg) \
+ ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
+#endif
+
+#define FIELD_PREP(_mask, _val) \
+ (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
/*
* Additional registers defined in GIC v3.
* Common GICD registers are defined in gic.h
diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
index dababe97cd..0e82625840 100644
--- a/xen/arch/arm/include/asm/gic_v3_its.h
+++ b/xen/arch/arm/include/asm/gic_v3_its.h
@@ -236,6 +236,11 @@ int its_inv_lpi(struct host_its *its, struct its_device *dev,
uint32_t eventid, unsigned int cpu);
int its_send_cmd_mapti(struct host_its *its, uint32_t deviceid,
uint32_t eventid, uint32_t pintid, uint16_t icid);
+struct its_device *its_create_device(struct host_its *hw_its,
+ uint32_t host_devid, uint64_t nr_events);
+int its_send_cmd_movi(struct host_its *its, uint32_t deviceid, uint32_t eventid,
+ uint16_t icid);
+int its_send_cmd_sync(struct host_its *its, unsigned int cpu);
#ifdef CONFIG_ACPI
unsigned long gicv3_its_make_hwdom_madt(const struct domain *d,
void *base_ptr);
@@ -326,6 +331,8 @@ void its_vpe_mask_db(struct its_vpe *vpe);
int gicv4_its_vlpi_unmap(struct pending_irq *pirq);
int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
bool needs_inv);
+void direct_lpi_inv(struct its_device *dev, uint32_t eventid,
+ uint32_t db_lpi, unsigned int cpu);
/* ITS quirks handling. */
uint64_t gicv3_its_get_cacheability(void);
diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
index 37b6b92f0c..1d800fdbaf 100644
--- a/xen/arch/arm/include/asm/gic_v4_its.h
+++ b/xen/arch/arm/include/asm/gic_v4_its.h
@@ -52,6 +52,7 @@ struct event_vlpi_map {
};
void gicv4_its_vpeid_allocator_init(void);
+int gicv4_init_vpe_proxy(void);
#define GICR_VPROPBASER 0x0070
#define GICR_VPENDBASER 0x0078
@@ -97,6 +98,10 @@ static inline void gits_write_vpendbaser(uint64_t val, void __iomem *addr)
}
#define gits_read_vpendbaser(c) readq_relaxed(c)
+#define GICR_INVLPIR_INTID GENMASK_ULL(31, 0)
+#define GICR_INVLPIR_VPEID GICR_INVALLR_VPEID
+#define GICR_INVLPIR_V GICR_INVALLR_V
+
#endif
/*
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 9ef667decb..558f81818c 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -407,9 +407,11 @@ extern void vgic_check_inflight_irqs_pending(struct vcpu *v,
/* GICV4 functions */
#ifdef CONFIG_GICV4
+bool gic_support_directLPI(void);
bool gic_support_vptValidDirty(void);
bool gic_is_gicv4(void);
#else
+#define gic_support_directLPI() (false)
#define gic_support_vptValidDirty() (false)
#define gic_is_gicv4() (false)
#endif
--
2.51.2
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 07/19] xen/domain: Alloc enough pages for VCPU struct
2026-02-02 16:14 ` [RFC PATCH 07/19] xen/domain: Alloc enough pages for VCPU struct Mykyta Poturai
@ 2026-02-02 16:45 ` Jan Beulich
0 siblings, 0 replies; 42+ messages in thread
From: Jan Beulich @ 2026-02-02 16:45 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xakep.amatop@gmail.com, Andrew Cooper, Anthony PERARD,
Michal Orzel, Julien Grall, Roger Pau Monné,
Stefano Stabellini, xen-devel@lists.xenproject.org
On 02.02.2026 17:14, Mykyta Poturai wrote:
> With introduction of GICv4 the size of struct vcpu can again be more
> than one page. Modify struct vcpu allocation to request enough pages
> again.
>
> Don't reintroduce the MAX_PAGES_PER_VCPU check.
> As per commit b77d774d8274183c2252f5fbc9fa3b3b7022ba06
>> It turns out that beyond efficiency, maybe, there is no real technical
>> reason this struct has to fit in one page
>
> Since there is no technical reason to limit struct vcpu size to one page,
> there also seems to be little reason the fiddle with 1 or 2 page limits.
Before writing this, did you check the recent discussion around the Arm
side change going back to a single page, and the moving of this code to
common/? Any ...
> --- a/xen/common/domain.c
> +++ b/xen/common/domain.c
> @@ -342,18 +342,24 @@ static struct vcpu *alloc_vcpu_struct(const struct domain *d)
> # define arch_vcpu_struct_memflags(d) ((void)(d), 0)
> #endif
> struct vcpu *v;
> + unsigned int order = get_order_from_bytes(sizeof(*v));
>
> - BUILD_BUG_ON(sizeof(*v) > PAGE_SIZE);
> - v = alloc_xenheap_pages(0, arch_vcpu_struct_memflags(d));
> + v = alloc_xenheap_pages(order, arch_vcpu_struct_memflags(d));
... non-order-0 allocation is at risk of failing despite there being
ample memory available, if there's heavy fragmentation. I'm sorry, but
without a much better justification this gets a NAK from me.
Jan
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 00/19] GICv4 Support for Xen
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
` (18 preceding siblings ...)
2026-02-02 16:14 ` [RFC PATCH 18/19] arm/gic: Add VPE proxy support Mykyta Poturai
@ 2026-02-03 10:01 ` Bertrand Marquis
2026-02-03 12:24 ` Mykyta Poturai
` (2 more replies)
19 siblings, 3 replies; 42+ messages in thread
From: Bertrand Marquis @ 2026-02-03 10:01 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, xakep.amatop@gmail.com,
Stefano Stabellini, Julien Grall, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
Hi Mykyta,
We have a number of series from you which have not been merged yet and
reviewing them all in parallel might be challenging.
Would you mind giving us a status and maybe priorities on them.
I could list the following series:
- GICv4
- CPU Hotplug on arm
- PCI enumeration on arm
- IPMMU for pci on arm
- dom0less for pci passthrough on arm
- SR-IOV for pvh
- SMMU for pci on arm
- MSI injection on arm
- suspend to ram on arm
There might be others feel free to complete the list.
On GICv4...
> On 2 Feb 2026, at 17:14, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> This series introduces GICv4 direct LPI injection for Xen.
>
> Direct LPI injection relies on the GIC tracking the mapping between physical and
> virtual CPUs. Each VCPU requires a VPE that is created and registered with the
> GIC via the `VMAPP` ITS command. The GIC is then informed of the current
> VPE-to-PCPU placement by programming `VPENDBASER` and `VPROPBASER` in the
> appropriate redistributor. LPIs are associated with VPEs through the `VMAPTI`
> ITS command, after which the GIC handles delivery without trapping into the
> hypervisor for each interrupt.
>
> When a VPE is not scheduled but has pending interrupts, the GIC raises a per-VPE
> doorbell LPI. Doorbells are owned by the hypervisor and prompt rescheduling so
> the VPE can drain its pending LPIs.
>
> Because GICv4 lacks a native doorbell invalidation mechanism, this series
> includes a helper that invalidates doorbell LPIs via synthetic “proxy” devices,
> following the approach used until GICv4.1.
>
> All of this work is mostly based on the work of Penny Zheng
> <penny.zheng@arm.com> and Luca Fancellu <luca.fancellu@arm.com>. And also from
> Linux patches by Mark Zyngier.
>
> Some patches are still a little rough and need some styling fixes and more
> testing, as all of them needed to be carved line by line from a giant ~4000 line
> patch. This RFC is directed mostly to get a general idea if the proposed
> approach is suitable and OK with everyone. And there is still an open question
> of how to handle Signed-off-by lines for Penny and Luca, since they have not
> indicated their preference yet.
I would like to ask how much performance benefits you could
have with this.
Adding GICv4 support is adding a lot of code which will have to be maintained
and tested and there should be a good improvement to justify this.
Did you do some benchmarks ? what are the results ?
At the time where we started to work on that at Arm, we ended up in the conclusion
that the complexity in Xen compared to the benefit was not justifying it hence why
this work was stopped in favor of other features that we thought would be more
beneficial to Xen (like PCI passthrough or SMMUv3).
Cheers
Bertrand
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 00/19] GICv4 Support for Xen
2026-02-03 10:01 ` [RFC PATCH 00/19] GICv4 Support for Xen Bertrand Marquis
@ 2026-02-03 12:24 ` Mykyta Poturai
2026-02-03 16:27 ` Bertrand Marquis
2026-02-13 11:36 ` Mykola Kvach
2026-03-17 9:11 ` Mykola Kvach
2 siblings, 1 reply; 42+ messages in thread
From: Mykyta Poturai @ 2026-02-03 12:24 UTC (permalink / raw)
To: Bertrand Marquis
Cc: xen-devel@lists.xenproject.org, xakep.amatop@gmail.com,
Stefano Stabellini, Julien Grall, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
On 03.02.26 12:01, Bertrand Marquis wrote:
> Hi Mykyta,
>
> We have a number of series from you which have not been merged yet and
> reviewing them all in parallel might be challenging.
>
> Would you mind giving us a status and maybe priorities on them.
>
> I could list the following series:
> - GICv4
> - CPU Hotplug on arm
> - PCI enumeration on arm
> - IPMMU for pci on arm
> - dom0less for pci passthrough on arm
> - SR-IOV for pvh
> - SMMU for pci on arm
> - MSI injection on arm
> - suspend to ram on arm
>
> There might be others feel free to complete the list.
>
> On GICv4...
>
>> On 2 Feb 2026, at 17:14, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>
>> This series introduces GICv4 direct LPI injection for Xen.
>>
>> Direct LPI injection relies on the GIC tracking the mapping between physical and
>> virtual CPUs. Each VCPU requires a VPE that is created and registered with the
>> GIC via the `VMAPP` ITS command. The GIC is then informed of the current
>> VPE-to-PCPU placement by programming `VPENDBASER` and `VPROPBASER` in the
>> appropriate redistributor. LPIs are associated with VPEs through the `VMAPTI`
>> ITS command, after which the GIC handles delivery without trapping into the
>> hypervisor for each interrupt.
>>
>> When a VPE is not scheduled but has pending interrupts, the GIC raises a per-VPE
>> doorbell LPI. Doorbells are owned by the hypervisor and prompt rescheduling so
>> the VPE can drain its pending LPIs.
>>
>> Because GICv4 lacks a native doorbell invalidation mechanism, this series
>> includes a helper that invalidates doorbell LPIs via synthetic “proxy” devices,
>> following the approach used until GICv4.1.
>>
>> All of this work is mostly based on the work of Penny Zheng
>> <penny.zheng@arm.com> and Luca Fancellu <luca.fancellu@arm.com>. And also from
>> Linux patches by Mark Zyngier.
>>
>> Some patches are still a little rough and need some styling fixes and more
>> testing, as all of them needed to be carved line by line from a giant ~4000 line
>> patch. This RFC is directed mostly to get a general idea if the proposed
>> approach is suitable and OK with everyone. And there is still an open question
>> of how to handle Signed-off-by lines for Penny and Luca, since they have not
>> indicated their preference yet.
>
> I would like to ask how much performance benefits you could
> have with this.
> Adding GICv4 support is adding a lot of code which will have to be maintained
> and tested and there should be a good improvement to justify this.
>
> Did you do some benchmarks ? what are the results ?
>
> At the time where we started to work on that at Arm, we ended up in the conclusion
> that the complexity in Xen compared to the benefit was not justifying it hence why
> this work was stopped in favor of other features that we thought would be more
> beneficial to Xen (like PCI passthrough or SMMUv3).
>
> Cheers
> Bertrand
>
Hi Bertrand
Current priorities are:
- CPU hotplug
- Suspend to RAM
- GICv4 (we will follow up with benchmarks)
- SR-IOV
MSI injection, dom0less for pci and PCI enumeration are low priority for now
Suspend to RAM is handled by Mykola Kvach
SMMU and IPMMU support are merged already AFAIU
--
Mykyta
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 00/19] GICv4 Support for Xen
2026-02-03 12:24 ` Mykyta Poturai
@ 2026-02-03 16:27 ` Bertrand Marquis
0 siblings, 0 replies; 42+ messages in thread
From: Bertrand Marquis @ 2026-02-03 16:27 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, xakep.amatop@gmail.com,
Stefano Stabellini, Julien Grall, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
Hi Mykyta,
> On 3 Feb 2026, at 13:24, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> On 03.02.26 12:01, Bertrand Marquis wrote:
>> Hi Mykyta,
>>
>> We have a number of series from you which have not been merged yet and
>> reviewing them all in parallel might be challenging.
>>
>> Would you mind giving us a status and maybe priorities on them.
>>
>> I could list the following series:
>> - GICv4
>> - CPU Hotplug on arm
>> - PCI enumeration on arm
>> - IPMMU for pci on arm
>> - dom0less for pci passthrough on arm
>> - SR-IOV for pvh
>> - SMMU for pci on arm
>> - MSI injection on arm
>> - suspend to ram on arm
>>
>> There might be others feel free to complete the list.
>>
>> On GICv4...
>>
>>> On 2 Feb 2026, at 17:14, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>>
>>> This series introduces GICv4 direct LPI injection for Xen.
>>>
>>> Direct LPI injection relies on the GIC tracking the mapping between physical and
>>> virtual CPUs. Each VCPU requires a VPE that is created and registered with the
>>> GIC via the `VMAPP` ITS command. The GIC is then informed of the current
>>> VPE-to-PCPU placement by programming `VPENDBASER` and `VPROPBASER` in the
>>> appropriate redistributor. LPIs are associated with VPEs through the `VMAPTI`
>>> ITS command, after which the GIC handles delivery without trapping into the
>>> hypervisor for each interrupt.
>>>
>>> When a VPE is not scheduled but has pending interrupts, the GIC raises a per-VPE
>>> doorbell LPI. Doorbells are owned by the hypervisor and prompt rescheduling so
>>> the VPE can drain its pending LPIs.
>>>
>>> Because GICv4 lacks a native doorbell invalidation mechanism, this series
>>> includes a helper that invalidates doorbell LPIs via synthetic “proxy” devices,
>>> following the approach used until GICv4.1.
>>>
>>> All of this work is mostly based on the work of Penny Zheng
>>> <penny.zheng@arm.com> and Luca Fancellu <luca.fancellu@arm.com>. And also from
>>> Linux patches by Mark Zyngier.
>>>
>>> Some patches are still a little rough and need some styling fixes and more
>>> testing, as all of them needed to be carved line by line from a giant ~4000 line
>>> patch. This RFC is directed mostly to get a general idea if the proposed
>>> approach is suitable and OK with everyone. And there is still an open question
>>> of how to handle Signed-off-by lines for Penny and Luca, since they have not
>>> indicated their preference yet.
>>
>> I would like to ask how much performance benefits you could
>> have with this.
>> Adding GICv4 support is adding a lot of code which will have to be maintained
>> and tested and there should be a good improvement to justify this.
>>
>> Did you do some benchmarks ? what are the results ?
>>
>> At the time where we started to work on that at Arm, we ended up in the conclusion
>> that the complexity in Xen compared to the benefit was not justifying it hence why
>> this work was stopped in favor of other features that we thought would be more
>> beneficial to Xen (like PCI passthrough or SMMUv3).
>>
>> Cheers
>> Bertrand
>>
>
> Hi Bertrand
>
> Current priorities are:
>
> - CPU hotplug
> - Suspend to RAM
> - GICv4 (we will follow up with benchmarks)
> - SR-IOV
>
Ok Let's focus on what is already there and being reviewed before GICv4.
I will follow up and your CPU hotplug review and suspend to RAM is already advanced so
we should focus on finishing those first.
Cheers
Bertrand
>
> MSI injection, dom0less for pci and PCI enumeration are low priority for now
>
> Suspend to RAM is handled by Mykola Kvach
>
> SMMU and IPMMU support are merged already AFAIU
>
> --
> Mykyta
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 01/19] arm/gicv4 add management structure definitions
2026-02-02 16:14 ` [RFC PATCH 01/19] arm/gicv4 add management structure definitions Mykyta Poturai
@ 2026-02-11 8:28 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-11 8:28 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Hi Mykyta
Thank you for the contribution.
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Add GICv4 specific structures to be used in later patches.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/include/asm/vgic.h | 43 +++++++++++++++++++++++++++++++++
> 1 file changed, 43 insertions(+)
>
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 6f9ab1c98c..a874a02d70 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -127,6 +127,42 @@ struct vgic_irq_rank {
> uint8_t vcpu[32];
> };
>
> +#ifdef CONFIG_GICV4
> +struct its_vm {
> + struct its_vpe **vpes;
> + /* Number of VPE. */
> + unsigned int nr_vpes;
> + uint32_t *db_lpi_bases;
> + unsigned int nr_db_lpis;
> + /* Property table per VM. */
> + void *vproptable;
> +};
> +
> +struct its_vpe {
> + rwlock_t lock;
> + uint32_t vpe_id;
> + /* Pending table per VCPU. */
> + void *vpendtable;
> + uint32_t vpe_db_lpi;
> + struct its_vm *its_vm;
> + unsigned int col_idx;
> + bool resident;
> + /* Pending VLPIs on schedule out? */
> + bool pending_last;
> + struct {
> + /* Implementation Defined Area Invalid */
> + bool idai;
> + /* VPE proxy mapping */
> + int vpe_proxy_event;
> + };
> + /*
> + * Ensure mutual exclusion between affinity setting of the vPE
> + * and vLPI operations using vpe->col_idx.
> + */
> + spinlock_t vpe_lock;
> +};
> +#endif
> +
> struct vgic_dist {
> /* Version of the vGIC */
> enum gic_version version;
> @@ -193,6 +229,10 @@ struct vgic_dist {
> */
> bool rdists_enabled; /* Is any redistributor enabled? */
> bool has_its;
> +#ifdef CONFIG_GICV4
> + struct its_vm *its_vm;
> +#endif
> + bool nassgireq;
nit: unused in this series
Best regards,
Mykola
> #endif
> };
>
> @@ -227,6 +267,9 @@ struct vgic_cpu {
> #define VGIC_V3_RDIST_LAST (1 << 0) /* last vCPU of the rdist */
> #define VGIC_V3_LPIS_ENABLED (1 << 1)
> uint8_t flags;
> +#ifdef CONFIG_GICV4
> + struct its_vpe *its_vpe;
> +#endif
> };
>
> struct sgi_target {
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 02/19] arm/gicv4-its: Add GICv4 ITS command definitions
2026-02-02 16:14 ` [RFC PATCH 02/19] arm/gicv4-its: Add GICv4 ITS command definitions Mykyta Poturai
@ 2026-02-11 8:28 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-11 8:28 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Add definitions for GICv4 specific ITS commands.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/include/asm/gic_v4_its.h | 41 +++++++++++++++++++++++++++
> 1 file changed, 41 insertions(+)
> create mode 100644 xen/arch/arm/include/asm/gic_v4_its.h
>
> diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
> new file mode 100644
> index 0000000000..f48eae60ad
> --- /dev/null
> +++ b/xen/arch/arm/include/asm/gic_v4_its.h
> @@ -0,0 +1,41 @@
> +/*
> + * ARM GICv4 ITS support
> + *
> + * Penny Zheng <penny.zheng@arm.com>
> + * Copyright (c) 2023 ARM Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; under version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef __ASM_ARM_GICV4_ITS_H__
> +#define __ASM_ARM_GICV4_ITS_H__
> +
> +#define GITS_CMD_VMOVI 0x21
> +#define GITS_CMD_VMOVP 0x22
> +#define GITS_CMD_VSGI 0x23
nit: This looks unrelated to the changes in this series
no vSGI handling in this series
Best regards,
Mykola
> +#define GITS_CMD_VSYNC 0x25
> +#define GITS_CMD_VMAPP 0x29
> +#define GITS_CMD_VMAPTI 0x2a
> +#define GITS_CMD_VINVALL 0x2d
> +#define GITS_CMD_INVDB 0x2e
> +
> +#endif
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 06/19] arm/gicv4-its: Add VLPI map/unmap operations
2026-02-02 16:14 ` [RFC PATCH 06/19] arm/gicv4-its: Add VLPI map/unmap operations Mykyta Poturai
@ 2026-02-11 8:31 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-11 8:31 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> For VLPI to be injected into a guest, it needs to be mapped or moved to
> a corresponding VPE first. Add a struct to handle the info about the
> VLPI mapping and a flag indicating whether the IRQ is tied to a HW one.
>
> Implement mapping/unmapping of VLPIs to VPEs, also handle moving. Tie
> them to emulated MAPTI/MOVI/DISCARD commands.
>
> Add GIC_IRQ_GUEST_FORWARDED IRQ status flag to keep track of which LPIs
> are mapped to virtual ones.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3-its.c | 14 ++
> xen/arch/arm/gic-v4-its.c | 292 ++++++++++++++++++++++++++
> xen/arch/arm/include/asm/gic_v3_its.h | 20 ++
> xen/arch/arm/include/asm/gic_v4_its.h | 20 ++
> xen/arch/arm/include/asm/vgic.h | 5 +
> xen/arch/arm/vgic-v3-its.c | 42 +++-
> 6 files changed, 387 insertions(+), 6 deletions(-)
> create mode 100644 xen/arch/arm/gic-v4-its.c
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 25c07eb861..25889445f5 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -315,6 +315,20 @@ int its_send_cmd_inv(struct host_its *its,
> return its_send_command(its, cmd);
> }
>
> +int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
> + uint32_t eventid)
> +{
> + uint64_t cmd[4];
> + uint32_t deviceid = dev->host_devid;
> +
> + cmd[0] = GITS_CMD_DISCARD | ((uint64_t)deviceid << 32);
> + cmd[1] = (uint64_t)eventid;
> + cmd[2] = 0x00;
> + cmd[3] = 0x00;
> +
> + return its_send_command(its, cmd);
> +}
> +
> /* Set up the (1:1) collection mapping for the given host CPU. */
> int gicv3_its_setup_collection(unsigned int cpu)
> {
> diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
> new file mode 100644
> index 0000000000..9bbd0d96b7
> --- /dev/null
> +++ b/xen/arch/arm/gic-v4-its.c
> @@ -0,0 +1,292 @@
> +/*
> + * xen/arch/arm/gic-v4-its.c
> + *
> + * ARM Generic Interrupt Controller support v4 version
> + * based on xen/arch/arm/gic-v3-its.c and kernel GICv4 driver
> + *
> + * Copyright (C) 2023 - ARM Ltd
> + * Penny Zheng <penny.zheng@arm.com>, ARM Ltd ported to Xen
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/errno.h>
> +#include <xen/sched.h>
> +#include <xen/spinlock.h>
> +#include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>
> +#include <asm/gic_v4_its.h>
> +#include <asm/vgic.h>
> +
> +
> +static int its_send_cmd_vsync(struct host_its *its, uint16_t vpeid)
> +{
> + uint64_t cmd[4];
> +
> + cmd[0] = GITS_CMD_VSYNC;
> + cmd[1] = (uint64_t)vpeid << 32;
> + cmd[2] = 0x00;
> + cmd[3] = 0x00;
> +
> + return its_send_command(its, cmd);
> +}
> +
> +static int its_send_cmd_vmapti(struct host_its *its, struct its_device *dev,
> + uint32_t eventid)
> +{
> + uint64_t cmd[4];
> + uint32_t deviceid = dev->host_devid;
> + struct its_vlpi_map *map = &dev->event_map.vlpi_maps[eventid];
> + uint16_t vpeid = map->vm->vpes[map->vpe_idx]->vpe_id;
> + uint32_t vintid = map->vintid;
> + uint32_t db_pintid;
> +
> + if ( map->db_enabled )
> + db_pintid = map->vm->vpes[map->vpe_idx]->vpe_db_lpi;
> + else
> + db_pintid = INVALID_LPI;
If we want to disable the doorbell, the VMAPTI encoding should use
Dbell_pINTID = 1023.
Arm IHI 0069H.b, section 5.3.20 VMAPTI states:
- If Dbell_pINTID is 1023, then no physical interrupt is generated.
- It is an error if Dbell_pINTID is not a valid doorbell INTID, where
a valid INTID is either:
* 1023, or
* within the supported range for LPIs.
So using 1023 is the architected way to represent “no doorbell”.
> +
> + cmd[0] = GITS_CMD_VMAPTI | ((uint64_t)deviceid << 32);
> + cmd[1] = eventid | ((uint64_t)vpeid << 32);
> + cmd[2] = vintid | ((uint64_t)db_pintid << 32);
> + cmd[3] = 0x00;
> +
> + return its_send_command(its, cmd);
> +}
> +
> +static bool pirq_is_forwarded_to_vcpu(struct pending_irq *pirq)
> +{
> + ASSERT(pirq);
> + return test_bit(GIC_IRQ_GUEST_FORWARDED, &pirq->status);
> +}
> +
> +bool event_is_forwarded_to_vcpu(struct its_device *dev, uint32_t eventid)
> +{
> + struct pending_irq *pirq;
> +
> + /* No vlpi maps at all ? */
> + if ( !dev->event_map.vlpi_maps)
> + return false;
> +
> + pirq = dev->event_map.vlpi_maps[eventid].pirq;
> + return pirq_is_forwarded_to_vcpu(pirq);
> +}
> +
> +static int its_send_cmd_vmovi(struct host_its *its, struct its_vlpi_map *map)
> +{
> + uint64_t cmd[4];
> + struct its_device *dev = map->dev;
> + uint32_t eventid = map->eventid;
> + uint32_t deviceid = dev->host_devid;
> + uint16_t vpeid = map->vm->vpes[map->vpe_idx]->vpe_id;
> + uint32_t db_pintid;
> +
> + if ( map->db_enabled )
> + db_pintid = map->vm->vpes[map->vpe_idx]->vpe_db_lpi;
> + else
> + db_pintid = INVALID_IRQ;
Same point as above: if the intention is “no doorbell”, the architected
encoding is Dbell_pINTID = 1023.
Per Arm IHI 0069H.b, section 5.3.21 VMOVI, an error is generated only when
D == 1 and Dbell_pINTID is not a valid doorbell INTID (valid values are
1023 or an INTID within the supported LPI range). When D == 0, Dbell_pINTID
is ignored.
> +
> + cmd[0] = GITS_CMD_VMOVI | ((uint64_t)deviceid << 32);
> + cmd[1] = eventid | ((uint64_t)vpeid << 32);
> + cmd[2] = (map->db_enabled ? 1UL : 0UL) | ((uint64_t)db_pintid << 32);
> + cmd[3] = 0x00;
> +
> + return its_send_command(its, cmd);
> +}
> +
> +static int gicv4_its_vlpi_map(struct its_vlpi_map *map)
> +{
> + struct its_device *dev;
> + struct host_its *its;
> + uint32_t eventid;
> + int ret;
> +
> + if ( !map )
> + return -EINVAL;
> + dev = map->dev;
> + its = map->dev->hw_its;
> + eventid = map->eventid;
> +
> + spin_lock(&dev->event_map.vlpi_lock);
> +
> + if ( !dev->event_map.vm )
> + {
> + struct its_vlpi_map *maps;
> +
> + maps = xzalloc_array(struct its_vlpi_map, dev->event_map.nr_lpis);
> + if ( !maps )
> + {
> + ret = -ENOMEM;
> + goto err;
nit: goto out;
> + }
> +
> + dev->event_map.vm = map->vm;
> + dev->event_map.vlpi_maps = maps;
> + }
> + else if ( dev->event_map.vm != map->vm )
> + {
> + ret = -EINVAL;
> + goto err;
> + }
> +
> + /* Get our private copy of the mapping information */
> + dev->event_map.vlpi_maps[eventid] = *map;
> +
> + if ( pirq_is_forwarded_to_vcpu(map->pirq) )
> + {
> + struct its_vlpi_map *old = &dev->event_map.vlpi_maps[eventid];
> + uint32_t old_vpeid = old->vm->vpes[old->vpe_idx]->vpe_id;
Nit/bug?: old_vpeid is read after vlpi_maps[eventid] has been overwritten
with *map, so it’s not the old mapping anymore.
Should old_vpeid be captured before the assignment?
> +
> + /* Already mapped, move it around */
> + ret = its_send_cmd_vmovi(dev->hw_its, map);
> + if ( ret )
> + goto err;
> +
> + /*
> + * ARM spec says that If, after using VMOVI to move an interrupt from
> + * vPE A to vPE B, software moves the same interrupt again, a VSYNC
> + * command must be issued to vPE A between the moves to ensure correct
> + * behavior.
> + * So each time we issue VMOVI, we VSYNC the old VPE for good measure.
> + */
> + ret = its_send_cmd_vsync(dev->hw_its, old_vpeid);
> + }
> + else
> + {
> + /* Drop the original physical mapping firstly */
> + ret = its_send_cmd_discard(its, dev, eventid);
> + if ( ret )
> + goto err;
> +
> + /* Then install the virtual one */
> + ret = its_send_cmd_vmapti(its, dev, eventid);
> + if ( ret )
> + goto err;
> +
> + /* Increment the number of VLPIs */
> + dev->event_map.nr_vlpis++;
> + }
> +
> + goto out;
> +
> + err:
> + xfree(dev->event_map.vlpi_maps);
1. Bug?: unconditionally frees dev->event_map.vlpi_maps, but it’s only
newly allocated on the !dev->event_map.vm path. If called with an existing
vm/maps, this can free live state and cause UAF.
2. Prefer XFREE(dev->event_map.vlpi_maps); it frees and NULLs the pointer.
> + out:
> + spin_unlock(&dev->event_map.vlpi_lock);
> + return ret;
> +}
nit: add new line between functions
> +int gicv4_its_vlpi_unmap(struct pending_irq *pirq)
> +{
> + struct its_vlpi_map *map = pirq->vlpi_map;
> + struct its_device *dev = map->dev;
> + int ret;
> + uint32_t host_lpi;
> +
> + spin_lock(&dev->event_map.vlpi_lock);
> +
> + if ( !dev->event_map.vm || !pirq_is_tied_to_hw(pirq) )
> + {
> + ret = -EINVAL;
> + goto out;
> + }
> +
> + /* Drop the virtual mapping */
> + ret = its_send_cmd_discard(dev->hw_its, dev, map->eventid);
> + if ( ret )
> + goto out;
> +
> + /* Restore the physical one */
> + clear_bit(GIC_IRQ_GUEST_FORWARDED, &pirq->status);
> + host_lpi = dev->host_lpi_blocks[map->eventid / LPI_BLOCK] +
> + (map->eventid % LPI_BLOCK);
> + /* Map every host LPI to host CPU 0 */
> + ret = its_send_cmd_mapti(dev->hw_its, dev->host_devid, map->eventid,
> + host_lpi, 0);
> + if ( ret )
> + goto out;
> +
> + lpi_write_config(lpi_data.lpi_property, host_lpi, 0xff, LPI_PROP_ENABLED);
Are we intentionally resetting host LPI priority here (same as allocation path)?
If yes, worth documenting.
> +
> + ret = its_inv_lpi(dev->hw_its, dev, map->eventid, 0);
> + if ( ret )
> + goto out;
> +
> + xfree(map);
> + /*
> + * Drop the refcount and make the device available again if
> + * this was the last VLPI.
> + */
> + if ( !--dev->event_map.nr_vlpis )
> + {
> + dev->event_map.vm = NULL;
> + xfree(dev->event_map.vlpi_maps);
> + }
> +
> +out:
> + spin_unlock(&dev->event_map.vlpi_lock);
> + return ret;
> +}
> +
> +int gicv4_assign_guest_event(struct domain *d, paddr_t vdoorbell_address,
> + uint32_t vdevid, uint32_t eventid,
> + struct pending_irq *pirq)
> +
> +{
> + int ret = ENODEV;
> + struct its_vm *vm = d->arch.vgic.its_vm;
> + struct its_vlpi_map *map;
> + struct its_device *dev;
> +
> + spin_lock(&d->arch.vgic.its_devices_lock);
> + dev = get_its_device(d, vdoorbell_address, vdevid);
> + if ( dev && eventid < dev->eventids )
> + {
> + /* Prepare the vlpi mapping info */
> + map = xzalloc(struct its_vlpi_map);
> + if ( !map )
> + goto out;
> + map->vm = vm;
> + map->vintid = pirq->irq;
> + map->db_enabled = true;
> + map->vpe_idx = pirq->lpi_vcpu_id;
> + map->properties = pirq->lpi_priority |
> + (test_bit(GIC_IRQ_GUEST_ENABLED, &pirq->status) ?
> + LPI_PROP_ENABLED : 0);
> + map->pirq = pirq;
> + map->dev = dev;
> + map->eventid = eventid;
> +
> + ret = gicv4_its_vlpi_map(map);
> + if ( ret )
> + {
> + xfree(map);
> + goto out;
> + }
> +
> + pirq->vlpi_map = map;
> + }
> +
> + out:
> + spin_unlock(&d->arch.vgic.its_devices_lock);
> + return ret;
> +}
> +
> +int gicv4_its_vlpi_move(struct pending_irq *pirq, struct vcpu *vcpu)
> +{
> + struct its_vlpi_map *map = pirq->vlpi_map;
> + struct its_device *dev = map->dev;
map is dereferenced before it’s validated
> +
> + if ( !dev->event_map.vm || !map )
> + return -EINVAL;
> +
> + map->vpe_idx = vcpu->vcpu_id;
> + return gicv4_its_vlpi_map(map);
> +}
> diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
> index 9f0ea9ccb1..75c91c0426 100644
> --- a/xen/arch/arm/include/asm/gic_v3_its.h
> +++ b/xen/arch/arm/include/asm/gic_v3_its.h
> @@ -116,6 +116,9 @@
> /* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
> #define LPI_BLOCK 32U
>
> +#ifdef CONFIG_GICV4
> +#include <asm/gic_v4_its.h>
> +#endif
> /*
> * Describes a device which is using the ITS and is used by a guest.
> * Since device IDs are per ITS (in contrast to vLPIs, which are per
> @@ -135,6 +138,9 @@ struct its_device {
> uint32_t eventids; /* Number of event IDs (MSIs) */
> uint32_t *host_lpi_blocks; /* Which LPIs are used on the host */
> struct pending_irq *pend_irqs; /* One struct per event */
> +#ifdef CONFIG_GICV4
> + struct event_vlpi_map event_map;
> +#endif
> };
>
> /* data structure for each hardware ITS */
> @@ -184,6 +190,8 @@ extern struct __lpi_data lpi_data;
>
> extern struct list_head host_its_list;
>
> +int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
> + uint32_t eventid);
> int its_send_cmd_inv(struct host_its *its, uint32_t deviceid, uint32_t eventid);
> int its_send_cmd_clear(struct host_its *its, uint32_t deviceid, uint32_t eventid);
> int its_send_cmd_mapti(struct host_its *its, uint32_t deviceid,
> @@ -254,6 +262,18 @@ int its_send_command(struct host_its *hw_its, const void *its_cmd);
>
> struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
> uint32_t vdevid);
> +/* GICv4 functions */
> +int gicv4_assign_guest_event(struct domain *d, paddr_t vdoorbell_address,
> + uint32_t vdevid, uint32_t eventid,
> + struct pending_irq *pirq);
> +int gicv4_its_vlpi_move(struct pending_irq *pirq, struct vcpu *vcpu);
> +#ifndef CONFIG_GICV4
> +#define event_is_forwarded_to_vcpu(dev, eventid) ((void)dev, (void)eventid, false)
> +#else
> +bool event_is_forwarded_to_vcpu(struct its_device *dev, uint32_t eventid);
> +void its_vpe_mask_db(struct its_vpe *vpe);
> +#endif
> +int gicv4_its_vlpi_unmap(struct pending_irq *pirq);
>
> /* ITS quirks handling. */
> uint64_t gicv3_its_get_cacheability(void);
> diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
> index f48eae60ad..722247ec60 100644
> --- a/xen/arch/arm/include/asm/gic_v4_its.h
> +++ b/xen/arch/arm/include/asm/gic_v4_its.h
> @@ -29,6 +29,26 @@
> #define GITS_CMD_VINVALL 0x2d
> #define GITS_CMD_INVDB 0x2e
>
> +/* Describes the mapping of a VLPI */
> +struct its_vlpi_map {
> + struct its_vm *vm;
> + unsigned int vpe_idx; /* Index of the VPE */
> + uint32_t vintid; /* Virtual LPI number */
> + bool db_enabled; /* Is the VPE doorbell to be generated? */
> + uint8_t properties;
> + struct pending_irq *pirq;
> + struct its_device *dev;
> + uint32_t eventid;
> +};
> +
> +struct event_vlpi_map {
> + unsigned int nr_lpis;
> + spinlock_t vlpi_lock;
> + struct its_vm *vm;
> + struct its_vlpi_map *vlpi_maps;
> + unsigned int nr_vlpis;
> +};
> +
> #endif
>
> /*
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 77323b2584..360f8a968e 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -70,6 +70,7 @@ struct pending_irq
> * LPI with the same number in an LR must be from an older LPI, which
> * has been unmapped before.
> *
> + * GIC_IRQ_GUEST_FORWARDED: the IRQ is forwarded to a VCPU(GICv4 only)
> */
> #define GIC_IRQ_GUEST_QUEUED 0
> #define GIC_IRQ_GUEST_ACTIVE 1
> @@ -77,6 +78,7 @@ struct pending_irq
> #define GIC_IRQ_GUEST_ENABLED 3
> #define GIC_IRQ_GUEST_MIGRATING 4
> #define GIC_IRQ_GUEST_PRISTINE_LPI 5
> +#define GIC_IRQ_GUEST_FORWARDED 6
> unsigned long status;
> struct irq_desc *desc; /* only set if the irq corresponds to a physical irq */
> unsigned int irq;
> @@ -95,6 +97,9 @@ struct pending_irq
> * vgic lock is not going to be enough. */
> struct list_head lr_queue;
> bool hw; /* Tied to HW IRQ */
> +#ifdef CONFIG_GICV4
> + struct its_vlpi_map *vlpi_map;
> +#endif
> };
>
> #define NR_INTERRUPT_PER_RANK 32
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 576e7fd4b0..94f7dd7d90 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -589,6 +589,14 @@ static int its_discard_event(struct virt_its *its,
> if ( vlpi == INVALID_LPI )
> return -ENOENT;
>
> + p = gicv3_its_get_event_pending_irq(its->d, its->doorbell_address,
> + vdevid, vevid);
> + if ( unlikely(!p) )
> + return -EINVAL;
> +
> + if ( pirq_is_tied_to_hw(p) )
> + if ( gicv4_its_vlpi_unmap(p) )
> + return -EINVAL;
> /*
> * TODO: This relies on the VCPU being correct in the ITS tables.
> * This can be fixed by either using a per-IRQ lock or by using
> @@ -751,6 +759,27 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
>
> vgic_init_pending_irq(pirq, intid, gic_is_gicv4());
>
> + pirq->lpi_vcpu_id = vcpu->vcpu_id;
pirq->lpi_vcpu_id = vcpu->vcpu_id; is assigned twice (here and again below).
Can we drop the earlier assignment and keep only the one after the
mapping/setup, unless something in between relies on it?
> +
> + if ( pirq_is_tied_to_hw(pirq) )
> + /*
> + * If on GICv4, we could let the VLPI being directly injected
> + * to the guest. To achieve that, the VLPI must be mapped using
> + * the VMAPTI command.
> + */
> + if ( gicv4_assign_guest_event(its->d, its->doorbell_address, devid,
> + eventid, pirq) )
> + goto out_remove_mapping;
1. looks like we should jump to "out_remove_host_entry" here, to roll back
the host mapping created by gicv3_assign_guest_event().
2. missing GICv4 rollback on error path: need gicv4_its_vlpi_unmap() before
removing host mapping
Best regards,
Mykola
> +
> + if ( pirq_is_tied_to_hw(pirq) )
> + set_bit(GIC_IRQ_GUEST_FORWARDED, &pirq->status);
> + else
> + /*
> + * Mark this LPI as new, so any older (now unmapped) LPI in any LR
> + * can be easily recognised as such.
> + */
> + set_bit(GIC_IRQ_GUEST_PRISTINE_LPI, &pirq->status);
> +
> /*
> * Now read the guest's property table to initialize our cached state.
> * We don't need the VGIC VCPU lock here, because the pending_irq isn't
> @@ -761,12 +790,6 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
> goto out_remove_host_entry;
>
> pirq->lpi_vcpu_id = vcpu->vcpu_id;
> - /*
> - * Mark this LPI as new, so any older (now unmapped) LPI in any LR
> - * can be easily recognised as such.
> - */
> - set_bit(GIC_IRQ_GUEST_PRISTINE_LPI, &pirq->status);
> -
> /*
> * Now insert the pending_irq into the domain's LPI tree, so that
> * it becomes live.
> @@ -824,6 +847,13 @@ static int its_handle_movi(struct virt_its *its, uint64_t *cmdptr)
> if ( unlikely(!p) )
> goto out_unlock;
>
> + if ( pirq_is_tied_to_hw(p) )
> + {
> + ret = gicv4_its_vlpi_move(p, nvcpu);
> + if ( ret )
> + goto out_unlock;
> + }
> +
> /*
> * TODO: This relies on the VCPU being correct in the ITS tables.
> * This can be fixed by either using a per-IRQ lock or by using
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 08/19] arm/gic: Keep track of GIC features
2026-02-02 16:14 ` [RFC PATCH 08/19] arm/gic: Keep track of GIC features Mykyta Poturai
@ 2026-02-12 7:00 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-12 7:00 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Different versions of GICv4 may support different features. Record them
> and provide functions to check for their availability.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3.c | 175 ++++++++++++++++++++-----
> xen/arch/arm/include/asm/gic.h | 2 +
> xen/arch/arm/include/asm/gic_v3_defs.h | 9 ++
> xen/arch/arm/include/asm/vgic.h | 9 ++
> 4 files changed, 162 insertions(+), 33 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 9b8b87078b..14852d18c2 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -100,6 +100,38 @@ static struct {
>
> static struct gic_info gicv3_info;
>
> +#ifdef CONFIG_GICV4
> +/* Global state */
> +static struct {
> + bool has_vlpis;
> + bool has_direct_lpi;
> + bool has_vpend_valid_dirty;
> + bool has_rvpeid;
> +} gicv4 = { .has_vlpis = true, .has_direct_lpi = true,
> + .has_vpend_valid_dirty = true, .has_rvpeid = true, };
> +
> +
> +bool gic_support_directLPI(void)
> +{
> + return gicv4.has_direct_lpi;
> +}
> +
> +bool gic_support_vptValidDirty(void)
> +{
> + return gicv4.has_vpend_valid_dirty;
> +}
> +
> +bool gic_has_v4_1_extension(void)
> +{
> + return gicv4.has_rvpeid;
> +}
> +
> +bool gic_is_gicv4(void)
> +{
> + return gicv4.has_vlpis;
gicv4.has_vlpis indicates vLPI/direct-injection support, not the GIC
architecture revision. A GICv4 implementation may have no vLPIs (only
pLPIs), so gic_is_gicv4() can return a false negative. If the intent
is to check the arch version, please use GICD_PIDR2/GICR_PIDR2.ArchRev
(e.g. ArchRev==4).
> +}
> +#endif
> +
> /* per-cpu re-distributor base */
> static DEFINE_PER_CPU(void __iomem*, rbase);
>
> @@ -914,7 +946,8 @@ static bool gicv3_enable_lpis(void)
> return true;
> }
>
> -static int __init gicv3_populate_rdist(void)
> +static int __init gic_iterate_rdists(int (*fn)(struct rdist_region *,
> + void __iomem *))
> {
> int i;
> uint32_t aff;
> @@ -958,40 +991,16 @@ static int __init gicv3_populate_rdist(void)
>
> if ( (typer >> 32) == aff )
> {
> + int ret;
> +
> this_cpu(rbase) = ptr;
>
> - if ( typer & GICR_TYPER_PLPIS )
> - {
> - paddr_t rdist_addr;
> - unsigned int procnum;
> - int ret;
> -
> - /*
> - * The ITS refers to redistributors either by their physical
> - * address or by their ID. Which one to use is an ITS
> - * choice. So determine those two values here (which we
> - * can do only here in GICv3 code) and tell the
> - * ITS code about it, so it can use them later to be able
> - * to address those redistributors accordingly.
> - */
> - rdist_addr = gicv3.rdist_regions[i].base;
> - rdist_addr += ptr - gicv3.rdist_regions[i].map_base;
> - procnum = (typer & GICR_TYPER_PROC_NUM_MASK);
> - procnum >>= GICR_TYPER_PROC_NUM_SHIFT;
> -
> - gicv3_set_redist_address(rdist_addr, procnum);
> -
> - ret = gicv3_lpi_init_rdist(ptr);
> - if ( ret && ret != -ENODEV )
> - {
> - printk("GICv3: CPU%d: Cannot initialize LPIs: %u\n",
> - smp_processor_id(), ret);
> - break;
> - }
> - }
nit: if we call the v4 capability update here we can avoid the extra
RD iteration and most of the diff
#ifdef CONFIG_GICV4
gicv4_update_lpi_properties()
#endif
> -
> - printk("GICv3: CPU%d: Found redistributor in region %d @%p\n",
> - smp_processor_id(), i, ptr);
> + ret = fn(gicv3.rdist_regions + i, ptr);
> + if ( ret )
> + return ret;
> +
> + printk("GICv3: CPU%d: Found redistributor @%p\n",
nit: maybe better to print correct version of GIC
the same applies to other prints
> + smp_processor_id(), ptr);
> return 0;
> }
>
> @@ -1010,11 +1019,107 @@ static int __init gicv3_populate_rdist(void)
> } while ( !(typer & GICR_TYPER_LAST) );
> }
>
> + return -ENODEV;
> +}
> +
> +static int __init __gicv3_populate_rdist(struct rdist_region *region,
> + void __iomem *ptr)
> +{
> + uint64_t typer;
> +
> + typer = readq_relaxed(ptr + GICR_TYPER);
> + if ( typer & GICR_TYPER_PLPIS )
> + {
> + paddr_t rdist_addr;
> + unsigned int procnum;
> + int ret;
> +
> + /*
> + * The ITS refers to redistributors either by their physical
> + * address or by their ID. Which one to use is an ITS
> + * choice. So determine those two values here (which we
> + * can do only here in GICv3 code) and tell the
> + * ITS code about it, so it can use them later to be able
> + * to address those redistributors accordingly.
> + */
> + rdist_addr = region->base;
> + rdist_addr += ptr - region->map_base;
> + procnum = (typer & GICR_TYPER_PROC_NUM_MASK);
> + procnum >>= GICR_TYPER_PROC_NUM_SHIFT;
> +
> + gicv3_set_redist_address(rdist_addr, procnum);
> +
> + ret = gicv3_lpi_init_rdist(ptr);
> + if ( ret && ret != -ENODEV )
> + {
> + printk("GICv3: CPU%d: Cannot initialize LPIs: %d\n",
> + smp_processor_id(), ret);
> + printk("%s %d\n", __func__, __LINE__);
> + return ret;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int __init gicv3_populate_rdist(void)
> +{
> + int ret = gic_iterate_rdists(__gicv3_populate_rdist);
> + if ( ret == 0)
> + return 0;
> +
> dprintk(XENLOG_ERR, "GICv3: CPU%d: mpidr 0x%"PRIregister" has no re-distributor!\n",
> smp_processor_id(), cpu_logical_map(smp_processor_id()));
> + return -ENODEV;
> +}
> +
> +#ifdef CONFIG_GICV4
> +static int __init __gicv4_update_vlpi_properties(struct rdist_region *region,
nit: the function updates more than vLPI properties (it also sets
has_direct_lpi),
so maybe rename it to include “lpi” rather than “vlpi”.
> + void __iomem *ptr)
> +{
> + uint64_t typer;
> +
> + typer = readq_relaxed(ptr + GICR_TYPER);
> + gicv4.has_vlpis &= !!(typer & GICR_TYPER_VLPIS);
Might be worth documenting that we treat these RD capabilities as global:
if any Redistributor lacks a feature, we disable it system-wide.
> + gicv4.has_rvpeid &= !!(typer & GICR_TYPER_RVPEID);
> + /* RVPEID implies some form of DirectLPI. */
> + gicv4.has_direct_lpi &= (!!(typer & GICR_TYPER_DirectLPIS) ||
It was not clear to me from the comment above why we use RVPEID here:
> + !!(typer & GICR_TYPER_RVPEID));
Looking at Arm IHI 0069H.b, the invalidate/sync registers are mandatory not
only when GICR_TYPER.Direct* is set, but also when GICR_CTLR.IR == 1, and
also whenever GICv4.1 is implemented. See 12.11.20 *GICR_INVLPIR*,
“Accessing the GICR_INVLPIR”.
RVPEID is a GICv4.1-only field, and it does not directly describe the
presence of the DirectLPI/invalidate registers. My understanding from the
GICR_VPENDBASER layout is that the v4.1 encoding uses a vPEID field, while
the v4.0 encoding uses a VPT physical address. If that interpretation is
correct, then RVPEID would always be set when GICv4.1 is present, which
makes the current check somewhat redundant (and it would be good to explain
that in the comment).
For reference, the GIC-700 / GIC-700AE TRMs describe RVPEID as indicating
whether GICR_VPENDBASER records the index (vPEID) into the vPE configuration
table (i.e. whether GICv4.1 support is present).
Given the above, I think it would be better to also include GICR_CTLR.IR
here (as the Linux kernel does), and add a short comment explaining the use
of RVPEID.
---
nit: we can just use gicv4.has_rvpeid here
> + gicv4.has_vpend_valid_dirty &= !!(typer & GICR_TYPER_DIRTY);
> +
> + /* Detect non-sensical configurations */
> + if ( gicv4.has_rvpeid && !gicv4.has_vlpis )
> + {
> + gicv4.has_direct_lpi = false;
> + gicv4.has_vlpis = false;
nit: 'gicv4.has_vlpis = false' is redundant here since the condition
already has '!gicv4.has_vlpis'
> + gicv4.has_rvpeid = false;
> + }
> +
> + printk("GICv4: CPU%d: %sVLPI support, %sdirect LPI support, %sValid+Dirty support, %sRVPEID support\n",
nit: Xen may run on GICv3 even with CONFIG_GICV4 code enabled; please
avoid hardcoding "GICv4"
> + smp_processor_id(), !!(typer & GICR_TYPER_VLPIS) ? "" : "no ",
> + (!!(typer & GICR_TYPER_DirectLPIS) ||
> + !!(typer & GICR_TYPER_RVPEID)) ? "" : "no ",
> + !!(typer & GICR_TYPER_DIRTY) ? "" : "no ",
> + !!(typer & GICR_TYPER_RVPEID) ? "" : "no ");
> +
> + return 0;
> +}
> +
> +static int __init gicv4_update_vlpi_properties(void)
> +{
> + int ret = gic_iterate_rdists(__gicv4_update_vlpi_properties);
> +
> + if ( ret == 0 )
> + return 0;
>
> return -ENODEV;
> }
> +#else
> +static int __init gicv4_update_vlpi_properties(void)
> +{
> + return 0;
> +}
> +#endif
>
> static int gicv3_cpu_init(void)
> {
> @@ -1024,6 +1129,10 @@ static int gicv3_cpu_init(void)
> if ( gicv3_populate_rdist() )
> return -ENODEV;
>
> + ret = gicv4_update_vlpi_properties();
> + if ( ret )
> + return ret;
> +
> if ( gicv3_enable_redist() )
> return -ENODEV;
>
> diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
> index 8e713aa477..afb1cc3751 100644
> --- a/xen/arch/arm/include/asm/gic.h
> +++ b/xen/arch/arm/include/asm/gic.h
> @@ -235,6 +235,8 @@ enum gic_version {
> GIC_INVALID = 0, /* the default until explicitly set up */
> GIC_V2,
> GIC_V3,
> + GIC_V4,
> + GIC_V4_1,
> };
>
> DECLARE_PER_CPU(uint64_t, lr_mask);
> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
> index c373b94d19..3a7d18ef59 100644
> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
> @@ -93,6 +93,12 @@
>
> #define GICD_TYPE_LPIS (1U << 17)
>
> +#define GICD_TYPER2 0x000c
> +
> +#define GICD_TYPER2_VIL (1U << 7)
> +#define GICD_TYPER2_VID GENMASK(4, 0)
> +#define GICD_TYPER2_nASSGIcap (1U << 8)
> +
nit: GICD_TYPER2 and related bit definitions are not used anywhere in this
series.
Best regards,
Mykola
> #define GICD_CTLR_RWP (1UL << 31)
> #define GICD_CTLR_ARE_NS (1U << 4)
> #define GICD_CTLR_ENABLE_G1A (1U << 1)
> @@ -149,7 +155,10 @@
>
> #define GICR_TYPER_PLPIS (1U << 0)
> #define GICR_TYPER_VLPIS (1U << 1)
> +#define GICR_TYPER_DIRTY (1U << 2)
> +#define GICR_TYPER_DirectLPIS (1U << 3)
> #define GICR_TYPER_LAST (1U << 4)
> +#define GICR_TYPER_RVPEID (1U << 7)
> #define GICR_TYPER_PROC_NUM_SHIFT 8
> #define GICR_TYPER_PROC_NUM_MASK (0xffff << GICR_TYPER_PROC_NUM_SHIFT)
>
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 360f8a968e..f12d736808 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -405,6 +405,15 @@ extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int ir
> extern void vgic_check_inflight_irqs_pending(struct vcpu *v,
> unsigned int rank, uint32_t r);
>
> +/* GICV4 functions */
> +#ifdef CONFIG_GICV4
> +bool gic_support_vptValidDirty(void);
> +bool gic_is_gicv4(void);
> +#else
> +#define gic_support_vptValidDirty() (false)
> +#define gic_is_gicv4() (false)
> +#endif
> +
> #endif /* !CONFIG_NEW_VGIC */
>
> /*** Common VGIC functions used by Xen arch code ****/
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 10/19] arm/its: Keep track of BASER regs
2026-02-02 16:14 ` [RFC PATCH 10/19] arm/its: Keep track of BASER regs Mykyta Poturai
@ 2026-02-12 7:52 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-12 7:52 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Keep all the info about BASER regs in host_its structure to be able to easily
> access it later.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3-its.c | 36 +++++++++++++++++++++------
> xen/arch/arm/include/asm/gic_v3_its.h | 14 +++++++++++
> 2 files changed, 43 insertions(+), 7 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index c628959f42..2328595a85 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -430,16 +430,32 @@ static void *its_map_cbaser(struct host_its *its)
> /* The ITS BASE registers work with page sizes of 4K, 16K or 64K. */
> #define BASER_PAGE_BITS(sz) ((sz) * 2 + 12)
>
> +struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
> + {
> + if (GITS_BASER_TYPE(hw_its->tables[i].val) == type)
> + return &hw_its->tables[i];
> + }
> +
> + return NULL;
> +}
> +
> static int its_map_baser(void __iomem *basereg, uint64_t regc,
> - unsigned int nr_items)
> + unsigned int nr_items, struct its_baser *baser)
> {
> uint64_t attr, reg;
> unsigned int entry_size = GITS_BASER_ENTRY_SIZE(regc);
> - unsigned int pagesz = 2; /* try 64K pages first, then go down. */
> + unsigned int page_size[4] = {SZ_4K, SZ_16K, SZ_64K, SZ_64K};
> + unsigned int pagesz = 0; /* try 64K pages first, then go down. */
pagesz=0 looks like test leftover
> unsigned int table_size;
> unsigned int order;
> void *buffer;
> + uint32_t type;
>
> + type = GITS_BASER_TYPE(regc);
nit: unused
> attr = gicv3_its_get_shareability() << GITS_BASER_SHAREABILITY_SHIFT;
> attr |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
> attr |= gicv3_its_get_cacheability() << GITS_BASER_INNER_CACHEABILITY_SHIFT;
> @@ -479,6 +495,11 @@ retry:
> writeq_relaxed(reg, basereg);
> regc = readq_relaxed(basereg);
>
> + baser->val = regc;
> + baser->base = buffer;
> + baser->table_size = table_size;
> + baser->pagesz = page_size[pagesz];
> +
> /* The host didn't like our attributes, just use what it returned. */
> if ( (regc & BASER_ATTR_MASK) != attr )
> {
> @@ -490,7 +511,7 @@ retry:
> }
> attr = regc & BASER_ATTR_MASK;
> }
> - if ( (regc & GITS_BASER_INNER_CACHEABILITY_MASK) <= GIC_BASER_CACHE_nC )
> + if ( gicv3_its_get_cacheability() <= GIC_BASER_CACHE_nC )
> clean_and_invalidate_dcache_va_range(buffer, table_size);
>
> /* If the host accepted our page size, we are done. */
> @@ -568,26 +589,27 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
> {
> void __iomem *basereg = hw_its->its_base + GITS_BASER0 + i * 8;
> unsigned int type;
> + struct its_baser *baser = hw_its->tables + i;
>
> reg = readq_relaxed(basereg);
> - type = (reg & GITS_BASER_TYPE_MASK) >> GITS_BASER_TYPE_SHIFT;
> + type = GITS_BASER_TYPE(reg);
> switch ( type )
> {
> case GITS_BASER_TYPE_NONE:
> continue;
> case GITS_BASER_TYPE_DEVICE:
> - ret = its_map_baser(basereg, reg, BIT(hw_its->devid_bits, UL));
> + ret = its_map_baser(basereg, reg, BIT(hw_its->devid_bits, UL), baser);
> if ( ret )
> return ret;
> break;
> case GITS_BASER_TYPE_COLLECTION:
> - ret = its_map_baser(basereg, reg, num_possible_cpus());
> + ret = its_map_baser(basereg, reg, num_possible_cpus(), baser);
> if ( ret )
> return ret;
> break;
> /* In case this is a GICv4, provide a (dummy) vPE table as well. */
> case GITS_BASER_TYPE_VCPU:
> - ret = its_map_baser(basereg, reg, 1);
> + ret = its_map_baser(basereg, reg, 32, baser);
Since MAX_VPEID is introduced later in the series, could you use it here
instead of the hardcoded 32 (or reorder the patches so it’s available)?
Also, if the BASER ends up indirect, the L1 sizing should be recalculated
from MAX_VPEID.
Best regards,
Mykola
> if ( ret )
> return ret;
> break;
> diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
> index 973ca6acdd..bd2696f354 100644
> --- a/xen/arch/arm/include/asm/gic_v3_its.h
> +++ b/xen/arch/arm/include/asm/gic_v3_its.h
> @@ -64,6 +64,7 @@
> #define GITS_BASER_INNER_CACHEABILITY_SHIFT 59
> #define GITS_BASER_TYPE_SHIFT 56
> #define GITS_BASER_TYPE_MASK (7ULL << GITS_BASER_TYPE_SHIFT)
> +#define GITS_BASER_TYPE(reg) ((reg & GITS_BASER_TYPE_MASK) >> GITS_BASER_TYPE_SHIFT)
> #define GITS_BASER_OUTER_CACHEABILITY_SHIFT 53
> #define GITS_BASER_TYPE_NONE 0UL
> #define GITS_BASER_TYPE_DEVICE 1UL
> @@ -143,6 +144,17 @@ struct its_device {
> #endif
> };
>
> +/*
> + * The ITS_BASER structure - contains memory information, cached
> + * value of BASER register configuration.
> + */
> +struct its_baser {
> + void *base;
> + uint64_t val;
> + unsigned int table_size;
> + unsigned int pagesz;
> +};
> +
> /* data structure for each hardware ITS */
> struct host_its {
> struct list_head entry;
> @@ -156,6 +168,7 @@ struct host_its {
> spinlock_t cmd_lock;
> void *cmd_buf;
> unsigned int flags;
> + struct its_baser tables[GITS_BASER_NR_REGS];
> };
>
> /* Map a collection for this host CPU to each host ITS. */
> @@ -259,6 +272,7 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
> uint32_t virt_lpi);
> void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> uint32_t virt_lpi);
> +struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type);
> void lpi_write_config(uint8_t *prop_table, uint32_t lpi, uint8_t clr,
> uint8_t set);
> int its_send_command(struct host_its *hw_its, const void *its_cmd);
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 00/19] GICv4 Support for Xen
2026-02-03 10:01 ` [RFC PATCH 00/19] GICv4 Support for Xen Bertrand Marquis
2026-02-03 12:24 ` Mykyta Poturai
@ 2026-02-13 11:36 ` Mykola Kvach
2026-02-13 11:48 ` Julien Grall
2026-03-17 9:11 ` Mykola Kvach
2 siblings, 1 reply; 42+ messages in thread
From: Mykola Kvach @ 2026-02-13 11:36 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Mykyta Poturai, xen-devel@lists.xenproject.org,
Stefano Stabellini, Julien Grall, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
Hi Mykyta,
Thank you for this patch series. I'll go through it and follow up with
comments shortly.
On Tue, Feb 3, 2026 at 12:02 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Mykyta,
>
> We have a number of series from you which have not been merged yet and
> reviewing them all in parallel might be challenging.
>
> Would you mind giving us a status and maybe priorities on them.
>
> I could list the following series:
> - GICv4
> - CPU Hotplug on arm
> - PCI enumeration on arm
> - IPMMU for pci on arm
> - dom0less for pci passthrough on arm
> - SR-IOV for pvh
> - SMMU for pci on arm
> - MSI injection on arm
> - suspend to ram on arm
>
> There might be others feel free to complete the list.
>
> On GICv4...
>
> > On 2 Feb 2026, at 17:14, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> >
> > This series introduces GICv4 direct LPI injection for Xen.
> >
> > Direct LPI injection relies on the GIC tracking the mapping between physical and
> > virtual CPUs. Each VCPU requires a VPE that is created and registered with the
> > GIC via the `VMAPP` ITS command. The GIC is then informed of the current
> > VPE-to-PCPU placement by programming `VPENDBASER` and `VPROPBASER` in the
> > appropriate redistributor. LPIs are associated with VPEs through the `VMAPTI`
> > ITS command, after which the GIC handles delivery without trapping into the
> > hypervisor for each interrupt.
> >
> > When a VPE is not scheduled but has pending interrupts, the GIC raises a per-VPE
> > doorbell LPI. Doorbells are owned by the hypervisor and prompt rescheduling so
> > the VPE can drain its pending LPIs.
> >
> > Because GICv4 lacks a native doorbell invalidation mechanism, this series
> > includes a helper that invalidates doorbell LPIs via synthetic “proxy” devices,
> > following the approach used until GICv4.1.
> >
> > All of this work is mostly based on the work of Penny Zheng
> > <penny.zheng@arm.com> and Luca Fancellu <luca.fancellu@arm.com>. And also from
> > Linux patches by Mark Zyngier.
> >
> > Some patches are still a little rough and need some styling fixes and more
> > testing, as all of them needed to be carved line by line from a giant ~4000 line
> > patch. This RFC is directed mostly to get a general idea if the proposed
> > approach is suitable and OK with everyone. And there is still an open question
> > of how to handle Signed-off-by lines for Penny and Luca, since they have not
> > indicated their preference yet.
>
> I would like to ask how much performance benefits you could
> have with this.
> Adding GICv4 support is adding a lot of code which will have to be maintained
> and tested and there should be a good improvement to justify this.
>
> Did you do some benchmarks ? what are the results ?
One more benchmarking note (and rationale): for meaningful performance
testing it may be necessary to disable WFI trapping (boot Xen with
`vwfi=native`).
If WFI is trapped, each guest idle instruction causes a VM-exit, and
Xen typically deschedules the vCPU. This makes the vCPU become
"non-resident" more often, so subsequent wakeups (e.g. vSGI/vLPI) tend
to go through a slower host-mediated path (waking the vCPU thread via
the scheduler and performing extra state transitions) instead of letting the
hardware wake and deliver to a running guest quickly.
For this reason it may be worth conditionally recommending (or even
auto-selecting) `vwfi=native` when direct injection is enabled for a
vCPU, so measurements reflect the actual delivery fast-path rather than
exit/scheduling overhead.
---
One more suggestion: it may be worth adding this as a small patch in the
series (or at least documenting it prominently). When direct injection
is enabled for a vCPU, trapping WFI can skew both behaviour and
benchmarks by pushing the vCPU into a "non-resident" state more often
and forcing wakeups to go through the host/scheduler path. A conditional
recommendation (or even auto-selecting `vwfi=native` in that mode) would
help keep the fast-path measurable and predictable.
Best regards,
Mykola
>
> At the time where we started to work on that at Arm, we ended up in the conclusion
> that the complexity in Xen compared to the benefit was not justifying it hence why
> this work was stopped in favor of other features that we thought would be more
> beneficial to Xen (like PCI passthrough or SMMUv3)>
> Cheers
> Bertrand
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 00/19] GICv4 Support for Xen
2026-02-13 11:36 ` Mykola Kvach
@ 2026-02-13 11:48 ` Julien Grall
2026-02-13 12:18 ` Mykola Kvach
0 siblings, 1 reply; 42+ messages in thread
From: Julien Grall @ 2026-02-13 11:48 UTC (permalink / raw)
To: Mykola Kvach, Bertrand Marquis
Cc: Mykyta Poturai, xen-devel@lists.xenproject.org,
Stefano Stabellini, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
Hi Mykola,
On 13/02/2026 11:36, Mykola Kvach wrote:
> For this reason it may be worth conditionally recommending (or even
> auto-selecting) `vwfi=native` when direct injection is enabled for a
> vCPU, so measurements reflect the actual delivery fast-path rather than
> exit/scheduling overhead.
I don't think this is a straightforward answer. "vwfi=native" is
beneficial when you have a single vCPU scheduled per pCPU. But if you
have multiple vCPUs running, then you may impair the overall performance
of the system as the scheduler will not be able to run another vCPU even
if the current vCPU is doing nothing (it is waiting for an interrupt).
As a data point, Xen didn't initially trapped WFI/WFE. But we noticed a
lot of slow down during boot if all the vCPUs for a guest were running
on the same pCPU. The difference was quite noticeable.
So instead of recommending to always set "vwfi=native", I would consider
an approach where Xen decides whether WFI/WFE is trapped based on the
number of vCPUs that can be scheduled on a given pCPU. This could be
adjusted on demand.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 00/19] GICv4 Support for Xen
2026-02-13 11:48 ` Julien Grall
@ 2026-02-13 12:18 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-13 12:18 UTC (permalink / raw)
To: Julien Grall
Cc: Bertrand Marquis, Mykyta Poturai, xen-devel@lists.xenproject.org,
Stefano Stabellini, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
Hi Julien,
On Fri, Feb 13, 2026 at 1:48 PM Julien Grall <julien@xen.org> wrote:
>
> Hi Mykola,
>
> On 13/02/2026 11:36, Mykola Kvach wrote:
> > For this reason it may be worth conditionally recommending (or even
> > auto-selecting) `vwfi=native` when direct injection is enabled for a
> > vCPU, so measurements reflect the actual delivery fast-path rather than
> > exit/scheduling overhead.
>
> I don't think this is a straightforward answer. "vwfi=native" is
> beneficial when you have a single vCPU scheduled per pCPU. But if you
> have multiple vCPUs running, then you may impair the overall performance
> of the system as the scheduler will not be able to run another vCPU even
> if the current vCPU is doing nothing (it is waiting for an interrupt).
>
> As a data point, Xen didn't initially trapped WFI/WFE. But we noticed a
> lot of slow down during boot if all the vCPUs for a guest were running
> on the same pCPU. The difference was quite noticeable.
>
> So instead of recommending to always set "vwfi=native", I would consider
> an approach where Xen decides whether WFI/WFE is trapped based on the
> number of vCPUs that can be scheduled on a given pCPU. This could be
> adjusted on demand.
Thanks for the clarification. I agree: recommending vwfi=native
unconditionally is not correct.
What I meant was specifically for benchmarking direct injection in a
1:1 vCPU:pCPU setup (or with vCPUs pinned), where trapping WFI/WFE adds
extra exits and can hide the fast-path benefit.
For general setups with oversubscription, vwfi=trap is the right
default, because it lets Xen schedule another runnable vCPU instead of
leaving a pCPU effectively idle while the guest sits in WFI.
I like your suggestion: make WFI/WFE trapping adaptive based on whether
the current pCPU has other runnable vCPUs.
Best regards,
Mykola
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 13/19] arm/gic: VPE scheduling
2026-02-02 16:14 ` [RFC PATCH 13/19] arm/gic: VPE scheduling Mykyta Poturai
@ 2026-02-18 11:30 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-18 11:30 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> GICv4 needs to know which VCPU is currently scheduled to be able to
> deliver VLPIs. Implement switching of VPEs on VCPU context switch by
> extending the existing save/restore mechanism used for GICv2 and GICv3.
>
> Scheduling a VPE is done by setting up the VPENDBASER and VPROPBASER
> registers to the appropriate tables for the currently running VCPU. When
> scheduling out, preserve the IDAI and PendingLast bits from VPENDBASER.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v2.c | 2 +-
> xen/arch/arm/gic-v3.c | 9 +-
> xen/arch/arm/gic-v4-its.c | 138 +++++++++++++++++++++++++
> xen/arch/arm/gic-vgic.c | 6 ++
> xen/arch/arm/include/asm/gic.h | 2 +-
> xen/arch/arm/include/asm/gic_v3_defs.h | 6 ++
> xen/arch/arm/include/asm/gic_v4_its.h | 9 ++
> xen/arch/arm/include/asm/vgic.h | 2 +
> 8 files changed, 170 insertions(+), 4 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 0cd41eac12..c16fa5d67e 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -181,7 +181,7 @@ static void gicv2_save_state(struct vcpu *v)
> writel_gich(0, GICH_HCR);
> }
>
> -static void gicv2_restore_state(const struct vcpu *v)
> +static void gicv2_restore_state(struct vcpu *v)
> {
> int i;
>
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index d4af332b0e..07736179db 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -133,7 +133,7 @@ bool gic_is_gicv4(void)
> #endif
>
> /* per-cpu re-distributor base */
> -static DEFINE_PER_CPU(void __iomem*, rbase);
> +DEFINE_PER_CPU(void __iomem*, rbase);
>
> #define GICD (gicv3.map_dbase)
> #define GICD_RDIST_BASE (this_cpu(rbase))
> @@ -475,13 +475,15 @@ static void gicv3_save_state(struct vcpu *v)
> * are now visible to the system register interface
> */
> dsb(sy);
> + if ( gic_is_gicv4() )
> + vgic_v4_put(v, false);
> gicv3_save_lrs(v);
> save_aprn_regs(&v->arch.gic);
> v->arch.gic.v3.vmcr = READ_SYSREG(ICH_VMCR_EL2);
> v->arch.gic.v3.sre_el1 = READ_SYSREG(ICC_SRE_EL1);
> }
>
> -static void gicv3_restore_state(const struct vcpu *v)
> +static void gicv3_restore_state(struct vcpu *v)
> {
> register_t val;
>
> @@ -510,6 +512,9 @@ static void gicv3_restore_state(const struct vcpu *v)
> restore_aprn_regs(&v->arch.gic);
> gicv3_restore_lrs(v);
>
> + if ( gic_is_gicv4() )
> + vgic_v4_load(v);
> +
> /*
> * Make sure all stores are visible the GIC
> */
> diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
> index fac3b44a94..6a550a65b2 100644
> --- a/xen/arch/arm/gic-v4-its.c
> +++ b/xen/arch/arm/gic-v4-its.c
> @@ -18,6 +18,7 @@
> * GNU General Public License for more details.
> */
>
> +#include <xen/delay.h>
> #include <xen/errno.h>
> #include <xen/sched.h>
> #include <xen/spinlock.h>
> @@ -44,6 +45,21 @@ void __init gicv4_its_vpeid_allocator_init(void)
> panic("Could not allocate VPEID bitmap space\n");
> }
>
> +static void __iomem *gic_data_rdist_vlpi_base(unsigned int cpu)
> +{
> + /*
> + * Each Redistributor defines two 64KB frames in the physical address map.
> + * In GICv4, there are two additional 64KB frames.
> + * The frames for each Redistributor must be contiguous and must be
> + * ordered as follows:
> + * 1. RD_base
> + * 2. SGI_base
> + * 3. VLPI_base
> + * 4. Reserved
> + */
> + return GICD_RDIST_BASE_CPU(cpu) + SZ_128K;
> +}
> +
> static int __init its_alloc_vpeid(struct its_vpe *vpe)
> {
> int id;
> @@ -571,3 +587,125 @@ int its_send_cmd_vinv(struct host_its *its, struct its_device *dev,
>
> return gicv3_its_wait_commands(its);
> }
> +
> +static uint64_t read_vpend_dirty_clean(void __iomem *vlpi_base,
> + unsigned int count)
> +{
> + uint64_t val;
> + bool clean;
> +
> + do {
> + val = gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER);
> + /* Poll GICR_VPENDBASER.Dirty until it reads 0. */
> + clean = !(val & GICR_VPENDBASER_Dirty);
> + if ( !clean )
> + {
> + count--;
> + cpu_relax();
> + udelay(1);
> + }
> + } while ( !clean && count );
> +
> + if ( !clean )
> + {
> + printk(XENLOG_WARNING "ITS virtual pending table not totally parsed\n");
> + val |= GICR_VPENDBASER_PendingLast;
> + }
> +
> + return val;
> +}
> +
> +/*
> + * When a vPE is made resident, the GIC starts parsing the virtual pending
> + * table to deliver pending interrupts. This takes place asynchronously,
> + * and can at times take a long while.
> + */
> +static void its_wait_vpt_parse_complete(void __iomem *vlpi_base)
> +{
> + if ( !gic_support_vptValidDirty() )
> + return;
> +
> + read_vpend_dirty_clean(vlpi_base, 500);
> +}
> +
> +static uint64_t its_clear_vpend_valid(void __iomem *vlpi_base, uint64_t clr,
> + uint64_t set)
> +{
> + unsigned int count = 1000000; /* 1s! */
The 1s timeout looks too large for a context‑switch path.
Consider a much smaller bound (or async handling) and warn on
timeout, rather than potentially stalling the scheduler.
> + uint64_t val;
> +
> + /*
> + * Clearing the Valid bit informs the Redistributor that a context
> + * switch is taking place.
> + */
> + val = gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER);
> + val &= ~GICR_VPENDBASER_Valid;
> + val &= ~clr;
> + val |= set;
> + gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
> +
> + return read_vpend_dirty_clean(vlpi_base, count);
> +}
> +
> +static void its_make_vpe_resident(struct its_vpe *vpe, unsigned int cpu)
> +{
> + void __iomem *vlpi_base = gic_data_rdist_vlpi_base(cpu);
> + uint64_t val;
> +
> + /* Switch in this VM's virtual property table. */
> + val = virt_to_maddr(vpe->its_vm->vproptable) & GENMASK(51, 12);
> + val |= gicv3_its_get_cacheability() << GICR_VPROPBASER_INNER_CACHEABILITY_SHIFT;
> + val |= gicv3_its_get_shareability() << GICR_VPROPBASER_SHAREABILITY_SHIFT;
> + val |= GIC_BASER_CACHE_SameAsInner << GICR_VPROPBASER_OUTER_CACHEABILITY_SHIFT;
Nit: cacheability/shareability are taken from a global ITS quirk.
If different ITSes can report different attributes, we would need
per‑ITS values here. Otherwise, a short note that Xen assumes
homogeneous ITS attributes would help.
> + val |= (HOST_LPIS_NRBITS - 1) & GICR_VPROPBASER_IDBITS_MASK;
> + gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
The VPROPBASER value is constant per VM. We could precompute
and store it in struct its_vm at init, then just write the
cached value here instead of recomputing each time.
> +
> + /* Switch in this VCPU's VPT. */
> + val = virt_to_maddr(vpe->vpendtable) & GENMASK(51, 16);
> + val |= gicv3_its_get_cacheability() << GICR_VPENDBASER_INNER_CACHEABILITY_SHIFT;
> + val |= gicv3_its_get_shareability() << GICR_VPENDBASER_SHAREABILITY_SHIFT;
> + val |= GIC_BASER_CACHE_SameAsInner << GICR_VPENDBASER_OUTER_CACHEABILITY_SHIFT;
> + /*
> + * When the GICR_VPENDBASER.Valid bit is written from 0 to 1,
> + * this bit is RES1.
> + */
> + val |= GICR_VPENDBASER_PendingLast;
> + val |= vpe->idai ? GICR_VPENDBASER_IDAI : 0;
> + val |= GICR_VPENDBASER_Valid;
> + gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
> +
> + its_wait_vpt_parse_complete(vlpi_base);
> +}
> +
> +static void its_make_vpe_non_resident(struct its_vpe *vpe, unsigned int cpu)
> +{
> + void __iomem *vlpi_base = gic_data_rdist_vlpi_base(cpu);
> + uint64_t val;
> +
> + val = its_clear_vpend_valid(vlpi_base, 0, 0);
> + vpe->idai = val & GICR_VPENDBASER_IDAI;
> + vpe->pending_last = val & GICR_VPENDBASER_PendingLast;
> +}
> +
> +void vgic_v4_load(struct vcpu *vcpu)
> +{
> + struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
> +
> +
> + if ( vpe->resident )
gicv3_restore_state() call vgic_v4_load when
gic_is_gicv4() is true, even if its_vpe is NULL.
The same story about vgic_v4_put.
> + return;
> +
> + its_make_vpe_resident(vpe, vcpu->processor);
> + vpe->resident = true;
> +}
> +
> +void vgic_v4_put(struct vcpu *vcpu, bool need_db)
nit: unused need_db
> +{
> + struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
> +
> + if ( !vpe->resident )
> + return;
> +
> + its_make_vpe_non_resident(vpe, vcpu->processor);
> + vpe->resident = false;
> +}
> diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
> index ea48c5375a..44db142dbd 100644
> --- a/xen/arch/arm/gic-vgic.c
> +++ b/xen/arch/arm/gic-vgic.c
> @@ -377,6 +377,12 @@ int vgic_vcpu_pending_irq(struct vcpu *v)
> }
> }
>
> +#ifdef CONFIG_GICV4
> + if ( gic_is_gicv4() )
> + if ( v->arch.vgic.its_vpe->pending_last )
Should we check its_vpe too?
> + rc = 1;
> +#endif
> +
> out:
> spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
> return rc;
> diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
> index afb1cc3751..04a20bdca5 100644
> --- a/xen/arch/arm/include/asm/gic.h
> +++ b/xen/arch/arm/include/asm/gic.h
> @@ -362,7 +362,7 @@ struct gic_hw_operations {
> /* Save GIC registers */
> void (*save_state)(struct vcpu *v);
> /* Restore GIC registers */
> - void (*restore_state)(const struct vcpu *v);
> + void (*restore_state)(struct vcpu *v);
> /* Dump GIC LR register information */
> void (*dump_state)(const struct vcpu *v);
>
> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
> index 3a7d18ef59..0db75309cf 100644
> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
> @@ -257,6 +257,12 @@ struct rdist_region {
> bool single_rdist;
> };
>
> +/* per-cpu re-distributor base */
> +DECLARE_PER_CPU(void __iomem*, rbase);
> +
> +#define GICD_RDIST_BASE (this_cpu(rbase))
looks like possible macro redefinition, see gic-v3.c
Best regards,
Mykola
> +#define GICD_RDIST_BASE_CPU(cpu) (per_cpu(rbase, cpu))
> +
> #endif /* __ASM_ARM_GIC_V3_DEFS_H__ */
>
> /*
> diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
> index ba81b25bde..37b6b92f0c 100644
> --- a/xen/arch/arm/include/asm/gic_v4_its.h
> +++ b/xen/arch/arm/include/asm/gic_v4_its.h
> @@ -56,6 +56,15 @@ void gicv4_its_vpeid_allocator_init(void);
> #define GICR_VPROPBASER 0x0070
> #define GICR_VPENDBASER 0x0078
>
> +#define GICR_VPROPBASER_OUTER_CACHEABILITY_SHIFT 56
> +#define GICR_VPROPBASER_SHAREABILITY_SHIFT 10
> +#define GICR_VPROPBASER_SHAREABILITY_MASK \
> + (3UL << GICR_VPROPBASER_SHAREABILITY_SHIFT)
> +#define GICR_VPROPBASER_INNER_CACHEABILITY_SHIFT 7
> +#define GICR_VPROPBASER_INNER_CACHEABILITY_MASK \
> + (7UL << GICR_VPROPBASER_INNER_CACHEABILITY_SHIFT)
> +#define GICR_VPROPBASER_IDBITS_MASK 0x1f
> +
> #define GICR_VPENDBASER_Dirty (1UL << 60)
> #define GICR_VPENDBASER_PendingLast (1UL << 61)
> #define GICR_VPENDBASER_IDAI (1UL << 62)
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 580310fec4..9ef667decb 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -417,6 +417,8 @@ bool gic_is_gicv4(void);
> int vgic_v4_its_vm_init(struct domain *d);
> void vgic_v4_free_its_vm(struct domain *d);
> int vgic_v4_its_vpe_init(struct vcpu *vcpu);
> +void vgic_v4_load(struct vcpu *vcpu);
> +void vgic_v4_put(struct vcpu *vcpu, bool need_db);
> #endif /* !CONFIG_NEW_VGIC */
>
> /*** Common VGIC functions used by Xen arch code ****/
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 12/19] arm/gic: Add VPENDBASER/VPROPBASER accessors
2026-02-02 16:14 ` [RFC PATCH 12/19] arm/gic: Add VPENDBASER/VPROPBASER accessors Mykyta Poturai
@ 2026-02-18 11:30 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-18 11:30 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Hi Mykyta,
Thank you for the patch!
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Implement accessors for GICv4 registers VPENDBASER and VPROPBASER.
> VPENDBASER access needs special handling to clear the Valid bit before
> writing a new value.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/include/asm/gic_v4_its.h | 38 +++++++++++++++++++++++++++
> 1 file changed, 38 insertions(+)
>
> diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
> index fb0ef37bbe..ba81b25bde 100644
> --- a/xen/arch/arm/include/asm/gic_v4_its.h
> +++ b/xen/arch/arm/include/asm/gic_v4_its.h
> @@ -17,6 +17,8 @@
> * along with this program; If not, see <http://www.gnu.org/licenses/>.
> */
>
> +#include <asm/arm64/io.h>
> +
> #ifndef __ASM_ARM_GICV4_ITS_H__
> #define __ASM_ARM_GICV4_ITS_H__
>
> @@ -50,6 +52,42 @@ struct event_vlpi_map {
> };
>
> void gicv4_its_vpeid_allocator_init(void);
> +
> +#define GICR_VPROPBASER 0x0070
> +#define GICR_VPENDBASER 0x0078
> +
> +#define GICR_VPENDBASER_Dirty (1UL << 60)
> +#define GICR_VPENDBASER_PendingLast (1UL << 61)
> +#define GICR_VPENDBASER_IDAI (1UL << 62)
> +#define GICR_VPENDBASER_Valid (1UL << 63)
> +
> +#define GICR_VPENDBASER_OUTER_CACHEABILITY_SHIFT 56
> +#define GICR_VPENDBASER_SHAREABILITY_SHIFT 10
> +#define GICR_VPENDBASER_INNER_CACHEABILITY_SHIFT 7
> +
> +#define gits_read_vpropbaser(c) readq_relaxed(c)
> +#define gits_write_vpropbaser(v, c) {writeq_relaxed(v, c);}
> +
> +/*
> + * GICR_VPENDBASER - the Valid bit must be cleared before changing
> + * anything else.
> + */
> +static inline void gits_write_vpendbaser(uint64_t val, void __iomem *addr)
> +{
> + uint64_t tmp;
> +
> + tmp = readq_relaxed(addr);
> + while ( tmp & GICR_VPENDBASER_Valid )
The loop clearing GICR_VPENDBASER_Valid has no timeout.
If the bit never clears, we can spin forever. Please add a bounded
retry (and warn on timeout), similar to other polling paths.
Best regards,
Mykola
> + {
> + tmp &= ~GICR_VPENDBASER_Valid;
> + writeq_relaxed(tmp, addr);
> + tmp = readq_relaxed(addr);
> + }
> +
> + writeq_relaxed(val, addr);
> +}
> +#define gits_read_vpendbaser(c) readq_relaxed(c)
> +
> #endif
>
> /*
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 11/19] arm/its: Add ITS VM and VPE allocation/teardown
2026-02-02 16:14 ` [RFC PATCH 11/19] arm/its: Add ITS VM and VPE allocation/teardown Mykyta Poturai
@ 2026-02-18 11:30 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-18 11:30 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Hi Mykyta,
Thank you for the patch!
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Do necessary allocations for GICv4 VLPI injection.
> When creating a domain allocate its_vm and property tables.
> For each VCPU allocate a VPe with a unique vpe id and separate pending table.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3-its.c | 157 ++++++++++++----
> xen/arch/arm/gic-v3-lpi.c | 61 +++++-
> xen/arch/arm/gic-v3.c | 18 ++
> xen/arch/arm/gic-v4-its.c | 259 ++++++++++++++++++++++++++
> xen/arch/arm/include/asm/gic_v3_its.h | 17 ++
> xen/arch/arm/include/asm/gic_v4_its.h | 1 +
> xen/arch/arm/include/asm/vgic.h | 3 +
> xen/arch/arm/vgic.c | 25 ++-
> 8 files changed, 496 insertions(+), 45 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 2328595a85..fb1d2709be 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -31,6 +31,8 @@
> LIST_HEAD(host_its_list);
>
>
> +unsigned int nvpeid = 16;
> +
> /*
> * It is unlikely that a platform implements ITSes with different quirks,
> * so assume they all share the same.
> @@ -228,7 +230,7 @@ int gicv3_its_wait_commands(struct host_its *hw_its)
> return -ETIMEDOUT;
> }
>
> -static uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
> +uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
> uint64_t reg)
> {
> reg &= ~GENMASK(51, 16);
> @@ -443,6 +445,54 @@ struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type)
> return NULL;
> }
>
> +bool its_alloc_table_entry(struct its_baser *baser, uint32_t id)
> +{
> + uint64_t reg = baser->val;
> + bool indirect = reg & GITS_BASER_INDIRECT;
> + unsigned int idx;
> + __le64 *table;
> + unsigned int entry_size = GITS_BASER_ENTRY_SIZE(reg);
> +
> + /* Don't allow id that exceeds single, flat table limit */
> + if ( !indirect )
> + return (id < (baser->table_size / entry_size));
> +
> + /* Compute 1st level table index & check if that exceeds table limit */
> + idx = id / (baser->pagesz / entry_size);
> + if ( idx >= (baser->pagesz / GITS_LVL1_ENTRY_SIZE) )
> + return false;
> +
> + table = baser->base;
> +
> + /* Allocate memory for 2nd level table */
> + if (!table[idx])
> + {
> + unsigned int page_size = baser->pagesz;
> + void *buffer;
> +
> + buffer = alloc_xenheap_pages(get_order_from_bytes(page_size),
> + gicv3_its_get_memflags());
> + if ( !buffer )
> + return -ENOMEM;
returns -ENOMEM from a bool function, so OOM looks like
success.
---
does not zero the new L2 table before publishing it.
See, Arm IHI 0069H.b, 5.2.1 The ITS tables:
Behavior is UNPREDICTABLE if:
- Memory that is used for the level 2 tables does not contain zeros at
the time of the new allocation for use by the ITS.
---
The L2 tables allocated here are never freed. If this is
intended as “allocate once for ITS lifetime”, please add a short
comment stating that, to avoid the appearance of a leak.
> +
> + /* Flush Lvl2 table to PoC if hw doesn't support coherency */
> + if ( gicv3_its_get_cacheability() <= GIC_BASER_CACHE_nC )
> + clean_and_invalidate_dcache_va_range(buffer, page_size);
> +
> + table[idx] = cpu_to_le64(virt_to_maddr(buffer) | GITS_VALID_BIT);
Before writing virt_to_maddr(buffer) into the L1 entry, should
we validate that the physical address is encodable?
> +
> + /* Flush Lvl1 entry to PoC if hw doesn't support coherency */
> + if ( gicv3_its_get_cacheability() <= GIC_BASER_CACHE_nC )
> + clean_and_invalidate_dcache_va_range(table + idx,
> + GITS_LVL1_ENTRY_SIZE);
> +
> + /* Ensure updated table contents are visible to ITS hardware */
> + dsb(sy);
> + }
> +
> + return true;
> +}
> +
> static int its_map_baser(void __iomem *basereg, uint64_t regc,
> unsigned int nr_items, struct its_baser *baser)
> {
> @@ -737,13 +787,75 @@ static int gicv3_its_map_host_events(struct host_its *its,
> return ret;
> }
>
> - /* TODO: Consider using INVALL here. Didn't work on the model, though. */
> + return 0;
> +}
> +
> +static bool its_alloc_device_table(struct host_its *hw_its, uint32_t dev_id)
> +{
> + struct its_baser *baser;
> +
> + baser = its_get_baser(hw_its, GITS_BASER_TYPE_DEVICE);
> + if ( !baser )
> + return false;
> +
> + return its_alloc_table_entry(baser, dev_id);
> +}
> +
> +struct its_device *its_create_device(struct host_its *hw_its,
> + uint32_t host_devid, uint64_t nr_events)
> +{
> + void *itt_addr = NULL;
> + struct its_device *dev = NULL;
> + int ret;
> +
> + /* Sanitise the provided hardware values against the host ITS. */
> + if ( host_devid >= BIT(hw_its->devid_bits, UL) )
> + return NULL;
> +
> + dev = xzalloc(struct its_device);
> + if ( !dev )
> + return NULL;
> +
> + /* An Interrupt Translation Table needs to be 256-byte aligned. */
> + dev->itt_order = get_order_from_bytes(nr_events * hw_its->itte_size);
> + itt_addr = alloc_xenheap_pages(dev->itt_order, gicv3_its_get_memflags());
> + if ( !itt_addr )
> + goto fail_dev;
> +
> + clean_and_invalidate_dcache_va_range(itt_addr,
> + nr_events * hw_its->itte_size);
clean_and_invalidate_dcache_va_range() is unconditional.
Should this be guarded by
gicv3_its_get_cacheability() <= GIC_BASER_CACHE_nC
as in other paths?
It avoids a redundant flush on coherent systems.
> +
>
> - ret = its_send_cmd_sync(its, 0);
> + if ( !its_alloc_device_table(hw_its, host_devid) )
> + goto fail_itt;
> +
> + ret = its_send_cmd_mapd(hw_its, host_devid, max(fls(nr_events - 1), 1U),
> + virt_to_maddr(itt_addr), true);
> if ( ret )
> - return ret;
> + goto fail_itt;
>
> - return gicv3_its_wait_commands(its);
> + dev->itt_addr = itt_addr;
> + dev->hw_its = hw_its;
> + dev->host_devid = host_devid;
> + dev->eventids = nr_events;
> +
> + return dev;
> +
> +fail_itt:
> + free_xenheap_pages(itt_addr, dev->itt_order);
> +fail_dev:
> + xfree(dev);
> +
> + return NULL;
> +}
> +
> +static void its_free_device(struct its_device *dev)
> +{
> + xfree(dev->host_lpi_blocks);
does not release host LPI blocks via
gicv3_free_host_lpi_block().
> + xfree(dev->itt_addr);
uses xfree() for itt_addr, but it comes from
alloc_xenheap_pages().
> + if ( dev->pend_irqs )
nit: xfree checks if the provided pointer is equal to NULL
> + xfree(dev->pend_irqs);
> + xfree(dev);
> }
>
> /*
> @@ -758,12 +870,10 @@ int gicv3_its_map_guest_device(struct domain *d,
> paddr_t guest_doorbell, uint32_t guest_devid,
> uint64_t nr_events, bool valid)
> {
> - void *itt_addr = NULL;
> struct host_its *hw_its;
> struct its_device *dev = NULL;
> struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
> int i, ret = -ENOENT; /* "i" must be signed to check for >= 0 below. */
> - unsigned int order;
>
> hw_its = gicv3_its_find_by_doorbell(host_doorbell);
> if ( !hw_its )
> @@ -823,23 +933,12 @@ int gicv3_its_map_guest_device(struct domain *d,
> if ( !valid )
> goto out_unlock;
>
> - ret = -ENOMEM;
> -
> - /* An Interrupt Translation Table needs to be 256-byte aligned. */
> - order = get_order_from_bytes(max(nr_events * hw_its->itte_size, 256UL));
> - itt_addr = alloc_xenheap_pages(order, gicv3_its_get_memflags());
> - if ( !itt_addr )
> - goto out_unlock;
> -
> - memset(itt_addr, 0, PAGE_SIZE << order);
> -
> - clean_and_invalidate_dcache_va_range(itt_addr,
> - nr_events * hw_its->itte_size);
> -
> - dev = xzalloc(struct its_device);
> + dev = its_create_device(hw_its, host_devid, nr_events);
> if ( !dev )
> goto out_unlock;
>
> + ret = -ENOMEM;
> +
> /*
> * Allocate the pending_irqs for each virtual LPI. They will be put
> * into the domain's radix tree upon the guest's MAPTI command.
> @@ -860,14 +959,6 @@ int gicv3_its_map_guest_device(struct domain *d,
> if ( !dev->host_lpi_blocks )
> goto out_unlock;
>
> - ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1),
> - virt_to_maddr(itt_addr), true);
> - if ( ret )
> - goto out_unlock;
> -
> - dev->itt_addr = itt_addr;
> - dev->itt_order = order;
> - dev->hw_its = hw_its;
> dev->guest_doorbell = guest_doorbell;
> dev->guest_devid = guest_devid;
> dev->host_devid = host_devid;
> @@ -920,13 +1011,7 @@ out_unlock:
>
> out:
> if ( dev )
> - {
> - xfree(dev->pend_irqs);
> - xfree(dev->host_lpi_blocks);
> - }
> - if ( itt_addr )
> - free_xenheap_pages(itt_addr, order);
> - xfree(dev);
> + its_free_device(dev);
>
> return ret;
> }
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index c029d5d7a4..3c2649b695 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -58,6 +58,7 @@ static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
>
> #define MAX_NR_HOST_LPIS (lpi_data.max_host_lpi_ids - LPI_OFFSET)
> #define HOST_LPIS_PER_PAGE (PAGE_SIZE / sizeof(union host_lpi))
> +uint32_t lpi_id_bits;
>
> static union host_lpi *gic_get_host_lpi(uint32_t plpi)
> {
> @@ -202,14 +203,11 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> write_u64_atomic(&hlpip->data, hlpi.data);
> }
>
> -static int gicv3_lpi_allocate_pendtable(unsigned int cpu)
> +struct page_info *lpi_allocate_pendtable(void)
> {
> void *pendtable;
> unsigned int order;
>
> - if ( per_cpu(lpi_redist, cpu).pending_table )
> - return -EBUSY;
> -
> /*
> * The pending table holds one bit per LPI and even covers bits for
> * interrupt IDs below 8192, so we allocate the full range.
> @@ -219,20 +217,34 @@ static int gicv3_lpi_allocate_pendtable(unsigned int cpu)
> order = get_order_from_bytes(max(lpi_data.max_host_lpi_ids / 8, (unsigned long)SZ_64K));
> pendtable = alloc_xenheap_pages(order, gicv3_its_get_memflags());
> if ( !pendtable )
> - return -ENOMEM;
> + return NULL;
>
> memset(pendtable, 0, PAGE_SIZE << order);
> /* Make sure the physical address can be encoded in the register. */
> if ( virt_to_maddr(pendtable) & ~GENMASK(51, 16) )
> {
> free_xenheap_pages(pendtable, order);
> - return -ERANGE;
> + return NULL;
> }
> clean_and_invalidate_dcache_va_range(pendtable,
> lpi_data.max_host_lpi_ids / 8);
>
> - per_cpu(lpi_redist, cpu).pending_table = pendtable;
> + return virt_to_page(pendtable);
> +}
> +
> +static int gicv3_lpi_allocate_pendtable(unsigned int cpu)
> +{
> + struct page_info *pendtable;
> +
> + if ( per_cpu(lpi_redist, cpu).pending_table )
> + return -EBUSY;
> +
> + pendtable = lpi_allocate_pendtable();
> + if ( !pendtable )
> + return -EINVAL;
>
> + per_cpu(lpi_redist, cpu).pending_table = page_to_virt(pendtable);
> +
> return 0;
> }
>
> @@ -274,6 +286,38 @@ static int gicv3_lpi_set_pendtable(void __iomem *rdist_base)
> return 0;
> }
>
> +void *lpi_allocate_proptable(void)
Since this helper is used to allocate the table programmed via
GICR_VPROPBASER (vLPI configuration table), it may be clearer to
name it lpi_allocate_vproptable() (and lpi_free_vproptable()) to avoid
confusion with the physical PROPBASER path.
> +{
> + void *table;
> + int order;
> +
> + /* The property table holds one byte per LPI. */
> + order = get_order_from_bytes(lpi_data.max_host_lpi_ids);
Do we really need to allocate one byte per *ID* here?
AFAIU for the (V)PROPBASER configuration table the entry for LPI N is
at (base + (N - 8192)), so the table needs only
(max_host_lpi_ids - 8192) bytes, not max_host_lpi_ids bytes.
Arm IHI 0069H.b, 5.1.1 "LPI Configuration tables" states:
For any LPI N, the location of the table entry is defined by
(base address + (N - 8192)).
Also see "GICv3 and GICv4 Software Overview" (DAI 0492B),
section "LPI Configuration table":
Size in bytes = 2^(GICR_PROPBASER.IDbits+1) – 8192
The "extra bytes" are relevant for the *pending* table semantics, not
for the configuration table. With the current allocation size, we also
init/flush only (max_host_lpi_ids - 8192) bytes, leaving a tail
uninitialized. Either allocate the adjusted size, or init/flush the
full allocated range.
---
nit: VPROPBASER is VM-wide (vLPI configuration is global to all vPEs in
a VM). It may be worth making the chosen IDbits/table size an explicit
per-VM value, i.e. based on programmed value from VPROPBASER.
> + table = alloc_xenheap_pages(order, gicv3_its_get_memflags());
> + if ( !table )
> + return NULL;
> +
> + /* Make sure the physical address can be encoded in the register. */
> + if ( (virt_to_maddr(table) & ~GENMASK(51, 12)) )
> + {
> + free_xenheap_pages(table, order);
> + return NULL;
> + }
> + memset(table, GIC_PRI_IRQ | LPI_PROP_RES1, MAX_NR_HOST_LPIS);
> + clean_and_invalidate_dcache_va_range(table, MAX_NR_HOST_LPIS);
> +
> + return table;
> +}
> +
> +void lpi_free_proptable(void *vproptable)
> +{
> + int order;
> +
> + /* The property table holds one byte per LPI. */
> + order = get_order_from_bytes(lpi_data.max_host_lpi_ids);
> + free_xenheap_pages(vproptable, order);
> +}
> +
> /*
> * Tell a redistributor about the (shared) property table, allocating one
> * if not already done.
> @@ -314,7 +358,8 @@ static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
> }
>
> /* Encode the number of bits needed, minus one */
> - reg |= fls(lpi_data.max_host_lpi_ids - 1) - 1;
> + lpi_id_bits = fls(lpi_data.max_host_lpi_ids - 1);
> + reg |= lpi_id_bits - 1;
>
> reg |= virt_to_maddr(lpi_data.lpi_property);
>
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 14852d18c2..d4af332b0e 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -2083,6 +2083,22 @@ static bool gic_dist_supports_lpis(void)
> return (readl_relaxed(GICD + GICD_TYPER) & GICD_TYPE_LPIS);
> }
>
> +#ifdef CONFIG_GICV4
> +static void __init gicv4_init(void)
> +{
> + gicv3_info.hw_version = GIC_V4;
We should derive the HW version from GICD/GICR/GITS_PIDR2 in the
common GICv3/4 init path instead of hard‑coding GIC_V4 based on config.
That avoids config‑dependent values and keeps version detection
consistent.
> +
> +
> + gicv4_its_vpeid_allocator_init();
> +
> +}
> +#else
> +static void __init gicv4_init(void)
> +{
> + ASSERT_UNREACHABLE();
> +}
> +#endif
> +
> /* Set up the GIC */
> static int __init gicv3_init(void)
> {
> @@ -2157,6 +2173,8 @@ static int __init gicv3_init(void)
>
> gicv3_hyp_init();
>
> + if ( gic_is_gicv4() )
> + gicv4_init();
> out:
> spin_unlock(&gicv3.lock);
>
> diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
> index 358d0bffb9..fac3b44a94 100644
> --- a/xen/arch/arm/gic-v4-its.c
> +++ b/xen/arch/arm/gic-v4-its.c
> @@ -27,6 +27,83 @@
> #include <asm/vgic.h>
>
>
> +/*
> + * VPE ID is at most 16 bits.
> + * Using a bitmap here limits us to 65536 concurrent VPEs.
> + */
> +static unsigned long *vpeid_mask;
> +
> +static spinlock_t vpeid_alloc_lock = SPIN_LOCK_UNLOCKED;
> +
> +void __init gicv4_its_vpeid_allocator_init(void)
> +{
> + /* Allocate space for vpeid_mask based on MAX_VPEID */
> + vpeid_mask = xzalloc_array(unsigned long, BITS_TO_LONGS(MAX_VPEID));
> +
> + if ( !vpeid_mask )
> + panic("Could not allocate VPEID bitmap space\n");
> +}
> +
> +static int __init its_alloc_vpeid(struct its_vpe *vpe)
Several helpers in gic-v4-its.c are marked __init (e.g.
its_alloc_vpeid(), its_free_vpeid(), its_vpe_init(),
its_vpe_teardown()), but they are called from vcpu_vgic_init() via
vgic_v4_its_vpe_init(). This is not init-only code, and may cause
section mismatch warnings or runtime issues once init sections are
freed. Please drop __init for these functions.
---
vpe is unused
> +{
> + int id;
> +
> + spin_lock(&vpeid_alloc_lock);
> +
> + id = find_first_zero_bit(vpeid_mask, MAX_VPEID);
> +
> + if ( id == MAX_VPEID )
> + {
> + id = -EBUSY;
> + printk(XENLOG_ERR "VPEID pool exhausted\n");
> + goto out;
> + }
> +
> + set_bit(id, vpeid_mask);
> +
> +out:
> + spin_unlock(&vpeid_alloc_lock);
> +
> + return id;
> +}
> +
> +static void __init its_free_vpeid(uint32_t vpe_id)
> +{
> + spin_lock(&vpeid_alloc_lock);
> +
> + clear_bit(vpe_id, vpeid_mask);
> +
> + spin_unlock(&vpeid_alloc_lock);
> +}
> +
> +static bool __init its_alloc_vpe_entry(uint32_t vpe_id)
> +{
> + struct host_its *hw_its;
> +
> + /*
> + * Make sure the L2 tables are allocated on *all* v4 ITSs. We
> + * could try and only do it on ITSs corresponding to devices
> + * that have interrupts targeted at this VPE, but the
> + * complexity becomes crazy.
> + */
> + list_for_each_entry(hw_its, &host_its_list, entry)
> + {
> + struct its_baser *baser;
> +
> + if ( !hw_its->is_v4 )
> + continue;
> +
> + baser = its_get_baser(hw_its, GITS_BASER_TYPE_VCPU);
> + if ( !baser )
> + return false;
> +
> + if ( !its_alloc_table_entry(baser, vpe_id) )
> + return false;
> + }
> +
> + return true;
> +}
> +
> static int its_send_cmd_vsync(struct host_its *its, uint16_t vpeid)
> {
> uint64_t cmd[4];
> @@ -39,6 +116,188 @@ static int its_send_cmd_vsync(struct host_its *its, uint16_t vpeid)
> return its_send_command(its, cmd);
> }
>
> +static int its_send_cmd_vmapp(struct host_its *its, struct its_vpe *vpe,
> + bool valid)
> +{
> + uint64_t cmd[4];
> + uint16_t vpeid = vpe->vpe_id;
> + uint64_t vpt_addr;
> + int ret;
> +
> + cmd[0] = GITS_CMD_VMAPP;
> + cmd[1] = (uint64_t)vpeid << 32;
> + cmd[2] = valid ? GITS_VALID_BIT : 0;
> +
> + /* Unmap command */
> + if ( !valid )
> + goto out;
> +
> + /* Target redistributor */
> + cmd[2] |= encode_rdbase(its, vpe->col_idx, 0x0);
> + vpt_addr = virt_to_maddr(vpe->vpendtable);
> + cmd[3] = (vpt_addr & GENMASK(51, 16)) |
> + ((HOST_LPIS_NRBITS - 1) & GENMASK(4, 0));
> +
> + out:
> + ret = its_send_command(its, cmd);
> +
> + return ret;
> +}
> +
> +static int its_send_cmd_vinvall(struct host_its *its, struct its_vpe *vpe)
> +{
> + uint64_t cmd[4];
> + uint16_t vpeid = vpe->vpe_id;
> +
> + cmd[0] = GITS_CMD_VINVALL;
> + cmd[1] = (uint64_t)vpeid << 32;
> + cmd[2] = 0x00;
> + cmd[3] = 0x00;
> +
> + return its_send_command(its, cmd);
> +}
> +
> +static int its_map_vpe(struct host_its *its, struct its_vpe *vpe)
> +{
> + int ret;
> +
> + /*
> + * VMAPP command maps the vPE to the target RDbase, including an
> + * associated virtual LPI Pending table.
> + */
> + ret = its_send_cmd_vmapp(its, vpe, true);
> + if ( ret )
> + return ret;
> +
> + ret = its_send_cmd_vinvall(its, vpe);
> + if ( ret )
> + return ret;
> +
> + ret = its_send_cmd_vsync(its, vpe->vpe_id);
> + if ( ret )
> + return ret;
Error handling for ITS commands is minimal. If VMAPP succeeds and
VINVALL/VSYNC fails, we return without rollback or recovery.
Should we consider retrying (e.g. via CWRITER.Retry), or detect
error/stall via GITS_TYPER.SEIS / GITS_CREADR.Stalled and
unmap the VPE on failure?
> +
> + return 0;
> +}
> +static int __init its_vpe_init(struct its_vpe *vpe)
> +{
> + int vpe_id, rc = -ENOMEM;
> + struct page_info *vpendtable;
> + struct host_its *hw_its;
> +
> + /* Allocate vpe id */
> + vpe_id = its_alloc_vpeid(vpe);
> + if ( vpe_id < 0 )
> + return rc;
> +
> + /* Allocate VPT */
> + vpendtable = lpi_allocate_pendtable();
> +
> + if ( !vpendtable )
> + goto fail_vpt;
> +
> + if ( !its_alloc_vpe_entry(vpe_id) )
> + goto fail_entry;
> +
> + rwlock_init(&vpe->lock);
> + vpe->vpe_id = vpe_id;
> + vpe->vpendtable = page_to_virt(vpendtable);
> + /*
> + * We eagerly inform all the v4 ITS and map vPE to the first
> + * possible CPU
> + */
> + vpe->col_idx = cpumask_first(&cpu_online_map);
> + list_for_each_entry(hw_its, &host_its_list, entry)
> + {
> + if ( !hw_its->is_v4 )
> + continue;
> +
> + if ( its_map_vpe(hw_its, vpe) )
If its_map_vpe() fails for a later ITS, we should unmap the
VPE on the ITSes already handled. its_vpe_teardown() frees
SW allocations (vpe too) but does not undo prior VMAPPs,
leaving stale mappings behind.
> + goto fail_entry;
> + }
> +
> + return 0;
> +
> + fail_entry:
> + xfree(page_to_virt(vpendtable));
failure path frees vpendtable with xfree(),
but it was allocated with alloc_xenheap_pages().
---
The fail path frees vpendtable locally, and the caller then
calls its_vpe_teardown() on error. This risks double‑free,
Please have a single owner for cleanup.
> + fail_vpt:
> + its_free_vpeid(vpe_id);
> +
> + return rc;
> +}
> +
> +static void __init its_vpe_teardown(struct its_vpe *vpe)
> +{
> + unsigned int order;
> +
> + order = get_order_from_bytes(max(lpi_data.max_host_lpi_ids / 8, (unsigned long)SZ_64K));
> + its_free_vpeid(vpe->vpe_id);
> + free_xenheap_pages(vpe->vpendtable, order);
> + xfree(vpe);
> +}
> +
> +int vgic_v4_its_vm_init(struct domain *d)
> +{
> + unsigned int nr_vcpus = d->max_vcpus;
> + int ret = -ENOMEM;
> +
> + if ( !gicv3_its_host_has_its() )
> + return 0;
> +
> + d->arch.vgic.its_vm = xzalloc(struct its_vm);
> + if ( !d->arch.vgic.its_vm )
> + return ret;
> +
> + d->arch.vgic.its_vm->vpes = xzalloc_array(struct its_vpe *, nr_vcpus);
> + if ( !d->arch.vgic.its_vm->vpes )
> + goto fail_vpes;
> + d->arch.vgic.its_vm->nr_vpes = nr_vcpus;
> +
> + d->arch.vgic.its_vm->vproptable = lpi_allocate_proptable();
> + if ( !d->arch.vgic.its_vm->vproptable )
> + goto fail_vprop;
> +
> + return 0;
> +
> +fail_vprop:
> + xfree(d->arch.vgic.its_vm->vpes)
> + fail_vpes:
> + xfree(d->arch.vgic.its_vm);
use XFREE to cleanup the its_vm ptr too
> +
> + return ret;
> +}
> +
> +void vgic_v4_free_its_vm(struct domain *d)
> +{
> + struct its_vm *its_vm = d->arch.vgic.its_vm;
> + if ( its_vm->vpes )
nit: no need to check the ptr it is done by xfree
> + xfree(its_vm->vpes);
> + if ( its_vm->vproptable )
nit: looks like it is safe to pass NULL ptr to lpi_free_proptable
> + lpi_free_proptable(its_vm);
passes its_vm to lpi_free_proptable(), not
its_vm->vproptable.
> +}
> +
> +int vgic_v4_its_vpe_init(struct vcpu *vcpu)
> +{
> + int ret;
> + struct its_vm *its_vm = vcpu->domain->arch.vgic.its_vm;
> + unsigned int vcpuid = vcpu->vcpu_id;
> +
> + vcpu->arch.vgic.its_vpe = xzalloc(struct its_vpe);
> + if ( !vcpu->arch.vgic.its_vpe )
> + return -ENOMEM;
> +
> + its_vm->vpes[vcpuid] = vcpu->arch.vgic.its_vpe;
its_vm->vpes[vcpuid] is assigned before its_vpe_init(). If
its_vpe_init() fails, the array keeps a stale pointer. Either
move the assignment after success, or clear it on failure.
> + vcpu->arch.vgic.its_vpe->its_vm = its_vm;
> +
> + ret = its_vpe_init(vcpu->arch.vgic.its_vpe);
> + if ( ret )
> + {
> + its_vpe_teardown(vcpu->arch.vgic.its_vpe);
> + return ret;
> + }
> + return 0;
> +}
> +
> static int its_send_cmd_vmapti(struct host_its *its, struct its_device *dev,
> uint32_t eventid)
> {
> diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
> index bd2696f354..411beb81c8 100644
> --- a/xen/arch/arm/include/asm/gic_v3_its.h
> +++ b/xen/arch/arm/include/asm/gic_v3_its.h
> @@ -77,6 +77,7 @@
> #define GITS_BASER_ENTRY_SIZE_SHIFT 48
> #define GITS_BASER_ENTRY_SIZE(reg) \
> ((((reg) >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
> +#define GITS_LVL1_ENTRY_SIZE 8UL
> #define GITS_BASER_SHAREABILITY_SHIFT 10
> #define GITS_BASER_PAGE_SIZE_SHIFT 8
> #define GITS_BASER_SIZE_MASK 0xff
> @@ -117,9 +118,19 @@
> /* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
> #define LPI_BLOCK 32U
>
> +extern unsigned int nvpeid;
> +/* The maximum number of VPEID bits supported by VLPI commands */
> +#define ITS_MAX_VPEID_BITS nvpeid
> +#define MAX_VPEID (1UL << ITS_MAX_VPEID_BITS)
> +
> #ifdef CONFIG_GICV4
> #include <asm/gic_v4_its.h>
> #endif
> +
> +extern uint32_t lpi_id_bits;
> +#define HOST_LPIS_NRBITS lpi_id_bits
> +#define MAX_HOST_LPIS BIT(lpi_id_bits, UL)
> +
> /*
> * Describes a device which is using the ITS and is used by a guest.
> * Since device IDs are per ITS (in contrast to vLPIs, which are per
> @@ -169,6 +180,7 @@ struct host_its {
> void *cmd_buf;
> unsigned int flags;
> struct its_baser tables[GITS_BASER_NR_REGS];
> + bool is_v4;
> };
>
> /* Map a collection for this host CPU to each host ITS. */
> @@ -273,8 +285,13 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
> void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> uint32_t virt_lpi);
> struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type);
> +bool its_alloc_table_entry(struct its_baser *baser, uint32_t id);
> +struct page_info *lpi_allocate_pendtable(void);
> +void *lpi_allocate_proptable(void);
> +void lpi_free_proptable(void *vproptable);
> void lpi_write_config(uint8_t *prop_table, uint32_t lpi, uint8_t clr,
> uint8_t set);
> +uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu, uint64_t reg);
> int its_send_command(struct host_its *hw_its, const void *its_cmd);
>
> struct its_device *get_its_device(struct domain *d, paddr_t vdoorbell,
> diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
> index 722247ec60..fb0ef37bbe 100644
> --- a/xen/arch/arm/include/asm/gic_v4_its.h
> +++ b/xen/arch/arm/include/asm/gic_v4_its.h
> @@ -49,6 +49,7 @@ struct event_vlpi_map {
> unsigned int nr_vlpis;
> };
>
> +void gicv4_its_vpeid_allocator_init(void);
> #endif
>
> /*
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index f12d736808..580310fec4 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -414,6 +414,9 @@ bool gic_is_gicv4(void);
> #define gic_is_gicv4() (false)
> #endif
>
> +int vgic_v4_its_vm_init(struct domain *d);
> +void vgic_v4_free_its_vm(struct domain *d);
> +int vgic_v4_its_vpe_init(struct vcpu *vcpu);
> #endif /* !CONFIG_NEW_VGIC */
>
> /*** Common VGIC functions used by Xen arch code ****/
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 0da8c1a425..6baf870ad5 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -22,6 +22,7 @@
>
> #include <asm/mmio.h>
> #include <asm/gic.h>
> +#include <asm/gic_v3_its.h>
> #include <asm/vgic.h>
>
>
> @@ -329,6 +330,15 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> for ( i = 0; i < NR_GIC_SGI; i++ )
> set_bit(i, d->arch.vgic.allocated_irqs);
>
> + if ( gic_is_gicv4() )
> + {
> + ret = vgic_v4_its_vm_init(d);
> + if ( ret )
> + {
> + printk(XENLOG_ERR "GICv4 its vm allocation failed\n");
> + return ret;
> + }
> + }
> return 0;
> }
>
> @@ -366,11 +376,14 @@ void domain_vgic_free(struct domain *d)
> #endif
> xfree(d->arch.vgic.pending_irqs);
> xfree(d->arch.vgic.allocated_irqs);
> +
> + if ( gic_is_gicv4() )
> + vgic_v4_free_its_vm(d);
vgic_v4_free_its_vm() on gic_is_gicv4() even when no
ITS, so its_vm can be NULL.
Best regards,
Mykola
> }
>
> int vcpu_vgic_init(struct vcpu *v)
> {
> - int i;
> + int i, ret;
>
> v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
> if ( v->arch.vgic.private_irqs == NULL )
> @@ -389,6 +402,16 @@ int vcpu_vgic_init(struct vcpu *v)
> INIT_LIST_HEAD(&v->arch.vgic.lr_pending);
> spin_lock_init(&v->arch.vgic.lock);
>
> + if ( gic_is_gicv4() && gicv3_its_host_has_its())
> + {
> + ret = vgic_v4_its_vpe_init(v);
> + if ( ret )
> + {
> + printk(XENLOG_ERR "GICv4 its vpe allocation failed\n");
> + return ret;
> + }
> + }
> +
> return 0;
> }
>
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 14/19] arm/its: VPE affinity changes
2026-02-02 16:14 ` [RFC PATCH 14/19] arm/its: VPE affinity changes Mykyta Poturai
@ 2026-02-18 21:00 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-18 21:00 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> When a VCPU is migrated to another PCPU, its VPE affinity must be
> updated. Hook into VPE scheduling to ensure that the VPE to be scheduled
> is located on the correct PCPU, if not, move it with VMOVP command.
>
> VMOVP needs to be issued on all ITSes in the system, and in the same
> order, unlsess single VMOVP capable ITS is used.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3-its.c | 55 ++++++++++++++
> xen/arch/arm/gic-v4-its.c | 105 ++++++++++++++++++++++++++
> xen/arch/arm/include/asm/gic_v3_its.h | 12 +++
> 3 files changed, 172 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index fb1d2709be..be840fbc8f 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -31,6 +31,8 @@
> LIST_HEAD(host_its_list);
>
>
> +unsigned long its_list_map;
> +
> unsigned int nvpeid = 16;
>
> /*
> @@ -612,10 +614,47 @@ static int gicv3_disable_its(struct host_its *hw_its)
> return -ETIMEDOUT;
> }
>
> +static int __init its_compute_its_list_map(struct host_its *hw_its)
> +{
> + int its_number;
> + uint32_t ctlr;
> +
> + its_number = find_first_zero_bit(&its_list_map, GICv4_ITS_LIST_MAX);
> + if ( its_number >= GICv4_ITS_LIST_MAX )
> + {
> + printk(XENLOG_ERR
> + "ITS@%lx: No ITSList entry available!\n", hw_its->addr);
> + return -EINVAL;
> + }
> +
> + ctlr = readl_relaxed(hw_its->its_base + GITS_CTLR);
> + ctlr &= ~GITS_CTLR_ITS_NUMBER;
> + ctlr |= its_number << GITS_CTLR_ITS_NUMBER_SHIFT;
> + writel_relaxed(ctlr, hw_its->its_base + GITS_CTLR);
> + ctlr = readl_relaxed(hw_its->its_base + GITS_CTLR);
> + if ( (ctlr & GITS_CTLR_ITS_NUMBER) !=
> + (its_number << GITS_CTLR_ITS_NUMBER_SHIFT) )
> + {
> + its_number = ctlr & GITS_CTLR_ITS_NUMBER;
> + its_number >>= GITS_CTLR_ITS_NUMBER_SHIFT;
> + }
> +
> + if ( test_and_set_bit(its_number, &its_list_map) )
> + {
> + printk(XENLOG_ERR
> + "ITS@%lx: Duplicate ITSList entry %d\n",
> + hw_its->addr, its_number);
> + return -EINVAL;
> + }
> +
> + return its_number;
> +}
> +
> static int gicv3_its_init_single_its(struct host_its *hw_its)
> {
> uint64_t reg;
> int i, ret;
> + int its_number;
>
> hw_its->its_base = ioremap_nocache(hw_its->addr, hw_its->size);
> if ( !hw_its->its_base )
> @@ -633,6 +672,22 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
> hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
> if ( reg & GITS_TYPER_PTA )
> hw_its->flags |= HOST_ITS_USES_PTA;
> + hw_its->is_v4 = reg & GITS_TYPER_VLPIS;
hw_its->is_v4 = reg & GITS_TYPER_VLPIS only tells us that VLPI
is supported, not the GIC architecture revision. The name is_v4
is misleading. Consider renaming to has_vlpis (or similar) and
use GITS_PIDR2 when you need the GIC arch revision.
> + if ( hw_its->is_v4 )
> + {
> + if ( !(reg & GITS_TYPER_VMOVP) )
> + {
> + its_number = its_compute_its_list_map(hw_its);
> + if ( its_number < 0 )
> + return its_number;
> + dprintk(XENLOG_INFO,
> + "ITS@%lx: Using ITS number %d\n",
> + hw_its->addr, its_number);
> + }
> + else
> + dprintk(XENLOG_INFO,
> + "ITS@%lx: Single VMOVP capable\n", hw_its->addr);
> + }
> spin_lock_init(&hw_its->cmd_lock);
>
> for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
> diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
> index 6a550a65b2..175fda7acb 100644
> --- a/xen/arch/arm/gic-v4-its.c
> +++ b/xen/arch/arm/gic-v4-its.c
> @@ -36,6 +36,9 @@ static unsigned long *vpeid_mask;
>
> static spinlock_t vpeid_alloc_lock = SPIN_LOCK_UNLOCKED;
>
> +static uint16_t vmovp_seq_num;
> +static spinlock_t vmovp_lock = SPIN_LOCK_UNLOCKED;
> +
> void __init gicv4_its_vpeid_allocator_init(void)
> {
> /* Allocate space for vpeid_mask based on MAX_VPEID */
> @@ -242,6 +245,57 @@ static int __init its_vpe_init(struct its_vpe *vpe)
> return rc;
> }
>
> +static int its_send_cmd_vmovp(struct its_vpe *vpe)
> +{
> + uint16_t vpeid = vpe->vpe_id;
> + int ret;
> + struct host_its *hw_its;
> +
> + if ( !its_list_map )
> + {
> + uint64_t cmd[4];
> +
> + hw_its = list_first_entry(&host_its_list, struct host_its, entry);
> + cmd[0] = GITS_CMD_VMOVP;
> + cmd[1] = (uint64_t)vpeid << 32;
> + cmd[2] = encode_rdbase(hw_its, vpe->col_idx, 0x0);
> + cmd[3] = 0x00;
> +
> + return its_send_command(hw_its, cmd);
Docs says VMOVP should be followed by VSYNC to synchronize the
context when moving a vPE across CommonLPIAff groups. Here we
issue VMOVP and return without VSYNC. Should we add a VSYNC (for
both the single‑ITS and ITSList paths)?
> + }
> +
> + /*
> + * If using the its_list "feature", we need to make sure that all ITSs
> + * receive all VMOVP commands in the same order. The only way
> + * to guarantee this is to make vmovp a serialization point.
> + */
> + spin_lock(&vmovp_lock);
> +
> + vmovp_seq_num++;
> +
> + /* Emit VMOVPs */
> + list_for_each_entry(hw_its, &host_its_list, entry)
> + {
> + uint64_t cmd[4];
> +
> + cmd[0] = GITS_CMD_VMOVP | ((uint64_t)vmovp_seq_num << 32);
> + cmd[1] = its_list_map | ((uint64_t)vpeid << 32);
> + cmd[2] = encode_rdbase(hw_its, vpe->col_idx, 0x0);
> + cmd[3] = 0x00;
> +
> + ret = its_send_command(hw_its, cmd);
We iterate over all ITSes and send VMOVP to each. Should this be
limited to ITSes participating in the ITSList (and/or only v4 ITSes
that can have the vPE mapping)? Sending to unrelated ITSes
looks unnecessary and may be incorrect.
---
If its_send_command() fails for one ITS in the loop, the VPE
affinity update becomes inconsistent across ITSes. Do we need a
rollback or a recovery path (e.g. retry, VSYNC, or mark vPE unusable)?
> + if ( ret )
> + {
> + spin_unlock(&vmovp_lock);
> + return ret;
> + }
> + }
> +
> + spin_unlock(&vmovp_lock);
> +
> + return 0;
> +}
> +
> static void __init its_vpe_teardown(struct its_vpe *vpe)
> {
> unsigned int order;
> @@ -687,6 +741,52 @@ static void its_make_vpe_non_resident(struct its_vpe *vpe, unsigned int cpu)
> vpe->pending_last = val & GICR_VPENDBASER_PendingLast;
> }
>
> +static int vpe_to_cpuid_lock(struct its_vpe *vpe, unsigned long *flags)
> +{
> + spin_lock_irqsave(&vpe->vpe_lock, *flags);
vpe_lock is used but never initialized.
> + return vpe->col_idx;
> +}
> +
> +static void vpe_to_cpuid_unlock(struct its_vpe *vpe, unsigned long *flags)
> +{
> + spin_unlock_irqrestore(&vpe->vpe_lock, *flags);
> +}
> +
> +static int gicv4_vpe_set_affinity(struct vcpu *vcpu)
> +{
> + struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
> + unsigned int from, to = vcpu->processor;
> + unsigned long flags;
> + int ret = 0;
> +
> + /*
> + * Changing affinity is mega expensive, so let's be as lazy as
> + * we can and only do it if we really have to. Also, if mapped
> + * into the proxy device, we need to move the doorbell interrupt
> + * to its new location.
> + *
> + * Another thing is that changing the affinity of a vPE affects
> + * *other interrupts* such as all the vLPIs that are routed to
> + * this vPE. This means that we must ensure nobody samples
> + * vpe->col_idx during the update, hence the lock below which
> + * must also be taken on any vLPI handling path that evaluates
> + * vpe->col_idx, such as reg-based vLPI invalidation.
> + */
> + from = vpe_to_cpuid_lock(vpe, &flags);
> + if ( from == to )
> + goto out;
> +
> + vpe->col_idx = to;
updates col_idx before VMOVP; on error it is not
restored.
> +
> + ret = its_send_cmd_vmovp(vpe);
> + if ( ret )
> + goto out;
> +
> + out:
> + vpe_to_cpuid_unlock(vpe, &flags);
> + return ret;
> +}
> +
> void vgic_v4_load(struct vcpu *vcpu)
> {
> struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
> @@ -695,6 +795,11 @@ void vgic_v4_load(struct vcpu *vcpu)
> if ( vpe->resident )
> return;
>
> + /*
> + * Before making the VPE resident, make sure the redistributor
> + * corresponding to our current CPU expects us here
> + */
> + WARN_ON(gicv4_vpe_set_affinity(vcpu));
If gicv4_vpe_set_affinity() fails, continuing to make the vPE resident
on the new CPU is very likely wrong: vLPI routing / doorbells can now
target the wrong redistributor.
Best regards,
Mykola
> its_make_vpe_resident(vpe, vcpu->processor);
> vpe->resident = true;
> }
> diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
> index 411beb81c8..f03a8fad47 100644
> --- a/xen/arch/arm/include/asm/gic_v3_its.h
> +++ b/xen/arch/arm/include/asm/gic_v3_its.h
> @@ -43,6 +43,9 @@
> #define GITS_CTLR_QUIESCENT BIT(31, UL)
> #define GITS_CTLR_ENABLE BIT(0, UL)
>
> +#define GITS_CTLR_ITS_NUMBER_SHIFT 4
> +#define GITS_CTLR_ITS_NUMBER (0xfUL << GITS_CTLR_ITS_NUMBER_SHIFT)
> +
> #define GITS_TYPER_PTA BIT(19, UL)
> #define GITS_TYPER_DEVIDS_SHIFT 13
> #define GITS_TYPER_DEVIDS_MASK (0x1fUL << GITS_TYPER_DEVIDS_SHIFT)
> @@ -60,6 +63,8 @@
> GITS_TYPER_ITT_SIZE_SHIFT) + 1)
> #define GITS_TYPER_PHYSICAL (1U << 0)
>
> +#define GITS_TYPER_VLPIS (1UL << 1)
> +#define GITS_TYPER_VMOVP (1UL << 37)
> #define GITS_BASER_INDIRECT BIT(62, UL)
> #define GITS_BASER_INNER_CACHEABILITY_SHIFT 59
> #define GITS_BASER_TYPE_SHIFT 56
> @@ -118,6 +123,12 @@
> /* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
> #define LPI_BLOCK 32U
>
> +/*
> + * Maximum number of ITSs when GITS_TYPER.VMOVP == 0, using the
> + * ITSList mechanism to perform inter-ITS synchronization.
> + */
> +#define GICv4_ITS_LIST_MAX 16
> +
> extern unsigned int nvpeid;
> /* The maximum number of VPEID bits supported by VLPI commands */
> #define ITS_MAX_VPEID_BITS nvpeid
> @@ -214,6 +225,7 @@ struct __lpi_data {
> extern struct __lpi_data lpi_data;
>
> extern struct list_head host_its_list;
> +extern unsigned long its_list_map;
>
> int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
> uint32_t eventid);
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 15/19] arm: Add gicv4 to domain creation
2026-02-02 16:14 ` [RFC PATCH 15/19] arm: Add gicv4 to domain creation Mykyta Poturai
@ 2026-02-18 21:00 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-18 21:00 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Allow creating domains with GIC_V4 version.
I am not convinced we should extend the public domctl ABI with
XEN_DOMCTL_CONFIG_GIC_V4/V4_1.
The GIC spec treats v4 as an extension of v3:
Because GICv4 is an extension of GICv3, all references to GICv3
in this manual apply equally to GICv4, unless explicitly indicated
otherwise. Any changes for GICv4.1 are indicated accordingly.
From a guest ABI perspective, a v4-capable system is still a GICv3
programmer's model, with additional optional features (e.g. vLPI/
vPE/ITS commands). Those features should be enabled/queried via the
relevant capability bits, not by inventing new "GIC version" domctl
values.
So I'd prefer:
* keep gic_hw_version accurate internally (v3/v4/v4.1 as needed),
but
* for domctl configuration, treat v4.x as XEN_DOMCTL_CONFIG_GIC_V3,
and keep dom0less DT generation using the gicv3 node.
This avoids growing the public ABI and toolstack churn for something
that is effectively a v3 + extensions model.
Best regards,
Mykola
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/dom0less-build.c | 1 +
> xen/arch/arm/domain.c | 16 ++++++++++++++++
> xen/arch/arm/vgic.c | 2 ++
> xen/include/public/arch-arm.h | 2 ++
> 4 files changed, 21 insertions(+)
>
> diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
> index 4181c10538..3c4b074b05 100644
> --- a/xen/arch/arm/dom0less-build.c
> +++ b/xen/arch/arm/dom0less-build.c
> @@ -157,6 +157,7 @@ int __init make_intc_domU_node(struct kernel_info *kinfo)
> {
> #ifdef CONFIG_GICV3
> case GIC_V3:
> + case GIC_V4:
> return make_gicv3_domU_node(kinfo);
> #endif
> #ifdef CONFIG_VGICV2
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 507df807ed..864e5511c9 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -604,6 +604,14 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
> config->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_V3;
> break;
>
> + case GIC_V4:
> + config->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_V4;
> + break;
> +
> + case GIC_V4_1:
> + config->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_V4_1;
> + break;
> +
> default:
> ASSERT_UNREACHABLE();
> return -EINVAL;
> @@ -679,6 +687,14 @@ int arch_domain_create(struct domain *d,
> d->arch.vgic.version = GIC_V3;
> break;
>
> + case XEN_DOMCTL_CONFIG_GIC_V4:
> + d->arch.vgic.version = GIC_V4;
> + break;
> +
> + case XEN_DOMCTL_CONFIG_GIC_V4_1:
> + d->arch.vgic.version = GIC_V4_1;
> + break;
> +
> default:
> BUG();
> }
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 6baf870ad5..e5752faeb7 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -153,6 +153,7 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
> {
> #ifdef CONFIG_GICV3
> case GIC_V3:
> + case GIC_V4:
> if ( vgic_v3_init(d, mmio_count) )
> return -ENODEV;
> break;
> @@ -929,6 +930,7 @@ unsigned int vgic_max_vcpus(unsigned int domctl_vgic_version)
>
> #ifdef CONFIG_GICV3
> case XEN_DOMCTL_CONFIG_GIC_V3:
> + case XEN_DOMCTL_CONFIG_GIC_V4:
> return 4096;
> #endif
>
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 96365e5603..12690cf3ac 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -322,6 +322,8 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> #define XEN_DOMCTL_CONFIG_GIC_NATIVE 0
> #define XEN_DOMCTL_CONFIG_GIC_V2 1
> #define XEN_DOMCTL_CONFIG_GIC_V3 2
> +#define XEN_DOMCTL_CONFIG_GIC_V4 3
> +#define XEN_DOMCTL_CONFIG_GIC_V4_1 4
>
> #define XEN_DOMCTL_CONFIG_TEE_NONE 0
> #define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 16/19] arm/gic: Fix LR group handling for GICv4
2026-02-02 16:14 ` [RFC PATCH 16/19] arm/gic: Fix LR group handling for GICv4 Mykyta Poturai
@ 2026-02-18 21:00 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-18 21:00 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
This commit can likely be dropped if we apply the approach from the
review of the previous one (i.e. do not introduce per-domain GIC_V4/V4_1
versions in the domctl ABI and keep guests on XEN_DOMCTL_CONFIG_GIC_V3).
In that case current->domain->arch.vgic.version would remain GIC_V3 for
v4-capable systems too, so the existing checks (== GIC_V3) already cover
the intended behaviour.
If we keep v4/v4.1 as distinct internal values, then using >= GIC_V3 is
fine, but please avoid relying on numeric ordering of enums unless that
ordering is explicitly guaranteed.
Thanks,
Mykola
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Extend the check to mark interrupts as Group1 for all GIC versions >= 3
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 07736179db..1cb3169b72 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -1366,10 +1366,10 @@ static void gicv3_update_lr(int lr, unsigned int virq, uint8_t priority,
> val = (((uint64_t)state & 0x3) << ICH_LR_STATE_SHIFT);
>
> /*
> - * When the guest is GICv3, all guest IRQs are Group 1, as Group0
> - * would result in a FIQ in the guest, which it wouldn't expect
> + * When the guest is GICv3/GICv4/GICv4.1, all guest IRQs are Group 1, as
> + * Group0 would result in a FIQ in the guest, which it wouldn't expect
> */
> - if ( current->domain->arch.vgic.version == GIC_V3 )
> + if ( current->domain->arch.vgic.version >= GIC_V3 )
> val |= ICH_LR_GRP1;
>
> val |= (uint64_t)priority << ICH_LR_PRIORITY_SHIFT;
> @@ -1455,10 +1455,10 @@ static void gicv3_write_lr(int lr, const struct gic_lr *lr_reg)
> }
>
> /*
> - * When the guest is using vGICv3, all the IRQs are Group 1. Group 0
> - * would result in a FIQ, which will not be expected by the guest OS.
> + * When the guest is using vGICv3/vGICv4/vGICv4.1, all the IRQs are Group 1.
> + * Group 0 would result in a FIQ, which will not be expected by the guest OS.
> */
> - if ( vgic_version == GIC_V3 )
> + if ( vgic_version >= GIC_V3 )
> lrv |= ICH_LR_GRP1;
>
> gicv3_ich_write_lr(lr, lrv);
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 19/19] arm/gicv4: Add GICv4 to the build system
2026-02-02 16:14 ` [RFC PATCH 19/19] arm/gicv4: Add GICv4 to the build system Mykyta Poturai
@ 2026-02-19 11:30 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-19 11:30 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Add a config option to enable GICv4 support and include the
> corresponding source file in the build system.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/Kconfig | 6 ++++++
> xen/arch/arm/Makefile | 1 +
> 2 files changed, 7 insertions(+)
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 442d353b43..c18eca9f9a 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -255,6 +255,12 @@ config ARM64_BTI
> Branch Target Identification support.
> This feature is not supported in Xen.
>
> +config GICV4
> + bool "GICv4 driver"
nit: I would rename this to something more specific, e.g.:
"GICv4 vLPI direct injection (experimental)",
as this is not a standalone GIC driver but v4 ITS extensions for direct
vLPI delivery.
It may also be worth gating it under "if UNSUPPORTED", since this builds
on the ITS code path which is currently marked as unsupported.
> + depends on GICV3 && HAS_ITS
> + help
> + Driver for ARM Generic Interrupt Controller v4 extension.
Maybe expand the help text a bit, for example:
Enable direct injection of vLPIs using the GICv4 ITS extensions.
This adds support for vPE/vLPI management via ITS commands such as
VMAPP, VMAPTI and (on vCPU migration) VMOVP, plus handling of per-vPE
doorbell LPIs used by Xen to reschedule vCPUs when vLPIs are pending.
Note: This relies on the (currently unsupported) ITS code path in Xen
and is considered experimental.
Best regards,
Mykola
> +
> source "arch/arm/tee/Kconfig"
>
> config PARTIAL_EMULATION
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 95bc7ad25e..8569df571b 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -27,6 +27,7 @@ obj-y += gic.o
> obj-$(CONFIG_GICV2) += gic-v2.o
> obj-$(CONFIG_GICV3) += gic-v3.o
> obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
> +obj-$(CONFIG_GICV4) += gic-v4-its.o
> obj-$(CONFIG_HAS_ITS) += gic-v3-lpi.o
> obj-y += guestcopy.o
> obj-y += guest_atomics.o
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 17/19] arm/gicv4: Handle doorbells
2026-02-02 16:14 ` [RFC PATCH 17/19] arm/gicv4: Handle doorbells Mykyta Poturai
@ 2026-02-19 11:30 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-19 11:30 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> When GIC wants to inject a virtual interrupt to a VCPU that is not
> currently scheduled, it sends a kick for a hypervisor to schedule it. To
> receive such kicks, we need to set up a doorbell interrupt for each VPE.
>
> Add changes necessary to allocate, mask/unmask and handle doorbell
> interrupts for each VPE. When a doorbell interrupt is received, set the
> pending_last flag for the corresponding VPE and kick it, so that the
> hypervisor schedules the VCPU to handle pending VLPIs.
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3-its.c | 13 ++-
> xen/arch/arm/gic-v3-lpi.c | 69 ++++++++++----
> xen/arch/arm/gic-v4-its.c | 127 ++++++++++++++++++++++++++
> xen/arch/arm/include/asm/gic_v3_its.h | 6 +-
> xen/arch/arm/vgic-v3-its.c | 14 ++-
> 5 files changed, 203 insertions(+), 26 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index be840fbc8f..fa5c1eb6d1 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -1016,8 +1016,11 @@ int gicv3_its_map_guest_device(struct domain *d,
>
> dev->guest_doorbell = guest_doorbell;
> dev->guest_devid = guest_devid;
> - dev->host_devid = host_devid;
> - dev->eventids = nr_events;
> +
> + #ifdef CONFIG_GICV4
> + spin_lock_init(&dev->event_map.vlpi_lock);
> + dev->event_map.nr_lpis = nr_events;
> + #endif
>
> rb_link_node(&dev->rbnode, parent, new);
> rb_insert_color(&dev->rbnode, &d->arch.vgic.its_devices);
> @@ -1142,7 +1145,8 @@ int gicv3_remove_guest_event(struct domain *d, paddr_t vdoorbell_address,
> if ( host_lpi == INVALID_LPI )
> return -EINVAL;
>
> - gicv3_lpi_update_host_entry(host_lpi, d->domain_id, INVALID_LPI);
> + gicv3_lpi_update_host_entry(host_lpi, d->domain_id, INVALID_LPI,
> + false, INVALID_VCPU_ID);
>
> return 0;
> }
> @@ -1169,7 +1173,8 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
> if ( !pirq )
> return NULL;
>
> - gicv3_lpi_update_host_entry(host_lpi, d->domain_id, virt_lpi);
> + gicv3_lpi_update_host_entry(host_lpi, d->domain_id, virt_lpi,
> + false, INVALID_VCPU_ID);
>
> return pirq;
> }
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 3c2649b695..37f1aa1064 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -39,7 +39,7 @@ union host_lpi {
> struct {
> uint32_t virt_lpi;
> uint16_t dom_id;
> - uint16_t pad;
> + uint16_t db_vcpu_id;
> };
> };
>
> @@ -161,24 +161,48 @@ void gicv3_do_LPI(unsigned int lpi)
> * ignore them, as they have no further state and no-one can expect
> * to see them if they have not been mapped.
> */
> - if ( hlpi.virt_lpi == INVALID_LPI )
> + if ( hlpi.virt_lpi == INVALID_LPI && hlpi.db_vcpu_id == INVALID_VCPU_ID )
> goto out;
>
> d = rcu_lock_domain_by_id(hlpi.dom_id);
> if ( !d )
> goto out;
>
> - /*
> - * TODO: Investigate what to do here for potential interrupt storms.
> - * As we keep all host LPIs enabled, for disabling LPIs we would need
> - * to queue a ITS host command, which we avoid so far during a guest's
> - * runtime. Also re-enabling would trigger a host command upon the
> - * guest sending a command, which could be an attack vector for
> - * hogging the host command queue.
> - * See the thread around here for some background:
> - * https://lists.xen.org/archives/html/xen-devel/2016-12/msg00003.html
> - */
> - vgic_vcpu_inject_lpi(d, hlpi.virt_lpi);
> + /* It is a doorbell interrupt. */
> + if ( hlpi.db_vcpu_id != INVALID_VCPU_ID )
> + {
> +#ifdef CONFIG_GICV4
> + struct vcpu *v = d->vcpu[hlpi.db_vcpu_id];
> +
> + /* We got the message, no need to fire again */
> + its_vpe_mask_db(v->arch.vgic.its_vpe);
> +
> + /*
> + * Update the pending_last flag that indicates that VLPIs are pending.
> + * And the corresponding vcpu is also kicked into action.
> + */
> + v->arch.vgic.its_vpe->pending_last = true;
This write is not synchronized. pending_last is touched from the LPI
handler path, so please use the appropriate vPE lock here, or make it
an atomic/WRITE_ONCE with a clear concurrency rationale. As-is this is
a data race.
> +
> + vcpu_kick(v);
> +#else
> + printk(XENLOG_WARNING
> + "Doorbell LPI is only suooprted on GICV4\n");
> +#endif
> + }
> + else
> + {
> + /*
> + * TODO: Investigate what to do here for potential interrupt storms.
> + * As we keep all host LPIs enabled, for disabling LPIs we would need
> + * to queue a ITS host command, which we avoid so far during a guest's
> + * runtime. Also re-enabling would trigger a host command upon the
> + * guest sending a command, which could be an attack vector for
> + * hogging the host command queue.
> + * See the thread around here for some background:
> + * https://lists.xen.org/archives/html/xen-devel/2016-12/msg00003.html
> + */
> + vgic_vcpu_inject_lpi(d, hlpi.virt_lpi);
> + }
>
> rcu_unlock_domain(d);
>
> @@ -187,7 +211,8 @@ out:
> }
>
> void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> - uint32_t virt_lpi)
> + uint32_t virt_lpi, bool is_db,
> + uint16_t db_vcpu_id)
> {
> union host_lpi *hlpip, hlpi;
>
> @@ -197,8 +222,16 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
>
> hlpip = &lpi_data.host_lpis[host_lpi / HOST_LPIS_PER_PAGE][host_lpi % HOST_LPIS_PER_PAGE];
>
> - hlpi.virt_lpi = virt_lpi;
> - hlpi.dom_id = domain_id;
> + if ( !is_db )
> + {
> + hlpi.virt_lpi = virt_lpi;
> + hlpi.dom_id = domain_id;
> + }
> + else
> + {
> + hlpi.dom_id = domain_id;
> + hlpi.db_vcpu_id = db_vcpu_id;
> + }
>
> write_u64_atomic(&hlpip->data, hlpi.data);
> }
> @@ -595,6 +628,7 @@ int gicv3_allocate_host_lpi_block(struct domain *d, uint32_t *first_lpi)
> */
> hlpi.virt_lpi = INVALID_LPI;
> hlpi.dom_id = d->domain_id;
> + hlpi.db_vcpu_id = INVALID_VCPU_ID;
> write_u64_atomic(&lpi_data.host_lpis[chunk][lpi_idx + i].data,
> hlpi.data);
>
> @@ -602,7 +636,8 @@ int gicv3_allocate_host_lpi_block(struct domain *d, uint32_t *first_lpi)
> * Enable this host LPI, so we don't have to do this during the
> * guest's runtime.
> */
> - lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
> + lpi_write_config(lpi_data.lpi_property, lpi + i + LPI_OFFSET, 0xff,
> + LPI_PROP_ENABLED);
> }
>
> lpi_data.next_free_lpi = lpi + LPI_BLOCK;
> diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
> index 175fda7acb..0462976b93 100644
> --- a/xen/arch/arm/gic-v4-its.c
> +++ b/xen/arch/arm/gic-v4-its.c
> @@ -157,6 +157,9 @@ static int its_send_cmd_vmapp(struct host_its *its, struct its_vpe *vpe,
> cmd[3] = (vpt_addr & GENMASK(51, 16)) |
> ((HOST_LPIS_NRBITS - 1) & GENMASK(4, 0));
>
> + /* Default doorbell interrupt */
> + cmd[1] |= (uint64_t)vpe->vpe_db_lpi;
Doorbell LPI is a GICv4.1 addition; for GICv4.0 this should not be set.
> +
> out:
> ret = its_send_command(its, cmd);
>
> @@ -296,6 +299,37 @@ static int its_send_cmd_vmovp(struct its_vpe *vpe)
> return 0;
> }
>
> +
> +static void its_vpe_send_inv_db(struct its_vpe *vpe)
> +{
> + // struct its_device *dev = vpe_proxy.dev;
> + // unsigned long flags;
> +
> + // spin_lock_irqsave(&vpe_proxy.lock, flags);
> + // gicv4_vpe_db_proxy_map_locked(vpe);
> + // its_send_cmd_inv(dev->hw_its, dev->host_devid, vpe->vpe_proxy_event);
> + // spin_unlock_irqrestore(&vpe_proxy.lock, flags);
> +}
> +
> +static void its_vpe_inv_db(struct its_vpe *vpe)
> +{
> + its_vpe_send_inv_db(vpe);
> +}
> +
> +void its_vpe_mask_db(struct its_vpe *vpe)
> +{
> + /* Only clear enable bit. */
> + lpi_write_config(lpi_data.lpi_property, vpe->vpe_db_lpi, LPI_PROP_ENABLED, 0);
> + its_vpe_inv_db(vpe);
> +}
> +
> +static void its_vpe_unmask_db(struct its_vpe *vpe)
> +{
> + /* Only set enable bit. */
> + lpi_write_config(lpi_data.lpi_property, vpe->vpe_db_lpi, 0, LPI_PROP_ENABLED);
> + its_vpe_inv_db(vpe);
> +}
> +
> static void __init its_vpe_teardown(struct its_vpe *vpe)
> {
> unsigned int order;
> @@ -309,6 +343,8 @@ static void __init its_vpe_teardown(struct its_vpe *vpe)
> int vgic_v4_its_vm_init(struct domain *d)
> {
> unsigned int nr_vcpus = d->max_vcpus;
> + unsigned int nr_db_lpis, nr_chunks, i = 0;
> + uint32_t *db_lpi_bases;
> int ret = -ENOMEM;
>
> if ( !gicv3_its_host_has_its() )
> @@ -326,9 +362,31 @@ int vgic_v4_its_vm_init(struct domain *d)
> d->arch.vgic.its_vm->vproptable = lpi_allocate_proptable();
> if ( !d->arch.vgic.its_vm->vproptable )
> goto fail_vprop;
> + /* Allocate a doorbell interrupt for each VPE. */
> + nr_db_lpis = d->arch.vgic.its_vm->nr_vpes;
> + nr_chunks = DIV_ROUND_UP(nr_db_lpis, LPI_BLOCK);
> + db_lpi_bases = xzalloc_array(uint32_t, nr_chunks);
> + if ( !db_lpi_bases )
> + goto fail_db_bases;
> +
> + do {
> + /* Allocate doorbell interrupts in chunks of LPI_BLOCK (=32). */
> + ret = gicv3_allocate_host_lpi_block(d, &db_lpi_bases[i]);
> + if ( ret )
> + goto fail_db;
> + } while ( ++i < nr_chunks );
> +
> + d->arch.vgic.its_vm->db_lpi_bases = db_lpi_bases;
> + d->arch.vgic.its_vm->nr_db_lpis = nr_db_lpis;
>
> return 0;
>
> +fail_db:
> + while ( --i >= 0 )
'i' is unsigned, so --i >= 0 is always true.
This will loop forever.
> + gicv3_free_host_lpi_block(d->arch.vgic.its_vm->db_lpi_bases[i]);
The fail path frees via d->arch.vgic.its_vm->db_lpi_bases, but this
pointer is only assigned on the success path. On failure it can be
NULL/uninitialized. Please free via the local db_lpi_bases[] (and then
xfree(db_lpi_bases)), or assign its_vm->db_lpi_bases before entering the
allocation loop.
> + xfree(db_lpi_bases);
> +fail_db_bases:
> + lpi_free_proptable(d->arch.vgic.its_vm->vproptable);
> fail_vprop:
> xfree(d->arch.vgic.its_vm->vpes);
> fail_vpes:
> @@ -340,8 +398,13 @@ fail_vprop:
> void vgic_v4_free_its_vm(struct domain *d)
> {
> struct its_vm *its_vm = d->arch.vgic.its_vm;
> + int nr_chunks = DIV_ROUND_UP(its_vm->nr_db_lpis, LPI_BLOCK);
> if ( its_vm->vpes )
> xfree(its_vm->vpes);
> + while ( --nr_chunks >= 0 )
> + gicv3_free_host_lpi_block(its_vm->db_lpi_bases[nr_chunks]);
> + if ( its_vm->db_lpi_bases )
> + xfree(its_vm->db_lpi_bases);
> if ( its_vm->vproptable )
> lpi_free_proptable(its_vm);
> }
> @@ -357,14 +420,29 @@ int vgic_v4_its_vpe_init(struct vcpu *vcpu)
> return -ENOMEM;
>
> its_vm->vpes[vcpuid] = vcpu->arch.vgic.its_vpe;
> + vcpu->arch.vgic.its_vpe = vcpu->arch.vgic.its_vpe;
This is a no-op self-assignment; please remove.
> + vcpu->arch.vgic.its_vpe->vpe_db_lpi = its_vm->db_lpi_bases[vcpuid/32] + (vcpuid % 32);
> + /*
> + * Sometimes vlpi gets firstly mapped before associated vpe
> + * becoming resident, so in case missing the interrupt, we intend to
> + * enable doorbell at the initialization stage
> + */
> +
> vcpu->arch.vgic.its_vpe->its_vm = its_vm;
>
> + gicv3_lpi_update_host_entry(vcpu->arch.vgic.its_vpe->vpe_db_lpi,
> + vcpu->domain->domain_id, INVALID_LPI, true,
> + vcpu->vcpu_id);
> +
> +
> ret = its_vpe_init(vcpu->arch.vgic.its_vpe);
> if ( ret )
> {
> its_vpe_teardown(vcpu->arch.vgic.its_vpe);
> return ret;
> }
> + its_vpe_unmask_db(vcpu->arch.vgic.its_vpe);
> +
> return 0;
> }
>
> @@ -800,6 +878,7 @@ void vgic_v4_load(struct vcpu *vcpu)
> * corresponding to our current CPU expects us here
> */
> WARN_ON(gicv4_vpe_set_affinity(vcpu));
> + its_vpe_mask_db(vpe);
> its_make_vpe_resident(vpe, vcpu->processor);
> vpe->resident = true;
> }
> @@ -812,5 +891,53 @@ void vgic_v4_put(struct vcpu *vcpu, bool need_db)
> return;
>
> its_make_vpe_non_resident(vpe, vcpu->processor);
> + if ( need_db )
> + /* Enable the doorbell, as the guest is going to block */
> + its_vpe_unmask_db(vpe);
> vpe->resident = false;
> }
> +
> +static int its_vlpi_set_doorbell(struct its_vlpi_map *map, bool enable)
> +{
> + if (map->db_enabled == enable)
> + return 0;
> +
> + map->db_enabled = enable;
> +
> + /*
> + * Ideally, we'd issue a VMAPTI to set the doorbell to its LPI
> + * value or to 1023, depending on the enable bit. But that
> + * would be issuing a mapping for an /existing/ DevID+EventID
> + * pair, which is UNPREDICTABLE. Instead, let's issue a VMOVI
> + * to the /same/ vPE, using this opportunity to adjust the doorbell.
> + */
> + return its_send_cmd_vmovi(map->dev->hw_its, map);
> +}
> +
> +int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
> + bool needs_inv)
> +{
> + struct its_vlpi_map *map;
> + unsigned int cpu;
> + int ret;
> +
> + if ( !pirq->vlpi_map )
> + return -EINVAL;
> +
> + map = pirq->vlpi_map;
> +
> + /* Cache the updated property and update the vproptable. */
> + map->properties = property;
> + lpi_write_config(map->vm->vproptable, pirq->irq, 0xff, property);
> +
> + if ( needs_inv )
> + {
> + cpu = map->vm->vpes[map->vpe_idx]->col_idx;
> + ret = its_inv_lpi(map->dev->hw_its, map->dev, map->eventid, cpu);
> + if ( ret )
> + return ret;
> + }
> +
> + return its_vlpi_set_doorbell(map, property & LPI_PROP_ENABLED);
> +}
> +
> diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
> index f03a8fad47..dababe97cd 100644
> --- a/xen/arch/arm/include/asm/gic_v3_its.h
> +++ b/xen/arch/arm/include/asm/gic_v3_its.h
> @@ -295,7 +295,9 @@ struct pending_irq *gicv3_assign_guest_event(struct domain *d,
> uint32_t vdevid, uint32_t eventid,
> uint32_t virt_lpi);
> void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> - uint32_t virt_lpi);
> + uint32_t virt_lpi, bool is_db,
> + uint16_t db_vcpu_id);
> +
> struct its_baser *its_get_baser(struct host_its *hw_its, uint32_t type);
> bool its_alloc_table_entry(struct its_baser *baser, uint32_t id);
> struct page_info *lpi_allocate_pendtable(void);
> @@ -322,6 +324,8 @@ bool event_is_forwarded_to_vcpu(struct its_device *dev, uint32_t eventid);
> void its_vpe_mask_db(struct its_vpe *vpe);
> #endif
> int gicv4_its_vlpi_unmap(struct pending_irq *pirq);
> +int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
> + bool needs_inv);
>
> /* ITS quirks handling. */
> uint64_t gicv3_its_get_cacheability(void);
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 94f7dd7d90..0a740ad68f 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -387,7 +387,7 @@ out_unlock:
> * property table and update the virtual IRQ's state in the given pending_irq.
> * Must be called with the respective VGIC VCPU lock held.
> */
> -static int update_lpi_property(struct domain *d, struct pending_irq *p)
> +int update_lpi_property(struct domain *d, struct pending_irq *p, bool needs_inv)
I don’t see any out-of-file users of update_lpi_property().
If it’s still file-local, keep it static.
Best regards,
Mykola
> {
> paddr_t addr;
> uint8_t property;
> @@ -417,6 +417,9 @@ static int update_lpi_property(struct domain *d, struct pending_irq *p)
> else
> clear_bit(GIC_IRQ_GUEST_ENABLED, &p->status);
>
> + if ( pirq_is_tied_to_hw(p) )
> + return its_vlpi_prop_update(p, property, needs_inv);
> +
> return 0;
> }
>
> @@ -430,6 +433,9 @@ static int update_lpi_property(struct domain *d, struct pending_irq *p)
> */
> static void update_lpi_vgic_status(struct vcpu *v, struct pending_irq *p)
> {
> + if ( pirq_is_tied_to_hw(p) )
> + return;
> +
> ASSERT(spin_is_locked(&v->arch.vgic.lock));
>
> if ( test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) )
> @@ -479,7 +485,7 @@ static int its_handle_inv(struct virt_its *its, uint64_t *cmdptr)
> spin_lock_irqsave(&vcpu->arch.vgic.lock, flags);
>
> /* Read the property table and update our cached status. */
> - if ( update_lpi_property(d, p) )
> + if ( update_lpi_property(d, p, true) )
> goto out_unlock;
>
> /* Check whether the LPI needs to go on a VCPU. */
> @@ -552,7 +558,7 @@ static int its_handle_invall(struct virt_its *its, uint64_t *cmdptr)
>
> vlpi = pirqs[i]->irq;
> /* If that fails for a single LPI, carry on to handle the rest. */
> - err = update_lpi_property(its->d, pirqs[i]);
> + err = update_lpi_property(its->d, pirqs[i], false);
> if ( !err )
> update_lpi_vgic_status(vcpu, pirqs[i]);
> else
> @@ -785,7 +791,7 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
> * We don't need the VGIC VCPU lock here, because the pending_irq isn't
> * in the radix tree yet.
> */
> - ret = update_lpi_property(its->d, pirq);
> + ret = update_lpi_property(its->d, pirq, true);
> if ( ret )
> goto out_remove_host_entry;
>
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 18/19] arm/gic: Add VPE proxy support
2026-02-02 16:14 ` [RFC PATCH 18/19] arm/gic: Add VPE proxy support Mykyta Poturai
@ 2026-02-19 16:00 ` Mykola Kvach
0 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-02-19 16:00 UTC (permalink / raw)
To: Mykyta Poturai
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
On Mon, Feb 2, 2026 at 6:14 PM Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> xen/arch/arm/gic-v3-its.c | 13 ++
> xen/arch/arm/gic-v3.c | 1 +
> xen/arch/arm/gic-v4-its.c | 207 ++++++++++++++++++++++++-
> xen/arch/arm/include/asm/gic_v3_defs.h | 7 +
> xen/arch/arm/include/asm/gic_v3_its.h | 7 +
> xen/arch/arm/include/asm/gic_v4_its.h | 5 +
> xen/arch/arm/include/asm/vgic.h | 2 +
> 7 files changed, 235 insertions(+), 7 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index fa5c1eb6d1..5979a82526 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -333,6 +333,19 @@ int its_send_cmd_discard(struct host_its *its, struct its_device *dev,
> return its_send_command(its, cmd);
> }
>
> +int its_send_cmd_movi(struct host_its *its, uint32_t deviceid, uint32_t eventid,
> + uint16_t icid)
> +{
> + uint64_t cmd[4];
> +
> + cmd[0] = GITS_CMD_MOVI | ((uint64_t)deviceid << 32);
> + cmd[1] = eventid;
> + cmd[2] = icid;
> + cmd[3] = 0x00;
> +
> + return its_send_command(its, cmd);
> +}
> +
> /* Set up the (1:1) collection mapping for the given host CPU. */
> int gicv3_its_setup_collection(unsigned int cpu)
> {
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 1cb3169b72..fb80038f17 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -2096,6 +2096,7 @@ static void __init gicv4_init(void)
>
> gicv4_its_vpeid_allocator_init();
>
> + gicv4_init_vpe_proxy();
This call ignores errors from gicv4_init_vpe_proxy(). If it returns an error
code, please handle it (propagate up, or at least log and bail out) to avoid
continuing with a partially initialized proxy.
> }
> #else
> static void __init gicv4_init(void)
> diff --git a/xen/arch/arm/gic-v4-its.c b/xen/arch/arm/gic-v4-its.c
> index 0462976b93..83ee0510ac 100644
> --- a/xen/arch/arm/gic-v4-its.c
> +++ b/xen/arch/arm/gic-v4-its.c
> @@ -39,6 +39,13 @@ static spinlock_t vpeid_alloc_lock = SPIN_LOCK_UNLOCKED;
> static uint16_t vmovp_seq_num;
> static spinlock_t vmovp_lock = SPIN_LOCK_UNLOCKED;
>
> +static struct {
> + spinlock_t lock;
> + struct its_device *dev;
> + struct its_vpe **vpes;
> + int next_victim;
> +} vpe_proxy;
> +
> void __init gicv4_its_vpeid_allocator_init(void)
> {
> /* Allocate space for vpeid_mask based on MAX_VPEID */
> @@ -201,6 +208,124 @@ static int its_map_vpe(struct host_its *its, struct its_vpe *vpe)
>
> return 0;
> }
> +static int gicv4_vpe_db_proxy_unmap_locked(struct its_vpe *vpe)
> +{
> + int ret;
> +
> + /* Already unmapped? */
> + if ( vpe->vpe_proxy_event == -1 )
> + return 0;
> +
> + ret = its_send_cmd_discard(vpe_proxy.dev->hw_its, vpe_proxy.dev,
> + vpe->vpe_proxy_event);
> + if ( ret )
> + return ret;
> + vpe_proxy.vpes[vpe->vpe_proxy_event] = NULL;
> +
> + /*
> + * We don't track empty slots at all, so let's move the
> + * next_victim pointer to quickly reuse the unmapped slot
> + */
> + if ( vpe_proxy.vpes[vpe_proxy.next_victim] )
> + vpe_proxy.next_victim = vpe->vpe_proxy_event;
> +
> + vpe->vpe_proxy_event = -1;
> +
> + return 0;
> +}
> +
> +static void gicv4_vpe_db_proxy_unmap(struct its_vpe *vpe)
> +{
> + if ( !gic_support_directLPI() )
> + {
> + unsigned long flags;
> +
> + spin_lock_irqsave(&vpe_proxy.lock, flags);
> + gicv4_vpe_db_proxy_unmap_locked(vpe);
Both return values are ignored. If map_locked() fails and leaves
vpe_proxy_event at the sentinel value, INV may use an invalid eventid.
At minimum: check map_locked() and INV return codes and avoid issuing
INV on failure.
> + spin_unlock_irqrestore(&vpe_proxy.lock, flags);
> + }
> +}
> +
> +/*
> + * If a GICv4.0 doesn't implement Direct LPIs (which is extremely
> + * likely), the only way to perform an invalidate is to use a fake
> + * device to issue an INV command, implying that the LPI has first
> + * been mapped to some event on that device. Since this is not exactly
> + * cheap, we try to keep that mapping around as long as possible, and
> + * only issue an UNMAP if we're short on available slots.
> + *
> + * GICv4.1 mandates that we're able to invalidate by writing to a
> + * MMIO register. And most of the time, we don't even have to invalidate
> + * vPE doorbell, as the redistributor can be told whether to generate a
> + * doorbell or not.
> + */
> +static int gicv4_vpe_db_proxy_map_locked(struct its_vpe *vpe)
> +{
> + int ret;
> +
> + /* Already mapped? */
> + if ( vpe->vpe_proxy_event != -1 )
> + return 0;
> +
> + /* This slot was already allocated. Kick the other VPE out. */
> + if ( vpe_proxy.vpes[vpe_proxy.next_victim] )
> + {
> + struct its_vpe *old_vpe = vpe_proxy.vpes[vpe_proxy.next_victim];
> +
> + ret = gicv4_vpe_db_proxy_unmap_locked(old_vpe);
> + if ( ret )
> + return ret;
> + }
> +
> + /* Map the new VPE instead */
> + vpe_proxy.vpes[vpe_proxy.next_victim] = vpe;
> + vpe->vpe_proxy_event = vpe_proxy.next_victim;
> + vpe_proxy.next_victim = (vpe_proxy.next_victim + 1) %
> + vpe_proxy.dev->eventids;
> +
> + return its_send_cmd_mapti(vpe_proxy.dev->hw_its, vpe_proxy.dev->host_devid,
The software state is committed before MAPTI completes. If MAPTI
fails, the vpe stays "mapped" in SW (and next_victim advances), but HW
doesn't have the translation. I think this needs rollback on failure
(or update state only after MAPTI succeeds).
> + vpe->vpe_proxy_event, vpe->vpe_db_lpi,
> + vpe->col_idx);
> +}
> +
> +int __init gicv4_init_vpe_proxy(void)
> +{
> + struct host_its *hw_its;
> + uint32_t devid;
> +
> + if ( gic_support_directLPI() )
> + {
> + printk("ITS: Using DirectLPI for GICv4 VPE invalidation\n");
> + return 0;
> + }
> +
> + /* Any ITS will do, even if not v4 */
> + hw_its = list_first_entry(&host_its_list, struct host_its, entry);
host_its_list may be empty here. list_first_entry() is unsafe in that
case please use list_first_entry_or_null() (or check list_empty()) and
handle the NULL/empty case
> +
> + vpe_proxy.vpes = xzalloc_array(struct its_vpe *, nr_cpu_ids);
> + if ( !vpe_proxy.vpes )
> + {
> + printk(XENLOG_ERR "ITS: Can't allocate GICv4 VPE proxy device array\n");
> + return -ENOMEM;
> + }
> +
> + /* Use the last possible DevID */
> + devid = BIT(hw_its->devid_bits, UL) - 1;
This "pick the last DevID" policy looks fragile: it can collide with
other Xen ITS users if they ever end up allocating across the full
DevID space.
> + vpe_proxy.dev = its_create_device(hw_its, devid, nr_cpu_ids);
> + if ( !vpe_proxy.dev )
> + {
> + printk(XENLOG_ERR "ITS: Can't allocate GICv4 VPE proxy device\n");
vpe_proxy.vpes is leaked when vpe_proxy.dev allocation fails
> + return -ENOMEM;
> + }
> +
> + spin_lock_init(&vpe_proxy.lock);
> + vpe_proxy.next_victim = 0;
> + printk(XENLOG_INFO
> + "ITS: Allocated DevID %u as GICv4 VPE proxy device\n", devid);
> +
> + return 0;
> +}
> +
> static int __init its_vpe_init(struct its_vpe *vpe)
> {
> int vpe_id, rc = -ENOMEM;
> @@ -224,6 +349,7 @@ static int __init its_vpe_init(struct its_vpe *vpe)
> rwlock_init(&vpe->lock);
> vpe->vpe_id = vpe_id;
> vpe->vpendtable = page_to_virt(vpendtable);
> + vpe->vpe_proxy_event = -1;
> /*
> * We eagerly inform all the v4 ITS and map vPE to the first
> * possible CPU
> @@ -299,16 +425,45 @@ static int its_send_cmd_vmovp(struct its_vpe *vpe)
> return 0;
> }
>
> +/* GICR_SYNCR.Busy == 1 until the invalidation completes. */
> +static void wait_for_syncr(void __iomem *rdbase)
> +{
> + while ( readl_relaxed(rdbase + GICR_SYNCR) & 1 )
> + cpu_relax();
This is an unbounded busy-wait. If SYNCR.Busy gets stuck (or RDbase is
wrong), Xen can hang. It likely needs a timeout + error path (similar
to other GIC polling loops).
> +}
> +
> +void direct_lpi_inv(struct its_device *dev, uint32_t eventid,
nit: dev and eventid is unused
> + uint32_t db_lpi, unsigned int cpu)
> +{
> + void __iomem *rdbase;
> + uint64_t val;
> + /* Register-based LPI invalidation for DB on GICv4.0 */
> + val = FIELD_PREP(GICR_INVLPIR_INTID, db_lpi);
> +
> + rdbase = per_cpu(rbase, cpu);
> + writeq_relaxed(val, rdbase + GICR_INVLPIR);
direct_lpi_inv() writes GICR_INVLPIR unconditionally.
Note that writing GICR_INVLPIR/GICR_INVALLR while GICR_SYNCR.Busy==1
is CONSTRAINED UNPREDICTABLE (the write may be ignored, or the
invalidate may be performed). So we need to ensure Busy==0 before
issuing a new invalidate, not only wait after the write.
Please add a pre-check loop (or serialize callers) to guarantee
GICR_SYNCR.Busy==0 before the INVLPIR write.
proxy lock protects only the ITS proxy path; direct INVLPIR must also
avoid writes while GICR_SYNCR.Busy==1 (pre-check or separate lock)
> + wait_for_syncr(rdbase);
> +}
>
> static void its_vpe_send_inv_db(struct its_vpe *vpe)
> {
> - // struct its_device *dev = vpe_proxy.dev;
> - // unsigned long flags;
> + if ( gic_support_directLPI() )
> + {
> + unsigned int cpu = vpe->col_idx;
We read vpe->col_idx without the vpe lock. Should we use
vpe_to_cpuid_lock() (or otherwise hold vpe_lock) to avoid a
race with affinity updates?
>
> - // spin_lock_irqsave(&vpe_proxy.lock, flags);
> - // gicv4_vpe_db_proxy_map_locked(vpe);
> - // its_send_cmd_inv(dev->hw_its, dev->host_devid, vpe->vpe_proxy_event);
> - // spin_unlock_irqrestore(&vpe_proxy.lock, flags);
> + /* Target the redistributor this VPE is currently known on */
> + direct_lpi_inv(NULL, 0, vpe->vpe_db_lpi, cpu);
I think we should document the ordering constraint at the point where
we select between direct_lpi_inv() and the ITS/proxy invalidation path
(its_vpe_send_inv_db()).
The architecture requires register-based invalidation (GICR_INVLPIR /
GICR_INVALLR + polling GICR_SYNCR.Busy) to be serialized against any
ITS command that affects the same interrupt. Overlapping these
operations is CONSTRAINED UNPREDICTABLE.
> + }
> + else
> + {
> + struct its_device *dev = vpe_proxy.dev;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&vpe_proxy.lock, flags);
> + gicv4_vpe_db_proxy_map_locked(vpe);
> + its_send_cmd_inv(dev->hw_its, dev->host_devid, vpe->vpe_proxy_event);
Both return values are ignored. If map_locked() fails and leaves
vpe_proxy_event at the sentinel value, INV may use an invalid eventid.
At minimum: check map_locked() and INV return codes and avoid issuing
INV on failure.
Need to review and handle errors in other places too.
> + spin_unlock_irqrestore(&vpe_proxy.lock, flags);
> + }
> }
>
> static void its_vpe_inv_db(struct its_vpe *vpe)
> @@ -335,6 +490,7 @@ static void __init its_vpe_teardown(struct its_vpe *vpe)
> unsigned int order;
>
> order = get_order_from_bytes(max(lpi_data.max_host_lpi_ids / 8, (unsigned long)SZ_64K));
> + gicv4_vpe_db_proxy_unmap(vpe);
> its_free_vpeid(vpe->vpe_id);
> free_xenheap_pages(vpe->vpendtable, order);
> xfree(vpe);
> @@ -830,6 +986,43 @@ static void vpe_to_cpuid_unlock(struct its_vpe *vpe, unsigned long *flags)
> spin_unlock_irqrestore(&vpe->vpe_lock, *flags);
> }
>
> +static void gicv4_vpe_db_proxy_move(struct its_vpe *vpe, unsigned int from,
> + unsigned int to)
> +{
> + unsigned long flags;
> +
> + if ( gic_support_directLPI() )
> + {
> + void __iomem *rdbase;
> +
> + rdbase = per_cpu(rbase, from);
> + /* Clear potential pending state on the old redistributor */
> + writeq_relaxed(vpe->vpe_db_lpi, rdbase + GICR_CLRLPIR);
> + wait_for_syncr(rdbase);
> + return;
> + }
> +
> + spin_lock_irqsave(&vpe_proxy.lock, flags);
> +
> + gicv4_vpe_db_proxy_map_locked(vpe);
> +
> + /* MOVI instructs the appropriate Redistributor to move the pending state */
> + its_send_cmd_movi(vpe_proxy.dev->hw_its, vpe_proxy.dev->host_devid,
> + vpe->vpe_proxy_event, to);
handle ret from its_send_cmd_movi
> +
> + /*
> + * ARM spec says that If, after using MOVI to move an interrupt from
> + * collection A to collection B, software moves the same interrupt again
> + * from collection B to collection C, a SYNC command must be used before
> + * the second MOVI for the Redistributor associated with collection A to
> + * ensure correct behavior.
> + * So each time we issue VMOVI, we VSYNC the old VPE for good measure.
> + */
> + WARN_ON(its_send_cmd_sync(vpe_proxy.dev->hw_its, from));
> +
> + spin_unlock_irqrestore(&vpe_proxy.lock, flags);
> +}
> +
> static int gicv4_vpe_set_affinity(struct vcpu *vcpu)
> {
> struct its_vpe *vpe = vcpu->arch.vgic.its_vpe;
> @@ -859,6 +1052,7 @@ static int gicv4_vpe_set_affinity(struct vcpu *vcpu)
> ret = its_send_cmd_vmovp(vpe);
> if ( ret )
> goto out;
> + gicv4_vpe_db_proxy_move(vpe, from, to);
>
> out:
> vpe_to_cpuid_unlock(vpe, &flags);
> @@ -940,4 +1134,3 @@ int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
>
> return its_vlpi_set_doorbell(map, property & LPI_PROP_ENABLED);
> }
> -
> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
> index 0db75309cf..b4d50516ef 100644
> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
> @@ -20,6 +20,13 @@
>
> #include <xen/sizes.h>
>
> +#ifndef FIELD_GET
> +#define FIELD_GET(_mask, _reg) \
> + ((typeof(_mask))(((_reg) & (_mask)) >> (ffs64(_mask) - 1)))
> +#endif
> +
> +#define FIELD_PREP(_mask, _val) \
> + (((typeof(_mask))(_val) << (ffs64(_mask) - 1)) & (_mask))
Guard this macro too, it can be redefined
> /*
> * Additional registers defined in GIC v3.
> * Common GICD registers are defined in gic.h
> diff --git a/xen/arch/arm/include/asm/gic_v3_its.h b/xen/arch/arm/include/asm/gic_v3_its.h
> index dababe97cd..0e82625840 100644
> --- a/xen/arch/arm/include/asm/gic_v3_its.h
> +++ b/xen/arch/arm/include/asm/gic_v3_its.h
> @@ -236,6 +236,11 @@ int its_inv_lpi(struct host_its *its, struct its_device *dev,
> uint32_t eventid, unsigned int cpu);
> int its_send_cmd_mapti(struct host_its *its, uint32_t deviceid,
> uint32_t eventid, uint32_t pintid, uint16_t icid);
> +struct its_device *its_create_device(struct host_its *hw_its,
> + uint32_t host_devid, uint64_t nr_events);
> +int its_send_cmd_movi(struct host_its *its, uint32_t deviceid, uint32_t eventid,
> + uint16_t icid);
> +int its_send_cmd_sync(struct host_its *its, unsigned int cpu);
> #ifdef CONFIG_ACPI
> unsigned long gicv3_its_make_hwdom_madt(const struct domain *d,
> void *base_ptr);
> @@ -326,6 +331,8 @@ void its_vpe_mask_db(struct its_vpe *vpe);
> int gicv4_its_vlpi_unmap(struct pending_irq *pirq);
> int its_vlpi_prop_update(struct pending_irq *pirq, uint8_t property,
> bool needs_inv);
> +void direct_lpi_inv(struct its_device *dev, uint32_t eventid,
> + uint32_t db_lpi, unsigned int cpu);
>
> /* ITS quirks handling. */
> uint64_t gicv3_its_get_cacheability(void);
> diff --git a/xen/arch/arm/include/asm/gic_v4_its.h b/xen/arch/arm/include/asm/gic_v4_its.h
> index 37b6b92f0c..1d800fdbaf 100644
> --- a/xen/arch/arm/include/asm/gic_v4_its.h
> +++ b/xen/arch/arm/include/asm/gic_v4_its.h
> @@ -52,6 +52,7 @@ struct event_vlpi_map {
> };
>
> void gicv4_its_vpeid_allocator_init(void);
> +int gicv4_init_vpe_proxy(void);
>
> #define GICR_VPROPBASER 0x0070
> #define GICR_VPENDBASER 0x0078
> @@ -97,6 +98,10 @@ static inline void gits_write_vpendbaser(uint64_t val, void __iomem *addr)
> }
> #define gits_read_vpendbaser(c) readq_relaxed(c)
>
> +#define GICR_INVLPIR_INTID GENMASK_ULL(31, 0)
> +#define GICR_INVLPIR_VPEID GICR_INVALLR_VPEID
> +#define GICR_INVLPIR_V GICR_INVALLR_V
GICR_INVALLR_VPEID and GICR_INVALLR_V are not defined, so this does
not build as-is.
Also note that the V/VPEID fields in GICR_INVLPIR/INVALLR exist only
for GICv4.1 (FEAT_GICv4p1). For GICv4.0 these bits are RES0.
Best regards,
Mykola
> +
> #endif
>
> /*
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 9ef667decb..558f81818c 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -407,9 +407,11 @@ extern void vgic_check_inflight_irqs_pending(struct vcpu *v,
>
> /* GICV4 functions */
> #ifdef CONFIG_GICV4
> +bool gic_support_directLPI(void);
> bool gic_support_vptValidDirty(void);
> bool gic_is_gicv4(void);
> #else
> +#define gic_support_directLPI() (false)
> #define gic_support_vptValidDirty() (false)
> #define gic_is_gicv4() (false)
> #endif
> --
> 2.51.2
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [RFC PATCH 00/19] GICv4 Support for Xen
2026-02-03 10:01 ` [RFC PATCH 00/19] GICv4 Support for Xen Bertrand Marquis
2026-02-03 12:24 ` Mykyta Poturai
2026-02-13 11:36 ` Mykola Kvach
@ 2026-03-17 9:11 ` Mykola Kvach
2 siblings, 0 replies; 42+ messages in thread
From: Mykola Kvach @ 2026-03-17 9:11 UTC (permalink / raw)
To: Bertrand Marquis
Cc: Mykyta Poturai, xen-devel@lists.xenproject.org,
Stefano Stabellini, Julien Grall, Michal Orzel, Volodymyr Babchuk,
Andrew Cooper, Anthony PERARD, Jan Beulich, Roger Pau Monné
Hi Bertrand,
On Tue, Feb 3, 2026 at 12:02 PM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi Mykyta,
>
> We have a number of series from you which have not been merged yet and
> reviewing them all in parallel might be challenging.
>
> Would you mind giving us a status and maybe priorities on them.
>
> I could list the following series:
> - GICv4
> - CPU Hotplug on arm
> - PCI enumeration on arm
> - IPMMU for pci on arm
> - dom0less for pci passthrough on arm
> - SR-IOV for pvh
> - SMMU for pci on arm
> - MSI injection on arm
> - suspend to ram on arm
>
> There might be others feel free to complete the list.
>
> On GICv4...
>
> > On 2 Feb 2026, at 17:14, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> >
> > This series introduces GICv4 direct LPI injection for Xen.
> >
> > Direct LPI injection relies on the GIC tracking the mapping between physical and
> > virtual CPUs. Each VCPU requires a VPE that is created and registered with the
> > GIC via the `VMAPP` ITS command. The GIC is then informed of the current
> > VPE-to-PCPU placement by programming `VPENDBASER` and `VPROPBASER` in the
> > appropriate redistributor. LPIs are associated with VPEs through the `VMAPTI`
> > ITS command, after which the GIC handles delivery without trapping into the
> > hypervisor for each interrupt.
> >
> > When a VPE is not scheduled but has pending interrupts, the GIC raises a per-VPE
> > doorbell LPI. Doorbells are owned by the hypervisor and prompt rescheduling so
> > the VPE can drain its pending LPIs.
> >
> > Because GICv4 lacks a native doorbell invalidation mechanism, this series
> > includes a helper that invalidates doorbell LPIs via synthetic “proxy” devices,
> > following the approach used until GICv4.1.
> >
> > All of this work is mostly based on the work of Penny Zheng
> > <penny.zheng@arm.com> and Luca Fancellu <luca.fancellu@arm.com>. And also from
> > Linux patches by Mark Zyngier.
> >
> > Some patches are still a little rough and need some styling fixes and more
> > testing, as all of them needed to be carved line by line from a giant ~4000 line
> > patch. This RFC is directed mostly to get a general idea if the proposed
> > approach is suitable and OK with everyone. And there is still an open question
> > of how to handle Signed-off-by lines for Penny and Luca, since they have not
> > indicated their preference yet.
>
> I would like to ask how much performance benefits you could
> have with this.
> Adding GICv4 support is adding a lot of code which will have to be maintained
> and tested and there should be a good improvement to justify this.
>
> Did you do some benchmarks ? what are the results ?
>
> At the time where we started to work on that at Arm, we ended up in the conclusion
> that the complexity in Xen compared to the benefit was not justifying it hence why
> this work was stopped in favor of other features that we thought would be more
> beneficial to Xen (like PCI passthrough or SMMUv3).
I have been asked to run benchmarks for this series, so here is a short
update from my side.
Test setup:
- AWS c7g bare metal
- Linux bare-metal reference and Xen dom0 runs
- fio random-read workloads on an NVMe-backed EBS volume (gp3, 160G, 80k iops)
- Main workloads:
- 4k, iodepth=1
- 16k, iodepth=1
- 4k, iodepth=4
- 4k, iodepth=1, numjobs=4
- 5 repetitions per configuration, looking mainly at median values
- Main Xen comparison was done with the default scheduler (credit2),
direct LPIs OFF vs ON
Summary:
- With credit2, enabling direct LPIs gave a small but repeatable IOPS
improvement across all tested workloads, roughly in the 0.8-1.1% range.
- Mean completion latency also improved consistently.
- The clearest gain was in tail latency. In the 4k randread,
iodepth=1, numjobs=4 case, p99.9 improved by about 41% and p99.99 by
about 34% with direct LPIs enabled.
- In this setup, switching from credit2 to null did not materially change
median throughput, so the observed improvement appears to come primarily
from the interrupt delivery path rather than from the scheduler choice.
A few caveats:
- This was a low-contention setup with only dom0 using 8 CPUs, so it did not
exercise heavy VCPU migration or scheduler pressure.
- I also tried an artificially constrained NVMe host queue depth
configuration, but I am treating that only as a stress/control case and
not as the main result.
A full benchmark report is available here:
https://github.com/xakep-amatop/giv4-benchmark/blob/main/report.pdf
The same repository also contains the raw benchmark result archives used for
the analysis.
So, based on these measurements, there does appear to be a measurable
benefit from direct LPI injection, with the strongest effect showing up in
tail latency rather than in median throughput.
If you need any additional benchmark results or specific test cases, please
let me know.
Best regards,
Mykola
>
> Cheers
> Bertrand
>
^ permalink raw reply [flat|nested] 42+ messages in thread
end of thread, other threads:[~2026-03-17 9:11 UTC | newest]
Thread overview: 42+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-02-02 16:14 [RFC PATCH 00/19] GICv4 Support for Xen Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 01/19] arm/gicv4 add management structure definitions Mykyta Poturai
2026-02-11 8:28 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 02/19] arm/gicv4-its: Add GICv4 ITS command definitions Mykyta Poturai
2026-02-11 8:28 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 03/19] arm/its: Export struct its_device Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 04/19] arm/its: Add vlpi configuration Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 06/19] arm/gicv4-its: Add VLPI map/unmap operations Mykyta Poturai
2026-02-11 8:31 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 07/19] xen/domain: Alloc enough pages for VCPU struct Mykyta Poturai
2026-02-02 16:45 ` Jan Beulich
2026-02-02 16:14 ` [RFC PATCH 05/19] arm/irq: Add hw flag to pending_irq Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 09/19] arm/its: Implement LPI invalidation Mykyta Poturai
2026-02-02 16:14 ` [RFC PATCH 08/19] arm/gic: Keep track of GIC features Mykyta Poturai
2026-02-12 7:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 10/19] arm/its: Keep track of BASER regs Mykyta Poturai
2026-02-12 7:52 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 11/19] arm/its: Add ITS VM and VPE allocation/teardown Mykyta Poturai
2026-02-18 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 12/19] arm/gic: Add VPENDBASER/VPROPBASER accessors Mykyta Poturai
2026-02-18 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 13/19] arm/gic: VPE scheduling Mykyta Poturai
2026-02-18 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 16/19] arm/gic: Fix LR group handling for GICv4 Mykyta Poturai
2026-02-18 21:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 15/19] arm: Add gicv4 to domain creation Mykyta Poturai
2026-02-18 21:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 14/19] arm/its: VPE affinity changes Mykyta Poturai
2026-02-18 21:00 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 17/19] arm/gicv4: Handle doorbells Mykyta Poturai
2026-02-19 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 19/19] arm/gicv4: Add GICv4 to the build system Mykyta Poturai
2026-02-19 11:30 ` Mykola Kvach
2026-02-02 16:14 ` [RFC PATCH 18/19] arm/gic: Add VPE proxy support Mykyta Poturai
2026-02-19 16:00 ` Mykola Kvach
2026-02-03 10:01 ` [RFC PATCH 00/19] GICv4 Support for Xen Bertrand Marquis
2026-02-03 12:24 ` Mykyta Poturai
2026-02-03 16:27 ` Bertrand Marquis
2026-02-13 11:36 ` Mykola Kvach
2026-02-13 11:48 ` Julien Grall
2026-02-13 12:18 ` Mykola Kvach
2026-03-17 9:11 ` Mykola Kvach
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.