* [PATCH v5 00/12] Introduce eSPI support
@ 2025-08-29 16:06 Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations Leonid Komarianskyi
` (11 more replies)
0 siblings, 12 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Anthony PERARD, Oleksii Kurochko, Community Manager
Hello everyone!
This is V5 of the patch series to introduce eSPI support, which contains
fixes and improvements based on the comments received in V4.
The main changes in this version compared to V4 are related to removing
unnecessary #ifdefs that previously prevented the use of certain defines
and macros, leading to an overuse of #ifdefs. After refactoring, it is
now possible to reuse the existing code for vGIC eSPI emulation, which
improves readability and simplifies future changes. As some patches that
had already been reviewed were modified, I would ask Volodymyr and
Oleksandr to take a look at the updated versions of:
[4/12] xen/arm/irq: add handling for IRQs in the eSPI range
[5/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI
The changes made in these patches allowed me to significantly simplify
the following patch:
[10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
For the other patches, I have addressed comments and added reviewed-by
or acked-by tags accordingly, based on the received comments.
Summarized description:
This patch series adds support for the extended shared peripheral
interrupt (eSPI) range (INTIDs 4096-5119 [2](ranges of INTIDs)) for Xen
and guest domains. The implementation uses a generic approach to handle
eSPIs, similar to regular SPIs, while maintaining compatibility with the
existing SPI range. Functionality remains unchanged for setups that do
not require eSPIs.
The series includes:
1) General refactoring of common IRQ operations with GIC registers to
improve code readability, simplify further maintenance and prepare the
key functions for eSPI implementation.
2) Introducing a new Kconfig option (default n) to enable or disable
eSPI support. Disabling this option prevents unnecessary resource
allocation for setups that do not require eSPIs.
3) Adding additional resources to store required information and operate
with up to 1024 interrupts from eSPI range.
4) Adjusting assertions and checks to pass verification for INTIDs in
the eSPI range.
5) Configuration of eSPI-specific registers during GIC initialization
for systems with GICv3.1+ hardware.
6) Enables eSPI MMIO emulation for vGIC, allowing guest domains to
access and operate within the eSPI's INTIDs.
7) Updating documentation and CHANGELOG to reflect the changes made for eSPI
support.
Also, to simplify reviewing, please find below link to unsquashed patches, that
are on top of every patch, that is changed in the series, compared to V4:
https://github.com/LKomaryanskiy/xen/commits/espi-support-master-upstream-v5-unsquashed/
Changes in V5:
- individual changes in patches
Link on V4:
- https://lists.xenproject.org/archives/html/xen-devel/2025-08/msg01767.html
Changes in V4:
- added a patch for documentation
- individual changes in patches
Link on V3:
- https://lists.xenproject.org/archives/html/xen-devel/2025-08/msg01628.html
Changes in V3:
- added a patch to update CHANGELOG.md
- individual changes in patches
Link on V2:
- https://lists.xenproject.org/archives/html/xen-devel/2025-08/msg00372.html
Changes in V2:
- added 2 more patches to implement helper
functions for gic/vgic:
xen/arm: gic: implement helper functions for INTID checks
xen/arm: vgic: implement helper functions for virq checks
- removed 2 patches:
xen/arm/irq: allow assignment/releasing of eSPI interrupts
xen/arm: gic/irq: permit routing of eSPI interrupts to Xen and domains
since their functionality can be moved to appropriate patches after
introducing patches with helper functions
- individual changes in patches
Link on V1:
- https://lists.xenproject.org/archives/html/xen-devel/2025-07/msg01809.html
Leonid Komarianskyi (12):
xen/arm: gicv3: refactor obtaining GIC addresses for common operations
xen/arm: gic: implement helper functions for INTID checks
xen/arm: vgic: implement helper functions for virq checks
xen/arm/irq: add handling for IRQs in the eSPI range
xen/arm: gicv3: implement handling of GICv3.1 eSPI
xen/arm/irq: allow eSPI processing in the gic_interrupt function
xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing
xen/arm: vgic: add resource management for extended SPIs
xen/arm: domain_build/dom0less-build: adjust domains config to support
eSPIs
xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
doc/man: update description for nr_spis with eSPI
CHANGELOG.md: add mention of GICv3.1 eSPI support
CHANGELOG.md | 2 +
docs/man/xl.cfg.5.pod.in | 13 +-
xen/arch/arm/Kconfig | 8 +
xen/arch/arm/dom0less-build.c | 2 +-
xen/arch/arm/domain_build.c | 2 +-
xen/arch/arm/gic-v3.c | 195 +++++++++++++++++----
xen/arch/arm/gic.c | 8 +-
xen/arch/arm/include/asm/gic.h | 31 ++++
xen/arch/arm/include/asm/gic_v3_defs.h | 40 ++++-
xen/arch/arm/include/asm/irq.h | 25 +++
xen/arch/arm/include/asm/vgic.h | 42 +++++
xen/arch/arm/irq.c | 62 ++++++-
xen/arch/arm/vgic-v3.c | 198 +++++++++++++++++----
xen/arch/arm/vgic.c | 232 ++++++++++++++++++++++++-
xen/arch/arm/vgic/vgic.c | 5 +
15 files changed, 778 insertions(+), 87 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-31 12:42 ` Oleksandr Tyshchenko
2025-08-29 16:06 ` [PATCH v5 02/12] xen/arm: gic: implement helper functions for INTID checks Leonid Komarianskyi
` (10 subsequent siblings)
11 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Volodymyr Babchuk, Julien Grall
Currently, many common functions perform the same operations to calculate
GIC register addresses. This patch consolidates the similar code into
a separate helper function to improve maintainability and reduce duplication.
This refactoring also simplifies the implementation of eSPI support in future
changes.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
Changes in V5:
- fixed a minor nit: changed %d to %u in the warning message because the
IRQ number cannot be less than zero
- added acked-by from Julien Grall
Changes in V4:
- no changes
Changes in V3:
- changed panic() in get_addr_by_offset() to printing warning and
ASSERT_UNREACHABLE()
- added verification of return pointer from get_addr_by_offset() in the
callers
- moved invocation of get_addr_by_offset() from spinlock guards, since
it is not necessarry
- added RB from Volodymyr Babchuk
Changes in V2:
- no changes
---
xen/arch/arm/gic-v3.c | 114 +++++++++++++++++++++++----------
xen/arch/arm/include/asm/irq.h | 1 +
2 files changed, 81 insertions(+), 34 deletions(-)
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index cd3e1acf79..29b7f68cba 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -445,17 +445,67 @@ static void gicv3_dump_state(const struct vcpu *v)
}
}
+static void __iomem *get_addr_by_offset(struct irq_desc *irqd, u32 offset)
+{
+ switch ( irqd->irq )
+ {
+ case 0 ... (NR_GIC_LOCAL_IRQS - 1):
+ switch ( offset )
+ {
+ case GICD_ISENABLER:
+ case GICD_ICENABLER:
+ case GICD_ISPENDR:
+ case GICD_ICPENDR:
+ case GICD_ISACTIVER:
+ case GICD_ICACTIVER:
+ return (GICD_RDIST_SGI_BASE + offset);
+ case GICD_ICFGR:
+ return (GICD_RDIST_SGI_BASE + GICR_ICFGR1);
+ case GICD_IPRIORITYR:
+ return (GICD_RDIST_SGI_BASE + GICR_IPRIORITYR0 + irqd->irq);
+ default:
+ break;
+ }
+ case NR_GIC_LOCAL_IRQS ... SPI_MAX_INTID:
+ switch ( offset )
+ {
+ case GICD_ISENABLER:
+ case GICD_ICENABLER:
+ case GICD_ISPENDR:
+ case GICD_ICPENDR:
+ case GICD_ISACTIVER:
+ case GICD_ICACTIVER:
+ return (GICD + offset + (irqd->irq / 32) * 4);
+ case GICD_ICFGR:
+ return (GICD + GICD_ICFGR + (irqd->irq / 16) * 4);
+ case GICD_IROUTER:
+ return (GICD + GICD_IROUTER + irqd->irq * 8);
+ case GICD_IPRIORITYR:
+ return (GICD + GICD_IPRIORITYR + irqd->irq);
+ default:
+ break;
+ }
+ default:
+ break;
+ }
+
+ /* Something went wrong, we shouldn't be able to reach here */
+ printk(XENLOG_WARNING "GICv3: WARNING: Invalid offset 0x%x for IRQ#%u",
+ offset, irqd->irq);
+ ASSERT_UNREACHABLE();
+
+ return NULL;
+}
+
static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
{
u32 mask = 1U << (irqd->irq % 32);
- void __iomem *base;
+ void __iomem *addr = get_addr_by_offset(irqd, offset);
- if ( irqd->irq < NR_GIC_LOCAL_IRQS )
- base = GICD_RDIST_SGI_BASE;
- else
- base = GICD;
+ if ( addr == NULL )
+ return;
- writel_relaxed(mask, base + offset + (irqd->irq / 32) * 4);
+ writel_relaxed(mask, addr);
if ( wait_for_rwp )
gicv3_wait_for_rwp(irqd->irq);
@@ -463,15 +513,12 @@ static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
static bool gicv3_peek_irq(struct irq_desc *irqd, u32 offset)
{
- void __iomem *base;
- unsigned int irq = irqd->irq;
+ void __iomem *addr = get_addr_by_offset(irqd, offset);
- if ( irq >= NR_GIC_LOCAL_IRQS)
- base = GICD + (irq / 32) * 4;
- else
- base = GICD_RDIST_SGI_BASE;
+ if ( addr == NULL )
+ return false;
- return !!(readl(base + offset) & (1U << (irq % 32)));
+ return !!(readl(addr) & (1U << (irqd->irq % 32)));
}
static void gicv3_unmask_irq(struct irq_desc *irqd)
@@ -558,30 +605,28 @@ static inline uint64_t gicv3_mpidr_to_affinity(int cpu)
static void gicv3_set_irq_type(struct irq_desc *desc, unsigned int type)
{
uint32_t cfg, actual, edgebit;
- void __iomem *base;
- unsigned int irq = desc->irq;
+ void __iomem *addr;
/* SGI's are always edge-triggered not need to call GICD_ICFGR0 */
- ASSERT(irq >= NR_GIC_SGI);
+ ASSERT(desc->irq >= NR_GIC_SGI);
- spin_lock(&gicv3.lock);
+ addr = get_addr_by_offset(desc, GICD_ICFGR);
+ if ( addr == NULL )
+ return;
- if ( irq >= NR_GIC_LOCAL_IRQS)
- base = GICD + GICD_ICFGR + (irq / 16) * 4;
- else
- base = GICD_RDIST_SGI_BASE + GICR_ICFGR1;
+ spin_lock(&gicv3.lock);
- cfg = readl_relaxed(base);
+ cfg = readl_relaxed(addr);
- edgebit = 2u << (2 * (irq % 16));
+ edgebit = 2u << (2 * (desc->irq % 16));
if ( type & IRQ_TYPE_LEVEL_MASK )
cfg &= ~edgebit;
else if ( type & IRQ_TYPE_EDGE_BOTH )
cfg |= edgebit;
- writel_relaxed(cfg, base);
+ writel_relaxed(cfg, addr);
- actual = readl_relaxed(base);
+ actual = readl_relaxed(addr);
if ( ( cfg & edgebit ) ^ ( actual & edgebit ) )
{
printk(XENLOG_WARNING "GICv3: WARNING: "
@@ -600,16 +645,13 @@ static void gicv3_set_irq_type(struct irq_desc *desc, unsigned int type)
static void gicv3_set_irq_priority(struct irq_desc *desc,
unsigned int priority)
{
- unsigned int irq = desc->irq;
+ void __iomem *addr = get_addr_by_offset(desc, GICD_IPRIORITYR);
- spin_lock(&gicv3.lock);
-
- /* Set priority */
- if ( irq < NR_GIC_LOCAL_IRQS )
- writeb_relaxed(priority, GICD_RDIST_SGI_BASE + GICR_IPRIORITYR0 + irq);
- else
- writeb_relaxed(priority, GICD + GICD_IPRIORITYR + irq);
+ if ( addr == NULL )
+ return;
+ spin_lock(&gicv3.lock);
+ writeb_relaxed(priority, addr);
spin_unlock(&gicv3.lock);
}
@@ -1273,6 +1315,10 @@ static void gicv3_irq_set_affinity(struct irq_desc *desc, const cpumask_t *mask)
{
unsigned int cpu;
uint64_t affinity;
+ void __iomem *addr = get_addr_by_offset(desc, GICD_IROUTER);
+
+ if ( addr == NULL )
+ return;
ASSERT(!cpumask_empty(mask));
@@ -1284,7 +1330,7 @@ static void gicv3_irq_set_affinity(struct irq_desc *desc, const cpumask_t *mask)
affinity &= ~GICD_IROUTER_SPI_MODE_ANY;
if ( desc->irq >= NR_GIC_LOCAL_IRQS )
- writeq_relaxed_non_atomic(affinity, (GICD + GICD_IROUTER + desc->irq * 8));
+ writeq_relaxed_non_atomic(affinity, addr);
spin_unlock(&gicv3.lock);
}
diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
index fce7e42a33..5bc6475eb4 100644
--- a/xen/arch/arm/include/asm/irq.h
+++ b/xen/arch/arm/include/asm/irq.h
@@ -29,6 +29,7 @@ struct arch_irq_desc {
*/
#define NR_IRQS 1024
+#define SPI_MAX_INTID 1019
#define LPI_OFFSET 8192
/* LPIs are always numbered starting at 8192, so 0 is a good invalid case. */
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 02/12] xen/arm: gic: implement helper functions for INTID checks
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 03/12] xen/arm: vgic: implement helper functions for virq checks Leonid Komarianskyi
` (9 subsequent siblings)
11 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Volodymyr Babchuk, Julien Grall
Introduced two new helper functions: gic_is_valid_line and
gic_is_spi. The first function helps determine whether an IRQ
number is less than the number of lines supported by hardware. The
second function additionally checks if the IRQ number falls within the
SPI range. Also, updated the appropriate checks to use these new helper
functions.
The current checks for the real GIC are very similar to those for the
vGIC but serve a different purpose. For GIC-related code, the interrupt
numbers should be validated based on whether the hardware can operate
with such interrupts. On the other hand, for the vGIC, the indexes must
also be verified to ensure they are available for a specific domain. The
first reason for introducing these helper functions is to avoid
potential confusion with vGIC-related checks. The second reason is to
consolidate similar code into separate functions, which can be more
easily extended by additional conditions, e.g., when implementing
extended SPI interrupts.
The changes, which replace open-coded checks with the use of the new
helper functions, do not introduce any functional changes, as the helper
functions follow the current IRQ index verification logic.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
Changes in V5:
- fixed a minor nit: moved the existing comment to the line above to fix
formatting that exceeded 80 characters
- added acked-by from Julien Grall
Changes in V4:
- removed redundant parentheses
- added reviewed-by from Volodymyr Babchuk
Changes in V3:
- renamed gic_is_valid_irq to gic_is_valid_line and gic_is_shared_irq to
gic_is_spi
- updated commit message
Changes in V2:
- introduced this patch
Changes for V4:
Changes in V4:
- removed redundant parentheses
- added reviewed-by from Volodymyr Babchuk
---
xen/arch/arm/gic.c | 3 ++-
xen/arch/arm/include/asm/gic.h | 9 +++++++++
xen/arch/arm/irq.c | 2 +-
3 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index e80fe0ca24..4bb11960ee 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -111,7 +111,8 @@ static void gic_set_irq_priority(struct irq_desc *desc, unsigned int priority)
void gic_route_irq_to_xen(struct irq_desc *desc, unsigned int priority)
{
ASSERT(priority <= 0xff); /* Only 8 bits of priority */
- ASSERT(desc->irq < gic_number_lines());/* Can't route interrupts that don't exist */
+ /* Can't route interrupts that don't exist */
+ ASSERT(gic_is_valid_line(desc->irq));
ASSERT(test_bit(_IRQ_DISABLED, &desc->status));
ASSERT(spin_is_locked(&desc->lock));
diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
index 541f0eeb80..3fcee42675 100644
--- a/xen/arch/arm/include/asm/gic.h
+++ b/xen/arch/arm/include/asm/gic.h
@@ -306,6 +306,15 @@ extern void gic_dump_vgic_info(struct vcpu *v);
/* Number of interrupt lines */
extern unsigned int gic_number_lines(void);
+static inline bool gic_is_valid_line(unsigned int irq)
+{
+ return irq < gic_number_lines();
+}
+
+static inline bool gic_is_spi(unsigned int irq)
+{
+ return irq >= NR_LOCAL_IRQS && gic_is_valid_line(irq);
+}
/* IRQ translation function for the device tree */
int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 03fbb90c6c..7dd5a2a453 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -415,7 +415,7 @@ err:
bool is_assignable_irq(unsigned int irq)
{
/* For now, we can only route SPIs to the guest */
- return (irq >= NR_LOCAL_IRQS) && (irq < gic_number_lines());
+ return gic_is_spi(irq);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 03/12] xen/arm: vgic: implement helper functions for virq checks
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 02/12] xen/arm: gic: implement helper functions for INTID checks Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range Leonid Komarianskyi
` (8 subsequent siblings)
11 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Oleksandr Tyshchenko, Volodymyr Babchuk, Julien Grall
Introduced two new helper functions for vGIC: vgic_is_valid_line and
vgic_is_spi. The functions are similar to the newly introduced
gic_is_valid_line and gic_is_spi, but they verify whether a vIRQ
is available for a specific domain, while GIC-specific functions
validate INTIDs for the real GIC hardware. For example, the GIC may
support all 992 SPI lines, but the domain may use only some part of them
(e.g., 640), depending on the highest IRQ number defined in the domain
configuration. Therefore, for vGIC-related code and checks, the
appropriate functions should be used. Also, updated the appropriate
checks to use these new helper functions.
The purpose of introducing new helper functions for vGIC is essentially
the same as for GIC: to avoid potential confusion with GIC-related
checks and to consolidate similar code into separate functions, which
can be more easily extended by additional conditions, e.g., when
implementing extended SPI interrupts.
Only the validation change in vgic_inject_irq may affect existing
functionality, as it currently checks whether the vIRQ is less than or
equal to vgic_num_irqs. Since IRQ indexes start from 0 (where 32 is the
first SPI), the check should behave consistently with similar logic in
other places and should check if the vIRQ number is less than
vgic_num_irqs. The remaining changes, which replace open-coded checks
with the use of these new helper functions, do not introduce any
functional changes, as the helper functions follow the current vIRQ
index verification logic.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
Reviewed-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
Changes in V5:
- added reviewed-by from Oleksandr Tyshchenko and from Volodymyr Babchuk
- added acked-by from Julien Grall
Changes in V4:
- removed redundant parentheses
Changes in V3:
- renamed vgic_is_valid_irq to vgic_is_valid_line and vgic_is_shared_irq
to vgic_is_spi
- added vgic_is_valid_line implementation for new-vgic, because
vgic_is_valid_line is called from generic code. It is necessary to fix
the build for new-vgic.
- updated commit message
Changes in V2:
- introduced this patch
---
xen/arch/arm/gic.c | 3 +--
xen/arch/arm/include/asm/vgic.h | 7 +++++++
xen/arch/arm/irq.c | 4 ++--
xen/arch/arm/vgic.c | 10 ++++++++--
xen/arch/arm/vgic/vgic.c | 5 +++++
5 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 4bb11960ee..9469c9d08c 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -134,8 +134,7 @@ int gic_route_irq_to_guest(struct domain *d, unsigned int virq,
ASSERT(spin_is_locked(&desc->lock));
/* Caller has already checked that the IRQ is an SPI */
- ASSERT(virq >= 32);
- ASSERT(virq < vgic_num_irqs(d));
+ ASSERT(vgic_is_spi(d, virq));
ASSERT(!is_lpi(virq));
ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 35c0c6a8b0..3e7cbbb196 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -335,6 +335,13 @@ extern void vgic_check_inflight_irqs_pending(struct vcpu *v,
/* Default number of vGIC SPIs. 32 are substracted to cover local IRQs. */
#define VGIC_DEF_NR_SPIS (min(gic_number_lines(), VGIC_MAX_IRQS) - 32)
+extern bool vgic_is_valid_line(struct domain *d, unsigned int virq);
+
+static inline bool vgic_is_spi(struct domain *d, unsigned int virq)
+{
+ return virq >= NR_LOCAL_IRQS && vgic_is_valid_line(d, virq);
+}
+
/*
* Allocate a guest VIRQ
* - spi == 0 => allocate a PPI. It will be the same on every vCPU
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 7dd5a2a453..b8eccfc924 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -442,7 +442,7 @@ int route_irq_to_guest(struct domain *d, unsigned int virq,
unsigned long flags;
int retval = 0;
- if ( virq >= vgic_num_irqs(d) )
+ if ( !vgic_is_valid_line(d, virq) )
{
printk(XENLOG_G_ERR
"the vIRQ number %u is too high for domain %u (max = %u)\n",
@@ -560,7 +560,7 @@ int release_guest_irq(struct domain *d, unsigned int virq)
int ret;
/* Only SPIs are supported */
- if ( virq < NR_LOCAL_IRQS || virq >= vgic_num_irqs(d) )
+ if ( !vgic_is_spi(d, virq) )
return -EINVAL;
desc = vgic_get_hw_irq_desc(d, NULL, virq);
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index c563ba93af..2bbf4d99aa 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -24,6 +24,12 @@
#include <asm/gic.h>
#include <asm/vgic.h>
+
+bool vgic_is_valid_line(struct domain *d, unsigned int virq)
+{
+ return virq < vgic_num_irqs(d);
+}
+
static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
unsigned int rank)
{
@@ -582,7 +588,7 @@ void vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
if ( !v )
{
/* The IRQ needs to be an SPI if no vCPU is specified. */
- ASSERT(virq >= 32 && virq <= vgic_num_irqs(d));
+ ASSERT(vgic_is_spi(d, virq));
v = vgic_get_target_vcpu(d->vcpu[0], virq);
};
@@ -659,7 +665,7 @@ bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr)
bool vgic_reserve_virq(struct domain *d, unsigned int virq)
{
- if ( virq >= vgic_num_irqs(d) )
+ if ( !vgic_is_valid_line(d, virq) )
return false;
return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 6cabd0496d..b2c0e1873a 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -718,6 +718,11 @@ bool vgic_reserve_virq(struct domain *d, unsigned int virq)
return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
}
+bool vgic_is_valid_line(struct domain *d, unsigned int virq)
+{
+ return virq < vgic_num_irqs(d);
+}
+
int vgic_allocate_virq(struct domain *d, bool spi)
{
int first, end;
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (2 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 03/12] xen/arm: vgic: implement helper functions for virq checks Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-29 19:45 ` Volodymyr Babchuk
2025-09-01 16:11 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI Leonid Komarianskyi
` (7 subsequent siblings)
11 siblings, 2 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Currently, Xen does not support eSPI interrupts, leading
to a data abort when such interrupts are defined in the DTS.
This patch introduces a separate array to initialize up to
1024 interrupt descriptors in the eSPI range and adds the
necessary defines and helper function. These changes lay the
groundwork for future implementation of full eSPI interrupt
support. As this GICv3.1 feature is not required by all vendors,
all changes are guarded by ifdefs, depending on the corresponding
Kconfig option.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
---
Changes in V5:
- no functional changes introduced by this version compared with V4, only
minor fixes and removal of ifdefs for macroses
- added TODO comment, suggested by Oleksandr Tyshchenko
- changed int to unsigned int for irqs
- removed ifdefs for eSPI-specific defines and macros to reduce the
number of ifdefs and code duplication in further changes
- removed reviewed-by as moving defines from ifdefs requires additional
confirmation from reviewers
Changes in V4:
- removed redundant line with 'default n' in Kconfig, as it is disabled
by default, without explicit specification
- added reviewed-by from Volodymyr Babchuk
Changes in V3:
- introduced a new define NR_ESPI_IRQS to avoid confusion, like in the
case of using NR_IRQS for espi_desc array
- implemented helper functions espi_to_desc and init_espi_data to make
it possible to add stubs with the same name, and as a result, reduce
the number of #ifdefs
- disable CONFIG_GICV3_ESPI default value to n
Changes in V2:
- use (ESPI_MAX_INTID + 1) instead of (ESPI_BASE_INTID + NR_IRQS)
- remove unnecessary comment for nr_irqs initialization
---
xen/arch/arm/Kconfig | 8 +++++
xen/arch/arm/include/asm/irq.h | 24 +++++++++++++++
xen/arch/arm/irq.c | 56 +++++++++++++++++++++++++++++++++-
3 files changed, 87 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 17df147b25..43b05533b1 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -135,6 +135,14 @@ config GICV3
Driver for the ARM Generic Interrupt Controller v3.
If unsure, use the default setting.
+config GICV3_ESPI
+ bool "Extended SPI range support"
+ depends on GICV3 && !NEW_VGIC
+ help
+ Allow Xen and domains to use interrupt numbers from the extended SPI
+ range, from 4096 to 5119. This feature is introduced in GICv3.1
+ architecture.
+
config HAS_ITS
bool "GICv3 ITS MSI controller support (UNSUPPORTED)" if UNSUPPORTED
depends on GICV3 && !NEW_VGIC && !ARM_32
diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
index 5bc6475eb4..4443799648 100644
--- a/xen/arch/arm/include/asm/irq.h
+++ b/xen/arch/arm/include/asm/irq.h
@@ -32,6 +32,13 @@ struct arch_irq_desc {
#define SPI_MAX_INTID 1019
#define LPI_OFFSET 8192
+#define ESPI_BASE_INTID 4096
+#define ESPI_MAX_INTID 5119
+#define NR_ESPI_IRQS 1024
+
+#define ESPI_INTID2IDX(intid) ((intid) - ESPI_BASE_INTID)
+#define ESPI_IDX2INTID(idx) ((idx) + ESPI_BASE_INTID)
+
/* LPIs are always numbered starting at 8192, so 0 is a good invalid case. */
#define INVALID_LPI 0
@@ -39,7 +46,15 @@ struct arch_irq_desc {
#define INVALID_IRQ 1023
extern const unsigned int nr_irqs;
+#ifdef CONFIG_GICV3_ESPI
+/*
+ * This will also cover the eSPI range, as some critical devices
+ * for booting Xen (e.g., serial) may use this type of interrupts.
+ */
+#define nr_static_irqs (ESPI_MAX_INTID + 1)
+#else
#define nr_static_irqs NR_IRQS
+#endif
struct irq_desc;
struct irqaction;
@@ -55,6 +70,15 @@ static inline bool is_lpi(unsigned int irq)
return irq >= LPI_OFFSET;
}
+static inline bool is_espi(unsigned int irq)
+{
+#ifdef CONFIG_GICV3_ESPI
+ return (irq >= ESPI_BASE_INTID && irq <= ESPI_MAX_INTID);
+#else
+ return false;
+#endif
+}
+
#define domain_pirq_to_irq(d, pirq) (pirq)
bool is_assignable_irq(unsigned int irq);
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index b8eccfc924..61c915c3f9 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -19,7 +19,11 @@
#include <asm/gic.h>
#include <asm/vgic.h>
+#ifdef CONFIG_GICV3_ESPI
+const unsigned int nr_irqs = ESPI_MAX_INTID + 1;
+#else
const unsigned int nr_irqs = NR_IRQS;
+#endif
static unsigned int local_irqs_type[NR_LOCAL_IRQS];
static DEFINE_SPINLOCK(local_irqs_type_lock);
@@ -46,6 +50,53 @@ void irq_end_none(struct irq_desc *irq)
}
static irq_desc_t irq_desc[NR_IRQS - NR_LOCAL_IRQS];
+#ifdef CONFIG_GICV3_ESPI
+/*
+ * TODO: Consider allocating an array dynamically if
+ * there is a need to enable GICV3_ESPI by default.
+ */
+static irq_desc_t espi_desc[NR_ESPI_IRQS];
+
+static struct irq_desc *espi_to_desc(unsigned int irq)
+{
+ return &espi_desc[ESPI_INTID2IDX(irq)];
+}
+
+static int __init init_espi_data(void)
+{
+ unsigned int irq;
+
+ for ( irq = ESPI_BASE_INTID; irq <= ESPI_MAX_INTID; irq++ )
+ {
+ struct irq_desc *desc = irq_to_desc(irq);
+ int rc = init_one_irq_desc(desc);
+
+ if ( rc )
+ return rc;
+
+ desc->irq = irq;
+ desc->action = NULL;
+ }
+
+ return 0;
+}
+#else
+/*
+ * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
+ * because in this case, is_espi will always return false.
+ */
+static struct irq_desc *espi_to_desc(unsigned int irq)
+{
+ ASSERT_UNREACHABLE();
+ return NULL;
+}
+
+static int __init init_espi_data(void)
+{
+ return 0;
+}
+#endif
+
static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
struct irq_desc *__irq_to_desc(unsigned int irq)
@@ -53,6 +104,9 @@ struct irq_desc *__irq_to_desc(unsigned int irq)
if ( irq < NR_LOCAL_IRQS )
return &this_cpu(local_irq_desc)[irq];
+ if ( is_espi(irq) )
+ return espi_to_desc(irq);
+
return &irq_desc[irq-NR_LOCAL_IRQS];
}
@@ -79,7 +133,7 @@ static int __init init_irq_data(void)
desc->action = NULL;
}
- return 0;
+ return init_espi_data();
}
static int init_local_irq_data(unsigned int cpu)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (3 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-29 19:55 ` Volodymyr Babchuk
2025-08-31 14:56 ` Oleksandr Tyshchenko
2025-08-29 16:06 ` [PATCH v5 06/12] xen/arm/irq: allow eSPI processing in the gic_interrupt function Leonid Komarianskyi
` (6 subsequent siblings)
11 siblings, 2 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Introduced appropriate register definitions, helper macros,
and initialization of required GICv3.1 distributor registers
to support eSPI. This type of interrupt is handled in the
same way as regular SPI interrupts, with the following
differences:
1) eSPIs can have up to 1024 interrupts, starting from the
beginning of the range, whereas regular SPIs use INTIDs from
32 to 1019, totaling 988 interrupts;
2) eSPIs start at INTID 4096, necessitating additional interrupt
index conversion during register operations.
In case if appropriate config is disabled, or GIC HW doesn't
support eSPI, the existing functionality will remain the same.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
---
Changes in V5:
- fixed minor nits, no functional changes: changed u32 to uint32_t and
added a comment noting that the configuration for eSPIs is the same as
for regular SPIs
- removed ifdefs for eSPI-specific offsets to reduce the number of
ifdefs and code duplication in further changes
- removed reviewed-by as moving offset from ifdefs requires additional
confirmation from reviewers
Changes in V4:
- added offsets for GICD_IGRPMODRnE and GICD_NSACRnE that are required
for vGIC emulation
- added a log banner with eSPI information, similar to the one for
regular SPI
- added newline after ifdef and before gic_is_valid_line
- added reviewed-by from Volodymyr Babchuk
Changes in V3:
- add __init attribute to gicv3_dist_espi_common_init
- change open-codded eSPI register initialization to the appropriate
gen-mask macro
- fixed formatting for lines with more than 80 symbols
- introduced gicv3_dist_espi_init_aff to be able to use stubs in case of
CONFIG_GICV3_ESPI disabled
- renamed parameter in the GICD_TYPER_ESPI_RANGE macro to espi_range
(name was taken from GIC specification) to avoid confusion
- changed type for i variable to unsigned int since it cannot be
negative
Changes in V2:
- move gic_number_espis function from
[PATCH 08/10] xen/arm: vgic: add resource management for extended SPIs
to use it in the newly introduced gic_is_valid_espi
- add gic_is_valid_espi which checks if IRQ number is in supported
by HW eSPI range
- update gic_is_valid_irq conditions to allow operations with eSPIs
---
xen/arch/arm/gic-v3.c | 83 ++++++++++++++++++++++++++
xen/arch/arm/include/asm/gic.h | 22 +++++++
xen/arch/arm/include/asm/gic_v3_defs.h | 38 ++++++++++++
3 files changed, 143 insertions(+)
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 29b7f68cba..4a7ce12f26 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -485,6 +485,36 @@ static void __iomem *get_addr_by_offset(struct irq_desc *irqd, u32 offset)
default:
break;
}
+#ifdef CONFIG_GICV3_ESPI
+ case ESPI_BASE_INTID ... ESPI_MAX_INTID:
+ {
+ uint32_t irq_index = ESPI_INTID2IDX(irqd->irq);
+
+ switch ( offset )
+ {
+ case GICD_ISENABLER:
+ return (GICD + GICD_ISENABLERnE + (irq_index / 32) * 4);
+ case GICD_ICENABLER:
+ return (GICD + GICD_ICENABLERnE + (irq_index / 32) * 4);
+ case GICD_ISPENDR:
+ return (GICD + GICD_ISPENDRnE + (irq_index / 32) * 4);
+ case GICD_ICPENDR:
+ return (GICD + GICD_ICPENDRnE + (irq_index / 32) * 4);
+ case GICD_ISACTIVER:
+ return (GICD + GICD_ISACTIVERnE + (irq_index / 32) * 4);
+ case GICD_ICACTIVER:
+ return (GICD + GICD_ICACTIVERnE + (irq_index / 32) * 4);
+ case GICD_ICFGR:
+ return (GICD + GICD_ICFGRnE + (irq_index / 16) * 4);
+ case GICD_IROUTER:
+ return (GICD + GICD_IROUTERnE + irq_index * 8);
+ case GICD_IPRIORITYR:
+ return (GICD + GICD_IPRIORITYRnE + irq_index);
+ default:
+ break;
+ }
+ }
+#endif
default:
break;
}
@@ -655,6 +685,55 @@ static void gicv3_set_irq_priority(struct irq_desc *desc,
spin_unlock(&gicv3.lock);
}
+#ifdef CONFIG_GICV3_ESPI
+unsigned int gic_number_espis(void)
+{
+ return gic_hw_ops->info->nr_espi;
+}
+
+static void __init gicv3_dist_espi_common_init(uint32_t type)
+{
+ unsigned int espi_nr, i;
+
+ espi_nr = min(1024U, GICD_TYPER_ESPIS_NUM(type));
+ gicv3_info.nr_espi = espi_nr;
+ /* The GIC HW doesn't support eSPI, so we can leave from here */
+ if ( gicv3_info.nr_espi == 0 )
+ return;
+
+ printk("GICv3: %d eSPI lines\n", gicv3_info.nr_espi);
+
+ /* The configuration for eSPIs is similar to that for regular SPIs */
+ for ( i = 0; i < espi_nr; i += 16 )
+ writel_relaxed(0, GICD + GICD_ICFGRnE + (i / 16) * 4);
+
+ for ( i = 0; i < espi_nr; i += 4 )
+ writel_relaxed(GIC_PRI_IRQ_ALL,
+ GICD + GICD_IPRIORITYRnE + (i / 4) * 4);
+
+ for ( i = 0; i < espi_nr; i += 32 )
+ {
+ writel_relaxed(GENMASK(31, 0), GICD + GICD_ICENABLERnE + (i / 32) * 4);
+ writel_relaxed(GENMASK(31, 0), GICD + GICD_ICACTIVERnE + (i / 32) * 4);
+ }
+
+ for ( i = 0; i < espi_nr; i += 32 )
+ writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPRnE + (i / 32) * 4);
+}
+
+static void __init gicv3_dist_espi_init_aff(uint64_t affinity)
+{
+ unsigned int i;
+
+ for ( i = 0; i < gicv3_info.nr_espi; i++ )
+ writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTERnE + i * 8);
+}
+#else
+static void __init gicv3_dist_espi_common_init(uint32_t type) { }
+
+static void __init gicv3_dist_espi_init_aff(uint64_t affinity) { }
+#endif
+
static void __init gicv3_dist_init(void)
{
uint32_t type;
@@ -700,6 +779,8 @@ static void __init gicv3_dist_init(void)
for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i += 32 )
writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPR + (i / 32) * 4);
+ gicv3_dist_espi_common_init(type);
+
gicv3_dist_wait_for_rwp();
/* Turn on the distributor */
@@ -713,6 +794,8 @@ static void __init gicv3_dist_init(void)
for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i++ )
writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTER + i * 8);
+
+ gicv3_dist_espi_init_aff(affinity);
}
static int gicv3_enable_redist(void)
diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
index 3fcee42675..1e747dcd99 100644
--- a/xen/arch/arm/include/asm/gic.h
+++ b/xen/arch/arm/include/asm/gic.h
@@ -306,8 +306,26 @@ extern void gic_dump_vgic_info(struct vcpu *v);
/* Number of interrupt lines */
extern unsigned int gic_number_lines(void);
+#ifdef CONFIG_GICV3_ESPI
+extern unsigned int gic_number_espis(void);
+
+static inline bool gic_is_valid_espi(unsigned int irq)
+{
+ return (irq >= ESPI_BASE_INTID &&
+ irq < ESPI_IDX2INTID(gic_number_espis()));
+}
+#else
+static inline bool gic_is_valid_espi(unsigned int irq)
+{
+ return false;
+}
+#endif
+
static inline bool gic_is_valid_line(unsigned int irq)
{
+ if ( gic_is_valid_espi(irq) )
+ return true;
+
return irq < gic_number_lines();
}
@@ -325,6 +343,10 @@ struct gic_info {
enum gic_version hw_version;
/* Number of GIC lines supported */
unsigned int nr_lines;
+#ifdef CONFIG_GICV3_ESPI
+ /* Number of GIC eSPI supported */
+ unsigned int nr_espi;
+#endif
/* Number of LR registers */
uint8_t nr_lrs;
/* Maintenance irq number */
diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
index 2af093e774..3370b4cd52 100644
--- a/xen/arch/arm/include/asm/gic_v3_defs.h
+++ b/xen/arch/arm/include/asm/gic_v3_defs.h
@@ -37,6 +37,44 @@
#define GICD_IROUTER1019 (0x7FD8)
#define GICD_PIDR2 (0xFFE8)
+/* Additional registers for GICv3.1 */
+#define GICD_IGROUPRnE (0x1000)
+#define GICD_IGROUPRnEN (0x107C)
+#define GICD_ISENABLERnE (0x1200)
+#define GICD_ISENABLERnEN (0x127C)
+#define GICD_ICENABLERnE (0x1400)
+#define GICD_ICENABLERnEN (0x147C)
+#define GICD_ISPENDRnE (0x1600)
+#define GICD_ISPENDRnEN (0x167C)
+#define GICD_ICPENDRnE (0x1800)
+#define GICD_ICPENDRnEN (0x187C)
+#define GICD_ISACTIVERnE (0x1A00)
+#define GICD_ISACTIVERnEN (0x1A7C)
+#define GICD_ICACTIVERnE (0x1C00)
+#define GICD_ICACTIVERnEN (0x1C7C)
+#define GICD_IPRIORITYRnE (0x2000)
+#define GICD_IPRIORITYRnEN (0x23FC)
+#define GICD_ICFGRnE (0x3000)
+#define GICD_ICFGRnEN (0x30FC)
+#define GICD_IGRPMODRnE (0x3400)
+#define GICD_IGRPMODRnEN (0x347C)
+#define GICD_NSACRnE (0x3600)
+#define GICD_NSACRnEN (0x36FC)
+#define GICD_IROUTERnE (0x8000)
+#define GICD_IROUTERnEN (0x9FFC)
+
+#ifdef CONFIG_GICV3_ESPI
+#define GICD_TYPER_ESPI_SHIFT 8
+#define GICD_TYPER_ESPI_RANGE_SHIFT 27
+#define GICD_TYPER_ESPI_RANGE_MASK (0x1F)
+#define GICD_TYPER_ESPI (1U << GICD_TYPER_ESPI_SHIFT)
+#define GICD_TYPER_ESPI_RANGE(espi_range) ((((espi_range) & \
+ GICD_TYPER_ESPI_RANGE_MASK) + 1) * 32)
+#define GICD_TYPER_ESPIS_NUM(typer) \
+ (((typer) & GICD_TYPER_ESPI) ? \
+ GICD_TYPER_ESPI_RANGE((typer) >> GICD_TYPER_ESPI_RANGE_SHIFT) : 0)
+#endif
+
/* Common between GICD_PIDR2 and GICR_PIDR2 */
#define GIC_PIDR2_ARCH_MASK (0xf0)
#define GIC_PIDR2_ARCH_GICv3 (0x30)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 06/12] xen/arm/irq: allow eSPI processing in the gic_interrupt function
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (4 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-09-02 13:42 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 07/12] xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing Leonid Komarianskyi
` (5 subsequent siblings)
11 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Volodymyr Babchuk
The gic_interrupt function is the main handler for processing IRQs.
Currently, due to restrictive checks, it does not process interrupt
numbers greater than 1019. This patch updates the condition to allow
the handling of interrupts from the eSPI range.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
Changes in V5:
- no changes
Changes in V4:
- fixed commit message
- added reviewed-by from Volodymyr Babchuk
Changes in V3:
- no changes
Changes in V2:
- no changes
---
xen/arch/arm/gic.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 9469c9d08c..260ee64cca 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -342,7 +342,7 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
/* Reading IRQ will ACK it */
irq = gic_hw_ops->read_irq();
- if ( likely(irq >= GIC_SGI_STATIC_MAX && irq < 1020) )
+ if ( likely(irq >= GIC_SGI_STATIC_MAX && irq < 1020) || is_espi(irq) )
{
isb();
do_IRQ(regs, irq, is_fiq);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 07/12] xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (5 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 06/12] xen/arm/irq: allow eSPI processing in the gic_interrupt function Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-09-02 13:53 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs Leonid Komarianskyi
` (4 subsequent siblings)
11 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Volodymyr Babchuk
To properly deactivate guest interrupts and allow them to be retriggered
after the initial trigger, the LR needs to be updated. The current
implementation ignores interrupts outside the range specified by the mask
0x3FF, which only covers IRQ numbers up to 1023. To enable processing of
eSPI interrupts, this patch updates the mask to 0x13FF.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
---
Changes in V5:
- no changes
Changes in V4:
- added reviewed-by from Volodymyr Babchuk
Changes in V3:
- no changes
Changes in V2:
- remove unnecessary CONFIG_GICV3_ESPI ifdef guard
---
xen/arch/arm/include/asm/gic_v3_defs.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
index 3370b4cd52..e70c1a5675 100644
--- a/xen/arch/arm/include/asm/gic_v3_defs.h
+++ b/xen/arch/arm/include/asm/gic_v3_defs.h
@@ -211,7 +211,7 @@
#define ICH_LR_VIRTUAL_SHIFT 0
#define ICH_LR_CPUID_MASK 0x7
#define ICH_LR_CPUID_SHIFT 10
-#define ICH_LR_PHYSICAL_MASK 0x3ff
+#define ICH_LR_PHYSICAL_MASK 0x13ff
#define ICH_LR_PHYSICAL_SHIFT 32
#define ICH_LR_STATE_MASK 0x3
#define ICH_LR_STATE_SHIFT 62
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (6 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 07/12] xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-29 20:45 ` Volodymyr Babchuk
2025-09-02 14:13 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs Leonid Komarianskyi
` (3 subsequent siblings)
11 siblings, 2 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
This change introduces resource management in the VGIC to handle
extended SPIs introduced in GICv3.1. The pending_irqs and
allocated_irqs arrays are resized to support the required
number of eSPIs, based on what is supported by the hardware and
requested by the guest. A new field, ext_shared_irqs, is added
to the VGIC structure to store information about eSPIs, similar
to how shared_irqs is used for regular SPIs.
Since the eSPI range starts at INTID 4096 and INTIDs between 1025
and 4095 are reserved, helper macros are introduced to simplify the
transformation of indices and to enable easier access to eSPI-specific
resources. These changes prepare the VGIC for processing eSPIs as
required by future functionality.
The initialization and deinitialization paths for vgic have been updated
to allocate and free these resources appropriately. Additionally,
updated handling of INTIDs greater than 1024, passed from the toolstack
during domain creation, and verification logic ensures only valid SPI or
eSPI INTIDs are used.
The existing SPI behavior remains unaffected when guests do not request
eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
option is disabled.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
---
Changes in V5:
- removed the has_espi field because it can be determined by checking
whether domain->arch.vgic.nr_espis is zero or not
- since vgic_ext_rank_offset is not used in this patch, it has been
moved to the appropriate patch in the patch series, which implements
vgic eSPI registers emulation and requires this function
- removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
and code duplication in further changes
- fixed minor nit: used %pd for printing domain with its ID
Changes in V4:
- added has_espi field to simplify determining whether a domain is able
to operate with eSPI
- fixed formatting issues and misspellings
Changes in V3:
- fixed formatting for lines with more than 80 symbols
- introduced helper functions to be able to use stubs in case of
CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
#ifdefs
- fixed checks for nr_spis in domain_vgic_init
- updated comment about nr_spis adjustments with dom0less mention
- moved comment with additional explanations before checks
- used unsigned int for indexes since they cannot be negative
- removed unnecessary parentheses
- move vgic_ext_rank_offset to the below ifdef guard, to reduce the
number of ifdefs
Changes in V2:
- change is_espi_rank to is_valid_espi_rank to verify whether the array
element ext_shared_irqs exists. The previous version, is_espi_rank,
only checked if the rank index was less than the maximum possible eSPI
rank index, but this could potentially result in accessing a
non-existing array element. To address this, is_valid_espi_rank was
introduced, which ensures that the required eSPI rank exists
- move gic_number_espis to
xen/arm: gicv3: implement handling of GICv3.1 eSPI
- update vgic_is_valid_irq checks to allow operating with eSPIs
- remove redundant newline in vgic_allocate_virq
---
xen/arch/arm/include/asm/vgic.h | 12 ++
xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
2 files changed, 208 insertions(+), 3 deletions(-)
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 3e7cbbb196..912d5b7694 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -146,6 +146,10 @@ struct vgic_dist {
int nr_spis; /* Number of SPIs */
unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
struct vgic_irq_rank *shared_irqs;
+#ifdef CONFIG_GICV3_ESPI
+ struct vgic_irq_rank *ext_shared_irqs;
+ int nr_espis; /* Number of extended SPIs */
+#endif
/*
* SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
* struct arch_vcpu.
@@ -243,6 +247,14 @@ struct vgic_ops {
/* Number of ranks of interrupt registers for a domain */
#define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
+#ifdef CONFIG_GICV3_ESPI
+#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
+#endif
+#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
+#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
+#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
+#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
+
#define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
#define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 2bbf4d99aa..c9b9528c66 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -27,9 +27,82 @@
bool vgic_is_valid_line(struct domain *d, unsigned int virq)
{
+#ifdef CONFIG_GICV3_ESPI
+ if ( virq >= ESPI_BASE_INTID &&
+ virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
+ return true;
+#endif
+
return virq < vgic_num_irqs(d);
}
+#ifdef CONFIG_GICV3_ESPI
+/*
+ * Since eSPI indexes start from 4096 and numbers from 1024 to
+ * 4095 are forbidden, we need to check both lower and upper
+ * limits for ranks.
+ */
+static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
+{
+ return rank >= EXT_RANK_MIN &&
+ EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
+}
+
+static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
+ unsigned int rank)
+{
+ return &v->domain->arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
+}
+
+static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
+{
+ return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
+ d->arch.vgic.allocated_irqs);
+}
+
+static void arch_move_espis(struct vcpu *v)
+{
+ const cpumask_t *cpu_mask = cpumask_of(v->processor);
+ struct domain *d = v->domain;
+ struct pending_irq *p;
+ struct vcpu *v_target;
+ unsigned int i;
+
+ for ( i = ESPI_BASE_INTID;
+ i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
+ {
+ v_target = vgic_get_target_vcpu(v, i);
+ p = irq_to_pending(v_target, i);
+
+ if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
+ irq_set_affinity(p->desc, cpu_mask);
+ }
+}
+#else
+static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
+{
+ return false;
+}
+
+/*
+ * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
+ * because in this case, is_valid_espi_rank will always return false.
+ */
+static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
+ unsigned int rank)
+{
+ ASSERT_UNREACHABLE();
+ return NULL;
+}
+
+static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
+{
+ return false;
+}
+
+static void arch_move_espis(struct vcpu *v) { }
+#endif
+
static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
unsigned int rank)
{
@@ -37,6 +110,8 @@ static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
return v->arch.vgic.private_irqs;
else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
return &v->domain->arch.vgic.shared_irqs[rank - 1];
+ else if ( is_valid_espi_rank(v->domain, rank) )
+ return vgic_get_espi_rank(v, rank);
else
return NULL;
}
@@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
return 0;
}
+#ifdef CONFIG_GICV3_ESPI
+static unsigned int vgic_num_spi_lines(struct domain *d)
+{
+ return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
+}
+
+static int init_vgic_espi(struct domain *d)
+{
+ unsigned int i, idx;
+
+ if ( d->arch.vgic.nr_espis == 0 )
+ return 0;
+
+ d->arch.vgic.ext_shared_irqs =
+ xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
+ if ( d->arch.vgic.ext_shared_irqs == NULL )
+ return -ENOMEM;
+
+ for ( i = d->arch.vgic.nr_spis, idx = 0;
+ i < vgic_num_spi_lines(d); i++, idx++ )
+ vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
+ ESPI_IDX2INTID(idx));
+
+ for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
+ vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
+
+ return 0;
+}
+
+struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
+{
+ irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
+ return &d->arch.vgic.pending_irqs[irq];
+}
+#else
+static unsigned int init_vgic_espi(struct domain *d)
+{
+ return 0;
+}
+
+static unsigned int vgic_num_spi_lines(struct domain *d)
+{
+ return d->arch.vgic.nr_spis;
+}
+
+struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
+{
+ return NULL;
+}
+#endif
+
+static unsigned int vgic_num_alloc_irqs(struct domain *d)
+{
+ return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
+}
+
int domain_vgic_init(struct domain *d, unsigned int nr_spis)
{
int i;
@@ -131,6 +262,36 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
*/
nr_spis = ROUNDUP(nr_spis, 32);
+#ifdef CONFIG_GICV3_ESPI
+ /*
+ * During domain creation, the dom0less DomUs code or toolstack specifies
+ * the maximum INTID, which is defined in the domain config subtracted by
+ * 32 to cover the local IRQs (please see the comment to VGIC_DEF_NR_SPIS).
+ * To compute the actual number of eSPI that will be usable for,
+ * add back 32.
+ */
+ if ( nr_spis + 32 > ESPI_IDX2INTID(NR_ESPI_IRQS) )
+ return -EINVAL;
+
+ if ( nr_spis + 32 >= ESPI_BASE_INTID )
+ {
+ d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32, 1024U);
+ /* Verify if GIC HW can handle provided INTID */
+ if ( d->arch.vgic.nr_espis > gic_number_espis() )
+ return -EINVAL;
+ /*
+ * Set the maximum available number for regular
+ * SPI to pass the next check
+ */
+ nr_spis = VGIC_DEF_NR_SPIS;
+ }
+ else
+ {
+ /* Domain will use the regular SPI range */
+ d->arch.vgic.nr_espis = 0;
+ }
+#endif
+
/* Limit the number of virtual SPIs supported to (1020 - 32) = 988 */
if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
return -EINVAL;
@@ -145,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
return -ENOMEM;
d->arch.vgic.pending_irqs =
- xzalloc_array(struct pending_irq, d->arch.vgic.nr_spis);
+ xzalloc_array(struct pending_irq, vgic_num_spi_lines(d));
if ( d->arch.vgic.pending_irqs == NULL )
return -ENOMEM;
@@ -156,12 +317,16 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
vgic_rank_init(&d->arch.vgic.shared_irqs[i], i + 1, 0);
+ ret = init_vgic_espi(d);
+ if ( ret )
+ return ret;
+
ret = d->arch.vgic.handler->domain_init(d);
if ( ret )
return ret;
d->arch.vgic.allocated_irqs =
- xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
+ xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_alloc_irqs(d)));
if ( !d->arch.vgic.allocated_irqs )
return -ENOMEM;
@@ -195,9 +360,27 @@ void domain_vgic_free(struct domain *d)
}
}
+#ifdef CONFIG_GICV3_ESPI
+ for ( i = 0; i < d->arch.vgic.nr_espis; i++ )
+ {
+ struct pending_irq *p = spi_to_pending(d, ESPI_IDX2INTID(i));
+
+ if ( p->desc )
+ {
+ ret = release_guest_irq(d, p->irq);
+ if ( ret )
+ dprintk(XENLOG_G_WARNING, "%pd: Failed to release virq %u ret = %d\n",
+ d, p->irq, ret);
+ }
+ }
+#endif
+
if ( d->arch.vgic.handler )
d->arch.vgic.handler->domain_free(d);
xfree(d->arch.vgic.shared_irqs);
+#ifdef CONFIG_GICV3_ESPI
+ xfree(d->arch.vgic.ext_shared_irqs);
+#endif
xfree(d->arch.vgic.pending_irqs);
xfree(d->arch.vgic.allocated_irqs);
}
@@ -331,6 +514,8 @@ void arch_move_irqs(struct vcpu *v)
if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
irq_set_affinity(p->desc, cpu_mask);
}
+
+ arch_move_espis(v);
}
void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
@@ -538,6 +723,8 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
n = &v->arch.vgic.pending_irqs[irq];
else if ( is_lpi(irq) )
n = v->domain->arch.vgic.handler->lpi_to_pending(v->domain, irq);
+ else if ( is_espi(irq) )
+ n = espi_to_pending(v->domain, irq);
else
n = &v->domain->arch.vgic.pending_irqs[irq - 32];
return n;
@@ -547,6 +734,9 @@ struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq)
{
ASSERT(irq >= NR_LOCAL_IRQS);
+ if ( is_espi(irq) )
+ return espi_to_pending(d, irq);
+
return &d->arch.vgic.pending_irqs[irq - 32];
}
@@ -668,6 +858,9 @@ bool vgic_reserve_virq(struct domain *d, unsigned int virq)
if ( !vgic_is_valid_line(d, virq) )
return false;
+ if ( is_espi(virq) )
+ return vgic_reserve_espi_virq(d, virq);
+
return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
}
@@ -685,7 +878,7 @@ int vgic_allocate_virq(struct domain *d, bool spi)
else
{
first = 32;
- end = vgic_num_irqs(d);
+ end = vgic_num_alloc_irqs(d);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (7 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-09-02 16:42 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers Leonid Komarianskyi
` (2 subsequent siblings)
11 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Volodymyr Babchuk, Oleksandr Tyshchenko
The Dom0 and DomUs logic for the dom0less configuration in create_dom0() and
arch_create_domUs() has been updated to account for extended SPIs when
supported by the hardware and enabled with CONFIG_GICV3_ESPI. These changes
ensure the proper calculation of the maximum number of SPIs and eSPIs available
to Dom0 and DomUs in dom0less setups.
When eSPIs are supported by the hardware and CONFIG_GICV3_ESPI is enabled, the
maximum number of eSPI interrupts is calculated using the ESPI_BASE_INTID
offset (4096) and is limited to 1024, with 32 IRQs subtracted. To ensure
compatibility with non-Dom0 domains, this adjustment is applied by the
toolstack during domain creation, while for Dom0 or DomUs in Dom0, it is
handled directly during VGIC initialization. If eSPIs are not supported, the
calculation defaults to using the standard SPI range, with a maximum value of
992 interrupt lines, as it works currently.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Reviewed-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
---
Changes in V5:
- fixed minor nits, no functional changes: updated the comment to make
it clearer and corrected a misspelling
- added reviewed-by from Volodymyr Babchuk and from Oleksandr Tyshchenko
Changes in V4:
- consolidated the eSPI and SPI logic into a new inline function,
vgic_def_nr_spis. Without eSPI support (either due to config being
disabled or hardware not supporting it), it will return the regular SPI
range, as it works currently. There are no functional changes compared
with the previous patch version
- removed VGIC_DEF_MAX_SPI macro, to reduce the number of ifdefs
Changes in V3:
- renamed macro VGIC_DEF_NR_ESPIS to more appropriate VGIC_DEF_MAX_SPI
- added eSPI initialization for dom0less setups
- fixed comment with mentions about dom0less builds
- fixed formatting for lines with more than 80 symbols
- updated commit message
Changes in V2:
- no changes
---
xen/arch/arm/dom0less-build.c | 2 +-
xen/arch/arm/domain_build.c | 2 +-
xen/arch/arm/include/asm/vgic.h | 19 +++++++++++++++++++
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
index 69b9ea22ce..02d5559102 100644
--- a/xen/arch/arm/dom0less-build.c
+++ b/xen/arch/arm/dom0less-build.c
@@ -285,7 +285,7 @@ void __init arch_create_domUs(struct dt_device_node *node,
{
int vpl011_virq = GUEST_VPL011_SPI;
- d_cfg->arch.nr_spis = VGIC_DEF_NR_SPIS;
+ d_cfg->arch.nr_spis = vgic_def_nr_spis();
/*
* The VPL011 virq is GUEST_VPL011_SPI, unless direct-map is
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index d91a71acfd..39eea0be00 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2054,7 +2054,7 @@ void __init create_dom0(void)
/* The vGIC for DOM0 is exactly emulating the hardware GIC */
dom0_cfg.arch.gic_version = XEN_DOMCTL_CONFIG_GIC_NATIVE;
- dom0_cfg.arch.nr_spis = VGIC_DEF_NR_SPIS;
+ dom0_cfg.arch.nr_spis = vgic_def_nr_spis();
dom0_cfg.arch.tee_type = tee_get_type();
dom0_cfg.max_vcpus = dom0_max_vcpus();
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 912d5b7694..3aa22114ba 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -347,6 +347,25 @@ extern void vgic_check_inflight_irqs_pending(struct vcpu *v,
/* Default number of vGIC SPIs. 32 are substracted to cover local IRQs. */
#define VGIC_DEF_NR_SPIS (min(gic_number_lines(), VGIC_MAX_IRQS) - 32)
+static inline unsigned int vgic_def_nr_spis(void)
+{
+#ifdef CONFIG_GICV3_ESPI
+ /*
+ * Check if the hardware supports extended SPIs (even if the appropriate
+ * config is set). If not, the common SPI range will be used. Otherwise
+ * return the maximum eSPI INTID, supported by HW GIC, subtracted by 32.
+ * For Dom0 and started at boot time DomUs we will add back this value
+ * during VGIC initialization. This ensures consistent handling for Dom0
+ * and other domains. For the regular SPI range interrupts in this case,
+ * the maximum value of VGIC_DEF_NR_SPIS will be used.
+ */
+ if ( gic_number_espis() > 0 )
+ return ESPI_BASE_INTID + min(gic_number_espis(), 1024U) - 32;
+#endif
+
+ return VGIC_DEF_NR_SPIS;
+}
+
extern bool vgic_is_valid_line(struct domain *d, unsigned int virq);
static inline bool vgic_is_spi(struct domain *d, unsigned int virq)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (8 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-29 20:12 ` Volodymyr Babchuk
` (2 more replies)
2025-08-29 16:06 ` [PATCH v5 11/12] doc/man: update description for nr_spis with eSPI Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 12/12] CHANGELOG.md: add mention of GICv3.1 eSPI support Leonid Komarianskyi
11 siblings, 3 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk
Implemented support for GICv3.1 extended SPI registers for vGICv3,
allowing the emulation of eSPI-specific behavior for guest domains.
The implementation includes read and write emulation for eSPI-related
registers (e.g., GICD_ISENABLERnE, GICD_IROUTERnE, and others),
following a similar approach to the handling of regular SPIs.
The eSPI registers, previously located in reserved address ranges,
are now adjusted to support MMIO read and write operations correctly
when CONFIG_GICV3_ESPI is enabled.
The availability of eSPIs and the number of emulated extended SPIs
for guest domains is reported by setting the appropriate bits in the
GICD_TYPER register, based on the number of eSPIs requested by the
domain and supported by the hardware. In cases where the configuration
option is disabled, the hardware does not support eSPIs, or the domain
does not request such interrupts, the functionality remains unchanged.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
---
Changes in V5:
- since eSPI-specific defines and macros are now available even when the
appropriate config is disabled, this allows us to remove many
#ifdef guards and reuse the existing code for regular SPIs for eSPIs as
well, as eSPIs are processed similarly. This improves code readability
and consolidates the register ranges for SPIs and eSPIs in a single
place, simplifying future changes or fixes for SPIs and their
counterparts from the extended range
- moved vgic_ext_rank_offset() from
[08/12] xen/arm: vgic: add resource management for extended SPIs
as the function was unused before this patch
- added stub implementation of vgic_ext_rank_offset() when CONFIG_GICV3_ESPI=n
- removed unnecessary defines for reserved ranges, which were introduced in
V4 to reduce the number of #ifdefs. The issue is now resolved by
allowing the use of SPI-specific offsets and reworking the logic
Changes in V4:
- added missing RAZ and write ignore eSPI-specific registers ranges:
GICD_NSACRnE and GICD_IGRPMODRnE
- changed previously reserved range to cover GICD_NSACRnE and
GICD_IGRPMODRnE
- introduced GICD_RESERVED_RANGE<n>_START/END defines to remove
hardcoded values and reduce the number of ifdefs
- fixed reserved ranges with eSPI option enabled: added missing range
0x0F30-0x0F7C
- updated the logic for domains that do not support eSPI, but Xen is
compiled with the eSPI option. Now, prior to other MMIO checks, we
verify whether eSPI is available for the domain or not. If not, it
behaves as it does currently - RAZ and WI
- fixed print for GICD_ICACTIVERnE
- fixed new lines formatting for switch-case
Changes in V3:
- changed vgic_store_irouter parameters - instead of offset virq is
used, to remove the additional bool espi parameter and simplify
checks. Also, adjusted parameters for regular SPI. Since the offset
parameter was used only for calculating virq number and then reused for
finding rank offset, it will not affect functionality.
- fixed formatting for goto lables - added newlines after condition
- fixed logs for GICD_ISACTIVERnE and GICD_ICACTIVERnE handlers
- removed #ifdefs in 2 places where they were adjacent and could be merged
Changes in V2:
- add missing rank index conversion for pending and inflight irqs
---
xen/arch/arm/include/asm/vgic.h | 4 +
xen/arch/arm/vgic-v3.c | 198 ++++++++++++++++++++++++++------
xen/arch/arm/vgic.c | 23 ++++
3 files changed, 192 insertions(+), 33 deletions(-)
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index 3aa22114ba..103bc3c74b 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -314,6 +314,10 @@ extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v,
unsigned int b,
unsigned int n,
unsigned int s);
+extern struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v,
+ unsigned int b,
+ unsigned int n,
+ unsigned int s);
extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 4369c55177..b5d766c98f 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -111,13 +111,10 @@ static uint64_t vgic_fetch_irouter(struct vgic_irq_rank *rank,
* Note the offset will be aligned to the appropriate boundary.
*/
static void vgic_store_irouter(struct domain *d, struct vgic_irq_rank *rank,
- unsigned int offset, uint64_t irouter)
+ unsigned int virq, uint64_t irouter)
{
struct vcpu *new_vcpu, *old_vcpu;
- unsigned int virq;
-
- /* There is 1 vIRQ per IROUTER */
- virq = offset / NR_BYTES_PER_IROUTER;
+ unsigned int offset;
/*
* The IROUTER0-31, used for SGIs/PPIs, are reserved and should
@@ -685,13 +682,20 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
{
case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
+ case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
+ case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
/* We do not implement security extensions for guests, read zero */
if ( dabt.size != DABT_WORD ) goto bad_width;
goto read_as_zero;
case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
+ case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
+ if ( reg >= GICD_ISENABLERnE )
+ rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
+ DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
if ( rank == NULL ) goto read_as_zero;
vgic_lock_rank(v, rank, flags);
*r = vreg_reg32_extract(rank->ienable, info);
@@ -699,8 +703,13 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
return 1;
case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
+ case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
+ if ( reg >= GICD_ICENABLERnE )
+ rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
+ DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
if ( rank == NULL ) goto read_as_zero;
vgic_lock_rank(v, rank, flags);
*r = vreg_reg32_extract(rank->ienable, info);
@@ -710,20 +719,29 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
/* Read the pending status of an IRQ via GICD/GICR is not supported */
case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
+ case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
+ case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
goto read_as_zero;
/* Read the active status of an IRQ via GICD/GICR is not supported */
case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
case VRANGE32(GICD_ICACTIVER, GICD_ICACTIVERN):
+ case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
+ case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
goto read_as_zero;
case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
+ case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
{
uint32_t ipriorityr;
uint8_t rank_index;
if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
+ if ( reg >= GICD_IPRIORITYRnE )
+ rank = vgic_ext_rank_offset(v, 8, reg - GICD_IPRIORITYRnE,
+ DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
if ( rank == NULL ) goto read_as_zero;
rank_index = REG_RANK_INDEX(8, reg - GICD_IPRIORITYR, DABT_WORD);
@@ -737,11 +755,15 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
}
case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
+ case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
{
uint32_t icfgr;
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
+ if ( reg >= GICD_ICFGRnE )
+ rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
if ( rank == NULL ) goto read_as_zero;
vgic_lock_rank(v, rank, flags);
icfgr = rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR, DABT_WORD)];
@@ -782,46 +804,81 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
{
case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
+ case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
+ case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
/* We do not implement security extensions for guests, write ignore */
goto write_ignore_32;
case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
+ case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
+ if ( reg >= GICD_ISENABLERnE )
+ rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
+ DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
if ( rank == NULL ) goto write_ignore;
vgic_lock_rank(v, rank, flags);
tr = rank->ienable;
vreg_reg32_setbits(&rank->ienable, r, info);
- vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
+ if ( reg >= GICD_ISENABLERnE )
+ vgic_enable_irqs(v, (rank->ienable) & (~tr),
+ EXT_RANK_IDX2NUM(rank->index));
+ else
+ vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
vgic_unlock_rank(v, rank, flags);
return 1;
case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
+ case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
+ if ( reg >= GICD_ICENABLERnE )
+ rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
+ DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
if ( rank == NULL ) goto write_ignore;
vgic_lock_rank(v, rank, flags);
tr = rank->ienable;
vreg_reg32_clearbits(&rank->ienable, r, info);
- vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
+ if ( reg >= GICD_ICENABLERnE )
+ vgic_disable_irqs(v, (~rank->ienable) & tr,
+ EXT_RANK_IDX2NUM(rank->index));
+ else
+ vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
vgic_unlock_rank(v, rank, flags);
return 1;
case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
+ case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
+ if ( reg >= GICD_ISPENDRnE )
+ rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISPENDRnE, DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
if ( rank == NULL ) goto write_ignore;
- vgic_set_irqs_pending(v, r, rank->index);
+ if ( reg >= GICD_ISPENDRnE )
+ vgic_set_irqs_pending(v, r, EXT_RANK_IDX2NUM(rank->index));
+ else
+ vgic_set_irqs_pending(v, r, rank->index);
return 1;
case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
+ case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
+ if ( reg >= GICD_ICPENDRnE )
+ rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICPENDRnE, DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
if ( rank == NULL ) goto write_ignore;
- vgic_check_inflight_irqs_pending(v, rank->index, r);
+ if ( reg >= GICD_ICPENDRnE )
+ vgic_check_inflight_irqs_pending(v,
+ EXT_RANK_IDX2NUM(rank->index), r);
+ else
+ vgic_check_inflight_irqs_pending(v, rank->index, r);
goto write_ignore;
@@ -838,16 +895,38 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
v, name, r, reg - GICD_ICACTIVER);
goto write_ignore_32;
+ case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
+ if ( dabt.size != DABT_WORD )
+ goto bad_width;
+ printk(XENLOG_G_ERR
+ "%pv: %s: unhandled word write %#"PRIregister" to ISACTIVER%dE\n",
+ v, name, r, reg - GICD_ISACTIVERnE);
+ return 0;
+
+ case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
+ printk(XENLOG_G_ERR
+ "%pv: %s: unhandled word write %#"PRIregister" to ICACTIVER%dE\n",
+ v, name, r, reg - GICD_ICACTIVERnE);
+ goto write_ignore_32;
+
case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
+ case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
{
- uint32_t *ipriorityr, priority;
+ uint32_t *ipriorityr, priority, offset;
if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
+ if ( reg >= GICD_IPRIORITYRnE ) {
+ offset = reg - GICD_IPRIORITYRnE;
+ rank = vgic_ext_rank_offset(v, 8, offset, DABT_WORD);
+ }
+ else
+ {
+ offset = reg - GICD_IPRIORITYR;
+ rank = vgic_rank_offset(v, 8, offset, DABT_WORD);
+ }
if ( rank == NULL ) goto write_ignore;
vgic_lock_rank(v, rank, flags);
- ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, reg - GICD_IPRIORITYR,
- DABT_WORD)];
+ ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, offset, DABT_WORD)];
priority = ACCESS_ONCE(*ipriorityr);
vreg_reg32_update(&priority, r, info);
ACCESS_ONCE(*ipriorityr) = priority;
@@ -859,10 +938,14 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
goto write_ignore_32;
case VRANGE32(GICD_ICFGR + 4, GICD_ICFGRN): /* PPI + SPIs */
+ case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
/* ICFGR1 for PPI's, which is implementation defined
if ICFGR1 is programmable or not. We chose to program */
if ( dabt.size != DABT_WORD ) goto bad_width;
- rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
+ if ( reg >= GICD_ICFGRnE )
+ rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
+ else
+ rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
if ( rank == NULL ) goto write_ignore;
vgic_lock_rank(v, rank, flags);
vreg_reg32_update(&rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR,
@@ -1129,6 +1212,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
typer |= GICD_TYPE_LPIS;
typer |= (v->domain->arch.vgic.intid_bits - 1) << GICD_TYPE_ID_BITS_SHIFT;
+#ifdef CONFIG_GICV3_ESPI
+ if ( v->domain->arch.vgic.nr_espis > 0 )
+ {
+ /* Set eSPI support bit for the domain */
+ typer |= GICD_TYPER_ESPI;
+ /* Set ESPI range bits */
+ typer |= (DIV_ROUND_UP(v->domain->arch.vgic.nr_espis, 32) - 1)
+ << GICD_TYPER_ESPI_RANGE_SHIFT;
+ }
+#endif
*r = vreg_reg32_extract(typer, info);
@@ -1194,6 +1287,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
+ case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
+ case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
+ case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
+ case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
+ case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
+ case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
+ case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
+ case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
+ case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
+ case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
/*
* Above all register are common with GICR and GICD
* Manage in common
@@ -1201,6 +1304,7 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
return __vgic_v3_distr_common_mmio_read("vGICD", v, info, gicd_reg, r);
case VRANGE32(GICD_NSACR, GICD_NSACRN):
+ case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
/* We do not implement security extensions for guests, read zero */
goto read_as_zero_32;
@@ -1216,16 +1320,21 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
/* Replaced with GICR_ISPENDR0. So ignore write */
goto read_as_zero_32;
- case VRANGE32(0x0F30, 0x60FC):
+ case VRANGE32(0x0F30, 0x0FFC):
goto read_reserved;
case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
+ case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
{
uint64_t irouter;
if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
- rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
- DABT_DOUBLE_WORD);
+ if ( gicd_reg >= GICD_IROUTERnE )
+ rank = vgic_ext_rank_offset(v, 64, gicd_reg - GICD_IROUTERnE,
+ DABT_DOUBLE_WORD);
+ else
+ rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
+ DABT_DOUBLE_WORD);
if ( rank == NULL ) goto read_as_zero;
vgic_lock_rank(v, rank, flags);
irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
@@ -1235,8 +1344,8 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
return 1;
}
-
- case VRANGE32(0x7FE0, 0xBFFC):
+ case VRANGE32(0x3700, 0x60FC):
+ case VRANGE32(0xA004, 0xBFFC):
goto read_reserved;
case VRANGE32(0xC000, 0xFFCC):
@@ -1382,12 +1491,23 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
+ case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
+ case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
+ case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
+ case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
+ case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
+ case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
+ case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
+ case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
+ case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
+ case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
/* Above registers are common with GICR and GICD
* Manage in common */
return __vgic_v3_distr_common_mmio_write("vGICD", v, info,
gicd_reg, r);
case VRANGE32(GICD_NSACR, GICD_NSACRN):
+ case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
/* We do not implement security extensions for guests, write ignore */
goto write_ignore_32;
@@ -1405,26 +1525,38 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
if ( dabt.size != DABT_WORD ) goto bad_width;
return 0;
- case VRANGE32(0x0F30, 0x60FC):
+ case VRANGE32(0x0F30, 0x0FFC):
goto write_reserved;
case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
+ case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
{
uint64_t irouter;
+ unsigned int offset, virq;
if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
- rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
- DABT_DOUBLE_WORD);
+ if ( gicd_reg >= GICD_IROUTERnE ) {
+ offset = gicd_reg - GICD_IROUTERnE;
+ rank = vgic_ext_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
+ } else {
+ offset = gicd_reg - GICD_IROUTER;
+ rank = vgic_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
+ }
if ( rank == NULL ) goto write_ignore;
vgic_lock_rank(v, rank, flags);
- irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
+ irouter = vgic_fetch_irouter(rank, offset);
vreg_reg64_update(&irouter, r, info);
- vgic_store_irouter(v->domain, rank, gicd_reg - GICD_IROUTER, irouter);
+ if ( gicd_reg >= GICD_IROUTERnE )
+ virq = ESPI_IDX2INTID(offset / NR_BYTES_PER_IROUTER);
+ else
+ virq = offset / NR_BYTES_PER_IROUTER;
+ vgic_store_irouter(v->domain, rank, virq, irouter);
vgic_unlock_rank(v, rank, flags);
return 1;
}
- case VRANGE32(0x7FE0, 0xBFFC):
+ case VRANGE32(0x3700, 0x60FC):
+ case VRANGE32(0xA004, 0xBFFC):
goto write_reserved;
case VRANGE32(0xC000, 0xFFCC):
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index c9b9528c66..27ffdf316c 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -193,6 +193,18 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
}
#ifdef CONFIG_GICV3_ESPI
+/*
+ * The function behavior is the same as for regular SPIs (vgic_rank_offset),
+ * but it operates with extended SPI ranks.
+ */
+struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
+ unsigned int n, unsigned int s)
+{
+ unsigned int rank = REG_RANK_NR(b, (n >> s));
+
+ return vgic_get_rank(v, rank + EXT_RANK_MIN);
+}
+
static unsigned int vgic_num_spi_lines(struct domain *d)
{
return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
@@ -241,6 +253,17 @@ struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
{
return NULL;
}
+
+/*
+ * It is expected that, in the case of CONFIG_GICV3_ESPI=n, the function will
+ * return NULL. In this scenario, mmio_read/mmio_write will be treated as
+ * RAZ/WI, as expected.
+ */
+struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
+ unsigned int n, unsigned int s)
+{
+ return NULL;
+}
#endif
static unsigned int vgic_num_alloc_irqs(struct domain *d)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 11/12] doc/man: update description for nr_spis with eSPI
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (9 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 12/12] CHANGELOG.md: add mention of GICv3.1 eSPI support Leonid Komarianskyi
11 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Anthony PERARD
Since eSPI support has been introduced, update the documentation with
the appropriate description.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
---
The discussion is ongoing and can be addressed in V5.
Clarification is needed from the maintainers.
Link:
- https://lore.kernel.org/xen-devel/87y0r4z717.fsf@epam.com/
Changes in V5:
- no changes
Changes in V4:
- introduced this patch
---
docs/man/xl.cfg.5.pod.in | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index 5362fb0e9a..292ab10331 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -3072,11 +3072,14 @@ interval of colors (such as "0-4").
=item B<nr_spis="NR_SPIS">
An optional integer parameter specifying the number of SPIs (Shared
-Peripheral Interrupts) to allocate for the domain. Max is 960 SPIs. If
-the `nr_spis` parameter is not specified, the value calculated by the toolstack
-will be used for the domain. Otherwise, the value specified by the `nr_spis`
-parameter will be used. The number of SPIs should match the highest interrupt
-ID that will be assigned to the domain.
+Peripheral Interrupts) to allocate for the domain. Max is 960 for regular SPIs
+or 5088 for eSPIs (Extended SPIs). The eSPIs includes an additional 1024 SPIs
+from the eSPI range (4096 to 5119) if the hardware supports extended SPIs
+(GICv3.1+) and CONFIG_GICV3_ESPI is enabled. If the `nr_spis` parameter is not
+specified, the value calculated by the toolstack will be used for the domain.
+Otherwise, the value specified by the `nr_spis` parameter will be used. The
+number of SPIs should match the highest interrupt ID that will be assigned to
+the domain.
=item B<trap_unmapped_accesses=BOOLEAN>
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v5 12/12] CHANGELOG.md: add mention of GICv3.1 eSPI support
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
` (10 preceding siblings ...)
2025-08-29 16:06 ` [PATCH v5 11/12] doc/man: update description for nr_spis with eSPI Leonid Komarianskyi
@ 2025-08-29 16:06 ` Leonid Komarianskyi
11 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-08-29 16:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Leonid Komarianskyi, Oleksii Kurochko,
Community Manager
The GICv3.1 eSPI (Extended Shared Peripheral Interrupts) range is
already supported with CONFIG_GICV3_ESPI enabled, so this feature should
be mentioned in CHANGELOG.md.
Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
Acked-by: Oleksii Kurochko<oleksii.kurochko@gmail.com>
---
Changes in V5:
- extended changelog update with a brief explanation of what eSPI is
- added acked-by from Oleksii Kurochko, which was received for V3:
https://lore.kernel.org/xen-devel/bce5e07c-eee7-481b-a833-4d5d8bbce78f@gmail.com/
Changes in V4:
- no changes
Changes in V3:
- introduced this patch
---
CHANGELOG.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5f31ca08fe..31b4ded444 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -29,6 +29,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- On Arm:
- Ability to enable stack protector
+ - GICv3.1 eSPI (Extended Shared Peripheral Interrupts) support for Xen
+ and guest domains.
### Removed
- On x86:
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-08-29 16:06 ` [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range Leonid Komarianskyi
@ 2025-08-29 19:45 ` Volodymyr Babchuk
2025-08-31 14:08 ` Oleksandr Tyshchenko
2025-09-01 16:11 ` Julien Grall
1 sibling, 1 reply; 49+ messages in thread
From: Volodymyr Babchuk @ 2025-08-29 19:45 UTC (permalink / raw)
To: Leonid Komarianskyi
Cc: xen-devel@lists.xenproject.org, olekstysh@gmail.com,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hi Leonid,
Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
> Currently, Xen does not support eSPI interrupts, leading
> to a data abort when such interrupts are defined in the DTS.
>
> This patch introduces a separate array to initialize up to
> 1024 interrupt descriptors in the eSPI range and adds the
> necessary defines and helper function. These changes lay the
> groundwork for future implementation of full eSPI interrupt
> support. As this GICv3.1 feature is not required by all vendors,
> all changes are guarded by ifdefs, depending on the corresponding
> Kconfig option.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> ---
> Changes in V5:
> - no functional changes introduced by this version compared with V4, only
> minor fixes and removal of ifdefs for macroses
> - added TODO comment, suggested by Oleksandr Tyshchenko
> - changed int to unsigned int for irqs
> - removed ifdefs for eSPI-specific defines and macros to reduce the
> number of ifdefs and code duplication in further changes
> - removed reviewed-by as moving defines from ifdefs requires additional
> confirmation from reviewers
>
> Changes in V4:
> - removed redundant line with 'default n' in Kconfig, as it is disabled
> by default, without explicit specification
> - added reviewed-by from Volodymyr Babchuk
>
> Changes in V3:
> - introduced a new define NR_ESPI_IRQS to avoid confusion, like in the
> case of using NR_IRQS for espi_desc array
> - implemented helper functions espi_to_desc and init_espi_data to make
> it possible to add stubs with the same name, and as a result, reduce
> the number of #ifdefs
> - disable CONFIG_GICV3_ESPI default value to n
>
> Changes in V2:
> - use (ESPI_MAX_INTID + 1) instead of (ESPI_BASE_INTID + NR_IRQS)
> - remove unnecessary comment for nr_irqs initialization
> ---
> xen/arch/arm/Kconfig | 8 +++++
> xen/arch/arm/include/asm/irq.h | 24 +++++++++++++++
> xen/arch/arm/irq.c | 56 +++++++++++++++++++++++++++++++++-
> 3 files changed, 87 insertions(+), 1 deletion(-)
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 17df147b25..43b05533b1 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -135,6 +135,14 @@ config GICV3
> Driver for the ARM Generic Interrupt Controller v3.
> If unsure, use the default setting.
>
> +config GICV3_ESPI
> + bool "Extended SPI range support"
> + depends on GICV3 && !NEW_VGIC
> + help
> + Allow Xen and domains to use interrupt numbers from the extended SPI
> + range, from 4096 to 5119. This feature is introduced in GICv3.1
> + architecture.
> +
> config HAS_ITS
> bool "GICv3 ITS MSI controller support (UNSUPPORTED)" if UNSUPPORTED
> depends on GICV3 && !NEW_VGIC && !ARM_32
> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
> index 5bc6475eb4..4443799648 100644
> --- a/xen/arch/arm/include/asm/irq.h
> +++ b/xen/arch/arm/include/asm/irq.h
> @@ -32,6 +32,13 @@ struct arch_irq_desc {
> #define SPI_MAX_INTID 1019
> #define LPI_OFFSET 8192
>
> +#define ESPI_BASE_INTID 4096
> +#define ESPI_MAX_INTID 5119
> +#define NR_ESPI_IRQS 1024
> +
> +#define ESPI_INTID2IDX(intid) ((intid) - ESPI_BASE_INTID)
> +#define ESPI_IDX2INTID(idx) ((idx) + ESPI_BASE_INTID)
> +
> /* LPIs are always numbered starting at 8192, so 0 is a good invalid case. */
> #define INVALID_LPI 0
>
> @@ -39,7 +46,15 @@ struct arch_irq_desc {
> #define INVALID_IRQ 1023
>
> extern const unsigned int nr_irqs;
> +#ifdef CONFIG_GICV3_ESPI
> +/*
> + * This will also cover the eSPI range, as some critical devices
> + * for booting Xen (e.g., serial) may use this type of interrupts.
> + */
> +#define nr_static_irqs (ESPI_MAX_INTID + 1)
> +#else
> #define nr_static_irqs NR_IRQS
> +#endif
>
> struct irq_desc;
> struct irqaction;
> @@ -55,6 +70,15 @@ static inline bool is_lpi(unsigned int irq)
> return irq >= LPI_OFFSET;
> }
>
> +static inline bool is_espi(unsigned int irq)
> +{
> +#ifdef CONFIG_GICV3_ESPI
Taking into account that with CONFIG_GICV3_ESPI=n we should never have
"irq" in eSPI range, do you really need this #ifdef? I think that
ASSERT_UNREACHABLE in espi_to_desc() is sufficient guard.
Also, IRQ line number belongs to eSPI range regardless of CONFIG_GICV3_ESPI,
value, so in my opinion is_espi() should always return correct value for
a given "irq".
> + return (irq >= ESPI_BASE_INTID && irq <= ESPI_MAX_INTID);
Also, you don't need parentheses here.
> +#else
> + return false;
> +#endif
> +}
> +
> #define domain_pirq_to_irq(d, pirq) (pirq)
>
> bool is_assignable_irq(unsigned int irq);
> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
> index b8eccfc924..61c915c3f9 100644
> --- a/xen/arch/arm/irq.c
> +++ b/xen/arch/arm/irq.c
> @@ -19,7 +19,11 @@
> #include <asm/gic.h>
> #include <asm/vgic.h>
>
> +#ifdef CONFIG_GICV3_ESPI
> +const unsigned int nr_irqs = ESPI_MAX_INTID + 1;
> +#else
> const unsigned int nr_irqs = NR_IRQS;
> +#endif
>
> static unsigned int local_irqs_type[NR_LOCAL_IRQS];
> static DEFINE_SPINLOCK(local_irqs_type_lock);
> @@ -46,6 +50,53 @@ void irq_end_none(struct irq_desc *irq)
> }
>
> static irq_desc_t irq_desc[NR_IRQS - NR_LOCAL_IRQS];
> +#ifdef CONFIG_GICV3_ESPI
> +/*
> + * TODO: Consider allocating an array dynamically if
> + * there is a need to enable GICV3_ESPI by default.
> + */
> +static irq_desc_t espi_desc[NR_ESPI_IRQS];
> +
> +static struct irq_desc *espi_to_desc(unsigned int irq)
> +{
> + return &espi_desc[ESPI_INTID2IDX(irq)];
> +}
> +
> +static int __init init_espi_data(void)
> +{
> + unsigned int irq;
> +
> + for ( irq = ESPI_BASE_INTID; irq <= ESPI_MAX_INTID; irq++ )
> + {
> + struct irq_desc *desc = irq_to_desc(irq);
> + int rc = init_one_irq_desc(desc);
> +
> + if ( rc )
> + return rc;
> +
> + desc->irq = irq;
> + desc->action = NULL;
> + }
> +
> + return 0;
> +}
> +#else
> +/*
> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
> + * because in this case, is_espi will always return false.
> + */
> +static struct irq_desc *espi_to_desc(unsigned int irq)
> +{
> + ASSERT_UNREACHABLE();
> + return NULL;
> +}
> +
> +static int __init init_espi_data(void)
> +{
> + return 0;
> +}
> +#endif
> +
> static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
>
> struct irq_desc *__irq_to_desc(unsigned int irq)
> @@ -53,6 +104,9 @@ struct irq_desc *__irq_to_desc(unsigned int irq)
> if ( irq < NR_LOCAL_IRQS )
> return &this_cpu(local_irq_desc)[irq];
>
> + if ( is_espi(irq) )
> + return espi_to_desc(irq);
> +
> return &irq_desc[irq-NR_LOCAL_IRQS];
> }
>
> @@ -79,7 +133,7 @@ static int __init init_irq_data(void)
> desc->action = NULL;
> }
>
> - return 0;
> + return init_espi_data();
> }
>
> static int init_local_irq_data(unsigned int cpu)
--
WBR, Volodymyr
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI
2025-08-29 16:06 ` [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI Leonid Komarianskyi
@ 2025-08-29 19:55 ` Volodymyr Babchuk
2025-09-01 17:30 ` Leonid Komarianskyi
2025-08-31 14:56 ` Oleksandr Tyshchenko
1 sibling, 1 reply; 49+ messages in thread
From: Volodymyr Babchuk @ 2025-08-29 19:55 UTC (permalink / raw)
To: Leonid Komarianskyi
Cc: xen-devel@lists.xenproject.org, olekstysh@gmail.com,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hi Leonid,
Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
> Introduced appropriate register definitions, helper macros,
> and initialization of required GICv3.1 distributor registers
> to support eSPI. This type of interrupt is handled in the
> same way as regular SPI interrupts, with the following
> differences:
>
> 1) eSPIs can have up to 1024 interrupts, starting from the
> beginning of the range, whereas regular SPIs use INTIDs from
> 32 to 1019, totaling 988 interrupts;
> 2) eSPIs start at INTID 4096, necessitating additional interrupt
> index conversion during register operations.
>
> In case if appropriate config is disabled, or GIC HW doesn't
> support eSPI, the existing functionality will remain the same.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> ---
> Changes in V5:
> - fixed minor nits, no functional changes: changed u32 to uint32_t and
> added a comment noting that the configuration for eSPIs is the same as
> for regular SPIs
> - removed ifdefs for eSPI-specific offsets to reduce the number of
> ifdefs and code duplication in further changes
> - removed reviewed-by as moving offset from ifdefs requires additional
> confirmation from reviewers
>
> Changes in V4:
> - added offsets for GICD_IGRPMODRnE and GICD_NSACRnE that are required
> for vGIC emulation
> - added a log banner with eSPI information, similar to the one for
> regular SPI
> - added newline after ifdef and before gic_is_valid_line
> - added reviewed-by from Volodymyr Babchuk
>
> Changes in V3:
> - add __init attribute to gicv3_dist_espi_common_init
> - change open-codded eSPI register initialization to the appropriate
> gen-mask macro
> - fixed formatting for lines with more than 80 symbols
> - introduced gicv3_dist_espi_init_aff to be able to use stubs in case of
> CONFIG_GICV3_ESPI disabled
> - renamed parameter in the GICD_TYPER_ESPI_RANGE macro to espi_range
> (name was taken from GIC specification) to avoid confusion
> - changed type for i variable to unsigned int since it cannot be
> negative
>
> Changes in V2:
> - move gic_number_espis function from
> [PATCH 08/10] xen/arm: vgic: add resource management for extended SPIs
> to use it in the newly introduced gic_is_valid_espi
> - add gic_is_valid_espi which checks if IRQ number is in supported
> by HW eSPI range
> - update gic_is_valid_irq conditions to allow operations with eSPIs
> ---
> xen/arch/arm/gic-v3.c | 83 ++++++++++++++++++++++++++
> xen/arch/arm/include/asm/gic.h | 22 +++++++
> xen/arch/arm/include/asm/gic_v3_defs.h | 38 ++++++++++++
> 3 files changed, 143 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 29b7f68cba..4a7ce12f26 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -485,6 +485,36 @@ static void __iomem *get_addr_by_offset(struct irq_desc *irqd, u32 offset)
> default:
> break;
> }
> +#ifdef CONFIG_GICV3_ESPI
> + case ESPI_BASE_INTID ... ESPI_MAX_INTID:
> + {
> + uint32_t irq_index = ESPI_INTID2IDX(irqd->irq);
> +
> + switch ( offset )
> + {
> + case GICD_ISENABLER:
> + return (GICD + GICD_ISENABLERnE + (irq_index / 32) * 4);
> + case GICD_ICENABLER:
> + return (GICD + GICD_ICENABLERnE + (irq_index / 32) * 4);
> + case GICD_ISPENDR:
> + return (GICD + GICD_ISPENDRnE + (irq_index / 32) * 4);
> + case GICD_ICPENDR:
> + return (GICD + GICD_ICPENDRnE + (irq_index / 32) * 4);
> + case GICD_ISACTIVER:
> + return (GICD + GICD_ISACTIVERnE + (irq_index / 32) * 4);
> + case GICD_ICACTIVER:
> + return (GICD + GICD_ICACTIVERnE + (irq_index / 32) * 4);
> + case GICD_ICFGR:
> + return (GICD + GICD_ICFGRnE + (irq_index / 16) * 4);
> + case GICD_IROUTER:
> + return (GICD + GICD_IROUTERnE + irq_index * 8);
> + case GICD_IPRIORITYR:
> + return (GICD + GICD_IPRIORITYRnE + irq_index);
> + default:
> + break;
> + }
> + }
> +#endif
> default:
> break;
> }
> @@ -655,6 +685,55 @@ static void gicv3_set_irq_priority(struct irq_desc *desc,
> spin_unlock(&gicv3.lock);
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> +unsigned int gic_number_espis(void)
> +{
> + return gic_hw_ops->info->nr_espi;
> +}
> +
> +static void __init gicv3_dist_espi_common_init(uint32_t type)
> +{
> + unsigned int espi_nr, i;
> +
> + espi_nr = min(1024U, GICD_TYPER_ESPIS_NUM(type));
> + gicv3_info.nr_espi = espi_nr;
> + /* The GIC HW doesn't support eSPI, so we can leave from here */
> + if ( gicv3_info.nr_espi == 0 )
> + return;
> +
> + printk("GICv3: %d eSPI lines\n", gicv3_info.nr_espi);
> +
> + /* The configuration for eSPIs is similar to that for regular SPIs */
> + for ( i = 0; i < espi_nr; i += 16 )
> + writel_relaxed(0, GICD + GICD_ICFGRnE + (i / 16) * 4);
> +
> + for ( i = 0; i < espi_nr; i += 4 )
> + writel_relaxed(GIC_PRI_IRQ_ALL,
> + GICD + GICD_IPRIORITYRnE + (i / 4) * 4);
> +
> + for ( i = 0; i < espi_nr; i += 32 )
> + {
> + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICENABLERnE + (i / 32) * 4);
> + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICACTIVERnE + (i / 32) * 4);
> + }
> +
> + for ( i = 0; i < espi_nr; i += 32 )
> + writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPRnE + (i / 32) * 4);
> +}
> +
> +static void __init gicv3_dist_espi_init_aff(uint64_t affinity)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < gicv3_info.nr_espi; i++ )
> + writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTERnE + i * 8);
> +}
> +#else
> +static void __init gicv3_dist_espi_common_init(uint32_t type) { }
> +
> +static void __init gicv3_dist_espi_init_aff(uint64_t affinity) { }
> +#endif
> +
> static void __init gicv3_dist_init(void)
> {
> uint32_t type;
> @@ -700,6 +779,8 @@ static void __init gicv3_dist_init(void)
> for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i += 32 )
> writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPR + (i / 32) * 4);
>
> + gicv3_dist_espi_common_init(type);
> +
> gicv3_dist_wait_for_rwp();
>
> /* Turn on the distributor */
> @@ -713,6 +794,8 @@ static void __init gicv3_dist_init(void)
>
> for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i++ )
> writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTER + i * 8);
> +
> + gicv3_dist_espi_init_aff(affinity);
> }
>
> static int gicv3_enable_redist(void)
> diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
> index 3fcee42675..1e747dcd99 100644
> --- a/xen/arch/arm/include/asm/gic.h
> +++ b/xen/arch/arm/include/asm/gic.h
> @@ -306,8 +306,26 @@ extern void gic_dump_vgic_info(struct vcpu *v);
>
> /* Number of interrupt lines */
> extern unsigned int gic_number_lines(void);
> +#ifdef CONFIG_GICV3_ESPI
> +extern unsigned int gic_number_espis(void);
> +
> +static inline bool gic_is_valid_espi(unsigned int irq)
> +{
> + return (irq >= ESPI_BASE_INTID &&
> + irq < ESPI_IDX2INTID(gic_number_espis()));
You don't need external () here.
> +}
> +#else
> +static inline bool gic_is_valid_espi(unsigned int irq)
> +{
> + return false;
> +}
> +#endif
> +
> static inline bool gic_is_valid_line(unsigned int irq)
> {
> + if ( gic_is_valid_espi(irq) )
> + return true;
> +
> return irq < gic_number_lines();
> }
As you are going to rework this patch anyways, my I ask to rewrite this
function in the following way?
static inline bool gic_is_valid_line(unsigned int irq)
{
return irq < gic_number_lines() || gic_is_valid_espi(irq);
}
My justification is that (irq < gic_number_lines()) case is more likely,
so it is better to evaluate it first, only then check for eSPIs.
I am sorry, I should asked it earlier, but only after removing #ifdef I
saw that this part could be more optimal.
>
> @@ -325,6 +343,10 @@ struct gic_info {
> enum gic_version hw_version;
> /* Number of GIC lines supported */
> unsigned int nr_lines;
> +#ifdef CONFIG_GICV3_ESPI
> + /* Number of GIC eSPI supported */
> + unsigned int nr_espi;
> +#endif
> /* Number of LR registers */
> uint8_t nr_lrs;
> /* Maintenance irq number */
> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
> index 2af093e774..3370b4cd52 100644
> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
> @@ -37,6 +37,44 @@
> #define GICD_IROUTER1019 (0x7FD8)
> #define GICD_PIDR2 (0xFFE8)
>
> +/* Additional registers for GICv3.1 */
> +#define GICD_IGROUPRnE (0x1000)
> +#define GICD_IGROUPRnEN (0x107C)
> +#define GICD_ISENABLERnE (0x1200)
> +#define GICD_ISENABLERnEN (0x127C)
> +#define GICD_ICENABLERnE (0x1400)
> +#define GICD_ICENABLERnEN (0x147C)
> +#define GICD_ISPENDRnE (0x1600)
> +#define GICD_ISPENDRnEN (0x167C)
> +#define GICD_ICPENDRnE (0x1800)
> +#define GICD_ICPENDRnEN (0x187C)
> +#define GICD_ISACTIVERnE (0x1A00)
> +#define GICD_ISACTIVERnEN (0x1A7C)
> +#define GICD_ICACTIVERnE (0x1C00)
> +#define GICD_ICACTIVERnEN (0x1C7C)
> +#define GICD_IPRIORITYRnE (0x2000)
> +#define GICD_IPRIORITYRnEN (0x23FC)
> +#define GICD_ICFGRnE (0x3000)
> +#define GICD_ICFGRnEN (0x30FC)
> +#define GICD_IGRPMODRnE (0x3400)
> +#define GICD_IGRPMODRnEN (0x347C)
> +#define GICD_NSACRnE (0x3600)
> +#define GICD_NSACRnEN (0x36FC)
> +#define GICD_IROUTERnE (0x8000)
> +#define GICD_IROUTERnEN (0x9FFC)
> +
> +#ifdef CONFIG_GICV3_ESPI
> +#define GICD_TYPER_ESPI_SHIFT 8
> +#define GICD_TYPER_ESPI_RANGE_SHIFT 27
> +#define GICD_TYPER_ESPI_RANGE_MASK (0x1F)
> +#define GICD_TYPER_ESPI (1U << GICD_TYPER_ESPI_SHIFT)
> +#define GICD_TYPER_ESPI_RANGE(espi_range) ((((espi_range) & \
> + GICD_TYPER_ESPI_RANGE_MASK) + 1) * 32)
> +#define GICD_TYPER_ESPIS_NUM(typer) \
> + (((typer) & GICD_TYPER_ESPI) ? \
> + GICD_TYPER_ESPI_RANGE((typer) >> GICD_TYPER_ESPI_RANGE_SHIFT) : 0)
> +#endif
> +
> /* Common between GICD_PIDR2 and GICR_PIDR2 */
> #define GIC_PIDR2_ARCH_MASK (0xf0)
> #define GIC_PIDR2_ARCH_GICv3 (0x30)
--
WBR, Volodymyr
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-08-29 16:06 ` [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers Leonid Komarianskyi
@ 2025-08-29 20:12 ` Volodymyr Babchuk
2025-09-01 18:32 ` Leonid Komarianskyi
2025-09-01 12:38 ` Oleksandr Tyshchenko
2025-09-02 16:55 ` Julien Grall
2 siblings, 1 reply; 49+ messages in thread
From: Volodymyr Babchuk @ 2025-08-29 20:12 UTC (permalink / raw)
To: Leonid Komarianskyi
Cc: xen-devel@lists.xenproject.org, olekstysh@gmail.com,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hi Leonid,
Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
> Implemented support for GICv3.1 extended SPI registers for vGICv3,
> allowing the emulation of eSPI-specific behavior for guest domains.
> The implementation includes read and write emulation for eSPI-related
> registers (e.g., GICD_ISENABLERnE, GICD_IROUTERnE, and others),
> following a similar approach to the handling of regular SPIs.
>
> The eSPI registers, previously located in reserved address ranges,
> are now adjusted to support MMIO read and write operations correctly
> when CONFIG_GICV3_ESPI is enabled.
>
> The availability of eSPIs and the number of emulated extended SPIs
> for guest domains is reported by setting the appropriate bits in the
> GICD_TYPER register, based on the number of eSPIs requested by the
> domain and supported by the hardware. In cases where the configuration
> option is disabled, the hardware does not support eSPIs, or the domain
> does not request such interrupts, the functionality remains unchanged.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
I have a couple of comments about coding style, but apart from that it
looks really good. With these issues fixed:
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>
> ---
> Changes in V5:
> - since eSPI-specific defines and macros are now available even when the
> appropriate config is disabled, this allows us to remove many
> #ifdef guards and reuse the existing code for regular SPIs for eSPIs as
> well, as eSPIs are processed similarly. This improves code readability
> and consolidates the register ranges for SPIs and eSPIs in a single
> place, simplifying future changes or fixes for SPIs and their
> counterparts from the extended range
> - moved vgic_ext_rank_offset() from
> [08/12] xen/arm: vgic: add resource management for extended SPIs
> as the function was unused before this patch
> - added stub implementation of vgic_ext_rank_offset() when CONFIG_GICV3_ESPI=n
> - removed unnecessary defines for reserved ranges, which were introduced in
> V4 to reduce the number of #ifdefs. The issue is now resolved by
> allowing the use of SPI-specific offsets and reworking the logic
>
> Changes in V4:
> - added missing RAZ and write ignore eSPI-specific registers ranges:
> GICD_NSACRnE and GICD_IGRPMODRnE
> - changed previously reserved range to cover GICD_NSACRnE and
> GICD_IGRPMODRnE
> - introduced GICD_RESERVED_RANGE<n>_START/END defines to remove
> hardcoded values and reduce the number of ifdefs
> - fixed reserved ranges with eSPI option enabled: added missing range
> 0x0F30-0x0F7C
> - updated the logic for domains that do not support eSPI, but Xen is
> compiled with the eSPI option. Now, prior to other MMIO checks, we
> verify whether eSPI is available for the domain or not. If not, it
> behaves as it does currently - RAZ and WI
> - fixed print for GICD_ICACTIVERnE
> - fixed new lines formatting for switch-case
>
> Changes in V3:
> - changed vgic_store_irouter parameters - instead of offset virq is
> used, to remove the additional bool espi parameter and simplify
> checks. Also, adjusted parameters for regular SPI. Since the offset
> parameter was used only for calculating virq number and then reused for
> finding rank offset, it will not affect functionality.
> - fixed formatting for goto lables - added newlines after condition
> - fixed logs for GICD_ISACTIVERnE and GICD_ICACTIVERnE handlers
> - removed #ifdefs in 2 places where they were adjacent and could be merged
>
> Changes in V2:
> - add missing rank index conversion for pending and inflight irqs
> ---
> xen/arch/arm/include/asm/vgic.h | 4 +
> xen/arch/arm/vgic-v3.c | 198 ++++++++++++++++++++++++++------
> xen/arch/arm/vgic.c | 23 ++++
> 3 files changed, 192 insertions(+), 33 deletions(-)
>
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 3aa22114ba..103bc3c74b 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -314,6 +314,10 @@ extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v,
> unsigned int b,
> unsigned int n,
> unsigned int s);
> +extern struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v,
> + unsigned int b,
> + unsigned int n,
> + unsigned int s);
> extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
> extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
> extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index 4369c55177..b5d766c98f 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -111,13 +111,10 @@ static uint64_t vgic_fetch_irouter(struct vgic_irq_rank *rank,
> * Note the offset will be aligned to the appropriate boundary.
> */
> static void vgic_store_irouter(struct domain *d, struct vgic_irq_rank *rank,
> - unsigned int offset, uint64_t irouter)
> + unsigned int virq, uint64_t irouter)
> {
> struct vcpu *new_vcpu, *old_vcpu;
> - unsigned int virq;
> -
> - /* There is 1 vIRQ per IROUTER */
> - virq = offset / NR_BYTES_PER_IROUTER;
> + unsigned int offset;
>
> /*
> * The IROUTER0-31, used for SGIs/PPIs, are reserved and should
> @@ -685,13 +682,20 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> {
> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /* We do not implement security extensions for guests, read zero */
> if ( dabt.size != DABT_WORD ) goto bad_width;
> goto read_as_zero;
>
> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> + if ( reg >= GICD_ISENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> *r = vreg_reg32_extract(rank->ienable, info);
> @@ -699,8 +703,13 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> return 1;
>
> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> + if ( reg >= GICD_ICENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> *r = vreg_reg32_extract(rank->ienable, info);
> @@ -710,20 +719,29 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> /* Read the pending status of an IRQ via GICD/GICR is not supported */
> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> goto read_as_zero;
>
> /* Read the active status of an IRQ via GICD/GICR is not supported */
> case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
> case VRANGE32(GICD_ICACTIVER, GICD_ICACTIVERN):
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> goto read_as_zero;
>
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> {
> uint32_t ipriorityr;
> uint8_t rank_index;
>
> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
> + if ( reg >= GICD_IPRIORITYRnE )
> + rank = vgic_ext_rank_offset(v, 8, reg - GICD_IPRIORITYRnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> rank_index = REG_RANK_INDEX(8, reg - GICD_IPRIORITYR, DABT_WORD);
>
> @@ -737,11 +755,15 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> }
>
> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> {
> uint32_t icfgr;
>
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> + if ( reg >= GICD_ICFGRnE )
> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> icfgr = rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR, DABT_WORD)];
> @@ -782,46 +804,81 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
> {
> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /* We do not implement security extensions for guests, write ignore */
> goto write_ignore_32;
>
> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> + if ( reg >= GICD_ISENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> tr = rank->ienable;
> vreg_reg32_setbits(&rank->ienable, r, info);
> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
> + if ( reg >= GICD_ISENABLERnE )
> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
> + EXT_RANK_IDX2NUM(rank->index));
> + else
> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
> vgic_unlock_rank(v, rank, flags);
> return 1;
>
> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> + if ( reg >= GICD_ICENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> tr = rank->ienable;
> vreg_reg32_clearbits(&rank->ienable, r, info);
> - vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
> + if ( reg >= GICD_ICENABLERnE )
> + vgic_disable_irqs(v, (~rank->ienable) & tr,
> + EXT_RANK_IDX2NUM(rank->index));
> + else
> + vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
> vgic_unlock_rank(v, rank, flags);
> return 1;
>
> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
> + if ( reg >= GICD_ISPENDRnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISPENDRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
>
> - vgic_set_irqs_pending(v, r, rank->index);
> + if ( reg >= GICD_ISPENDRnE )
> + vgic_set_irqs_pending(v, r, EXT_RANK_IDX2NUM(rank->index));
> + else
> + vgic_set_irqs_pending(v, r, rank->index);
>
> return 1;
>
> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
> + if ( reg >= GICD_ICPENDRnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICPENDRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
>
> - vgic_check_inflight_irqs_pending(v, rank->index, r);
> + if ( reg >= GICD_ICPENDRnE )
> + vgic_check_inflight_irqs_pending(v,
> + EXT_RANK_IDX2NUM(rank->index), r);
> + else
> + vgic_check_inflight_irqs_pending(v, rank->index, r);
>
> goto write_ignore;
>
> @@ -838,16 +895,38 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
> v, name, r, reg - GICD_ICACTIVER);
> goto write_ignore_32;
>
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + if ( dabt.size != DABT_WORD )
> + goto bad_width;
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to ISACTIVER%dE\n",
> + v, name, r, reg - GICD_ISACTIVERnE);
> + return 0;
> +
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to ICACTIVER%dE\n",
> + v, name, r, reg - GICD_ICACTIVERnE);
> + goto write_ignore_32;
> +
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> {
> - uint32_t *ipriorityr, priority;
> + uint32_t *ipriorityr, priority, offset;
>
> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
> + if ( reg >= GICD_IPRIORITYRnE ) {
Brace should go on new line
> + offset = reg - GICD_IPRIORITYRnE;
> + rank = vgic_ext_rank_offset(v, 8, offset, DABT_WORD);
> + }
> + else
> + {
> + offset = reg - GICD_IPRIORITYR;
> + rank = vgic_rank_offset(v, 8, offset, DABT_WORD);
> + }
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> - ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, reg - GICD_IPRIORITYR,
> - DABT_WORD)];
> + ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, offset, DABT_WORD)];
> priority = ACCESS_ONCE(*ipriorityr);
> vreg_reg32_update(&priority, r, info);
> ACCESS_ONCE(*ipriorityr) = priority;
> @@ -859,10 +938,14 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
> goto write_ignore_32;
>
> case VRANGE32(GICD_ICFGR + 4, GICD_ICFGRN): /* PPI + SPIs */
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> /* ICFGR1 for PPI's, which is implementation defined
> if ICFGR1 is programmable or not. We chose to program */
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> + if ( reg >= GICD_ICFGRnE )
> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> vreg_reg32_update(&rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR,
> @@ -1129,6 +1212,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> typer |= GICD_TYPE_LPIS;
>
> typer |= (v->domain->arch.vgic.intid_bits - 1) << GICD_TYPE_ID_BITS_SHIFT;
> +#ifdef CONFIG_GICV3_ESPI
> + if ( v->domain->arch.vgic.nr_espis > 0 )
> + {
> + /* Set eSPI support bit for the domain */
> + typer |= GICD_TYPER_ESPI;
> + /* Set ESPI range bits */
> + typer |= (DIV_ROUND_UP(v->domain->arch.vgic.nr_espis, 32) - 1)
> + << GICD_TYPER_ESPI_RANGE_SHIFT;
> + }
> +#endif
>
> *r = vreg_reg32_extract(typer, info);
>
> @@ -1194,6 +1287,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /*
> * Above all register are common with GICR and GICD
> * Manage in common
> @@ -1201,6 +1304,7 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> return __vgic_v3_distr_common_mmio_read("vGICD", v, info, gicd_reg, r);
>
> case VRANGE32(GICD_NSACR, GICD_NSACRN):
> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
> /* We do not implement security extensions for guests, read zero */
> goto read_as_zero_32;
>
> @@ -1216,16 +1320,21 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> /* Replaced with GICR_ISPENDR0. So ignore write */
> goto read_as_zero_32;
>
> - case VRANGE32(0x0F30, 0x60FC):
> + case VRANGE32(0x0F30, 0x0FFC):
> goto read_reserved;
>
> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
> {
> uint64_t irouter;
>
> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
> - DABT_DOUBLE_WORD);
> + if ( gicd_reg >= GICD_IROUTERnE )
> + rank = vgic_ext_rank_offset(v, 64, gicd_reg - GICD_IROUTERnE,
> + DABT_DOUBLE_WORD);
> + else
> + rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
> + DABT_DOUBLE_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
> @@ -1235,8 +1344,8 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>
> return 1;
> }
> -
> - case VRANGE32(0x7FE0, 0xBFFC):
> + case VRANGE32(0x3700, 0x60FC):
> + case VRANGE32(0xA004, 0xBFFC):
> goto read_reserved;
>
> case VRANGE32(0xC000, 0xFFCC):
> @@ -1382,12 +1491,23 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /* Above registers are common with GICR and GICD
> * Manage in common */
> return __vgic_v3_distr_common_mmio_write("vGICD", v, info,
> gicd_reg, r);
>
> case VRANGE32(GICD_NSACR, GICD_NSACRN):
> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
> /* We do not implement security extensions for guests, write ignore */
> goto write_ignore_32;
>
> @@ -1405,26 +1525,38 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
> if ( dabt.size != DABT_WORD ) goto bad_width;
> return 0;
>
> - case VRANGE32(0x0F30, 0x60FC):
> + case VRANGE32(0x0F30, 0x0FFC):
> goto write_reserved;
>
> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
> {
> uint64_t irouter;
> + unsigned int offset, virq;
>
> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
> - DABT_DOUBLE_WORD);
> + if ( gicd_reg >= GICD_IROUTERnE ) {
Braces should go into separate line
> + offset = gicd_reg - GICD_IROUTERnE;
> + rank = vgic_ext_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
> + } else {
... so "else" also should be on a separate line
> + offset = gicd_reg - GICD_IROUTER;
> + rank = vgic_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
> + }
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> - irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
> + irouter = vgic_fetch_irouter(rank, offset);
> vreg_reg64_update(&irouter, r, info);
> - vgic_store_irouter(v->domain, rank, gicd_reg - GICD_IROUTER, irouter);
> + if ( gicd_reg >= GICD_IROUTERnE )
> + virq = ESPI_IDX2INTID(offset / NR_BYTES_PER_IROUTER);
> + else
> + virq = offset / NR_BYTES_PER_IROUTER;
> + vgic_store_irouter(v->domain, rank, virq, irouter);
> vgic_unlock_rank(v, rank, flags);
> return 1;
> }
>
> - case VRANGE32(0x7FE0, 0xBFFC):
> + case VRANGE32(0x3700, 0x60FC):
> + case VRANGE32(0xA004, 0xBFFC):
> goto write_reserved;
>
> case VRANGE32(0xC000, 0xFFCC):
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index c9b9528c66..27ffdf316c 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -193,6 +193,18 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
> }
>
> #ifdef CONFIG_GICV3_ESPI
> +/*
> + * The function behavior is the same as for regular SPIs (vgic_rank_offset),
> + * but it operates with extended SPI ranks.
> + */
> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
> + unsigned int n, unsigned int s)
> +{
> + unsigned int rank = REG_RANK_NR(b, (n >> s));
> +
> + return vgic_get_rank(v, rank + EXT_RANK_MIN);
> +}
> +
> static unsigned int vgic_num_spi_lines(struct domain *d)
> {
> return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
> @@ -241,6 +253,17 @@ struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
> {
> return NULL;
> }
> +
> +/*
> + * It is expected that, in the case of CONFIG_GICV3_ESPI=n, the function will
> + * return NULL. In this scenario, mmio_read/mmio_write will be treated as
> + * RAZ/WI, as expected.
> + */
> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
> + unsigned int n, unsigned int s)
> +{
> + return NULL;
> +}
> #endif
>
> static unsigned int vgic_num_alloc_irqs(struct domain *d)
--
WBR, Volodymyr
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-08-29 16:06 ` [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs Leonid Komarianskyi
@ 2025-08-29 20:45 ` Volodymyr Babchuk
2025-08-31 15:58 ` Oleksandr Tyshchenko
2025-09-01 17:38 ` Leonid Komarianskyi
2025-09-02 14:13 ` Julien Grall
1 sibling, 2 replies; 49+ messages in thread
From: Volodymyr Babchuk @ 2025-08-29 20:45 UTC (permalink / raw)
To: Leonid Komarianskyi
Cc: xen-devel@lists.xenproject.org, olekstysh@gmail.com,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hi Leonid,
Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
> This change introduces resource management in the VGIC to handle
> extended SPIs introduced in GICv3.1. The pending_irqs and
> allocated_irqs arrays are resized to support the required
> number of eSPIs, based on what is supported by the hardware and
> requested by the guest. A new field, ext_shared_irqs, is added
> to the VGIC structure to store information about eSPIs, similar
> to how shared_irqs is used for regular SPIs.
>
> Since the eSPI range starts at INTID 4096 and INTIDs between 1025
> and 4095 are reserved, helper macros are introduced to simplify the
> transformation of indices and to enable easier access to eSPI-specific
> resources. These changes prepare the VGIC for processing eSPIs as
> required by future functionality.
>
> The initialization and deinitialization paths for vgic have been updated
> to allocate and free these resources appropriately. Additionally,
> updated handling of INTIDs greater than 1024, passed from the toolstack
> during domain creation, and verification logic ensures only valid SPI or
> eSPI INTIDs are used.
>
> The existing SPI behavior remains unaffected when guests do not request
> eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
> option is disabled.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> ---
> Changes in V5:
> - removed the has_espi field because it can be determined by checking
> whether domain->arch.vgic.nr_espis is zero or not
> - since vgic_ext_rank_offset is not used in this patch, it has been
> moved to the appropriate patch in the patch series, which implements
> vgic eSPI registers emulation and requires this function
> - removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
> and code duplication in further changes
> - fixed minor nit: used %pd for printing domain with its ID
>
> Changes in V4:
> - added has_espi field to simplify determining whether a domain is able
> to operate with eSPI
> - fixed formatting issues and misspellings
>
> Changes in V3:
> - fixed formatting for lines with more than 80 symbols
> - introduced helper functions to be able to use stubs in case of
> CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
> #ifdefs
> - fixed checks for nr_spis in domain_vgic_init
> - updated comment about nr_spis adjustments with dom0less mention
> - moved comment with additional explanations before checks
> - used unsigned int for indexes since they cannot be negative
> - removed unnecessary parentheses
> - move vgic_ext_rank_offset to the below ifdef guard, to reduce the
> number of ifdefs
>
> Changes in V2:
> - change is_espi_rank to is_valid_espi_rank to verify whether the array
> element ext_shared_irqs exists. The previous version, is_espi_rank,
> only checked if the rank index was less than the maximum possible eSPI
> rank index, but this could potentially result in accessing a
> non-existing array element. To address this, is_valid_espi_rank was
> introduced, which ensures that the required eSPI rank exists
> - move gic_number_espis to
> xen/arm: gicv3: implement handling of GICv3.1 eSPI
> - update vgic_is_valid_irq checks to allow operating with eSPIs
> - remove redundant newline in vgic_allocate_virq
> ---
> xen/arch/arm/include/asm/vgic.h | 12 ++
> xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
> 2 files changed, 208 insertions(+), 3 deletions(-)
>
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 3e7cbbb196..912d5b7694 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -146,6 +146,10 @@ struct vgic_dist {
> int nr_spis; /* Number of SPIs */
> unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
> struct vgic_irq_rank *shared_irqs;
> +#ifdef CONFIG_GICV3_ESPI
> + struct vgic_irq_rank *ext_shared_irqs;
> + int nr_espis; /* Number of extended SPIs */
> +#endif
> /*
> * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
> * struct arch_vcpu.
> @@ -243,6 +247,14 @@ struct vgic_ops {
> /* Number of ranks of interrupt registers for a domain */
> #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
>
> +#ifdef CONFIG_GICV3_ESPI
> +#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
> +#endif
> +#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
> +#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
> +#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
> +#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
> +
> #define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
> #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
>
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 2bbf4d99aa..c9b9528c66 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -27,9 +27,82 @@
>
> bool vgic_is_valid_line(struct domain *d, unsigned int virq)
> {
> +#ifdef CONFIG_GICV3_ESPI
> + if ( virq >= ESPI_BASE_INTID &&
> + virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
> + return true;
> +#endif
> +
> return virq < vgic_num_irqs(d);
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> +/*
> + * Since eSPI indexes start from 4096 and numbers from 1024 to
> + * 4095 are forbidden, we need to check both lower and upper
> + * limits for ranks.
> + */
> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
> +{
> + return rank >= EXT_RANK_MIN &&
> + EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
> +}
> +
> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
> + unsigned int rank)
> +{
> + return &v->domain->arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
> +}
> +
> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
> +{
> + return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
> + d->arch.vgic.allocated_irqs);
> +}
> +
> +static void arch_move_espis(struct vcpu *v)
I don't need you need a copy of arch_move_irqs(). Se below for more info.
> +{
> + const cpumask_t *cpu_mask = cpumask_of(v->processor);
> + struct domain *d = v->domain;
> + struct pending_irq *p;
> + struct vcpu *v_target;
> + unsigned int i;
> +
> + for ( i = ESPI_BASE_INTID;
> + i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
> + {
> + v_target = vgic_get_target_vcpu(v, i);
> + p = irq_to_pending(v_target, i);
> +
> + if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
> + irq_set_affinity(p->desc, cpu_mask);
> + }
> +}
> +#else
> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
> +{
> + return false;
> +}
> +
> +/*
> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
> + * because in this case, is_valid_espi_rank will always return false.
> + */
> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
> + unsigned int rank)
> +{
> + ASSERT_UNREACHABLE();
> + return NULL;
> +}
> +
> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
> +{
> + return false;
> +}
> +
> +static void arch_move_espis(struct vcpu *v) { }
> +#endif
> +
> static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
> unsigned int rank)
> {
> @@ -37,6 +110,8 @@ static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
> return v->arch.vgic.private_irqs;
> else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
> return &v->domain->arch.vgic.shared_irqs[rank - 1];
> + else if ( is_valid_espi_rank(v->domain, rank) )
> + return vgic_get_espi_rank(v, rank);
> else
> return NULL;
> }
> @@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
> return 0;
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> +static unsigned int vgic_num_spi_lines(struct domain *d)
> +{
> + return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
> +}
> +
> +static int init_vgic_espi(struct domain *d)
> +{
> + unsigned int i, idx;
> +
> + if ( d->arch.vgic.nr_espis == 0 )
> + return 0;
> +
> + d->arch.vgic.ext_shared_irqs =
> + xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
> + if ( d->arch.vgic.ext_shared_irqs == NULL )
> + return -ENOMEM;
> +
> + for ( i = d->arch.vgic.nr_spis, idx = 0;
> + i < vgic_num_spi_lines(d); i++, idx++ )
> + vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
> + ESPI_IDX2INTID(idx));
> +
> + for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
> + vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
> +
> + return 0;
> +}
> +
> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
I know that I should made this observation in previous version, but I
didn't, sorry for that. Anyways, I don't think that this is a good idea
to introduce this function and vgic_reserve_espi_virq(), as well as
arch_move_espis(), actually, because in each case this is a code
duplication, which is not good.
I think that instead you need to introduce a pair of helpers that will
map any (e)SPI number to pending_irq[]/allocate_irqs index and back.
somethink like
static inline unsigned virq_to_index(int virq)
{
if (is_espi(virq))
return ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
return virq;
}
See below for examples.
> +{
> + irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
> + return &d->arch.vgic.pending_irqs[irq];
> +}
> +#else
> +static unsigned int init_vgic_espi(struct domain *d)
> +{
> + return 0;
> +}
> +
> +static unsigned int vgic_num_spi_lines(struct domain *d)
> +{
> + return d->arch.vgic.nr_spis;
> +}
> +
> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
> +{
> + return NULL;
> +}
> +#endif
> +
> +static unsigned int vgic_num_alloc_irqs(struct domain *d)
> +{
> + return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
> +}
> +
> int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> {
> int i;
> @@ -131,6 +262,36 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> */
> nr_spis = ROUNDUP(nr_spis, 32);
>
> +#ifdef CONFIG_GICV3_ESPI
> + /*
> + * During domain creation, the dom0less DomUs code or toolstack specifies
> + * the maximum INTID, which is defined in the domain config subtracted by
> + * 32 to cover the local IRQs (please see the comment to VGIC_DEF_NR_SPIS).
> + * To compute the actual number of eSPI that will be usable for,
> + * add back 32.
> + */
> + if ( nr_spis + 32 > ESPI_IDX2INTID(NR_ESPI_IRQS) )
> + return -EINVAL;
> +
> + if ( nr_spis + 32 >= ESPI_BASE_INTID )
> + {
> + d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32, 1024U);
> + /* Verify if GIC HW can handle provided INTID */
> + if ( d->arch.vgic.nr_espis > gic_number_espis() )
> + return -EINVAL;
> + /*
> + * Set the maximum available number for regular
> + * SPI to pass the next check
> + */
> + nr_spis = VGIC_DEF_NR_SPIS;
> + }
> + else
> + {
> + /* Domain will use the regular SPI range */
> + d->arch.vgic.nr_espis = 0;
> + }
> +#endif
> +
> /* Limit the number of virtual SPIs supported to (1020 - 32) = 988 */
> if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
> return -EINVAL;
> @@ -145,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> return -ENOMEM;
>
> d->arch.vgic.pending_irqs =
> - xzalloc_array(struct pending_irq, d->arch.vgic.nr_spis);
> + xzalloc_array(struct pending_irq, vgic_num_spi_lines(d));
> if ( d->arch.vgic.pending_irqs == NULL )
> return -ENOMEM;
>
> @@ -156,12 +317,16 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
> vgic_rank_init(&d->arch.vgic.shared_irqs[i], i + 1, 0);
>
> + ret = init_vgic_espi(d);
> + if ( ret )
> + return ret;
> +
> ret = d->arch.vgic.handler->domain_init(d);
> if ( ret )
> return ret;
>
> d->arch.vgic.allocated_irqs =
> - xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
> + xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_alloc_irqs(d)));
> if ( !d->arch.vgic.allocated_irqs )
> return -ENOMEM;
>
> @@ -195,9 +360,27 @@ void domain_vgic_free(struct domain *d)
> }
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> + for ( i = 0; i < d->arch.vgic.nr_espis; i++ )
> + {
> + struct pending_irq *p = spi_to_pending(d, ESPI_IDX2INTID(i));
> +
> + if ( p->desc )
> + {
> + ret = release_guest_irq(d, p->irq);
> + if ( ret )
> + dprintk(XENLOG_G_WARNING, "%pd: Failed to release virq %u ret = %d\n",
> + d, p->irq, ret);
> + }
> + }
> +#endif
> +
> if ( d->arch.vgic.handler )
> d->arch.vgic.handler->domain_free(d);
> xfree(d->arch.vgic.shared_irqs);
> +#ifdef CONFIG_GICV3_ESPI
> + xfree(d->arch.vgic.ext_shared_irqs);
> +#endif
> xfree(d->arch.vgic.pending_irqs);
> xfree(d->arch.vgic.allocated_irqs);
> }
> @@ -331,6 +514,8 @@ void arch_move_irqs(struct vcpu *v)
> if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
> irq_set_affinity(p->desc, cpu_mask);
> }
> +
> + arch_move_espis(v);
> }
>
> void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
> @@ -538,6 +723,8 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
> n = &v->arch.vgic.pending_irqs[irq];
> else if ( is_lpi(irq) )
> n = v->domain->arch.vgic.handler->lpi_to_pending(v->domain, irq);
> + else if ( is_espi(irq) )
> + n = espi_to_pending(v->domain, irq);
> else
> n = &v->domain->arch.vgic.pending_irqs[irq - 32];
> return n;
> @@ -547,6 +734,9 @@ struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq)
> {
> ASSERT(irq >= NR_LOCAL_IRQS);
>
> + if ( is_espi(irq) )
> + return espi_to_pending(d, irq);
> +
here you can just do
idx = virq_to_idx(virq);
> return &d->arch.vgic.pending_irqs[irq - 32];
and
return &d->arch.vgic.pending_irqs[idx];
instead
> }
>
> @@ -668,6 +858,9 @@ bool vgic_reserve_virq(struct domain *d, unsigned int virq)
> if ( !vgic_is_valid_line(d, virq) )
> return false;
>
> + if ( is_espi(virq) )
> + return vgic_reserve_espi_virq(d, virq);
> +
here you can just do
idx = virq_to_idx(virq)
> return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
and then just
return !test_and_set_bit(idx, d->arch.vgic.allocated_irqs);
> }
>
> @@ -685,7 +878,7 @@ int vgic_allocate_virq(struct domain *d, bool spi)
> else
> {
> first = 32;
> - end = vgic_num_irqs(d);
> + end = vgic_num_alloc_irqs(d);
> }
I thinj you need to recalculate "virq" value at the end of this
function. You'll return index in bitfield, but this is not the same is
IRQ number in case of eSPIs. The helpers I mentioned before can help here.
>
> /*
Lastly, I think that it is very wasteful to allocate pending_irqs as
continuous array, because it will consist mostly of unused entries,
especially with eSPIs enable. Probably, better approach will be to use radix
tree. As a bonus, you can use IRQ line number as a key, and get rid of
all these is_espi() checks and mappings between IRQ number and index in
the array. But this is a much more drastic change, and I don't think that it
should be done in this patch series...
--
WBR, Volodymyr
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations
2025-08-29 16:06 ` [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations Leonid Komarianskyi
@ 2025-08-31 12:42 ` Oleksandr Tyshchenko
2025-09-01 14:36 ` Leonid Komarianskyi
0 siblings, 1 reply; 49+ messages in thread
From: Oleksandr Tyshchenko @ 2025-08-31 12:42 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel,
Volodymyr Babchuk, Julien Grall
On 29.08.25 19:06, Leonid Komarianskyi wrote:
Hello Leonid
> Currently, many common functions perform the same operations to calculate
> GIC register addresses. This patch consolidates the similar code into
> a separate helper function to improve maintainability and reduce duplication.
> This refactoring also simplifies the implementation of eSPI support in future
> changes.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
> Acked-by: Julien Grall <jgrall@amazon.com>
Reviewed-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
with one NIT ...
>
> ---
> Changes in V5:
> - fixed a minor nit: changed %d to %u in the warning message because the
> IRQ number cannot be less than zero
> - added acked-by from Julien Grall
>
> Changes in V4:
> - no changes
>
> Changes in V3:
> - changed panic() in get_addr_by_offset() to printing warning and
> ASSERT_UNREACHABLE()
> - added verification of return pointer from get_addr_by_offset() in the
> callers
> - moved invocation of get_addr_by_offset() from spinlock guards, since
> it is not necessarry
> - added RB from Volodymyr Babchuk
>
> Changes in V2:
> - no changes
> ---
> xen/arch/arm/gic-v3.c | 114 +++++++++++++++++++++++----------
> xen/arch/arm/include/asm/irq.h | 1 +
> 2 files changed, 81 insertions(+), 34 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index cd3e1acf79..29b7f68cba 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -445,17 +445,67 @@ static void gicv3_dump_state(const struct vcpu *v)
> }
> }
>
> +static void __iomem *get_addr_by_offset(struct irq_desc *irqd, u32 offset)
... please use uint32_t for new code
(could probably updated on commit)
> +{
> + switch ( irqd->irq )
> + {
> + case 0 ... (NR_GIC_LOCAL_IRQS - 1):
> + switch ( offset )
> + {
> + case GICD_ISENABLER:
> + case GICD_ICENABLER:
> + case GICD_ISPENDR:
> + case GICD_ICPENDR:
> + case GICD_ISACTIVER:
> + case GICD_ICACTIVER:
> + return (GICD_RDIST_SGI_BASE + offset);
> + case GICD_ICFGR:
> + return (GICD_RDIST_SGI_BASE + GICR_ICFGR1);
> + case GICD_IPRIORITYR:
> + return (GICD_RDIST_SGI_BASE + GICR_IPRIORITYR0 + irqd->irq);
> + default:
> + break;
> + }
> + case NR_GIC_LOCAL_IRQS ... SPI_MAX_INTID:
> + switch ( offset )
> + {
> + case GICD_ISENABLER:
> + case GICD_ICENABLER:
> + case GICD_ISPENDR:
> + case GICD_ICPENDR:
> + case GICD_ISACTIVER:
> + case GICD_ICACTIVER:
> + return (GICD + offset + (irqd->irq / 32) * 4);
> + case GICD_ICFGR:
> + return (GICD + GICD_ICFGR + (irqd->irq / 16) * 4);
> + case GICD_IROUTER:
> + return (GICD + GICD_IROUTER + irqd->irq * 8);
> + case GICD_IPRIORITYR:
> + return (GICD + GICD_IPRIORITYR + irqd->irq);
> + default:
> + break;
> + }
> + default:
> + break;
> + }
> +
> + /* Something went wrong, we shouldn't be able to reach here */
> + printk(XENLOG_WARNING "GICv3: WARNING: Invalid offset 0x%x for IRQ#%u",
> + offset, irqd->irq);
> + ASSERT_UNREACHABLE();
> +
> + return NULL;
> +}
> +
> static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
> {
> u32 mask = 1U << (irqd->irq % 32);
> - void __iomem *base;
> + void __iomem *addr = get_addr_by_offset(irqd, offset);
>
> - if ( irqd->irq < NR_GIC_LOCAL_IRQS )
> - base = GICD_RDIST_SGI_BASE;
> - else
> - base = GICD;
> + if ( addr == NULL )
> + return;
>
> - writel_relaxed(mask, base + offset + (irqd->irq / 32) * 4);
> + writel_relaxed(mask, addr);
>
> if ( wait_for_rwp )
> gicv3_wait_for_rwp(irqd->irq);
> @@ -463,15 +513,12 @@ static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
>
> static bool gicv3_peek_irq(struct irq_desc *irqd, u32 offset)
> {
> - void __iomem *base;
> - unsigned int irq = irqd->irq;
> + void __iomem *addr = get_addr_by_offset(irqd, offset);
>
> - if ( irq >= NR_GIC_LOCAL_IRQS)
> - base = GICD + (irq / 32) * 4;
> - else
> - base = GICD_RDIST_SGI_BASE;
> + if ( addr == NULL )
> + return false;
>
> - return !!(readl(base + offset) & (1U << (irq % 32)));
> + return !!(readl(addr) & (1U << (irqd->irq % 32)));
> }
>
> static void gicv3_unmask_irq(struct irq_desc *irqd)
> @@ -558,30 +605,28 @@ static inline uint64_t gicv3_mpidr_to_affinity(int cpu)
> static void gicv3_set_irq_type(struct irq_desc *desc, unsigned int type)
> {
> uint32_t cfg, actual, edgebit;
> - void __iomem *base;
> - unsigned int irq = desc->irq;
> + void __iomem *addr;
>
> /* SGI's are always edge-triggered not need to call GICD_ICFGR0 */
> - ASSERT(irq >= NR_GIC_SGI);
> + ASSERT(desc->irq >= NR_GIC_SGI);
>
> - spin_lock(&gicv3.lock);
> + addr = get_addr_by_offset(desc, GICD_ICFGR);
> + if ( addr == NULL )
> + return;
>
> - if ( irq >= NR_GIC_LOCAL_IRQS)
> - base = GICD + GICD_ICFGR + (irq / 16) * 4;
> - else
> - base = GICD_RDIST_SGI_BASE + GICR_ICFGR1;
> + spin_lock(&gicv3.lock);
>
> - cfg = readl_relaxed(base);
> + cfg = readl_relaxed(addr);
>
> - edgebit = 2u << (2 * (irq % 16));
> + edgebit = 2u << (2 * (desc->irq % 16));
> if ( type & IRQ_TYPE_LEVEL_MASK )
> cfg &= ~edgebit;
> else if ( type & IRQ_TYPE_EDGE_BOTH )
> cfg |= edgebit;
>
> - writel_relaxed(cfg, base);
> + writel_relaxed(cfg, addr);
>
> - actual = readl_relaxed(base);
> + actual = readl_relaxed(addr);
> if ( ( cfg & edgebit ) ^ ( actual & edgebit ) )
> {
> printk(XENLOG_WARNING "GICv3: WARNING: "
> @@ -600,16 +645,13 @@ static void gicv3_set_irq_type(struct irq_desc *desc, unsigned int type)
> static void gicv3_set_irq_priority(struct irq_desc *desc,
> unsigned int priority)
> {
> - unsigned int irq = desc->irq;
> + void __iomem *addr = get_addr_by_offset(desc, GICD_IPRIORITYR);
>
> - spin_lock(&gicv3.lock);
> -
> - /* Set priority */
> - if ( irq < NR_GIC_LOCAL_IRQS )
> - writeb_relaxed(priority, GICD_RDIST_SGI_BASE + GICR_IPRIORITYR0 + irq);
> - else
> - writeb_relaxed(priority, GICD + GICD_IPRIORITYR + irq);
> + if ( addr == NULL )
> + return;
>
> + spin_lock(&gicv3.lock);
> + writeb_relaxed(priority, addr);
> spin_unlock(&gicv3.lock);
> }
>
> @@ -1273,6 +1315,10 @@ static void gicv3_irq_set_affinity(struct irq_desc *desc, const cpumask_t *mask)
> {
> unsigned int cpu;
> uint64_t affinity;
> + void __iomem *addr = get_addr_by_offset(desc, GICD_IROUTER);
> +
> + if ( addr == NULL )
> + return;
>
> ASSERT(!cpumask_empty(mask));
>
> @@ -1284,7 +1330,7 @@ static void gicv3_irq_set_affinity(struct irq_desc *desc, const cpumask_t *mask)
> affinity &= ~GICD_IROUTER_SPI_MODE_ANY;
>
> if ( desc->irq >= NR_GIC_LOCAL_IRQS )
> - writeq_relaxed_non_atomic(affinity, (GICD + GICD_IROUTER + desc->irq * 8));
> + writeq_relaxed_non_atomic(affinity, addr);
>
> spin_unlock(&gicv3.lock);
> }
> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
> index fce7e42a33..5bc6475eb4 100644
> --- a/xen/arch/arm/include/asm/irq.h
> +++ b/xen/arch/arm/include/asm/irq.h
> @@ -29,6 +29,7 @@ struct arch_irq_desc {
> */
> #define NR_IRQS 1024
>
> +#define SPI_MAX_INTID 1019
> #define LPI_OFFSET 8192
>
> /* LPIs are always numbered starting at 8192, so 0 is a good invalid case. */
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-08-29 19:45 ` Volodymyr Babchuk
@ 2025-08-31 14:08 ` Oleksandr Tyshchenko
2025-09-01 14:42 ` Leonid Komarianskyi
0 siblings, 1 reply; 49+ messages in thread
From: Oleksandr Tyshchenko @ 2025-08-31 14:08 UTC (permalink / raw)
To: Leonid Komarianskyi
Cc: Volodymyr Babchuk, xen-devel@lists.xenproject.org,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
On 29.08.25 22:45, Volodymyr Babchuk wrote:
>
> Hi Leonid,
Hello Leonid
>
> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>
>> Currently, Xen does not support eSPI interrupts, leading
>> to a data abort when such interrupts are defined in the DTS.
>>
>> This patch introduces a separate array to initialize up to
>> 1024 interrupt descriptors in the eSPI range and adds the
>> necessary defines and helper function. These changes lay the
>> groundwork for future implementation of full eSPI interrupt
>> support. As this GICv3.1 feature is not required by all vendors,
>> all changes are guarded by ifdefs, depending on the corresponding
>> Kconfig option.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>
>> ---
>> Changes in V5:
>> - no functional changes introduced by this version compared with V4, only
>> minor fixes and removal of ifdefs for macroses
>> - added TODO comment, suggested by Oleksandr Tyshchenko
>> - changed int to unsigned int for irqs
>> - removed ifdefs for eSPI-specific defines and macros to reduce the
>> number of ifdefs and code duplication in further changes
>> - removed reviewed-by as moving defines from ifdefs requires additional
>> confirmation from reviewers
Reviewed-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
with the following addressed ...
>>
>> Changes in V4:
>> - removed redundant line with 'default n' in Kconfig, as it is disabled
>> by default, without explicit specification
>> - added reviewed-by from Volodymyr Babchuk
>>
>> Changes in V3:
>> - introduced a new define NR_ESPI_IRQS to avoid confusion, like in the
>> case of using NR_IRQS for espi_desc array
>> - implemented helper functions espi_to_desc and init_espi_data to make
>> it possible to add stubs with the same name, and as a result, reduce
>> the number of #ifdefs
>> - disable CONFIG_GICV3_ESPI default value to n
>>
>> Changes in V2:
>> - use (ESPI_MAX_INTID + 1) instead of (ESPI_BASE_INTID + NR_IRQS)
>> - remove unnecessary comment for nr_irqs initialization
>> ---
>> xen/arch/arm/Kconfig | 8 +++++
>> xen/arch/arm/include/asm/irq.h | 24 +++++++++++++++
>> xen/arch/arm/irq.c | 56 +++++++++++++++++++++++++++++++++-
>> 3 files changed, 87 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 17df147b25..43b05533b1 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -135,6 +135,14 @@ config GICV3
>> Driver for the ARM Generic Interrupt Controller v3.
>> If unsure, use the default setting.
>>
>> +config GICV3_ESPI
>> + bool "Extended SPI range support"
>> + depends on GICV3 && !NEW_VGIC
>> + help
>> + Allow Xen and domains to use interrupt numbers from the extended SPI
>> + range, from 4096 to 5119. This feature is introduced in GICv3.1
>> + architecture.
>> +
>> config HAS_ITS
>> bool "GICv3 ITS MSI controller support (UNSUPPORTED)" if UNSUPPORTED
>> depends on GICV3 && !NEW_VGIC && !ARM_32
>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
>> index 5bc6475eb4..4443799648 100644
>> --- a/xen/arch/arm/include/asm/irq.h
>> +++ b/xen/arch/arm/include/asm/irq.h
>> @@ -32,6 +32,13 @@ struct arch_irq_desc {
>> #define SPI_MAX_INTID 1019
>> #define LPI_OFFSET 8192
>>
>> +#define ESPI_BASE_INTID 4096
>> +#define ESPI_MAX_INTID 5119
>> +#define NR_ESPI_IRQS 1024
>> +
>> +#define ESPI_INTID2IDX(intid) ((intid) - ESPI_BASE_INTID)
>> +#define ESPI_IDX2INTID(idx) ((idx) + ESPI_BASE_INTID)
>> +
>> /* LPIs are always numbered starting at 8192, so 0 is a good invalid case. */
>> #define INVALID_LPI 0
>>
>> @@ -39,7 +46,15 @@ struct arch_irq_desc {
>> #define INVALID_IRQ 1023
>>
>> extern const unsigned int nr_irqs;
>> +#ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * This will also cover the eSPI range, as some critical devices
>> + * for booting Xen (e.g., serial) may use this type of interrupts.
>> + */
>> +#define nr_static_irqs (ESPI_MAX_INTID + 1)
>> +#else
>> #define nr_static_irqs NR_IRQS
>> +#endif
>>
>> struct irq_desc;
>> struct irqaction;
>> @@ -55,6 +70,15 @@ static inline bool is_lpi(unsigned int irq)
>> return irq >= LPI_OFFSET;
>> }
>>
>> +static inline bool is_espi(unsigned int irq)
>> +{
>> +#ifdef CONFIG_GICV3_ESPI
>
> Taking into account that with CONFIG_GICV3_ESPI=n we should never have
> "irq" in eSPI range, do you really need this #ifdef? I think that
> ASSERT_UNREACHABLE in espi_to_desc() is sufficient guard.
>
> Also, IRQ line number belongs to eSPI range regardless of CONFIG_GICV3_ESPI,
> value, so in my opinion is_espi() should always return correct value for
> a given "irq".
... I agree with Volodymyr's suggestion for is_espi() to always return
correct value for a given "irq".
>
>> + return (irq >= ESPI_BASE_INTID && irq <= ESPI_MAX_INTID);
>
> Also, you don't need parentheses here.
>
>> +#else
>> + return false;
>> +#endif
>> +}
>> +
>> #define domain_pirq_to_irq(d, pirq) (pirq)
>>
>> bool is_assignable_irq(unsigned int irq);
>> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
>> index b8eccfc924..61c915c3f9 100644
>> --- a/xen/arch/arm/irq.c
>> +++ b/xen/arch/arm/irq.c
>> @@ -19,7 +19,11 @@
>> #include <asm/gic.h>
>> #include <asm/vgic.h>
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +const unsigned int nr_irqs = ESPI_MAX_INTID + 1;
>> +#else
>> const unsigned int nr_irqs = NR_IRQS;
>> +#endif
>>
>> static unsigned int local_irqs_type[NR_LOCAL_IRQS];
>> static DEFINE_SPINLOCK(local_irqs_type_lock);
>> @@ -46,6 +50,53 @@ void irq_end_none(struct irq_desc *irq)
>> }
>>
>> static irq_desc_t irq_desc[NR_IRQS - NR_LOCAL_IRQS];
>> +#ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * TODO: Consider allocating an array dynamically if
>> + * there is a need to enable GICV3_ESPI by default.
>> + */
>> +static irq_desc_t espi_desc[NR_ESPI_IRQS];
>> +
>> +static struct irq_desc *espi_to_desc(unsigned int irq)
>> +{
>> + return &espi_desc[ESPI_INTID2IDX(irq)];
>> +}
>> +
>> +static int __init init_espi_data(void)
>> +{
>> + unsigned int irq;
>> +
>> + for ( irq = ESPI_BASE_INTID; irq <= ESPI_MAX_INTID; irq++ )
>> + {
>> + struct irq_desc *desc = irq_to_desc(irq);
>> + int rc = init_one_irq_desc(desc);
>> +
>> + if ( rc )
>> + return rc;
>> +
>> + desc->irq = irq;
>> + desc->action = NULL;
>> + }
>> +
>> + return 0;
>> +}
>> +#else
>> +/*
>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>> + * because in this case, is_espi will always return false.
This comment should also be updated.
>> + */
>> +static struct irq_desc *espi_to_desc(unsigned int irq)
>> +{
>> + ASSERT_UNREACHABLE();
>> + return NULL;
>> +}
>> +
>> +static int __init init_espi_data(void)
>> +{
>> + return 0;
>> +}
>> +#endif
>> +
>> static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
>>
>> struct irq_desc *__irq_to_desc(unsigned int irq)
>> @@ -53,6 +104,9 @@ struct irq_desc *__irq_to_desc(unsigned int irq)
>> if ( irq < NR_LOCAL_IRQS )
>> return &this_cpu(local_irq_desc)[irq];
>>
>> + if ( is_espi(irq) )
>> + return espi_to_desc(irq);
>> +
>> return &irq_desc[irq-NR_LOCAL_IRQS];
>> }
>>
>> @@ -79,7 +133,7 @@ static int __init init_irq_data(void)
>> desc->action = NULL;
>> }
>>
>> - return 0;
>> + return init_espi_data();
>> }
>>
>> static int init_local_irq_data(unsigned int cpu)
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI
2025-08-29 16:06 ` [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI Leonid Komarianskyi
2025-08-29 19:55 ` Volodymyr Babchuk
@ 2025-08-31 14:56 ` Oleksandr Tyshchenko
1 sibling, 0 replies; 49+ messages in thread
From: Oleksandr Tyshchenko @ 2025-08-31 14:56 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel,
Volodymyr Babchuk
On 29.08.25 19:06, Leonid Komarianskyi wrote:
Hello Leonid
> Introduced appropriate register definitions, helper macros,
> and initialization of required GICv3.1 distributor registers
> to support eSPI. This type of interrupt is handled in the
> same way as regular SPI interrupts, with the following
> differences:
>
> 1) eSPIs can have up to 1024 interrupts, starting from the
> beginning of the range, whereas regular SPIs use INTIDs from
> 32 to 1019, totaling 988 interrupts;
> 2) eSPIs start at INTID 4096, necessitating additional interrupt
> index conversion during register operations.
>
> In case if appropriate config is disabled, or GIC HW doesn't
> support eSPI, the existing functionality will remain the same.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> ---
> Changes in V5:
> - fixed minor nits, no functional changes: changed u32 to uint32_t and
> added a comment noting that the configuration for eSPIs is the same as
> for regular SPIs
> - removed ifdefs for eSPI-specific offsets to reduce the number of
> ifdefs and code duplication in further changes
> - removed reviewed-by as moving offset from ifdefs requires additional
> confirmation from reviewers
Reviewed-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>
> Changes in V4:
> - added offsets for GICD_IGRPMODRnE and GICD_NSACRnE that are required
> for vGIC emulation
> - added a log banner with eSPI information, similar to the one for
> regular SPI
> - added newline after ifdef and before gic_is_valid_line
> - added reviewed-by from Volodymyr Babchuk
>
> Changes in V3:
> - add __init attribute to gicv3_dist_espi_common_init
> - change open-codded eSPI register initialization to the appropriate
> gen-mask macro
> - fixed formatting for lines with more than 80 symbols
> - introduced gicv3_dist_espi_init_aff to be able to use stubs in case of
> CONFIG_GICV3_ESPI disabled
> - renamed parameter in the GICD_TYPER_ESPI_RANGE macro to espi_range
> (name was taken from GIC specification) to avoid confusion
> - changed type for i variable to unsigned int since it cannot be
> negative
>
> Changes in V2:
> - move gic_number_espis function from
> [PATCH 08/10] xen/arm: vgic: add resource management for extended SPIs
> to use it in the newly introduced gic_is_valid_espi
> - add gic_is_valid_espi which checks if IRQ number is in supported
> by HW eSPI range
> - update gic_is_valid_irq conditions to allow operations with eSPIs
> ---
> xen/arch/arm/gic-v3.c | 83 ++++++++++++++++++++++++++
> xen/arch/arm/include/asm/gic.h | 22 +++++++
> xen/arch/arm/include/asm/gic_v3_defs.h | 38 ++++++++++++
> 3 files changed, 143 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 29b7f68cba..4a7ce12f26 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -485,6 +485,36 @@ static void __iomem *get_addr_by_offset(struct irq_desc *irqd, u32 offset)
> default:
> break;
> }
> +#ifdef CONFIG_GICV3_ESPI
> + case ESPI_BASE_INTID ... ESPI_MAX_INTID:
> + {
> + uint32_t irq_index = ESPI_INTID2IDX(irqd->irq);
> +
> + switch ( offset )
> + {
> + case GICD_ISENABLER:
> + return (GICD + GICD_ISENABLERnE + (irq_index / 32) * 4);
> + case GICD_ICENABLER:
> + return (GICD + GICD_ICENABLERnE + (irq_index / 32) * 4);
> + case GICD_ISPENDR:
> + return (GICD + GICD_ISPENDRnE + (irq_index / 32) * 4);
> + case GICD_ICPENDR:
> + return (GICD + GICD_ICPENDRnE + (irq_index / 32) * 4);
> + case GICD_ISACTIVER:
> + return (GICD + GICD_ISACTIVERnE + (irq_index / 32) * 4);
> + case GICD_ICACTIVER:
> + return (GICD + GICD_ICACTIVERnE + (irq_index / 32) * 4);
> + case GICD_ICFGR:
> + return (GICD + GICD_ICFGRnE + (irq_index / 16) * 4);
> + case GICD_IROUTER:
> + return (GICD + GICD_IROUTERnE + irq_index * 8);
> + case GICD_IPRIORITYR:
> + return (GICD + GICD_IPRIORITYRnE + irq_index);
> + default:
> + break;
> + }
> + }
> +#endif
> default:
> break;
> }
> @@ -655,6 +685,55 @@ static void gicv3_set_irq_priority(struct irq_desc *desc,
> spin_unlock(&gicv3.lock);
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> +unsigned int gic_number_espis(void)
> +{
> + return gic_hw_ops->info->nr_espi;
> +}
> +
> +static void __init gicv3_dist_espi_common_init(uint32_t type)
> +{
> + unsigned int espi_nr, i;
> +
> + espi_nr = min(1024U, GICD_TYPER_ESPIS_NUM(type));
> + gicv3_info.nr_espi = espi_nr;
> + /* The GIC HW doesn't support eSPI, so we can leave from here */
> + if ( gicv3_info.nr_espi == 0 )
> + return;
> +
> + printk("GICv3: %d eSPI lines\n", gicv3_info.nr_espi);
> +
> + /* The configuration for eSPIs is similar to that for regular SPIs */
> + for ( i = 0; i < espi_nr; i += 16 )
> + writel_relaxed(0, GICD + GICD_ICFGRnE + (i / 16) * 4);
> +
> + for ( i = 0; i < espi_nr; i += 4 )
> + writel_relaxed(GIC_PRI_IRQ_ALL,
> + GICD + GICD_IPRIORITYRnE + (i / 4) * 4);
> +
> + for ( i = 0; i < espi_nr; i += 32 )
> + {
> + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICENABLERnE + (i / 32) * 4);
> + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICACTIVERnE + (i / 32) * 4);
> + }
> +
> + for ( i = 0; i < espi_nr; i += 32 )
> + writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPRnE + (i / 32) * 4);
> +}
> +
> +static void __init gicv3_dist_espi_init_aff(uint64_t affinity)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < gicv3_info.nr_espi; i++ )
> + writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTERnE + i * 8);
> +}
> +#else
> +static void __init gicv3_dist_espi_common_init(uint32_t type) { }
> +
> +static void __init gicv3_dist_espi_init_aff(uint64_t affinity) { }
> +#endif
> +
> static void __init gicv3_dist_init(void)
> {
> uint32_t type;
> @@ -700,6 +779,8 @@ static void __init gicv3_dist_init(void)
> for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i += 32 )
> writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPR + (i / 32) * 4);
>
> + gicv3_dist_espi_common_init(type);
> +
> gicv3_dist_wait_for_rwp();
>
> /* Turn on the distributor */
> @@ -713,6 +794,8 @@ static void __init gicv3_dist_init(void)
>
> for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i++ )
> writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTER + i * 8);
> +
> + gicv3_dist_espi_init_aff(affinity);
> }
>
> static int gicv3_enable_redist(void)
> diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
> index 3fcee42675..1e747dcd99 100644
> --- a/xen/arch/arm/include/asm/gic.h
> +++ b/xen/arch/arm/include/asm/gic.h
> @@ -306,8 +306,26 @@ extern void gic_dump_vgic_info(struct vcpu *v);
>
> /* Number of interrupt lines */
> extern unsigned int gic_number_lines(void);
> +#ifdef CONFIG_GICV3_ESPI
> +extern unsigned int gic_number_espis(void);
> +
> +static inline bool gic_is_valid_espi(unsigned int irq)
> +{
> + return (irq >= ESPI_BASE_INTID &&
> + irq < ESPI_IDX2INTID(gic_number_espis()));
> +}
> +#else
> +static inline bool gic_is_valid_espi(unsigned int irq)
> +{
> + return false;
> +}
> +#endif
> +
> static inline bool gic_is_valid_line(unsigned int irq)
> {
> + if ( gic_is_valid_espi(irq) )
> + return true;
> +
> return irq < gic_number_lines();
> }
>
> @@ -325,6 +343,10 @@ struct gic_info {
> enum gic_version hw_version;
> /* Number of GIC lines supported */
> unsigned int nr_lines;
> +#ifdef CONFIG_GICV3_ESPI
> + /* Number of GIC eSPI supported */
> + unsigned int nr_espi;
> +#endif
> /* Number of LR registers */
> uint8_t nr_lrs;
> /* Maintenance irq number */
> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
> index 2af093e774..3370b4cd52 100644
> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
> @@ -37,6 +37,44 @@
> #define GICD_IROUTER1019 (0x7FD8)
> #define GICD_PIDR2 (0xFFE8)
>
> +/* Additional registers for GICv3.1 */
> +#define GICD_IGROUPRnE (0x1000)
> +#define GICD_IGROUPRnEN (0x107C)
> +#define GICD_ISENABLERnE (0x1200)
> +#define GICD_ISENABLERnEN (0x127C)
> +#define GICD_ICENABLERnE (0x1400)
> +#define GICD_ICENABLERnEN (0x147C)
> +#define GICD_ISPENDRnE (0x1600)
> +#define GICD_ISPENDRnEN (0x167C)
> +#define GICD_ICPENDRnE (0x1800)
> +#define GICD_ICPENDRnEN (0x187C)
> +#define GICD_ISACTIVERnE (0x1A00)
> +#define GICD_ISACTIVERnEN (0x1A7C)
> +#define GICD_ICACTIVERnE (0x1C00)
> +#define GICD_ICACTIVERnEN (0x1C7C)
> +#define GICD_IPRIORITYRnE (0x2000)
> +#define GICD_IPRIORITYRnEN (0x23FC)
> +#define GICD_ICFGRnE (0x3000)
> +#define GICD_ICFGRnEN (0x30FC)
> +#define GICD_IGRPMODRnE (0x3400)
> +#define GICD_IGRPMODRnEN (0x347C)
> +#define GICD_NSACRnE (0x3600)
> +#define GICD_NSACRnEN (0x36FC)
> +#define GICD_IROUTERnE (0x8000)
> +#define GICD_IROUTERnEN (0x9FFC)
> +
> +#ifdef CONFIG_GICV3_ESPI
> +#define GICD_TYPER_ESPI_SHIFT 8
> +#define GICD_TYPER_ESPI_RANGE_SHIFT 27
> +#define GICD_TYPER_ESPI_RANGE_MASK (0x1F)
> +#define GICD_TYPER_ESPI (1U << GICD_TYPER_ESPI_SHIFT)
> +#define GICD_TYPER_ESPI_RANGE(espi_range) ((((espi_range) & \
> + GICD_TYPER_ESPI_RANGE_MASK) + 1) * 32)
> +#define GICD_TYPER_ESPIS_NUM(typer) \
> + (((typer) & GICD_TYPER_ESPI) ? \
> + GICD_TYPER_ESPI_RANGE((typer) >> GICD_TYPER_ESPI_RANGE_SHIFT) : 0)
> +#endif
> +
> /* Common between GICD_PIDR2 and GICR_PIDR2 */
> #define GIC_PIDR2_ARCH_MASK (0xf0)
> #define GIC_PIDR2_ARCH_GICv3 (0x30)
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-08-29 20:45 ` Volodymyr Babchuk
@ 2025-08-31 15:58 ` Oleksandr Tyshchenko
2025-09-01 18:00 ` Leonid Komarianskyi
2025-09-01 17:38 ` Leonid Komarianskyi
1 sibling, 1 reply; 49+ messages in thread
From: Oleksandr Tyshchenko @ 2025-08-31 15:58 UTC (permalink / raw)
To: Leonid Komarianskyi, Volodymyr Babchuk
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel
On 29.08.25 23:45, Volodymyr Babchuk wrote:
Hello Leonid, Volodymyr
>
> Hi Leonid,
>
> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>
>> This change introduces resource management in the VGIC to handle
>> extended SPIs introduced in GICv3.1. The pending_irqs and
>> allocated_irqs arrays are resized to support the required
>> number of eSPIs, based on what is supported by the hardware and
>> requested by the guest. A new field, ext_shared_irqs, is added
>> to the VGIC structure to store information about eSPIs, similar
>> to how shared_irqs is used for regular SPIs.
>>
>> Since the eSPI range starts at INTID 4096 and INTIDs between 1025
>> and 4095 are reserved, helper macros are introduced to simplify the
>> transformation of indices and to enable easier access to eSPI-specific
>> resources. These changes prepare the VGIC for processing eSPIs as
>> required by future functionality.
>>
>> The initialization and deinitialization paths for vgic have been updated
>> to allocate and free these resources appropriately. Additionally,
>> updated handling of INTIDs greater than 1024, passed from the toolstack
>> during domain creation, and verification logic ensures only valid SPI or
>> eSPI INTIDs are used.
>>
>> The existing SPI behavior remains unaffected when guests do not request
>> eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
>> option is disabled.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>
>> ---
>> Changes in V5:
>> - removed the has_espi field because it can be determined by checking
>> whether domain->arch.vgic.nr_espis is zero or not
>> - since vgic_ext_rank_offset is not used in this patch, it has been
>> moved to the appropriate patch in the patch series, which implements
>> vgic eSPI registers emulation and requires this function
>> - removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
>> and code duplication in further changes
>> - fixed minor nit: used %pd for printing domain with its ID
@Leonid, thanks for optimizing the series, now it looks much better (the
number of #ifdef-s is reduced, code is reused).
>>
>> Changes in V4:
>> - added has_espi field to simplify determining whether a domain is able
>> to operate with eSPI
>> - fixed formatting issues and misspellings
>>
>> Changes in V3:
>> - fixed formatting for lines with more than 80 symbols
>> - introduced helper functions to be able to use stubs in case of
>> CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
>> #ifdefs
>> - fixed checks for nr_spis in domain_vgic_init
>> - updated comment about nr_spis adjustments with dom0less mention
>> - moved comment with additional explanations before checks
>> - used unsigned int for indexes since they cannot be negative
>> - removed unnecessary parentheses
>> - move vgic_ext_rank_offset to the below ifdef guard, to reduce the
>> number of ifdefs
>>
>> Changes in V2:
>> - change is_espi_rank to is_valid_espi_rank to verify whether the array
>> element ext_shared_irqs exists. The previous version, is_espi_rank,
>> only checked if the rank index was less than the maximum possible eSPI
>> rank index, but this could potentially result in accessing a
>> non-existing array element. To address this, is_valid_espi_rank was
>> introduced, which ensures that the required eSPI rank exists
>> - move gic_number_espis to
>> xen/arm: gicv3: implement handling of GICv3.1 eSPI
>> - update vgic_is_valid_irq checks to allow operating with eSPIs
>> - remove redundant newline in vgic_allocate_virq
>> ---
>> xen/arch/arm/include/asm/vgic.h | 12 ++
>> xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
>> 2 files changed, 208 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
>> index 3e7cbbb196..912d5b7694 100644
>> --- a/xen/arch/arm/include/asm/vgic.h
>> +++ b/xen/arch/arm/include/asm/vgic.h
>> @@ -146,6 +146,10 @@ struct vgic_dist {
>> int nr_spis; /* Number of SPIs */
>> unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
>> struct vgic_irq_rank *shared_irqs;
>> +#ifdef CONFIG_GICV3_ESPI
>> + struct vgic_irq_rank *ext_shared_irqs;
>> + int nr_espis; /* Number of extended SPIs */
It seems you have agreed (V4) that nr_espis could not be negative.
>> +#endif
>> /*
>> * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
>> * struct arch_vcpu.
>> @@ -243,6 +247,14 @@ struct vgic_ops {
>> /* Number of ranks of interrupt registers for a domain */
>> #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
>> +#endif
>> +#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
>> +#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
>> +#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
>> +#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
>> +
>> #define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
>> #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
>>
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 2bbf4d99aa..c9b9528c66 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -27,9 +27,82 @@
>>
>> bool vgic_is_valid_line(struct domain *d, unsigned int virq)
>> {
>> +#ifdef CONFIG_GICV3_ESPI
>> + if ( virq >= ESPI_BASE_INTID &&
>> + virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
>> + return true;
>> +#endif
>> +
>> return virq < vgic_num_irqs(d);
>> }
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * Since eSPI indexes start from 4096 and numbers from 1024 to
>> + * 4095 are forbidden, we need to check both lower and upper
>> + * limits for ranks.
>> + */
>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
>> +{
>> + return rank >= EXT_RANK_MIN &&
>> + EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
>> +}
>> +
>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>> + unsigned int rank)
>> +{
>> + return &v->domain->arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
>> +}
>> +
>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
>> +{
>> + return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
>> + d->arch.vgic.allocated_irqs);
>> +}
>> +
>> +static void arch_move_espis(struct vcpu *v)
>
> I don't need you need a copy of arch_move_irqs(). Se below for more info.
>
>> +{
>> + const cpumask_t *cpu_mask = cpumask_of(v->processor);
>> + struct domain *d = v->domain;
>> + struct pending_irq *p;
>> + struct vcpu *v_target;
>> + unsigned int i;
>> +
>> + for ( i = ESPI_BASE_INTID;
>> + i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
>> + {
>> + v_target = vgic_get_target_vcpu(v, i);
>> + p = irq_to_pending(v_target, i);
>> +
>> + if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
>> + irq_set_affinity(p->desc, cpu_mask);
>> + }
>> +}
>> +#else
>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
>> +{
>> + return false;
>> +}
>> +
>> +/*
>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>> + * because in this case, is_valid_espi_rank will always return false.
>> + */
>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>> + unsigned int rank)
>> +{
>> + ASSERT_UNREACHABLE();
>> + return NULL;
>> +}
>> +
>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
>> +{
>> + return false;
>> +}
>> +
>> +static void arch_move_espis(struct vcpu *v) { }
>> +#endif
>> +
>> static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
>> unsigned int rank)
>> {
>> @@ -37,6 +110,8 @@ static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
>> return v->arch.vgic.private_irqs;
>> else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
>> return &v->domain->arch.vgic.shared_irqs[rank - 1];
>> + else if ( is_valid_espi_rank(v->domain, rank) )
>> + return vgic_get_espi_rank(v, rank);
>> else
>> return NULL;
>> }
>> @@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
>> return 0;
>> }
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>> +{
>> + return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
>> +}
>> +
>> +static int init_vgic_espi(struct domain *d)
>> +{
>> + unsigned int i, idx;
>> +
>> + if ( d->arch.vgic.nr_espis == 0 )
>> + return 0;
>> +
>> + d->arch.vgic.ext_shared_irqs =
>> + xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
>> + if ( d->arch.vgic.ext_shared_irqs == NULL )
>> + return -ENOMEM;
>> +
>> + for ( i = d->arch.vgic.nr_spis, idx = 0;
>> + i < vgic_num_spi_lines(d); i++, idx++ )
>> + vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
>> + ESPI_IDX2INTID(idx));
>> +
>> + for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
>> + vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
>> +
>> + return 0;
>> +}
>> +
>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>
> I know that I should made this observation in previous version, but I
> didn't, sorry for that. Anyways, I don't think that this is a good idea
> to introduce this function and vgic_reserve_espi_virq(), as well as
> arch_move_espis(), actually, because in each case this is a code
> duplication, which is not good.
>
> I think that instead you need to introduce a pair of helpers that will
> map any (e)SPI number to pending_irq[]/allocate_irqs index and back.
>
> somethink like
>
> static inline unsigned virq_to_index(int virq)
> {
> if (is_espi(virq))
> return ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
> return virq;
> }
>
> See below for examples.
>
>> +{
>> + irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
>> + return &d->arch.vgic.pending_irqs[irq];
>> +}
>> +#else
>> +static unsigned int init_vgic_espi(struct domain *d)
>> +{
>> + return 0;
>> +}
>> +
>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>> +{
>> + return d->arch.vgic.nr_spis;
>> +}
>> +
>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>> +{
>> + return NULL;
>> +}
>> +#endif
>> +
>> +static unsigned int vgic_num_alloc_irqs(struct domain *d)
>> +{
>> + return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
>> +}
I do not know where it would be better to put a comment related to
non-visible in the patch context route_irq_to_guest(), but put it here.
I am afraid, the vgic_num_irqs(d) printed in the following error message
is not entirely correct with your changes:
route_irq_to_guest():
...
if ( !vgic_is_valid_line(d, virq) )
{
printk(XENLOG_G_ERR
"the vIRQ number %u is too high for domain %u (max = %u)\n",
irq, d->domain_id, vgic_num_irqs(d));
return -EINVAL;
}
>> +
>> int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> {
>> int i;
>> @@ -131,6 +262,36 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> */
>> nr_spis = ROUNDUP(nr_spis, 32);
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> + /*
>> + * During domain creation, the dom0less DomUs code or toolstack specifies
>> + * the maximum INTID, which is defined in the domain config subtracted by
>> + * 32 to cover the local IRQs (please see the comment to VGIC_DEF_NR_SPIS).
>> + * To compute the actual number of eSPI that will be usable for,
>> + * add back 32.
>> + */
>> + if ( nr_spis + 32 > ESPI_IDX2INTID(NR_ESPI_IRQS) )
>> + return -EINVAL;
>> +
>> + if ( nr_spis + 32 >= ESPI_BASE_INTID )
>> + {
>> + d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32, 1024U);
>> + /* Verify if GIC HW can handle provided INTID */
>> + if ( d->arch.vgic.nr_espis > gic_number_espis() )
>> + return -EINVAL;
>> + /*
>> + * Set the maximum available number for regular
>> + * SPI to pass the next check
>> + */
>> + nr_spis = VGIC_DEF_NR_SPIS;
>> + }
>> + else
>> + {
>> + /* Domain will use the regular SPI range */
>> + d->arch.vgic.nr_espis = 0;
>> + }
>> +#endif
>> +
>> /* Limit the number of virtual SPIs supported to (1020 - 32) = 988 */
>> if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>> return -EINVAL;
>> @@ -145,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> return -ENOMEM;
>>
>> d->arch.vgic.pending_irqs =
>> - xzalloc_array(struct pending_irq, d->arch.vgic.nr_spis);
>> + xzalloc_array(struct pending_irq, vgic_num_spi_lines(d));
>> if ( d->arch.vgic.pending_irqs == NULL )
>> return -ENOMEM;
>>
>> @@ -156,12 +317,16 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
>> vgic_rank_init(&d->arch.vgic.shared_irqs[i], i + 1, 0);
>>
>> + ret = init_vgic_espi(d);
>> + if ( ret )
>> + return ret;
>> +
>> ret = d->arch.vgic.handler->domain_init(d);
>> if ( ret )
>> return ret;
>>
>> d->arch.vgic.allocated_irqs =
>> - xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
>> + xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_alloc_irqs(d)));
>> if ( !d->arch.vgic.allocated_irqs )
>> return -ENOMEM;
>>
>> @@ -195,9 +360,27 @@ void domain_vgic_free(struct domain *d)
>> }
>> }
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> + for ( i = 0; i < d->arch.vgic.nr_espis; i++ )
>> + {
>> + struct pending_irq *p = spi_to_pending(d, ESPI_IDX2INTID(i));
>> +
>> + if ( p->desc )
>> + {
>> + ret = release_guest_irq(d, p->irq);
>> + if ( ret )
>> + dprintk(XENLOG_G_WARNING, "%pd: Failed to release virq %u ret = %d\n",
>> + d, p->irq, ret);
>> + }
>> + }
>> +#endif
>> +
>> if ( d->arch.vgic.handler )
>> d->arch.vgic.handler->domain_free(d);
>> xfree(d->arch.vgic.shared_irqs);
>> +#ifdef CONFIG_GICV3_ESPI
>> + xfree(d->arch.vgic.ext_shared_irqs);
>> +#endif
>> xfree(d->arch.vgic.pending_irqs);
>> xfree(d->arch.vgic.allocated_irqs);
>> }
>> @@ -331,6 +514,8 @@ void arch_move_irqs(struct vcpu *v)
>> if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
>> irq_set_affinity(p->desc, cpu_mask);
>> }
>> +
>> + arch_move_espis(v);
>> }
>>
>> void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
>> @@ -538,6 +723,8 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
>> n = &v->arch.vgic.pending_irqs[irq];
>> else if ( is_lpi(irq) )
>> n = v->domain->arch.vgic.handler->lpi_to_pending(v->domain, irq);
>> + else if ( is_espi(irq) )
>> + n = espi_to_pending(v->domain, irq);
>> else
>> n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>> return n;
>> @@ -547,6 +734,9 @@ struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq)
>> {
>> ASSERT(irq >= NR_LOCAL_IRQS);
>>
>> + if ( is_espi(irq) )
>> + return espi_to_pending(d, irq);
>> +
>
> here you can just do
>
> idx = virq_to_idx(virq);
>
>> return &d->arch.vgic.pending_irqs[irq - 32];
>
> and
>
> return &d->arch.vgic.pending_irqs[idx];
>
> instead
>
>> }
>>
>> @@ -668,6 +858,9 @@ bool vgic_reserve_virq(struct domain *d, unsigned int virq)
>> if ( !vgic_is_valid_line(d, virq) )
>> return false;
>>
>> + if ( is_espi(virq) )
>> + return vgic_reserve_espi_virq(d, virq);
>> +
>
> here you can just do
>
> idx = virq_to_idx(virq)
>
>> return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
>
> and then just
>
> return !test_and_set_bit(idx, d->arch.vgic.allocated_irqs);
>
>
>> }
>>
>> @@ -685,7 +878,7 @@ int vgic_allocate_virq(struct domain *d, bool spi)
>> else
>> {
>> first = 32;
>> - end = vgic_num_irqs(d);
>> + end = vgic_num_alloc_irqs(d);
>> }
>
> I thinj you need to recalculate "virq" value at the end of this
> function. You'll return index in bitfield, but this is not the same is
> IRQ number in case of eSPIs.
+1
The helpers I mentioned before can help here.
>
>>
>> /*
>
> Lastly, I think that it is very wasteful to allocate pending_irqs as
> continuous array, because it will consist mostly of unused entries,
> especially with eSPIs enable. Probably, better approach will be to use radix
> tree. As a bonus, you can use IRQ line number as a key, and get rid of
> all these is_espi() checks and mappings between IRQ number and index in
> the array. But this is a much more drastic change, and I don't think that it
> should be done in this patch series...
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-08-29 16:06 ` [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers Leonid Komarianskyi
2025-08-29 20:12 ` Volodymyr Babchuk
@ 2025-09-01 12:38 ` Oleksandr Tyshchenko
2025-09-01 18:27 ` Leonid Komarianskyi
2025-09-02 16:55 ` Julien Grall
2 siblings, 1 reply; 49+ messages in thread
From: Oleksandr Tyshchenko @ 2025-09-01 12:38 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel,
Volodymyr Babchuk
On 29.08.25 19:06, Leonid Komarianskyi wrote:
Hello Leonid
> Implemented support for GICv3.1 extended SPI registers for vGICv3,
> allowing the emulation of eSPI-specific behavior for guest domains.
> The implementation includes read and write emulation for eSPI-related
> registers (e.g., GICD_ISENABLERnE, GICD_IROUTERnE, and others),
> following a similar approach to the handling of regular SPIs.
>
> The eSPI registers, previously located in reserved address ranges,
> are now adjusted to support MMIO read and write operations correctly
> when CONFIG_GICV3_ESPI is enabled.
>
> The availability of eSPIs and the number of emulated extended SPIs
> for guest domains is reported by setting the appropriate bits in the
> GICD_TYPER register, based on the number of eSPIs requested by the
> domain and supported by the hardware. In cases where the configuration
> option is disabled, the hardware does not support eSPIs, or the domain
> does not request such interrupts, the functionality remains unchanged.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> ---
> Changes in V5:
> - since eSPI-specific defines and macros are now available even when the
> appropriate config is disabled, this allows us to remove many
> #ifdef guards and reuse the existing code for regular SPIs for eSPIs as
> well, as eSPIs are processed similarly. This improves code readability
> and consolidates the register ranges for SPIs and eSPIs in a single
> place, simplifying future changes or fixes for SPIs and their
> counterparts from the extended range
> - moved vgic_ext_rank_offset() from
> [08/12] xen/arm: vgic: add resource management for extended SPIs
> as the function was unused before this patch
> - added stub implementation of vgic_ext_rank_offset() when CONFIG_GICV3_ESPI=n
> - removed unnecessary defines for reserved ranges, which were introduced in
> V4 to reduce the number of #ifdefs. The issue is now resolved by
> allowing the use of SPI-specific offsets and reworking the logic
Looks very good now, thanks. Just one NIT and one question below ...
>
> Changes in V4:
> - added missing RAZ and write ignore eSPI-specific registers ranges:
> GICD_NSACRnE and GICD_IGRPMODRnE
> - changed previously reserved range to cover GICD_NSACRnE and
> GICD_IGRPMODRnE
> - introduced GICD_RESERVED_RANGE<n>_START/END defines to remove
> hardcoded values and reduce the number of ifdefs
> - fixed reserved ranges with eSPI option enabled: added missing range
> 0x0F30-0x0F7C
> - updated the logic for domains that do not support eSPI, but Xen is
> compiled with the eSPI option. Now, prior to other MMIO checks, we
> verify whether eSPI is available for the domain or not. If not, it
> behaves as it does currently - RAZ and WI
> - fixed print for GICD_ICACTIVERnE
> - fixed new lines formatting for switch-case
>
> Changes in V3:
> - changed vgic_store_irouter parameters - instead of offset virq is
> used, to remove the additional bool espi parameter and simplify
> checks. Also, adjusted parameters for regular SPI. Since the offset
> parameter was used only for calculating virq number and then reused for
> finding rank offset, it will not affect functionality.
> - fixed formatting for goto lables - added newlines after condition
> - fixed logs for GICD_ISACTIVERnE and GICD_ICACTIVERnE handlers
> - removed #ifdefs in 2 places where they were adjacent and could be merged
>
> Changes in V2:
> - add missing rank index conversion for pending and inflight irqs
> ---
> xen/arch/arm/include/asm/vgic.h | 4 +
> xen/arch/arm/vgic-v3.c | 198 ++++++++++++++++++++++++++------
> xen/arch/arm/vgic.c | 23 ++++
> 3 files changed, 192 insertions(+), 33 deletions(-)
>
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 3aa22114ba..103bc3c74b 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -314,6 +314,10 @@ extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v,
> unsigned int b,
> unsigned int n,
> unsigned int s);
> +extern struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v,
> + unsigned int b,
> + unsigned int n,
> + unsigned int s);
> extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
> extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
> extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index 4369c55177..b5d766c98f 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -111,13 +111,10 @@ static uint64_t vgic_fetch_irouter(struct vgic_irq_rank *rank,
> * Note the offset will be aligned to the appropriate boundary.
> */
> static void vgic_store_irouter(struct domain *d, struct vgic_irq_rank *rank,
> - unsigned int offset, uint64_t irouter)
> + unsigned int virq, uint64_t irouter)
> {
> struct vcpu *new_vcpu, *old_vcpu;
> - unsigned int virq;
> -
> - /* There is 1 vIRQ per IROUTER */
> - virq = offset / NR_BYTES_PER_IROUTER;
> + unsigned int offset;
>
> /*
> * The IROUTER0-31, used for SGIs/PPIs, are reserved and should
> @@ -685,13 +682,20 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> {
> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /* We do not implement security extensions for guests, read zero */
> if ( dabt.size != DABT_WORD ) goto bad_width;
> goto read_as_zero;
>
> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> + if ( reg >= GICD_ISENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> *r = vreg_reg32_extract(rank->ienable, info);
> @@ -699,8 +703,13 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> return 1;
>
> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> + if ( reg >= GICD_ICENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> *r = vreg_reg32_extract(rank->ienable, info);
> @@ -710,20 +719,29 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> /* Read the pending status of an IRQ via GICD/GICR is not supported */
> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> goto read_as_zero;
>
> /* Read the active status of an IRQ via GICD/GICR is not supported */
> case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
> case VRANGE32(GICD_ICACTIVER, GICD_ICACTIVERN):
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> goto read_as_zero;
>
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> {
> uint32_t ipriorityr;
> uint8_t rank_index;
>
> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
> + if ( reg >= GICD_IPRIORITYRnE )
> + rank = vgic_ext_rank_offset(v, 8, reg - GICD_IPRIORITYRnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> rank_index = REG_RANK_INDEX(8, reg - GICD_IPRIORITYR, DABT_WORD);
>
> @@ -737,11 +755,15 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
> }
>
> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> {
> uint32_t icfgr;
>
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> + if ( reg >= GICD_ICFGRnE )
> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> icfgr = rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR, DABT_WORD)];
> @@ -782,46 +804,81 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
> {
> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /* We do not implement security extensions for guests, write ignore */
> goto write_ignore_32;
>
> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> + if ( reg >= GICD_ISENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> tr = rank->ienable;
> vreg_reg32_setbits(&rank->ienable, r, info);
> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
> + if ( reg >= GICD_ISENABLERnE )
> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
> + EXT_RANK_IDX2NUM(rank->index));
> + else
> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
> vgic_unlock_rank(v, rank, flags);
> return 1;
>
> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> + if ( reg >= GICD_ICENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> tr = rank->ienable;
> vreg_reg32_clearbits(&rank->ienable, r, info);
> - vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
> + if ( reg >= GICD_ICENABLERnE )
> + vgic_disable_irqs(v, (~rank->ienable) & tr,
> + EXT_RANK_IDX2NUM(rank->index));
> + else
> + vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
> vgic_unlock_rank(v, rank, flags);
> return 1;
>
> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
> + if ( reg >= GICD_ISPENDRnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISPENDRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
>
> - vgic_set_irqs_pending(v, r, rank->index);
> + if ( reg >= GICD_ISPENDRnE )
> + vgic_set_irqs_pending(v, r, EXT_RANK_IDX2NUM(rank->index));
> + else
> + vgic_set_irqs_pending(v, r, rank->index);
>
> return 1;
>
> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
> + if ( reg >= GICD_ICPENDRnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICPENDRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
>
> - vgic_check_inflight_irqs_pending(v, rank->index, r);
> + if ( reg >= GICD_ICPENDRnE )
> + vgic_check_inflight_irqs_pending(v,
> + EXT_RANK_IDX2NUM(rank->index), r);
> + else
> + vgic_check_inflight_irqs_pending(v, rank->index, r);
>
> goto write_ignore;
>
> @@ -838,16 +895,38 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
> v, name, r, reg - GICD_ICACTIVER);
> goto write_ignore_32;
>
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + if ( dabt.size != DABT_WORD )
> + goto bad_width;
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to ISACTIVER%dE\n",
> + v, name, r, reg - GICD_ISACTIVERnE);
> + return 0;
> +
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to ICACTIVER%dE\n",
> + v, name, r, reg - GICD_ICACTIVERnE);
> + goto write_ignore_32;
NIT: I would group with regular SPI ranges (taking into account that all
other ranges were already grouped including the read accesses),
something like that (non tested):
case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
+ case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
if ( dabt.size != DABT_WORD ) goto bad_width;
- printk(XENLOG_G_ERR
- "%pv: %s: unhandled word write %#"PRIregister" to
ISACTIVER%d\n",
- v, name, r, reg - GICD_ISACTIVER);
+ if ( reg >= GICD_ISACTIVERnE )
+ printk(XENLOG_G_ERR
+ "%pv: %s: unhandled word write %#"PRIregister" to
ISACTIVER%dE\n",
+ v, name, r, reg - GICD_ISACTIVERnE);
+ else
+ printk(XENLOG_G_ERR
+ "%pv: %s: unhandled word write %#"PRIregister" to
ISACTIVER%d\n",
+ v, name, r, reg - GICD_ISACTIVER);
return 0;
case VRANGE32(GICD_ICACTIVER, GICD_ICACTIVERN):
- printk(XENLOG_G_ERR
- "%pv: %s: unhandled word write %#"PRIregister" to
ICACTIVER%d\n",
- v, name, r, reg - GICD_ICACTIVER);
+ case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
+ if ( reg >= GICD_ICACTIVERnE )
+ printk(XENLOG_G_ERR
+ "%pv: %s: unhandled word write %#"PRIregister" to
ICACTIVER%dE\n",
+ v, name, r, reg - GICD_ICACTIVERnE);
+ else
+ printk(XENLOG_G_ERR
+ "%pv: %s: unhandled word write %#"PRIregister" to
ICACTIVER%d\n",
+ v, name, r, reg - GICD_ICACTIVER);
goto write_ignore_32;
> +
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> {
> - uint32_t *ipriorityr, priority;
> + uint32_t *ipriorityr, priority, offset;
>
> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
> + if ( reg >= GICD_IPRIORITYRnE ) {
> + offset = reg - GICD_IPRIORITYRnE;
> + rank = vgic_ext_rank_offset(v, 8, offset, DABT_WORD);
> + }
> + else
> + {
> + offset = reg - GICD_IPRIORITYR;
> + rank = vgic_rank_offset(v, 8, offset, DABT_WORD);
> + }
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> - ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, reg - GICD_IPRIORITYR,
> - DABT_WORD)];
> + ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, offset, DABT_WORD)];
Here
> priority = ACCESS_ONCE(*ipriorityr);
> vreg_reg32_update(&priority, r, info);
> ACCESS_ONCE(*ipriorityr) = priority;
> @@ -859,10 +938,14 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
> goto write_ignore_32;
>
> case VRANGE32(GICD_ICFGR + 4, GICD_ICFGRN): /* PPI + SPIs */
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> /* ICFGR1 for PPI's, which is implementation defined
> if ICFGR1 is programmable or not. We chose to program */
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> + if ( reg >= GICD_ICFGRnE )
> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> vreg_reg32_update(&rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR,
> @@ -1129,6 +1212,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> typer |= GICD_TYPE_LPIS;
>
> typer |= (v->domain->arch.vgic.intid_bits - 1) << GICD_TYPE_ID_BITS_SHIFT;
> +#ifdef CONFIG_GICV3_ESPI
> + if ( v->domain->arch.vgic.nr_espis > 0 )
> + {
> + /* Set eSPI support bit for the domain */
> + typer |= GICD_TYPER_ESPI;
> + /* Set ESPI range bits */
> + typer |= (DIV_ROUND_UP(v->domain->arch.vgic.nr_espis, 32) - 1)
> + << GICD_TYPER_ESPI_RANGE_SHIFT;
> + }
> +#endif
>
> *r = vreg_reg32_extract(typer, info);
>
> @@ -1194,6 +1287,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /*
> * Above all register are common with GICR and GICD
> * Manage in common
> @@ -1201,6 +1304,7 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> return __vgic_v3_distr_common_mmio_read("vGICD", v, info, gicd_reg, r);
>
> case VRANGE32(GICD_NSACR, GICD_NSACRN):
> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
> /* We do not implement security extensions for guests, read zero */
> goto read_as_zero_32;
>
> @@ -1216,16 +1320,21 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
> /* Replaced with GICR_ISPENDR0. So ignore write */
> goto read_as_zero_32;
>
> - case VRANGE32(0x0F30, 0x60FC):
> + case VRANGE32(0x0F30, 0x0FFC):
> goto read_reserved;
>
> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
> {
> uint64_t irouter;
>
> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
> - DABT_DOUBLE_WORD);
> + if ( gicd_reg >= GICD_IROUTERnE )
> + rank = vgic_ext_rank_offset(v, 64, gicd_reg - GICD_IROUTERnE,
> + DABT_DOUBLE_WORD);
> + else
> + rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
> + DABT_DOUBLE_WORD);
> if ( rank == NULL ) goto read_as_zero;
> vgic_lock_rank(v, rank, flags);
> irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
Here you use the same offset for regular and extended SPI ranges
(gicd_reg - GICD_IROUTER) ...
> @@ -1235,8 +1344,8 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>
> return 1;
> }
> -
> - case VRANGE32(0x7FE0, 0xBFFC):
> + case VRANGE32(0x3700, 0x60FC):
> + case VRANGE32(0xA004, 0xBFFC):
> goto read_reserved;
>
> case VRANGE32(0xC000, 0xFFCC):
> @@ -1382,12 +1491,23 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /* Above registers are common with GICR and GICD
> * Manage in common */
> return __vgic_v3_distr_common_mmio_write("vGICD", v, info,
> gicd_reg, r);
>
> case VRANGE32(GICD_NSACR, GICD_NSACRN):
> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
> /* We do not implement security extensions for guests, write ignore */
> goto write_ignore_32;
>
> @@ -1405,26 +1525,38 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
> if ( dabt.size != DABT_WORD ) goto bad_width;
> return 0;
>
> - case VRANGE32(0x0F30, 0x60FC):
> + case VRANGE32(0x0F30, 0x0FFC):
> goto write_reserved;
>
> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
> {
> uint64_t irouter;
> + unsigned int offset, virq;
>
> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
> - DABT_DOUBLE_WORD);
> + if ( gicd_reg >= GICD_IROUTERnE ) {
> + offset = gicd_reg - GICD_IROUTERnE;
> + rank = vgic_ext_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
> + } else {
> + offset = gicd_reg - GICD_IROUTER;
> + rank = vgic_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
> + }
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> - irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
> + irouter = vgic_fetch_irouter(rank, offset);
... But here you use different offsets for regular and extended SPI
ranges (gicd_reg - GICD_IROUTER vs gicd_reg - GICD_IROUTERnE). Could you
please clarify why (what did I miss)?
> vreg_reg64_update(&irouter, r, info);
> - vgic_store_irouter(v->domain, rank, gicd_reg - GICD_IROUTER, irouter);
> + if ( gicd_reg >= GICD_IROUTERnE )
> + virq = ESPI_IDX2INTID(offset / NR_BYTES_PER_IROUTER);
> + else
> + virq = offset / NR_BYTES_PER_IROUTER;
> + vgic_store_irouter(v->domain, rank, virq, irouter);
> vgic_unlock_rank(v, rank, flags);
> return 1;
> }
>
> - case VRANGE32(0x7FE0, 0xBFFC):
> + case VRANGE32(0x3700, 0x60FC):
> + case VRANGE32(0xA004, 0xBFFC):
> goto write_reserved;
>
> case VRANGE32(0xC000, 0xFFCC):
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index c9b9528c66..27ffdf316c 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -193,6 +193,18 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
> }
>
> #ifdef CONFIG_GICV3_ESPI
> +/*
> + * The function behavior is the same as for regular SPIs (vgic_rank_offset),
> + * but it operates with extended SPI ranks.
> + */
> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
> + unsigned int n, unsigned int s)
> +{
> + unsigned int rank = REG_RANK_NR(b, (n >> s));
> +
> + return vgic_get_rank(v, rank + EXT_RANK_MIN);
> +}
> +
> static unsigned int vgic_num_spi_lines(struct domain *d)
> {
> return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
> @@ -241,6 +253,17 @@ struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
> {
> return NULL;
> }
> +
> +/*
> + * It is expected that, in the case of CONFIG_GICV3_ESPI=n, the function will
> + * return NULL. In this scenario, mmio_read/mmio_write will be treated as
> + * RAZ/WI, as expected.
> + */
> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
> + unsigned int n, unsigned int s)
> +{
> + return NULL;
> +}
> #endif
>
> static unsigned int vgic_num_alloc_irqs(struct domain *d)
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations
2025-08-31 12:42 ` Oleksandr Tyshchenko
@ 2025-09-01 14:36 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-01 14:36 UTC (permalink / raw)
To: Oleksandr Tyshchenko, xen-devel@lists.xenproject.org
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel,
Volodymyr Babchuk, Julien Grall
Hello Oleksandr,
Thank you for your comment and RB.
On 31.08.25 15:42, Oleksandr Tyshchenko wrote:
>
>
> On 29.08.25 19:06, Leonid Komarianskyi wrote:
>
> Hello Leonid
>
>
>> Currently, many common functions perform the same operations to calculate
>> GIC register addresses. This patch consolidates the similar code into
>> a separate helper function to improve maintainability and reduce
>> duplication.
>> This refactoring also simplifies the implementation of eSPI support in
>> future
>> changes.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>> Acked-by: Julien Grall <jgrall@amazon.com>
>
> Reviewed-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>
> with one NIT ...
>
>>
>> ---
>> Changes in V5:
>> - fixed a minor nit: changed %d to %u in the warning message because the
>> IRQ number cannot be less than zero
>> - added acked-by from Julien Grall
>>
>> Changes in V4:
>> - no changes
>>
>> Changes in V3:
>> - changed panic() in get_addr_by_offset() to printing warning and
>> ASSERT_UNREACHABLE()
>> - added verification of return pointer from get_addr_by_offset() in the
>> callers
>> - moved invocation of get_addr_by_offset() from spinlock guards, since
>> it is not necessarry
>> - added RB from Volodymyr Babchuk
>>
>> Changes in V2:
>> - no changes
>> ---
>> xen/arch/arm/gic-v3.c | 114 +++++++++++++++++++++++----------
>> xen/arch/arm/include/asm/irq.h | 1 +
>> 2 files changed, 81 insertions(+), 34 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>> index cd3e1acf79..29b7f68cba 100644
>> --- a/xen/arch/arm/gic-v3.c
>> +++ b/xen/arch/arm/gic-v3.c
>> @@ -445,17 +445,67 @@ static void gicv3_dump_state(const struct vcpu *v)
>> }
>> }
>> +static void __iomem *get_addr_by_offset(struct irq_desc *irqd, u32
>> offset)
>
> ... please use uint32_t for new code
>
> (could probably updated on commit)
>
>
I will fix that in V6 and review the other patches to avoid the use of
u32/u64.
>> +{
>> + switch ( irqd->irq )
>> + {
>> + case 0 ... (NR_GIC_LOCAL_IRQS - 1):
>> + switch ( offset )
>> + {
>> + case GICD_ISENABLER:
>> + case GICD_ICENABLER:
>> + case GICD_ISPENDR:
>> + case GICD_ICPENDR:
>> + case GICD_ISACTIVER:
>> + case GICD_ICACTIVER:
>> + return (GICD_RDIST_SGI_BASE + offset);
>> + case GICD_ICFGR:
>> + return (GICD_RDIST_SGI_BASE + GICR_ICFGR1);
>> + case GICD_IPRIORITYR:
>> + return (GICD_RDIST_SGI_BASE + GICR_IPRIORITYR0 + irqd->irq);
>> + default:
>> + break;
>> + }
>> + case NR_GIC_LOCAL_IRQS ... SPI_MAX_INTID:
>> + switch ( offset )
>> + {
>> + case GICD_ISENABLER:
>> + case GICD_ICENABLER:
>> + case GICD_ISPENDR:
>> + case GICD_ICPENDR:
>> + case GICD_ISACTIVER:
>> + case GICD_ICACTIVER:
>> + return (GICD + offset + (irqd->irq / 32) * 4);
>> + case GICD_ICFGR:
>> + return (GICD + GICD_ICFGR + (irqd->irq / 16) * 4);
>> + case GICD_IROUTER:
>> + return (GICD + GICD_IROUTER + irqd->irq * 8);
>> + case GICD_IPRIORITYR:
>> + return (GICD + GICD_IPRIORITYR + irqd->irq);
>> + default:
>> + break;
>> + }
>> + default:
>> + break;
>> + }
>> +
>> + /* Something went wrong, we shouldn't be able to reach here */
>> + printk(XENLOG_WARNING "GICv3: WARNING: Invalid offset 0x%x for
>> IRQ#%u",
>> + offset, irqd->irq);
>> + ASSERT_UNREACHABLE();
>> +
>> + return NULL;
>> +}
>> +
>> static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool
>> wait_for_rwp)
>> {
>> u32 mask = 1U << (irqd->irq % 32);
>> - void __iomem *base;
>> + void __iomem *addr = get_addr_by_offset(irqd, offset);
>> - if ( irqd->irq < NR_GIC_LOCAL_IRQS )
>> - base = GICD_RDIST_SGI_BASE;
>> - else
>> - base = GICD;
>> + if ( addr == NULL )
>> + return;
>> - writel_relaxed(mask, base + offset + (irqd->irq / 32) * 4);
>> + writel_relaxed(mask, addr);
>> if ( wait_for_rwp )
>> gicv3_wait_for_rwp(irqd->irq);
>> @@ -463,15 +513,12 @@ static void gicv3_poke_irq(struct irq_desc
>> *irqd, u32 offset, bool wait_for_rwp)
>> static bool gicv3_peek_irq(struct irq_desc *irqd, u32 offset)
>> {
>> - void __iomem *base;
>> - unsigned int irq = irqd->irq;
>> + void __iomem *addr = get_addr_by_offset(irqd, offset);
>> - if ( irq >= NR_GIC_LOCAL_IRQS)
>> - base = GICD + (irq / 32) * 4;
>> - else
>> - base = GICD_RDIST_SGI_BASE;
>> + if ( addr == NULL )
>> + return false;
>> - return !!(readl(base + offset) & (1U << (irq % 32)));
>> + return !!(readl(addr) & (1U << (irqd->irq % 32)));
>> }
>> static void gicv3_unmask_irq(struct irq_desc *irqd)
>> @@ -558,30 +605,28 @@ static inline uint64_t
>> gicv3_mpidr_to_affinity(int cpu)
>> static void gicv3_set_irq_type(struct irq_desc *desc, unsigned int
>> type)
>> {
>> uint32_t cfg, actual, edgebit;
>> - void __iomem *base;
>> - unsigned int irq = desc->irq;
>> + void __iomem *addr;
>> /* SGI's are always edge-triggered not need to call GICD_ICFGR0 */
>> - ASSERT(irq >= NR_GIC_SGI);
>> + ASSERT(desc->irq >= NR_GIC_SGI);
>> - spin_lock(&gicv3.lock);
>> + addr = get_addr_by_offset(desc, GICD_ICFGR);
>> + if ( addr == NULL )
>> + return;
>> - if ( irq >= NR_GIC_LOCAL_IRQS)
>> - base = GICD + GICD_ICFGR + (irq / 16) * 4;
>> - else
>> - base = GICD_RDIST_SGI_BASE + GICR_ICFGR1;
>> + spin_lock(&gicv3.lock);
>> - cfg = readl_relaxed(base);
>> + cfg = readl_relaxed(addr);
>> - edgebit = 2u << (2 * (irq % 16));
>> + edgebit = 2u << (2 * (desc->irq % 16));
>> if ( type & IRQ_TYPE_LEVEL_MASK )
>> cfg &= ~edgebit;
>> else if ( type & IRQ_TYPE_EDGE_BOTH )
>> cfg |= edgebit;
>> - writel_relaxed(cfg, base);
>> + writel_relaxed(cfg, addr);
>> - actual = readl_relaxed(base);
>> + actual = readl_relaxed(addr);
>> if ( ( cfg & edgebit ) ^ ( actual & edgebit ) )
>> {
>> printk(XENLOG_WARNING "GICv3: WARNING: "
>> @@ -600,16 +645,13 @@ static void gicv3_set_irq_type(struct irq_desc
>> *desc, unsigned int type)
>> static void gicv3_set_irq_priority(struct irq_desc *desc,
>> unsigned int priority)
>> {
>> - unsigned int irq = desc->irq;
>> + void __iomem *addr = get_addr_by_offset(desc, GICD_IPRIORITYR);
>> - spin_lock(&gicv3.lock);
>> -
>> - /* Set priority */
>> - if ( irq < NR_GIC_LOCAL_IRQS )
>> - writeb_relaxed(priority, GICD_RDIST_SGI_BASE +
>> GICR_IPRIORITYR0 + irq);
>> - else
>> - writeb_relaxed(priority, GICD + GICD_IPRIORITYR + irq);
>> + if ( addr == NULL )
>> + return;
>> + spin_lock(&gicv3.lock);
>> + writeb_relaxed(priority, addr);
>> spin_unlock(&gicv3.lock);
>> }
>> @@ -1273,6 +1315,10 @@ static void gicv3_irq_set_affinity(struct
>> irq_desc *desc, const cpumask_t *mask)
>> {
>> unsigned int cpu;
>> uint64_t affinity;
>> + void __iomem *addr = get_addr_by_offset(desc, GICD_IROUTER);
>> +
>> + if ( addr == NULL )
>> + return;
>> ASSERT(!cpumask_empty(mask));
>> @@ -1284,7 +1330,7 @@ static void gicv3_irq_set_affinity(struct
>> irq_desc *desc, const cpumask_t *mask)
>> affinity &= ~GICD_IROUTER_SPI_MODE_ANY;
>> if ( desc->irq >= NR_GIC_LOCAL_IRQS )
>> - writeq_relaxed_non_atomic(affinity, (GICD + GICD_IROUTER +
>> desc->irq * 8));
>> + writeq_relaxed_non_atomic(affinity, addr);
>> spin_unlock(&gicv3.lock);
>> }
>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/
>> asm/irq.h
>> index fce7e42a33..5bc6475eb4 100644
>> --- a/xen/arch/arm/include/asm/irq.h
>> +++ b/xen/arch/arm/include/asm/irq.h
>> @@ -29,6 +29,7 @@ struct arch_irq_desc {
>> */
>> #define NR_IRQS 1024
>> +#define SPI_MAX_INTID 1019
>> #define LPI_OFFSET 8192
>> /* LPIs are always numbered starting at 8192, so 0 is a good invalid
>> case. */
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-08-31 14:08 ` Oleksandr Tyshchenko
@ 2025-09-01 14:42 ` Leonid Komarianskyi
2025-09-01 16:16 ` Julien Grall
0 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-01 14:42 UTC (permalink / raw)
To: Oleksandr Tyshchenko
Cc: Volodymyr Babchuk, xen-devel@lists.xenproject.org,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hello Oleksandr and Volodymyr,
Thank you for your comments.
On 31.08.25 17:08, Oleksandr Tyshchenko wrote:
>
>
> On 29.08.25 22:45, Volodymyr Babchuk wrote:
>>
>> Hi Leonid,
>
> Hello Leonid
>
>>
>> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>>
>>> Currently, Xen does not support eSPI interrupts, leading
>>> to a data abort when such interrupts are defined in the DTS.
>>>
>>> This patch introduces a separate array to initialize up to
>>> 1024 interrupt descriptors in the eSPI range and adds the
>>> necessary defines and helper function. These changes lay the
>>> groundwork for future implementation of full eSPI interrupt
>>> support. As this GICv3.1 feature is not required by all vendors,
>>> all changes are guarded by ifdefs, depending on the corresponding
>>> Kconfig option.
>>>
>>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>>
>>> ---
>>> Changes in V5:
>>> - no functional changes introduced by this version compared with V4,
>>> only
>>> minor fixes and removal of ifdefs for macroses
>>> - added TODO comment, suggested by Oleksandr Tyshchenko
>>> - changed int to unsigned int for irqs
>>> - removed ifdefs for eSPI-specific defines and macros to reduce the
>>> number of ifdefs and code duplication in further changes
>>> - removed reviewed-by as moving defines from ifdefs requires additional
>>> confirmation from reviewers
>
>
> Reviewed-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>
> with the following addressed ...
>
>
>>>
>>> Changes in V4:
>>> - removed redundant line with 'default n' in Kconfig, as it is disabled
>>> by default, without explicit specification
>>> - added reviewed-by from Volodymyr Babchuk
>>>
>>> Changes in V3:
>>> - introduced a new define NR_ESPI_IRQS to avoid confusion, like in the
>>> case of using NR_IRQS for espi_desc array
>>> - implemented helper functions espi_to_desc and init_espi_data to make
>>> it possible to add stubs with the same name, and as a result, reduce
>>> the number of #ifdefs
>>> - disable CONFIG_GICV3_ESPI default value to n
>>>
>>> Changes in V2:
>>> - use (ESPI_MAX_INTID + 1) instead of (ESPI_BASE_INTID + NR_IRQS)
>>> - remove unnecessary comment for nr_irqs initialization
>>> ---
>>> xen/arch/arm/Kconfig | 8 +++++
>>> xen/arch/arm/include/asm/irq.h | 24 +++++++++++++++
>>> xen/arch/arm/irq.c | 56 +++++++++++++++++++++++++++++++++-
>>> 3 files changed, 87 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>>> index 17df147b25..43b05533b1 100644
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -135,6 +135,14 @@ config GICV3
>>> Driver for the ARM Generic Interrupt Controller v3.
>>> If unsure, use the default setting.
>>> +config GICV3_ESPI
>>> + bool "Extended SPI range support"
>>> + depends on GICV3 && !NEW_VGIC
>>> + help
>>> + Allow Xen and domains to use interrupt numbers from the
>>> extended SPI
>>> + range, from 4096 to 5119. This feature is introduced in GICv3.1
>>> + architecture.
>>> +
>>> config HAS_ITS
>>> bool "GICv3 ITS MSI controller support (UNSUPPORTED)" if
>>> UNSUPPORTED
>>> depends on GICV3 && !NEW_VGIC && !ARM_32
>>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/
>>> asm/irq.h
>>> index 5bc6475eb4..4443799648 100644
>>> --- a/xen/arch/arm/include/asm/irq.h
>>> +++ b/xen/arch/arm/include/asm/irq.h
>>> @@ -32,6 +32,13 @@ struct arch_irq_desc {
>>> #define SPI_MAX_INTID 1019
>>> #define LPI_OFFSET 8192
>>> +#define ESPI_BASE_INTID 4096
>>> +#define ESPI_MAX_INTID 5119
>>> +#define NR_ESPI_IRQS 1024
>>> +
>>> +#define ESPI_INTID2IDX(intid) ((intid) - ESPI_BASE_INTID)
>>> +#define ESPI_IDX2INTID(idx) ((idx) + ESPI_BASE_INTID)
>>> +
>>> /* LPIs are always numbered starting at 8192, so 0 is a good
>>> invalid case. */
>>> #define INVALID_LPI 0
>>> @@ -39,7 +46,15 @@ struct arch_irq_desc {
>>> #define INVALID_IRQ 1023
>>> extern const unsigned int nr_irqs;
>>> +#ifdef CONFIG_GICV3_ESPI
>>> +/*
>>> + * This will also cover the eSPI range, as some critical devices
>>> + * for booting Xen (e.g., serial) may use this type of interrupts.
>>> + */
>>> +#define nr_static_irqs (ESPI_MAX_INTID + 1)
>>> +#else
>>> #define nr_static_irqs NR_IRQS
>>> +#endif
>>> struct irq_desc;
>>> struct irqaction;
>>> @@ -55,6 +70,15 @@ static inline bool is_lpi(unsigned int irq)
>>> return irq >= LPI_OFFSET;
>>> }
>>> +static inline bool is_espi(unsigned int irq)
>>> +{
>>> +#ifdef CONFIG_GICV3_ESPI
>>
>> Taking into account that with CONFIG_GICV3_ESPI=n we should never have
>> "irq" in eSPI range, do you really need this #ifdef? I think that
>> ASSERT_UNREACHABLE in espi_to_desc() is sufficient guard.
>>
>> Also, IRQ line number belongs to eSPI range regardless of
>> CONFIG_GICV3_ESPI,
>> value, so in my opinion is_espi() should always return correct value for
>> a given "irq".
>
> ... I agree with Volodymyr's suggestion for is_espi() to always return
> correct value for a given "irq".
>
>
I will fix that in V6.
>>
>>> + return (irq >= ESPI_BASE_INTID && irq <= ESPI_MAX_INTID);
>>
>> Also, you don't need parentheses here.
I will remove the redundant parentheses in V6 as well.
>>
>>> +#else
>>> + return false;
>>> +#endif
>>> +}
>>> +
>>> #define domain_pirq_to_irq(d, pirq) (pirq)
>>> bool is_assignable_irq(unsigned int irq);
>>> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
>>> index b8eccfc924..61c915c3f9 100644
>>> --- a/xen/arch/arm/irq.c
>>> +++ b/xen/arch/arm/irq.c
>>> @@ -19,7 +19,11 @@
>>> #include <asm/gic.h>
>>> #include <asm/vgic.h>
>>> +#ifdef CONFIG_GICV3_ESPI
>>> +const unsigned int nr_irqs = ESPI_MAX_INTID + 1;
>>> +#else
>>> const unsigned int nr_irqs = NR_IRQS;
>>> +#endif
>>> static unsigned int local_irqs_type[NR_LOCAL_IRQS];
>>> static DEFINE_SPINLOCK(local_irqs_type_lock);
>>> @@ -46,6 +50,53 @@ void irq_end_none(struct irq_desc *irq)
>>> }
>>> static irq_desc_t irq_desc[NR_IRQS - NR_LOCAL_IRQS];
>>> +#ifdef CONFIG_GICV3_ESPI
>>> +/*
>>> + * TODO: Consider allocating an array dynamically if
>>> + * there is a need to enable GICV3_ESPI by default.
>>> + */
>>> +static irq_desc_t espi_desc[NR_ESPI_IRQS];
>>> +
>>> +static struct irq_desc *espi_to_desc(unsigned int irq)
>>> +{
>>> + return &espi_desc[ESPI_INTID2IDX(irq)];
>>> +}
>>> +
>>> +static int __init init_espi_data(void)
>>> +{
>>> + unsigned int irq;
>>> +
>>> + for ( irq = ESPI_BASE_INTID; irq <= ESPI_MAX_INTID; irq++ )
>>> + {
>>> + struct irq_desc *desc = irq_to_desc(irq);
>>> + int rc = init_one_irq_desc(desc);
>>> +
>>> + if ( rc )
>>> + return rc;
>>> +
>>> + desc->irq = irq;
>>> + desc->action = NULL;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +#else
>>> +/*
>>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>>> + * because in this case, is_espi will always return false.
>
> This comment should also be updated.
>
Sure, I will update the comment accordingly.
>>> + */
>>> +static struct irq_desc *espi_to_desc(unsigned int irq)
>>> +{
>>> + ASSERT_UNREACHABLE();
>>> + return NULL;
>>> +}
>>> +
>>> +static int __init init_espi_data(void)
>>> +{
>>> + return 0;
>>> +}
>>> +#endif
>>> +
>>> static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
>>> struct irq_desc *__irq_to_desc(unsigned int irq)
>>> @@ -53,6 +104,9 @@ struct irq_desc *__irq_to_desc(unsigned int irq)
>>> if ( irq < NR_LOCAL_IRQS )
>>> return &this_cpu(local_irq_desc)[irq];
>>> + if ( is_espi(irq) )
>>> + return espi_to_desc(irq);
>>> +
>>> return &irq_desc[irq-NR_LOCAL_IRQS];
>>> }
>>> @@ -79,7 +133,7 @@ static int __init init_irq_data(void)
>>> desc->action = NULL;
>>> }
>>> - return 0;
>>> + return init_espi_data();
>>> }
>>> static int init_local_irq_data(unsigned int cpu)
>>
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-08-29 16:06 ` [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range Leonid Komarianskyi
2025-08-29 19:45 ` Volodymyr Babchuk
@ 2025-09-01 16:11 ` Julien Grall
2025-09-02 8:56 ` Leonid Komarianskyi
1 sibling, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-01 16:11 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Leonid,
On 29/08/2025 17:06, Leonid Komarianskyi wrote:
> Currently, Xen does not support eSPI interrupts, leading
> to a data abort when such interrupts are defined in the DTS.
>
> This patch introduces a separate array to initialize up to
> 1024 interrupt descriptors in the eSPI range and adds the
> necessary defines and helper function. These changes lay the
> groundwork for future implementation of full eSPI interrupt
> support. As this GICv3.1 feature is not required by all vendors,
> all changes are guarded by ifdefs, depending on the corresponding
> Kconfig option.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> ---
> Changes in V5:
> - no functional changes introduced by this version compared with V4, only
> minor fixes and removal of ifdefs for macroses
> - added TODO comment, suggested by Oleksandr Tyshchenko
> - changed int to unsigned int for irqs
> - removed ifdefs for eSPI-specific defines and macros to reduce the
> number of ifdefs and code duplication in further changes
> - removed reviewed-by as moving defines from ifdefs requires additional
> confirmation from reviewers
>
> Changes in V4:
> - removed redundant line with 'default n' in Kconfig, as it is disabled
> by default, without explicit specification
> - added reviewed-by from Volodymyr Babchuk
>
> Changes in V3:
> - introduced a new define NR_ESPI_IRQS to avoid confusion, like in the
> case of using NR_IRQS for espi_desc array
> - implemented helper functions espi_to_desc and init_espi_data to make
> it possible to add stubs with the same name, and as a result, reduce
> the number of #ifdefs
> - disable CONFIG_GICV3_ESPI default value to n
>
> Changes in V2:
> - use (ESPI_MAX_INTID + 1) instead of (ESPI_BASE_INTID + NR_IRQS)
> - remove unnecessary comment for nr_irqs initialization
> ---
> xen/arch/arm/Kconfig | 8 +++++
> xen/arch/arm/include/asm/irq.h | 24 +++++++++++++++
> xen/arch/arm/irq.c | 56 +++++++++++++++++++++++++++++++++-
> 3 files changed, 87 insertions(+), 1 deletion(-)
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 17df147b25..43b05533b1 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -135,6 +135,14 @@ config GICV3
> Driver for the ARM Generic Interrupt Controller v3.
> If unsure, use the default setting.
>
> +config GICV3_ESPI
> + bool "Extended SPI range support"
> + depends on GICV3 && !NEW_VGIC
> + help
> + Allow Xen and domains to use interrupt numbers from the extended SPI
> + range, from 4096 to 5119. This feature is introduced in GICv3.1
> + architecture.
> +
> config HAS_ITS
> bool "GICv3 ITS MSI controller support (UNSUPPORTED)" if UNSUPPORTED
> depends on GICV3 && !NEW_VGIC && !ARM_32
> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/asm/irq.h
> index 5bc6475eb4..4443799648 100644
> --- a/xen/arch/arm/include/asm/irq.h
> +++ b/xen/arch/arm/include/asm/irq.h
> @@ -32,6 +32,13 @@ struct arch_irq_desc {
> #define SPI_MAX_INTID 1019
> #define LPI_OFFSET 8192
>
> +#define ESPI_BASE_INTID 4096
> +#define ESPI_MAX_INTID 5119
> +#define NR_ESPI_IRQS 1024
> +
> +#define ESPI_INTID2IDX(intid) ((intid) - ESPI_BASE_INTID)
> +#define ESPI_IDX2INTID(idx) ((idx) + ESPI_BASE_INTID)
NIT: I would consider adding sanity check (i.e. ASSERT()) to confirm
that both ``intid`` and ``idx`` are within the bounds.
> +
> /* LPIs are always numbered starting at 8192, so 0 is a good invalid case. */
> #define INVALID_LPI 0
>
> @@ -39,7 +46,15 @@ struct arch_irq_desc {
> #define INVALID_IRQ 1023
>
> extern const unsigned int nr_irqs;
> +#ifdef CONFIG_GICV3_ESPI
> +/*
> + * This will also cover the eSPI range, as some critical devices
> + * for booting Xen (e.g., serial) may use this type of interrupts.
> + */
Reading this again, I still don't quite understand why we are mentioning
Xen devices. Looking at the code, for Arm, we only seem to use
nr_static_irqs to configure nr_pirqs and XSM. Both are ony used by domains.
So I think this needs to be clarified.
> +#define nr_static_irqs (ESPI_MAX_INTID + 1)
> +#else
> #define nr_static_irqs NR_IRQS
> +#endif
>
> struct irq_desc;
> struct irqaction;
> @@ -55,6 +70,15 @@ static inline bool is_lpi(unsigned int irq)
> return irq >= LPI_OFFSET;
> }
>
> +static inline bool is_espi(unsigned int irq)
> +{
> +#ifdef CONFIG_GICV3_ESPI
> + return (irq >= ESPI_BASE_INTID && irq <= ESPI_MAX_INTID);
> +#else
> + return false;
> +#endif
> +}
> +
> #define domain_pirq_to_irq(d, pirq) (pirq)
>
> bool is_assignable_irq(unsigned int irq);
> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
> index b8eccfc924..61c915c3f9 100644
> --- a/xen/arch/arm/irq.c
> +++ b/xen/arch/arm/irq.c
> @@ -19,7 +19,11 @@
> #include <asm/gic.h>
> #incl#ude <asm/vgic.h>
>
> +#ifdef CONFIG_GICV3_ESPI
> +const unsigned int nr_irqs = ESPI_MAX_INTID + 1;
> +#else
> const unsigned int nr_irqs = NR_IRQS;
> +#endif
NIT: I think you can use:
const unsigned int nr_irqs = IS_ENABLED(CONFIG_GICV3_ESPI)?
(ESPI_MAX_INTID + 1) : NR_IRQS;
That said, I think we need to rethink about the use of nr_irqs and
nr_static_irqs because they don't entirely make sense for Arm as we
don't support PIRQs.
I would at least try to get rid of one of the variable (maybe nr_irqs)
if not both.
This could be done as a follow-up.
>
> static unsigned int local_irqs_type[NR_LOCAL_IRQS];
> static DEFINE_SPINLOCK(local_irqs_type_lock);
> @@ -46,6 +50,53 @@ void irq_end_none(struct irq_desc *irq)
> }
>
> static irq_desc_t irq_desc[NR_IRQS - NR_LOCAL_IRQS];
> +#ifdef CONFIG_GICV3_ESPI
> +/*
> + * TODO: Consider allocating an array dynamically if
> + * there is a need to enable GICV3_ESPI by default.
> + */
I know this was suggested by Oleksandr, however most likely distro will
want to enable this feature so they can be booted on a wide range of
platform. So I think "if there is a need to enable GICV3_ESPI by
default" should be removed.
> +static irq_desc_t espi_desc[NR_ESPI_IRQS];> +
> +static struct irq_desc *espi_to_desc(unsigned int irq)
> +{
> + return &espi_desc[ESPI_INTID2IDX(irq)];
> +}
> +
> +static int __init init_espi_data(void)
> +{
> + unsigned int irq;
> +
> + for ( irq = ESPI_BASE_INTID; irq <= ESPI_MAX_INTID; irq++ )
> + {
> + struct irq_desc *desc = irq_to_desc(irq);
> + int rc = init_one_irq_desc(desc);
> +
> + if ( rc )
> + return rc;
> +
> + desc->irq = irq;
> + desc->action = NULL;
> + }
> +
> + return 0;
> +}
> +#else
> +/*
> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
> + * because in this case, is_espi will always return false.
> + */
Is this is not mean to be called, then can we only define the prototype
like we do for __bad_atomic_read()?
> +static struct irq_desc *espi_to_desc(unsigned int irq)
> +{
> + ASSERT_UNREACHABLE();
> + return NULL;
> +}
> +
> +static int __init init_espi_data(void)
> +{
> + return 0;
> +}
> +#endif
> +
> static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
>
> struct irq_desc *__irq_to_desc(unsigned int irq)
> @@ -53,6 +104,9 @@ struct irq_desc *__irq_to_desc(unsigned int irq)
> if ( irq < NR_LOCAL_IRQS )
> return &this_cpu(local_irq_desc)[irq];
>
> + if ( is_espi(irq) )
> + return espi_to_desc(irq);
> +
> return &irq_desc[irq-NR_LOCAL_IRQS];
> }
>
> @@ -79,7 +133,7 @@ static int __init init_irq_data(void)
> desc->action = NULL;
> }
>
> - return 0;
> + return init_espi_data();
> }
>
> static int init_local_irq_data(unsigned int cpu)
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-09-01 14:42 ` Leonid Komarianskyi
@ 2025-09-01 16:16 ` Julien Grall
2025-09-02 6:56 ` Leonid Komarianskyi
0 siblings, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-01 16:16 UTC (permalink / raw)
To: Leonid Komarianskyi, Oleksandr Tyshchenko
Cc: Volodymyr Babchuk, xen-devel@lists.xenproject.org,
Stefano Stabellini, Bertrand Marquis, Michal Orzel
Hi,
On 01/09/2025 15:42, Leonid Komarianskyi wrote:
>>> Taking into account that with CONFIG_GICV3_ESPI=n we should never have
>>> "irq" in eSPI range, do you really need this #ifdef? I think that
>>> ASSERT_UNREACHABLE in espi_to_desc() is sufficient guard.
>>>
>>> Also, IRQ line number belongs to eSPI range regardless of
>>> CONFIG_GICV3_ESPI,
>>> value, so in my opinion is_espi() should always return correct value for
>>> a given "irq".
>>
>> ... I agree with Volodymyr's suggestion for is_espi() to always return
>> correct value for a given "irq".
>>
>>
>
> I will fix that in V6.
I am not sure about this. If is_espi() is not returning false with
CONFIG_GICV3_EPSI, then the compiler would not be able to optimize code
like:
if (is_espi(...)) {
return espi_to_desc(irq);
}
return &irq_desc[irq-NR_LOCAL_IRQS];
irq_to_desc() is called fairly often, so I would like to keep the code
fairly optimized. An alternative would be to use #ifdef CONFIG_*. I
don't like it, but it could be a compromise if Oleksandr and Volodymyr
wants to push to remove #ifdef from CONFIG_IS_ESPI.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI
2025-08-29 19:55 ` Volodymyr Babchuk
@ 2025-09-01 17:30 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-01 17:30 UTC (permalink / raw)
To: Volodymyr Babchuk
Cc: xen-devel@lists.xenproject.org, olekstysh@gmail.com,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hi Volodymyr,
Thank tou for your review.
On 29.08.25 22:55, Volodymyr Babchuk wrote:
>
> Hi Leonid,
>
> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>
>> Introduced appropriate register definitions, helper macros,
>> and initialization of required GICv3.1 distributor registers
>> to support eSPI. This type of interrupt is handled in the
>> same way as regular SPI interrupts, with the following
>> differences:
>>
>> 1) eSPIs can have up to 1024 interrupts, starting from the
>> beginning of the range, whereas regular SPIs use INTIDs from
>> 32 to 1019, totaling 988 interrupts;
>> 2) eSPIs start at INTID 4096, necessitating additional interrupt
>> index conversion during register operations.
>>
>> In case if appropriate config is disabled, or GIC HW doesn't
>> support eSPI, the existing functionality will remain the same.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>
>> ---
>> Changes in V5:
>> - fixed minor nits, no functional changes: changed u32 to uint32_t and
>> added a comment noting that the configuration for eSPIs is the same as
>> for regular SPIs
>> - removed ifdefs for eSPI-specific offsets to reduce the number of
>> ifdefs and code duplication in further changes
>> - removed reviewed-by as moving offset from ifdefs requires additional
>> confirmation from reviewers
>>
>> Changes in V4:
>> - added offsets for GICD_IGRPMODRnE and GICD_NSACRnE that are required
>> for vGIC emulation
>> - added a log banner with eSPI information, similar to the one for
>> regular SPI
>> - added newline after ifdef and before gic_is_valid_line
>> - added reviewed-by from Volodymyr Babchuk
>>
>> Changes in V3:
>> - add __init attribute to gicv3_dist_espi_common_init
>> - change open-codded eSPI register initialization to the appropriate
>> gen-mask macro
>> - fixed formatting for lines with more than 80 symbols
>> - introduced gicv3_dist_espi_init_aff to be able to use stubs in case of
>> CONFIG_GICV3_ESPI disabled
>> - renamed parameter in the GICD_TYPER_ESPI_RANGE macro to espi_range
>> (name was taken from GIC specification) to avoid confusion
>> - changed type for i variable to unsigned int since it cannot be
>> negative
>>
>> Changes in V2:
>> - move gic_number_espis function from
>> [PATCH 08/10] xen/arm: vgic: add resource management for extended SPIs
>> to use it in the newly introduced gic_is_valid_espi
>> - add gic_is_valid_espi which checks if IRQ number is in supported
>> by HW eSPI range
>> - update gic_is_valid_irq conditions to allow operations with eSPIs
>> ---
>> xen/arch/arm/gic-v3.c | 83 ++++++++++++++++++++++++++
>> xen/arch/arm/include/asm/gic.h | 22 +++++++
>> xen/arch/arm/include/asm/gic_v3_defs.h | 38 ++++++++++++
>> 3 files changed, 143 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>> index 29b7f68cba..4a7ce12f26 100644
>> --- a/xen/arch/arm/gic-v3.c
>> +++ b/xen/arch/arm/gic-v3.c
>> @@ -485,6 +485,36 @@ static void __iomem *get_addr_by_offset(struct irq_desc *irqd, u32 offset)
>> default:
>> break;
>> }
>> +#ifdef CONFIG_GICV3_ESPI
>> + case ESPI_BASE_INTID ... ESPI_MAX_INTID:
>> + {
>> + uint32_t irq_index = ESPI_INTID2IDX(irqd->irq);
>> +
>> + switch ( offset )
>> + {
>> + case GICD_ISENABLER:
>> + return (GICD + GICD_ISENABLERnE + (irq_index / 32) * 4);
>> + case GICD_ICENABLER:
>> + return (GICD + GICD_ICENABLERnE + (irq_index / 32) * 4);
>> + case GICD_ISPENDR:
>> + return (GICD + GICD_ISPENDRnE + (irq_index / 32) * 4);
>> + case GICD_ICPENDR:
>> + return (GICD + GICD_ICPENDRnE + (irq_index / 32) * 4);
>> + case GICD_ISACTIVER:
>> + return (GICD + GICD_ISACTIVERnE + (irq_index / 32) * 4);
>> + case GICD_ICACTIVER:
>> + return (GICD + GICD_ICACTIVERnE + (irq_index / 32) * 4);
>> + case GICD_ICFGR:
>> + return (GICD + GICD_ICFGRnE + (irq_index / 16) * 4);
>> + case GICD_IROUTER:
>> + return (GICD + GICD_IROUTERnE + irq_index * 8);
>> + case GICD_IPRIORITYR:
>> + return (GICD + GICD_IPRIORITYRnE + irq_index);
>> + default:
>> + break;
>> + }
>> + }
>> +#endif
>> default:
>> break;
>> }
>> @@ -655,6 +685,55 @@ static void gicv3_set_irq_priority(struct irq_desc *desc,
>> spin_unlock(&gicv3.lock);
>> }
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +unsigned int gic_number_espis(void)
>> +{
>> + return gic_hw_ops->info->nr_espi;
>> +}
>> +
>> +static void __init gicv3_dist_espi_common_init(uint32_t type)
>> +{
>> + unsigned int espi_nr, i;
>> +
>> + espi_nr = min(1024U, GICD_TYPER_ESPIS_NUM(type));
>> + gicv3_info.nr_espi = espi_nr;
>> + /* The GIC HW doesn't support eSPI, so we can leave from here */
>> + if ( gicv3_info.nr_espi == 0 )
>> + return;
>> +
>> + printk("GICv3: %d eSPI lines\n", gicv3_info.nr_espi);
>> +
>> + /* The configuration for eSPIs is similar to that for regular SPIs */
>> + for ( i = 0; i < espi_nr; i += 16 )
>> + writel_relaxed(0, GICD + GICD_ICFGRnE + (i / 16) * 4);
>> +
>> + for ( i = 0; i < espi_nr; i += 4 )
>> + writel_relaxed(GIC_PRI_IRQ_ALL,
>> + GICD + GICD_IPRIORITYRnE + (i / 4) * 4);
>> +
>> + for ( i = 0; i < espi_nr; i += 32 )
>> + {
>> + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICENABLERnE + (i / 32) * 4);
>> + writel_relaxed(GENMASK(31, 0), GICD + GICD_ICACTIVERnE + (i / 32) * 4);
>> + }
>> +
>> + for ( i = 0; i < espi_nr; i += 32 )
>> + writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPRnE + (i / 32) * 4);
>> +}
>> +
>> +static void __init gicv3_dist_espi_init_aff(uint64_t affinity)
>> +{
>> + unsigned int i;
>> +
>> + for ( i = 0; i < gicv3_info.nr_espi; i++ )
>> + writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTERnE + i * 8);
>> +}
>> +#else
>> +static void __init gicv3_dist_espi_common_init(uint32_t type) { }
>> +
>> +static void __init gicv3_dist_espi_init_aff(uint64_t affinity) { }
>> +#endif
>> +
>> static void __init gicv3_dist_init(void)
>> {
>> uint32_t type;
>> @@ -700,6 +779,8 @@ static void __init gicv3_dist_init(void)
>> for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i += 32 )
>> writel_relaxed(GENMASK(31, 0), GICD + GICD_IGROUPR + (i / 32) * 4);
>>
>> + gicv3_dist_espi_common_init(type);
>> +
>> gicv3_dist_wait_for_rwp();
>>
>> /* Turn on the distributor */
>> @@ -713,6 +794,8 @@ static void __init gicv3_dist_init(void)
>>
>> for ( i = NR_GIC_LOCAL_IRQS; i < nr_lines; i++ )
>> writeq_relaxed_non_atomic(affinity, GICD + GICD_IROUTER + i * 8);
>> +
>> + gicv3_dist_espi_init_aff(affinity);
>> }
>>
>> static int gicv3_enable_redist(void)
>> diff --git a/xen/arch/arm/include/asm/gic.h b/xen/arch/arm/include/asm/gic.h
>> index 3fcee42675..1e747dcd99 100644
>> --- a/xen/arch/arm/include/asm/gic.h
>> +++ b/xen/arch/arm/include/asm/gic.h
>> @@ -306,8 +306,26 @@ extern void gic_dump_vgic_info(struct vcpu *v);
>>
>> /* Number of interrupt lines */
>> extern unsigned int gic_number_lines(void);
>> +#ifdef CONFIG_GICV3_ESPI
>> +extern unsigned int gic_number_espis(void);
>> +
>> +static inline bool gic_is_valid_espi(unsigned int irq)
>> +{
>> + return (irq >= ESPI_BASE_INTID &&
>> + irq < ESPI_IDX2INTID(gic_number_espis()));
>
> You don't need external () here.
>
Oh, I missed that, sorry:( I will remove them in V6 and recheck the
other patches for such isses.
>> +}
>> +#else
>> +static inline bool gic_is_valid_espi(unsigned int irq)
>> +{
>> + return false;
>> +}
>> +#endif
>> +
>> static inline bool gic_is_valid_line(unsigned int irq)
>> {
>> + if ( gic_is_valid_espi(irq) )
>> + return true;
>> +
>> return irq < gic_number_lines();
>> }
>
> As you are going to rework this patch anyways, my I ask to rewrite this
> function in the following way?
>
> static inline bool gic_is_valid_line(unsigned int irq)
> {
> return irq < gic_number_lines() || gic_is_valid_espi(irq);
> }
>
> My justification is that (irq < gic_number_lines()) case is more likely,
> so it is better to evaluate it first, only then check for eSPIs.
>
> I am sorry, I should asked it earlier, but only after removing #ifdef I
> saw that this part could be more optimal.
>
Thank you for pointing this out:) It is really make sence, so I will fix
this in V6.
>>
>> @@ -325,6 +343,10 @@ struct gic_info {
>> enum gic_version hw_version;
>> /* Number of GIC lines supported */
>> unsigned int nr_lines;
>> +#ifdef CONFIG_GICV3_ESPI
>> + /* Number of GIC eSPI supported */
>> + unsigned int nr_espi;
>> +#endif
>> /* Number of LR registers */
>> uint8_t nr_lrs;
>> /* Maintenance irq number */
>> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
>> index 2af093e774..3370b4cd52 100644
>> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
>> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
>> @@ -37,6 +37,44 @@
>> #define GICD_IROUTER1019 (0x7FD8)
>> #define GICD_PIDR2 (0xFFE8)
>>
>> +/* Additional registers for GICv3.1 */
>> +#define GICD_IGROUPRnE (0x1000)
>> +#define GICD_IGROUPRnEN (0x107C)
>> +#define GICD_ISENABLERnE (0x1200)
>> +#define GICD_ISENABLERnEN (0x127C)
>> +#define GICD_ICENABLERnE (0x1400)
>> +#define GICD_ICENABLERnEN (0x147C)
>> +#define GICD_ISPENDRnE (0x1600)
>> +#define GICD_ISPENDRnEN (0x167C)
>> +#define GICD_ICPENDRnE (0x1800)
>> +#define GICD_ICPENDRnEN (0x187C)
>> +#define GICD_ISACTIVERnE (0x1A00)
>> +#define GICD_ISACTIVERnEN (0x1A7C)
>> +#define GICD_ICACTIVERnE (0x1C00)
>> +#define GICD_ICACTIVERnEN (0x1C7C)
>> +#define GICD_IPRIORITYRnE (0x2000)
>> +#define GICD_IPRIORITYRnEN (0x23FC)
>> +#define GICD_ICFGRnE (0x3000)
>> +#define GICD_ICFGRnEN (0x30FC)
>> +#define GICD_IGRPMODRnE (0x3400)
>> +#define GICD_IGRPMODRnEN (0x347C)
>> +#define GICD_NSACRnE (0x3600)
>> +#define GICD_NSACRnEN (0x36FC)
>> +#define GICD_IROUTERnE (0x8000)
>> +#define GICD_IROUTERnEN (0x9FFC)
>> +
>> +#ifdef CONFIG_GICV3_ESPI
>> +#define GICD_TYPER_ESPI_SHIFT 8
>> +#define GICD_TYPER_ESPI_RANGE_SHIFT 27
>> +#define GICD_TYPER_ESPI_RANGE_MASK (0x1F)
>> +#define GICD_TYPER_ESPI (1U << GICD_TYPER_ESPI_SHIFT)
>> +#define GICD_TYPER_ESPI_RANGE(espi_range) ((((espi_range) & \
>> + GICD_TYPER_ESPI_RANGE_MASK) + 1) * 32)
>> +#define GICD_TYPER_ESPIS_NUM(typer) \
>> + (((typer) & GICD_TYPER_ESPI) ? \
>> + GICD_TYPER_ESPI_RANGE((typer) >> GICD_TYPER_ESPI_RANGE_SHIFT) : 0)
>> +#endif
>> +
>> /* Common between GICD_PIDR2 and GICR_PIDR2 */
>> #define GIC_PIDR2_ARCH_MASK (0xf0)
>> #define GIC_PIDR2_ARCH_GICv3 (0x30)
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-08-29 20:45 ` Volodymyr Babchuk
2025-08-31 15:58 ` Oleksandr Tyshchenko
@ 2025-09-01 17:38 ` Leonid Komarianskyi
1 sibling, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-01 17:38 UTC (permalink / raw)
To: Volodymyr Babchuk
Cc: xen-devel@lists.xenproject.org, olekstysh@gmail.com,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hello Volodymyr,
Thank you for your close review and suggestions.
On 29.08.25 23:45, Volodymyr Babchuk wrote:
>
> Hi Leonid,
>
> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>
>> This change introduces resource management in the VGIC to handle
>> extended SPIs introduced in GICv3.1. The pending_irqs and
>> allocated_irqs arrays are resized to support the required
>> number of eSPIs, based on what is supported by the hardware and
>> requested by the guest. A new field, ext_shared_irqs, is added
>> to the VGIC structure to store information about eSPIs, similar
>> to how shared_irqs is used for regular SPIs.
>>
>> Since the eSPI range starts at INTID 4096 and INTIDs between 1025
>> and 4095 are reserved, helper macros are introduced to simplify the
>> transformation of indices and to enable easier access to eSPI-specific
>> resources. These changes prepare the VGIC for processing eSPIs as
>> required by future functionality.
>>
>> The initialization and deinitialization paths for vgic have been updated
>> to allocate and free these resources appropriately. Additionally,
>> updated handling of INTIDs greater than 1024, passed from the toolstack
>> during domain creation, and verification logic ensures only valid SPI or
>> eSPI INTIDs are used.
>>
>> The existing SPI behavior remains unaffected when guests do not request
>> eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
>> option is disabled.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>
>> ---
>> Changes in V5:
>> - removed the has_espi field because it can be determined by checking
>> whether domain->arch.vgic.nr_espis is zero or not
>> - since vgic_ext_rank_offset is not used in this patch, it has been
>> moved to the appropriate patch in the patch series, which implements
>> vgic eSPI registers emulation and requires this function
>> - removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
>> and code duplication in further changes
>> - fixed minor nit: used %pd for printing domain with its ID
>>
>> Changes in V4:
>> - added has_espi field to simplify determining whether a domain is able
>> to operate with eSPI
>> - fixed formatting issues and misspellings
>>
>> Changes in V3:
>> - fixed formatting for lines with more than 80 symbols
>> - introduced helper functions to be able to use stubs in case of
>> CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
>> #ifdefs
>> - fixed checks for nr_spis in domain_vgic_init
>> - updated comment about nr_spis adjustments with dom0less mention
>> - moved comment with additional explanations before checks
>> - used unsigned int for indexes since they cannot be negative
>> - removed unnecessary parentheses
>> - move vgic_ext_rank_offset to the below ifdef guard, to reduce the
>> number of ifdefs
>>
>> Changes in V2:
>> - change is_espi_rank to is_valid_espi_rank to verify whether the array
>> element ext_shared_irqs exists. The previous version, is_espi_rank,
>> only checked if the rank index was less than the maximum possible eSPI
>> rank index, but this could potentially result in accessing a
>> non-existing array element. To address this, is_valid_espi_rank was
>> introduced, which ensures that the required eSPI rank exists
>> - move gic_number_espis to
>> xen/arm: gicv3: implement handling of GICv3.1 eSPI
>> - update vgic_is_valid_irq checks to allow operating with eSPIs
>> - remove redundant newline in vgic_allocate_virq
>> ---
>> xen/arch/arm/include/asm/vgic.h | 12 ++
>> xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
>> 2 files changed, 208 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
>> index 3e7cbbb196..912d5b7694 100644
>> --- a/xen/arch/arm/include/asm/vgic.h
>> +++ b/xen/arch/arm/include/asm/vgic.h
>> @@ -146,6 +146,10 @@ struct vgic_dist {
>> int nr_spis; /* Number of SPIs */
>> unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
>> struct vgic_irq_rank *shared_irqs;
>> +#ifdef CONFIG_GICV3_ESPI
>> + struct vgic_irq_rank *ext_shared_irqs;
>> + int nr_espis; /* Number of extended SPIs */
>> +#endif
>> /*
>> * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
>> * struct arch_vcpu.
>> @@ -243,6 +247,14 @@ struct vgic_ops {
>> /* Number of ranks of interrupt registers for a domain */
>> #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
>> +#endif
>> +#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
>> +#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
>> +#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
>> +#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
>> +
>> #define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
>> #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
>>
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 2bbf4d99aa..c9b9528c66 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -27,9 +27,82 @@
>>
>> bool vgic_is_valid_line(struct domain *d, unsigned int virq)
>> {
>> +#ifdef CONFIG_GICV3_ESPI
>> + if ( virq >= ESPI_BASE_INTID &&
>> + virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
>> + return true;
>> +#endif
>> +
>> return virq < vgic_num_irqs(d);
>> }
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * Since eSPI indexes start from 4096 and numbers from 1024 to
>> + * 4095 are forbidden, we need to check both lower and upper
>> + * limits for ranks.
>> + */
>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
>> +{
>> + return rank >= EXT_RANK_MIN &&
>> + EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
>> +}
>> +
>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>> + unsigned int rank)
>> +{
>> + return &v->domain->arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
>> +}
>> +
>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
>> +{
>> + return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
>> + d->arch.vgic.allocated_irqs);
>> +}
>> +
>> +static void arch_move_espis(struct vcpu *v)
>
> I don't need you need a copy of arch_move_irqs(). Se below for more info.
>
>> +{
>> + const cpumask_t *cpu_mask = cpumask_of(v->processor);
>> + struct domain *d = v->domain;
>> + struct pending_irq *p;
>> + struct vcpu *v_target;
>> + unsigned int i;
>> +
>> + for ( i = ESPI_BASE_INTID;
>> + i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
>> + {
>> + v_target = vgic_get_target_vcpu(v, i);
>> + p = irq_to_pending(v_target, i);
>> +
>> + if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
>> + irq_set_affinity(p->desc, cpu_mask);
>> + }
>> +}
>> +#else
>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
>> +{
>> + return false;
>> +}
>> +
>> +/*
>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>> + * because in this case, is_valid_espi_rank will always return false.
>> + */
>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>> + unsigned int rank)
>> +{
>> + ASSERT_UNREACHABLE();
>> + return NULL;
>> +}
>> +
>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
>> +{
>> + return false;
>> +}
>> +
>> +static void arch_move_espis(struct vcpu *v) { }
>> +#endif
>> +
>> static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
>> unsigned int rank)
>> {
>> @@ -37,6 +110,8 @@ static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
>> return v->arch.vgic.private_irqs;
>> else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
>> return &v->domain->arch.vgic.shared_irqs[rank - 1];
>> + else if ( is_valid_espi_rank(v->domain, rank) )
>> + return vgic_get_espi_rank(v, rank);
>> else
>> return NULL;
>> }
>> @@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
>> return 0;
>> }
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>> +{
>> + return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
>> +}
>> +
>> +static int init_vgic_espi(struct domain *d)
>> +{
>> + unsigned int i, idx;
>> +
>> + if ( d->arch.vgic.nr_espis == 0 )
>> + return 0;
>> +
>> + d->arch.vgic.ext_shared_irqs =
>> + xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
>> + if ( d->arch.vgic.ext_shared_irqs == NULL )
>> + return -ENOMEM;
>> +
>> + for ( i = d->arch.vgic.nr_spis, idx = 0;
>> + i < vgic_num_spi_lines(d); i++, idx++ )
>> + vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
>> + ESPI_IDX2INTID(idx));
>> +
>> + for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
>> + vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
>> +
>> + return 0;
>> +}
>> +
>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>
> I know that I should made this observation in previous version, but I
> didn't, sorry for that. Anyways, I don't think that this is a good idea
> to introduce this function and vgic_reserve_espi_virq(), as well as
> arch_move_espis(), actually, because in each case this is a code
> duplication, which is not good.
>
> I think that instead you need to introduce a pair of helpers that will
> map any (e)SPI number to pending_irq[]/allocate_irqs index and back.
>
> somethink like
>
> static inline unsigned virq_to_index(int virq)
> {
> if (is_espi(virq))
> return ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
> return virq;
> }
>
> See below for examples.
>
You do not need to say sorry for that :) You provided a very good
solution with this helper function. So thank you again for that - I will
add the helper function in V6 to consolidate similar code into it and,
as a result, reuse existing code without duplication.
>> +{
>> + irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
>> + return &d->arch.vgic.pending_irqs[irq];
>> +}
>> +#else
>> +static unsigned int init_vgic_espi(struct domain *d)
>> +{
>> + return 0;
>> +}
>> +
>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>> +{
>> + return d->arch.vgic.nr_spis;
>> +}
>> +
>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>> +{
>> + return NULL;
>> +}
>> +#endif
>> +
>> +static unsigned int vgic_num_alloc_irqs(struct domain *d)
>> +{
>> + return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
>> +}
>> +
>> int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> {
>> int i;
>> @@ -131,6 +262,36 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> */
>> nr_spis = ROUNDUP(nr_spis, 32);
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> + /*
>> + * During domain creation, the dom0less DomUs code or toolstack specifies
>> + * the maximum INTID, which is defined in the domain config subtracted by
>> + * 32 to cover the local IRQs (please see the comment to VGIC_DEF_NR_SPIS).
>> + * To compute the actual number of eSPI that will be usable for,
>> + * add back 32.
>> + */
>> + if ( nr_spis + 32 > ESPI_IDX2INTID(NR_ESPI_IRQS) )
>> + return -EINVAL;
>> +
>> + if ( nr_spis + 32 >= ESPI_BASE_INTID )
>> + {
>> + d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32, 1024U);
>> + /* Verify if GIC HW can handle provided INTID */
>> + if ( d->arch.vgic.nr_espis > gic_number_espis() )
>> + return -EINVAL;
>> + /*
>> + * Set the maximum available number for regular
>> + * SPI to pass the next check
>> + */
>> + nr_spis = VGIC_DEF_NR_SPIS;
>> + }
>> + else
>> + {
>> + /* Domain will use the regular SPI range */
>> + d->arch.vgic.nr_espis = 0;
>> + }
>> +#endif
>> +
>> /* Limit the number of virtual SPIs supported to (1020 - 32) = 988 */
>> if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>> return -EINVAL;
>> @@ -145,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> return -ENOMEM;
>>
>> d->arch.vgic.pending_irqs =
>> - xzalloc_array(struct pending_irq, d->arch.vgic.nr_spis);
>> + xzalloc_array(struct pending_irq, vgic_num_spi_lines(d));
>> if ( d->arch.vgic.pending_irqs == NULL )
>> return -ENOMEM;
>>
>> @@ -156,12 +317,16 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
>> vgic_rank_init(&d->arch.vgic.shared_irqs[i], i + 1, 0);
>>
>> + ret = init_vgic_espi(d);
>> + if ( ret )
>> + return ret;
>> +
>> ret = d->arch.vgic.handler->domain_init(d);
>> if ( ret )
>> return ret;
>>
>> d->arch.vgic.allocated_irqs =
>> - xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
>> + xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_alloc_irqs(d)));
>> if ( !d->arch.vgic.allocated_irqs )
>> return -ENOMEM;
>>
>> @@ -195,9 +360,27 @@ void domain_vgic_free(struct domain *d)
>> }
>> }
>>
>> +#ifdef CONFIG_GICV3_ESPI
>> + for ( i = 0; i < d->arch.vgic.nr_espis; i++ )
>> + {
>> + struct pending_irq *p = spi_to_pending(d, ESPI_IDX2INTID(i));
>> +
>> + if ( p->desc )
>> + {
>> + ret = release_guest_irq(d, p->irq);
>> + if ( ret )
>> + dprintk(XENLOG_G_WARNING, "%pd: Failed to release virq %u ret = %d\n",
>> + d, p->irq, ret);
>> + }
>> + }
>> +#endif
>> +
>> if ( d->arch.vgic.handler )
>> d->arch.vgic.handler->domain_free(d);
>> xfree(d->arch.vgic.shared_irqs);
>> +#ifdef CONFIG_GICV3_ESPI
>> + xfree(d->arch.vgic.ext_shared_irqs);
>> +#endif
>> xfree(d->arch.vgic.pending_irqs);
>> xfree(d->arch.vgic.allocated_irqs);
>> }
>> @@ -331,6 +514,8 @@ void arch_move_irqs(struct vcpu *v)
>> if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
>> irq_set_affinity(p->desc, cpu_mask);
>> }
>> +
>> + arch_move_espis(v);
>> }
>>
>> void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
>> @@ -538,6 +723,8 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
>> n = &v->arch.vgic.pending_irqs[irq];
>> else if ( is_lpi(irq) )
>> n = v->domain->arch.vgic.handler->lpi_to_pending(v->domain, irq);
>> + else if ( is_espi(irq) )
>> + n = espi_to_pending(v->domain, irq);
>> else
>> n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>> return n;
>> @@ -547,6 +734,9 @@ struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq)
>> {
>> ASSERT(irq >= NR_LOCAL_IRQS);
>>
>> + if ( is_espi(irq) )
>> + return espi_to_pending(d, irq);
>> +
>
> here you can just do
>
> idx = virq_to_idx(virq);
>
>> return &d->arch.vgic.pending_irqs[irq - 32];
>
> and
>
> return &d->arch.vgic.pending_irqs[idx];
>
> instead
>
>> }
>>
>> @@ -668,6 +858,9 @@ bool vgic_reserve_virq(struct domain *d, unsigned int virq)
>> if ( !vgic_is_valid_line(d, virq) )
>> return false;
>>
>> + if ( is_espi(virq) )
>> + return vgic_reserve_espi_virq(d, virq);
>> +
>
> here you can just do
>
> idx = virq_to_idx(virq)
>
>> return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
>
> and then just
>
> return !test_and_set_bit(idx, d->arch.vgic.allocated_irqs);
>
>
>> }
>>
>> @@ -685,7 +878,7 @@ int vgic_allocate_virq(struct domain *d, bool spi)
>> else
>> {
>> first = 32;
>> - end = vgic_num_irqs(d);
>> + end = vgic_num_alloc_irqs(d);
>> }
>
> I thinj you need to recalculate "virq" value at the end of this
> function. You'll return index in bitfield, but this is not the same is
> IRQ number in case of eSPIs. The helpers I mentioned before can help here.
>
Oh, I missed this. It definitely needs to be recalculated for eSPIs. I
will add a fix for this in V6.
>>
>> /*
>
> Lastly, I think that it is very wasteful to allocate pending_irqs as
> continuous array, because it will consist mostly of unused entries,
> especially with eSPIs enable. Probably, better approach will be to use radix
> tree. As a bonus, you can use IRQ line number as a key, and get rid of
> all these is_espi() checks and mappings between IRQ number and index in
> the array. But this is a much more drastic change, and I don't think that it
> should be done in this patch series...
>
Yes, I agree with that. It can be improved, but first, I need to prepare
the solution with dynamic allocation for IRQ descriptors, as I promised
previously. Of course, after merging the current eSPI patch series.
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-08-31 15:58 ` Oleksandr Tyshchenko
@ 2025-09-01 18:00 ` Leonid Komarianskyi
2025-09-02 8:16 ` Oleksandr Tyshchenko
0 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-01 18:00 UTC (permalink / raw)
To: Oleksandr Tyshchenko, Volodymyr Babchuk
Cc: xen-devel@lists.xenproject.org, Stefano Stabellini, Julien Grall,
Bertrand Marquis, Michal Orzel
Hello Oleksandr,
Thank you for your review.
On 31.08.25 18:58, Oleksandr Tyshchenko wrote:
>
>
> On 29.08.25 23:45, Volodymyr Babchuk wrote:
>
> Hello Leonid, Volodymyr
>
>>
>> Hi Leonid,
>>
>> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>>
>>> This change introduces resource management in the VGIC to handle
>>> extended SPIs introduced in GICv3.1. The pending_irqs and
>>> allocated_irqs arrays are resized to support the required
>>> number of eSPIs, based on what is supported by the hardware and
>>> requested by the guest. A new field, ext_shared_irqs, is added
>>> to the VGIC structure to store information about eSPIs, similar
>>> to how shared_irqs is used for regular SPIs.
>>>
>>> Since the eSPI range starts at INTID 4096 and INTIDs between 1025
>>> and 4095 are reserved, helper macros are introduced to simplify the
>>> transformation of indices and to enable easier access to eSPI-specific
>>> resources. These changes prepare the VGIC for processing eSPIs as
>>> required by future functionality.
>>>
>>> The initialization and deinitialization paths for vgic have been updated
>>> to allocate and free these resources appropriately. Additionally,
>>> updated handling of INTIDs greater than 1024, passed from the toolstack
>>> during domain creation, and verification logic ensures only valid SPI or
>>> eSPI INTIDs are used.
>>>
>>> The existing SPI behavior remains unaffected when guests do not request
>>> eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
>>> option is disabled.
>>>
>>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>>
>>> ---
>>> Changes in V5:
>>> - removed the has_espi field because it can be determined by checking
>>> whether domain->arch.vgic.nr_espis is zero or not
>>> - since vgic_ext_rank_offset is not used in this patch, it has been
>>> moved to the appropriate patch in the patch series, which implements
>>> vgic eSPI registers emulation and requires this function
>>> - removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
>>> and code duplication in further changes
>>> - fixed minor nit: used %pd for printing domain with its ID
>
> @Leonid, thanks for optimizing the series, now it looks much better (the
> number of #ifdef-s is reduced, code is reused).
>
>
I am doing my best, and you and the other reviewers are helping me
improve the code. Thank you for that!
>>>
>>> Changes in V4:
>>> - added has_espi field to simplify determining whether a domain is able
>>> to operate with eSPI
>>> - fixed formatting issues and misspellings
>>>
>>> Changes in V3:
>>> - fixed formatting for lines with more than 80 symbols
>>> - introduced helper functions to be able to use stubs in case of
>>> CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
>>> #ifdefs
>>> - fixed checks for nr_spis in domain_vgic_init
>>> - updated comment about nr_spis adjustments with dom0less mention
>>> - moved comment with additional explanations before checks
>>> - used unsigned int for indexes since they cannot be negative
>>> - removed unnecessary parentheses
>>> - move vgic_ext_rank_offset to the below ifdef guard, to reduce the
>>> number of ifdefs
>>>
>>> Changes in V2:
>>> - change is_espi_rank to is_valid_espi_rank to verify whether the array
>>> element ext_shared_irqs exists. The previous version, is_espi_rank,
>>> only checked if the rank index was less than the maximum possible
>>> eSPI
>>> rank index, but this could potentially result in accessing a
>>> non-existing array element. To address this, is_valid_espi_rank was
>>> introduced, which ensures that the required eSPI rank exists
>>> - move gic_number_espis to
>>> xen/arm: gicv3: implement handling of GICv3.1 eSPI
>>> - update vgic_is_valid_irq checks to allow operating with eSPIs
>>> - remove redundant newline in vgic_allocate_virq
>>> ---
>>> xen/arch/arm/include/asm/vgic.h | 12 ++
>>> xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
>>> 2 files changed, 208 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/
>>> asm/vgic.h
>>> index 3e7cbbb196..912d5b7694 100644
>>> --- a/xen/arch/arm/include/asm/vgic.h
>>> +++ b/xen/arch/arm/include/asm/vgic.h
>>> @@ -146,6 +146,10 @@ struct vgic_dist {
>>> int nr_spis; /* Number of SPIs */
>>> unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
>>> struct vgic_irq_rank *shared_irqs;
>>> +#ifdef CONFIG_GICV3_ESPI
>>> + struct vgic_irq_rank *ext_shared_irqs;
>>> + int nr_espis; /* Number of extended SPIs */
>
> It seems you have agreed (V4) that nr_espis could not be negative.
>
I appologize for that, I missed this change. I will fix it in V6.
>>> +#endif
>>> /*
>>> * SPIs are domain global, SGIs and PPIs are per-VCPU and
>>> stored in
>>> * struct arch_vcpu.
>>> @@ -243,6 +247,14 @@ struct vgic_ops {
>>> /* Number of ranks of interrupt registers for a domain */
>>> #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
>>> +#ifdef CONFIG_GICV3_ESPI
>>> +#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
>>> +#endif
>>> +#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
>>> +#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
>>> +#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
>>> +#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
>>> +
>>> #define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
>>> #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
>>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>>> index 2bbf4d99aa..c9b9528c66 100644
>>> --- a/xen/arch/arm/vgic.c
>>> +++ b/xen/arch/arm/vgic.c
>>> @@ -27,9 +27,82 @@
>>> bool vgic_is_valid_line(struct domain *d, unsigned int virq)
>>> {
>>> +#ifdef CONFIG_GICV3_ESPI
>>> + if ( virq >= ESPI_BASE_INTID &&
>>> + virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
>>> + return true;
>>> +#endif
>>> +
>>> return virq < vgic_num_irqs(d);
>>> }
>>> +#ifdef CONFIG_GICV3_ESPI
>>> +/*
>>> + * Since eSPI indexes start from 4096 and numbers from 1024 to
>>> + * 4095 are forbidden, we need to check both lower and upper
>>> + * limits for ranks.
>>> + */
>>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int
>>> rank)
>>> +{
>>> + return rank >= EXT_RANK_MIN &&
>>> + EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
>>> +}
>>> +
>>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>>> + unsigned int
>>> rank)
>>> +{
>>> + return &v->domain-
>>> >arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
>>> +}
>>> +
>>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned
>>> int virq)
>>> +{
>>> + return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
>>> + d->arch.vgic.allocated_irqs);
>>> +}
>>> +
>>> +static void arch_move_espis(struct vcpu *v)
>>
>> I don't need you need a copy of arch_move_irqs(). Se below for more info.
>>
>>> +{
>>> + const cpumask_t *cpu_mask = cpumask_of(v->processor);
>>> + struct domain *d = v->domain;
>>> + struct pending_irq *p;
>>> + struct vcpu *v_target;
>>> + unsigned int i;
>>> +
>>> + for ( i = ESPI_BASE_INTID;
>>> + i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
>>> + {
>>> + v_target = vgic_get_target_vcpu(v, i);
>>> + p = irq_to_pending(v_target, i);
>>> +
>>> + if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p-
>>> >status) )
>>> + irq_set_affinity(p->desc, cpu_mask);
>>> + }
>>> +}
>>> +#else
>>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int
>>> rank)
>>> +{
>>> + return false;
>>> +}
>>> +
>>> +/*
>>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>>> + * because in this case, is_valid_espi_rank will always return false.
>>> + */
>>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>>> + unsigned int
>>> rank)
>>> +{
>>> + ASSERT_UNREACHABLE();
>>> + return NULL;
>>> +}
>>> +
>>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned
>>> int virq)
>>> +{
>>> + return false;
>>> +}
>>> +
>>> +static void arch_move_espis(struct vcpu *v) { }
>>> +#endif
>>> +
>>> static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
>>> unsigned int rank)
>>> {
>>> @@ -37,6 +110,8 @@ static inline struct vgic_irq_rank
>>> *vgic_get_rank(struct vcpu *v,
>>> return v->arch.vgic.private_irqs;
>>> else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
>>> return &v->domain->arch.vgic.shared_irqs[rank - 1];
>>> + else if ( is_valid_espi_rank(v->domain, rank) )
>>> + return vgic_get_espi_rank(v, rank);
>>> else
>>> return NULL;
>>> }
>>> @@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d,
>>> unsigned int *mmio_count)
>>> return 0;
>>> }
>>> +#ifdef CONFIG_GICV3_ESPI
>>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>>> +{
>>> + return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
>>> +}
>>> +
>>> +static int init_vgic_espi(struct domain *d)
>>> +{
>>> + unsigned int i, idx;
>>> +
>>> + if ( d->arch.vgic.nr_espis == 0 )
>>> + return 0;
>>> +
>>> + d->arch.vgic.ext_shared_irqs =
>>> + xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
>>> + if ( d->arch.vgic.ext_shared_irqs == NULL )
>>> + return -ENOMEM;
>>> +
>>> + for ( i = d->arch.vgic.nr_spis, idx = 0;
>>> + i < vgic_num_spi_lines(d); i++, idx++ )
>>> + vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
>>> + ESPI_IDX2INTID(idx));
>>> +
>>> + for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
>>> + vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>>
>> I know that I should made this observation in previous version, but I
>> didn't, sorry for that. Anyways, I don't think that this is a good idea
>> to introduce this function and vgic_reserve_espi_virq(), as well as
>> arch_move_espis(), actually, because in each case this is a code
>> duplication, which is not good.
>>
>> I think that instead you need to introduce a pair of helpers that will
>> map any (e)SPI number to pending_irq[]/allocate_irqs index and back.
>>
>> somethink like
>>
>> static inline unsigned virq_to_index(int virq)
>> {
>> if (is_espi(virq))
>> return ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
>> return virq;
>> }
>>
>> See below for examples.
>>
>>> +{
>>> + irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
>>> + return &d->arch.vgic.pending_irqs[irq];
>>> +}
>>> +#else
>>> +static unsigned int init_vgic_espi(struct domain *d)
>>> +{
>>> + return 0;
>>> +}
>>> +
>>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>>> +{
>>> + return d->arch.vgic.nr_spis;
>>> +}
>>> +
>>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>>> +{
>>> + return NULL;
>>> +}
>>> +#endif
>>> +
>>> +static unsigned int vgic_num_alloc_irqs(struct domain *d)
>>> +{
>>> + return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
>>> +}
>
> I do not know where it would be better to put a comment related to non-
> visible in the patch context route_irq_to_guest(), but put it here.
>
> I am afraid, the vgic_num_irqs(d) printed in the following error message
> is not entirely correct with your changes:
>
> route_irq_to_guest():
>
> ...
>
> if ( !vgic_is_valid_line(d, virq) )
> {
> printk(XENLOG_G_ERR
> "the vIRQ number %u is too high for domain %u (max =
> %u)\n",
> irq, d->domain_id, vgic_num_irqs(d));
> return -EINVAL;
> }
>
>
Would it be okay to change the error message to something like:
"invalid vIRQ number %u for domain %pd\n"
I understand that it is a more generic error message, but I think it
might become overly complicated if I add more information stating that
the IRQ should be within the range 0...vgic_num_irqs(d) or
4096...ESPI_IDX2INTID(d->arch.vgic.nr_espis).
>>> +
>>> int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>>> {
>>> int i;
>>> @@ -131,6 +262,36 @@ int domain_vgic_init(struct domain *d, unsigned
>>> int nr_spis)
>>> */
>>> nr_spis = ROUNDUP(nr_spis, 32);
>>> +#ifdef CONFIG_GICV3_ESPI
>>> + /*
>>> + * During domain creation, the dom0less DomUs code or toolstack
>>> specifies
>>> + * the maximum INTID, which is defined in the domain config
>>> subtracted by
>>> + * 32 to cover the local IRQs (please see the comment to
>>> VGIC_DEF_NR_SPIS).
>>> + * To compute the actual number of eSPI that will be usable for,
>>> + * add back 32.
>>> + */
>>> + if ( nr_spis + 32 > ESPI_IDX2INTID(NR_ESPI_IRQS) )
>>> + return -EINVAL;
>>> +
>>> + if ( nr_spis + 32 >= ESPI_BASE_INTID )
>>> + {
>>> + d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32,
>>> 1024U);
>>> + /* Verify if GIC HW can handle provided INTID */
>>> + if ( d->arch.vgic.nr_espis > gic_number_espis() )
>>> + return -EINVAL;
>>> + /*
>>> + * Set the maximum available number for regular
>>> + * SPI to pass the next check
>>> + */
>>> + nr_spis = VGIC_DEF_NR_SPIS;
>>> + }
>>> + else
>>> + {
>>> + /* Domain will use the regular SPI range */
>>> + d->arch.vgic.nr_espis = 0;
>>> + }
>>> +#endif
>>> +
>>> /* Limit the number of virtual SPIs supported to (1020 - 32) =
>>> 988 */
>>> if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>>> return -EINVAL;
>>> @@ -145,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned
>>> int nr_spis)
>>> return -ENOMEM;
>>> d->arch.vgic.pending_irqs =
>>> - xzalloc_array(struct pending_irq, d->arch.vgic.nr_spis);
>>> + xzalloc_array(struct pending_irq, vgic_num_spi_lines(d));
>>> if ( d->arch.vgic.pending_irqs == NULL )
>>> return -ENOMEM;
>>> @@ -156,12 +317,16 @@ int domain_vgic_init(struct domain *d, unsigned
>>> int nr_spis)
>>> for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
>>> vgic_rank_init(&d->arch.vgic.shared_irqs[i], i + 1, 0);
>>> + ret = init_vgic_espi(d);
>>> + if ( ret )
>>> + return ret;
>>> +
>>> ret = d->arch.vgic.handler->domain_init(d);
>>> if ( ret )
>>> return ret;
>>> d->arch.vgic.allocated_irqs =
>>> - xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
>>> + xzalloc_array(unsigned long,
>>> BITS_TO_LONGS(vgic_num_alloc_irqs(d)));
>>> if ( !d->arch.vgic.allocated_irqs )
>>> return -ENOMEM;
>>> @@ -195,9 +360,27 @@ void domain_vgic_free(struct domain *d)
>>> }
>>> }
>>> +#ifdef CONFIG_GICV3_ESPI
>>> + for ( i = 0; i < d->arch.vgic.nr_espis; i++ )
>>> + {
>>> + struct pending_irq *p = spi_to_pending(d, ESPI_IDX2INTID(i));
>>> +
>>> + if ( p->desc )
>>> + {
>>> + ret = release_guest_irq(d, p->irq);
>>> + if ( ret )
>>> + dprintk(XENLOG_G_WARNING, "%pd: Failed to release
>>> virq %u ret = %d\n",
>>> + d, p->irq, ret);
>>> + }
>>> + }
>>> +#endif
>>> +
>>> if ( d->arch.vgic.handler )
>>> d->arch.vgic.handler->domain_free(d);
>>> xfree(d->arch.vgic.shared_irqs);
>>> +#ifdef CONFIG_GICV3_ESPI
>>> + xfree(d->arch.vgic.ext_shared_irqs);
>>> +#endif
>>> xfree(d->arch.vgic.pending_irqs);
>>> xfree(d->arch.vgic.allocated_irqs);
>>> }
>>> @@ -331,6 +514,8 @@ void arch_move_irqs(struct vcpu *v)
>>> if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING,
>>> &p->status) )
>>> irq_set_affinity(p->desc, cpu_mask);
>>> }
>>> +
>>> + arch_move_espis(v);
>>> }
>>> void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
>>> @@ -538,6 +723,8 @@ struct pending_irq *irq_to_pending(struct vcpu
>>> *v, unsigned int irq)
>>> n = &v->arch.vgic.pending_irqs[irq];
>>> else if ( is_lpi(irq) )
>>> n = v->domain->arch.vgic.handler->lpi_to_pending(v->domain,
>>> irq);
>>> + else if ( is_espi(irq) )
>>> + n = espi_to_pending(v->domain, irq);
>>> else
>>> n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>>> return n;
>>> @@ -547,6 +734,9 @@ struct pending_irq *spi_to_pending(struct domain
>>> *d, unsigned int irq)
>>> {
>>> ASSERT(irq >= NR_LOCAL_IRQS);
>>> + if ( is_espi(irq) )
>>> + return espi_to_pending(d, irq);
>>> +
>>
>> here you can just do
>>
>> idx = virq_to_idx(virq);
>>
>>> return &d->arch.vgic.pending_irqs[irq - 32];
>>
>> and
>>
>> return &d->arch.vgic.pending_irqs[idx];
>>
>> instead
>>
>>> }
>>> @@ -668,6 +858,9 @@ bool vgic_reserve_virq(struct domain *d, unsigned
>>> int virq)
>>> if ( !vgic_is_valid_line(d, virq) )
>>> return false;
>>> + if ( is_espi(virq) )
>>> + return vgic_reserve_espi_virq(d, virq);
>>> +
>>
>> here you can just do
>>
>> idx = virq_to_idx(virq)
>>
>>> return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
>>
>> and then just
>>
>> return !test_and_set_bit(idx, d->arch.vgic.allocated_irqs);
>>
>>
>>> }
>>> @@ -685,7 +878,7 @@ int vgic_allocate_virq(struct domain *d, bool spi)
>>> else
>>> {
>>> first = 32;
>>> - end = vgic_num_irqs(d);
>>> + end = vgic_num_alloc_irqs(d);
>>> }
>>
>> I thinj you need to recalculate "virq" value at the end of this
>> function. You'll return index in bitfield, but this is not the same is
>> IRQ number in case of eSPIs.
>
> +1
>
> The helpers I mentioned before can help here.
>>
>>> /*
>>
>> Lastly, I think that it is very wasteful to allocate pending_irqs as
>> continuous array, because it will consist mostly of unused entries,
>> especially with eSPIs enable. Probably, better approach will be to use
>> radix
>> tree. As a bonus, you can use IRQ line number as a key, and get rid of
>> all these is_espi() checks and mappings between IRQ number and index in
>> the array. But this is a much more drastic change, and I don't think
>> that it
>> should be done in this patch series...
>>
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-09-01 12:38 ` Oleksandr Tyshchenko
@ 2025-09-01 18:27 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-01 18:27 UTC (permalink / raw)
To: Oleksandr Tyshchenko, xen-devel@lists.xenproject.org
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel,
Volodymyr Babchuk
Hello Oleksandr,
Thank you for your review and your good qestion.
On 01.09.25 15:38, Oleksandr Tyshchenko wrote:
>
>
> On 29.08.25 19:06, Leonid Komarianskyi wrote:
>
>
> Hello Leonid
>
>> Implemented support for GICv3.1 extended SPI registers for vGICv3,
>> allowing the emulation of eSPI-specific behavior for guest domains.
>> The implementation includes read and write emulation for eSPI-related
>> registers (e.g., GICD_ISENABLERnE, GICD_IROUTERnE, and others),
>> following a similar approach to the handling of regular SPIs.
>>
>> The eSPI registers, previously located in reserved address ranges,
>> are now adjusted to support MMIO read and write operations correctly
>> when CONFIG_GICV3_ESPI is enabled.
>>
>> The availability of eSPIs and the number of emulated extended SPIs
>> for guest domains is reported by setting the appropriate bits in the
>> GICD_TYPER register, based on the number of eSPIs requested by the
>> domain and supported by the hardware. In cases where the configuration
>> option is disabled, the hardware does not support eSPIs, or the domain
>> does not request such interrupts, the functionality remains unchanged.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>
>> ---
>> Changes in V5:
>> - since eSPI-specific defines and macros are now available even when the
>> appropriate config is disabled, this allows us to remove many
>> #ifdef guards and reuse the existing code for regular SPIs for
>> eSPIs as
>> well, as eSPIs are processed similarly. This improves code readability
>> and consolidates the register ranges for SPIs and eSPIs in a single
>> place, simplifying future changes or fixes for SPIs and their
>> counterparts from the extended range
>> - moved vgic_ext_rank_offset() from
>> [08/12] xen/arm: vgic: add resource management for extended SPIs
>> as the function was unused before this patch
>> - added stub implementation of vgic_ext_rank_offset() when
>> CONFIG_GICV3_ESPI=n
>> - removed unnecessary defines for reserved ranges, which were
>> introduced in
>> V4 to reduce the number of #ifdefs. The issue is now resolved by
>> allowing the use of SPI-specific offsets and reworking the logic
>
>
> Looks very good now, thanks. Just one NIT and one question below ...
>
>>
>> Changes in V4:
>> - added missing RAZ and write ignore eSPI-specific registers ranges:
>> GICD_NSACRnE and GICD_IGRPMODRnE
>> - changed previously reserved range to cover GICD_NSACRnE and
>> GICD_IGRPMODRnE
>> - introduced GICD_RESERVED_RANGE<n>_START/END defines to remove
>> hardcoded values and reduce the number of ifdefs
>> - fixed reserved ranges with eSPI option enabled: added missing range
>> 0x0F30-0x0F7C
>> - updated the logic for domains that do not support eSPI, but Xen is
>> compiled with the eSPI option. Now, prior to other MMIO checks, we
>> verify whether eSPI is available for the domain or not. If not, it
>> behaves as it does currently - RAZ and WI
>> - fixed print for GICD_ICACTIVERnE
>> - fixed new lines formatting for switch-case
>>
>> Changes in V3:
>> - changed vgic_store_irouter parameters - instead of offset virq is
>> used, to remove the additional bool espi parameter and simplify
>> checks. Also, adjusted parameters for regular SPI. Since the offset
>> parameter was used only for calculating virq number and then reused
>> for
>> finding rank offset, it will not affect functionality.
>> - fixed formatting for goto lables - added newlines after condition
>> - fixed logs for GICD_ISACTIVERnE and GICD_ICACTIVERnE handlers
>> - removed #ifdefs in 2 places where they were adjacent and could be
>> merged
>>
>> Changes in V2:
>> - add missing rank index conversion for pending and inflight irqs
>> ---
>> xen/arch/arm/include/asm/vgic.h | 4 +
>> xen/arch/arm/vgic-v3.c | 198 ++++++++++++++++++++++++++------
>> xen/arch/arm/vgic.c | 23 ++++
>> 3 files changed, 192 insertions(+), 33 deletions(-)
>>
>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/
>> asm/vgic.h
>> index 3aa22114ba..103bc3c74b 100644
>> --- a/xen/arch/arm/include/asm/vgic.h
>> +++ b/xen/arch/arm/include/asm/vgic.h
>> @@ -314,6 +314,10 @@ extern struct vgic_irq_rank
>> *vgic_rank_offset(struct vcpu *v,
>> unsigned int b,
>> unsigned int n,
>> unsigned int s);
>> +extern struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v,
>> + unsigned int b,
>> + unsigned int n,
>> + unsigned int s);
>> extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned
>> int irq);
>> extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned
>> int n);
>> extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned
>> int n);
>> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
>> index 4369c55177..b5d766c98f 100644
>> --- a/xen/arch/arm/vgic-v3.c
>> +++ b/xen/arch/arm/vgic-v3.c
>> @@ -111,13 +111,10 @@ static uint64_t vgic_fetch_irouter(struct
>> vgic_irq_rank *rank,
>> * Note the offset will be aligned to the appropriate boundary.
>> */
>> static void vgic_store_irouter(struct domain *d, struct
>> vgic_irq_rank *rank,
>> - unsigned int offset, uint64_t irouter)
>> + unsigned int virq, uint64_t irouter)
>> {
>> struct vcpu *new_vcpu, *old_vcpu;
>> - unsigned int virq;
>> -
>> - /* There is 1 vIRQ per IROUTER */
>> - virq = offset / NR_BYTES_PER_IROUTER;
>> + unsigned int offset;
>> /*
>> * The IROUTER0-31, used for SGIs/PPIs, are reserved and should
>> @@ -685,13 +682,20 @@ static int
>> __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
>> {
>> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /* We do not implement security extensions for guests, read
>> zero */
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> goto read_as_zero;
>> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>> + if ( reg >= GICD_ISENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER,
>> DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> *r = vreg_reg32_extract(rank->ienable, info);
>> @@ -699,8 +703,13 @@ static int __vgic_v3_distr_common_mmio_read(const
>> char *name, struct vcpu *v,
>> return 1;
>> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
>> + if ( reg >= GICD_ICENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER,
>> DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> *r = vreg_reg32_extract(rank->ienable, info);
>> @@ -710,20 +719,29 @@ static int
>> __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
>> /* Read the pending status of an IRQ via GICD/GICR is not
>> supported */
>> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
>> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> goto read_as_zero;
>> /* Read the active status of an IRQ via GICD/GICR is not
>> supported */
>> case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
>> case VRANGE32(GICD_ICACTIVER, GICD_ICACTIVERN):
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> goto read_as_zero;
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> {
>> uint32_t ipriorityr;
>> uint8_t rank_index;
>> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto
>> bad_width;
>> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
>> + if ( reg >= GICD_IPRIORITYRnE )
>> + rank = vgic_ext_rank_offset(v, 8, reg - GICD_IPRIORITYRnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR,
>> DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> rank_index = REG_RANK_INDEX(8, reg - GICD_IPRIORITYR,
>> DABT_WORD);
>> @@ -737,11 +755,15 @@ static int
>> __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
>> }
>> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> {
>> uint32_t icfgr;
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> + if ( reg >= GICD_ICFGRnE )
>> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE,
>> DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> icfgr = rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR,
>> DABT_WORD)];
>> @@ -782,46 +804,81 @@ static int
>> __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>> {
>> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /* We do not implement security extensions for guests, write
>> ignore */
>> goto write_ignore_32;
>> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>> + if ( reg >= GICD_ISENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER,
>> DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> tr = rank->ienable;
>> vreg_reg32_setbits(&rank->ienable, r, info);
>> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>> + if ( reg >= GICD_ISENABLERnE )
>> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
>> + EXT_RANK_IDX2NUM(rank->index));
>> + else
>> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>> vgic_unlock_rank(v, rank, flags);
>> return 1;
>> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
>> + if ( reg >= GICD_ICENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER,
>> DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> tr = rank->ienable;
>> vreg_reg32_clearbits(&rank->ienable, r, info);
>> - vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
>> + if ( reg >= GICD_ICENABLERnE )
>> + vgic_disable_irqs(v, (~rank->ienable) & tr,
>> + EXT_RANK_IDX2NUM(rank->index));
>> + else
>> + vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
>> vgic_unlock_rank(v, rank, flags);
>> return 1;
>> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
>> + if ( reg >= GICD_ISPENDRnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISPENDRnE,
>> DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR,
>> DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> - vgic_set_irqs_pending(v, r, rank->index);
>> + if ( reg >= GICD_ISPENDRnE )
>> + vgic_set_irqs_pending(v, r, EXT_RANK_IDX2NUM(rank->index));
>> + else
>> + vgic_set_irqs_pending(v, r, rank->index);
>> return 1;
>> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
>> + if ( reg >= GICD_ICPENDRnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICPENDRnE,
>> DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR,
>> DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> - vgic_check_inflight_irqs_pending(v, rank->index, r);
>> + if ( reg >= GICD_ICPENDRnE )
>> + vgic_check_inflight_irqs_pending(v,
>> + EXT_RANK_IDX2NUM(rank-
>> >index), r);
>> + else
>> + vgic_check_inflight_irqs_pending(v, rank->index, r);
>> goto write_ignore;
>> @@ -838,16 +895,38 @@ static int
>> __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>> v, name, r, reg - GICD_ICACTIVER);
>> goto write_ignore_32;
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + if ( dabt.size != DABT_WORD )
>> + goto bad_width;
>> + printk(XENLOG_G_ERR
>> + "%pv: %s: unhandled word write %#"PRIregister" to
>> ISACTIVER%dE\n",
>> + v, name, r, reg - GICD_ISACTIVERnE);
>> + return 0;
>> +
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> + printk(XENLOG_G_ERR
>> + "%pv: %s: unhandled word write %#"PRIregister" to
>> ICACTIVER%dE\n",
>> + v, name, r, reg - GICD_ICACTIVERnE);
>> + goto write_ignore_32;
>
>
> NIT: I would group with regular SPI ranges (taking into account that all
> other ranges were already grouped including the read accesses),
> something like that (non tested):
>
> case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - printk(XENLOG_G_ERR
> - "%pv: %s: unhandled word write %#"PRIregister" to
> ISACTIVER%d\n",
> - v, name, r, reg - GICD_ISACTIVER);
> + if ( reg >= GICD_ISACTIVERnE )
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to
> ISACTIVER%dE\n",
> + v, name, r, reg - GICD_ISACTIVERnE);
> + else
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to
> ISACTIVER%d\n",
> + v, name, r, reg - GICD_ISACTIVER);
> return 0;
>
> case VRANGE32(GICD_ICACTIVER, GICD_ICACTIVERN):
> - printk(XENLOG_G_ERR
> - "%pv: %s: unhandled word write %#"PRIregister" to
> ICACTIVER%d\n",
> - v, name, r, reg - GICD_ICACTIVER);
> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
> + if ( reg >= GICD_ICACTIVERnE )
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to
> ICACTIVER%dE\n",
> + v, name, r, reg - GICD_ICACTIVERnE);
> + else
> + printk(XENLOG_G_ERR
> + "%pv: %s: unhandled word write %#"PRIregister" to
> ICACTIVER%d\n",
> + v, name, r, reg - GICD_ICACTIVER);
> goto write_ignore_32;
>
>
I agree with that. With such changes, it will be much easier to modify
the code for SPI and eSPI counterparts if needed. I will regroup them in V6.
>> +
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> {
>> - uint32_t *ipriorityr, priority;
>> + uint32_t *ipriorityr, priority, offset;
>> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto
>> bad_width;
>> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
>> + if ( reg >= GICD_IPRIORITYRnE ) {
>> + offset = reg - GICD_IPRIORITYRnE;
>> + rank = vgic_ext_rank_offset(v, 8, offset, DABT_WORD);
>> + }
>> + else
>> + {
>> + offset = reg - GICD_IPRIORITYR;
>> + rank = vgic_rank_offset(v, 8, offset, DABT_WORD);
>> + }
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> - ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, reg -
>> GICD_IPRIORITYR,
>> - DABT_WORD)];
>> + ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, offset,
>> DABT_WORD)];
>
> Here
>
>
>> priority = ACCESS_ONCE(*ipriorityr);
>> vreg_reg32_update(&priority, r, info);
>> ACCESS_ONCE(*ipriorityr) = priority;
>> @@ -859,10 +938,14 @@ static int
>> __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>> goto write_ignore_32;
>> case VRANGE32(GICD_ICFGR + 4, GICD_ICFGRN): /* PPI + SPIs */
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> /* ICFGR1 for PPI's, which is implementation defined
>> if ICFGR1 is programmable or not. We chose to program */
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> + if ( reg >= GICD_ICFGRnE )
>> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE,
>> DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> vreg_reg32_update(&rank->icfg[REG_RANK_INDEX(2, reg -
>> GICD_ICFGR,
>> @@ -1129,6 +1212,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu
>> *v, mmio_info_t *info,
>> typer |= GICD_TYPE_LPIS;
>> typer |= (v->domain->arch.vgic.intid_bits - 1) <<
>> GICD_TYPE_ID_BITS_SHIFT;
>> +#ifdef CONFIG_GICV3_ESPI
>> + if ( v->domain->arch.vgic.nr_espis > 0 )
>> + {
>> + /* Set eSPI support bit for the domain */
>> + typer |= GICD_TYPER_ESPI;
>> + /* Set ESPI range bits */
>> + typer |= (DIV_ROUND_UP(v->domain->arch.vgic.nr_espis, 32)
>> - 1)
>> + << GICD_TYPER_ESPI_RANGE_SHIFT;
>> + }
>> +#endif
>> *r = vreg_reg32_extract(typer, info);
>> @@ -1194,6 +1287,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu
>> *v, mmio_info_t *info,
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /*
>> * Above all register are common with GICR and GICD
>> * Manage in common
>> @@ -1201,6 +1304,7 @@ static int vgic_v3_distr_mmio_read(struct vcpu
>> *v, mmio_info_t *info,
>> return __vgic_v3_distr_common_mmio_read("vGICD", v, info,
>> gicd_reg, r);
>> case VRANGE32(GICD_NSACR, GICD_NSACRN):
>> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
>> /* We do not implement security extensions for guests, read
>> zero */
>> goto read_as_zero_32;
>> @@ -1216,16 +1320,21 @@ static int vgic_v3_distr_mmio_read(struct vcpu
>> *v, mmio_info_t *info,
>> /* Replaced with GICR_ISPENDR0. So ignore write */
>> goto read_as_zero_32;
>> - case VRANGE32(0x0F30, 0x60FC):
>> + case VRANGE32(0x0F30, 0x0FFC):
>> goto read_reserved;
>> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
>> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
>> {
>> uint64_t irouter;
>> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
>> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
>> - DABT_DOUBLE_WORD);
>> + if ( gicd_reg >= GICD_IROUTERnE )
>> + rank = vgic_ext_rank_offset(v, 64, gicd_reg -
>> GICD_IROUTERnE,
>> + DABT_DOUBLE_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
>> + DABT_DOUBLE_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
>
> Here you use the same offset for regular and extended SPI ranges
> (gicd_reg - GICD_IROUTER) ...
>
>
Oh, this is an error, and it only works because GICD_IROUTER has an
offset of 0x6000, GICD_IROUTERnE has 0x8000, and vgic_fetch_irouter
applies the mask INTERRUPT_RANK_MASK (0x1F):
/* There is exactly 1 vIRQ per IROUTER */
offset /= NR_BYTES_PER_IROUTER;
/* Get the index in the rank */
offset &= INTERRUPT_RANK_MASK;
After applying the mask, the 'additional' difference of 0x2000/8 was
removed..
The offset should be recalculated according to the register being
processed, as in the code below.
I will fix this and review other places with similar code.
>> @@ -1235,8 +1344,8 @@ static int vgic_v3_distr_mmio_read(struct vcpu
>> *v, mmio_info_t *info,
>> return 1;
>> }
>> -
>> - case VRANGE32(0x7FE0, 0xBFFC):
>> + case VRANGE32(0x3700, 0x60FC):
>> + case VRANGE32(0xA004, 0xBFFC):
>> goto read_reserved;
>> case VRANGE32(0xC000, 0xFFCC):
>> @@ -1382,12 +1491,23 @@ static int vgic_v3_distr_mmio_write(struct
>> vcpu *v, mmio_info_t *info,
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /* Above registers are common with GICR and GICD
>> * Manage in common */
>> return __vgic_v3_distr_common_mmio_write("vGICD", v, info,
>> gicd_reg, r);
>> case VRANGE32(GICD_NSACR, GICD_NSACRN):
>> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
>> /* We do not implement security extensions for guests, write
>> ignore */
>> goto write_ignore_32;
>> @@ -1405,26 +1525,38 @@ static int vgic_v3_distr_mmio_write(struct
>> vcpu *v, mmio_info_t *info,
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> return 0;
>> - case VRANGE32(0x0F30, 0x60FC):
>> + case VRANGE32(0x0F30, 0x0FFC):
>> goto write_reserved;
>> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
>> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
>> {
>> uint64_t irouter;
>> + unsigned int offset, virq;
>> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
>> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
>> - DABT_DOUBLE_WORD);
>> + if ( gicd_reg >= GICD_IROUTERnE ) {
>> + offset = gicd_reg - GICD_IROUTERnE;
>> + rank = vgic_ext_rank_offset(v, 64, offset,
>> DABT_DOUBLE_WORD);
>> + } else {
>> + offset = gicd_reg - GICD_IROUTER;
>> + rank = vgic_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
>> + }
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> - irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
>> + irouter = vgic_fetch_irouter(rank, offset);
>
> ... But here you use different offsets for regular and extended SPI
> ranges (gicd_reg - GICD_IROUTER vs gicd_reg - GICD_IROUTERnE). Could you
> please clarify why (what did I miss)?
>
>
>> vreg_reg64_update(&irouter, r, info);
>> - vgic_store_irouter(v->domain, rank, gicd_reg - GICD_IROUTER,
>> irouter);
>> + if ( gicd_reg >= GICD_IROUTERnE )
>> + virq = ESPI_IDX2INTID(offset / NR_BYTES_PER_IROUTER);
>> + else
>> + virq = offset / NR_BYTES_PER_IROUTER;
>> + vgic_store_irouter(v->domain, rank, virq, irouter);
>> vgic_unlock_rank(v, rank, flags);
>> return 1;
>> }
>> - case VRANGE32(0x7FE0, 0xBFFC):
>> + case VRANGE32(0x3700, 0x60FC):
>> + case VRANGE32(0xA004, 0xBFFC):
>> goto write_reserved;
>> case VRANGE32(0xC000, 0xFFCC):
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index c9b9528c66..27ffdf316c 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -193,6 +193,18 @@ int domain_vgic_register(struct domain *d,
>> unsigned int *mmio_count)
>> }
>> #ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * The function behavior is the same as for regular SPIs
>> (vgic_rank_offset),
>> + * but it operates with extended SPI ranks.
>> + */
>> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned
>> int b,
>> + unsigned int n, unsigned
>> int s)
>> +{
>> + unsigned int rank = REG_RANK_NR(b, (n >> s));
>> +
>> + return vgic_get_rank(v, rank + EXT_RANK_MIN);
>> +}
>> +
>> static unsigned int vgic_num_spi_lines(struct domain *d)
>> {
>> return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
>> @@ -241,6 +253,17 @@ struct pending_irq *espi_to_pending(struct domain
>> *d, unsigned int irq)
>> {
>> return NULL;
>> }
>> +
>> +/*
>> + * It is expected that, in the case of CONFIG_GICV3_ESPI=n, the
>> function will
>> + * return NULL. In this scenario, mmio_read/mmio_write will be
>> treated as
>> + * RAZ/WI, as expected.
>> + */
>> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned
>> int b,
>> + unsigned int n, unsigned
>> int s)
>> +{
>> + return NULL;
>> +}
>> #endif
>> static unsigned int vgic_num_alloc_irqs(struct domain *d)
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-08-29 20:12 ` Volodymyr Babchuk
@ 2025-09-01 18:32 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-01 18:32 UTC (permalink / raw)
To: Volodymyr Babchuk
Cc: xen-devel@lists.xenproject.org, olekstysh@gmail.com,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
Hi Volodymyr,
Thank you for your review and RB.
Unfortunately, the code contains an error related to offset calculation
for GICD_IROUTERnE, which was observed by Oleksandr. I will fix it in
V6, along with the formatting issues you discovered. I apologize for that.
On 29.08.25 23:12, Volodymyr Babchuk wrote:
> Hi Leonid,
>
> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>
>> Implemented support for GICv3.1 extended SPI registers for vGICv3,
>> allowing the emulation of eSPI-specific behavior for guest domains.
>> The implementation includes read and write emulation for eSPI-related
>> registers (e.g., GICD_ISENABLERnE, GICD_IROUTERnE, and others),
>> following a similar approach to the handling of regular SPIs.
>>
>> The eSPI registers, previously located in reserved address ranges,
>> are now adjusted to support MMIO read and write operations correctly
>> when CONFIG_GICV3_ESPI is enabled.
>>
>> The availability of eSPIs and the number of emulated extended SPIs
>> for guest domains is reported by setting the appropriate bits in the
>> GICD_TYPER register, based on the number of eSPIs requested by the
>> domain and supported by the hardware. In cases where the configuration
>> option is disabled, the hardware does not support eSPIs, or the domain
>> does not request such interrupts, the functionality remains unchanged.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> I have a couple of comments about coding style, but apart from that it
> looks really good. With these issues fixed:
>
> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>
>>
>> ---
>> Changes in V5:
>> - since eSPI-specific defines and macros are now available even when the
>> appropriate config is disabled, this allows us to remove many
>> #ifdef guards and reuse the existing code for regular SPIs for eSPIs as
>> well, as eSPIs are processed similarly. This improves code readability
>> and consolidates the register ranges for SPIs and eSPIs in a single
>> place, simplifying future changes or fixes for SPIs and their
>> counterparts from the extended range
>> - moved vgic_ext_rank_offset() from
>> [08/12] xen/arm: vgic: add resource management for extended SPIs
>> as the function was unused before this patch
>> - added stub implementation of vgic_ext_rank_offset() when CONFIG_GICV3_ESPI=n
>> - removed unnecessary defines for reserved ranges, which were introduced in
>> V4 to reduce the number of #ifdefs. The issue is now resolved by
>> allowing the use of SPI-specific offsets and reworking the logic
>>
>> Changes in V4:
>> - added missing RAZ and write ignore eSPI-specific registers ranges:
>> GICD_NSACRnE and GICD_IGRPMODRnE
>> - changed previously reserved range to cover GICD_NSACRnE and
>> GICD_IGRPMODRnE
>> - introduced GICD_RESERVED_RANGE<n>_START/END defines to remove
>> hardcoded values and reduce the number of ifdefs
>> - fixed reserved ranges with eSPI option enabled: added missing range
>> 0x0F30-0x0F7C
>> - updated the logic for domains that do not support eSPI, but Xen is
>> compiled with the eSPI option. Now, prior to other MMIO checks, we
>> verify whether eSPI is available for the domain or not. If not, it
>> behaves as it does currently - RAZ and WI
>> - fixed print for GICD_ICACTIVERnE
>> - fixed new lines formatting for switch-case
>>
>> Changes in V3:
>> - changed vgic_store_irouter parameters - instead of offset virq is
>> used, to remove the additional bool espi parameter and simplify
>> checks. Also, adjusted parameters for regular SPI. Since the offset
>> parameter was used only for calculating virq number and then reused for
>> finding rank offset, it will not affect functionality.
>> - fixed formatting for goto lables - added newlines after condition
>> - fixed logs for GICD_ISACTIVERnE and GICD_ICACTIVERnE handlers
>> - removed #ifdefs in 2 places where they were adjacent and could be merged
>>
>> Changes in V2:
>> - add missing rank index conversion for pending and inflight irqs
>> ---
>> xen/arch/arm/include/asm/vgic.h | 4 +
>> xen/arch/arm/vgic-v3.c | 198 ++++++++++++++++++++++++++------
>> xen/arch/arm/vgic.c | 23 ++++
>> 3 files changed, 192 insertions(+), 33 deletions(-)
>>
>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
>> index 3aa22114ba..103bc3c74b 100644
>> --- a/xen/arch/arm/include/asm/vgic.h
>> +++ b/xen/arch/arm/include/asm/vgic.h
>> @@ -314,6 +314,10 @@ extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v,
>> unsigned int b,
>> unsigned int n,
>> unsigned int s);
>> +extern struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v,
>> + unsigned int b,
>> + unsigned int n,
>> + unsigned int s);
>> extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
>> extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
>> extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, unsigned int n);
>> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
>> index 4369c55177..b5d766c98f 100644
>> --- a/xen/arch/arm/vgic-v3.c
>> +++ b/xen/arch/arm/vgic-v3.c
>> @@ -111,13 +111,10 @@ static uint64_t vgic_fetch_irouter(struct vgic_irq_rank *rank,
>> * Note the offset will be aligned to the appropriate boundary.
>> */
>> static void vgic_store_irouter(struct domain *d, struct vgic_irq_rank *rank,
>> - unsigned int offset, uint64_t irouter)
>> + unsigned int virq, uint64_t irouter)
>> {
>> struct vcpu *new_vcpu, *old_vcpu;
>> - unsigned int virq;
>> -
>> - /* There is 1 vIRQ per IROUTER */
>> - virq = offset / NR_BYTES_PER_IROUTER;
>> + unsigned int offset;
>>
>> /*
>> * The IROUTER0-31, used for SGIs/PPIs, are reserved and should
>> @@ -685,13 +682,20 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
>> {
>> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /* We do not implement security extensions for guests, read zero */
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> goto read_as_zero;
>>
>> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>> + if ( reg >= GICD_ISENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> *r = vreg_reg32_extract(rank->ienable, info);
>> @@ -699,8 +703,13 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
>> return 1;
>>
>> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
>> + if ( reg >= GICD_ICENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> *r = vreg_reg32_extract(rank->ienable, info);
>> @@ -710,20 +719,29 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
>> /* Read the pending status of an IRQ via GICD/GICR is not supported */
>> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
>> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> goto read_as_zero;
>>
>> /* Read the active status of an IRQ via GICD/GICR is not supported */
>> case VRANGE32(GICD_ISACTIVER, GICD_ISACTIVERN):
>> case VRANGE32(GICD_ICACTIVER, GICD_ICACTIVERN):
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> goto read_as_zero;
>>
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> {
>> uint32_t ipriorityr;
>> uint8_t rank_index;
>>
>> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
>> + if ( reg >= GICD_IPRIORITYRnE )
>> + rank = vgic_ext_rank_offset(v, 8, reg - GICD_IPRIORITYRnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> rank_index = REG_RANK_INDEX(8, reg - GICD_IPRIORITYR, DABT_WORD);
>>
>> @@ -737,11 +755,15 @@ static int __vgic_v3_distr_common_mmio_read(const char *name, struct vcpu *v,
>> }
>>
>> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> {
>> uint32_t icfgr;
>>
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> + if ( reg >= GICD_ICFGRnE )
>> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> icfgr = rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR, DABT_WORD)];
>> @@ -782,46 +804,81 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>> {
>> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /* We do not implement security extensions for guests, write ignore */
>> goto write_ignore_32;
>>
>> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>> + if ( reg >= GICD_ISENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> tr = rank->ienable;
>> vreg_reg32_setbits(&rank->ienable, r, info);
>> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>> + if ( reg >= GICD_ISENABLERnE )
>> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
>> + EXT_RANK_IDX2NUM(rank->index));
>> + else
>> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>> vgic_unlock_rank(v, rank, flags);
>> return 1;
>>
>> case VRANGE32(GICD_ICENABLER, GICD_ICENABLERN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
>> + if ( reg >= GICD_ICENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ICENABLER, DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> tr = rank->ienable;
>> vreg_reg32_clearbits(&rank->ienable, r, info);
>> - vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
>> + if ( reg >= GICD_ICENABLERnE )
>> + vgic_disable_irqs(v, (~rank->ienable) & tr,
>> + EXT_RANK_IDX2NUM(rank->index));
>> + else
>> + vgic_disable_irqs(v, (~rank->ienable) & tr, rank->index);
>> vgic_unlock_rank(v, rank, flags);
>> return 1;
>>
>> case VRANGE32(GICD_ISPENDR, GICD_ISPENDRN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
>> + if ( reg >= GICD_ISPENDRnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISPENDRnE, DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISPENDR, DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>>
>> - vgic_set_irqs_pending(v, r, rank->index);
>> + if ( reg >= GICD_ISPENDRnE )
>> + vgic_set_irqs_pending(v, r, EXT_RANK_IDX2NUM(rank->index));
>> + else
>> + vgic_set_irqs_pending(v, r, rank->index);
>>
>> return 1;
>>
>> case VRANGE32(GICD_ICPENDR, GICD_ICPENDRN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
>> + if ( reg >= GICD_ICPENDRnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ICPENDRnE, DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ICPENDR, DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>>
>> - vgic_check_inflight_irqs_pending(v, rank->index, r);
>> + if ( reg >= GICD_ICPENDRnE )
>> + vgic_check_inflight_irqs_pending(v,
>> + EXT_RANK_IDX2NUM(rank->index), r);
>> + else
>> + vgic_check_inflight_irqs_pending(v, rank->index, r);
>>
>> goto write_ignore;
>>
>> @@ -838,16 +895,38 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>> v, name, r, reg - GICD_ICACTIVER);
>> goto write_ignore_32;
>>
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + if ( dabt.size != DABT_WORD )
>> + goto bad_width;
>> + printk(XENLOG_G_ERR
>> + "%pv: %s: unhandled word write %#"PRIregister" to ISACTIVER%dE\n",
>> + v, name, r, reg - GICD_ISACTIVERnE);
>> + return 0;
>> +
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> + printk(XENLOG_G_ERR
>> + "%pv: %s: unhandled word write %#"PRIregister" to ICACTIVER%dE\n",
>> + v, name, r, reg - GICD_ICACTIVERnE);
>> + goto write_ignore_32;
>> +
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> {
>> - uint32_t *ipriorityr, priority;
>> + uint32_t *ipriorityr, priority, offset;
>>
>> if ( dabt.size != DABT_BYTE && dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 8, reg - GICD_IPRIORITYR, DABT_WORD);
>> + if ( reg >= GICD_IPRIORITYRnE ) {
>
> Brace should go on new line
>
>> + offset = reg - GICD_IPRIORITYRnE;
>> + rank = vgic_ext_rank_offset(v, 8, offset, DABT_WORD);
>> + }
>> + else
>> + {
>> + offset = reg - GICD_IPRIORITYR;
>> + rank = vgic_rank_offset(v, 8, offset, DABT_WORD);
>> + }
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> - ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, reg - GICD_IPRIORITYR,
>> - DABT_WORD)];
>> + ipriorityr = &rank->ipriorityr[REG_RANK_INDEX(8, offset, DABT_WORD)];
>> priority = ACCESS_ONCE(*ipriorityr);
>> vreg_reg32_update(&priority, r, info);
>> ACCESS_ONCE(*ipriorityr) = priority;
>> @@ -859,10 +938,14 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>> goto write_ignore_32;
>>
>> case VRANGE32(GICD_ICFGR + 4, GICD_ICFGRN): /* PPI + SPIs */
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> /* ICFGR1 for PPI's, which is implementation defined
>> if ICFGR1 is programmable or not. We chose to program */
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> + if ( reg >= GICD_ICFGRnE )
>> + rank = vgic_ext_rank_offset(v, 2, reg - GICD_ICFGRnE, DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 2, reg - GICD_ICFGR, DABT_WORD);
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> vreg_reg32_update(&rank->icfg[REG_RANK_INDEX(2, reg - GICD_ICFGR,
>> @@ -1129,6 +1212,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>> typer |= GICD_TYPE_LPIS;
>>
>> typer |= (v->domain->arch.vgic.intid_bits - 1) << GICD_TYPE_ID_BITS_SHIFT;
>> +#ifdef CONFIG_GICV3_ESPI
>> + if ( v->domain->arch.vgic.nr_espis > 0 )
>> + {
>> + /* Set eSPI support bit for the domain */
>> + typer |= GICD_TYPER_ESPI;
>> + /* Set ESPI range bits */
>> + typer |= (DIV_ROUND_UP(v->domain->arch.vgic.nr_espis, 32) - 1)
>> + << GICD_TYPER_ESPI_RANGE_SHIFT;
>> + }
>> +#endif
>>
>> *r = vreg_reg32_extract(typer, info);
>>
>> @@ -1194,6 +1287,16 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /*
>> * Above all register are common with GICR and GICD
>> * Manage in common
>> @@ -1201,6 +1304,7 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>> return __vgic_v3_distr_common_mmio_read("vGICD", v, info, gicd_reg, r);
>>
>> case VRANGE32(GICD_NSACR, GICD_NSACRN):
>> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
>> /* We do not implement security extensions for guests, read zero */
>> goto read_as_zero_32;
>>
>> @@ -1216,16 +1320,21 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>> /* Replaced with GICR_ISPENDR0. So ignore write */
>> goto read_as_zero_32;
>>
>> - case VRANGE32(0x0F30, 0x60FC):
>> + case VRANGE32(0x0F30, 0x0FFC):
>> goto read_reserved;
>>
>> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
>> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
>> {
>> uint64_t irouter;
>>
>> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
>> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
>> - DABT_DOUBLE_WORD);
>> + if ( gicd_reg >= GICD_IROUTERnE )
>> + rank = vgic_ext_rank_offset(v, 64, gicd_reg - GICD_IROUTERnE,
>> + DABT_DOUBLE_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
>> + DABT_DOUBLE_WORD);
>> if ( rank == NULL ) goto read_as_zero;
>> vgic_lock_rank(v, rank, flags);
>> irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
>> @@ -1235,8 +1344,8 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>>
>> return 1;
>> }
>> -
>> - case VRANGE32(0x7FE0, 0xBFFC):
>> + case VRANGE32(0x3700, 0x60FC):
>> + case VRANGE32(0xA004, 0xBFFC):
>> goto read_reserved;
>>
>> case VRANGE32(0xC000, 0xFFCC):
>> @@ -1382,12 +1491,23 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
>> case VRANGE32(GICD_IPRIORITYR, GICD_IPRIORITYRN):
>> case VRANGE32(GICD_ICFGR, GICD_ICFGRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> + case VRANGE32(GICD_ICENABLERnE, GICD_ICENABLERnEN):
>> + case VRANGE32(GICD_ISPENDRnE, GICD_ISPENDRnEN):
>> + case VRANGE32(GICD_ICPENDRnE, GICD_ICPENDRnEN):
>> + case VRANGE32(GICD_ISACTIVERnE, GICD_ISACTIVERnEN):
>> + case VRANGE32(GICD_ICACTIVERnE, GICD_ICACTIVERnEN):
>> + case VRANGE32(GICD_IPRIORITYRnE, GICD_IPRIORITYRnEN):
>> + case VRANGE32(GICD_ICFGRnE, GICD_ICFGRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /* Above registers are common with GICR and GICD
>> * Manage in common */
>> return __vgic_v3_distr_common_mmio_write("vGICD", v, info,
>> gicd_reg, r);
>>
>> case VRANGE32(GICD_NSACR, GICD_NSACRN):
>> + case VRANGE32(GICD_NSACRnE, GICD_NSACRnEN):
>> /* We do not implement security extensions for guests, write ignore */
>> goto write_ignore_32;
>>
>> @@ -1405,26 +1525,38 @@ static int vgic_v3_distr_mmio_write(struct vcpu *v, mmio_info_t *info,
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> return 0;
>>
>> - case VRANGE32(0x0F30, 0x60FC):
>> + case VRANGE32(0x0F30, 0x0FFC):
>> goto write_reserved;
>>
>> case VRANGE64(GICD_IROUTER32, GICD_IROUTER1019):
>> + case VRANGE64(GICD_IROUTERnE, GICD_IROUTERnEN):
>> {
>> uint64_t irouter;
>> + unsigned int offset, virq;
>>
>> if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
>> - rank = vgic_rank_offset(v, 64, gicd_reg - GICD_IROUTER,
>> - DABT_DOUBLE_WORD);
>> + if ( gicd_reg >= GICD_IROUTERnE ) {
>
> Braces should go into separate line
>
>> + offset = gicd_reg - GICD_IROUTERnE;
>> + rank = vgic_ext_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
>> + } else {
>
> ... so "else" also should be on a separate line
>
>> + offset = gicd_reg - GICD_IROUTER;
>> + rank = vgic_rank_offset(v, 64, offset, DABT_DOUBLE_WORD);
>> + }
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> - irouter = vgic_fetch_irouter(rank, gicd_reg - GICD_IROUTER);
>> + irouter = vgic_fetch_irouter(rank, offset);
>> vreg_reg64_update(&irouter, r, info);
>> - vgic_store_irouter(v->domain, rank, gicd_reg - GICD_IROUTER, irouter);
>> + if ( gicd_reg >= GICD_IROUTERnE )
>> + virq = ESPI_IDX2INTID(offset / NR_BYTES_PER_IROUTER);
>> + else
>> + virq = offset / NR_BYTES_PER_IROUTER;
>> + vgic_store_irouter(v->domain, rank, virq, irouter);
>> vgic_unlock_rank(v, rank, flags);
>> return 1;
>> }
>>
>> - case VRANGE32(0x7FE0, 0xBFFC):
>> + case VRANGE32(0x3700, 0x60FC):
>> + case VRANGE32(0xA004, 0xBFFC):
>> goto write_reserved;
>>
>> case VRANGE32(0xC000, 0xFFCC):
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index c9b9528c66..27ffdf316c 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -193,6 +193,18 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
>> }
>>
>> #ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * The function behavior is the same as for regular SPIs (vgic_rank_offset),
>> + * but it operates with extended SPI ranks.
>> + */
>> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
>> + unsigned int n, unsigned int s)
>> +{
>> + unsigned int rank = REG_RANK_NR(b, (n >> s));
>> +
>> + return vgic_get_rank(v, rank + EXT_RANK_MIN);
>> +}
>> +
>> static unsigned int vgic_num_spi_lines(struct domain *d)
>> {
>> return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
>> @@ -241,6 +253,17 @@ struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>> {
>> return NULL;
>> }
>> +
>> +/*
>> + * It is expected that, in the case of CONFIG_GICV3_ESPI=n, the function will
>> + * return NULL. In this scenario, mmio_read/mmio_write will be treated as
>> + * RAZ/WI, as expected.
>> + */
>> +struct vgic_irq_rank *vgic_ext_rank_offset(struct vcpu *v, unsigned int b,
>> + unsigned int n, unsigned int s)
>> +{
>> + return NULL;
>> +}
>> #endif
>>
>> static unsigned int vgic_num_alloc_irqs(struct domain *d)
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-09-01 16:16 ` Julien Grall
@ 2025-09-02 6:56 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-02 6:56 UTC (permalink / raw)
To: Julien Grall, Oleksandr Tyshchenko
Cc: Volodymyr Babchuk, xen-devel@lists.xenproject.org,
Stefano Stabellini, Bertrand Marquis, Michal Orzel
Hello Julien, Volodymyr and Oleksandr,
On 01.09.25 19:16, Julien Grall wrote:
> Hi,
>
> On 01/09/2025 15:42, Leonid Komarianskyi wrote:
>>>> Taking into account that with CONFIG_GICV3_ESPI=n we should never have
>>>> "irq" in eSPI range, do you really need this #ifdef? I think that
>>>> ASSERT_UNREACHABLE in espi_to_desc() is sufficient guard.
>>>>
>>>> Also, IRQ line number belongs to eSPI range regardless of
>>>> CONFIG_GICV3_ESPI,
>>>> value, so in my opinion is_espi() should always return correct value
>>>> for
>>>> a given "irq".
>>>
>>> ... I agree with Volodymyr's suggestion for is_espi() to always
>>> return
>>> correct value for a given "irq".
>>>
>>>
>>
>> I will fix that in V6.
>
> I am not sure about this. If is_espi() is not returning false with
> CONFIG_GICV3_EPSI, then the compiler would not be able to optimize code
> like:
>
> if (is_espi(...)) {
> return espi_to_desc(irq);
> }
>
> return &irq_desc[irq-NR_LOCAL_IRQS];
>
> irq_to_desc() is called fairly often, so I would like to keep the code
> fairly optimized. An alternative would be to use #ifdef CONFIG_*. I
> don't like it, but it could be a compromise if Oleksandr and Volodymyr
> wants to push to remove #ifdef from CONFIG_IS_ESPI.
>
> Cheers,
>
I am just thinking about a possible compromise between writing logical
code where the function name reflects its actual functionality and
allowing for compiler optimization. Perhaps it would be better to keep
the #ifdef but rename the function to `is_valid_espi()` or something
similar.
In that case, I think there would be less confusion, as it seems
reasonable for a function with such a name to return false when
CONFIG_GICV3_ESPI=n, and also the compiler would be able to optimize the
code.
What do you think about this approach?
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-09-01 18:00 ` Leonid Komarianskyi
@ 2025-09-02 8:16 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 49+ messages in thread
From: Oleksandr Tyshchenko @ 2025-09-02 8:16 UTC (permalink / raw)
To: Leonid Komarianskyi
Cc: xen-devel@lists.xenproject.org, Volodymyr Babchuk,
Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel
On 01.09.25 21:00, Leonid Komarianskyi wrote:
> Hello Oleksandr,
Hello Leonid
>
> Thank you for your review.
>
> On 31.08.25 18:58, Oleksandr Tyshchenko wrote:
>>
>>
>> On 29.08.25 23:45, Volodymyr Babchuk wrote:
>>
>> Hello Leonid, Volodymyr
>>
>>>
>>> Hi Leonid,
>>>
>>> Leonid Komarianskyi <Leonid_Komarianskyi@epam.com> writes:
>>>
>>>> This change introduces resource management in the VGIC to handle
>>>> extended SPIs introduced in GICv3.1. The pending_irqs and
>>>> allocated_irqs arrays are resized to support the required
>>>> number of eSPIs, based on what is supported by the hardware and
>>>> requested by the guest. A new field, ext_shared_irqs, is added
>>>> to the VGIC structure to store information about eSPIs, similar
>>>> to how shared_irqs is used for regular SPIs.
>>>>
>>>> Since the eSPI range starts at INTID 4096 and INTIDs between 1025
>>>> and 4095 are reserved, helper macros are introduced to simplify the
>>>> transformation of indices and to enable easier access to eSPI-specific
>>>> resources. These changes prepare the VGIC for processing eSPIs as
>>>> required by future functionality.
>>>>
>>>> The initialization and deinitialization paths for vgic have been updated
>>>> to allocate and free these resources appropriately. Additionally,
>>>> updated handling of INTIDs greater than 1024, passed from the toolstack
>>>> during domain creation, and verification logic ensures only valid SPI or
>>>> eSPI INTIDs are used.
>>>>
>>>> The existing SPI behavior remains unaffected when guests do not request
>>>> eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
>>>> option is disabled.
>>>>
>>>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>>>
>>>> ---
>>>> Changes in V5:
>>>> - removed the has_espi field because it can be determined by checking
>>>> whether domain->arch.vgic.nr_espis is zero or not
>>>> - since vgic_ext_rank_offset is not used in this patch, it has been
>>>> moved to the appropriate patch in the patch series, which implements
>>>> vgic eSPI registers emulation and requires this function
>>>> - removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
>>>> and code duplication in further changes
>>>> - fixed minor nit: used %pd for printing domain with its ID
>>
>> @Leonid, thanks for optimizing the series, now it looks much better (the
>> number of #ifdef-s is reduced, code is reused).
>>
>>
>
> I am doing my best, and you and the other reviewers are helping me
> improve the code. Thank you for that!
>
>>>>
>>>> Changes in V4:
>>>> - added has_espi field to simplify determining whether a domain is able
>>>> to operate with eSPI
>>>> - fixed formatting issues and misspellings
>>>>
>>>> Changes in V3:
>>>> - fixed formatting for lines with more than 80 symbols
>>>> - introduced helper functions to be able to use stubs in case of
>>>> CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
>>>> #ifdefs
>>>> - fixed checks for nr_spis in domain_vgic_init
>>>> - updated comment about nr_spis adjustments with dom0less mention
>>>> - moved comment with additional explanations before checks
>>>> - used unsigned int for indexes since they cannot be negative
>>>> - removed unnecessary parentheses
>>>> - move vgic_ext_rank_offset to the below ifdef guard, to reduce the
>>>> number of ifdefs
>>>>
>>>> Changes in V2:
>>>> - change is_espi_rank to is_valid_espi_rank to verify whether the array
>>>> element ext_shared_irqs exists. The previous version, is_espi_rank,
>>>> only checked if the rank index was less than the maximum possible
>>>> eSPI
>>>> rank index, but this could potentially result in accessing a
>>>> non-existing array element. To address this, is_valid_espi_rank was
>>>> introduced, which ensures that the required eSPI rank exists
>>>> - move gic_number_espis to
>>>> xen/arm: gicv3: implement handling of GICv3.1 eSPI
>>>> - update vgic_is_valid_irq checks to allow operating with eSPIs
>>>> - remove redundant newline in vgic_allocate_virq
>>>> ---
>>>> xen/arch/arm/include/asm/vgic.h | 12 ++
>>>> xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
>>>> 2 files changed, 208 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/
>>>> asm/vgic.h
>>>> index 3e7cbbb196..912d5b7694 100644
>>>> --- a/xen/arch/arm/include/asm/vgic.h
>>>> +++ b/xen/arch/arm/include/asm/vgic.h
>>>> @@ -146,6 +146,10 @@ struct vgic_dist {
>>>> int nr_spis; /* Number of SPIs */
>>>> unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
>>>> struct vgic_irq_rank *shared_irqs;
>>>> +#ifdef CONFIG_GICV3_ESPI
>>>> + struct vgic_irq_rank *ext_shared_irqs;
>>>> + int nr_espis; /* Number of extended SPIs */
>>
>> It seems you have agreed (V4) that nr_espis could not be negative.
>>
>
> I appologize for that, I missed this change. I will fix it in V6.
everything is fine, no need to apologize
>
>>>> +#endif
>>>> /*
>>>> * SPIs are domain global, SGIs and PPIs are per-VCPU and
>>>> stored in
>>>> * struct arch_vcpu.
>>>> @@ -243,6 +247,14 @@ struct vgic_ops {
>>>> /* Number of ranks of interrupt registers for a domain */
>>>> #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
>>>> +#ifdef CONFIG_GICV3_ESPI
>>>> +#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
>>>> +#endif
>>>> +#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
>>>> +#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
>>>> +#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
>>>> +#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
>>>> +
>>>> #define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
>>>> #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
>>>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>>>> index 2bbf4d99aa..c9b9528c66 100644
>>>> --- a/xen/arch/arm/vgic.c
>>>> +++ b/xen/arch/arm/vgic.c
>>>> @@ -27,9 +27,82 @@
>>>> bool vgic_is_valid_line(struct domain *d, unsigned int virq)
>>>> {
>>>> +#ifdef CONFIG_GICV3_ESPI
>>>> + if ( virq >= ESPI_BASE_INTID &&
>>>> + virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
>>>> + return true;
>>>> +#endif
>>>> +
>>>> return virq < vgic_num_irqs(d);
>>>> }
>>>> +#ifdef CONFIG_GICV3_ESPI
>>>> +/*
>>>> + * Since eSPI indexes start from 4096 and numbers from 1024 to
>>>> + * 4095 are forbidden, we need to check both lower and upper
>>>> + * limits for ranks.
>>>> + */
>>>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int
>>>> rank)
>>>> +{
>>>> + return rank >= EXT_RANK_MIN &&
>>>> + EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
>>>> +}
>>>> +
>>>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>>>> + unsigned int
>>>> rank)
>>>> +{
>>>> + return &v->domain-
>>>>> arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
>>>> +}
>>>> +
>>>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned
>>>> int virq)
>>>> +{
>>>> + return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
>>>> + d->arch.vgic.allocated_irqs);
>>>> +}
>>>> +
>>>> +static void arch_move_espis(struct vcpu *v)
>>>
>>> I don't need you need a copy of arch_move_irqs(). Se below for more info.
>>>
>>>> +{
>>>> + const cpumask_t *cpu_mask = cpumask_of(v->processor);
>>>> + struct domain *d = v->domain;
>>>> + struct pending_irq *p;
>>>> + struct vcpu *v_target;
>>>> + unsigned int i;
>>>> +
>>>> + for ( i = ESPI_BASE_INTID;
>>>> + i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
>>>> + {
>>>> + v_target = vgic_get_target_vcpu(v, i);
>>>> + p = irq_to_pending(v_target, i);
>>>> +
>>>> + if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p-
>>>>> status) )
>>>> + irq_set_affinity(p->desc, cpu_mask);
>>>> + }
>>>> +}
>>>> +#else
>>>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int
>>>> rank)
>>>> +{
>>>> + return false;
>>>> +}
>>>> +
>>>> +/*
>>>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>>>> + * because in this case, is_valid_espi_rank will always return false.
>>>> + */
>>>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>>>> + unsigned int
>>>> rank)
>>>> +{
>>>> + ASSERT_UNREACHABLE();
>>>> + return NULL;
>>>> +}
>>>> +
>>>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned
>>>> int virq)
>>>> +{
>>>> + return false;
>>>> +}
>>>> +
>>>> +static void arch_move_espis(struct vcpu *v) { }
>>>> +#endif
>>>> +
>>>> static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
>>>> unsigned int rank)
>>>> {
>>>> @@ -37,6 +110,8 @@ static inline struct vgic_irq_rank
>>>> *vgic_get_rank(struct vcpu *v,
>>>> return v->arch.vgic.private_irqs;
>>>> else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
>>>> return &v->domain->arch.vgic.shared_irqs[rank - 1];
>>>> + else if ( is_valid_espi_rank(v->domain, rank) )
>>>> + return vgic_get_espi_rank(v, rank);
>>>> else
>>>> return NULL;
>>>> }
>>>> @@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d,
>>>> unsigned int *mmio_count)
>>>> return 0;
>>>> }
>>>> +#ifdef CONFIG_GICV3_ESPI
>>>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>>>> +{
>>>> + return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
>>>> +}
>>>> +
>>>> +static int init_vgic_espi(struct domain *d)
>>>> +{
>>>> + unsigned int i, idx;
>>>> +
>>>> + if ( d->arch.vgic.nr_espis == 0 )
>>>> + return 0;
>>>> +
>>>> + d->arch.vgic.ext_shared_irqs =
>>>> + xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
>>>> + if ( d->arch.vgic.ext_shared_irqs == NULL )
>>>> + return -ENOMEM;
>>>> +
>>>> + for ( i = d->arch.vgic.nr_spis, idx = 0;
>>>> + i < vgic_num_spi_lines(d); i++, idx++ )
>>>> + vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
>>>> + ESPI_IDX2INTID(idx));
>>>> +
>>>> + for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
>>>> + vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>>>
>>> I know that I should made this observation in previous version, but I
>>> didn't, sorry for that. Anyways, I don't think that this is a good idea
>>> to introduce this function and vgic_reserve_espi_virq(), as well as
>>> arch_move_espis(), actually, because in each case this is a code
>>> duplication, which is not good.
>>>
>>> I think that instead you need to introduce a pair of helpers that will
>>> map any (e)SPI number to pending_irq[]/allocate_irqs index and back.
>>>
>>> somethink like
>>>
>>> static inline unsigned virq_to_index(int virq)
>>> {
>>> if (is_espi(virq))
>>> return ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
>>> return virq;
>>> }
>>>
>>> See below for examples.
>>>
>>>> +{
>>>> + irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
>>>> + return &d->arch.vgic.pending_irqs[irq];
>>>> +}
>>>> +#else
>>>> +static unsigned int init_vgic_espi(struct domain *d)
>>>> +{
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>>>> +{
>>>> + return d->arch.vgic.nr_spis;
>>>> +}
>>>> +
>>>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>>>> +{
>>>> + return NULL;
>>>> +}
>>>> +#endif
>>>> +
>>>> +static unsigned int vgic_num_alloc_irqs(struct domain *d)
>>>> +{
>>>> + return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
>>>> +}
>>
>> I do not know where it would be better to put a comment related to non-
>> visible in the patch context route_irq_to_guest(), but put it here.
>>
>> I am afraid, the vgic_num_irqs(d) printed in the following error message
>> is not entirely correct with your changes:
>>
>> route_irq_to_guest():
>>
>> ...
>>
>> if ( !vgic_is_valid_line(d, virq) )
>> {
>> printk(XENLOG_G_ERR
>> "the vIRQ number %u is too high for domain %u (max =
>> %u)\n",
>> irq, d->domain_id, vgic_num_irqs(d));
>> return -EINVAL;
>> }
>>
>>
>
> Would it be okay to change the error message to something like:
> "invalid vIRQ number %u for domain %pd\n"
>
> I understand that it is a more generic error message, but I think it
> might become overly complicated if I add more information stating that
> the IRQ should be within the range 0...vgic_num_irqs(d) or
> 4096...ESPI_IDX2INTID(d->arch.vgic.nr_espis).
>
I see, so it would not be precise to just let's say print
vgic_num_irqs(d) or ESPI_IDX2INTID(d->arch.vgic.nr_espis) as "(max =
%u)" since the vIRQ range is not contiguous.
Well, removing extra information (max) that could help the user to
figure out what was wrong is not ideal, but if you think it would
overcomplicate things, then I (not a maintainer of this code) would be
okay with the proposed simplified error message.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-09-01 16:11 ` Julien Grall
@ 2025-09-02 8:56 ` Leonid Komarianskyi
2025-09-02 13:11 ` Julien Grall
0 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-02 8:56 UTC (permalink / raw)
To: Julien Grall, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Julien,
Thank you for your review.
On 01.09.25 19:11, Julien Grall wrote:
> Hi Leonid,
>
> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>> Currently, Xen does not support eSPI interrupts, leading
>> to a data abort when such interrupts are defined in the DTS.
>>
>> This patch introduces a separate array to initialize up to
>> 1024 interrupt descriptors in the eSPI range and adds the
>> necessary defines and helper function. These changes lay the
>> groundwork for future implementation of full eSPI interrupt
>> support. As this GICv3.1 feature is not required by all vendors,
>> all changes are guarded by ifdefs, depending on the corresponding
>> Kconfig option.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>
>> ---
>> Changes in V5:
>> - no functional changes introduced by this version compared with V4, only
>> minor fixes and removal of ifdefs for macroses
>> - added TODO comment, suggested by Oleksandr Tyshchenko
>> - changed int to unsigned int for irqs
>> - removed ifdefs for eSPI-specific defines and macros to reduce the
>> number of ifdefs and code duplication in further changes
>> - removed reviewed-by as moving defines from ifdefs requires additional
>> confirmation from reviewers
>>
>> Changes in V4:
>> - removed redundant line with 'default n' in Kconfig, as it is disabled
>> by default, without explicit specification
>> - added reviewed-by from Volodymyr Babchuk
>>
>> Changes in V3:
>> - introduced a new define NR_ESPI_IRQS to avoid confusion, like in the
>> case of using NR_IRQS for espi_desc array
>> - implemented helper functions espi_to_desc and init_espi_data to make
>> it possible to add stubs with the same name, and as a result, reduce
>> the number of #ifdefs
>> - disable CONFIG_GICV3_ESPI default value to n
>>
>> Changes in V2:
>> - use (ESPI_MAX_INTID + 1) instead of (ESPI_BASE_INTID + NR_IRQS)
>> - remove unnecessary comment for nr_irqs initialization
>> ---
>> xen/arch/arm/Kconfig | 8 +++++
>> xen/arch/arm/include/asm/irq.h | 24 +++++++++++++++
>> xen/arch/arm/irq.c | 56 +++++++++++++++++++++++++++++++++-
>> 3 files changed, 87 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 17df147b25..43b05533b1 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -135,6 +135,14 @@ config GICV3
>> Driver for the ARM Generic Interrupt Controller v3.
>> If unsure, use the default setting.
>> +config GICV3_ESPI
>> + bool "Extended SPI range support"
>> + depends on GICV3 && !NEW_VGIC
>> + help
>> + Allow Xen and domains to use interrupt numbers from the
>> extended SPI
>> + range, from 4096 to 5119. This feature is introduced in GICv3.1
>> + architecture.
>> +
>> config HAS_ITS
>> bool "GICv3 ITS MSI controller support (UNSUPPORTED)" if
>> UNSUPPORTED
>> depends on GICV3 && !NEW_VGIC && !ARM_32
>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/
>> asm/irq.h
>> index 5bc6475eb4..4443799648 100644
>> --- a/xen/arch/arm/include/asm/irq.h
>> +++ b/xen/arch/arm/include/asm/irq.h
>> @@ -32,6 +32,13 @@ struct arch_irq_desc {
>> #define SPI_MAX_INTID 1019
>> #define LPI_OFFSET 8192
>> +#define ESPI_BASE_INTID 4096
>> +#define ESPI_MAX_INTID 5119
>> +#define NR_ESPI_IRQS 1024
>> +
>> +#define ESPI_INTID2IDX(intid) ((intid) - ESPI_BASE_INTID)
>> +#define ESPI_IDX2INTID(idx) ((idx) + ESPI_BASE_INTID)
>
> NIT: I would consider adding sanity check (i.e. ASSERT()) to confirm
> that both ``intid`` and ``idx`` are within the bounds.
>
Okay, I will add sanity check with ASSERTs in V6 (similar to
GNTPIN_incr2oflow_mask):
#define ESPI_INTID2IDX(intid) \
({ \
ASSERT(((intid) >= ESPI_BASE_INTID) && \
((intid) <= ESPI_MAX_INTID)); \
((intid) - ESPI_BASE_INTID); \
})
#define ESPI_IDX2INTID(idx) \
({ \
ASSERT(((idx) >= 0) && \
((idx) <= NR_ESPI_IRQS)); \
((idx) + ESPI_BASE_INTID); \
})
>> +
>> /* LPIs are always numbered starting at 8192, so 0 is a good invalid
>> case. */
>> #define INVALID_LPI 0
>> @@ -39,7 +46,15 @@ struct arch_irq_desc {
>> #define INVALID_IRQ 1023
>> extern const unsigned int nr_irqs;
>> +#ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * This will also cover the eSPI range, as some critical devices
>> + * for booting Xen (e.g., serial) may use this type of interrupts.
>> + */
>
> Reading this again, I still don't quite understand why we are mentioning
> Xen devices. Looking at the code, for Arm, we only seem to use
> nr_static_irqs to configure nr_pirqs and XSM. Both are ony used by domains.
>
> So I think this needs to be clarified.
>
Sure, I will update the comment.
>> +#define nr_static_irqs (ESPI_MAX_INTID + 1)
>> +#else
>> #define nr_static_irqs NR_IRQS
>> +#endif
>> struct irq_desc;
>> struct irqaction;
>> @@ -55,6 +70,15 @@ static inline bool is_lpi(unsigned int irq)
>> return irq >= LPI_OFFSET;
>> }
>> +static inline bool is_espi(unsigned int irq)
>> +{
>> +#ifdef CONFIG_GICV3_ESPI
>> + return (irq >= ESPI_BASE_INTID && irq <= ESPI_MAX_INTID);
>> +#else
>> + return false;
>> +#endif
>> +}
>> +
>> #define domain_pirq_to_irq(d, pirq) (pirq)
>> bool is_assignable_irq(unsigned int irq);
>> diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
>> index b8eccfc924..61c915c3f9 100644
>> --- a/xen/arch/arm/irq.c
>> +++ b/xen/arch/arm/irq.c
>> @@ -19,7 +19,11 @@
>> #include <asm/gic.h>
>> #incl#ude <asm/vgic.h>
>> +#ifdef CONFIG_GICV3_ESPI
>> +const unsigned int nr_irqs = ESPI_MAX_INTID + 1;
>> +#else
>> const unsigned int nr_irqs = NR_IRQS;
>> +#endif
>
> NIT: I think you can use:
>
> const unsigned int nr_irqs = IS_ENABLED(CONFIG_GICV3_ESPI)?
> (ESPI_MAX_INTID + 1) : NR_IRQS;
>
I will update it to proposed above in V6, thanks.
> That said, I think we need to rethink about the use of nr_irqs and
> nr_static_irqs because they don't entirely make sense for Arm as we
> don't support PIRQs.
>
> I would at least try to get rid of one of the variable (maybe nr_irqs)
> if not both.
>
> This could be done as a follow-up.
>
>> static unsigned int local_irqs_type[NR_LOCAL_IRQS];
>> static DEFINE_SPINLOCK(local_irqs_type_lock);
>> @@ -46,6 +50,53 @@ void irq_end_none(struct irq_desc *irq)
>> }
>> static irq_desc_t irq_desc[NR_IRQS - NR_LOCAL_IRQS];
>> +#ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * TODO: Consider allocating an array dynamically if
>> + * there is a need to enable GICV3_ESPI by default.
>> + */
>
> I know this was suggested by Oleksandr, however most likely distro will
> want to enable this feature so they can be booted on a wide range of
> platform. So I think "if there is a need to enable GICV3_ESPI by
> default" should be removed.
>
I will remove the second part of comment in V6.
> > +static irq_desc_t espi_desc[NR_ESPI_IRQS];> +
>> +static struct irq_desc *espi_to_desc(unsigned int irq)
>> +{
>> + return &espi_desc[ESPI_INTID2IDX(irq)];
>> +}
>> +
>> +static int __init init_espi_data(void)
>> +{
>> + unsigned int irq;
>> +
>> + for ( irq = ESPI_BASE_INTID; irq <= ESPI_MAX_INTID; irq++ )
>> + {
>> + struct irq_desc *desc = irq_to_desc(irq);
>> + int rc = init_one_irq_desc(desc);
>> +
>> + if ( rc )
>> + return rc;
>> +
>> + desc->irq = irq;
>> + desc->action = NULL;
>> + }
>> +
>> + return 0;
>> +}
>> +#else
>> +/*
>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>> + * because in this case, is_espi will always return false.
>> + */
>
> Is this is not mean to be called, then can we only define the prototype
> like we do for __bad_atomic_read()?
>
This would only be possible if is_espi() contains #ifdefs (when
CONFIG_GICV3_ESPI=n, it returns false unconditionally), but we need to
agree on this in the parallel thread discussion related to this patch.
If it is feasible, I will define it as a prototype in v6.
>> +static struct irq_desc *espi_to_desc(unsigned int irq)
>> +{
>> + ASSERT_UNREACHABLE();
>> + return NULL;
>> +}
>> +
>> +static int __init init_espi_data(void)
>> +{
>> + return 0;
>> +}
>> +#endif
>> +
>> static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
>> struct irq_desc *__irq_to_desc(unsigned int irq)
>> @@ -53,6 +104,9 @@ struct irq_desc *__irq_to_desc(unsigned int irq)
>> if ( irq < NR_LOCAL_IRQS )
>> return &this_cpu(local_irq_desc)[irq];
>> + if ( is_espi(irq) )
>> + return espi_to_desc(irq);
>> +
>> return &irq_desc[irq-NR_LOCAL_IRQS];
>> }
>> @@ -79,7 +133,7 @@ static int __init init_irq_data(void)
>> desc->action = NULL;
>> }
>> - return 0;
>> + return init_espi_data();
>> }
>> static int init_local_irq_data(unsigned int cpu)
>
> Cheers,
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range
2025-09-02 8:56 ` Leonid Komarianskyi
@ 2025-09-02 13:11 ` Julien Grall
0 siblings, 0 replies; 49+ messages in thread
From: Julien Grall @ 2025-09-02 13:11 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi,
On 02/09/2025 09:56, Leonid Komarianskyi wrote:
>>> diff --git a/xen/arch/arm/include/asm/irq.h b/xen/arch/arm/include/
>>> asm/irq.h
>>> index 5bc6475eb4..4443799648 100644
>>> --- a/xen/arch/arm/include/asm/irq.h
>>> +++ b/xen/arch/arm/include/asm/irq.h
>>> @@ -32,6 +32,13 @@ struct arch_irq_desc {
>>> #define SPI_MAX_INTID 1019
>>> #define LPI_OFFSET 8192
>>> +#define ESPI_BASE_INTID 4096
>>> +#define ESPI_MAX_INTID 5119
>>> +#define NR_ESPI_IRQS 1024
>>> +
>>> +#define ESPI_INTID2IDX(intid) ((intid) - ESPI_BASE_INTID)
>>> +#define ESPI_IDX2INTID(idx) ((idx) + ESPI_BASE_INTID)
>>
>> NIT: I would consider adding sanity check (i.e. ASSERT()) to confirm
>> that both ``intid`` and ``idx`` are within the bounds.
>>
>
> Okay, I will add sanity check with ASSERTs in V6 (similar to
> GNTPIN_incr2oflow_mask):
>
> #define ESPI_INTID2IDX(intid) \
> ({ \
> ASSERT(((intid) >= ESPI_BASE_INTID) && \
> ((intid) <= ESPI_MAX_INTID)); \
> ((intid) - ESPI_BASE_INTID); \
> })
If you are using a macro, then you will need to stash "intid" in a local
variable. Otherwise, it would be evaluated multiple time.
The alternative is to use a static inline helper which is usually preferred.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 06/12] xen/arm/irq: allow eSPI processing in the gic_interrupt function
2025-08-29 16:06 ` [PATCH v5 06/12] xen/arm/irq: allow eSPI processing in the gic_interrupt function Leonid Komarianskyi
@ 2025-09-02 13:42 ` Julien Grall
0 siblings, 0 replies; 49+ messages in thread
From: Julien Grall @ 2025-09-02 13:42 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi,
On 29/08/2025 17:06, Leonid Komarianskyi wrote:
> The gic_interrupt function is the main handler for processing IRQs.
> Currently, due to restrictive checks, it does not process interrupt
> numbers greater than 1019. This patch updates the condition to allow
> the handling of interrupts from the eSPI range.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Acked-by: Julien Grall <jgrall@amazon.com>
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 07/12] xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing
2025-08-29 16:06 ` [PATCH v5 07/12] xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing Leonid Komarianskyi
@ 2025-09-02 13:53 ` Julien Grall
2025-09-02 17:06 ` Leonid Komarianskyi
0 siblings, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-02 13:53 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi,
On 29/08/2025 17:06, Leonid Komarianskyi wrote:
> To properly deactivate guest interrupts and allow them to be retriggered
> after the initial trigger, the LR needs to be updated. The current
Why guest specifically? Isn't the problem the same if a physical eSPI is
routed to dom0? IOW, shouldn't the explaination be:
"To properly deactivate physical eSPI routed to a domain and ..."
> implementation ignores interrupts outside the range specified by the mask
> 0x3FF, which only covers IRQ numbers up to 1023. To enable processing of
> eSPI interrupts, this patch updates the mask to 0x13FF.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>
> ---
> Changes in V5:
> - no changes
>
> Changes in V4:
> - added reviewed-by from Volodymyr Babchuk
>
> Changes in V3:
> - no changes
>
> Changes in V2:
> - remove unnecessary CONFIG_GICV3_ESPI ifdef guard
> ---
> xen/arch/arm/include/asm/gic_v3_defs.h | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/include/asm/gic_v3_defs.h
> index 3370b4cd52..e70c1a5675 100644
> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
> @@ -211,7 +211,7 @@
> #define ICH_LR_VIRTUAL_SHIFT 0
> #define ICH_LR_CPUID_MASK 0x7
> #define ICH_LR_CPUID_SHIFT 10
> -#define ICH_LR_PHYSICAL_MASK 0x3ff
> +#define ICH_LR_PHYSICAL_MASK 0x13ff
It took me a while to understand why we are using 0x13ff rather than
0x1fff. It is because eSPI range is 4096 - 5519. So in theory, it would
be ok to just add '0x1000'. But I think this is more confusion that it
is worth. So I would rather prefer if we use 0x1fff as this matches the
specification.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-08-29 16:06 ` [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs Leonid Komarianskyi
2025-08-29 20:45 ` Volodymyr Babchuk
@ 2025-09-02 14:13 ` Julien Grall
2025-09-02 20:24 ` Leonid Komarianskyi
1 sibling, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-02 14:13 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Leonid,
On 29/08/2025 17:06, Leonid Komarianskyi wrote:
> This change introduces resource management in the VGIC to handle
> extended SPIs introduced in GICv3.1. The pending_irqs and
> allocated_irqs arrays are resized to support the required
> number of eSPIs, based on what is supported by the hardware and
> requested by the guest. A new field, ext_shared_irqs, is added
> to the VGIC structure to store information about eSPIs, similar
> to how shared_irqs is used for regular SPIs.
>
> Since the eSPI range starts at INTID 4096 and INTIDs between 1025
> and 4095 are reserved, helper macros are introduced to simplify the
> transformation of indices and to enable easier access to eSPI-specific
> resources. These changes prepare the VGIC for processing eSPIs as
> required by future functionality.
>
> The initialization and deinitialization paths for vgic have been updated
> to allocate and free these resources appropriately. Additionally,
> updated handling of INTIDs greater than 1024, passed from the toolstack
> during domain creation, and verification logic ensures only valid SPI or
> eSPI INTIDs are used.
>
> The existing SPI behavior remains unaffected when guests do not request
> eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
> option is disabled.
>
> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>
> ---
> Changes in V5:
> - removed the has_espi field because it can be determined by checking
> whether domain->arch.vgic.nr_espis is zero or not
> - since vgic_ext_rank_offset is not used in this patch, it has been
> moved to the appropriate patch in the patch series, which implements
> vgic eSPI registers emulation and requires this function
> - removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
> and code duplication in further changes
> - fixed minor nit: used %pd for printing domain with its ID
>
> Changes in V4:
> - added has_espi field to simplify determining whether a domain is able
> to operate with eSPI
> - fixed formatting issues and misspellings
>
> Changes in V3:
> - fixed formatting for lines with more than 80 symbols
> - introduced helper functions to be able to use stubs in case of
> CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
> #ifdefs
> - fixed checks for nr_spis in domain_vgic_init
> - updated comment about nr_spis adjustments with dom0less mention
> - moved comment with additional explanations before checks
> - used unsigned int for indexes since they cannot be negative
> - removed unnecessary parentheses
> - move vgic_ext_rank_offset to the below ifdef guard, to reduce the
> number of ifdefs
>
> Changes in V2:
> - change is_espi_rank to is_valid_espi_rank to verify whether the array
> element ext_shared_irqs exists. The previous version, is_espi_rank,
> only checked if the rank index was less than the maximum possible eSPI
> rank index, but this could potentially result in accessing a
> non-existing array element. To address this, is_valid_espi_rank was
> introduced, which ensures that the required eSPI rank exists
> - move gic_number_espis to
> xen/arm: gicv3: implement handling of GICv3.1 eSPI
> - update vgic_is_valid_irq checks to allow operating with eSPIs
> - remove redundant newline in vgic_allocate_virq
> ---
> xen/arch/arm/include/asm/vgic.h | 12 ++
> xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
> 2 files changed, 208 insertions(+), 3 deletions(-)
>
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 3e7cbbb196..912d5b7694 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -146,6 +146,10 @@ struct vgic_dist {
> int nr_spis; /* Number of SPIs */
> unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
> struct vgic_irq_rank *shared_irqs;
> +#ifdef CONFIG_GICV3_ESPI
> + struct vgic_irq_rank *ext_shared_irqs;
> + int nr_espis; /* Number of extended SPIs */
For new code, we prefer using "unsigned int" when the value is meant to
be >= 0.
> +#endif
> /*
> * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
> * struct arch_vcpu.
> @@ -243,6 +247,14 @@ struct vgic_ops {
> /* Number of ranks of interrupt registers for a domain */
> #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
>
> +#ifdef CONFIG_GICV3_ESPI
> +#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
> +#endif
> +#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
> +#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
> +#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
> +#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
For the 6 lines above, please add space before and after the operators.
> +
> #define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
> #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
>
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 2bbf4d99aa..c9b9528c66 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -27,9 +27,82 @@
>
> bool vgic_is_valid_line(struct domain *d, unsigned int virq)
> {
> +#ifdef CONFIG_GICV3_ESPI
> + if ( virq >= ESPI_BASE_INTID &&
> + virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
> + return true;
> +#endif
> +
> return virq < vgic_num_irqs(d);
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> +/*
> + * Since eSPI indexes start from 4096 and numbers from 1024 to
> + * 4095 are forbidden, we need to check both lower and upper
> + * limits for ranks.
> + */
> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
> +{
> + return rank >= EXT_RANK_MIN &&
> + EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
> +}
> +
> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
> + unsigned int rank)
> +{
> + return &v->domain->arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
> +}
> +
> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
> +{
> + return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
> + d->arch.vgic.allocated_irqs);
> +}
> +
> +static void arch_move_espis(struct vcpu *v)
> +{
> + const cpumask_t *cpu_mask = cpumask_of(v->processor);
> + struct domain *d = v->domain;
> + struct pending_irq *p;
> + struct vcpu *v_target;
cpu_mask, p and v_target only seems to be used within the loop. If
that's correct, then please move the definition within the loop.
> + unsigned int i;
> +
> + for ( i = ESPI_BASE_INTID;
> + i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
> + {
> + v_target = vgic_get_target_vcpu(v, i);
> + p = irq_to_pending(v_target, i);
> +
> + if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
> + irq_set_affinity(p->desc, cpu_mask);
> + }
> +}
> +#else
> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int rank)
> +{
> + return false;
> +}
> +
> +/*
> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
> + * because in this case, is_valid_espi_rank will always return false.
> + */
> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
> + unsigned int rank)
> +{
> + ASSERT_UNREACHABLE();
> + return NULL;
> +}
> +
> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned int virq)
> +{
> + return false;
> +}
> +
> +static void arch_move_espis(struct vcpu *v) { }
> +#endif
> +
> static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
> unsigned int rank)
> {
> @@ -37,6 +110,8 @@ static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
> return v->arch.vgic.private_irqs;
> else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
> return &v->domain->arch.vgic.shared_irqs[rank - 1];
> + else if ( is_valid_espi_rank(v->domain, rank) )
> + return vgic_get_espi_rank(v, rank);
> else
> return NULL;
> }
> @@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d, unsigned int *mmio_count)
> return 0;
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> +static unsigned int vgic_num_spi_lines(struct domain *d)
> +{
> + return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
> +}
> +
> +static int init_vgic_espi(struct domain *d)
> +{
> + unsigned int i, idx;
> +
> + if ( d->arch.vgic.nr_espis == 0 )
> + return 0;
> +
> + d->arch.vgic.ext_shared_irqs =
> + xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
> + if ( d->arch.vgic.ext_shared_irqs == NULL )
> + return -ENOMEM;
> +
> + for ( i = d->arch.vgic.nr_spis, idx = 0;
> + i < vgic_num_spi_lines(d); i++, idx++ )
> + vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
> + ESPI_IDX2INTID(idx));
> +
> + for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
> + vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
> +
> + return 0;
> +}
> +
> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
> +{
> + irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
> + return &d->arch.vgic.pending_irqs[irq];
> +}
> +#else
> +static unsigned int init_vgic_espi(struct domain *d)
> +{
> + return 0;
> +}
> +
> +static unsigned int vgic_num_spi_lines(struct domain *d)
> +{
> + return d->arch.vgic.nr_spis;
> +}
> +
> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
> +{
> + return NULL;
> +}
> +#endif
> +
> +static unsigned int vgic_num_alloc_irqs(struct domain *d)
> +{
> + return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
> +}
> +
> int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> {
> int i;
> @@ -131,6 +262,36 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> */
> nr_spis = ROUNDUP(nr_spis, 32);
>
> +#ifdef CONFIG_GICV3_ESPI
> + /*
> + * During domain creation, the dom0less DomUs code or toolstack specifies
> + * the maximum INTID, which is defined in the domain config subtracted by
> + * 32 to cover the local IRQs (please see the comment to VGIC_DEF_NR_SPIS).
> + * To compute the actual number of eSPI that will be usable for,
> + * add back 32.
> + */
> + if ( nr_spis + 32 > ESPI_IDX2INTID(NR_ESPI_IRQS) )
> + return -EINVAL;
> +
> + if ( nr_spis + 32 >= ESPI_BASE_INTID )
> + {
> + d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32, 1024U);
We should return an error rather than silently capping the value.
> + /* Verify if GIC HW can handle provided INTID */
> + if ( d->arch.vgic.nr_espis > gic_number_espis() )
> + return -EINVAL;
> + /*
> + * Set the maximum available number for regular
> + * SPI to pass the next check
> + */
I think it would be better to rework the code so the check is not called.
> + nr_spis = VGIC_DEF_NR_SPIS;
> + }
> + else
> + {
> + /* Domain will use the regular SPI range */
> + d->arch.vgic.nr_espis = 0;
> + }
> +#endif
> +
> /* Limit the number of virtual SPIs supported to (1020 - 32) = 988 */
> if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
> return -EINVAL;
> @@ -145,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> return -ENOMEM;
>
> d->arch.vgic.pending_irqs =
> - xzalloc_array(struct pending_irq, d->arch.vgic.nr_spis);
> + xzalloc_array(struct pending_irq, vgic_num_spi_lines(d));
> if ( d->arch.vgic.pending_irqs == NULL )
> return -ENOMEM;
>
> @@ -156,12 +317,16 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
> vgic_rank_init(&d->arch.vgic.shared_irqs[i], i + 1, 0);
>
> + ret = init_vgic_espi(d);
> + if ( ret )
> + return ret;
> +
> ret = d->arch.vgic.handler->domain_init(d);
> if ( ret )
> return ret;
>
> d->arch.vgic.allocated_irqs =
> - xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
> + xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_alloc_irqs(d)));
> if ( !d->arch.vgic.allocated_irqs )
> return -ENOMEM;
>
> @@ -195,9 +360,27 @@ void domain_vgic_free(struct domain *d)
> }
> }
>
> +#ifdef CONFIG_GICV3_ESPI
> + for ( i = 0; i < d->arch.vgic.nr_espis; i++ )
Now we are potentially doubling up the number of IRQs we have to
release. I am not entirely sure this is still ok to do it without any
preemption. Do you have any data?
> + {
> + struct pending_irq *p = spi_to_pending(d, ESPI_IDX2INTID(i));
> +
> + if ( p->desc )
> + {
> + ret = release_guest_irq(d, p->irq);
> + if ( ret )
> + dprintk(XENLOG_G_WARNING, "%pd: Failed to release virq %u ret = %d\n",
> + d, p->irq, ret);
> + }
> + }
> +#endif
> +
> if ( d->arch.vgic.handler )
> d->arch.vgic.handler->domain_free(d);
> xfree(d->arch.vgic.shared_irqs);
> +#ifdef CONFIG_GICV3_ESPI
> + xfree(d->arch.vgic.ext_shared_irqs);
> +#endif
> xfree(d->arch.vgic.pending_irqs);
> xfree(d->arch.vgic.allocated_irqs);
> }
> @@ -331,6 +514,8 @@ void arch_move_irqs(struct vcpu *v)
> if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p->status) )
> irq_set_affinity(p->desc, cpu_mask);
> }
> +
> + arch_move_espis(v);
> }
>
> void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
> @@ -538,6 +723,8 @@ struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
> n = &v->arch.vgic.pending_irqs[irq];
> else if ( is_lpi(irq) )
> n = v->domain->arch.vgic.handler->lpi_to_pending(v->domain, irq);
> + else if ( is_espi(irq) )
> + n = espi_to_pending(v->domain, irq);
> else
> n = &v->domain->arch.vgic.pending_irqs[irq - 32];
> return n;
> @@ -547,6 +734,9 @@ struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq)
> {
> ASSERT(irq >= NR_LOCAL_IRQS);
>
> + if ( is_espi(irq) )
> + return espi_to_pending(d, irq);
> +
> return &d->arch.vgic.pending_irqs[irq - 32];
> }
>
> @@ -668,6 +858,9 @@ bool vgic_reserve_virq(struct domain *d, unsigned int virq)
> if ( !vgic_is_valid_line(d, virq) )
> return false;
>
> + if ( is_espi(virq) )
> + return vgic_reserve_espi_virq(d, virq);
> +
> return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
> }
>
> @@ -685,7 +878,7 @@ int vgic_allocate_virq(struct domain *d, bool spi)
> else
> {
> first = 32;
> - end = vgic_num_irqs(d);
> + end = vgic_num_alloc_irqs(d);
> }
>
> /*
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs
2025-08-29 16:06 ` [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs Leonid Komarianskyi
@ 2025-09-02 16:42 ` Julien Grall
2025-09-02 17:15 ` Leonid Komarianskyi
0 siblings, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-02 16:42 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Oleksandr Tyshchenko
Hi Leonid,
On 29/08/2025 17:06, Leonid Komarianskyi wrote:
> The Dom0 and DomUs logic for the dom0less configuration in create_dom0() and
> arch_create_domUs() has been updated to account for extended SPIs when
> supported by the hardware and enabled with CONFIG_GICV3_ESPI.
Style: We don't commonly use past tense to describe the new behavior.
> xen/arch/arm/dom0less-build.c | 2 +-
> xen/arch/arm/domain_build.c | 2 +-
> xen/arch/arm/include/asm/vgic.h | 19 +++++++++++++++++++
> 3 files changed, 21 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
> index 69b9ea22ce..02d5559102 100644
> --- a/xen/arch/arm/dom0less-build.c
> +++ b/xen/arch/arm/dom0less-build.c
> @@ -285,7 +285,7 @@ void __init arch_create_domUs(struct dt_device_node *node,
> {
> int vpl011_virq = GUEST_VPL011_SPI;
>
> - d_cfg->arch.nr_spis = VGIC_DEF_NR_SPIS;
> + d_cfg->arch.nr_spis = vgic_def_nr_spis();
>
> /*
> * The VPL011 virq is GUEST_VPL011_SPI, unless direct-map is
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index d91a71acfd..39eea0be00 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -2054,7 +2054,7 @@ void __init create_dom0(void)
>
> /* The vGIC for DOM0 is exactly emulating the hardware GIC */
> dom0_cfg.arch.gic_version = XEN_DOMCTL_CONFIG_GIC_NATIVE;
> - dom0_cfg.arch.nr_spis = VGIC_DEF_NR_SPIS;
> + dom0_cfg.arch.nr_spis = vgic_def_nr_spis();
> dom0_cfg.arch.tee_type = tee_get_type();
> dom0_cfg.max_vcpus = dom0_max_vcpus();
>
> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
> index 912d5b7694..3aa22114ba 100644
> --- a/xen/arch/arm/include/asm/vgic.h
> +++ b/xen/arch/arm/include/asm/vgic.h
> @@ -347,6 +347,25 @@ extern void vgic_check_inflight_irqs_pending(struct vcpu *v,
> /* Default number of vGIC SPIs. 32 are substracted to cover local IRQs. */
> #define VGIC_DEF_NR_SPIS (min(gic_number_lines(), VGIC_MAX_IRQS) - 32)>
> +static inline unsigned int vgic_def_nr_spis(void)
> +{
> +#ifdef CONFIG_GICV3_ESPI
> + /*
> + * Check if the hardware supports extended SPIs (even if the appropriate
> + * config is set). If not, the common SPI range will be used. Otherwise
> + * return the maximum eSPI INTID, supported by HW GIC, subtracted by 32.
> + * For Dom0 and started at boot time DomUs we will add back this value
> + * during VGIC initialization. This ensures consistent handling for Dom0
> + * and other domains. For the regular SPI range interrupts in this case,
> + * the maximum value of VGIC_DEF_NR_SPIS will be used.
> + */
> + if ( gic_number_espis() > 0 )
> + return ESPI_BASE_INTID + min(gic_number_espis(), 1024U) - 32;
> +#endif
> +
> + return VGIC_DEF_NR_SPIS;
This is the only user of VGIC_DEF_NR_SPIS. Therefore, I would prefer if
we remove the define. This will avoid any confusion between the helper
and the define.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-08-29 16:06 ` [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers Leonid Komarianskyi
2025-08-29 20:12 ` Volodymyr Babchuk
2025-09-01 12:38 ` Oleksandr Tyshchenko
@ 2025-09-02 16:55 ` Julien Grall
2025-09-02 17:26 ` Leonid Komarianskyi
2 siblings, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-02 16:55 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Leonid,
On 29/08/2025 17:06, Leonid Komarianskyi wrote:
> @@ -782,46 +804,81 @@ static int __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
> {
> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
> /* We do not implement security extensions for guests, write ignore */
> goto write_ignore_32;
>
> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
> if ( dabt.size != DABT_WORD ) goto bad_width;
> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
> + if ( reg >= GICD_ISENABLERnE )
> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
> + DABT_WORD);
> + else
> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
While I understand the desire to try to avoid code duplication. I feel
this is making the code a lot more complicating and in fact riskier
because...
> if ( rank == NULL ) goto write_ignore;
> vgic_lock_rank(v, rank, flags);
> tr = rank->ienable;
> vreg_reg32_setbits(&rank->ienable, r, info);
> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
> + if ( reg >= GICD_ISENABLERnE )
> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
> + EXT_RANK_IDX2NUM(rank->index));
> + else
> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
... you now have to keep both "if" the same. Unless we can have a
version to avoid "ifs" everywhere (maybe using macros), I would rather
create a separate funciton to handle eSPIs.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 07/12] xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing
2025-09-02 13:53 ` Julien Grall
@ 2025-09-02 17:06 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-02 17:06 UTC (permalink / raw)
To: Julien Grall, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Julien,
Thank you for your review.
On 02.09.25 16:53, Julien Grall wrote:
> Hi,
>
> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>> To properly deactivate guest interrupts and allow them to be retriggered
>> after the initial trigger, the LR needs to be updated. The current
>
> Why guest specifically? Isn't the problem the same if a physical eSPI is
> routed to dom0? IOW, shouldn't the explaination be:
>
> "To properly deactivate physical eSPI routed to a domain and ..."
>
I will update the commit in V6.
>> implementation ignores interrupts outside the range specified by the mask
>> 0x3FF, which only covers IRQ numbers up to 1023. To enable processing of
>> eSPI interrupts, this patch updates the mask to 0x13FF.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
>>
>> ---
>> Changes in V5:
>> - no changes
>>
>> Changes in V4:
>> - added reviewed-by from Volodymyr Babchuk
>>
>> Changes in V3:
>> - no changes
>>
>> Changes in V2:
>> - remove unnecessary CONFIG_GICV3_ESPI ifdef guard
>> ---
>> xen/arch/arm/include/asm/gic_v3_defs.h | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/include/asm/gic_v3_defs.h b/xen/arch/arm/
>> include/asm/gic_v3_defs.h
>> index 3370b4cd52..e70c1a5675 100644
>> --- a/xen/arch/arm/include/asm/gic_v3_defs.h
>> +++ b/xen/arch/arm/include/asm/gic_v3_defs.h
>> @@ -211,7 +211,7 @@
>> #define ICH_LR_VIRTUAL_SHIFT 0
>> #define ICH_LR_CPUID_MASK 0x7
>> #define ICH_LR_CPUID_SHIFT 10
>> -#define ICH_LR_PHYSICAL_MASK 0x3ff
>> +#define ICH_LR_PHYSICAL_MASK 0x13ff
>
> It took me a while to understand why we are using 0x13ff rather than
> 0x1fff. It is because eSPI range is 4096 - 5519. So in theory, it would
> be ok to just add '0x1000'. But I think this is more confusion that it
> is worth. So I would rather prefer if we use 0x1fff as this matches the
> specification.
>
> Cheers,
>
Yes, I agree with that - it will be clearer, so I will update the mask
to 0x1fff.
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs
2025-09-02 16:42 ` Julien Grall
@ 2025-09-02 17:15 ` Leonid Komarianskyi
2025-09-02 19:59 ` Julien Grall
0 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-02 17:15 UTC (permalink / raw)
To: Julien Grall, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Oleksandr Tyshchenko
Hi Julien,
Thank you for your review.
On 02.09.25 19:42, Julien Grall wrote:
> Hi Leonid,
>
> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>> The Dom0 and DomUs logic for the dom0less configuration in
>> create_dom0() and
>> arch_create_domUs() has been updated to account for extended SPIs when
>> supported by the hardware and enabled with CONFIG_GICV3_ESPI.
>
> Style: We don't commonly use past tense to describe the new behavior.
I will update it in V6.
>> xen/arch/arm/dom0less-build.c | 2 +-
>> xen/arch/arm/domain_build.c | 2 +-
>> xen/arch/arm/include/asm/vgic.h | 19 +++++++++++++++++++
>> 3 files changed, 21 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-
>> build.c
>> index 69b9ea22ce..02d5559102 100644
>> --- a/xen/arch/arm/dom0less-build.c
>> +++ b/xen/arch/arm/dom0less-build.c
>> @@ -285,7 +285,7 @@ void __init arch_create_domUs(struct
>> dt_device_node *node,
>> {
>> int vpl011_virq = GUEST_VPL011_SPI;
>> - d_cfg->arch.nr_spis = VGIC_DEF_NR_SPIS;
>> + d_cfg->arch.nr_spis = vgic_def_nr_spis();
>> /*
>> * The VPL011 virq is GUEST_VPL011_SPI, unless direct-map is
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index d91a71acfd..39eea0be00 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -2054,7 +2054,7 @@ void __init create_dom0(void)
>> /* The vGIC for DOM0 is exactly emulating the hardware GIC */
>> dom0_cfg.arch.gic_version = XEN_DOMCTL_CONFIG_GIC_NATIVE;
>> - dom0_cfg.arch.nr_spis = VGIC_DEF_NR_SPIS;
>> + dom0_cfg.arch.nr_spis = vgic_def_nr_spis();
>> dom0_cfg.arch.tee_type = tee_get_type();
>> dom0_cfg.max_vcpus = dom0_max_vcpus();
>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/
>> asm/vgic.h
>> index 912d5b7694..3aa22114ba 100644
>> --- a/xen/arch/arm/include/asm/vgic.h
>> +++ b/xen/arch/arm/include/asm/vgic.h
>> @@ -347,6 +347,25 @@ extern void
>> vgic_check_inflight_irqs_pending(struct vcpu *v,
>> /* Default number of vGIC SPIs. 32 are substracted to cover local
>> IRQs. */
>> #define VGIC_DEF_NR_SPIS (min(gic_number_lines(), VGIC_MAX_IRQS) - 32)>
>> +static inline unsigned int vgic_def_nr_spis(void)
>> +{
>> +#ifdef CONFIG_GICV3_ESPI
>> + /*
>> + * Check if the hardware supports extended SPIs (even if the
>> appropriate
>> + * config is set). If not, the common SPI range will be used.
>> Otherwise
>> + * return the maximum eSPI INTID, supported by HW GIC, subtracted
>> by 32.
>> + * For Dom0 and started at boot time DomUs we will add back this
>> value
>> + * during VGIC initialization. This ensures consistent handling
>> for Dom0
>> + * and other domains. For the regular SPI range interrupts in
>> this case,
>> + * the maximum value of VGIC_DEF_NR_SPIS will be used.
>> + */
>> + if ( gic_number_espis() > 0 )
>> + return ESPI_BASE_INTID + min(gic_number_espis(), 1024U) - 32;
>> +#endif
>> +
>> + return VGIC_DEF_NR_SPIS;
>
> This is the only user of VGIC_DEF_NR_SPIS. Therefore, I would prefer if
> we remove the define. This will avoid any confusion between the helper
> and the define.
>
> Cheers,
>
Actually, we need this macro in the previous patch in the series:
[08/12] xen/arm: vgic: add resource management for extended SPIs
(domain_vgic_init):
if ( nr_spis + 32 >= ESPI_BASE_INTID )
{
d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32, 1024U);
/* Verify if GIC HW can handle provided INTID */
if ( d->arch.vgic.nr_espis > gic_number_espis() )
return -EINVAL;
/*
* Set the maximum available number for regular
* SPI to pass the next check
*/
nr_spis = VGIC_DEF_NR_SPIS;
}
I have seen your comment regarding reworking the checks, but in any
case, I still need this macro to assign the value to nr_spis, so I would
prefer to leave it.
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-09-02 16:55 ` Julien Grall
@ 2025-09-02 17:26 ` Leonid Komarianskyi
2025-09-02 20:20 ` Julien Grall
0 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-02 17:26 UTC (permalink / raw)
To: Julien Grall, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Julien,
Thank you for your review and suggestions.
On 02.09.25 19:55, Julien Grall wrote:
> Hi Leonid,
>
> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>> @@ -782,46 +804,81 @@ static int
>> __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>> {
>> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>> /* We do not implement security extensions for guests, write
>> ignore */
>> goto write_ignore_32;
>> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>> if ( dabt.size != DABT_WORD ) goto bad_width;
>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>> + if ( reg >= GICD_ISENABLERnE )
>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
>> + DABT_WORD);
>> + else
>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER,
>> DABT_WORD);
>
> While I understand the desire to try to avoid code duplication. I feel
> this is making the code a lot more complicating and in fact riskier
> because...
>
>> if ( rank == NULL ) goto write_ignore;
>> vgic_lock_rank(v, rank, flags);
>> tr = rank->ienable;
>> vreg_reg32_setbits(&rank->ienable, r, info);
>> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>> + if ( reg >= GICD_ISENABLERnE )
>> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
>> + EXT_RANK_IDX2NUM(rank->index));
>> + else
>> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>
> ... you now have to keep both "if" the same. Unless we can have a
> version to avoid "ifs" everywhere (maybe using macros), I would rather
> create a separate funciton to handle eSPIs.
>
> Cheers,
>
The main idea of V5 of this patch is to consolidate similar code and
keep the pairs of regular SPIs and their eSPI counterparts in the same
place to simplify any future modifications of these pairs. If
eSPI-specific registers are moved to a separate function, it would
result in code duplication. Additionally, I think it would be easier to
miss updating the code for one register of the SPI/eSPI pair while
modifying the code for the other..
So, I will revise the code and try to avoid ifs where possible.
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs
2025-09-02 17:15 ` Leonid Komarianskyi
@ 2025-09-02 19:59 ` Julien Grall
0 siblings, 0 replies; 49+ messages in thread
From: Julien Grall @ 2025-09-02 19:59 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Oleksandr Tyshchenko
On 02/09/2025 18:15, Leonid Komarianskyi wrote:
> Hi Julien,
Hi Leonid,
> On 02.09.25 19:42, Julien Grall wrote:
>> Hi Leonid,
>>
>> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>>> The Dom0 and DomUs logic for the dom0less configuration in
>>> create_dom0() and
>>> arch_create_domUs() has been updated to account for extended SPIs when
>>> supported by the hardware and enabled with CONFIG_GICV3_ESPI.
>>
>> Style: We don't commonly use past tense to describe the new behavior.
>
> I will update it in V6.
>
>>> xen/arch/arm/dom0less-build.c | 2 +-
>>> xen/arch/arm/domain_build.c | 2 +-
>>> xen/arch/arm/include/asm/vgic.h | 19 +++++++++++++++++++
>>> 3 files changed, 21 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-
>>> build.c
>>> index 69b9ea22ce..02d5559102 100644
>>> --- a/xen/arch/arm/dom0less-build.c
>>> +++ b/xen/arch/arm/dom0less-build.c
>>> @@ -285,7 +285,7 @@ void __init arch_create_domUs(struct
>>> dt_device_node *node,
>>> {
>>> int vpl011_virq = GUEST_VPL011_SPI;
>>> - d_cfg->arch.nr_spis = VGIC_DEF_NR_SPIS;
>>> + d_cfg->arch.nr_spis = vgic_def_nr_spis();
>>> /*
>>> * The VPL011 virq is GUEST_VPL011_SPI, unless direct-map is
>>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>> index d91a71acfd..39eea0be00 100644
>>> --- a/xen/arch/arm/domain_build.c
>>> +++ b/xen/arch/arm/domain_build.c
>>> @@ -2054,7 +2054,7 @@ void __init create_dom0(void)
>>> /* The vGIC for DOM0 is exactly emulating the hardware GIC */
>>> dom0_cfg.arch.gic_version = XEN_DOMCTL_CONFIG_GIC_NATIVE;
>>> - dom0_cfg.arch.nr_spis = VGIC_DEF_NR_SPIS;
>>> + dom0_cfg.arch.nr_spis = vgic_def_nr_spis();
>>> dom0_cfg.arch.tee_type = tee_get_type();
>>> dom0_cfg.max_vcpus = dom0_max_vcpus();
>>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/
>>> asm/vgic.h
>>> index 912d5b7694..3aa22114ba 100644
>>> --- a/xen/arch/arm/include/asm/vgic.h
>>> +++ b/xen/arch/arm/include/asm/vgic.h
>>> @@ -347,6 +347,25 @@ extern void
>>> vgic_check_inflight_irqs_pending(struct vcpu *v,
>>> /* Default number of vGIC SPIs. 32 are substracted to cover local
>>> IRQs. */
>>> #define VGIC_DEF_NR_SPIS (min(gic_number_lines(), VGIC_MAX_IRQS) - 32)>
>>> +static inline unsigned int vgic_def_nr_spis(void)
>>> +{
>>> +#ifdef CONFIG_GICV3_ESPI
>>> + /*
>>> + * Check if the hardware supports extended SPIs (even if the
>>> appropriate
>>> + * config is set). If not, the common SPI range will be used.
>>> Otherwise
>>> + * return the maximum eSPI INTID, supported by HW GIC, subtracted
>>> by 32.
>>> + * For Dom0 and started at boot time DomUs we will add back this
>>> value
>>> + * during VGIC initialization. This ensures consistent handling
>>> for Dom0
>>> + * and other domains. For the regular SPI range interrupts in
>>> this case,
>>> + * the maximum value of VGIC_DEF_NR_SPIS will be used.
>>> + */
>>> + if ( gic_number_espis() > 0 )
>>> + return ESPI_BASE_INTID + min(gic_number_espis(), 1024U) - 32;
>>> +#endif
>>> +
>>> + return VGIC_DEF_NR_SPIS;
>>
>> This is the only user of VGIC_DEF_NR_SPIS. Therefore, I would prefer if
>> we remove the define. This will avoid any confusion between the helper
>> and the define.
>>
>> Cheers,
>>
>
> Actually, we need this macro in the previous patch in the series:
> [08/12] xen/arm: vgic: add resource management for extended SPIs
> (domain_vgic_init):
>
> if ( nr_spis + 32 >= ESPI_BASE_INTID )
> {
> d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32, 1024U);
> /* Verify if GIC HW can handle provided INTID */
> if ( d->arch.vgic.nr_espis > gic_number_espis() )
> return -EINVAL;
> /*
> * Set the maximum available number for regular
> * SPI to pass the next check
> */
> nr_spis = VGIC_DEF_NR_SPIS;
> }
>
> I have seen your comment regarding reworking the checks, but in any
> case, I still need this macro to assign the value to nr_spis, so I would
> prefer to leave it.
Ah! I didn't spot the other use. Then let's keep it for now.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-09-02 17:26 ` Leonid Komarianskyi
@ 2025-09-02 20:20 ` Julien Grall
2025-09-02 20:55 ` Leonid Komarianskyi
0 siblings, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-02 20:20 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Leonid,
On 02/09/2025 18:26, Leonid Komarianskyi wrote:
> Hi Julien,
>
> Thank you for your review and suggestions.
>
> On 02.09.25 19:55, Julien Grall wrote:
>> Hi Leonid,
>>
>> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>>> @@ -782,46 +804,81 @@ static int
>>> __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>>> {
>>> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
>>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>>> /* We do not implement security extensions for guests, write
>>> ignore */
>>> goto write_ignore_32;
>>> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
>>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>>> if ( dabt.size != DABT_WORD ) goto bad_width;
>>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER, DABT_WORD);
>>> + if ( reg >= GICD_ISENABLERnE )
>>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
>>> + DABT_WORD);
>>> + else
>>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER,
>>> DABT_WORD);
>>
>> While I understand the desire to try to avoid code duplication. I feel
>> this is making the code a lot more complicating and in fact riskier
>> because...
>>
>>> if ( rank == NULL ) goto write_ignore;
>>> vgic_lock_rank(v, rank, flags);
>>> tr = rank->ienable;
>>> vreg_reg32_setbits(&rank->ienable, r, info);
>>> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>>> + if ( reg >= GICD_ISENABLERnE )
>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
>>> + EXT_RANK_IDX2NUM(rank->index));
>>> + else
>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>>
>> ... you now have to keep both "if" the same. Unless we can have a
>> version to avoid "ifs" everywhere (maybe using macros), I would rather
>> create a separate funciton to handle eSPIs.
>>
>> Cheers,
>>
>
> The main idea of V5 of this patch is to consolidate similar code and
> keep the pairs of regular SPIs and their eSPI counterparts in the same
> place to simplify any future modifications of these pairs. If
> eSPI-specific registers are moved to a separate function, it would
> result in code duplication. Additionally, I think it would be easier to
> miss updating the code for one register of the SPI/eSPI pair while
> modifying the code for the other..
I understand your reasoning, but in this case, we need to try to keep
the code as common as possible and reduce the number of ifs. Looking at
the patch again, we seem to often use "EXT_RANK_IDX2NUM(rank->index)"
and this is why we have the second "if", for instance here. I think it
would be preferable if "index" store the proper value.
An alternative would be to process the 3rd parameters separately.
The next big one to takle is:
if ( reg >= ... )
vgic_ext_rank_...(...)
else
vgic_rank(...)
Ideally I would like to provide an helper that can figure out whether
this is an eSPI or not. Looking at the pattern, I think we would
introduce a new helper which rather than taking a relative offset take
the reg offset, the base for SPIs and the base for eSPIs.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs
2025-09-02 14:13 ` Julien Grall
@ 2025-09-02 20:24 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-02 20:24 UTC (permalink / raw)
To: Julien Grall, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Julien,
Thank you for your review comments and your question.
On 02.09.25 17:13, Julien Grall wrote:
> Hi Leonid,
>
> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>> This change introduces resource management in the VGIC to handle
>> extended SPIs introduced in GICv3.1. The pending_irqs and
>> allocated_irqs arrays are resized to support the required
>> number of eSPIs, based on what is supported by the hardware and
>> requested by the guest. A new field, ext_shared_irqs, is added
>> to the VGIC structure to store information about eSPIs, similar
>> to how shared_irqs is used for regular SPIs.
>>
>> Since the eSPI range starts at INTID 4096 and INTIDs between 1025
>> and 4095 are reserved, helper macros are introduced to simplify the
>> transformation of indices and to enable easier access to eSPI-specific
>> resources. These changes prepare the VGIC for processing eSPIs as
>> required by future functionality.
>>
>> The initialization and deinitialization paths for vgic have been updated
>> to allocate and free these resources appropriately. Additionally,
>> updated handling of INTIDs greater than 1024, passed from the toolstack
>> during domain creation, and verification logic ensures only valid SPI or
>> eSPI INTIDs are used.
>>
>> The existing SPI behavior remains unaffected when guests do not request
>> eSPIs, GIC hardware does not support them, or the CONFIG_GICV3_ESPI
>> option is disabled.
>>
>> Signed-off-by: Leonid Komarianskyi <leonid_komarianskyi@epam.com>
>>
>> ---
>> Changes in V5:
>> - removed the has_espi field because it can be determined by checking
>> whether domain->arch.vgic.nr_espis is zero or not
>> - since vgic_ext_rank_offset is not used in this patch, it has been
>> moved to the appropriate patch in the patch series, which implements
>> vgic eSPI registers emulation and requires this function
>> - removed ifdefs for eSPI-specific macros to reduce the number of ifdefs
>> and code duplication in further changes
>> - fixed minor nit: used %pd for printing domain with its ID
>>
>> Changes in V4:
>> - added has_espi field to simplify determining whether a domain is able
>> to operate with eSPI
>> - fixed formatting issues and misspellings
>>
>> Changes in V3:
>> - fixed formatting for lines with more than 80 symbols
>> - introduced helper functions to be able to use stubs in case of
>> CONFIG_GICV3_ESPI disabled, and as a result, reduce the number of
>> #ifdefs
>> - fixed checks for nr_spis in domain_vgic_init
>> - updated comment about nr_spis adjustments with dom0less mention
>> - moved comment with additional explanations before checks
>> - used unsigned int for indexes since they cannot be negative
>> - removed unnecessary parentheses
>> - move vgic_ext_rank_offset to the below ifdef guard, to reduce the
>> number of ifdefs
>>
>> Changes in V2:
>> - change is_espi_rank to is_valid_espi_rank to verify whether the array
>> element ext_shared_irqs exists. The previous version, is_espi_rank,
>> only checked if the rank index was less than the maximum possible eSPI
>> rank index, but this could potentially result in accessing a
>> non-existing array element. To address this, is_valid_espi_rank was
>> introduced, which ensures that the required eSPI rank exists
>> - move gic_number_espis to
>> xen/arm: gicv3: implement handling of GICv3.1 eSPI
>> - update vgic_is_valid_irq checks to allow operating with eSPIs
>> - remove redundant newline in vgic_allocate_virq
>> ---
>> xen/arch/arm/include/asm/vgic.h | 12 ++
>> xen/arch/arm/vgic.c | 199 +++++++++++++++++++++++++++++++-
>> 2 files changed, 208 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/
>> asm/vgic.h
>> index 3e7cbbb196..912d5b7694 100644
>> --- a/xen/arch/arm/include/asm/vgic.h
>> +++ b/xen/arch/arm/include/asm/vgic.h
>> @@ -146,6 +146,10 @@ struct vgic_dist {
>> int nr_spis; /* Number of SPIs */
>> unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
>> struct vgic_irq_rank *shared_irqs;
>> +#ifdef CONFIG_GICV3_ESPI
>> + struct vgic_irq_rank *ext_shared_irqs;
>> + int nr_espis; /* Number of extended SPIs */
>
> For new code, we prefer using "unsigned int" when the value is meant to
> be >= 0.
>
Sure, Oleksandr has already spotted this. I will change the type to
unsigned in V6.
>> +#endif
>> /*
>> * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
>> * struct arch_vcpu.
>> @@ -243,6 +247,14 @@ struct vgic_ops {
>> /* Number of ranks of interrupt registers for a domain */
>> #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_spis+31)/32)
>> +#ifdef CONFIG_GICV3_ESPI
>> +#define DOMAIN_NR_EXT_RANKS(d) (((d)->arch.vgic.nr_espis+31)/32)
>> +#endif
>> +#define EXT_RANK_MIN (ESPI_BASE_INTID/32)
>> +#define EXT_RANK_MAX ((ESPI_MAX_INTID+31)/32)
>> +#define EXT_RANK_NUM2IDX(num) ((num)-EXT_RANK_MIN)
>> +#define EXT_RANK_IDX2NUM(idx) ((idx)+EXT_RANK_MIN)
>
> For the 6 lines above, please add space before and after the operators.
>
I will fix formatting in V6.
>> +
>> #define vgic_lock(v) spin_lock_irq(&(v)->domain->arch.vgic.lock)
>> #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 2bbf4d99aa..c9b9528c66 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -27,9 +27,82 @@
>> bool vgic_is_valid_line(struct domain *d, unsigned int virq)
>> {
>> +#ifdef CONFIG_GICV3_ESPI
>> + if ( virq >= ESPI_BASE_INTID &&
>> + virq < ESPI_IDX2INTID(d->arch.vgic.nr_espis) )
>> + return true;
>> +#endif
>> +
>> return virq < vgic_num_irqs(d);
>> }
>> +#ifdef CONFIG_GICV3_ESPI
>> +/*
>> + * Since eSPI indexes start from 4096 and numbers from 1024 to
>> + * 4095 are forbidden, we need to check both lower and upper
>> + * limits for ranks.
>> + */
>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int
>> rank)
>> +{
>> + return rank >= EXT_RANK_MIN &&
>> + EXT_RANK_NUM2IDX(rank) < DOMAIN_NR_EXT_RANKS(d);
>> +}
>> +
>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>> + unsigned int
>> rank)
>> +{
>> + return &v->domain-
>> >arch.vgic.ext_shared_irqs[EXT_RANK_NUM2IDX(rank)];
>> +}
>> +
>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned
>> int virq)
>> +{
>> + return !test_and_set_bit(ESPI_INTID2IDX(virq) + vgic_num_irqs(d),
>> + d->arch.vgic.allocated_irqs);
>> +}
>> +
>> +static void arch_move_espis(struct vcpu *v)
>> +{
>> + const cpumask_t *cpu_mask = cpumask_of(v->processor);
>> + struct domain *d = v->domain;
>> + struct pending_irq *p;
>> + struct vcpu *v_target;
>
> cpu_mask, p and v_target only seems to be used within the loop. If
> that's correct, then please move the definition within the loop.
>
This function will be removed (thanks to Volodymyr's suggestion with
helper function), and I will reuse the existing code in V6.
>> + unsigned int i;
>> +
>> + for ( i = ESPI_BASE_INTID;
>> + i < EXT_RANK_IDX2NUM(d->arch.vgic.nr_espis); i++ )
>> + {
>> + v_target = vgic_get_target_vcpu(v, i);
>> + p = irq_to_pending(v_target, i);
>> +
>> + if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p-
>> >status) )
>> + irq_set_affinity(p->desc, cpu_mask);
>> + }
>> +}
>> +#else
>> +static inline bool is_valid_espi_rank(struct domain *d, unsigned int
>> rank)
>> +{
>> + return false;
>> +}
>> +
>> +/*
>> + * This function is stub and will not be called if CONFIG_GICV3_ESPI=n,
>> + * because in this case, is_valid_espi_rank will always return false.
>> + */
>> +static inline struct vgic_irq_rank *vgic_get_espi_rank(struct vcpu *v,
>> + unsigned int
>> rank)
>> +{
>> + ASSERT_UNREACHABLE();
>> + return NULL;
>> +}
>> +
>> +static inline bool vgic_reserve_espi_virq(struct domain *d, unsigned
>> int virq)
>> +{
>> + return false;
>> +}
>> +
>> +static void arch_move_espis(struct vcpu *v) { }
>> +#endif
>> +
>> static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v,
>> unsigned int rank)
>> {
>> @@ -37,6 +110,8 @@ static inline struct vgic_irq_rank
>> *vgic_get_rank(struct vcpu *v,
>> return v->arch.vgic.private_irqs;
>> else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
>> return &v->domain->arch.vgic.shared_irqs[rank - 1];
>> + else if ( is_valid_espi_rank(v->domain, rank) )
>> + return vgic_get_espi_rank(v, rank);
>> else
>> return NULL;
>> }
>> @@ -117,6 +192,62 @@ int domain_vgic_register(struct domain *d,
>> unsigned int *mmio_count)
>> return 0;
>> }
>> +#ifdef CONFIG_GICV3_ESPI
>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>> +{
>> + return d->arch.vgic.nr_spis + d->arch.vgic.nr_espis;
>> +}
>> +
>> +static int init_vgic_espi(struct domain *d)
>> +{
>> + unsigned int i, idx;
>> +
>> + if ( d->arch.vgic.nr_espis == 0 )
>> + return 0;
>> +
>> + d->arch.vgic.ext_shared_irqs =
>> + xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_EXT_RANKS(d));
>> + if ( d->arch.vgic.ext_shared_irqs == NULL )
>> + return -ENOMEM;
>> +
>> + for ( i = d->arch.vgic.nr_spis, idx = 0;
>> + i < vgic_num_spi_lines(d); i++, idx++ )
>> + vgic_init_pending_irq(&d->arch.vgic.pending_irqs[i],
>> + ESPI_IDX2INTID(idx));
>> +
>> + for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
>> + vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
>> +
>> + return 0;
>> +}
>> +
>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>> +{
>> + irq = ESPI_INTID2IDX(irq) + d->arch.vgic.nr_spis;
>> + return &d->arch.vgic.pending_irqs[irq];
>> +}
>> +#else
>> +static unsigned int init_vgic_espi(struct domain *d)
>> +{
>> + return 0;
>> +}
>> +
>> +static unsigned int vgic_num_spi_lines(struct domain *d)
>> +{
>> + return d->arch.vgic.nr_spis;
>> +}
>> +
>> +struct pending_irq *espi_to_pending(struct domain *d, unsigned int irq)
>> +{
>> + return NULL;
>> +}
>> +#endif
>> +
>> +static unsigned int vgic_num_alloc_irqs(struct domain *d)
>> +{
>> + return vgic_num_spi_lines(d) + NR_LOCAL_IRQS;
>> +}
>> +
>> int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> {
>> int i;
>> @@ -131,6 +262,36 @@ int domain_vgic_init(struct domain *d, unsigned
>> int nr_spis)
>> */
>> nr_spis = ROUNDUP(nr_spis, 32);
>> +#ifdef CONFIG_GICV3_ESPI
>> + /*
>> + * During domain creation, the dom0less DomUs code or toolstack
>> specifies
>> + * the maximum INTID, which is defined in the domain config
>> subtracted by
>> + * 32 to cover the local IRQs (please see the comment to
>> VGIC_DEF_NR_SPIS).
>> + * To compute the actual number of eSPI that will be usable for,
>> + * add back 32.
>> + */
>> + if ( nr_spis + 32 > ESPI_IDX2INTID(NR_ESPI_IRQS) )
>> + return -EINVAL;
>> +
>> + if ( nr_spis + 32 >= ESPI_BASE_INTID )
>> + {
>> + d->arch.vgic.nr_espis = min(nr_spis - ESPI_BASE_INTID + 32,
>> 1024U);
>
> We should return an error rather than silently capping the value.
>
>> + /* Verify if GIC HW can handle provided INTID */
>> + if ( d->arch.vgic.nr_espis > gic_number_espis() )
>> + return -EINVAL;
>> + /*
>> + * Set the maximum available number for regular
>> + * SPI to pass the next check
>> + */
>
> I think it would be better to rework the code so the check is not called.
>
I will rework the code in V6, thanks.
>> + nr_spis = VGIC_DEF_NR_SPIS;
>> + }
>> + else
>> + {
>> + /* Domain will use the regular SPI range */
>> + d->arch.vgic.nr_espis = 0;
>> + }
>> +#endif
>> +
>> /* Limit the number of virtual SPIs supported to (1020 - 32) =
>> 988 */
>> if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>> return -EINVAL;
>> @@ -145,7 +306,7 @@ int domain_vgic_init(struct domain *d, unsigned
>> int nr_spis)
>> return -ENOMEM;
>> d->arch.vgic.pending_irqs =
>> - xzalloc_array(struct pending_irq, d->arch.vgic.nr_spis);
>> + xzalloc_array(struct pending_irq, vgic_num_spi_lines(d));
>> if ( d->arch.vgic.pending_irqs == NULL )
>> return -ENOMEM;
>> @@ -156,12 +317,16 @@ int domain_vgic_init(struct domain *d, unsigned
>> int nr_spis)
>> for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
>> vgic_rank_init(&d->arch.vgic.shared_irqs[i], i + 1, 0);
>> + ret = init_vgic_espi(d);
>> + if ( ret )
>> + return ret;
>> +
>> ret = d->arch.vgic.handler->domain_init(d);
>> if ( ret )
>> return ret;
>> d->arch.vgic.allocated_irqs =
>> - xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
>> + xzalloc_array(unsigned long,
>> BITS_TO_LONGS(vgic_num_alloc_irqs(d)));
>> if ( !d->arch.vgic.allocated_irqs )
>> return -ENOMEM;
>> @@ -195,9 +360,27 @@ void domain_vgic_free(struct domain *d)
>> }
>> }
>> +#ifdef CONFIG_GICV3_ESPI
>> + for ( i = 0; i < d->arch.vgic.nr_espis; i++ )
>
> Now we are potentially doubling up the number of IRQs we have to
> release. I am not entirely sure this is still ok to do it without any
> preemption. Do you have any data?
>
I made some measurements on our hardware (unfortunately, we currently
cannot enable all devices to use more IRQs). However, I observed a
linear time dependency based on the number of IRQs, so I believe we can
extrapolate the data.
I prepared some changes for V6 according to the comments (this is not
the final solution, but I can still measure the time). I added a few
more lines of code to measure the elapsed time:
void domain_vgic_free(struct domain *d)
{
struct pending_irq *p;
unsigned int i, virq;
int ret;
unsigned int relsesed = 0;
s_time_t start = NOW(), diff;
for ( i = 32; i < vgic_num_alloc_irqs(d); i++ )
{
virq = idx_to_virq(d, i);
p = spi_to_pending(d, virq);
if ( p->desc )
{
relsesed++;
ret = release_guest_irq(d, p->irq);
if ( ret )
dprintk(XENLOG_G_WARNING, "d%u: Failed to release virq
%u ret = %d\n",
d->domain_id, p->irq, ret);
}
}
diff = NOW() - start;
printk("time elapsed = %lu ns (%lu us, %lu ms) released = %u IRQs\n",
diff, diff / 1000, diff / 1000000, relsesed);
Please note that vgic_num_alloc_irqs(d) in V6 will operate across the
full IRQ range, including the eSPIs. I also checked multiple times and
obtained consistent results.
Here are the measurements:
time elapsed = 485145 ns (485 us, 0 ms) released = 100 IRQs
time elapsed = 954196 ns (954 us, 0 ms) released = 200 IRQs
time elapsed = 1921712 ns (1921 us, 1 ms) released = 400 IRQs
Also, the index conversion and spi_to_pending() do not take much time,
as they are quite simple operations. The most time-consuming operation
is release_guest_irq(), but it is invoked only when an IRQ is assigned
to a domain.
So, releasing every 100 IRQs while destroying domain takes approximately
500 us on our setup. Therefore, releasing the maximum IRQs - 960 SPIs +
1024 eSPIs = 1984 IRQs - would take approximately 10 ms. From my
understanding, this is not critical, especially considering that nearly
2000 IRQs typically will not be assigned to a single domain.
>> + {
>> + struct pending_irq *p = spi_to_pending(d, ESPI_IDX2INTID(i));
>> +
>> + if ( p->desc )
>> + {
>> + ret = release_guest_irq(d, p->irq);
>> + if ( ret )
>> + dprintk(XENLOG_G_WARNING, "%pd: Failed to release
>> virq %u ret = %d\n",
>> + d, p->irq, ret);
>> + }
>> + }
>> +#endif
>> +
>> if ( d->arch.vgic.handler )
>> d->arch.vgic.handler->domain_free(d);
>> xfree(d->arch.vgic.shared_irqs);
>> +#ifdef CONFIG_GICV3_ESPI
>> + xfree(d->arch.vgic.ext_shared_irqs);
>> +#endif
>> xfree(d->arch.vgic.pending_irqs);
>> xfree(d->arch.vgic.allocated_irqs);
>> }
>> @@ -331,6 +514,8 @@ void arch_move_irqs(struct vcpu *v)
>> if ( v_target == v && !test_bit(GIC_IRQ_GUEST_MIGRATING, &p-
>> >status) )
>> irq_set_affinity(p->desc, cpu_mask);
>> }
>> +
>> + arch_move_espis(v);
>> }
>> void vgic_disable_irqs(struct vcpu *v, uint32_t r, unsigned int n)
>> @@ -538,6 +723,8 @@ struct pending_irq *irq_to_pending(struct vcpu *v,
>> unsigned int irq)
>> n = &v->arch.vgic.pending_irqs[irq];
>> else if ( is_lpi(irq) )
>> n = v->domain->arch.vgic.handler->lpi_to_pending(v->domain,
>> irq);
>> + else if ( is_espi(irq) )
>> + n = espi_to_pending(v->domain, irq);
>> else
>> n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>> return n;
>> @@ -547,6 +734,9 @@ struct pending_irq *spi_to_pending(struct domain
>> *d, unsigned int irq)
>> {
>> ASSERT(irq >= NR_LOCAL_IRQS);
>> + if ( is_espi(irq) )
>> + return espi_to_pending(d, irq);
>> +
>> return &d->arch.vgic.pending_irqs[irq - 32];
>> }
>> @@ -668,6 +858,9 @@ bool vgic_reserve_virq(struct domain *d, unsigned
>> int virq)
>> if ( !vgic_is_valid_line(d, virq) )
>> return false;
>> + if ( is_espi(virq) )
>> + return vgic_reserve_espi_virq(d, virq);
>> +
>> return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
>> }
>> @@ -685,7 +878,7 @@ int vgic_allocate_virq(struct domain *d, bool spi)
>> else
>> {
>> first = 32;
>> - end = vgic_num_irqs(d);
>> + end = vgic_num_alloc_irqs(d);
>> }
>> /*
>
> Cheers,
>
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-09-02 20:20 ` Julien Grall
@ 2025-09-02 20:55 ` Leonid Komarianskyi
2025-09-03 11:26 ` Julien Grall
0 siblings, 1 reply; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-02 20:55 UTC (permalink / raw)
To: Julien Grall, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Julien,
On 02.09.25 23:20, Julien Grall wrote:
> Hi Leonid,
>
> On 02/09/2025 18:26, Leonid Komarianskyi wrote:
>> Hi Julien,
>>
>> Thank you for your review and suggestions.
>>
>> On 02.09.25 19:55, Julien Grall wrote:
>>> Hi Leonid,
>>>
>>> On 29/08/2025 17:06, Leonid Komarianskyi wrote:
>>>> @@ -782,46 +804,81 @@ static int
>>>> __vgic_v3_distr_common_mmio_write(const char *name, struct vcpu *v,
>>>> {
>>>> case VRANGE32(GICD_IGROUPR, GICD_IGROUPRN):
>>>> case VRANGE32(GICD_IGRPMODR, GICD_IGRPMODRN):
>>>> + case VRANGE32(GICD_IGROUPRnE, GICD_IGROUPRnEN):
>>>> + case VRANGE32(GICD_IGRPMODRnE, GICD_IGRPMODRnEN):
>>>> /* We do not implement security extensions for guests, write
>>>> ignore */
>>>> goto write_ignore_32;
>>>> case VRANGE32(GICD_ISENABLER, GICD_ISENABLERN):
>>>> + case VRANGE32(GICD_ISENABLERnE, GICD_ISENABLERnEN):
>>>> if ( dabt.size != DABT_WORD ) goto bad_width;
>>>> - rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER,
>>>> DABT_WORD);
>>>> + if ( reg >= GICD_ISENABLERnE )
>>>> + rank = vgic_ext_rank_offset(v, 1, reg - GICD_ISENABLERnE,
>>>> + DABT_WORD);
>>>> + else
>>>> + rank = vgic_rank_offset(v, 1, reg - GICD_ISENABLER,
>>>> DABT_WORD);
>>>
>>> While I understand the desire to try to avoid code duplication. I feel
>>> this is making the code a lot more complicating and in fact riskier
>>> because...
>>>
>>>> if ( rank == NULL ) goto write_ignore;
>>>> vgic_lock_rank(v, rank, flags);
>>>> tr = rank->ienable;
>>>> vreg_reg32_setbits(&rank->ienable, r, info);
>>>> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>>>> + if ( reg >= GICD_ISENABLERnE )
>>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
>>>> + EXT_RANK_IDX2NUM(rank->index));
>>>> + else
>>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>>>
>>> ... you now have to keep both "if" the same. Unless we can have a
>>> version to avoid "ifs" everywhere (maybe using macros), I would rather
>>> create a separate funciton to handle eSPIs.
>>>
>>> Cheers,
>>>
>>
>> The main idea of V5 of this patch is to consolidate similar code and
>> keep the pairs of regular SPIs and their eSPI counterparts in the same
>> place to simplify any future modifications of these pairs. If
>> eSPI-specific registers are moved to a separate function, it would
>> result in code duplication. Additionally, I think it would be easier to
>> miss updating the code for one register of the SPI/eSPI pair while
>> modifying the code for the other..
>
> I understand your reasoning, but in this case, we need to try to keep
> the code as common as possible and reduce the number of ifs. Looking at
> the patch again, we seem to often use "EXT_RANK_IDX2NUM(rank->index)"
> and this is why we have the second "if", for instance here. I think it
> would be preferable if "index" store the proper value.
>
Actually, there are 4 specific cases where I need to use EXT_RANK_IDX2NUM:
vgic_enable_irqs, vgic_disable_irqs, vgic_set_irqs_pending, and
vgic_check_inflight_irqs_pending. The reason I made this transformation
is that these functions are generic and calculate the virq based on the
rank number, e.g. vgic_check_inflight_irqs_pending():
unsigned int irq = i + 32 * rank;
As a result, I decided to introduce EXT_RANK_IDX2NUM with ifs rather
than modifying very generic code that would also affect vGICv2 and be
more difficult to upstream..
> An alternative would be to process the 3rd parameters separately.
>
> The next big one to takle is:
>
> if ( reg >= ... )
> vgic_ext_rank_...(...)
> else
> vgic_rank(...)
>
> Ideally I would like to provide an helper that can figure out whether
> this is an eSPI or not. Looking at the pattern, I think we would
> introduce a new helper which rather than taking a relative offset take
> the reg offset, the base for SPIs and the base for eSPIs.
>
> Cheers,
>
Hm, that's a good idea. I will try to implement this. I agree that it
will reduce the boilerplate code.
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-09-02 20:55 ` Leonid Komarianskyi
@ 2025-09-03 11:26 ` Julien Grall
2025-09-03 11:45 ` Leonid Komarianskyi
0 siblings, 1 reply; 49+ messages in thread
From: Julien Grall @ 2025-09-03 11:26 UTC (permalink / raw)
To: Leonid Komarianskyi, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Leonid,
On 02/09/2025 21:55, Leonid Komarianskyi wrote:
>>>>> if ( rank == NULL ) goto write_ignore;
>>>>> vgic_lock_rank(v, rank, flags);
>>>>> tr = rank->ienable;
>>>>> vreg_reg32_setbits(&rank->ienable, r, info);
>>>>> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>>>>> + if ( reg >= GICD_ISENABLERnE )
>>>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
>>>>> + EXT_RANK_IDX2NUM(rank->index));
>>>>> + else
>>>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>>>>
>>>> ... you now have to keep both "if" the same. Unless we can have a
>>>> version to avoid "ifs" everywhere (maybe using macros), I would rather
>>>> create a separate funciton to handle eSPIs.
>>>>
>>>> Cheers,
>>>>
>>>
>>> The main idea of V5 of this patch is to consolidate similar code and
>>> keep the pairs of regular SPIs and their eSPI counterparts in the same
>>> place to simplify any future modifications of these pairs. If
>>> eSPI-specific registers are moved to a separate function, it would
>>> result in code duplication. Additionally, I think it would be easier to
>>> miss updating the code for one register of the SPI/eSPI pair while
>>> modifying the code for the other..
>>
>> I understand your reasoning, but in this case, we need to try to keep
>> the code as common as possible and reduce the number of ifs. Looking at
>> the patch again, we seem to often use "EXT_RANK_IDX2NUM(rank->index)"
>> and this is why we have the second "if", for instance here. I think it
>> would be preferable if "index" store the proper value.
>>
>
> Actually, there are 4 specific cases where I need to use EXT_RANK_IDX2NUM:
> vgic_enable_irqs, vgic_disable_irqs, vgic_set_irqs_pending, and
> vgic_check_inflight_irqs_pending. The reason I made this transformation
> is that these functions are generic and calculate the virq based on the
> rank number, e.g. vgic_check_inflight_irqs_pending():
>
> unsigned int irq = i + 32 * rank;
>
> As a result, I decided to introduce EXT_RANK_IDX2NUM with ifs rather
> than modifying very generic code that would also affect vGICv2 and be
> more difficult to upstream..
I wasn't asking to modify the code in vgic_enable_irqs() & co. But
rather how it is called.
Right now, rank->index for eSPI, will be starting at 0. But what if
instead, it is starting at 128 (i.e. EXT_RANK_MIN)?
Effectively, rather than initializating the eSPI ranks with:
for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
You could do:
for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i],
EXT_RANK_IDX2NUM(i), 0);
This would remove all the "if"s and the "EXT_RANK_IDX2NUM(rank->index)".
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers
2025-09-03 11:26 ` Julien Grall
@ 2025-09-03 11:45 ` Leonid Komarianskyi
0 siblings, 0 replies; 49+ messages in thread
From: Leonid Komarianskyi @ 2025-09-03 11:45 UTC (permalink / raw)
To: Julien Grall, xen-devel@lists.xenproject.org
Cc: olekstysh@gmail.com, Stefano Stabellini, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
Hi Julien,
Thank you for your suggestion.
On 03.09.25 14:26, Julien Grall wrote:
> Hi Leonid,
>
> On 02/09/2025 21:55, Leonid Komarianskyi wrote:
>>>>>> if ( rank == NULL ) goto write_ignore;
>>>>>> vgic_lock_rank(v, rank, flags);
>>>>>> tr = rank->ienable;
>>>>>> vreg_reg32_setbits(&rank->ienable, r, info);
>>>>>> - vgic_enable_irqs(v, (rank->ienable) & (~tr), rank->index);
>>>>>> + if ( reg >= GICD_ISENABLERnE )
>>>>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr),
>>>>>> + EXT_RANK_IDX2NUM(rank->index));
>>>>>> + else
>>>>>> + vgic_enable_irqs(v, (rank->ienable) & (~tr), rank-
>>>>>> >index);
>>>>>
>>>>> ... you now have to keep both "if" the same. Unless we can have a
>>>>> version to avoid "ifs" everywhere (maybe using macros), I would rather
>>>>> create a separate funciton to handle eSPIs.
>>>>>
>>>>> Cheers,
>>>>>
>>>>
>>>> The main idea of V5 of this patch is to consolidate similar code and
>>>> keep the pairs of regular SPIs and their eSPI counterparts in the same
>>>> place to simplify any future modifications of these pairs. If
>>>> eSPI-specific registers are moved to a separate function, it would
>>>> result in code duplication. Additionally, I think it would be easier to
>>>> miss updating the code for one register of the SPI/eSPI pair while
>>>> modifying the code for the other..
>>>
>>> I understand your reasoning, but in this case, we need to try to keep
>>> the code as common as possible and reduce the number of ifs. Looking at
>>> the patch again, we seem to often use "EXT_RANK_IDX2NUM(rank->index)"
>>> and this is why we have the second "if", for instance here. I think it
>>> would be preferable if "index" store the proper value.
>>>
>>
>> Actually, there are 4 specific cases where I need to use
>> EXT_RANK_IDX2NUM:
>> vgic_enable_irqs, vgic_disable_irqs, vgic_set_irqs_pending, and
>> vgic_check_inflight_irqs_pending. The reason I made this transformation
>> is that these functions are generic and calculate the virq based on the
>> rank number, e.g. vgic_check_inflight_irqs_pending():
>>
>> unsigned int irq = i + 32 * rank;
>>
>> As a result, I decided to introduce EXT_RANK_IDX2NUM with ifs rather
>> than modifying very generic code that would also affect vGICv2 and be
>> more difficult to upstream..
>
> I wasn't asking to modify the code in vgic_enable_irqs() & co. But
> rather how it is called.
>
> Right now, rank->index for eSPI, will be starting at 0. But what if
> instead, it is starting at 128 (i.e. EXT_RANK_MIN)?
>
> Effectively, rather than initializating the eSPI ranks with:
>
> for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
> vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i], i, 0);
>
> You could do:
>
> for ( i = 0; i < DOMAIN_NR_EXT_RANKS(d); i++ )
> vgic_rank_init(&d->arch.vgic.ext_shared_irqs[i],
> EXT_RANK_IDX2NUM(i), 0);
>
> This would remove all the "if"s and the "EXT_RANK_IDX2NUM(rank->index)".
>
> Cheers,
>
Yesterday evening, I realized the same :) I fixed this while preparing
the next version of the patch series. Also, I found a way to remove many
ifs in this patch by introducing just 2 helper functions. I will push v6
soon with these updates. I hope it will look much better.
Best regards,
Leonid
^ permalink raw reply [flat|nested] 49+ messages in thread
end of thread, other threads:[~2025-09-03 11:46 UTC | newest]
Thread overview: 49+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-29 16:06 [PATCH v5 00/12] Introduce eSPI support Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 01/12] xen/arm: gicv3: refactor obtaining GIC addresses for common operations Leonid Komarianskyi
2025-08-31 12:42 ` Oleksandr Tyshchenko
2025-09-01 14:36 ` Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 02/12] xen/arm: gic: implement helper functions for INTID checks Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 03/12] xen/arm: vgic: implement helper functions for virq checks Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 04/12] xen/arm/irq: add handling for IRQs in the eSPI range Leonid Komarianskyi
2025-08-29 19:45 ` Volodymyr Babchuk
2025-08-31 14:08 ` Oleksandr Tyshchenko
2025-09-01 14:42 ` Leonid Komarianskyi
2025-09-01 16:16 ` Julien Grall
2025-09-02 6:56 ` Leonid Komarianskyi
2025-09-01 16:11 ` Julien Grall
2025-09-02 8:56 ` Leonid Komarianskyi
2025-09-02 13:11 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 05/12] xen/arm: gicv3: implement handling of GICv3.1 eSPI Leonid Komarianskyi
2025-08-29 19:55 ` Volodymyr Babchuk
2025-09-01 17:30 ` Leonid Komarianskyi
2025-08-31 14:56 ` Oleksandr Tyshchenko
2025-08-29 16:06 ` [PATCH v5 06/12] xen/arm/irq: allow eSPI processing in the gic_interrupt function Leonid Komarianskyi
2025-09-02 13:42 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 07/12] xen/arm: gicv3: modify ICH_LR_PHYSICAL_MASK to allow eSPI processing Leonid Komarianskyi
2025-09-02 13:53 ` Julien Grall
2025-09-02 17:06 ` Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 08/12] xen/arm: vgic: add resource management for extended SPIs Leonid Komarianskyi
2025-08-29 20:45 ` Volodymyr Babchuk
2025-08-31 15:58 ` Oleksandr Tyshchenko
2025-09-01 18:00 ` Leonid Komarianskyi
2025-09-02 8:16 ` Oleksandr Tyshchenko
2025-09-01 17:38 ` Leonid Komarianskyi
2025-09-02 14:13 ` Julien Grall
2025-09-02 20:24 ` Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 09/12] xen/arm: domain_build/dom0less-build: adjust domains config to support eSPIs Leonid Komarianskyi
2025-09-02 16:42 ` Julien Grall
2025-09-02 17:15 ` Leonid Komarianskyi
2025-09-02 19:59 ` Julien Grall
2025-08-29 16:06 ` [PATCH v5 10/12] xen/arm: vgic-v3: add emulation of GICv3.1 eSPI registers Leonid Komarianskyi
2025-08-29 20:12 ` Volodymyr Babchuk
2025-09-01 18:32 ` Leonid Komarianskyi
2025-09-01 12:38 ` Oleksandr Tyshchenko
2025-09-01 18:27 ` Leonid Komarianskyi
2025-09-02 16:55 ` Julien Grall
2025-09-02 17:26 ` Leonid Komarianskyi
2025-09-02 20:20 ` Julien Grall
2025-09-02 20:55 ` Leonid Komarianskyi
2025-09-03 11:26 ` Julien Grall
2025-09-03 11:45 ` Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 11/12] doc/man: update description for nr_spis with eSPI Leonid Komarianskyi
2025-08-29 16:06 ` [PATCH v5 12/12] CHANGELOG.md: add mention of GICv3.1 eSPI support Leonid Komarianskyi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).