* [PATCH v4 01/22] xen/arm: smmuv3: Maintain a SID->device structure
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-07 0:06 ` [PATCH v4 02/22] xen/arm: smmuv3: Add support for stage-1 and nested stage translation Milan Djokic
` (20 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Jean-Philippe Brucker, Bertrand Marquis, Rahul Singh,
Stefano Stabellini, Julien Grall, Michal Orzel, Volodymyr Babchuk,
Joerg Roedel, Nicolin Chen, Will Deacon, Milan Djokic
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Backport Linux commits cdf315f907d4 and b00d24997a11, adapted to the Xen
SMMUv3 implementation.
When handling faults from the event or PRI queue, we need to find the
struct device associated with a SID. Add an rb_tree to keep track of
SIDs.
Also handle duplicate SIDs, as bridged PCI devices may have multiple
streams with the same SID.
The backport is adapted to Xen's SMMUv3 driver structure and types.
Fixes are also included in the master insertion rollback
and teardown paths to correctly handle partially
inserted stream entries and avoid leaving stale rb_tree nodes.
Linux commits:
cdf315f907d4 ("iommu/arm-smmu-v3: Maintain a SID->device structure")
b00d24997a11 ("iommu/arm-smmu-v3: Fix iommu_device_probe bug due to duplicated stream ids")
Link: https://lore.kernel.org/r/20210401154718.307519-8-jean-philippe@linaro.org
Link: https://lore.kernel.org/r/20250415185620.504299-1-nicolinc@nvidia.com
Origin: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cdf315f907d4
Origin: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git b00d24997a11
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/smmu-v3.c | 176 ++++++++++++++++++++++----
xen/drivers/passthrough/arm/smmu-v3.h | 13 +-
2 files changed, 163 insertions(+), 26 deletions(-)
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index bf153227db..7162473d62 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -809,6 +809,27 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid)
return 0;
}
+__maybe_unused
+static struct arm_smmu_master *
+arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid)
+{
+ struct rb_node *node;
+ struct arm_smmu_stream *stream;
+
+ node = smmu->streams.rb_node;
+ while (node) {
+ stream = rb_entry(node, struct arm_smmu_stream, node);
+ if (stream->id < sid)
+ node = node->rb_right;
+ else if (stream->id > sid)
+ node = node->rb_left;
+ else
+ return stream->master;
+ }
+
+ return NULL;
+}
+
/* IRQ and event handlers */
static void arm_smmu_evtq_tasklet(void *dev)
{
@@ -1042,8 +1063,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,
if (!master->ats_enabled)
return 0;
- for (i = 0; i < master->num_sids; i++) {
- cmd->atc.sid = master->sids[i];
+ for (i = 0; i < master->num_streams; i++) {
+ cmd->atc.sid = master->streams[i].id;
arm_smmu_cmdq_issue_cmd(master->smmu, cmd);
}
@@ -1271,13 +1292,13 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master)
int i, j;
struct arm_smmu_device *smmu = master->smmu;
- for (i = 0; i < master->num_sids; ++i) {
- u32 sid = master->sids[i];
+ for (i = 0; i < master->num_streams; ++i) {
+ u32 sid = master->streams[i].id;
__le64 *step = arm_smmu_get_step_for_sid(smmu, sid);
/* Bridged PCI devices may end up with duplicated IDs */
for (j = 0; j < i; j++)
- if (master->sids[j] == sid)
+ if (master->streams[j].id == sid)
break;
if (j < i)
continue;
@@ -1486,6 +1507,123 @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid)
return sid < limit;
}
+
+static int arm_smmu_insert_master(struct arm_smmu_device *smmu,
+ struct arm_smmu_master *master)
+{
+ int i;
+ int ret = 0;
+ struct arm_smmu_stream *new_stream, *cur_stream;
+ struct rb_node **new_node, *parent_node = NULL;
+ struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev);
+ bool *insert;
+
+ master->streams = _xzalloc_array(sizeof(*master->streams), sizeof(void *),
+ fwspec->num_ids);
+ if (!master->streams)
+ return -ENOMEM;
+
+ insert = _xzalloc_array(sizeof(bool), sizeof(bool *),
+ fwspec->num_ids);
+ if (!insert) {
+ xfree(master->streams);
+ master->streams = NULL;
+ return -ENOMEM;
+ }
+
+ master->num_streams = fwspec->num_ids;
+
+ mutex_lock(&smmu->streams_mutex);
+ for (i = 0; i < fwspec->num_ids; i++) {
+ u32 sid = fwspec->ids[i];
+ insert[i] = false;
+
+ new_stream = &master->streams[i];
+ new_stream->id = sid;
+ new_stream->master = master;
+ RB_CLEAR_NODE(&new_stream->node);
+
+ /*
+ * Check the SIDs are in range of the SMMU and our stream table
+ */
+ if (!arm_smmu_sid_in_range(smmu, sid)) {
+ ret = -ERANGE;
+ break;
+ }
+
+ /* Ensure l2 strtab is initialised */
+ if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
+ ret = arm_smmu_init_l2_strtab(smmu, sid);
+ if (ret)
+ break;
+ }
+
+ /* Insert into SID tree */
+ insert[i] = true;
+ new_node = &(smmu->streams.rb_node);
+ while (*new_node) {
+ cur_stream = rb_entry(*new_node, struct arm_smmu_stream,
+ node);
+ parent_node = *new_node;
+ if (cur_stream->id > new_stream->id) {
+ new_node = &((*new_node)->rb_left);
+ } else if (cur_stream->id < new_stream->id) {
+ new_node = &((*new_node)->rb_right);
+ } else if (cur_stream->master == master) {
+ /* Bridged PCI devices may end up with duplicated IDs */
+ insert[i] = false;
+ break;
+ }
+ else {
+ dev_warn(master->dev,
+ "stream %u already in tree\n",
+ cur_stream->id);
+ ret = -EINVAL;
+ insert[i] = false;
+ break;
+ }
+ }
+ if (ret)
+ break;
+
+ if(insert[i]) {
+ rb_link_node(&new_stream->node, parent_node, new_node);
+ rb_insert_color(&new_stream->node, &smmu->streams);
+ }
+ }
+
+ if (ret) {
+ for (i--; i >= 0; i--)
+ if (insert[i])
+ rb_erase(&master->streams[i].node, &smmu->streams);
+ xfree(master->streams);
+ master->streams = NULL;
+ }
+ mutex_unlock(&smmu->streams_mutex);
+
+ xfree(insert);
+
+ return ret;
+}
+
+static void arm_smmu_remove_master(struct arm_smmu_master *master)
+{
+ int i;
+ struct arm_smmu_device *smmu = master->smmu;
+ struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev);
+
+ if (!smmu || !master->streams)
+ return;
+
+ mutex_lock(&smmu->streams_mutex);
+ for (i = 0; i < fwspec->num_ids; i++)
+ if(!RB_EMPTY_NODE(&master->streams[i].node))
+ rb_erase(&master->streams[i].node, &smmu->streams);
+ mutex_unlock(&smmu->streams_mutex);
+
+ xfree(master->streams);
+}
+
/* Forward declaration */
static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev);
static int arm_smmu_assign_dev(struct domain *d, u8 devfn, struct device *dev,
@@ -1495,7 +1633,7 @@ static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
static int arm_smmu_add_device(u8 devfn, struct device *dev)
{
- int i, ret;
+ int ret;
struct arm_smmu_device *smmu;
struct arm_smmu_master *master;
struct iommu_fwspec *fwspec;
@@ -1532,26 +1670,11 @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
master->dev = dev;
master->smmu = smmu;
- master->sids = fwspec->ids;
- master->num_sids = fwspec->num_ids;
dev_iommu_priv_set(dev, master);
- /* Check the SIDs are in range of the SMMU and our stream table */
- for (i = 0; i < master->num_sids; i++) {
- u32 sid = master->sids[i];
-
- if (!arm_smmu_sid_in_range(smmu, sid)) {
- ret = -ERANGE;
- goto err_free_master;
- }
-
- /* Ensure l2 strtab is initialised */
- if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
- ret = arm_smmu_init_l2_strtab(smmu, sid);
- if (ret)
- goto err_free_master;
- }
- }
+ ret = arm_smmu_insert_master(smmu, master);
+ if (ret)
+ goto err_free_master;
/*
* Note that PASID must be enabled before, and disabled after ATS:
@@ -1598,6 +1721,8 @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
return 0;
err_free_master:
+ arm_smmu_disable_pasid(master);
+ arm_smmu_remove_master(master);
xfree(master);
dev_iommu_priv_set(dev, NULL);
return ret;
@@ -1796,6 +1921,9 @@ static int __init arm_smmu_init_structures(struct arm_smmu_device *smmu)
{
int ret;
+ mutex_init(&smmu->streams_mutex);
+ smmu->streams = RB_ROOT;
+
ret = arm_smmu_init_queues(smmu);
if (ret)
return ret;
diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
index ab07366294..ab1f29f6c7 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.h
+++ b/xen/drivers/passthrough/arm/smmu-v3.h
@@ -639,6 +639,15 @@ struct arm_smmu_device {
struct tasklet evtq_irq_tasklet;
struct tasklet priq_irq_tasklet;
struct tasklet combined_irq_tasklet;
+
+ struct rb_root streams;
+ struct mutex streams_mutex;
+};
+
+struct arm_smmu_stream {
+ u32 id;
+ struct arm_smmu_master *master;
+ struct rb_node node;
};
/* SMMU private data for each master */
@@ -647,8 +656,8 @@ struct arm_smmu_master {
struct device *dev;
struct arm_smmu_domain *domain;
struct list_head domain_head;
- u32 *sids;
- unsigned int num_sids;
+ struct arm_smmu_stream *streams;
+ unsigned int num_streams;
bool ats_enabled;
};
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 02/22] xen/arm: smmuv3: Add support for stage-1 and nested stage translation
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
2026-10-07 0:06 ` [PATCH v4 01/22] xen/arm: smmuv3: Maintain a SID->device structure Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-09 17:37 ` Oleksandr Tyshchenko
2026-10-07 0:06 ` [PATCH v4 03/22] xen/arm: smmuv3: Alloc io_domain for each device Milan Djokic
` (19 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Bertrand Marquis, Stefano Stabellini, Julien Grall,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Xen SMMUv3 driver only supports stage-2 translation. Add support for
Stage-1 translation that is required to support nested stage
translation.
In true nested mode, both s1_cfg and s2_cfg will coexist.
Let's remove the union. When nested stage translation is setup, both
s1_cfg and s2_cfg are valid.
We introduce a new smmu_domain abort field that will be set
upon guest stage-1 configuration passing. If no guest stage-1
config has been attached, it is ignored when writing the STE.
arm_smmu_write_strtab_ent() is modified to write both stage
fields in the STE and deal with the abort field.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/smmu-v3.c | 96 +++++++++++++++++++++++----
xen/drivers/passthrough/arm/smmu-v3.h | 9 +++
2 files changed, 93 insertions(+), 12 deletions(-)
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index 7162473d62..d3e88d5a55 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -683,8 +683,10 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
* 3. Update Config, sync
*/
u64 val = le64_to_cpu(dst[0]);
- bool ste_live = false;
+ bool s1_live = false, s2_live = false, ste_live = false;
+ bool abort, translate = false;
struct arm_smmu_device *smmu = NULL;
+ struct arm_smmu_s1_cfg *s1_cfg = NULL;
struct arm_smmu_s2_cfg *s2_cfg = NULL;
struct arm_smmu_domain *smmu_domain = NULL;
struct arm_smmu_cmdq_ent prefetch_cmd = {
@@ -699,30 +701,57 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
smmu = master->smmu;
}
- if (smmu_domain)
- s2_cfg = &smmu_domain->s2_cfg;
+ if (smmu_domain) {
+ switch (smmu_domain->stage) {
+ case ARM_SMMU_DOMAIN_S1:
+ s1_cfg = &smmu_domain->s1_cfg;
+ break;
+ case ARM_SMMU_DOMAIN_NESTED:
+ s1_cfg = &smmu_domain->s1_cfg;
+ fallthrough;
+ case ARM_SMMU_DOMAIN_S2:
+ s2_cfg = &smmu_domain->s2_cfg;
+ break;
+ default:
+ break;
+ }
+ translate = s1_cfg || s2_cfg;
+ }
if (val & STRTAB_STE_0_V) {
switch (FIELD_GET(STRTAB_STE_0_CFG, val)) {
case STRTAB_STE_0_CFG_BYPASS:
break;
+ case STRTAB_STE_0_CFG_S1_TRANS:
+ s1_live = true;
+ break;
case STRTAB_STE_0_CFG_S2_TRANS:
- ste_live = true;
+ s2_live = true;
+ break;
+ case STRTAB_STE_0_CFG_NESTED:
+ s1_live = true;
+ s2_live = true;
break;
case STRTAB_STE_0_CFG_ABORT:
- BUG_ON(!disable_bypass);
break;
default:
BUG(); /* STE corruption */
}
}
+ ste_live = s1_live || s2_live;
+
/* Nuke the existing STE_0 value, as we're going to rewrite it */
val = STRTAB_STE_0_V;
/* Bypass/fault */
- if (!smmu_domain || !(s2_cfg)) {
- if (!smmu_domain && disable_bypass)
+ if (!smmu_domain)
+ abort = disable_bypass;
+ else
+ abort = smmu_domain->abort;
+
+ if (abort || !translate) {
+ if (abort)
val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_ABORT);
else
val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_BYPASS);
@@ -740,7 +769,32 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
return;
}
+ if (ste_live) {
+ /* First invalidate the live STE */
+ dst[0] = cpu_to_le64(STRTAB_STE_0_CFG_ABORT);
+ arm_smmu_sync_ste_for_sid(smmu, sid);
+ }
+
+ if (s1_cfg) {
+ dst[1] = cpu_to_le64(
+ FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) |
+ FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) |
+ FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) |
+ FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) |
+ FIELD_PREP(STRTAB_STE_1_STRW, STRTAB_STE_1_STRW_NSEL1));
+
+ if (smmu->features & ARM_SMMU_FEAT_STALLS &&
+ !(smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
+ dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD);
+
+ val |= (s1_cfg->s1ctxptr & STRTAB_STE_0_S1CTXPTR_MASK) |
+ FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS) |
+ FIELD_PREP(STRTAB_STE_0_S1CDMAX, s1_cfg->s1cdmax) |
+ FIELD_PREP(STRTAB_STE_0_S1FMT, s1_cfg->s1fmt);
+ }
+
if (s2_cfg) {
+ u64 vttbr = s2_cfg->vttbr & STRTAB_STE_3_S2TTB_MASK;
u64 strtab =
FIELD_PREP(STRTAB_STE_2_S2VMID, s2_cfg->vmid) |
FIELD_PREP(STRTAB_STE_2_VTCR, s2_cfg->vtcr) |
@@ -750,12 +804,19 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2AA64 |
STRTAB_STE_2_S2R;
- BUG_ON(ste_live);
+ if (s2_live) {
+ u64 s2ttb = le64_to_cpu(dst[3]) & STRTAB_STE_3_S2TTB_MASK;
+ BUG_ON(s2ttb != vttbr);
+ }
+
dst[2] = cpu_to_le64(strtab);
- dst[3] = cpu_to_le64(s2_cfg->vttbr & STRTAB_STE_3_S2TTB_MASK);
+ dst[3] = cpu_to_le64(vttbr);
val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S2_TRANS);
+ } else {
+ dst[2] = 0;
+ dst[3] = 0;
}
if (master->ats_enabled)
@@ -1254,8 +1315,16 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain,
{
int ret;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_device *smmu = smmu_domain->smmu;
+
+ if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S2))
+ return 0;
- /* Restrict the stage to what we can actually support */
+ /*
+ * Default to stage-2 support and initialize it. Stage-1 will be
+ * initialized during guest config attachment, if nested or stage-1
+ * mode is required
+ */
smmu_domain->stage = ARM_SMMU_DOMAIN_S2;
ret = arm_smmu_domain_finalise_s2(smmu_domain, master);
@@ -2398,11 +2467,14 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
break;
}
+ if (reg & IDR0_S1P)
+ smmu->features |= ARM_SMMU_FEAT_TRANS_S1;
+
if (reg & IDR0_S2P)
smmu->features |= ARM_SMMU_FEAT_TRANS_S2;
- if (!(reg & IDR0_S2P)) {
- dev_err(smmu->dev, "no stage-2 translation support!\n");
+ if (!(reg & (IDR0_S1P | IDR0_S2P))) {
+ dev_err(smmu->dev, "no translation support!\n");
return -ENXIO;
}
diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
index ab1f29f6c7..e257de6db9 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.h
+++ b/xen/drivers/passthrough/arm/smmu-v3.h
@@ -197,6 +197,7 @@
#define STRTAB_STE_0_CFG_BYPASS 4
#define STRTAB_STE_0_CFG_S1_TRANS 5
#define STRTAB_STE_0_CFG_S2_TRANS 6
+#define STRTAB_STE_0_CFG_NESTED 7
#define STRTAB_STE_0_S1FMT GENMASK_ULL(5, 4)
#define STRTAB_STE_0_S1FMT_LINEAR 0
@@ -549,6 +550,12 @@ struct arm_smmu_strtab_l1_desc {
dma_addr_t l2ptr_dma;
};
+struct arm_smmu_s1_cfg {
+ u64 s1ctxptr;
+ u8 s1fmt;
+ u8 s1cdmax;
+};
+
struct arm_smmu_s2_cfg {
u16 vmid;
u64 vttbr;
@@ -669,6 +676,8 @@ struct arm_smmu_domain {
atomic_t nr_ats_masters;
enum arm_smmu_domain_stage stage;
+ bool abort;
+ struct arm_smmu_s1_cfg s1_cfg;
struct arm_smmu_s2_cfg s2_cfg;
/* Xen domain associated with this SMMU domain */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 02/22] xen/arm: smmuv3: Add support for stage-1 and nested stage translation
2026-10-07 0:06 ` [PATCH v4 02/22] xen/arm: smmuv3: Add support for stage-1 and nested stage translation Milan Djokic
@ 2026-10-09 17:37 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-09 17:37 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Bertrand Marquis, Stefano Stabellini, Julien Grall,
Michal Orzel, Volodymyr Babchuk
On 10/7/26 03:06, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> Xen SMMUv3 driver only supports stage-2 translation. Add support for
> Stage-1 translation that is required to support nested stage
> translation.
>
> In true nested mode, both s1_cfg and s2_cfg will coexist.
> Let's remove the union. When nested stage translation is setup, both
> s1_cfg and s2_cfg are valid.
>
> We introduce a new smmu_domain abort field that will be set
> upon guest stage-1 configuration passing. If no guest stage-1
> config has been attached, it is ignored when writing the STE.
>
> arm_smmu_write_strtab_ent() is modified to write both stage
> fields in the STE and deal with the abort field.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/drivers/passthrough/arm/smmu-v3.c | 96 +++++++++++++++++++++++----
> xen/drivers/passthrough/arm/smmu-v3.h | 9 +++
> 2 files changed, 93 insertions(+), 12 deletions(-)
>
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
> index 7162473d62..d3e88d5a55 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.c
> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
> @@ -683,8 +683,10 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
> * 3. Update Config, sync
> */
> u64 val = le64_to_cpu(dst[0]);
> - bool ste_live = false;
> + bool s1_live = false, s2_live = false, ste_live = false;
> + bool abort, translate = false;
> struct arm_smmu_device *smmu = NULL;
> + struct arm_smmu_s1_cfg *s1_cfg = NULL;
> struct arm_smmu_s2_cfg *s2_cfg = NULL;
> struct arm_smmu_domain *smmu_domain = NULL;
> struct arm_smmu_cmdq_ent prefetch_cmd = {
> @@ -699,30 +701,57 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
> smmu = master->smmu;
> }
>
> - if (smmu_domain)
> - s2_cfg = &smmu_domain->s2_cfg;
> + if (smmu_domain) {
> + switch (smmu_domain->stage) {
> + case ARM_SMMU_DOMAIN_S1:
> + s1_cfg = &smmu_domain->s1_cfg;
> + break;
> + case ARM_SMMU_DOMAIN_NESTED:
> + s1_cfg = &smmu_domain->s1_cfg;
> + fallthrough;
> + case ARM_SMMU_DOMAIN_S2:
> + s2_cfg = &smmu_domain->s2_cfg;
> + break;
> + default:
> + break;
> + }
> + translate = s1_cfg || s2_cfg;
> + }
>
> if (val & STRTAB_STE_0_V) {
> switch (FIELD_GET(STRTAB_STE_0_CFG, val)) {
> case STRTAB_STE_0_CFG_BYPASS:
> break;
> + case STRTAB_STE_0_CFG_S1_TRANS:
> + s1_live = true;
> + break;
> case STRTAB_STE_0_CFG_S2_TRANS:
> - ste_live = true;
> + s2_live = true;
> + break;
> + case STRTAB_STE_0_CFG_NESTED:
> + s1_live = true;
> + s2_live = true;
> break;
> case STRTAB_STE_0_CFG_ABORT:
> - BUG_ON(!disable_bypass);
> break;
> default:
> BUG(); /* STE corruption */
> }
> }
>
> + ste_live = s1_live || s2_live;
> +
> /* Nuke the existing STE_0 value, as we're going to rewrite it */
> val = STRTAB_STE_0_V;
>
> /* Bypass/fault */
> - if (!smmu_domain || !(s2_cfg)) {
> - if (!smmu_domain && disable_bypass)
> + if (!smmu_domain)
> + abort = disable_bypass;
> + else
> + abort = smmu_domain->abort;
> +
> + if (abort || !translate) {
> + if (abort)
> val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_ABORT);
> else
> val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_BYPASS);
> @@ -740,7 +769,32 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
> return;
> }
>
> + if (ste_live) {
> + /* First invalidate the live STE */
> + dst[0] = cpu_to_le64(STRTAB_STE_0_CFG_ABORT);
> + arm_smmu_sync_ste_for_sid(smmu, sid);
> + }
> +
> + if (s1_cfg) {
> + dst[1] = cpu_to_le64(
> + FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) |
> + FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) |
> + FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) |
> + FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) |
> + FIELD_PREP(STRTAB_STE_1_STRW, STRTAB_STE_1_STRW_NSEL1));
> +
> + if (smmu->features & ARM_SMMU_FEAT_STALLS &&
> + !(smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
> + dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD);
> +
> + val |= (s1_cfg->s1ctxptr & STRTAB_STE_0_S1CTXPTR_MASK) |
> + FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S1_TRANS) |
> + FIELD_PREP(STRTAB_STE_0_S1CDMAX, s1_cfg->s1cdmax) |
> + FIELD_PREP(STRTAB_STE_0_S1FMT, s1_cfg->s1fmt);
> + }
> +
> if (s2_cfg) {
> + u64 vttbr = s2_cfg->vttbr & STRTAB_STE_3_S2TTB_MASK;
> u64 strtab =
> FIELD_PREP(STRTAB_STE_2_S2VMID, s2_cfg->vmid) |
> FIELD_PREP(STRTAB_STE_2_VTCR, s2_cfg->vtcr) |
> @@ -750,12 +804,19 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
> STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2AA64 |
> STRTAB_STE_2_S2R;
>
> - BUG_ON(ste_live);
> + if (s2_live) {
> + u64 s2ttb = le64_to_cpu(dst[3]) & STRTAB_STE_3_S2TTB_MASK;
> + BUG_ON(s2ttb != vttbr);
> + }
> +
> dst[2] = cpu_to_le64(strtab);
>
> - dst[3] = cpu_to_le64(s2_cfg->vttbr & STRTAB_STE_3_S2TTB_MASK);
> + dst[3] = cpu_to_le64(vttbr);
>
> val |= FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S2_TRANS);
> + } else {
> + dst[2] = 0;
> + dst[3] = 0;
> }
>
> if (master->ats_enabled)
> @@ -1254,8 +1315,16 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain,
> {
> int ret;
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> + struct arm_smmu_device *smmu = smmu_domain->smmu;
> +
> + if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S2))
> + return 0;
I am not fully confident I understood all the magic here correctly, but
it appears the initial state may be broken on SMMUs without stage-2
(stage-1-only hosts).
Here we return early without setting a stage, and the zeroed domain has
stage-1 (enum value 0). So a new domain is left in a default stage-1
state with no real config. The first attach then writes a live S1 STE
with s1ctxptr = 0 in arm_smmu_write_strtab_ent(), so the hardware reads
a context descriptor from physical address 0. I wonder we should default
the domain to abort.
Could you please double-check?
Also, it is my understanding that without stage-2 translation the
guest’s s1ctxptr and the stage-1 output address are used as host
physical addresses. The guest decides where the hardware reads and
writes. I wonder whether we should refuse this mode for non-hardware
(non-direct-mapped) domains.
Apologies if this has already been discussed.
>
> - /* Restrict the stage to what we can actually support */
> + /*
> + * Default to stage-2 support and initialize it. Stage-1 will be
> + * initialized during guest config attachment, if nested or stage-1
> + * mode is required
> + */
> smmu_domain->stage = ARM_SMMU_DOMAIN_S2;
>
> ret = arm_smmu_domain_finalise_s2(smmu_domain, master);
> @@ -2398,11 +2467,14 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
> break;
> }
>
> + if (reg & IDR0_S1P)
> + smmu->features |= ARM_SMMU_FEAT_TRANS_S1;
> +
> if (reg & IDR0_S2P)
> smmu->features |= ARM_SMMU_FEAT_TRANS_S2;
>
> - if (!(reg & IDR0_S2P)) {
> - dev_err(smmu->dev, "no stage-2 translation support!\n");
> + if (!(reg & (IDR0_S1P | IDR0_S2P))) {
> + dev_err(smmu->dev, "no translation support!\n");
> return -ENXIO;
> }
>
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
> index ab1f29f6c7..e257de6db9 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.h
> +++ b/xen/drivers/passthrough/arm/smmu-v3.h
> @@ -197,6 +197,7 @@
> #define STRTAB_STE_0_CFG_BYPASS 4
> #define STRTAB_STE_0_CFG_S1_TRANS 5
> #define STRTAB_STE_0_CFG_S2_TRANS 6
> +#define STRTAB_STE_0_CFG_NESTED 7
>
> #define STRTAB_STE_0_S1FMT GENMASK_ULL(5, 4)
> #define STRTAB_STE_0_S1FMT_LINEAR 0
> @@ -549,6 +550,12 @@ struct arm_smmu_strtab_l1_desc {
> dma_addr_t l2ptr_dma;
> };
>
> +struct arm_smmu_s1_cfg {
> + u64 s1ctxptr;
> + u8 s1fmt;
> + u8 s1cdmax;
> +};
> +
> struct arm_smmu_s2_cfg {
> u16 vmid;
> u64 vttbr;
> @@ -669,6 +676,8 @@ struct arm_smmu_domain {
> atomic_t nr_ats_masters;
>
> enum arm_smmu_domain_stage stage;
> + bool abort;
> + struct arm_smmu_s1_cfg s1_cfg;
> struct arm_smmu_s2_cfg s2_cfg;
>
> /* Xen domain associated with this SMMU domain */
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 03/22] xen/arm: smmuv3: Alloc io_domain for each device
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
2026-10-07 0:06 ` [PATCH v4 01/22] xen/arm: smmuv3: Maintain a SID->device structure Milan Djokic
2026-10-07 0:06 ` [PATCH v4 02/22] xen/arm: smmuv3: Add support for stage-1 and nested stage translation Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-07 0:06 ` [PATCH v4 04/22] xen/arm: vIOMMU: add generic vIOMMU framework Milan Djokic
` (18 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Bertrand Marquis, Stefano Stabellini, Julien Grall,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
In current implementation io_domain is allocated once for each xen
domain as Stage2 translation is common for all devices in same xen
domain.
Nested stage supports S1 and S2 configuration at the same time. Stage1
translation will be different for each device as linux kernel will
allocate page-table for each device.
Alloc io_domain for each device so that each device can have different
Stage-1 and Stage-2 configuration structure.
The stage-2 configuration is now shared between devices belonging to
the same domain. Update io_domain initialization to create a new stage-2
configuration only if a VMID has not already been allocated for the
target Xen domain. Similarly, update io_domain teardown to free the
VMID only when no other devices are assigned to the target domain.
Also optimize the IOTLB flush procedure to invalidate each VMID only
once.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/smmu-v3.c | 102 ++++++++++++++++++++++++--
1 file changed, 94 insertions(+), 8 deletions(-)
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index d3e88d5a55..bd0689664b 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -1229,14 +1229,49 @@ static void arm_smmu_bitmap_free(unsigned long *map, int idx)
clear_bit(idx, map);
}
+static struct arm_smmu_domain *arm_smmu_get_domain_s2(struct domain *d,
+ struct arm_smmu_device *smmu)
+{
+ struct iommu_domain *io_domain;
+ struct arm_smmu_domain *smmu_domain;
+ struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+
+ if (!smmu)
+ return NULL;
+
+ /*
+ * Loop through the &xen_domain->contexts to locate a context
+ * assigned to this SMMU
+ */
+ list_for_each_entry(io_domain, &xen_domain->contexts, list) {
+ smmu_domain = to_smmu_domain(io_domain);
+ if (smmu_domain->smmu == smmu &&
+ smmu_domain->s2_cfg.vmid != INVALID_VMID)
+ return smmu_domain;
+ }
+
+ return NULL;
+}
+
static void arm_smmu_domain_free(struct iommu_domain *domain)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_device *smmu = smmu_domain->smmu;
struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
+ struct arm_smmu_domain *existing;
- if (cfg->vmid)
- arm_smmu_bitmap_free(smmu->vmid_map, cfg->vmid);
+ if (cfg->vmid) {
+ /*
+ * VMIDs are shared across contexts belonging to the same
+ * Xen domain and SMMU. The current domain has already been
+ * removed from the domain list, so this lookup only finds
+ * remaining users of the VMID.
+ */
+ existing = arm_smmu_get_domain_s2(smmu_domain->d, smmu);
+
+ if (!existing)
+ arm_smmu_bitmap_free(smmu->vmid_map, cfg->vmid);
+ }
xfree(smmu_domain);
}
@@ -1316,6 +1351,7 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain,
int ret;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_device *smmu = smmu_domain->smmu;
+ struct arm_smmu_domain *existing;
if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S2))
return 0;
@@ -1327,6 +1363,26 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain,
*/
smmu_domain->stage = ARM_SMMU_DOMAIN_S2;
+ /*
+ * Look for an already initialized stage-2 configuration for this
+ * Xen domain and SMMU.
+ */
+ existing = arm_smmu_get_domain_s2(smmu_domain->d, smmu);
+
+ if (existing) {
+ /*
+ * Reuse the stage-2 configuration.
+ */
+ smmu_domain->s2_cfg.vmid = existing->s2_cfg.vmid;
+ smmu_domain->s2_cfg.vttbr = existing->s2_cfg.vttbr;
+ smmu_domain->s2_cfg.vtcr = existing->s2_cfg.vtcr;
+ return 0;
+ }
+
+ /*
+ * If no existing stage-2 configuration for this
+ * (Xen domain, SMMU), create it
+ */
ret = arm_smmu_domain_finalise_s2(smmu_domain, master);
if (ret < 0)
return ret;
@@ -2807,20 +2863,40 @@ static uint32_t __ro_after_init platform_features = ARM_SMMU_FEAT_COHERENCY;
static int __must_check arm_smmu_iotlb_flush_all(struct domain *d)
{
struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
- struct iommu_domain *io_domain;
+ struct iommu_domain *io_domain, *prev;
spin_lock(&xen_domain->lock);
list_for_each_entry(io_domain, &xen_domain->contexts, list) {
+ struct arm_smmu_domain *smmu_domain = to_smmu_domain(io_domain);
+
/*
* Only invalidate the context when SMMU is present.
* This is because the context initialization is delayed
* until a master has been added.
*/
- if (unlikely(!ACCESS_ONCE(to_smmu_domain(io_domain)->smmu)))
+ if (unlikely(!ACCESS_ONCE(smmu_domain->smmu)))
continue;
- arm_smmu_tlb_inv_context(to_smmu_domain(io_domain));
+ /*
+ * Skip contexts that share the same SMMU and VMID with an
+ * earlier context.
+ */
+ list_for_each_entry(prev, &xen_domain->contexts, list) {
+ struct arm_smmu_domain *prev_domain = to_smmu_domain(prev);
+
+ if (prev == io_domain)
+ break;
+
+ if (prev_domain->smmu == smmu_domain->smmu &&
+ prev_domain->s2_cfg.vmid == smmu_domain->s2_cfg.vmid)
+ break;
+ }
+
+ if (prev != io_domain)
+ continue;
+
+ arm_smmu_tlb_inv_context(smmu_domain);
}
spin_unlock(&xen_domain->lock);
@@ -2866,12 +2942,22 @@ static struct iommu_domain *arm_smmu_get_domain(struct domain *d,
/*
* Loop through the &xen_domain->contexts to locate a context
- * assigned to this SMMU
+ * assigned to this device
*/
list_for_each_entry(io_domain, &xen_domain->contexts, list) {
+ unsigned long flags;
+ const struct arm_smmu_master *master;
+
smmu_domain = to_smmu_domain(io_domain);
- if (smmu_domain->smmu == smmu)
- return io_domain;
+
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ if (master->dev == dev) {
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+ return io_domain;
+ }
+ }
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
}
return NULL;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 04/22] xen/arm: vIOMMU: add generic vIOMMU framework
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (2 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 03/22] xen/arm: smmuv3: Alloc io_domain for each device Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-09 11:41 ` Oleksandr Tyshchenko
2026-10-07 0:06 ` [PATCH v4 05/22] xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests Milan Djokic
` (17 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Jan Beulich,
Roger Pau Monné, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
This patch adds basic framework for vIOMMU.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/arch/arm/dom0less-build.c | 2 +
xen/arch/arm/domain.c | 31 +++++++++++-
xen/arch/arm/domain_build.c | 2 +
xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++
xen/drivers/passthrough/Kconfig | 5 ++
xen/drivers/passthrough/arm/Makefile | 1 +
xen/drivers/passthrough/arm/viommu.c | 49 +++++++++++++++++++
xen/include/public/arch-arm.h | 6 ++-
8 files changed, 164 insertions(+), 2 deletions(-)
create mode 100644 xen/arch/arm/include/asm/viommu.h
create mode 100644 xen/drivers/passthrough/arm/viommu.c
diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
index 3f48f74226..f87437915e 100644
--- a/xen/arch/arm/dom0less-build.c
+++ b/xen/arch/arm/dom0less-build.c
@@ -23,6 +23,7 @@
#include <asm/arm64/sve.h>
#include <asm/domain_build.h>
#include <asm/firmware/sci.h>
+#include <asm/viommu.h>
#include <asm/grant_table.h>
#include <asm/setup.h>
@@ -369,6 +370,7 @@ int __init arch_parse_dom0less_node(struct dt_device_node *node,
uint32_t val;
d_cfg->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_NATIVE;
+ d_cfg->arch.viommu_type = viommu_get_type();
d_cfg->flags |= XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap;
if ( domu_dt_sci_parse(node, d_cfg) )
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index a739dd157e..5c411a2310 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -33,6 +33,7 @@
#include <asm/vgic.h>
#include <asm/vpci.h>
#include <asm/vpsci.h>
+#include <asm/viommu.h>
#include <asm/vtimer.h>
#include "vuart.h"
@@ -663,6 +664,21 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
return -EINVAL;
}
+ if ( !(config->flags & XEN_DOMCTL_CDF_iommu) &&
+ config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
+ {
+ dprintk(XENLOG_INFO,
+ "vIOMMU requested while iommu not enabled for domain\n");
+ return -EINVAL;
+ }
+
+ if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
+ {
+ dprintk(XENLOG_INFO,
+ "vIOMMU type requested not supported by the platform or Xen\n");
+ return -EINVAL;
+ }
+
return sci_domain_sanitise_config(config);
}
@@ -762,6 +778,9 @@ int arch_domain_create(struct domain *d,
if ( (rc = sci_domain_init(d, config)) != 0 )
goto fail;
+ if ( (rc = domain_viommu_init(d, config->arch.viommu_type)) != 0 )
+ goto fail;
+
return 0;
fail:
@@ -822,7 +841,8 @@ static void resume_ctx_reset(struct resume_info *ctx)
void arch_domain_destroy(struct domain *d)
{
resume_ctx_reset(&d->arch.resume_ctx);
-
+ if ( viommu_relinquish_resources(d) )
+ printk(XENLOG_INFO "%pd: Domain vIOMMU teardown failed\n", d);
tee_free_domain_ctx(d);
/* IOMMU page table is shared with P2M, always call
* iommu_domain_destroy() before p2m_final_teardown().
@@ -1058,6 +1078,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)
*/
enum {
PROG_pci = 1,
+ PROG_viommu,
PROG_sci,
PROG_tee,
PROG_xen,
@@ -1105,6 +1126,14 @@ int domain_relinquish_resources(struct domain *d)
if ( ret )
return ret;
#endif
+ /*
+ * PCI devices may rely on the vIOMMU, so its resources
+ * should be relinquished only after the PCI devices are released.
+ */
+ PROGRESS(viommu):
+ ret = viommu_relinquish_resources(d);
+ if (ret )
+ return ret;
PROGRESS(sci):
ret = sci_relinquish_resources(d);
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 72d5316180..cd2e09f005 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -36,6 +36,7 @@
#include <asm/arm64/sve.h>
#include <asm/cpufeature.h>
#include <asm/domain_build.h>
+#include <asm/viommu.h>
#include <xen/event.h>
#include <xen/irq.h>
@@ -1964,6 +1965,7 @@ void __init create_dom0(void)
dom0_cfg.arch.nr_spis = vgic_def_nr_spis();
dom0_cfg.arch.tee_type = tee_get_type();
dom0_cfg.max_vcpus = dom0_max_vcpus();
+ dom0_cfg.arch.viommu_type = viommu_get_type();
if ( iommu_enabled )
dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
new file mode 100644
index 0000000000..95f82d9cf5
--- /dev/null
+++ b/xen/arch/arm/include/asm/viommu.h
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
+#ifndef __ARCH_ARM_VIOMMU_H__
+#define __ARCH_ARM_VIOMMU_H__
+
+#include <xen/lib.h>
+#include <xen/types.h>
+#include <public/xen.h>
+
+struct viommu_ops {
+ /*
+ * Called during domain construction if toolstack requests to enable
+ * vIOMMU support.
+ */
+ int (*domain_init)(struct domain *d);
+
+ /*
+ * Called during domain destruction to free resources used by vIOMMU.
+ */
+ int (*relinquish_resources)(struct domain *d);
+};
+
+struct viommu_desc {
+ /* vIOMMU domains init/free operations described above. */
+ const struct viommu_ops *ops;
+
+ /*
+ * ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type.
+ * Should be one of XEN_DOMCTL_CONFIG_VIOMMU_xxx
+ */
+ uint8_t viommu_type;
+};
+
+#ifdef CONFIG_ARM_VIRTUAL_IOMMU
+
+int domain_viommu_init(struct domain *d, uint8_t viommu_type);
+int viommu_relinquish_resources(struct domain *d);
+uint8_t viommu_get_type(void);
+
+#else
+
+static inline uint8_t viommu_get_type(void)
+{
+ return XEN_DOMCTL_CONFIG_VIOMMU_NONE;
+}
+
+static inline int domain_viommu_init(struct domain *d, uint8_t viommu_type)
+{
+ if ( likely(viommu_type == XEN_DOMCTL_CONFIG_VIOMMU_NONE) )
+ return 0;
+
+ return -ENODEV;
+}
+
+static inline int viommu_relinquish_resources(struct domain *d)
+{
+ return 0;
+}
+
+#endif /* CONFIG_ARM_VIRTUAL_IOMMU */
+
+#endif /* __ARCH_ARM_VIOMMU_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/Kconfig b/xen/drivers/passthrough/Kconfig
index b413c33a4c..a92cfbea98 100644
--- a/xen/drivers/passthrough/Kconfig
+++ b/xen/drivers/passthrough/Kconfig
@@ -35,6 +35,11 @@ config IPMMU_VMSA
(H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2
translation table format and is able to use CPU's P2M table as is.
+config ARM_VIRTUAL_IOMMU
+ bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED
+ help
+ Support virtual IOMMU infrastructure to implement vIOMMU.
+
endif
config AMD_IOMMU
diff --git a/xen/drivers/passthrough/arm/Makefile b/xen/drivers/passthrough/arm/Makefile
index c5fb3b58a5..c3783188e3 100644
--- a/xen/drivers/passthrough/arm/Makefile
+++ b/xen/drivers/passthrough/arm/Makefile
@@ -2,3 +2,4 @@ obj-y += iommu.o iommu_helpers.o iommu_fwspec.o
obj-$(CONFIG_ARM_SMMU) += smmu.o
obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o
+obj-$(CONFIG_ARM_VIRTUAL_IOMMU) += viommu.o
diff --git a/xen/drivers/passthrough/arm/viommu.c b/xen/drivers/passthrough/arm/viommu.c
new file mode 100644
index 0000000000..0c0efd668e
--- /dev/null
+++ b/xen/drivers/passthrough/arm/viommu.c
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
+
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/types.h>
+
+#include <asm/viommu.h>
+
+/* Struct to hold the vIOMMU ops and vIOMMU type */
+static const struct viommu_desc __ro_after_init *cur_viommu;
+
+int domain_viommu_init(struct domain *d, uint8_t viommu_type)
+{
+ if ( viommu_type == XEN_DOMCTL_CONFIG_VIOMMU_NONE )
+ return 0;
+
+ if ( !cur_viommu )
+ return -ENODEV;
+
+ if ( cur_viommu->viommu_type != viommu_type )
+ return -EINVAL;
+
+ return cur_viommu->ops->domain_init(d);
+}
+
+int viommu_relinquish_resources(struct domain *d)
+{
+ if ( !cur_viommu )
+ return 0;
+
+ return cur_viommu->ops->relinquish_resources(d);
+}
+
+uint8_t viommu_get_type(void)
+{
+ if ( !cur_viommu )
+ return XEN_DOMCTL_CONFIG_VIOMMU_NONE;
+
+ return cur_viommu->viommu_type;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 00de30b896..a1491f9cf7 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -334,6 +334,8 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
#define XEN_DOMCTL_CONFIG_ARM_V8R_EL1_MSA_PMSA 1
#define XEN_DOMCTL_CONFIG_ARM_V8R_EL1_MSA_VMSA 2
+#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0
+
struct xen_arch_domainconfig {
/* IN/OUT */
uint8_t gic_version;
@@ -361,7 +363,9 @@ struct xen_arch_domainconfig {
uint8_t arm_sci_type;
/* IN */
uint8_t v8r_el1_msa;
- uint16_t pad;
+ /* IN */
+ uint8_t viommu_type;
+ uint8_t pad;
};
#endif /* __XEN__ || __XEN_TOOLS__ */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 04/22] xen/arm: vIOMMU: add generic vIOMMU framework
2026-10-07 0:06 ` [PATCH v4 04/22] xen/arm: vIOMMU: add generic vIOMMU framework Milan Djokic
@ 2026-10-09 11:41 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-09 11:41 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Jan Beulich,
Roger Pau Monné
On 10/7/26 03:06, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> This patch adds basic framework for vIOMMU.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/arch/arm/dom0less-build.c | 2 +
> xen/arch/arm/domain.c | 31 +++++++++++-
> xen/arch/arm/domain_build.c | 2 +
> xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++
> xen/drivers/passthrough/Kconfig | 5 ++
> xen/drivers/passthrough/arm/Makefile | 1 +
> xen/drivers/passthrough/arm/viommu.c | 49 +++++++++++++++++++
> xen/include/public/arch-arm.h | 6 ++-
> 8 files changed, 164 insertions(+), 2 deletions(-)
> create mode 100644 xen/arch/arm/include/asm/viommu.h
> create mode 100644 xen/drivers/passthrough/arm/viommu.c
>
> diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
> index 3f48f74226..f87437915e 100644
> --- a/xen/arch/arm/dom0less-build.c
> +++ b/xen/arch/arm/dom0less-build.c
> @@ -23,6 +23,7 @@
> #include <asm/arm64/sve.h>
> #include <asm/domain_build.h>
> #include <asm/firmware/sci.h>
> +#include <asm/viommu.h>
> #include <asm/grant_table.h>
> #include <asm/setup.h>
>
> @@ -369,6 +370,7 @@ int __init arch_parse_dom0less_node(struct dt_device_node *node,
> uint32_t val;
>
> d_cfg->arch.gic_version = XEN_DOMCTL_CONFIG_GIC_NATIVE;
> + d_cfg->arch.viommu_type = viommu_get_type();
> d_cfg->flags |= XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap;
>
> if ( domu_dt_sci_parse(node, d_cfg) )
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index a739dd157e..5c411a2310 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -33,6 +33,7 @@
> #include <asm/vgic.h>
> #include <asm/vpci.h>
> #include <asm/vpsci.h>
> +#include <asm/viommu.h>
> #include <asm/vtimer.h>
>
> #include "vuart.h"
> @@ -663,6 +664,21 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
> return -EINVAL;
> }
>
> + if ( !(config->flags & XEN_DOMCTL_CDF_iommu) &&
> + config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
> + {
> + dprintk(XENLOG_INFO,
> + "vIOMMU requested while iommu not enabled for domain\n");
> + return -EINVAL;
> + }
I might miss something, but if CONFIG_ARM_VIRTUAL_IOMMU=y and we have
SMMUv3 the dom0less domains "without passthrough" will fail to build.
arch_parse_dom0less_node() sets viommu_type for every guest. But
CDF_iommu is only set when passthrough = "enabled" or the node has a
multiboot DTB. So a plain guest without passthrough is therefore
rejected here.
> +
> + if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
> + {
> + dprintk(XENLOG_INFO,
> + "vIOMMU type requested not supported by the platform or Xen\n");
> + return -EINVAL;
> + }
> +
> return sci_domain_sanitise_config(config);
> }
>
> @@ -762,6 +778,9 @@ int arch_domain_create(struct domain *d,
> if ( (rc = sci_domain_init(d, config)) != 0 )
> goto fail;
>
> + if ( (rc = domain_viommu_init(d, config->arch.viommu_type)) != 0 )
> + goto fail;
> +
> return 0;
>
> fail:
> @@ -822,7 +841,8 @@ static void resume_ctx_reset(struct resume_info *ctx)
> void arch_domain_destroy(struct domain *d)
> {
> resume_ctx_reset(&d->arch.resume_ctx);
> -
> + if ( viommu_relinquish_resources(d) )
> + printk(XENLOG_INFO "%pd: Domain vIOMMU teardown failed\n", d);
> tee_free_domain_ctx(d);
> /* IOMMU page table is shared with P2M, always call
> * iommu_domain_destroy() before p2m_final_teardown().
> @@ -1058,6 +1078,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)
> */
> enum {
> PROG_pci = 1,
> + PROG_viommu,
> PROG_sci,
> PROG_tee,
> PROG_xen,
> @@ -1105,6 +1126,14 @@ int domain_relinquish_resources(struct domain *d)
> if ( ret )
> return ret;
> #endif
> + /*
> + * PCI devices may rely on the vIOMMU, so its resources
> + * should be relinquished only after the PCI devices are released.
> + */
> + PROGRESS(viommu):
> + ret = viommu_relinquish_resources(d);
> + if (ret )
> + return ret;
>
> PROGRESS(sci):
> ret = sci_relinquish_resources(d);
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 72d5316180..cd2e09f005 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -36,6 +36,7 @@
> #include <asm/arm64/sve.h>
> #include <asm/cpufeature.h>
> #include <asm/domain_build.h>
> +#include <asm/viommu.h>
> #include <xen/event.h>
>
> #include <xen/irq.h>
> @@ -1964,6 +1965,7 @@ void __init create_dom0(void)
> dom0_cfg.arch.nr_spis = vgic_def_nr_spis();
> dom0_cfg.arch.tee_type = tee_get_type();
> dom0_cfg.max_vcpus = dom0_max_vcpus();
> + dom0_cfg.arch.viommu_type = viommu_get_type();
>
> if ( iommu_enabled )
> dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
> diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
> new file mode 100644
> index 0000000000..95f82d9cf5
> --- /dev/null
> +++ b/xen/arch/arm/include/asm/viommu.h
> @@ -0,0 +1,70 @@
> +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
> +#ifndef __ARCH_ARM_VIOMMU_H__
> +#define __ARCH_ARM_VIOMMU_H__
> +
> +#include <xen/lib.h>
> +#include <xen/types.h>
> +#include <public/xen.h>
> +
> +struct viommu_ops {
> + /*
> + * Called during domain construction if toolstack requests to enable
> + * vIOMMU support.
> + */
> + int (*domain_init)(struct domain *d);
> +
> + /*
> + * Called during domain destruction to free resources used by vIOMMU.
> + */
> + int (*relinquish_resources)(struct domain *d);
> +};
> +
> +struct viommu_desc {
> + /* vIOMMU domains init/free operations described above. */
> + const struct viommu_ops *ops;
> +
> + /*
> + * ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type.
> + * Should be one of XEN_DOMCTL_CONFIG_VIOMMU_xxx
> + */
> + uint8_t viommu_type;
> +};
> +
> +#ifdef CONFIG_ARM_VIRTUAL_IOMMU
> +
> +int domain_viommu_init(struct domain *d, uint8_t viommu_type);
> +int viommu_relinquish_resources(struct domain *d);
> +uint8_t viommu_get_type(void);
> +
> +#else
> +
> +static inline uint8_t viommu_get_type(void)
> +{
> + return XEN_DOMCTL_CONFIG_VIOMMU_NONE;
> +}
> +
> +static inline int domain_viommu_init(struct domain *d, uint8_t viommu_type)
> +{
> + if ( likely(viommu_type == XEN_DOMCTL_CONFIG_VIOMMU_NONE) )
> + return 0;
> +
> + return -ENODEV;
> +}
> +
> +static inline int viommu_relinquish_resources(struct domain *d)
> +{
> + return 0;
> +}
> +
> +#endif /* CONFIG_ARM_VIRTUAL_IOMMU */
> +
> +#endif /* __ARCH_ARM_VIOMMU_H__ */
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/Kconfig b/xen/drivers/passthrough/Kconfig
> index b413c33a4c..a92cfbea98 100644
> --- a/xen/drivers/passthrough/Kconfig
> +++ b/xen/drivers/passthrough/Kconfig
> @@ -35,6 +35,11 @@ config IPMMU_VMSA
> (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2
> translation table format and is able to use CPU's P2M table as is.
>
> +config ARM_VIRTUAL_IOMMU
> + bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED
> + help
> + Support virtual IOMMU infrastructure to implement vIOMMU.
> +
> endif
>
> config AMD_IOMMU
> diff --git a/xen/drivers/passthrough/arm/Makefile b/xen/drivers/passthrough/arm/Makefile
> index c5fb3b58a5..c3783188e3 100644
> --- a/xen/drivers/passthrough/arm/Makefile
> +++ b/xen/drivers/passthrough/arm/Makefile
> @@ -2,3 +2,4 @@ obj-y += iommu.o iommu_helpers.o iommu_fwspec.o
> obj-$(CONFIG_ARM_SMMU) += smmu.o
> obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
> obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o
> +obj-$(CONFIG_ARM_VIRTUAL_IOMMU) += viommu.o
> diff --git a/xen/drivers/passthrough/arm/viommu.c b/xen/drivers/passthrough/arm/viommu.c
> new file mode 100644
> index 0000000000..0c0efd668e
> --- /dev/null
> +++ b/xen/drivers/passthrough/arm/viommu.c
> @@ -0,0 +1,49 @@
> +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
> +
> +#include <xen/errno.h>
> +#include <xen/init.h>
> +#include <xen/types.h>
> +
> +#include <asm/viommu.h>
> +
> +/* Struct to hold the vIOMMU ops and vIOMMU type */
> +static const struct viommu_desc __ro_after_init *cur_viommu;
> +
> +int domain_viommu_init(struct domain *d, uint8_t viommu_type)
> +{
> + if ( viommu_type == XEN_DOMCTL_CONFIG_VIOMMU_NONE )
> + return 0;
> +
> + if ( !cur_viommu )
> + return -ENODEV;
> +
> + if ( cur_viommu->viommu_type != viommu_type )
> + return -EINVAL;
> +
> + return cur_viommu->ops->domain_init(d);
> +}
> +
> +int viommu_relinquish_resources(struct domain *d)
> +{
> + if ( !cur_viommu )
> + return 0;
> +
> + return cur_viommu->ops->relinquish_resources(d);
> +}
> +
> +uint8_t viommu_get_type(void)
> +{
> + if ( !cur_viommu )
> + return XEN_DOMCTL_CONFIG_VIOMMU_NONE;
> +
> + return cur_viommu->viommu_type;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 00de30b896..a1491f9cf7 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -334,6 +334,8 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> #define XEN_DOMCTL_CONFIG_ARM_V8R_EL1_MSA_PMSA 1
> #define XEN_DOMCTL_CONFIG_ARM_V8R_EL1_MSA_VMSA 2
>
> +#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0
> +
> struct xen_arch_domainconfig {
> /* IN/OUT */
> uint8_t gic_version;
> @@ -361,7 +363,9 @@ struct xen_arch_domainconfig {
> uint8_t arm_sci_type;
> /* IN */
> uint8_t v8r_el1_msa;
> - uint16_t pad;
> + /* IN */
> + uint8_t viommu_type;
> + uint8_t pad;
> };
> #endif /* __XEN__ || __XEN_TOOLS__ */
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 05/22] xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (3 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 04/22] xen/arm: vIOMMU: add generic vIOMMU framework Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-11 14:06 ` Oleksandr Tyshchenko
2026-10-07 0:06 ` [PATCH v4 06/22] tools: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param Milan Djokic
` (16 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Jan Beulich,
Roger Pau Monné, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
domain_viommu_init() will be called during domain creation and will add
the dummy trap handler for virtual IOMMUs for guests.
A host IOMMU list will be created when host IOMMU devices are probed
and this list will be used to create the IOMMU device tree node for
dom0. For dom0, 1-1 mapping will be established between vIOMMU in dom0
and physical IOMMU.
For domUs, the 1-N mapping will be established between domU and physical
IOMMUs. A new area has been reserved in the arm guest physical map at
which the emulated vIOMMU node is created in the device tree.
Also set the vIOMMU type to vSMMUv3 to enable vIOMMU framework to call
vSMMUv3 domain creation/destroy functions.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/arch/arm/domain.c | 6 +-
xen/arch/arm/include/asm/domain.h | 5 ++
xen/arch/arm/include/asm/viommu.h | 36 ++++++++
xen/drivers/passthrough/Kconfig | 8 ++
xen/drivers/passthrough/arm/Makefile | 1 +
xen/drivers/passthrough/arm/smmu-v3.c | 7 ++
xen/drivers/passthrough/arm/viommu.c | 83 ++++++++++++++++-
xen/drivers/passthrough/arm/vsmmu-v3.c | 118 +++++++++++++++++++++++++
xen/drivers/passthrough/arm/vsmmu-v3.h | 20 +++++
xen/include/public/arch-arm.h | 5 ++
10 files changed, 287 insertions(+), 2 deletions(-)
create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.c
create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.h
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 5c411a2310..003a8d45b9 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -672,7 +672,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
return -EINVAL;
}
- if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
+ if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE &&
+ config->arch.viommu_type != viommu_get_type() )
{
dprintk(XENLOG_INFO,
"vIOMMU type requested not supported by the platform or Xen\n");
@@ -728,6 +729,9 @@ int arch_domain_create(struct domain *d,
count += domain_vpci_get_num_mmio_handlers(d);
+ if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
+ count += domain_viommu_get_num_mmio_handlers(d);
+
if ( (rc = domain_io_init(d, count + MAX_IO_HANDLER)) != 0 )
goto fail;
diff --git a/xen/arch/arm/include/asm/domain.h b/xen/arch/arm/include/asm/domain.h
index c3d2c4981f..70af0bf8b7 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -102,6 +102,11 @@ struct arch_domain
#ifdef CONFIG_MPU
uint8_t v8r_el1_msa;
#endif
+
+#ifdef CONFIG_ARM_VIRTUAL_IOMMU
+ struct list_head viommu_list; /* List of virtual IOMMUs */
+#endif
+
} __cacheline_aligned;
struct arch_vcpu
diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
index 95f82d9cf5..287595068b 100644
--- a/xen/arch/arm/include/asm/viommu.h
+++ b/xen/arch/arm/include/asm/viommu.h
@@ -3,9 +3,23 @@
#define __ARCH_ARM_VIOMMU_H__
#include <xen/lib.h>
+#include <xen/list.h>
#include <xen/types.h>
#include <public/xen.h>
+/* data structure for each hardware IOMMU */
+struct host_iommu {
+ struct list_head entry;
+ /* IOMMU DT node used to establish virtual to physical IOMMU/SID mapping */
+ const struct dt_device_node *dt_node;
+ /* Used for dom0 vIOMMU instantiation (1-1 MMIO regions mappings) */
+ paddr_t addr;
+ /* Used for dom0 vIOMMU instantiation (1-1 MMIO regions mappings) */
+ paddr_t size;
+ /* Used for dom0 EVTQ and GERROR interrupts (1-1 IRQ mapping) */
+ uint32_t irq;
+};
+
struct viommu_ops {
/*
* Called during domain construction if toolstack requests to enable
@@ -32,9 +46,15 @@ struct viommu_desc {
#ifdef CONFIG_ARM_VIRTUAL_IOMMU
+extern struct list_head host_iommu_list;
+
int domain_viommu_init(struct domain *d, uint8_t viommu_type);
int viommu_relinquish_resources(struct domain *d);
uint8_t viommu_get_type(void);
+void add_to_host_iommu_list(paddr_t addr, paddr_t size,
+ const struct dt_device_node *node);
+void set_cur_viommu(const struct viommu_desc *desc);
+unsigned int domain_viommu_get_num_mmio_handlers(struct domain *d);
#else
@@ -56,6 +76,22 @@ static inline int viommu_relinquish_resources(struct domain *d)
return 0;
}
+static inline void __init add_to_host_iommu_list(paddr_t addr, paddr_t size,
+ const struct dt_device_node *node)
+{
+ return;
+}
+
+static inline unsigned int domain_viommu_get_num_mmio_handlers(struct domain *d)
+{
+ return 0;
+}
+
+static inline void __init set_cur_viommu(const struct viommu_desc *desc)
+{
+ return;
+}
+
#endif /* CONFIG_ARM_VIRTUAL_IOMMU */
#endif /* __ARCH_ARM_VIOMMU_H__ */
diff --git a/xen/drivers/passthrough/Kconfig b/xen/drivers/passthrough/Kconfig
index a92cfbea98..40d072822f 100644
--- a/xen/drivers/passthrough/Kconfig
+++ b/xen/drivers/passthrough/Kconfig
@@ -40,6 +40,14 @@ config ARM_VIRTUAL_IOMMU
help
Support virtual IOMMU infrastructure to implement vIOMMU.
+config VIRTUAL_ARM_SMMU_V3
+ bool "ARM Ltd. Virtual SMMUv3 Support (UNSUPPORTED)"
+ depends on ARM_SMMU_V3 && ARM_VIRTUAL_IOMMU
+ help
+ Support for implementations of the virtual ARM System MMU architecture
+ version 3. Virtual SMMUv3 is unsupported feature and should not be used
+ in production.
+
endif
config AMD_IOMMU
diff --git a/xen/drivers/passthrough/arm/Makefile b/xen/drivers/passthrough/arm/Makefile
index c3783188e3..c8f0a5f802 100644
--- a/xen/drivers/passthrough/arm/Makefile
+++ b/xen/drivers/passthrough/arm/Makefile
@@ -3,3 +3,4 @@ obj-$(CONFIG_ARM_SMMU) += smmu.o
obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o
obj-$(CONFIG_ARM_VIRTUAL_IOMMU) += viommu.o
+obj-$(CONFIG_VIRTUAL_ARM_SMMU_V3) += vsmmu-v3.o
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index bd0689664b..21b82eba85 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -93,6 +93,7 @@
#include <asm/platform.h>
#include "smmu-v3.h"
+#include "vsmmu-v3.h"
#define ARM_SMMU_VTCR_SH_IS 3
#define ARM_SMMU_VTCR_RGN_WBWA 1
@@ -2829,6 +2830,9 @@ static int __init arm_smmu_device_probe(struct platform_device *pdev)
list_add(&smmu->devices, &arm_smmu_devices);
spin_unlock(&arm_smmu_devices_lock);
+ /* Add to host IOMMU list to initialize vIOMMU for dom0 */
+ add_to_host_iommu_list(ioaddr, iosize, dev_to_dt(pdev));
+
return 0;
@@ -3181,6 +3185,9 @@ static __init int arm_smmu_dt_init(struct dt_device_node *dev,
platform_features &= smmu->features;
+ /* Set vIOMMU type to SMMUv3 */
+ vsmmuv3_set_type();
+
return 0;
}
diff --git a/xen/drivers/passthrough/arm/viommu.c b/xen/drivers/passthrough/arm/viommu.c
index 0c0efd668e..3401e4f7d9 100644
--- a/xen/drivers/passthrough/arm/viommu.c
+++ b/xen/drivers/passthrough/arm/viommu.c
@@ -2,12 +2,54 @@
#include <xen/errno.h>
#include <xen/init.h>
+#include <xen/irq.h>
#include <xen/types.h>
+#include <xen/sched.h>
#include <asm/viommu.h>
+/* List of all host IOMMUs */
+LIST_HEAD(host_iommu_list);
+
/* Struct to hold the vIOMMU ops and vIOMMU type */
static const struct viommu_desc __ro_after_init *cur_viommu;
+static bool host_iommu_list_incomplete;
+
+/* Common function for adding to host_iommu_list */
+void __init add_to_host_iommu_list(paddr_t addr, paddr_t size,
+ const struct dt_device_node *node)
+{
+ struct host_iommu *iommu_data;
+ int irq;
+
+ iommu_data = xzalloc(struct host_iommu);
+ if ( !iommu_data )
+ {
+ host_iommu_list_incomplete = true;
+ printk(XENLOG_WARNING
+ "vIOMMU: Cannot allocate memory for host IOMMU data\n");
+ return;
+ }
+
+ iommu_data->addr = addr;
+ iommu_data->size = size;
+ iommu_data->dt_node = node;
+ irq = platform_get_irq(node, 0);
+ if ( irq < 0 )
+ {
+ host_iommu_list_incomplete = true;
+ printk(XENLOG_WARNING
+ "vIOMMU: disabling support because IOMMU @0x%"PRIx64
+ " has no usable IRQ (%d)\n", addr, irq);
+ xfree(iommu_data);
+ return;
+ }
+ iommu_data->irq = irq;
+
+ printk("vIOMMU: Found IOMMU @0x%"PRIx64"\n", addr);
+
+ list_add_tail(&iommu_data->entry, &host_iommu_list);
+}
int domain_viommu_init(struct domain *d, uint8_t viommu_type)
{
@@ -33,12 +75,51 @@ int viommu_relinquish_resources(struct domain *d)
uint8_t viommu_get_type(void)
{
- if ( !cur_viommu )
+ if ( !cur_viommu || host_iommu_list_incomplete ||
+ list_empty(&host_iommu_list) )
return XEN_DOMCTL_CONFIG_VIOMMU_NONE;
return cur_viommu->viommu_type;
}
+void __init set_cur_viommu(const struct viommu_desc *desc)
+{
+ if ( cur_viommu && (cur_viommu != desc) )
+ {
+ printk("WARNING: Cannot set vIOMMU, already set to a different value\n");
+ return;
+ }
+
+ cur_viommu = desc;
+}
+
+unsigned int domain_viommu_get_num_mmio_handlers(struct domain *d)
+{
+ if ( is_hardware_domain(d) )
+ {
+ struct host_iommu *hw_iommu;
+ int count = 0;
+
+ /*
+ * For hardware domain, vIOMMU per pIOMMU is exposed. Number of
+ * regions required for the configuration space is equal to number of
+ * host IOMMUs.
+ */
+ list_for_each_entry(hw_iommu, &host_iommu_list, entry)
+ {
+ count++;
+ }
+
+ return count;
+ }
+
+ /*
+ * For unpriviliged domains, single vIOMMU is exposed and requires one
+ * region to cover the configuration space.
+ */
+ return 1;
+}
+
/*
* Local variables:
* mode: C
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
new file mode 100644
index 0000000000..ea3b5bb34d
--- /dev/null
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -0,0 +1,118 @@
+/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
+
+#include <xen/param.h>
+#include <xen/sched.h>
+#include <asm/mmio.h>
+#include <asm/viommu.h>
+
+struct virt_smmu {
+ struct domain *d;
+ struct list_head viommu_list;
+};
+
+static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
+ register_t r, void *priv)
+{
+ BUG_ON("unimplemented");
+ return IO_HANDLED;
+}
+
+static int vsmmuv3_mmio_read(struct vcpu *v, mmio_info_t *info,
+ register_t *r, void *priv)
+{
+ BUG_ON("unimplemented");
+ return IO_HANDLED;
+}
+
+static const struct mmio_handler_ops vsmmuv3_mmio_handler = {
+ .read = vsmmuv3_mmio_read,
+ .write = vsmmuv3_mmio_write,
+};
+
+static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size)
+{
+ struct virt_smmu *smmu;
+
+ smmu = xzalloc(struct virt_smmu);
+ if ( !smmu )
+ return -ENOMEM;
+
+ smmu->d = d;
+
+ register_mmio_handler(d, &vsmmuv3_mmio_handler, addr, size, smmu);
+
+ /* Register the vIOMMU to be able to clean it up later. */
+ list_add_tail(&smmu->viommu_list, &d->arch.viommu_list);
+
+ return 0;
+}
+
+static int domain_vsmmuv3_init(struct domain *d)
+{
+ int ret;
+
+ INIT_LIST_HEAD(&d->arch.viommu_list);
+
+ if ( is_hardware_domain(d) )
+ {
+ struct host_iommu *hw_iommu;
+
+ list_for_each_entry(hw_iommu, &host_iommu_list, entry)
+ {
+ ret = vsmmuv3_init_single(d, hw_iommu->addr, hw_iommu->size);
+ if ( ret )
+ return ret;
+ }
+ }
+ else
+ {
+ ret = vsmmuv3_init_single(d, GUEST_VSMMUV3_BASE, GUEST_VSMMUV3_SIZE);
+ if ( ret )
+ return ret;
+ }
+
+ return 0;
+}
+
+static int vsmmuv3_relinquish_resources(struct domain *d)
+{
+ struct virt_smmu *pos, *temp;
+
+ /* Cope with uninitialized vIOMMU */
+ if ( list_head_is_null(&d->arch.viommu_list) )
+ return 0;
+
+ list_for_each_entry_safe(pos, temp, &d->arch.viommu_list, viommu_list )
+ {
+ list_del(&pos->viommu_list);
+ xfree(pos);
+ }
+
+ return 0;
+}
+
+static const struct viommu_ops vsmmuv3_ops = {
+ .domain_init = domain_vsmmuv3_init,
+ .relinquish_resources = vsmmuv3_relinquish_resources,
+};
+
+static const struct viommu_desc vsmmuv3_desc = {
+ .ops = &vsmmuv3_ops,
+ .viommu_type = XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3,
+};
+
+void __init vsmmuv3_set_type(void)
+{
+ const struct viommu_desc *desc = &vsmmuv3_desc;
+
+ set_cur_viommu(desc);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.h b/xen/drivers/passthrough/arm/vsmmu-v3.h
new file mode 100644
index 0000000000..9f4c86abd5
--- /dev/null
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
+#ifndef __ARCH_ARM_VSMMU_V3_H__
+#define __ARCH_ARM_VSMMU_V3_H__
+
+#include <asm/viommu.h>
+
+#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3
+
+void vsmmuv3_set_type(void);
+
+#else
+
+static inline void __init vsmmuv3_set_type(void)
+{
+ return;
+}
+
+#endif /* CONFIG_VIRTUAL_ARM_SMMU_V3 */
+
+#endif /* __ARCH_ARM_VSMMU_V3_H__ */
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index a1491f9cf7..8863e2e457 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -335,6 +335,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
#define XEN_DOMCTL_CONFIG_ARM_V8R_EL1_MSA_VMSA 2
#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0
+#define XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 1
struct xen_arch_domainconfig {
/* IN/OUT */
@@ -463,6 +464,10 @@ typedef uint64_t xen_callback_t;
#define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
#define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
+/* vsmmuv3 MMIO mappings */
+#define GUEST_VSMMUV3_BASE xen_mk_ullong(0x04040000)
+#define GUEST_VSMMUV3_SIZE xen_mk_ullong(0x00020000)
+
/*
* 256 MB is reserved for VPCI configuration space based on calculation
* 256 buses x 32 devices x 8 functions x 4 KB = 256 MB
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 05/22] xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests
2026-10-07 0:06 ` [PATCH v4 05/22] xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests Milan Djokic
@ 2026-10-11 14:06 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-11 14:06 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Jan Beulich,
Roger Pau Monné
On 10/7/26 03:06, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> domain_viommu_init() will be called during domain creation and will add
> the dummy trap handler for virtual IOMMUs for guests.
>
> A host IOMMU list will be created when host IOMMU devices are probed
> and this list will be used to create the IOMMU device tree node for
> dom0. For dom0, 1-1 mapping will be established between vIOMMU in dom0
> and physical IOMMU.
>
> For domUs, the 1-N mapping will be established between domU and physical
> IOMMUs. A new area has been reserved in the arm guest physical map at
> which the emulated vIOMMU node is created in the device tree.
>
> Also set the vIOMMU type to vSMMUv3 to enable vIOMMU framework to call
> vSMMUv3 domain creation/destroy functions.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/arch/arm/domain.c | 6 +-
> xen/arch/arm/include/asm/domain.h | 5 ++
> xen/arch/arm/include/asm/viommu.h | 36 ++++++++
> xen/drivers/passthrough/Kconfig | 8 ++
> xen/drivers/passthrough/arm/Makefile | 1 +
> xen/drivers/passthrough/arm/smmu-v3.c | 7 ++
> xen/drivers/passthrough/arm/viommu.c | 83 ++++++++++++++++-
> xen/drivers/passthrough/arm/vsmmu-v3.c | 118 +++++++++++++++++++++++++
> xen/drivers/passthrough/arm/vsmmu-v3.h | 20 +++++
> xen/include/public/arch-arm.h | 5 ++
> 10 files changed, 287 insertions(+), 2 deletions(-)
> create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.c
> create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.h
>
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 5c411a2310..003a8d45b9 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -672,7 +672,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
> return -EINVAL;
> }
>
> - if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
> + if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE &&
> + config->arch.viommu_type != viommu_get_type() )
> {
> dprintk(XENLOG_INFO,
> "vIOMMU type requested not supported by the platform or Xen\n");
> @@ -728,6 +729,9 @@ int arch_domain_create(struct domain *d,
>
> count += domain_vpci_get_num_mmio_handlers(d);
>
> + if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
> + count += domain_viommu_get_num_mmio_handlers(d);
> +
> if ( (rc = domain_io_init(d, count + MAX_IO_HANDLER)) != 0 )
> goto fail;
>
> diff --git a/xen/arch/arm/include/asm/domain.h b/xen/arch/arm/include/asm/domain.h
> index c3d2c4981f..70af0bf8b7 100644
> --- a/xen/arch/arm/include/asm/domain.h
> +++ b/xen/arch/arm/include/asm/domain.h
> @@ -102,6 +102,11 @@ struct arch_domain
> #ifdef CONFIG_MPU
> uint8_t v8r_el1_msa;
> #endif
> +
> +#ifdef CONFIG_ARM_VIRTUAL_IOMMU
> + struct list_head viommu_list; /* List of virtual IOMMUs */
> +#endif
> +
> } __cacheline_aligned;
>
> struct arch_vcpu
> diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
> index 95f82d9cf5..287595068b 100644
> --- a/xen/arch/arm/include/asm/viommu.h
> +++ b/xen/arch/arm/include/asm/viommu.h
> @@ -3,9 +3,23 @@
> #define __ARCH_ARM_VIOMMU_H__
>
> #include <xen/lib.h>
> +#include <xen/list.h>
> #include <xen/types.h>
> #include <public/xen.h>
>
> +/* data structure for each hardware IOMMU */
> +struct host_iommu {
> + struct list_head entry;
> + /* IOMMU DT node used to establish virtual to physical IOMMU/SID mapping */
> + const struct dt_device_node *dt_node;
> + /* Used for dom0 vIOMMU instantiation (1-1 MMIO regions mappings) */
> + paddr_t addr;
> + /* Used for dom0 vIOMMU instantiation (1-1 MMIO regions mappings) */
> + paddr_t size;
> + /* Used for dom0 EVTQ and GERROR interrupts (1-1 IRQ mapping) */
> + uint32_t irq;
> +};
> +
> struct viommu_ops {
> /*
> * Called during domain construction if toolstack requests to enable
> @@ -32,9 +46,15 @@ struct viommu_desc {
>
> #ifdef CONFIG_ARM_VIRTUAL_IOMMU
>
> +extern struct list_head host_iommu_list;
> +
> int domain_viommu_init(struct domain *d, uint8_t viommu_type);
> int viommu_relinquish_resources(struct domain *d);
> uint8_t viommu_get_type(void);
> +void add_to_host_iommu_list(paddr_t addr, paddr_t size,
> + const struct dt_device_node *node);
> +void set_cur_viommu(const struct viommu_desc *desc);
> +unsigned int domain_viommu_get_num_mmio_handlers(struct domain *d);
>
> #else
>
> @@ -56,6 +76,22 @@ static inline int viommu_relinquish_resources(struct domain *d)
> return 0;
> }
>
> +static inline void __init add_to_host_iommu_list(paddr_t addr, paddr_t size,
> + const struct dt_device_node *node)
> +{
> + return;
> +}
> +
> +static inline unsigned int domain_viommu_get_num_mmio_handlers(struct domain *d)
> +{
> + return 0;
> +}
> +
> +static inline void __init set_cur_viommu(const struct viommu_desc *desc)
> +{
> + return;
> +}
> +
> #endif /* CONFIG_ARM_VIRTUAL_IOMMU */
>
> #endif /* __ARCH_ARM_VIOMMU_H__ */
> diff --git a/xen/drivers/passthrough/Kconfig b/xen/drivers/passthrough/Kconfig
> index a92cfbea98..40d072822f 100644
> --- a/xen/drivers/passthrough/Kconfig
> +++ b/xen/drivers/passthrough/Kconfig
> @@ -40,6 +40,14 @@ config ARM_VIRTUAL_IOMMU
> help
> Support virtual IOMMU infrastructure to implement vIOMMU.
>
> +config VIRTUAL_ARM_SMMU_V3
> + bool "ARM Ltd. Virtual SMMUv3 Support (UNSUPPORTED)"
> + depends on ARM_SMMU_V3 && ARM_VIRTUAL_IOMMU
> + help
> + Support for implementations of the virtual ARM System MMU architecture
> + version 3. Virtual SMMUv3 is unsupported feature and should not be used
> + in production.
> +
> endif
>
> config AMD_IOMMU
> diff --git a/xen/drivers/passthrough/arm/Makefile b/xen/drivers/passthrough/arm/Makefile
> index c3783188e3..c8f0a5f802 100644
> --- a/xen/drivers/passthrough/arm/Makefile
> +++ b/xen/drivers/passthrough/arm/Makefile
> @@ -3,3 +3,4 @@ obj-$(CONFIG_ARM_SMMU) += smmu.o
> obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
> obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o
> obj-$(CONFIG_ARM_VIRTUAL_IOMMU) += viommu.o
> +obj-$(CONFIG_VIRTUAL_ARM_SMMU_V3) += vsmmu-v3.o
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
> index bd0689664b..21b82eba85 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.c
> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
> @@ -93,6 +93,7 @@
> #include <asm/platform.h>
>
> #include "smmu-v3.h"
> +#include "vsmmu-v3.h"
>
> #define ARM_SMMU_VTCR_SH_IS 3
> #define ARM_SMMU_VTCR_RGN_WBWA 1
> @@ -2829,6 +2830,9 @@ static int __init arm_smmu_device_probe(struct platform_device *pdev)
> list_add(&smmu->devices, &arm_smmu_devices);
> spin_unlock(&arm_smmu_devices_lock);
>
> + /* Add to host IOMMU list to initialize vIOMMU for dom0 */
> + add_to_host_iommu_list(ioaddr, iosize, dev_to_dt(pdev));
> +
> return 0;
>
>
> @@ -3181,6 +3185,9 @@ static __init int arm_smmu_dt_init(struct dt_device_node *dev,
>
> platform_features &= smmu->features;
>
> + /* Set vIOMMU type to SMMUv3 */
> + vsmmuv3_set_type();
> +
> return 0;
> }
>
> diff --git a/xen/drivers/passthrough/arm/viommu.c b/xen/drivers/passthrough/arm/viommu.c
> index 0c0efd668e..3401e4f7d9 100644
> --- a/xen/drivers/passthrough/arm/viommu.c
> +++ b/xen/drivers/passthrough/arm/viommu.c
> @@ -2,12 +2,54 @@
>
> #include <xen/errno.h>
> #include <xen/init.h>
> +#include <xen/irq.h>
> #include <xen/types.h>
> +#include <xen/sched.h>
>
> #include <asm/viommu.h>
>
> +/* List of all host IOMMUs */
> +LIST_HEAD(host_iommu_list);
> +
> /* Struct to hold the vIOMMU ops and vIOMMU type */
> static const struct viommu_desc __ro_after_init *cur_viommu;
> +static bool host_iommu_list_incomplete;
> +
> +/* Common function for adding to host_iommu_list */
> +void __init add_to_host_iommu_list(paddr_t addr, paddr_t size,
> + const struct dt_device_node *node)
> +{
> + struct host_iommu *iommu_data;
> + int irq;
> +
> + iommu_data = xzalloc(struct host_iommu);
> + if ( !iommu_data )
> + {
> + host_iommu_list_incomplete = true;
> + printk(XENLOG_WARNING
> + "vIOMMU: Cannot allocate memory for host IOMMU data\n");
> + return;
> + }
> +
> + iommu_data->addr = addr;
> + iommu_data->size = size;
> + iommu_data->dt_node = node;
> + irq = platform_get_irq(node, 0);
> + if ( irq < 0 )
> + {
> + host_iommu_list_incomplete = true;
> + printk(XENLOG_WARNING
> + "vIOMMU: disabling support because IOMMU @0x%"PRIx64
> + " has no usable IRQ (%d)\n", addr, irq);
> + xfree(iommu_data);
> + return;
> + }
> + iommu_data->irq = irq;
> +
> + printk("vIOMMU: Found IOMMU @0x%"PRIx64"\n", addr);
> +
> + list_add_tail(&iommu_data->entry, &host_iommu_list);
> +}
>
> int domain_viommu_init(struct domain *d, uint8_t viommu_type)
> {
> @@ -33,12 +75,51 @@ int viommu_relinquish_resources(struct domain *d)
>
> uint8_t viommu_get_type(void)
> {
> - if ( !cur_viommu )
> + if ( !cur_viommu || host_iommu_list_incomplete ||
> + list_empty(&host_iommu_list) )
> return XEN_DOMCTL_CONFIG_VIOMMU_NONE;
>
> return cur_viommu->viommu_type;
> }
>
> +void __init set_cur_viommu(const struct viommu_desc *desc)
> +{
> + if ( cur_viommu && (cur_viommu != desc) )
> + {
> + printk("WARNING: Cannot set vIOMMU, already set to a different value\n");
> + return;
> + }
> +
> + cur_viommu = desc;
> +}
> +
> +unsigned int domain_viommu_get_num_mmio_handlers(struct domain *d)
> +{
> + if ( is_hardware_domain(d) )
> + {
> + struct host_iommu *hw_iommu;
> + int count = 0;
> +
> + /*
> + * For hardware domain, vIOMMU per pIOMMU is exposed. Number of
> + * regions required for the configuration space is equal to number of
> + * host IOMMUs.
> + */
> + list_for_each_entry(hw_iommu, &host_iommu_list, entry)
> + {
> + count++;
> + }
> +
> + return count;
> + }
> +
> + /*
> + * For unpriviliged domains, single vIOMMU is exposed and requires one
> + * region to cover the configuration space.
> + */
> + return 1;
> +}
> +
> /*
> * Local variables:
> * mode: C
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
> new file mode 100644
> index 0000000000..ea3b5bb34d
> --- /dev/null
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
> @@ -0,0 +1,118 @@
> +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
> +
> +#include <xen/param.h>
> +#include <xen/sched.h>
> +#include <asm/mmio.h>
> +#include <asm/viommu.h>
> +
> +struct virt_smmu {
> + struct domain *d;
> + struct list_head viommu_list;
> +};
> +
> +static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
> + register_t r, void *priv)
> +{
> + BUG_ON("unimplemented");
> + return IO_HANDLED;
> +}
> +
> +static int vsmmuv3_mmio_read(struct vcpu *v, mmio_info_t *info,
> + register_t *r, void *priv)
> +{
> + BUG_ON("unimplemented");
> + return IO_HANDLED;
> +}
> +
> +static const struct mmio_handler_ops vsmmuv3_mmio_handler = {
> + .read = vsmmuv3_mmio_read,
> + .write = vsmmuv3_mmio_write,
> +};
> +
> +static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size)
> +{
> + struct virt_smmu *smmu;
> +
> + smmu = xzalloc(struct virt_smmu);
> + if ( !smmu )
> + return -ENOMEM;
> +
> + smmu->d = d;
> +
> + register_mmio_handler(d, &vsmmuv3_mmio_handler, addr, size, smmu);
> +
> + /* Register the vIOMMU to be able to clean it up later. */
> + list_add_tail(&smmu->viommu_list, &d->arch.viommu_list);
> +
> + return 0;
> +}
> +
> +static int domain_vsmmuv3_init(struct domain *d)
> +{
> + int ret;
> +
> + INIT_LIST_HEAD(&d->arch.viommu_list);
> +
> + if ( is_hardware_domain(d) )
Looking over other device emulations in Xen (vpl011, vgic, etc.), I see
that the host (hardware) MMIO range is also exposed to direct-mapped
domains, as there might be a non-hwdom 1:1 domain. Xen even has a
specific macro for that: domain_use_host_layout. Here, however, the
guest MMIO range will be exposed to a regular direct-mapped domain. Is
there a specific reason for that (is this intentional)?
> + {
> + struct host_iommu *hw_iommu;
> +
> + list_for_each_entry(hw_iommu, &host_iommu_list, entry)
> + {
> + ret = vsmmuv3_init_single(d, hw_iommu->addr, hw_iommu->size);
> + if ( ret )
> + return ret;
> + }
> + }
> + else
> + {
> + ret = vsmmuv3_init_single(d, GUEST_VSMMUV3_BASE, GUEST_VSMMUV3_SIZE);
> + if ( ret )
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static int vsmmuv3_relinquish_resources(struct domain *d)
> +{
> + struct virt_smmu *pos, *temp;
> +
> + /* Cope with uninitialized vIOMMU */
> + if ( list_head_is_null(&d->arch.viommu_list) )
> + return 0;
> +
> + list_for_each_entry_safe(pos, temp, &d->arch.viommu_list, viommu_list )
> + {
> + list_del(&pos->viommu_list);
> + xfree(pos);
> + }
> +
> + return 0;
> +}
> +
> +static const struct viommu_ops vsmmuv3_ops = {
> + .domain_init = domain_vsmmuv3_init,
> + .relinquish_resources = vsmmuv3_relinquish_resources,
> +};
> +
> +static const struct viommu_desc vsmmuv3_desc = {
> + .ops = &vsmmuv3_ops,
> + .viommu_type = XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3,
> +};
> +
> +void __init vsmmuv3_set_type(void)
> +{
> + const struct viommu_desc *desc = &vsmmuv3_desc;
> +
> + set_cur_viommu(desc);
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.h b/xen/drivers/passthrough/arm/vsmmu-v3.h
> new file mode 100644
> index 0000000000..9f4c86abd5
> --- /dev/null
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.h
> @@ -0,0 +1,20 @@
> +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
> +#ifndef __ARCH_ARM_VSMMU_V3_H__
> +#define __ARCH_ARM_VSMMU_V3_H__
> +
> +#include <asm/viommu.h>
> +
> +#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3
> +
> +void vsmmuv3_set_type(void);
> +
> +#else
> +
> +static inline void __init vsmmuv3_set_type(void)
> +{
> + return;
> +}
> +
> +#endif /* CONFIG_VIRTUAL_ARM_SMMU_V3 */
> +
> +#endif /* __ARCH_ARM_VSMMU_V3_H__ */
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index a1491f9cf7..8863e2e457 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -335,6 +335,7 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
> #define XEN_DOMCTL_CONFIG_ARM_V8R_EL1_MSA_VMSA 2
>
> #define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0
> +#define XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 1
>
> struct xen_arch_domainconfig {
> /* IN/OUT */
> @@ -463,6 +464,10 @@ typedef uint64_t xen_callback_t;
> #define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000) /* vCPU0..127 */
> #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000)
>
> +/* vsmmuv3 MMIO mappings */
> +#define GUEST_VSMMUV3_BASE xen_mk_ullong(0x04040000)
> +#define GUEST_VSMMUV3_SIZE xen_mk_ullong(0x00020000)
> +
> /*
> * 256 MB is reserved for VPCI configuration space based on calculation
> * 256 buses x 32 devices x 8 functions x 4 KB = 256 MB
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 06/22] tools: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (4 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 05/22] xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-07 0:06 ` [PATCH v4 07/22] xen/arm: smmuv3: Extend HW features detection and apply errata Milan Djokic
` (15 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Anthony PERARD, Nick Rosbrook, George Dunlap,
Juergen Gross, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Add new viommu_type field and field values XEN_DOMCTL_CONFIG_VIOMMU_NONE
XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 in xen_arch_domainconfig to
enable/disable vIOMMU support for domains.
Also add viommu="N" parameter to xl domain configuration to enable the
vIOMMU for the domains. Currently, only the "smmuv3" type is supported
for ARM.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
Acked-by: Nick Rosbrook <enr0n@ubuntu.com>
---
docs/man/xl.cfg.5.pod.in | 22 ++++++++++++++++++++++
tools/golang/xenlight/helpers.gen.go | 2 ++
tools/golang/xenlight/types.gen.go | 7 +++++++
tools/include/libxl.h | 5 +++++
tools/libs/light/libxl_arm.c | 13 +++++++++++++
tools/libs/light/libxl_types.idl | 6 ++++++
tools/xl/xl_parse.c | 9 +++++++++
7 files changed, 64 insertions(+)
diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index d34951edb9..355a7381c8 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -3185,6 +3185,28 @@ option.
=back
+=over 4
+
+=item B<viommu="STRING">
+
+Enable an emulated IOMMU for the guest, allowing the guest to configure
+IOMMU stage-1 translation for guest-assigned devices protected by the
+IOMMU.
+
+To enable the emulated IOMMU, specify the following option in the VM
+configuration file:
+
+viommu = "smmuv3"
+
+Currently, only the "smmuv3" type is supported on ARM.
+
+For this option to take effect, the passthrough option must also be
+enabled. E.g. if passthrough is explicitly disabled, or implicitly when no
+devices passed through to the guest while the vIOMMU is enabled, the
+domain configuration is invalid and will be refused by Xen.
+
+=back
+
=head3 x86
=over 4
diff --git a/tools/golang/xenlight/helpers.gen.go b/tools/golang/xenlight/helpers.gen.go
index b0c09da910..b24cfd0533 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -1273,6 +1273,7 @@ x.ArchArm.NrSpis = uint32(xc.arch_arm.nr_spis)
if err := x.ArchArm.ArmSci.fromC(&xc.arch_arm.arm_sci);err != nil {
return fmt.Errorf("converting field ArchArm.ArmSci: %v", err)
}
+x.ArchArm.ViommuType = ViommuType(xc.arch_arm.viommu_type)
if err := x.ArchX86.MsrRelaxed.fromC(&xc.arch_x86.msr_relaxed);err != nil {
return fmt.Errorf("converting field ArchX86.MsrRelaxed: %v", err)
}
@@ -1815,6 +1816,7 @@ xc.arch_arm.nr_spis = C.uint32_t(x.ArchArm.NrSpis)
if err := x.ArchArm.ArmSci.toC(&xc.arch_arm.arm_sci); err != nil {
return fmt.Errorf("converting field ArchArm.ArmSci: %v", err)
}
+xc.arch_arm.viommu_type = C.libxl_viommu_type(x.ArchArm.ViommuType)
if err := x.ArchX86.MsrRelaxed.toC(&xc.arch_x86.msr_relaxed); err != nil {
return fmt.Errorf("converting field ArchX86.MsrRelaxed: %v", err)
}
diff --git a/tools/golang/xenlight/types.gen.go b/tools/golang/xenlight/types.gen.go
index e0fd78ec03..e306f9c1ac 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight/types.gen.go
@@ -530,6 +530,12 @@ type ArmSci struct {
Type ArmSciType
}
+type ViommuType int
+const(
+ViommuTypeNone ViommuType = 0
+ViommuTypeSmmuv3 ViommuType = 1
+)
+
type RdmReserve struct {
Strategy RdmReserveStrategy
Policy RdmReservePolicy
@@ -619,6 +625,7 @@ Vuart VuartType
SveVl SveType
NrSpis uint32
ArmSci ArmSci
+ViommuType ViommuType
}
ArchX86 struct {
MsrRelaxed Defbool
diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index 7c098edab6..1af8613499 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -318,6 +318,11 @@
*/
#define LIBXL_HAVE_BUILDINFO_ARCH_ARM_SCI 1
+/*
+ * libxl_domain_build_info has the arch_arm.viommu_type field.
+ */
+#define LIBXL_HAVE_BUILDINFO_ARCH_ARM_VIOMMU 1
+
/*
* LIBXL_HAVE_SOFT_RESET indicates that libxl supports performing
* 'soft reset' for domains and there is 'soft_reset' shutdown reason
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index 7e9f8a1bc3..a248793588 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -247,6 +247,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
}
LOG(DEBUG, " - SCI type=%u", config->arch.arm_sci_type);
+ switch (d_config->b_info.arch_arm.viommu_type) {
+ case LIBXL_VIOMMU_TYPE_NONE:
+ config->arch.viommu_type = XEN_DOMCTL_CONFIG_VIOMMU_NONE;
+ break;
+ case LIBXL_VIOMMU_TYPE_SMMUV3:
+ config->arch.viommu_type = XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3;
+ break;
+ default:
+ LOG(ERROR, "Unknown vIOMMU type %d",
+ d_config->b_info.arch_arm.viommu_type);
+ return ERROR_FAIL;
+ }
+
return 0;
}
diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index a7893460f0..f0a9a21ba4 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -561,6 +561,11 @@ libxl_arm_sci = Struct("arm_sci", [
("type", libxl_arm_sci_type),
])
+libxl_viommu_type = Enumeration("viommu_type", [
+ (0, "none"),
+ (1, "smmuv3")
+ ], init_val = "LIBXL_VIOMMU_TYPE_NONE")
+
libxl_rdm_reserve = Struct("rdm_reserve", [
("strategy", libxl_rdm_reserve_strategy),
("policy", libxl_rdm_reserve_policy),
@@ -745,6 +750,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
("sve_vl", libxl_sve_type),
("nr_spis", uint32, {'init_val': 'LIBXL_NR_SPIS_DEFAULT'}),
("arm_sci", libxl_arm_sci),
+ ("viommu_type", libxl_viommu_type),
])),
("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool),
])),
diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
index 0ccece3e8a..4e5bc84ca3 100644
--- a/tools/xl/xl_parse.c
+++ b/tools/xl/xl_parse.c
@@ -3080,6 +3080,15 @@ skip_usbdev:
}
}
+ if (!xlu_cfg_get_string (config, "viommu", &buf, 1)) {
+ e = libxl_viommu_type_from_string(buf, &b_info->arch_arm.viommu_type);
+ if (e) {
+ fprintf(stderr,
+ "Unknown vIOMMU type \"%s\" specified\n", buf);
+ exit(-ERROR_FAIL);
+ }
+ }
+
parse_vkb_list(config, d_config);
d_config->virtios = NULL;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 07/22] xen/arm: smmuv3: Extend HW features detection and apply errata
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (5 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 06/22] tools: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-07 0:06 ` [PATCH v4 08/22] xen/arm: vsmmuv3: Add support for registers emulation Milan Djokic
` (14 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Milan Djokic, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Rahul Singh, Robin Murphy,
Will Deacon
Backport Linux commits 4bbd63eb685b and fd86b5944215, adapted to the
Xen SMMUv3 implementation, applying errata workarounds to prevent
nesting on MMU-600 and MMU-700 products.
Detect IOMMU hardware features and store them in the IOMMU host list
for each probed SMMUv3 device, allowing the emulated IOMMU to advertise
features supported by the physical IOMMU.
Linux commits:
4bbd63eb685b ("iommu/arm-smmu-v3: Work around MMU-600 erratum 1076982")
fd86b5944215 ("iommu/arm-smmu-v3: Add explicit feature for nesting")
Link: https://lore.kernel.org/all/08adbe3d01024d8382a478325f73b56851f76e49.1683731256.git.robin.murphy@arm.com/
Link: https://lore.kernel.org/all/136c3f4a3a84cc14a5a1978ace57dfd3ed67b688.1683731256.git.robin.murphy@arm.com/
Origin: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 4bbd63eb685b
Origin: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git fd86b5944215
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/arch/arm/include/asm/viommu.h | 8 ++-
xen/drivers/passthrough/arm/smmu-v3.c | 81 ++++++++++++++++++++++++--
xen/drivers/passthrough/arm/smmu-v3.h | 32 ++++++++--
xen/drivers/passthrough/arm/viommu.c | 4 +-
xen/drivers/passthrough/arm/vsmmu-v3.c | 44 ++++++++++++--
5 files changed, 152 insertions(+), 17 deletions(-)
diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
index 287595068b..844cf34d56 100644
--- a/xen/arch/arm/include/asm/viommu.h
+++ b/xen/arch/arm/include/asm/viommu.h
@@ -12,6 +12,8 @@ struct host_iommu {
struct list_head entry;
/* IOMMU DT node used to establish virtual to physical IOMMU/SID mapping */
const struct dt_device_node *dt_node;
+ /* Physical IOMMU device structure used to detect supported features */
+ const void *iommu;
/* Used for dom0 vIOMMU instantiation (1-1 MMIO regions mappings) */
paddr_t addr;
/* Used for dom0 vIOMMU instantiation (1-1 MMIO regions mappings) */
@@ -52,7 +54,8 @@ int domain_viommu_init(struct domain *d, uint8_t viommu_type);
int viommu_relinquish_resources(struct domain *d);
uint8_t viommu_get_type(void);
void add_to_host_iommu_list(paddr_t addr, paddr_t size,
- const struct dt_device_node *node);
+ const struct dt_device_node *node,
+ const void *iommu);
void set_cur_viommu(const struct viommu_desc *desc);
unsigned int domain_viommu_get_num_mmio_handlers(struct domain *d);
@@ -77,7 +80,8 @@ static inline int viommu_relinquish_resources(struct domain *d)
}
static inline void __init add_to_host_iommu_list(paddr_t addr, paddr_t size,
- const struct dt_device_node *node)
+ const struct dt_device_node *node,
+ const void *iommu)
{
return;
}
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index 21b82eba85..9d2b8a708b 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -2451,6 +2451,37 @@ err_free_irqs:
return ret;
}
+static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu)
+{
+ u32 reg;
+ unsigned int implementer, productid, variant, revision;
+
+ reg = readl_relaxed(smmu->base + ARM_SMMU_IIDR);
+ implementer = FIELD_GET(IIDR_IMPLEMENTER, reg);
+ productid = FIELD_GET(IIDR_PRODUCTID, reg);
+ variant = FIELD_GET(IIDR_VARIANT, reg);
+ revision = FIELD_GET(IIDR_REVISION, reg);
+
+ switch (implementer) {
+ case IIDR_IMPLEMENTER_ARM:
+ switch (productid) {
+ case IIDR_PRODUCTID_ARM_MMU_600:
+ /* Arm erratum 1076982 */
+ if (variant == 0 && revision <= 2)
+ smmu->features &= ~ARM_SMMU_FEAT_SEV;
+ /* Arm erratum 1209401 */
+ if (variant < 2)
+ smmu->features &= ~ARM_SMMU_FEAT_NESTING;
+ break;
+ case IIDR_PRODUCTID_ARM_MMU_700:
+ /* Arm errata 2268618, 2812531 */
+ smmu->features &= ~ARM_SMMU_FEAT_NESTING;
+ break;
+ }
+ break;
+ }
+}
+
static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
{
u32 reg;
@@ -2539,8 +2570,10 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
switch (FIELD_GET(IDR0_TTF, reg)) {
case IDR0_TTF_AARCH32_64:
smmu->ias = 40;
+ smmu->features |= ARM_SMMU_FEAT_TTF_AARCH32_64;
fallthrough;
case IDR0_TTF_AARCH64:
+ smmu->features |= ARM_SMMU_FEAT_TTF_AARCH64;
break;
default:
dev_err(smmu->dev, "AArch64 table format not supported!\n");
@@ -2549,6 +2582,8 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
/* ASID/VMID sizes */
smmu->vmid_bits = reg & IDR0_VMID16 ? 16 : 8;
+ if (reg & IDR0_ASID16)
+ smmu->features |= ARM_SMMU_FEAT_ASID_16;
/* IDR1 */
reg = readl_relaxed(smmu->base + ARM_SMMU_IDR1);
@@ -2588,12 +2623,18 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
smmu->evtq.max_stalls = FIELD_GET(IDR5_STALL_MAX, reg);
/* Page sizes */
- if (reg & IDR5_GRAN64K)
+ if (reg & IDR5_GRAN64K) {
smmu->pgsize_bitmap |= SZ_64K | SZ_512M;
- if (reg & IDR5_GRAN16K)
+ smmu->features |= ARM_SMMU_FEAT_GRAN64K;
+ }
+ if (reg & IDR5_GRAN16K) {
smmu->pgsize_bitmap |= SZ_16K | SZ_32M;
- if (reg & IDR5_GRAN4K)
+ smmu->features |= ARM_SMMU_FEAT_GRAN16K;
+ }
+ if (reg & IDR5_GRAN4K) {
smmu->pgsize_bitmap |= SZ_4K | SZ_2M | SZ_1G;
+ smmu->features |= ARM_SMMU_FEAT_GRAN4K;
+ }
/* Input address size */
if (FIELD_GET(IDR5_VAX, reg) == IDR5_VAX_52_BIT)
@@ -2626,15 +2667,46 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
fallthrough;
case IDR5_OAS_48_BIT:
smmu->oas = 48;
+ smmu->features |= ARM_SMMU_FEAT_OAS_48_BIT;
break;
}
smmu->oas = min_t(unsigned long, PADDR_BITS, smmu->oas);
smmu->ias = max(smmu->ias, smmu->oas);
+ switch (smmu->oas) {
+ case 32:
+ smmu->features |= ARM_SMMU_FEAT_OAS_32_BIT;
+ break;
+ case 36:
+ smmu->features |= ARM_SMMU_FEAT_OAS_36_BIT;
+ break;
+ case 40:
+ smmu->features |= ARM_SMMU_FEAT_OAS_40_BIT;
+ break;
+ case 42:
+ smmu->features |= ARM_SMMU_FEAT_OAS_42_BIT;
+ break;
+ case 44:
+ smmu->features |= ARM_SMMU_FEAT_OAS_44_BIT;
+ break;
+ case 48:
+ smmu->features |= ARM_SMMU_FEAT_OAS_48_BIT;
+ break;
+ case 52:
+ smmu->features |= ARM_SMMU_FEAT_OAS_52_BIT;
+ break;
+ }
+
/* Xen: Set maximum Stage-2 input size supported by the SMMU. */
p2m_restrict_ipa_bits(smmu->ias);
+ if ((smmu->features & ARM_SMMU_FEAT_TRANS_S1) &&
+ (smmu->features & ARM_SMMU_FEAT_TRANS_S2))
+ smmu->features |= ARM_SMMU_FEAT_NESTING;
+
+ arm_smmu_device_iidr_probe(smmu);
+
dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n",
smmu->ias, smmu->oas, smmu->features);
return 0;
@@ -2829,9 +2901,8 @@ static int __init arm_smmu_device_probe(struct platform_device *pdev)
spin_lock(&arm_smmu_devices_lock);
list_add(&smmu->devices, &arm_smmu_devices);
spin_unlock(&arm_smmu_devices_lock);
-
/* Add to host IOMMU list to initialize vIOMMU for dom0 */
- add_to_host_iommu_list(ioaddr, iosize, dev_to_dt(pdev));
+ add_to_host_iommu_list(ioaddr, iosize, dev_to_dt(pdev), (void *) smmu);
return 0;
diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
index e257de6db9..ef492098ae 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.h
+++ b/xen/drivers/passthrough/arm/smmu-v3.h
@@ -60,6 +60,16 @@
#define IDR5_VAX GENMASK(11, 10)
#define IDR5_VAX_52_BIT 1
+#define ARM_SMMU_IIDR 0x18
+#define IIDR_PRODUCTID GENMASK(31, 20)
+#define IIDR_VARIANT GENMASK(19, 16)
+#define IIDR_REVISION GENMASK(15, 12)
+#define IIDR_IMPLEMENTER GENMASK(11, 0)
+
+#define IIDR_IMPLEMENTER_ARM 0x43b
+#define IIDR_PRODUCTID_ARM_MMU_600 0x483
+#define IIDR_PRODUCTID_ARM_MMU_700 0x487
+
#define ARM_SMMU_CR0 0x20
#define CR0_ATSCHK (1 << 4)
#define CR0_CMDQEN (1 << 3)
@@ -602,10 +612,24 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_COHERENCY (1 << 8)
#define ARM_SMMU_FEAT_TRANS_S1 (1 << 9)
#define ARM_SMMU_FEAT_TRANS_S2 (1 << 10)
-#define ARM_SMMU_FEAT_STALLS (1 << 11)
-#define ARM_SMMU_FEAT_HYP (1 << 12)
-#define ARM_SMMU_FEAT_STALL_FORCE (1 << 13)
-#define ARM_SMMU_FEAT_VAX (1 << 14)
+#define ARM_SMMU_FEAT_NESTING (1 << 11)
+#define ARM_SMMU_FEAT_STALLS (1 << 12)
+#define ARM_SMMU_FEAT_HYP (1 << 13)
+#define ARM_SMMU_FEAT_STALL_FORCE (1 << 14)
+#define ARM_SMMU_FEAT_VAX (1 << 15)
+#define ARM_SMMU_FEAT_ASID_16 (1 << 16)
+#define ARM_SMMU_FEAT_TTF_AARCH64 (1 << 17)
+#define ARM_SMMU_FEAT_TTF_AARCH32_64 (1 << 18)
+#define ARM_SMMU_FEAT_GRAN64K (1 << 19)
+#define ARM_SMMU_FEAT_GRAN16K (1 << 20)
+#define ARM_SMMU_FEAT_GRAN4K (1 << 21)
+#define ARM_SMMU_FEAT_OAS_32_BIT (1 << 22)
+#define ARM_SMMU_FEAT_OAS_36_BIT (1 << 23)
+#define ARM_SMMU_FEAT_OAS_40_BIT (1 << 24)
+#define ARM_SMMU_FEAT_OAS_42_BIT (1 << 25)
+#define ARM_SMMU_FEAT_OAS_44_BIT (1 << 26)
+#define ARM_SMMU_FEAT_OAS_48_BIT (1 << 27)
+#define ARM_SMMU_FEAT_OAS_52_BIT (1 << 28)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
diff --git a/xen/drivers/passthrough/arm/viommu.c b/xen/drivers/passthrough/arm/viommu.c
index 3401e4f7d9..407e68e62d 100644
--- a/xen/drivers/passthrough/arm/viommu.c
+++ b/xen/drivers/passthrough/arm/viommu.c
@@ -17,7 +17,8 @@ static bool host_iommu_list_incomplete;
/* Common function for adding to host_iommu_list */
void __init add_to_host_iommu_list(paddr_t addr, paddr_t size,
- const struct dt_device_node *node)
+ const struct dt_device_node *node,
+ const void *iommu)
{
struct host_iommu *iommu_data;
int irq;
@@ -45,6 +46,7 @@ void __init add_to_host_iommu_list(paddr_t addr, paddr_t size,
return;
}
iommu_data->irq = irq;
+ iommu_data->iommu = iommu;
printk("vIOMMU: Found IOMMU @0x%"PRIx64"\n", addr);
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index ea3b5bb34d..7001f8c23f 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -5,9 +5,13 @@
#include <asm/mmio.h>
#include <asm/viommu.h>
+#include "smmu-v3.h"
+#include "vsmmu-v3.h"
+
struct virt_smmu {
struct domain *d;
struct list_head viommu_list;
+ uint32_t features;
};
static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
@@ -29,7 +33,8 @@ static const struct mmio_handler_ops vsmmuv3_mmio_handler = {
.write = vsmmuv3_mmio_write,
};
-static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size)
+static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
+ uint32_t features)
{
struct virt_smmu *smmu;
@@ -38,6 +43,7 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size)
return -ENOMEM;
smmu->d = d;
+ smmu->features = features;
register_mmio_handler(d, &vsmmuv3_mmio_handler, addr, size, smmu);
@@ -50,23 +56,51 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size)
static int domain_vsmmuv3_init(struct domain *d)
{
int ret;
+ struct host_iommu *hw_iommu;
+ const struct arm_smmu_device *smmu;
+
+ if ( list_head_is_null(&host_iommu_list) )
+ return -ENODEV;
INIT_LIST_HEAD(&d->arch.viommu_list);
if ( is_hardware_domain(d) )
{
- struct host_iommu *hw_iommu;
-
list_for_each_entry(hw_iommu, &host_iommu_list, entry)
{
- ret = vsmmuv3_init_single(d, hw_iommu->addr, hw_iommu->size);
+ smmu = (const struct arm_smmu_device *) hw_iommu->iommu;
+ ret = vsmmuv3_init_single(d, hw_iommu->addr, hw_iommu->size,
+ smmu->features);
if ( ret )
return ret;
}
}
else
{
- ret = vsmmuv3_init_single(d, GUEST_VSMMUV3_BASE, GUEST_VSMMUV3_SIZE);
+ /*
+ * Require: For non-hardware domains, features must be uniform across
+ * physical SMMUs to avoid advertising features that are unsupported by
+ * certain units.
+ */
+ uint32_t features;
+ hw_iommu = list_first_entry(&host_iommu_list, struct host_iommu, entry);
+ smmu = (const struct arm_smmu_device *) hw_iommu->iommu;
+ features = smmu->features;
+ list_for_each_entry(hw_iommu, &host_iommu_list, entry)
+ {
+ smmu = (const struct arm_smmu_device *) hw_iommu->iommu;
+
+ if ( smmu->features != features )
+ {
+ printk(XENLOG_G_ERR
+ "vSMMUv3: Unsupported HW layout: physical SMMUs must " \
+ "have a uniform set of features\n");
+ return -EINVAL;
+ }
+ }
+
+ ret = vsmmuv3_init_single(d, GUEST_VSMMUV3_BASE, GUEST_VSMMUV3_SIZE,
+ features);
if ( ret )
return ret;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 08/22] xen/arm: vsmmuv3: Add support for registers emulation
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (6 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 07/22] xen/arm: smmuv3: Extend HW features detection and apply errata Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-07 0:06 ` [PATCH v4 09/22] xen/arm: vsmmuv3: Add support for cmdqueue handling Milan Djokic
` (13 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Add initial support for various emulated registers for virtual SMMUv3
for guests and also add support for virtual cmdq and eventq.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/vsmmu-v3.c | 573 ++++++++++++++++++++++++-
1 file changed, 571 insertions(+), 2 deletions(-)
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index 7001f8c23f..c4fef6e8bd 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -1,30 +1,588 @@
/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */
+/*
+ * IOMMU stage-1 emulation for ARM SMMUv3.
+ *
+ * TODO: Using ARM generic implementer/product/variant to avoid activating
+ * errata workarounds in the guest driver due to emulation layer.
+ * This needs to be fully handled in the future.
+ */
+
#include <xen/param.h>
#include <xen/sched.h>
#include <asm/mmio.h>
+#include <asm/vgic-emul.h>
#include <asm/viommu.h>
+#include <asm/vreg.h>
#include "smmu-v3.h"
#include "vsmmu-v3.h"
+/* Register Definition */
+#define ARM_SMMU_IDR2 0x8
+#define ARM_SMMU_IDR3 0xc
+#define ARM_SMMU_IDR4 0x10
+#define IDR0_TERM_MODEL (1 << 26)
+#define IDR3_RIL (1 << 10)
+/* Only CMDQ, EVENTQ and SMMUEN supported */
+#define CR0_RESERVED 0xFFFFFFF2U
+/* TODO: Right now only 16-bit SID size is supported */
+#define SMMU_IDR1_SIDSIZE 16
+/* The CMDQ/EVTQ sizes are currently set to the architectural maximum. */
+#define SMMU_CMDQS 19
+#define SMMU_EVTQS 19
+#define DWORDS_BYTES 8
+
+/* virtual smmu queue */
+struct arm_vsmmu_queue {
+ uint64_t q_base; /* base register */
+ uint32_t prod;
+ uint32_t cons;
+ uint8_t ent_size;
+ uint8_t max_n_shift;
+};
+
struct virt_smmu {
struct domain *d;
struct list_head viommu_list;
uint32_t features;
+ paddr_t addr;
+ uint32_t cr[3];
+ uint32_t cr0ack;
+ uint32_t gerror;
+ uint32_t gerrorn;
+ uint32_t strtab_base_cfg;
+ uint64_t strtab_base;
+ uint32_t irq_ctrl;
+ uint64_t gerror_irq_cfg0;
+ uint64_t evtq_irq_cfg0;
+ struct arm_vsmmu_queue evtq, cmdq;
+ spinlock_t cmd_queue_lock;
+ spinlock_t evt_queue_lock;
+ spinlock_t gerror_lock;
+ spinlock_t cr0_lock;
+ spinlock_t cr1_lock;
+ spinlock_t cr2_lock;
+ spinlock_t strtab_cfg_lock;
+ spinlock_t irq_cfg_lock;
};
+/* Helper functions */
+static inline bool smmu_get_smmu_enabled(struct virt_smmu *smmu)
+{
+ bool enabled;
+
+ spin_lock(&smmu->cr0_lock);
+ enabled = FIELD_GET(CR0_SMMUEN, smmu->cr[0]);
+ spin_unlock(&smmu->cr0_lock);
+
+ return enabled;
+}
+
+static inline bool smmu_get_cmdq_enabled(struct virt_smmu *smmu)
+{
+ bool enabled;
+
+ spin_lock(&smmu->cr0_lock);
+ enabled = FIELD_GET(CR0_CMDQEN, smmu->cr[0]);
+ spin_unlock(&smmu->cr0_lock);
+
+ return enabled;
+}
+
+static inline bool smmu_get_evtq_enabled(struct virt_smmu *smmu)
+{
+ bool enabled;
+
+ spin_lock(&smmu->cr0_lock);
+ enabled = FIELD_GET(CR0_EVTQEN, smmu->cr[0]);
+ spin_unlock(&smmu->cr0_lock);
+
+ return enabled;
+}
+
static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
register_t r, void *priv)
{
- BUG_ON("unimplemented");
+ struct virt_smmu *smmu = priv;
+ uint64_t reg;
+ uint32_t reg32;
+ struct hsr_dabt dabt = info->dabt;
+ uint64_t offset = info->gpa - smmu->addr;
+
+ if ( offset & 0x10000 )
+ {
+ /* page 1 */
+ switch ( info->gpa & 0xffff )
+ {
+ case VREG32(ARM_SMMU_EVTQ_PROD):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ /* Write allowed only when SMMU_CR0.EVTQEN == 0 */
+ if ( smmu_get_evtq_enabled(smmu) )
+ break;
+ spin_lock(&smmu->evt_queue_lock);
+ reg32 = smmu->evtq.prod;
+ vreg_reg32_update(®32, r, info);
+ smmu->evtq.prod = reg32;
+ spin_unlock(&smmu->evt_queue_lock);
+ break;
+ case VREG32(ARM_SMMU_EVTQ_CONS):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->evt_queue_lock);
+ reg32 = smmu->evtq.cons;
+ vreg_reg32_update(®32, r, info);
+ smmu->evtq.cons = reg32;
+ spin_unlock(&smmu->evt_queue_lock);
+ break;
+ default:
+ printk(XENLOG_G_ERR
+ "%pd: vSMMUv3: unhandled write r%d offset %"PRIpaddr"\n",
+ v->domain, info->dabt.reg, (unsigned long)info->gpa & 0xffff);
+ return IO_HANDLED;
+ }
+ }
+ else
+ {
+ /* page 0 */
+ switch ( info->gpa & 0xffff )
+ {
+ case VREG32(ARM_SMMU_CR0):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cr0_lock);
+ reg32 = smmu->cr[0];
+ vreg_reg32_update(®32, r, info);
+ smmu->cr[0] = reg32;
+ smmu->cr0ack = reg32 & ~CR0_RESERVED;
+ spin_unlock(&smmu->cr0_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CR1):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cr1_lock);
+ reg32 = smmu->cr[1];
+ vreg_reg32_update(®32, r, info);
+ smmu->cr[1] = reg32;
+ spin_unlock(&smmu->cr1_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CR2):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cr2_lock);
+ reg32 = smmu->cr[2];
+ vreg_reg32_update(®32, r, info);
+ smmu->cr[2] = reg32;
+ spin_unlock(&smmu->cr2_lock);
+ break;
+
+ case VREG64(ARM_SMMU_STRTAB_BASE):
+ /* Must only be written when SMMU_CR0.SMMUEN == 0, otherwise ignore */
+ if ( smmu_get_smmu_enabled(smmu) )
+ break;
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->strtab_cfg_lock);
+ reg = smmu->strtab_base;
+ vreg_reg64_update(®, r, info);
+ smmu->strtab_base = reg;
+ spin_unlock(&smmu->strtab_cfg_lock);
+ break;
+
+ case VREG32(ARM_SMMU_STRTAB_BASE_CFG):
+ /* Must only be written when SMMU_CR0.SMMUEN == 0, otherwise ignore */
+ if ( smmu_get_smmu_enabled(smmu) )
+ break;
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->strtab_cfg_lock);
+ reg32 = smmu->strtab_base_cfg;
+ vreg_reg32_update(®32, r, info);
+ smmu->strtab_base_cfg = reg32;
+ spin_unlock(&smmu->strtab_cfg_lock);
+ break;
+
+ case VREG64(ARM_SMMU_CMDQ_BASE):
+ /* Must only be written when SMMU_CR0.CMDQEN == 0, otherwise ignore */
+ if ( smmu_get_cmdq_enabled(smmu) )
+ break;
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->cmd_queue_lock);
+ reg = smmu->cmdq.q_base;
+ vreg_reg64_update(®, r, info);
+ smmu->cmdq.q_base = reg;
+ smmu->cmdq.max_n_shift = FIELD_GET(Q_BASE_LOG2SIZE, smmu->cmdq.q_base);
+ if ( smmu->cmdq.max_n_shift > SMMU_CMDQS )
+ smmu->cmdq.max_n_shift = SMMU_CMDQS;
+ spin_unlock(&smmu->cmd_queue_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CMDQ_PROD):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cmd_queue_lock);
+ reg32 = smmu->cmdq.prod;
+ vreg_reg32_update(®32, r, info);
+ smmu->cmdq.prod = reg32;
+ spin_unlock(&smmu->cmd_queue_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CMDQ_CONS):
+ /* Must only be written when SMMU_CR0.CMDQEN == 0, otherwise ignore */
+ if ( smmu_get_cmdq_enabled(smmu) )
+ break;
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cmd_queue_lock);
+ reg32 = smmu->cmdq.cons;
+ vreg_reg32_update(®32, r, info);
+ smmu->cmdq.cons = reg32;
+ spin_unlock(&smmu->cmd_queue_lock);
+ break;
+
+ case VREG64(ARM_SMMU_EVTQ_BASE):
+ /* Must only be written when SMMU_CR0.EVTQEN == 0, otherwise ignore */
+ if ( smmu_get_evtq_enabled(smmu) )
+ break;
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->evt_queue_lock);
+ reg = smmu->evtq.q_base;
+ vreg_reg64_update(®, r, info);
+ smmu->evtq.q_base = reg;
+ smmu->evtq.max_n_shift = FIELD_GET(Q_BASE_LOG2SIZE, smmu->evtq.q_base);
+ if ( smmu->evtq.max_n_shift > SMMU_EVTQS )
+ smmu->evtq.max_n_shift = SMMU_EVTQS;
+ spin_unlock(&smmu->evt_queue_lock);
+ break;
+
+ case VREG32(ARM_SMMU_IRQ_CTRL):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->irq_cfg_lock);
+ reg32 = smmu->irq_ctrl;
+ vreg_reg32_update(®32, r, info);
+ smmu->irq_ctrl = reg32;
+ spin_unlock(&smmu->irq_cfg_lock);
+ break;
+
+ case VREG64(ARM_SMMU_GERROR_IRQ_CFG0):
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->irq_cfg_lock);
+ reg = smmu->gerror_irq_cfg0;
+ vreg_reg64_update(®, r, info);
+ smmu->gerror_irq_cfg0 = reg;
+ spin_unlock(&smmu->irq_cfg_lock);
+ break;
+
+ case VREG64(ARM_SMMU_EVTQ_IRQ_CFG0):
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->irq_cfg_lock);
+ reg = smmu->evtq_irq_cfg0;
+ vreg_reg64_update(®, r, info);
+ smmu->evtq_irq_cfg0 = reg;
+ spin_unlock(&smmu->irq_cfg_lock);
+ break;
+
+ case VREG32(ARM_SMMU_GERRORN):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->gerror_lock);
+ reg32 = smmu->gerrorn;
+ vreg_reg32_update(®32, r, info);
+ smmu->gerrorn = reg32;
+ spin_unlock(&smmu->gerror_lock);
+ break;
+
+ default:
+ printk(XENLOG_G_ERR
+ "%pd: vSMMUv3: unhandled write r%d offset %"PRIpaddr"\n",
+ v->domain, info->dabt.reg, (unsigned long)info->gpa & 0xffff);
+ return IO_HANDLED;
+ }
+ }
+
+ return IO_HANDLED;
+
+ bad_width:
+ gprintk(XENLOG_G_ERR,
+ "%pd: vSMMUv3: bad write width %d r%d offset %"PRIpaddr"\n",
+ v->domain, dabt.size, dabt.reg, info->gpa & 0xffff);
return IO_HANDLED;
}
static int vsmmuv3_mmio_read(struct vcpu *v, mmio_info_t *info,
register_t *r, void *priv)
{
- BUG_ON("unimplemented");
+ struct virt_smmu *smmu = priv;
+ uint64_t reg;
+ struct hsr_dabt dabt = info->dabt;
+ uint64_t offset = info->gpa - smmu->addr;
+
+ if ( offset & 0x10000 )
+ {
+ /* page 1 */
+ switch ( info->gpa & 0xffff )
+ {
+ case VREG32(ARM_SMMU_EVTQ_PROD):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->evt_queue_lock);
+ *r = vreg_reg32_extract(smmu->evtq.prod, info);
+ spin_unlock(&smmu->evt_queue_lock);
+ break;
+
+ case VREG32(ARM_SMMU_EVTQ_CONS):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->evt_queue_lock);
+ *r = vreg_reg32_extract(smmu->evtq.cons, info);
+ spin_unlock(&smmu->evt_queue_lock);
+ break;
+
+ default:
+ printk(XENLOG_G_ERR
+ "%pd: vSMMUv3: unhandled read r%d offset %"PRIpaddr"\n",
+ v->domain, info->dabt.reg, (unsigned long)info->gpa & 0xffff);
+ return IO_HANDLED;
+ }
+ }
+ else
+ {
+ /* page 0 */
+ switch ( info->gpa & 0xffff )
+ {
+ case VREG32(ARM_SMMU_IDR0):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ /*
+ * Emulated SMMU capabilities.
+ *
+ * Some architectural capabilities are currently not implemented
+ * by the stage-1 emulation layer or underlying Xen SMMU support,
+ * therefore they are not advertised atm even if supported by the HW
+ */
+ reg = FIELD_PREP(IDR0_S1P, 1) |
+ /*
+ * TODO: Coherent access not supported by the emulation atm.
+ * Guest must perform explicit cache clean
+ */
+ FIELD_PREP(IDR0_COHACC, 0) |
+ /* Transaction stall on fault not implemented */
+ FIELD_PREP(IDR0_STALL_MODEL, 1) |
+ /* Terminating a transaction with RAZ/WI behavior not implemented */
+ FIELD_PREP(IDR0_TERM_MODEL, 1);
+
+ /* HW reflected capabilities */
+ reg |= FIELD_PREP(IDR0_ST_LVL,
+ smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB ?
+ 1 : 0);
+ reg |= FIELD_PREP(IDR0_TTF,
+ smmu->features & ARM_SMMU_FEAT_TTF_AARCH32_64 ?
+ IDR0_TTF_AARCH32_64 : IDR0_TTF_AARCH64);
+
+ reg |= FIELD_PREP(IDR0_ASID16,
+ smmu->features & ARM_SMMU_FEAT_ASID_16 ? 1 : 0);
+ if ( (smmu->features & ARM_SMMU_FEAT_TT_LE) &&
+ (smmu->features & ARM_SMMU_FEAT_TT_BE) )
+ reg |= FIELD_PREP(IDR0_TTENDIAN, IDR0_TTENDIAN_MIXED);
+ else if ( smmu->features & ARM_SMMU_FEAT_TT_LE )
+ reg |= FIELD_PREP(IDR0_TTENDIAN, IDR0_TTENDIAN_LE);
+ #ifdef __BIG_ENDIAN
+ else if ( smmu->features & ARM_SMMU_FEAT_TT_BE )
+ reg |= FIELD_PREP(IDR0_TTENDIAN, IDR0_TTENDIAN_BE);
+ #endif
+ *r = vreg_reg32_extract(reg, info);
+ break;
+
+ case VREG32(ARM_SMMU_IDR1):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ reg = FIELD_PREP(IDR1_SIDSIZE, SMMU_IDR1_SIDSIZE) |
+ FIELD_PREP(IDR1_CMDQS, SMMU_CMDQS) |
+ FIELD_PREP(IDR1_EVTQS, SMMU_EVTQS);
+ *r = vreg_reg32_extract(reg, info);
+ break;
+
+ case VREG32(ARM_SMMU_IDR2):
+ goto read_reserved;
+
+ case VREG32(ARM_SMMU_IDR3):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ reg = 0;
+ *r = vreg_reg32_extract(reg, info);
+ break;
+
+ case VREG32(ARM_SMMU_IDR4):
+ goto read_impl_defined;
+
+ case VREG32(ARM_SMMU_IDR5):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ reg = 0;
+ if (smmu->features & ARM_SMMU_FEAT_GRAN64K)
+ reg |= IDR5_GRAN64K;
+ if (smmu->features & ARM_SMMU_FEAT_GRAN16K)
+ reg |= IDR5_GRAN16K;
+ if (smmu->features & ARM_SMMU_FEAT_GRAN4K)
+ reg |= IDR5_GRAN4K;
+
+ if (smmu->features & ARM_SMMU_FEAT_OAS_32_BIT)
+ reg |= IDR5_OAS_32_BIT;
+ if (smmu->features & ARM_SMMU_FEAT_OAS_36_BIT)
+ reg |= IDR5_OAS_36_BIT;
+ if (smmu->features & ARM_SMMU_FEAT_OAS_40_BIT)
+ reg |= IDR5_OAS_40_BIT;
+ if (smmu->features & ARM_SMMU_FEAT_OAS_42_BIT)
+ reg |= IDR5_OAS_42_BIT;
+ if (smmu->features & ARM_SMMU_FEAT_OAS_44_BIT)
+ reg |= IDR5_OAS_44_BIT;
+ if (smmu->features & ARM_SMMU_FEAT_OAS_48_BIT)
+ reg |= IDR5_OAS_48_BIT;
+ if (smmu->features & ARM_SMMU_FEAT_OAS_52_BIT)
+ reg |= IDR5_OAS_52_BIT;
+
+ *r = vreg_reg32_extract(reg, info);
+ break;
+
+ case VREG32(ARM_SMMU_IIDR):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ /* Use generic ARM IIDR value to avoid activating errata workarounds */
+ reg = FIELD_PREP(IIDR_IMPLEMENTER, IIDR_IMPLEMENTER_ARM) |
+ FIELD_PREP(IIDR_REVISION, 0) |
+ FIELD_PREP(IIDR_VARIANT, 0) |
+ FIELD_PREP(IIDR_PRODUCTID, 0);
+ *r = vreg_reg32_extract(reg, info);
+ break;
+
+ case VREG32(ARM_SMMU_CR0):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cr0_lock);
+ *r = vreg_reg32_extract(smmu->cr[0], info);
+ spin_unlock(&smmu->cr0_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CR0ACK):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cr0_lock);
+ *r = vreg_reg32_extract(smmu->cr0ack, info);
+ spin_unlock(&smmu->cr0_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CR1):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cr1_lock);
+ *r = vreg_reg32_extract(smmu->cr[1], info);
+ spin_unlock(&smmu->cr1_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CR2):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cr2_lock);
+ *r = vreg_reg32_extract(smmu->cr[2], info);
+ spin_unlock(&smmu->cr2_lock);
+ break;
+
+ case VREG64(ARM_SMMU_STRTAB_BASE):
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->strtab_cfg_lock);
+ *r = vreg_reg64_extract(smmu->strtab_base, info);
+ spin_unlock(&smmu->strtab_cfg_lock);
+ break;
+
+ case VREG32(ARM_SMMU_STRTAB_BASE_CFG):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->strtab_cfg_lock);
+ *r = vreg_reg32_extract(smmu->strtab_base_cfg, info);
+ spin_unlock(&smmu->strtab_cfg_lock);
+ break;
+
+ case VREG64(ARM_SMMU_CMDQ_BASE):
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->cmd_queue_lock);
+ *r = vreg_reg64_extract(smmu->cmdq.q_base, info);
+ spin_unlock(&smmu->cmd_queue_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CMDQ_PROD):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cmd_queue_lock);
+ *r = vreg_reg32_extract(smmu->cmdq.prod, info);
+ spin_unlock(&smmu->cmd_queue_lock);
+ break;
+
+ case VREG32(ARM_SMMU_CMDQ_CONS):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->cmd_queue_lock);
+ *r = vreg_reg32_extract(smmu->cmdq.cons, info);
+ spin_unlock(&smmu->cmd_queue_lock);
+ break;
+
+ case VREG64(ARM_SMMU_EVTQ_BASE):
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->evt_queue_lock);
+ *r = vreg_reg64_extract(smmu->evtq.q_base, info);
+ spin_unlock(&smmu->evt_queue_lock);
+ break;
+
+ case VREG32(ARM_SMMU_IRQ_CTRL):
+ case VREG32(ARM_SMMU_IRQ_CTRLACK):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->irq_cfg_lock);
+ *r = vreg_reg32_extract(smmu->irq_ctrl, info);
+ spin_unlock(&smmu->irq_cfg_lock);
+ break;
+
+ case VREG64(ARM_SMMU_GERROR_IRQ_CFG0):
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->irq_cfg_lock);
+ *r = vreg_reg64_extract(smmu->gerror_irq_cfg0, info);
+ spin_unlock(&smmu->irq_cfg_lock);
+ break;
+
+ case VREG64(ARM_SMMU_EVTQ_IRQ_CFG0):
+ if ( dabt.size != DABT_DOUBLE_WORD && dabt.size != DABT_WORD )
+ goto bad_width;
+ spin_lock(&smmu->irq_cfg_lock);
+ *r = vreg_reg64_extract(smmu->evtq_irq_cfg0, info);
+ spin_unlock(&smmu->irq_cfg_lock);
+ break;
+
+ case VREG32(ARM_SMMU_GERROR):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->gerror_lock);
+ *r = vreg_reg32_extract(smmu->gerror, info);
+ spin_unlock(&smmu->gerror_lock);
+ break;
+
+ case VREG32(ARM_SMMU_GERRORN):
+ if ( dabt.size != DABT_WORD ) goto bad_width;
+ spin_lock(&smmu->gerror_lock);
+ *r = vreg_reg32_extract(smmu->gerrorn, info);
+ spin_unlock(&smmu->gerror_lock);
+ break;
+
+ default:
+ printk(XENLOG_G_ERR
+ "%pd: vSMMUv3: unhandled read r%d offset %"PRIpaddr"\n",
+ v->domain, info->dabt.reg, (unsigned long)info->gpa & 0xffff);
+ return IO_HANDLED;
+ }
+ }
+ return IO_HANDLED;
+
+ read_impl_defined:
+ printk(XENLOG_G_DEBUG
+ "%pd: vSMMUv3: RAZ on implementation defined register offset %"PRIpaddr"\n",
+ v->domain, info->gpa & 0xffff);
+ *r = 0;
+ return IO_HANDLED;
+
+ read_reserved:
+ printk(XENLOG_G_DEBUG
+ "%pd: vSMMUv3: RAZ on reserved register offset %"PRIpaddr"\n",
+ v->domain, info->gpa & 0xffff);
+ *r = 0;
+ return IO_HANDLED;
+ bad_width:
+ gprintk(XENLOG_G_ERR,
+ "%pd: vSMMUv3: bad read width %d r%d offset %"PRIpaddr"\n",
+ v->domain, dabt.size, dabt.reg, info->gpa & 0xffff);
return IO_HANDLED;
}
@@ -44,6 +602,17 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
smmu->d = d;
smmu->features = features;
+ smmu->cmdq.ent_size = CMDQ_ENT_DWORDS * DWORDS_BYTES;
+ smmu->evtq.ent_size = EVTQ_ENT_DWORDS * DWORDS_BYTES;
+
+ spin_lock_init(&smmu->cmd_queue_lock);
+ spin_lock_init(&smmu->evt_queue_lock);
+ spin_lock_init(&smmu->gerror_lock);
+ spin_lock_init(&smmu->cr0_lock);
+ spin_lock_init(&smmu->cr1_lock);
+ spin_lock_init(&smmu->cr2_lock);
+ spin_lock_init(&smmu->strtab_cfg_lock);
+ spin_lock_init(&smmu->irq_cfg_lock);
register_mmio_handler(d, &vsmmuv3_mmio_handler, addr, size, smmu);
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 09/22] xen/arm: vsmmuv3: Add support for cmdqueue handling
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (7 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 08/22] xen/arm: vsmmuv3: Add support for registers emulation Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-10 16:58 ` Oleksandr Tyshchenko
2026-10-07 0:06 ` [PATCH v4 10/22] xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE Milan Djokic
` (12 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Add support for virtual cmdqueue handling for guests
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/vsmmu-v3.c | 127 +++++++++++++++++++++++++
1 file changed, 127 insertions(+)
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index c4fef6e8bd..10143927e7 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -8,6 +8,7 @@
* This needs to be fully handled in the future.
*/
+#include <xen/guest_access.h>
#include <xen/param.h>
#include <xen/sched.h>
#include <asm/mmio.h>
@@ -33,6 +34,51 @@
#define SMMU_EVTQS 19
#define DWORDS_BYTES 8
+/*
+ * SMMUv3 command definitions
+ * Some commands are fully handled by the emulation layer, while others are
+ * currently treated as architectural no-ops because the required behavior
+ * is either implicitly guaranteed by Xen or not yet modeled explicitly.
+ *
+ * Emulation handled commands:
+ *
+ * - CMD_CFGI_STE
+ * - CMD_TLBI_NH_ASID
+ * - CMD_TLBI_NSNH_ALL
+ * - CMD_TLBI_NH_VA
+ *
+ * No-op/Implicitly handled commands:
+ *
+ * - CMD_SYNC
+ * - CMD_PREFETCH_CFG
+ * - CMD_CFGI_CD
+ * - CMD_CFGI_CD_ALL
+ * - CMD_CFGI_ALL
+ *
+ * TODO: Remaining architecture-defined commands are not supported (error
+ * produced), due to lack of support in SMMUv3 driver / emulation layer
+ * TODO: Range / per-device invalidation not supported atm
+ * TODO: Long-running command queue handling is not implemented (command
+ * consumption should be budgeted with asynchronous resumption).
+ */
+#define CMDQ_OP_PREFETCH_CFG 0x1
+#define CMDQ_OP_CFGI_STE 0x3
+#define CMDQ_OP_CFGI_ALL 0x4
+#define CMDQ_OP_CFGI_CD 0x5
+#define CMDQ_OP_CFGI_CD_ALL 0x6
+#define CMDQ_OP_TLBI_NH_ASID 0x11
+#define CMDQ_OP_TLBI_NH_VA 0x12
+#define CMDQ_OP_TLBI_NSNH_ALL 0x30
+#define CMDQ_OP_CMD_SYNC 0x46
+
+/* Queue Handling */
+#define Q_BASE(q) ((q)->q_base & Q_BASE_ADDR_MASK)
+#define Q_CONS_ENT(q) (Q_BASE(q) + Q_IDX(q, (q)->cons) * (q)->ent_size)
+#define Q_PROD_ENT(q) (Q_BASE(q) + Q_IDX(q, (q)->prod) * (q)->ent_size)
+
+/* Helper Macros */
+#define smmu_cmd_get_command(x) FIELD_GET(CMDQ_0_OP, x)
+
/* virtual smmu queue */
struct arm_vsmmu_queue {
uint64_t q_base; /* base register */
@@ -99,6 +145,83 @@ static inline bool smmu_get_evtq_enabled(struct virt_smmu *smmu)
spin_unlock(&smmu->cr0_lock);
return enabled;
+};
+
+/* Queue manipulation functions */
+static bool queue_empty(struct arm_vsmmu_queue *q)
+{
+ return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
+ Q_WRP(q, q->prod) == Q_WRP(q, q->cons);
+}
+
+static void queue_inc_cons(struct arm_vsmmu_queue *q)
+{
+ uint32_t cons = (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1;
+ q->cons = Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons);
+}
+
+static void dump_smmu_command(uint64_t *command)
+{
+ gprintk(XENLOG_ERR, "cmd 0x%02llx: %016"PRIx64" %016"PRIx64"\n",
+ smmu_cmd_get_command(command[0]), command[0], command[1]);
+}
+static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
+{
+ struct arm_vsmmu_queue *q = &smmu->cmdq;
+ struct domain *d = smmu->d;
+ uint64_t command[CMDQ_ENT_DWORDS];
+ paddr_t addr;
+ int ret = 0;
+
+ if ( !smmu_get_cmdq_enabled(smmu) )
+ return 0;
+
+ while ( !queue_empty(q) )
+ {
+ int ret;
+
+ addr = Q_CONS_ENT(q);
+ ret = access_guest_memory_by_gpa(d, addr, command,
+ sizeof(command), false);
+ if ( ret ) {
+ queue_inc_cons(q);
+ return ret;
+ }
+
+ switch ( smmu_cmd_get_command(command[0]) )
+ {
+ case CMDQ_OP_CFGI_STE:
+ break;
+ case CMDQ_OP_PREFETCH_CFG:
+ case CMDQ_OP_CFGI_CD:
+ case CMDQ_OP_CFGI_CD_ALL:
+ case CMDQ_OP_CFGI_ALL:
+ case CMDQ_OP_CMD_SYNC:
+ break;
+ case CMDQ_OP_TLBI_NH_ASID:
+ case CMDQ_OP_TLBI_NSNH_ALL:
+ case CMDQ_OP_TLBI_NH_VA:
+ ret = iommu_iotlb_flush_all(smmu->d, 1);
+ if ( !ret )
+ break;
+ fallthrough;
+ default:
+ gdprintk(XENLOG_ERR, "vSMMUv3: unhandled command\n");
+ dump_smmu_command(command);
+ break;
+ }
+
+ if ( ret )
+ {
+ gdprintk(XENLOG_ERR,
+ "vSMMUv3: command error %d while handling command\n",
+ ret);
+ dump_smmu_command(command);
+ }
+ queue_inc_cons(q);
+ }
+
+ return ret;
}
static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
@@ -221,6 +344,10 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
reg32 = smmu->cmdq.prod;
vreg_reg32_update(®32, r, info);
smmu->cmdq.prod = reg32;
+
+ if ( arm_vsmmu_handle_cmds(smmu) )
+ gdprintk(XENLOG_ERR, "error handling vSMMUv3 commands\n");
+
spin_unlock(&smmu->cmd_queue_lock);
break;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 09/22] xen/arm: vsmmuv3: Add support for cmdqueue handling
2026-10-07 0:06 ` [PATCH v4 09/22] xen/arm: vsmmuv3: Add support for cmdqueue handling Milan Djokic
@ 2026-10-10 16:58 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-10 16:58 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
On 10/7/26 03:06, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> Add support for virtual cmdqueue handling for guests
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/drivers/passthrough/arm/vsmmu-v3.c | 127 +++++++++++++++++++++++++
> 1 file changed, 127 insertions(+)
>
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
> index c4fef6e8bd..10143927e7 100644
> --- a/xen/drivers/passthrough/arm/vsmmu-v3.c
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
> @@ -8,6 +8,7 @@
> * This needs to be fully handled in the future.
> */
>
> +#include <xen/guest_access.h>
> #include <xen/param.h>
> #include <xen/sched.h>
> #include <asm/mmio.h>
> @@ -33,6 +34,51 @@
> #define SMMU_EVTQS 19
> #define DWORDS_BYTES 8
>
> +/*
> + * SMMUv3 command definitions
> + * Some commands are fully handled by the emulation layer, while others are
> + * currently treated as architectural no-ops because the required behavior
> + * is either implicitly guaranteed by Xen or not yet modeled explicitly.
> + *
> + * Emulation handled commands:
> + *
> + * - CMD_CFGI_STE
> + * - CMD_TLBI_NH_ASID
> + * - CMD_TLBI_NSNH_ALL
> + * - CMD_TLBI_NH_VA
> + *
> + * No-op/Implicitly handled commands:
> + *
> + * - CMD_SYNC
> + * - CMD_PREFETCH_CFG
> + * - CMD_CFGI_CD
> + * - CMD_CFGI_CD_ALL
> + * - CMD_CFGI_ALL
> + *
> + * TODO: Remaining architecture-defined commands are not supported (error
> + * produced), due to lack of support in SMMUv3 driver / emulation layer
> + * TODO: Range / per-device invalidation not supported atm
> + * TODO: Long-running command queue handling is not implemented (command
> + * consumption should be budgeted with asynchronous resumption).
> + */
> +#define CMDQ_OP_PREFETCH_CFG 0x1
> +#define CMDQ_OP_CFGI_STE 0x3
> +#define CMDQ_OP_CFGI_ALL 0x4
> +#define CMDQ_OP_CFGI_CD 0x5
> +#define CMDQ_OP_CFGI_CD_ALL 0x6
> +#define CMDQ_OP_TLBI_NH_ASID 0x11
> +#define CMDQ_OP_TLBI_NH_VA 0x12
> +#define CMDQ_OP_TLBI_NSNH_ALL 0x30
> +#define CMDQ_OP_CMD_SYNC 0x46
> +
> +/* Queue Handling */
> +#define Q_BASE(q) ((q)->q_base & Q_BASE_ADDR_MASK)
> +#define Q_CONS_ENT(q) (Q_BASE(q) + Q_IDX(q, (q)->cons) * (q)->ent_size)
> +#define Q_PROD_ENT(q) (Q_BASE(q) + Q_IDX(q, (q)->prod) * (q)->ent_size)
> +
> +/* Helper Macros */
> +#define smmu_cmd_get_command(x) FIELD_GET(CMDQ_0_OP, x)
> +
> /* virtual smmu queue */
> struct arm_vsmmu_queue {
> uint64_t q_base; /* base register */
> @@ -99,6 +145,83 @@ static inline bool smmu_get_evtq_enabled(struct virt_smmu *smmu)
> spin_unlock(&smmu->cr0_lock);
>
> return enabled;
> +};
> +
> +/* Queue manipulation functions */
> +static bool queue_empty(struct arm_vsmmu_queue *q)
> +{
> + return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
> + Q_WRP(q, q->prod) == Q_WRP(q, q->cons);
> +}
> +
> +static void queue_inc_cons(struct arm_vsmmu_queue *q)
> +{
> + uint32_t cons = (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1;
> + q->cons = Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons);
> +}
> +
> +static void dump_smmu_command(uint64_t *command)
> +{
> + gprintk(XENLOG_ERR, "cmd 0x%02llx: %016"PRIx64" %016"PRIx64"\n",
> + smmu_cmd_get_command(command[0]), command[0], command[1]);
> +}
> +static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
> +{
> + struct arm_vsmmu_queue *q = &smmu->cmdq;
> + struct domain *d = smmu->d;
> + uint64_t command[CMDQ_ENT_DWORDS];
> + paddr_t addr;
> + int ret = 0;
> +
> + if ( !smmu_get_cmdq_enabled(smmu) )
> + return 0;
> +
> + while ( !queue_empty(q) )
> + {
This is unbounded loop, guest controls PROD and the advertised queue
size is big enough. Below ...
> + int ret;
> +
> + addr = Q_CONS_ENT(q);
> + ret = access_guest_memory_by_gpa(d, addr, command,
> + sizeof(command), false);
> + if ( ret ) {
> + queue_inc_cons(q);
> + return ret;
> + }
> +
> + switch ( smmu_cmd_get_command(command[0]) )
> + {
> + case CMDQ_OP_CFGI_STE:
> + break;
> + case CMDQ_OP_PREFETCH_CFG:
> + case CMDQ_OP_CFGI_CD:
> + case CMDQ_OP_CFGI_CD_ALL:
> + case CMDQ_OP_CFGI_ALL:
> + case CMDQ_OP_CMD_SYNC:
> + break;
> + case CMDQ_OP_TLBI_NH_ASID:
> + case CMDQ_OP_TLBI_NSNH_ALL:
> + case CMDQ_OP_TLBI_NH_VA:
> + ret = iommu_iotlb_flush_all(smmu->d, 1);
... TLBI does full-VMID flush and waits for the hardware. All of this
runs inside one MMIO trap with cmd_queue_lock held, so CPU could stall.
Yes, this is honestly mentioned in the design doc (Security
Considerations, Observation ...) and doc calls it future work. Yes, this
should be sorted out before driver can be treated as supported feature.
So please treat my comment as a remark. For now, I would put a TODO
(e.g., above the while loop so it does not get forgotten.
> + if ( !ret )
> + break;
> + fallthrough;
> + default:
> + gdprintk(XENLOG_ERR, "vSMMUv3: unhandled command\n");
> + dump_smmu_command(command);
> + break;
> + }
> +
> + if ( ret )
> + {
> + gdprintk(XENLOG_ERR,
> + "vSMMUv3: command error %d while handling command\n",
> + ret);
> + dump_smmu_command(command);
> + }
> + queue_inc_cons(q);
> + }
> +
> + return ret;
> }
>
> static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
> @@ -221,6 +344,10 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
> reg32 = smmu->cmdq.prod;
> vreg_reg32_update(®32, r, info);
> smmu->cmdq.prod = reg32;
> +
> + if ( arm_vsmmu_handle_cmds(smmu) )
> + gdprintk(XENLOG_ERR, "error handling vSMMUv3 commands\n");
> +
> spin_unlock(&smmu->cmd_queue_lock);
> break;
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 10/22] xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (8 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 09/22] xen/arm: vsmmuv3: Add support for cmdqueue handling Milan Djokic
@ 2026-10-07 0:06 ` Milan Djokic
2026-10-10 16:14 ` Oleksandr Tyshchenko
2026-10-07 0:07 ` [PATCH v4 11/22] xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware Milan Djokic
` (11 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:06 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
CMD_CFGI_STE is used to invalidate/validate the STE. Emulated vSMMUv3
driver in XEN will read the STE from the guest memory space and capture
the Stage-1 configuration required to support nested translation.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/vsmmu-v3.c | 169 +++++++++++++++++++++++++
1 file changed, 169 insertions(+)
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index 10143927e7..25ba734fa7 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -78,6 +78,21 @@
/* Helper Macros */
#define smmu_cmd_get_command(x) FIELD_GET(CMDQ_0_OP, x)
+#define smmu_cmd_get_sid(x) FIELD_GET(CMDQ_CFGI_0_SID, x)
+#define smmu_get_ste_s1cdmax(x) FIELD_GET(STRTAB_STE_0_S1CDMAX, x)
+#define smmu_get_ste_s1fmt(x) FIELD_GET(STRTAB_STE_0_S1FMT, x)
+#define smmu_get_ste_s1stalld(x) FIELD_GET(STRTAB_STE_1_S1STALLD, x)
+#define smmu_get_ste_s1ctxptr(x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \
+ FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x))
+
+/* stage-1 translation configuration */
+struct arm_vsmmu_s1_trans_cfg {
+ paddr_t s1ctxptr;
+ uint8_t s1fmt;
+ uint8_t s1cdmax;
+ bool bypassed; /* translation is bypassed */
+ bool aborted; /* translation is aborted */
+};
/* virtual smmu queue */
struct arm_vsmmu_queue {
@@ -165,6 +180,159 @@ static void dump_smmu_command(uint64_t *command)
gprintk(XENLOG_ERR, "cmd 0x%02llx: %016"PRIx64" %016"PRIx64"\n",
smmu_cmd_get_command(command[0]), command[0], command[1]);
}
+
+static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
+ uint64_t *ste)
+{
+ paddr_t addr, strtab_base;
+ struct domain *d = smmu->d;
+ uint32_t log2size;
+ int strtab_size_shift;
+ int ret;
+
+ spin_lock(&smmu->strtab_cfg_lock);
+ log2size = MIN(FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, smmu->strtab_base_cfg),
+ SMMU_IDR1_SIDSIZE);
+ spin_unlock(&smmu->strtab_cfg_lock);
+
+ if ( sid >= (1U << log2size) )
+ return -EINVAL;
+
+ spin_lock(&smmu->strtab_cfg_lock);
+ if ( FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->strtab_base_cfg) ==
+ STRTAB_BASE_CFG_FMT_2LVL )
+ {
+ int idx, max_l2_ste, span;
+ paddr_t l1ptr, l2ptr;
+ uint64_t l1std;
+ uint8_t sid_split = FIELD_GET(STRTAB_BASE_CFG_SPLIT,
+ smmu->strtab_base_cfg);
+
+ /*
+ * Reserved SPLIT values behave as 6 according to the architecture.
+ */
+ if ( sid_split != 0x6 && sid_split != 0x8 && sid_split != 0xA )
+ {
+ sid_split = 0x6;
+ }
+
+ strtab_size_shift = MAX(5, (int)log2size - sid_split - 1 + 3);
+ strtab_base = smmu->strtab_base & STRTAB_BASE_ADDR_MASK &
+ ~GENMASK_ULL(strtab_size_shift, 0);
+ spin_unlock(&smmu->strtab_cfg_lock);
+ idx = (sid >> sid_split) * STRTAB_L1_DESC_DWORDS;
+ l1ptr = (paddr_t)(strtab_base + idx * sizeof(l1std));
+
+ ret = access_guest_memory_by_gpa(d, l1ptr, &l1std,
+ sizeof(l1std), false);
+ if ( ret )
+ {
+ gdprintk(XENLOG_ERR,
+ "Could not read L1PTR at 0X%"PRIx64"\n", l1ptr);
+ return ret;
+ }
+
+ span = FIELD_GET(STRTAB_L1_DESC_SPAN, l1std);
+
+ /* Validate SPAN against the architecturally defined range and SPLIT. */
+ if ( !span || span > 0xB || span > sid_split + 1 )
+ {
+ /* L2Ptr is invalid; this descriptor matches no valid STEs. */
+ return -EINVAL;
+ }
+
+ max_l2_ste = (1U << (span - 1)) - 1;
+ l2ptr = FIELD_PREP(STRTAB_L1_DESC_L2PTR_MASK,
+ FIELD_GET(STRTAB_L1_DESC_L2PTR_MASK, l1std));
+ idx = sid & ((1 << sid_split) - 1);
+ if ( idx > max_l2_ste )
+ {
+ gdprintk(XENLOG_ERR, "idx=%d > max_l2_ste=%d\n",
+ idx, max_l2_ste);
+ return -EINVAL;
+ }
+ addr = l2ptr + idx * sizeof(*ste) * STRTAB_STE_DWORDS;
+ }
+ else
+ {
+ strtab_size_shift = log2size + 5;
+ strtab_base = smmu->strtab_base & STRTAB_BASE_ADDR_MASK &
+ ~GENMASK_ULL(strtab_size_shift, 0);
+ spin_unlock(&smmu->strtab_cfg_lock);
+ addr = strtab_base + sid * sizeof(*ste) * STRTAB_STE_DWORDS;
+ }
+ ret = access_guest_memory_by_gpa(d, addr, ste, sizeof(*ste), false);
+ if ( ret )
+ {
+ gdprintk(XENLOG_ERR,
+ "Cannot fetch pte at address=0x%"PRIx64"\n", addr);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int arm_vsmmu_decode_ste(struct virt_smmu *smmu, uint32_t sid,
+ struct arm_vsmmu_s1_trans_cfg *cfg,
+ uint64_t *ste)
+{
+ uint64_t val = ste[0];
+
+ if ( !(val & STRTAB_STE_0_V) )
+ return -EAGAIN;
+
+ switch ( FIELD_GET(STRTAB_STE_0_CFG, val) )
+ {
+ case STRTAB_STE_0_CFG_BYPASS:
+ cfg->bypassed = true;
+ return 0;
+ case STRTAB_STE_0_CFG_ABORT:
+ cfg->aborted = true;
+ return 0;
+ case STRTAB_STE_0_CFG_S1_TRANS:
+ break;
+ case STRTAB_STE_0_CFG_S2_TRANS:
+ gdprintk(XENLOG_ERR, "vSMMUv3 does not support stage 2 yet\n");
+ goto bad_ste;
+ default:
+ gdprintk(XENLOG_ERR, "Invalid STE config type\n");
+ goto bad_ste;
+ }
+
+ cfg->s1ctxptr = smmu_get_ste_s1ctxptr(val);
+ cfg->s1fmt = smmu_get_ste_s1fmt(val);
+ cfg->s1cdmax = smmu_get_ste_s1cdmax(val);
+ if ( cfg->s1cdmax != 0 )
+ {
+ gdprintk(XENLOG_ERR,
+ "vSMMUv3 does not support multiple context descriptors\n");
+ goto bad_ste;
+ }
+
+ return 0;
+
+ bad_ste:
+ return -EINVAL;
+}
+
+static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
+{
+ int ret;
+ uint64_t ste[STRTAB_STE_DWORDS];
+ struct arm_vsmmu_s1_trans_cfg s1_cfg = {0};
+ uint32_t sid = smmu_cmd_get_sid(cmdptr[0]);
+
+ ret = arm_vsmmu_find_ste(smmu, sid, ste);
+ if ( ret )
+ return ret;
+
+ ret = arm_vsmmu_decode_ste(smmu, sid, &s1_cfg, ste);
+ if ( ret )
+ return (ret == -EAGAIN ) ? 0 : ret;
+
+ return 0;
+}
+
static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
{
struct arm_vsmmu_queue *q = &smmu->cmdq;
@@ -191,6 +359,7 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
switch ( smmu_cmd_get_command(command[0]) )
{
case CMDQ_OP_CFGI_STE:
+ ret = arm_vsmmu_handle_cfgi_ste(smmu, command);
break;
case CMDQ_OP_PREFETCH_CFG:
case CMDQ_OP_CFGI_CD:
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 10/22] xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE
2026-10-07 0:06 ` [PATCH v4 10/22] xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE Milan Djokic
@ 2026-10-10 16:14 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-10 16:14 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
On 10/7/26 03:06, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> CMD_CFGI_STE is used to invalidate/validate the STE. Emulated vSMMUv3
> driver in XEN will read the STE from the guest memory space and capture
> the Stage-1 configuration required to support nested translation.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/drivers/passthrough/arm/vsmmu-v3.c | 169 +++++++++++++++++++++++++
> 1 file changed, 169 insertions(+)
>
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
> index 10143927e7..25ba734fa7 100644
> --- a/xen/drivers/passthrough/arm/vsmmu-v3.c
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
> @@ -78,6 +78,21 @@
>
> /* Helper Macros */
> #define smmu_cmd_get_command(x) FIELD_GET(CMDQ_0_OP, x)
> +#define smmu_cmd_get_sid(x) FIELD_GET(CMDQ_CFGI_0_SID, x)
> +#define smmu_get_ste_s1cdmax(x) FIELD_GET(STRTAB_STE_0_S1CDMAX, x)
> +#define smmu_get_ste_s1fmt(x) FIELD_GET(STRTAB_STE_0_S1FMT, x)
> +#define smmu_get_ste_s1stalld(x) FIELD_GET(STRTAB_STE_1_S1STALLD, x)
> +#define smmu_get_ste_s1ctxptr(x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \
> + FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x))
> +
> +/* stage-1 translation configuration */
> +struct arm_vsmmu_s1_trans_cfg {
> + paddr_t s1ctxptr;
> + uint8_t s1fmt;
> + uint8_t s1cdmax;
> + bool bypassed; /* translation is bypassed */
> + bool aborted; /* translation is aborted */
> +};
>
> /* virtual smmu queue */
> struct arm_vsmmu_queue {
> @@ -165,6 +180,159 @@ static void dump_smmu_command(uint64_t *command)
> gprintk(XENLOG_ERR, "cmd 0x%02llx: %016"PRIx64" %016"PRIx64"\n",
> smmu_cmd_get_command(command[0]), command[0], command[1]);
> }
> +
> +static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
> + uint64_t *ste)
> +{
> + paddr_t addr, strtab_base;
> + struct domain *d = smmu->d;
> + uint32_t log2size;
> + int strtab_size_shift;
> + int ret;
> +
> + spin_lock(&smmu->strtab_cfg_lock);
> + log2size = MIN(FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, smmu->strtab_base_cfg),
> + SMMU_IDR1_SIDSIZE);
> + spin_unlock(&smmu->strtab_cfg_lock);
> +
> + if ( sid >= (1U << log2size) )
> + return -EINVAL;
> +
> + spin_lock(&smmu->strtab_cfg_lock);
> + if ( FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->strtab_base_cfg) ==
> + STRTAB_BASE_CFG_FMT_2LVL )
> + {
> + int idx, max_l2_ste, span;
> + paddr_t l1ptr, l2ptr;
> + uint64_t l1std;
> + uint8_t sid_split = FIELD_GET(STRTAB_BASE_CFG_SPLIT,
> + smmu->strtab_base_cfg);
> +
> + /*
> + * Reserved SPLIT values behave as 6 according to the architecture.
> + */
> + if ( sid_split != 0x6 && sid_split != 0x8 && sid_split != 0xA )
> + {
> + sid_split = 0x6;
> + }
> +
> + strtab_size_shift = MAX(5, (int)log2size - sid_split - 1 + 3);
> + strtab_base = smmu->strtab_base & STRTAB_BASE_ADDR_MASK &
> + ~GENMASK_ULL(strtab_size_shift, 0);
> + spin_unlock(&smmu->strtab_cfg_lock);
> + idx = (sid >> sid_split) * STRTAB_L1_DESC_DWORDS;
> + l1ptr = (paddr_t)(strtab_base + idx * sizeof(l1std));
> +
> + ret = access_guest_memory_by_gpa(d, l1ptr, &l1std,
> + sizeof(l1std), false);
> + if ( ret )
> + {
> + gdprintk(XENLOG_ERR,
> + "Could not read L1PTR at 0X%"PRIx64"\n", l1ptr);
> + return ret;
> + }
> +
> + span = FIELD_GET(STRTAB_L1_DESC_SPAN, l1std);
> +
> + /* Validate SPAN against the architecturally defined range and SPLIT. */
> + if ( !span || span > 0xB || span > sid_split + 1 )
> + {
> + /* L2Ptr is invalid; this descriptor matches no valid STEs. */
> + return -EINVAL;
> + }
> +
> + max_l2_ste = (1U << (span - 1)) - 1;
> + l2ptr = FIELD_PREP(STRTAB_L1_DESC_L2PTR_MASK,
> + FIELD_GET(STRTAB_L1_DESC_L2PTR_MASK, l1std));
> + idx = sid & ((1 << sid_split) - 1);
> + if ( idx > max_l2_ste )
> + {
> + gdprintk(XENLOG_ERR, "idx=%d > max_l2_ste=%d\n",
> + idx, max_l2_ste);
> + return -EINVAL;
> + }
> + addr = l2ptr + idx * sizeof(*ste) * STRTAB_STE_DWORDS;
> + }
> + else
> + {
> + strtab_size_shift = log2size + 5;
> + strtab_base = smmu->strtab_base & STRTAB_BASE_ADDR_MASK &
> + ~GENMASK_ULL(strtab_size_shift, 0);
> + spin_unlock(&smmu->strtab_cfg_lock);
> + addr = strtab_base + sid * sizeof(*ste) * STRTAB_STE_DWORDS;
> + }
> + ret = access_guest_memory_by_gpa(d, addr, ste, sizeof(*ste), false);
> + if ( ret )
> + {
> + gdprintk(XENLOG_ERR,
> + "Cannot fetch pte at address=0x%"PRIx64"\n", addr);
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static int arm_vsmmu_decode_ste(struct virt_smmu *smmu, uint32_t sid,
> + struct arm_vsmmu_s1_trans_cfg *cfg,
> + uint64_t *ste)
> +{
> + uint64_t val = ste[0];
> +
> + if ( !(val & STRTAB_STE_0_V) )
> + return -EAGAIN;
I am sorry, but I do not follow why invalid entry ...
> +
> + switch ( FIELD_GET(STRTAB_STE_0_CFG, val) )
> + {
> + case STRTAB_STE_0_CFG_BYPASS:
> + cfg->bypassed = true;
> + return 0;
> + case STRTAB_STE_0_CFG_ABORT:
> + cfg->aborted = true;
> + return 0;
> + case STRTAB_STE_0_CFG_S1_TRANS:
> + break;
> + case STRTAB_STE_0_CFG_S2_TRANS:
> + gdprintk(XENLOG_ERR, "vSMMUv3 does not support stage 2 yet\n");
> + goto bad_ste;
> + default:
> + gdprintk(XENLOG_ERR, "Invalid STE config type\n");
> + goto bad_ste;
> + }
> +
> + cfg->s1ctxptr = smmu_get_ste_s1ctxptr(val);
> + cfg->s1fmt = smmu_get_ste_s1fmt(val);
> + cfg->s1cdmax = smmu_get_ste_s1cdmax(val);
> + if ( cfg->s1cdmax != 0 )
> + {
> + gdprintk(XENLOG_ERR,
> + "vSMMUv3 does not support multiple context descriptors\n");
> + goto bad_ste;
> + }
> +
> + return 0;
> +
> + bad_ste:
> + return -EINVAL;
> +}
> +
> +static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
> +{
> + int ret;
> + uint64_t ste[STRTAB_STE_DWORDS];
NIT: Looks like ste[0] is only used today (first 8 bytes), the rest is
uninitialised.
> + struct arm_vsmmu_s1_trans_cfg s1_cfg = {0};
> + uint32_t sid = smmu_cmd_get_sid(cmdptr[0]);
> +
> + ret = arm_vsmmu_find_ste(smmu, sid, ste);
> + if ( ret )
> + return ret;
> +
> + ret = arm_vsmmu_decode_ste(smmu, sid, &s1_cfg, ste);
> + if ( ret )
> + return (ret == -EAGAIN ) ? 0 : ret;
... is treated as success an no-op. The physical STE keeps its old
guest_cfg (added by subsequent commit) after the guest invalidates it.
Could you please clarify why this is ok. Should not it become abort?
> +
> + return 0;
> +}
> +
> static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
> {
> struct arm_vsmmu_queue *q = &smmu->cmdq;
> @@ -191,6 +359,7 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
> switch ( smmu_cmd_get_command(command[0]) )
> {
> case CMDQ_OP_CFGI_STE:
> + ret = arm_vsmmu_handle_cfgi_ste(smmu, command);
> break;
> case CMDQ_OP_PREFETCH_CFG:
> case CMDQ_OP_CFGI_CD:
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 11/22] xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (9 preceding siblings ...)
2026-10-07 0:06 ` [PATCH v4 10/22] xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-10 14:47 ` Oleksandr Tyshchenko
2026-10-07 0:07 ` [PATCH v4 12/22] xen/arm: vsmmuv3: Add support for event queue and global error Milan Djokic
` (10 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Jan Beulich,
Roger Pau Monné, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Attach the Stage-1 configuration to device STE to support nested
translation for the guests.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/arch/arm/include/asm/iommu.h | 7 ++
xen/drivers/passthrough/arm/smmu-v3.c | 105 +++++++++++++++++++++++++
xen/drivers/passthrough/arm/smmu-v3.h | 1 +
xen/drivers/passthrough/arm/vsmmu-v3.c | 18 +++++
xen/include/xen/iommu.h | 6 ++
5 files changed, 137 insertions(+)
diff --git a/xen/arch/arm/include/asm/iommu.h b/xen/arch/arm/include/asm/iommu.h
index ad15477e24..56bc9314a7 100644
--- a/xen/arch/arm/include/asm/iommu.h
+++ b/xen/arch/arm/include/asm/iommu.h
@@ -20,6 +20,13 @@ struct arch_iommu
void *priv;
};
+struct iommu_guest_config {
+ paddr_t s1ctxptr;
+ uint8_t config;
+ uint8_t s1fmt;
+ uint8_t s1cdmax;
+};
+
const struct iommu_ops *iommu_get_ops(void);
void iommu_set_ops(const struct iommu_ops *ops);
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index 9d2b8a708b..196ac660ef 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -3003,6 +3003,37 @@ static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev)
return NULL;
}
+static struct iommu_domain *arm_smmu_get_domain_by_sid(struct domain *d,
+ u32 sid)
+{
+ int i;
+ unsigned long flags;
+ struct iommu_domain *io_domain;
+ struct arm_smmu_domain *smmu_domain;
+ struct arm_smmu_master *master;
+ struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+
+ /*
+ * Loop through the &xen_domain->contexts to locate a context
+ * associated with the target SMMU and device SID
+ */
+ list_for_each_entry(io_domain, &xen_domain->contexts, list) {
+ smmu_domain = to_smmu_domain(io_domain);
+
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ list_for_each_entry(master, &smmu_domain->devices, domain_head) {
+ for (i = 0; i < master->num_streams; i++) {
+ if (sid != master->streams[i].id)
+ continue;
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+ return io_domain;
+ }
+ }
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+ }
+ return NULL;
+}
+
static struct iommu_domain *arm_smmu_get_domain(struct domain *d,
struct device *dev)
{
@@ -3216,6 +3247,79 @@ static void arm_smmu_iommu_xen_domain_teardown(struct domain *d)
xfree(xen_domain);
}
+static int arm_smmu_attach_guest_config(struct domain *d, uint32_t sid,
+ const struct iommu_guest_config *cfg)
+{
+ int ret = -EINVAL;
+ unsigned long flags;
+ struct arm_smmu_master *master;
+ struct arm_smmu_domain *smmu_domain;
+ struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+ struct iommu_domain *io_domain = arm_smmu_get_domain_by_sid(d, sid);
+
+ if (!io_domain)
+ return -ENODEV;
+
+ smmu_domain = to_smmu_domain(io_domain);
+
+ spin_lock(&xen_domain->lock);
+
+ switch (cfg->config) {
+ case ARM_SMMU_DOMAIN_ABORT:
+ smmu_domain->abort = true;
+ smmu_domain->stage = cfg->config;
+ break;
+ case ARM_SMMU_DOMAIN_BYPASS:
+ smmu_domain->abort = false;
+ smmu_domain->stage = cfg->config;
+ break;
+ case ARM_SMMU_DOMAIN_S1:
+ /* Check stage-1 support */
+ if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_TRANS_S1)) {
+ dev_info(smmu_domain->smmu->dev,
+ "stage-1 not implemented\n");
+ goto out;
+ }
+
+ /* If stage-2 is not supported, configure stage-1 only */
+ if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_TRANS_S2)) {
+ dev_warn(smmu_domain->smmu->dev,
+ "SMMU does not implement stage-2 translation support. "
+ "Fallback to stage-1-only translation configuration\n");
+ /* Enable Stage-1 translation. */
+ smmu_domain->stage = ARM_SMMU_DOMAIN_S1;
+ }
+ else {
+ /* Check nested support */
+ if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_NESTING)) {
+ dev_info(smmu_domain->smmu->dev,
+ "Nested translation not supported\n");
+ goto out;
+ }
+ smmu_domain->stage = ARM_SMMU_DOMAIN_NESTED;
+ }
+
+ /* Enable Stage-1 translation. */
+ smmu_domain->s1_cfg.s1ctxptr = cfg->s1ctxptr;
+ smmu_domain->s1_cfg.s1fmt = cfg->s1fmt;
+ smmu_domain->s1_cfg.s1cdmax = cfg->s1cdmax;
+ smmu_domain->abort = false;
+ break;
+ default:
+ goto out;
+ }
+
+ spin_lock_irqsave(&smmu_domain->devices_lock, flags);
+ list_for_each_entry(master, &smmu_domain->devices, domain_head)
+ arm_smmu_install_ste_for_dev(master);
+ spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
+
+ ret = 0;
+out:
+ spin_unlock(&xen_domain->lock);
+ return ret;
+}
+
static const struct iommu_ops arm_smmu_iommu_ops = {
.page_sizes = PAGE_SIZE_4K,
.init = arm_smmu_iommu_xen_domain_init,
@@ -3228,6 +3332,7 @@ static const struct iommu_ops arm_smmu_iommu_ops = {
.unmap_page = arm_iommu_unmap_page,
.dt_xlate = arm_smmu_dt_xlate,
.add_device = arm_smmu_add_device,
+ .attach_guest_config = arm_smmu_attach_guest_config,
};
static __init int arm_smmu_dt_init(struct dt_device_node *dev,
diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
index ef492098ae..4a912be740 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.h
+++ b/xen/drivers/passthrough/arm/smmu-v3.h
@@ -402,6 +402,7 @@ enum arm_smmu_domain_stage {
ARM_SMMU_DOMAIN_S2,
ARM_SMMU_DOMAIN_NESTED,
ARM_SMMU_DOMAIN_BYPASS,
+ ARM_SMMU_DOMAIN_ABORT,
};
/* Xen specific code. */
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index 25ba734fa7..3f557d1c4d 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -319,8 +319,11 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
{
int ret;
uint64_t ste[STRTAB_STE_DWORDS];
+ struct domain *d = smmu->d;
+ struct domain_iommu *hd = dom_iommu(d);
struct arm_vsmmu_s1_trans_cfg s1_cfg = {0};
uint32_t sid = smmu_cmd_get_sid(cmdptr[0]);
+ struct iommu_guest_config guest_cfg = {0};
ret = arm_vsmmu_find_ste(smmu, sid, ste);
if ( ret )
@@ -330,6 +333,21 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
if ( ret )
return (ret == -EAGAIN ) ? 0 : ret;
+ guest_cfg.s1ctxptr = s1_cfg.s1ctxptr;
+ guest_cfg.s1fmt = s1_cfg.s1fmt;
+ guest_cfg.s1cdmax = s1_cfg.s1cdmax;
+
+ if ( s1_cfg.bypassed )
+ guest_cfg.config = ARM_SMMU_DOMAIN_BYPASS;
+ else if ( s1_cfg.aborted )
+ guest_cfg.config = ARM_SMMU_DOMAIN_ABORT;
+ else
+ guest_cfg.config = ARM_SMMU_DOMAIN_S1;
+
+ ret = hd->platform_ops->attach_guest_config(d, sid, &guest_cfg);
+ if ( ret )
+ return ret;
+
return 0;
}
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 37c4a1dc82..253f8bb6cb 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -312,6 +312,7 @@ static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
#endif /* HAS_DEVICE_TREE_DISCOVERY */
struct page_info;
+struct iommu_guest_config;
/*
* Any non-zero value returned from callbacks of this type will cause the
@@ -387,6 +388,11 @@ struct iommu_ops {
#endif
/* Inhibit all interrupt generation, to be used at shutdown. */
void (*quiesce)(void);
+
+#ifdef CONFIG_ARM
+ int (*attach_guest_config)(struct domain *d, uint32_t sid,
+ const struct iommu_guest_config *cfg);
+#endif
};
/*
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 11/22] xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware
2026-10-07 0:07 ` [PATCH v4 11/22] xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware Milan Djokic
@ 2026-10-10 14:47 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-10 14:47 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Jan Beulich,
Roger Pau Monné
On 10/7/26 03:07, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> Attach the Stage-1 configuration to device STE to support nested
> translation for the guests.
My review of this is based on the full series being applied and includes
the surrounding code context.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/arch/arm/include/asm/iommu.h | 7 ++
> xen/drivers/passthrough/arm/smmu-v3.c | 105 +++++++++++++++++++++++++
> xen/drivers/passthrough/arm/smmu-v3.h | 1 +
> xen/drivers/passthrough/arm/vsmmu-v3.c | 18 +++++
> xen/include/xen/iommu.h | 6 ++
> 5 files changed, 137 insertions(+)
>
> diff --git a/xen/arch/arm/include/asm/iommu.h b/xen/arch/arm/include/asm/iommu.h
> index ad15477e24..56bc9314a7 100644
> --- a/xen/arch/arm/include/asm/iommu.h
> +++ b/xen/arch/arm/include/asm/iommu.h
> @@ -20,6 +20,13 @@ struct arch_iommu
> void *priv;
> };
>
> +struct iommu_guest_config {
> + paddr_t s1ctxptr;
> + uint8_t config;
> + uint8_t s1fmt;
> + uint8_t s1cdmax;
> +};
> +
> const struct iommu_ops *iommu_get_ops(void);
> void iommu_set_ops(const struct iommu_ops *ops);
>
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
> index 9d2b8a708b..196ac660ef 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.c
> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
> @@ -3003,6 +3003,37 @@ static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev)
> return NULL;
> }
>
> +static struct iommu_domain *arm_smmu_get_domain_by_sid(struct domain *d,
> + u32 sid)
> +{
> + int i;
> + unsigned long flags;
> + struct iommu_domain *io_domain;
> + struct arm_smmu_domain *smmu_domain;
> + struct arm_smmu_master *master;
> + struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> +
> + /*
> + * Loop through the &xen_domain->contexts to locate a context
> + * associated with the target SMMU and device SID
> + */
> + list_for_each_entry(io_domain, &xen_domain->contexts, list) {
> + smmu_domain = to_smmu_domain(io_domain);
> +
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> + for (i = 0; i < master->num_streams; i++) {
> + if (sid != master->streams[i].id)
> + continue;
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> + return io_domain;
> + }
> + }
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> + }
> + return NULL;
> +}
> +
> static struct iommu_domain *arm_smmu_get_domain(struct domain *d,
> struct device *dev)
> {
> @@ -3216,6 +3247,79 @@ static void arm_smmu_iommu_xen_domain_teardown(struct domain *d)
> xfree(xen_domain);
> }
>
> +static int arm_smmu_attach_guest_config(struct domain *d, uint32_t sid,
> + const struct iommu_guest_config *cfg)
> +{
> + int ret = -EINVAL;
> + unsigned long flags;
> + struct arm_smmu_master *master;
> + struct arm_smmu_domain *smmu_domain;
> + struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> + struct iommu_domain *io_domain = arm_smmu_get_domain_by_sid(d, sid);
> +
> + if (!io_domain)
> + return -ENODEV;
> +
> + smmu_domain = to_smmu_domain(io_domain);
I might be wrong, but it feels to me that io_domain should be fetched
after acquiring the lock below since concurrent deassign can free the
io_domain. arm_smmu_deassign_dev() takes xen_domain->lock, decrements
io_domain->ref, and if it hits 0, calls arm_smmu_destroy_iommu_domain()
which frees the memory.
> +
> + spin_lock(&xen_domain->lock);
> +
> + switch (cfg->config) {
> + case ARM_SMMU_DOMAIN_ABORT:
> + smmu_domain->abort = true;
> + smmu_domain->stage = cfg->config;
> + break;
> + case ARM_SMMU_DOMAIN_BYPASS:
> + smmu_domain->abort = false;
> + smmu_domain->stage = cfg->config;
I am not quite sure where the best place is to leave this comment, but
unless I am missing something, it looks like a guest can trivially
disable stage-2 translation.
If the guest requests CFG_BYPASS, arm_vsmmu_decode_ste() sets bypassed
and guest_cfg.config becomes ARM_SMMU_DOMAIN_BYPASS.
arm_smmu_attach_guest_config() then sets that stage. Consequently,
arm_smmu_write_strtab_ent() finds neither s1_cfg nor s2_cfg and writes
the physical STE as CFG_BYPASS, which bypasses both stages.
So any device the guest puts in bypass (which Linux might legitimately
do for identity domains) can read and write any physical memory,
including Xen’s memory. Even for well-behaved guests, this is
problematic: the device would use IOVA (which equals IPA) directly as a
host physical address instead of going through the domain’s P2M.
I think that guest bypass request should map to S2-only physical STE.
> + break;
> + case ARM_SMMU_DOMAIN_S1:
> + /* Check stage-1 support */
> + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_TRANS_S1)) {
> + dev_info(smmu_domain->smmu->dev,
> + "stage-1 not implemented\n");
> + goto out;
> + }
> +
> + /* If stage-2 is not supported, configure stage-1 only */
> + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_TRANS_S2)) {
> + dev_warn(smmu_domain->smmu->dev,
> + "SMMU does not implement stage-2 translation support. "
> + "Fallback to stage-1-only translation configuration\n");
> + /* Enable Stage-1 translation. */
> + smmu_domain->stage = ARM_SMMU_DOMAIN_S1;
As I already mentioned for commit #02/22, Without stage-2 there is no
layer to turn the stage-1 output's IPA into PA. Should we allow this
fallback only for privileged, direct-mapped domains?
And apologies if this has already been discussed.
> + }
> + else {
> + /* Check nested support */
> + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_NESTING)) {
> + dev_info(smmu_domain->smmu->dev,
> + "Nested translation not supported\n");
> + goto out;
According to commit #07/22 "MMU-600 variant < 2 and all MMU-700" are
affected. But it seems that Xen still offers SMMU, a guest still sees
SMMU and might believe it is working. The attach fails with only a log
line (gdprintk in arm_vsmmu_handle_cmds()) and the guest is never told.
Device DMA then might land in the wrong guest memory.
Please consider gating this at viommu_get_type() or at domain creation.
> + }
> + smmu_domain->stage = ARM_SMMU_DOMAIN_NESTED;
> + }
> +
> + /* Enable Stage-1 translation. */
> + smmu_domain->s1_cfg.s1ctxptr = cfg->s1ctxptr;
> + smmu_domain->s1_cfg.s1fmt = cfg->s1fmt;
> + smmu_domain->s1_cfg.s1cdmax = cfg->s1cdmax;
> + smmu_domain->abort = false;
Since all this stored per smmu_domain, one domain-wide S1 config
(bypass, abort, context pointer) is applied to all of device’s SIDs
(from the design doc I got this as intentional for current driver
implementation) ...
> + break;
> + default:
> + goto out;
> + }
> +
> + spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> + list_for_each_entry(master, &smmu_domain->devices, domain_head)
> + arm_smmu_install_ste_for_dev(master);
... ^^^ loops over all masters and applies the identical config to all
of them. If, lets say, a device (PCI?) exposes several SIDs, and the
guest tries to program SID 0 with one mappings and SID 1 with a
different mappings, the second command will silently overwrite SID 0's
mappings.
I do not want to ask for extra work, but should Xen at least report an
error when a guest programs different configs, rather than silently
overwriting the previous one? Or at least leave a TODO?
> + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags);
> +
> + ret = 0;
> +out:
> + spin_unlock(&xen_domain->lock);
> + return ret;
> +}
> +
> static const struct iommu_ops arm_smmu_iommu_ops = {
> .page_sizes = PAGE_SIZE_4K,
> .init = arm_smmu_iommu_xen_domain_init,
> @@ -3228,6 +3332,7 @@ static const struct iommu_ops arm_smmu_iommu_ops = {
> .unmap_page = arm_iommu_unmap_page,
> .dt_xlate = arm_smmu_dt_xlate,
> .add_device = arm_smmu_add_device,
> + .attach_guest_config = arm_smmu_attach_guest_config,
> };
>
> static __init int arm_smmu_dt_init(struct dt_device_node *dev,
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
> index ef492098ae..4a912be740 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.h
> +++ b/xen/drivers/passthrough/arm/smmu-v3.h
> @@ -402,6 +402,7 @@ enum arm_smmu_domain_stage {
> ARM_SMMU_DOMAIN_S2,
> ARM_SMMU_DOMAIN_NESTED,
> ARM_SMMU_DOMAIN_BYPASS,
> + ARM_SMMU_DOMAIN_ABORT,
> };
>
> /* Xen specific code. */
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
> index 25ba734fa7..3f557d1c4d 100644
> --- a/xen/drivers/passthrough/arm/vsmmu-v3.c
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
> @@ -319,8 +319,11 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
> {
> int ret;
> uint64_t ste[STRTAB_STE_DWORDS];
> + struct domain *d = smmu->d;
> + struct domain_iommu *hd = dom_iommu(d);
> struct arm_vsmmu_s1_trans_cfg s1_cfg = {0};
> uint32_t sid = smmu_cmd_get_sid(cmdptr[0]);
> + struct iommu_guest_config guest_cfg = {0};
>
> ret = arm_vsmmu_find_ste(smmu, sid, ste);
> if ( ret )
> @@ -330,6 +333,21 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
> if ( ret )
> return (ret == -EAGAIN ) ? 0 : ret;
>
> + guest_cfg.s1ctxptr = s1_cfg.s1ctxptr;
> + guest_cfg.s1fmt = s1_cfg.s1fmt;
> + guest_cfg.s1cdmax = s1_cfg.s1cdmax;
> +
> + if ( s1_cfg.bypassed )
> + guest_cfg.config = ARM_SMMU_DOMAIN_BYPASS;
> + else if ( s1_cfg.aborted )
> + guest_cfg.config = ARM_SMMU_DOMAIN_ABORT;
> + else
> + guest_cfg.config = ARM_SMMU_DOMAIN_S1;
> +
> + ret = hd->platform_ops->attach_guest_config(d, sid, &guest_cfg);
> + if ( ret )
> + return ret;
> +
> return 0;
> }
>
> diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
> index 37c4a1dc82..253f8bb6cb 100644
> --- a/xen/include/xen/iommu.h
> +++ b/xen/include/xen/iommu.h
> @@ -312,6 +312,7 @@ static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
> #endif /* HAS_DEVICE_TREE_DISCOVERY */
>
> struct page_info;
> +struct iommu_guest_config;
>
> /*
> * Any non-zero value returned from callbacks of this type will cause the
> @@ -387,6 +388,11 @@ struct iommu_ops {
> #endif
> /* Inhibit all interrupt generation, to be used at shutdown. */
> void (*quiesce)(void);
> +
> +#ifdef CONFIG_ARM
> + int (*attach_guest_config)(struct domain *d, uint32_t sid,
> + const struct iommu_guest_config *cfg);
> +#endif
> };
>
> /*
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 12/22] xen/arm: vsmmuv3: Add support for event queue and global error
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (10 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 11/22] xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-11 13:27 ` Oleksandr Tyshchenko
2026-10-07 0:07 ` [PATCH v4 13/22] xen/arm: vsmmuv3: IOMMU device tree node for dom0 Milan Djokic
` (9 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Event queue is used to send the events to guest when there is an events/
faults. Add support for event queue to send events to guest.
Global error in SMMUv3 hw will be updated in smmu_gerror and
smmu_gerrorn register. Add support for global error registers to send
global error to guest.
Add architectural error handling to the emulated command queue,
updating the global error register and stopping command consumption
on error.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/smmu-v3.h | 20 +++
xen/drivers/passthrough/arm/vsmmu-v3.c | 233 +++++++++++++++++++++++--
xen/include/public/arch-arm.h | 2 +
3 files changed, 244 insertions(+), 11 deletions(-)
diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
index 4a912be740..44ecb5d89c 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.h
+++ b/xen/drivers/passthrough/arm/smmu-v3.h
@@ -358,6 +358,26 @@
#define EVTQ_0_ID GENMASK_ULL(7, 0)
+#define EVT_ID_BAD_STREAMID 0x02
+#define EVT_ID_BAD_STE 0x04
+#define EVT_ID_TRANSLATION_FAULT 0x10
+#define EVT_ID_ADDR_SIZE_FAULT 0x11
+#define EVT_ID_ACCESS_FAULT 0x12
+#define EVT_ID_PERMISSION_FAULT 0x13
+
+#define EVTQ_0_SSV (1UL << 11)
+#define EVTQ_0_SSID GENMASK_ULL(31, 12)
+#define EVTQ_0_SID GENMASK_ULL(63, 32)
+#define EVTQ_1_STAG GENMASK_ULL(15, 0)
+#define EVTQ_1_STALL (1UL << 31)
+#define EVTQ_1_PnU (1UL << 33)
+#define EVTQ_1_InD (1UL << 34)
+#define EVTQ_1_RnW (1UL << 35)
+#define EVTQ_1_S2 (1UL << 39)
+#define EVTQ_1_CLASS GENMASK_ULL(41, 40)
+#define EVTQ_1_TT_READ (1UL << 44)
+#define EVTQ_2_ADDR GENMASK_ULL(63, 0)
+#define EVTQ_3_IPA GENMASK_ULL(51, 12)
/* PRI queue */
#define PRIQ_ENT_SZ_SHIFT 4
#define PRIQ_ENT_DWORDS ((1 << PRIQ_ENT_SZ_SHIFT) >> 3)
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index 3f557d1c4d..87342fdf69 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -85,6 +85,35 @@
#define smmu_get_ste_s1ctxptr(x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \
FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x))
+/* event queue entry */
+struct arm_smmu_evtq_ent {
+ /* Common fields */
+ uint8_t opcode;
+ uint32_t sid;
+
+ /* Event-specific fields */
+ union {
+ struct {
+ uint32_t ssid;
+ bool ssv;
+ } c_bad_ste_streamid;
+
+ struct {
+ bool stall;
+ uint16_t stag;
+ uint32_t ssid;
+ bool ssv;
+ bool s2;
+ uint64_t addr;
+ bool rnw;
+ bool pnu;
+ bool ind;
+ uint8_t class;
+ uint64_t addr2;
+ } f_translation;
+ };
+};
+
/* stage-1 translation configuration */
struct arm_vsmmu_s1_trans_cfg {
paddr_t s1ctxptr;
@@ -115,6 +144,7 @@ struct virt_smmu {
uint32_t strtab_base_cfg;
uint64_t strtab_base;
uint32_t irq_ctrl;
+ uint32_t virq;
uint64_t gerror_irq_cfg0;
uint64_t evtq_irq_cfg0;
struct arm_vsmmu_queue evtq, cmdq;
@@ -160,9 +190,15 @@ static inline bool smmu_get_evtq_enabled(struct virt_smmu *smmu)
spin_unlock(&smmu->cr0_lock);
return enabled;
-};
+}
/* Queue manipulation functions */
+static bool queue_full(struct arm_vsmmu_queue *q)
+{
+ return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
+ Q_WRP(q, q->prod) != Q_WRP(q, q->cons);
+}
+
static bool queue_empty(struct arm_vsmmu_queue *q)
{
return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
@@ -175,12 +211,133 @@ static void queue_inc_cons(struct arm_vsmmu_queue *q)
q->cons = Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons);
}
+static void queue_inc_prod(struct arm_vsmmu_queue *q)
+{
+ u32 prod = (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + 1;
+ q->prod = Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod);
+}
+
static void dump_smmu_command(uint64_t *command)
{
gprintk(XENLOG_ERR, "cmd 0x%02llx: %016"PRIx64" %016"PRIx64"\n",
smmu_cmd_get_command(command[0]), command[0], command[1]);
}
+static void arm_vsmmu_update_irq(struct virt_smmu *smmu)
+{
+ bool evt_pending = false;
+ bool gerr_pending = false;
+ uint32_t irq_ctrl;
+
+ spin_lock(&smmu->irq_cfg_lock);
+ irq_ctrl = smmu->irq_ctrl;
+ spin_unlock(&smmu->irq_cfg_lock);
+
+ spin_lock(&smmu->evt_queue_lock);
+ evt_pending = !queue_empty(&smmu->evtq);
+ spin_unlock(&smmu->evt_queue_lock);
+
+ spin_lock(&smmu->gerror_lock);
+ gerr_pending = (smmu->gerror ^ smmu->gerrorn) != 0;
+ spin_unlock(&smmu->gerror_lock);
+
+ vgic_inject_irq(smmu->d, NULL, smmu->virq,
+ ((irq_ctrl & IRQ_CTRL_EVTQ_IRQEN) && evt_pending) ||
+ ((irq_ctrl & IRQ_CTRL_GERROR_IRQEN) && gerr_pending));
+}
+
+static void arm_vsmmu_set_gerror(struct virt_smmu *smmu, uint32_t gerror_err)
+{
+ uint32_t pending, new_gerrors;
+
+ spin_lock(&smmu->gerror_lock);
+
+ pending = smmu->gerror ^ smmu->gerrorn;
+ new_gerrors = ~pending & gerror_err;
+
+ if ( new_gerrors )
+ smmu->gerror ^= new_gerrors;
+
+ spin_unlock(&smmu->gerror_lock);
+
+ arm_vsmmu_update_irq(smmu);
+}
+
+static int arm_vsmmu_write_evtq(struct virt_smmu *smmu, uint64_t *evt)
+{
+ struct arm_vsmmu_queue *q = &smmu->evtq;
+ struct domain *d = smmu->d;
+ paddr_t addr;
+ int ret;
+
+ if ( !smmu_get_evtq_enabled(smmu) )
+ return -EINVAL;
+
+ spin_lock(&smmu->evt_queue_lock);
+
+ if ( queue_full(q) )
+ {
+ spin_unlock(&smmu->evt_queue_lock);
+ return -EINVAL;
+ }
+
+ addr = Q_PROD_ENT(q);
+
+ ret = access_guest_memory_by_gpa(d, addr, evt,
+ sizeof(*evt) * EVTQ_ENT_DWORDS, true);
+ if ( ret )
+ {
+ spin_unlock(&smmu->evt_queue_lock);
+ return ret;
+ }
+
+ queue_inc_prod(q);
+
+ spin_unlock(&smmu->evt_queue_lock);
+
+ arm_vsmmu_update_irq(smmu);
+
+ return 0;
+}
+
+static void arm_vsmmu_send_event(struct virt_smmu *smmu,
+ struct arm_smmu_evtq_ent *ent)
+{
+ uint64_t evt[EVTQ_ENT_DWORDS];
+ int ret;
+
+ memset(evt, 0, 1 << EVTQ_ENT_SZ_SHIFT);
+
+ if ( !smmu_get_evtq_enabled(smmu) )
+ return;
+
+ evt[0] |= FIELD_PREP(EVTQ_0_ID, ent->opcode);
+ evt[0] |= FIELD_PREP(EVTQ_0_SID, ent->sid);
+
+ switch ( ent->opcode )
+ {
+ case EVT_ID_BAD_STREAMID:
+ case EVT_ID_BAD_STE:
+ evt[0] |= FIELD_PREP(EVTQ_0_SSID, ent->c_bad_ste_streamid.ssid);
+ evt[0] |= FIELD_PREP(EVTQ_0_SSV, ent->c_bad_ste_streamid.ssv);
+ break;
+ case EVT_ID_TRANSLATION_FAULT:
+ case EVT_ID_ADDR_SIZE_FAULT:
+ case EVT_ID_ACCESS_FAULT:
+ case EVT_ID_PERMISSION_FAULT:
+ break;
+ default:
+ gdprintk(XENLOG_WARNING, "vSMMUv3: event opcode is bad\n");
+ break;
+ }
+
+ ret = arm_vsmmu_write_evtq(smmu, evt);
+ if ( ret )
+ arm_vsmmu_set_gerror(smmu, GERROR_EVTQ_ABT_ERR);
+
+ return;
+}
+
static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
uint64_t *ste)
{
@@ -189,6 +346,14 @@ static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
uint32_t log2size;
int strtab_size_shift;
int ret;
+ struct arm_smmu_evtq_ent ent = {
+ .opcode = EVT_ID_BAD_STREAMID,
+ .sid = sid,
+ .c_bad_ste_streamid = {
+ .ssid = 0,
+ .ssv = false,
+ },
+ };
spin_lock(&smmu->strtab_cfg_lock);
log2size = MIN(FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, smmu->strtab_base_cfg),
@@ -196,7 +361,10 @@ static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
spin_unlock(&smmu->strtab_cfg_lock);
if ( sid >= (1U << log2size) )
+ {
+ arm_vsmmu_send_event(smmu, &ent);
return -EINVAL;
+ }
spin_lock(&smmu->strtab_cfg_lock);
if ( FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->strtab_base_cfg) ==
@@ -249,6 +417,7 @@ static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
{
gdprintk(XENLOG_ERR, "idx=%d > max_l2_ste=%d\n",
idx, max_l2_ste);
+ arm_vsmmu_send_event(smmu, &ent);
return -EINVAL;
}
addr = l2ptr + idx * sizeof(*ste) * STRTAB_STE_DWORDS;
@@ -277,6 +446,14 @@ static int arm_vsmmu_decode_ste(struct virt_smmu *smmu, uint32_t sid,
uint64_t *ste)
{
uint64_t val = ste[0];
+ struct arm_smmu_evtq_ent ent = {
+ .opcode = EVT_ID_BAD_STE,
+ .sid = sid,
+ .c_bad_ste_streamid = {
+ .ssid = 0,
+ .ssv = false,
+ },
+ };
if ( !(val & STRTAB_STE_0_V) )
return -EAGAIN;
@@ -312,6 +489,7 @@ static int arm_vsmmu_decode_ste(struct virt_smmu *smmu, uint32_t sid,
return 0;
bad_ste:
+ arm_vsmmu_send_event(smmu, &ent);
return -EINVAL;
}
@@ -362,16 +540,19 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
if ( !smmu_get_cmdq_enabled(smmu) )
return 0;
- while ( !queue_empty(q) )
+ while ( !queue_empty(q) &&
+ ((smmu->gerror ^ smmu->gerrorn) & (GERROR_CMDQ_ERR)) == 0 )
{
- int ret;
-
addr = Q_CONS_ENT(q);
ret = access_guest_memory_by_gpa(d, addr, command,
sizeof(command), false);
- if ( ret ) {
- queue_inc_cons(q);
- return ret;
+ if ( ret )
+ {
+ smmu->cmdq.cons &= ~CMDQ_CONS_ERR;
+ smmu->cmdq.cons |= FIELD_PREP(CMDQ_CONS_ERR,
+ CMDQ_ERR_CERROR_ABT_IDX);
+ arm_vsmmu_set_gerror(smmu, GERROR_CMDQ_ERR);
+ break;
}
switch ( smmu_cmd_get_command(command[0]) )
@@ -395,6 +576,10 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
default:
gdprintk(XENLOG_ERR, "vSMMUv3: unhandled command\n");
dump_smmu_command(command);
+ smmu->cmdq.cons &= ~CMDQ_CONS_ERR;
+ smmu->cmdq.cons |= FIELD_PREP(CMDQ_CONS_ERR,
+ CMDQ_ERR_CERROR_ILL_IDX);
+ arm_vsmmu_set_gerror(smmu, GERROR_CMDQ_ERR);
break;
}
@@ -405,6 +590,15 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
ret);
dump_smmu_command(command);
}
+
+ /*
+ * Stop consuming the command queue if a command error occurred.
+ * CMDQ_CONS continues to point at the offending command until the
+ * error is acknowledged.
+ */
+ if ( (smmu->gerror ^ smmu->gerrorn) & GERROR_CMDQ_ERR )
+ break;
+
queue_inc_cons(q);
}
@@ -443,6 +637,7 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
vreg_reg32_update(®32, r, info);
smmu->evtq.cons = reg32;
spin_unlock(&smmu->evt_queue_lock);
+ arm_vsmmu_update_irq(smmu);
break;
default:
printk(XENLOG_G_ERR
@@ -573,6 +768,7 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
vreg_reg32_update(®32, r, info);
smmu->irq_ctrl = reg32;
spin_unlock(&smmu->irq_cfg_lock);
+ arm_vsmmu_update_irq(smmu);
break;
case VREG64(ARM_SMMU_GERROR_IRQ_CFG0):
@@ -602,6 +798,10 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
vreg_reg32_update(®32, r, info);
smmu->gerrorn = reg32;
spin_unlock(&smmu->gerror_lock);
+ arm_vsmmu_update_irq(smmu);
+ spin_lock(&smmu->cmd_queue_lock);
+ arm_vsmmu_handle_cmds(smmu);
+ spin_unlock(&smmu->cmd_queue_lock);
break;
default:
@@ -905,9 +1105,10 @@ static const struct mmio_handler_ops vsmmuv3_mmio_handler = {
.write = vsmmuv3_mmio_write,
};
-static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
- uint32_t features)
+static int vsmmuv3_init_single(struct domain *d, paddr_t addr,
+ paddr_t size, uint32_t virq, uint32_t features)
{
+ int ret;
struct virt_smmu *smmu;
smmu = xzalloc(struct virt_smmu);
@@ -916,6 +1117,7 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
smmu->d = d;
smmu->features = features;
+ smmu->virq = virq;
smmu->cmdq.ent_size = CMDQ_ENT_DWORDS * DWORDS_BYTES;
smmu->evtq.ent_size = EVTQ_ENT_DWORDS * DWORDS_BYTES;
@@ -928,6 +1130,15 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
spin_lock_init(&smmu->strtab_cfg_lock);
spin_lock_init(&smmu->irq_cfg_lock);
+ ret = vgic_reserve_virq(d, virq);
+ if ( !ret )
+ {
+ printk(XENLOG_G_ERR
+ "vSMMUv3: Failed to reserve domain vIRQ %u\n", virq);
+ xfree(smmu);
+ return -EINVAL;
+ }
+
register_mmio_handler(d, &vsmmuv3_mmio_handler, addr, size, smmu);
/* Register the vIOMMU to be able to clean it up later. */
@@ -953,7 +1164,7 @@ static int domain_vsmmuv3_init(struct domain *d)
{
smmu = (const struct arm_smmu_device *) hw_iommu->iommu;
ret = vsmmuv3_init_single(d, hw_iommu->addr, hw_iommu->size,
- smmu->features);
+ hw_iommu->irq, smmu->features);
if ( ret )
return ret;
}
@@ -983,7 +1194,7 @@ static int domain_vsmmuv3_init(struct domain *d)
}
ret = vsmmuv3_init_single(d, GUEST_VSMMUV3_BASE, GUEST_VSMMUV3_SIZE,
- features);
+ GUEST_VSMMU_SPI, features);
if ( ret )
return ret;
}
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 8863e2e457..c43b8b7b71 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -538,6 +538,8 @@ typedef uint64_t xen_callback_t;
#define GUEST_VIRTIO_MMIO_SPI_FIRST 33
#define GUEST_VIRTIO_MMIO_SPI_LAST 43
+#define GUEST_VSMMU_SPI 44
+
/*
* SGI is the preferred delivery mechanism of FF-A pending notifications or
* schedule recveive interrupt. SGIs 8-15 are normally not used by a guest
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 12/22] xen/arm: vsmmuv3: Add support for event queue and global error
2026-10-07 0:07 ` [PATCH v4 12/22] xen/arm: vsmmuv3: Add support for event queue and global error Milan Djokic
@ 2026-10-11 13:27 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-11 13:27 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
On 10/7/26 03:07, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> Event queue is used to send the events to guest when there is an events/
> faults. Add support for event queue to send events to guest.
>
> Global error in SMMUv3 hw will be updated in smmu_gerror and
> smmu_gerrorn register. Add support for global error registers to send
> global error to guest.
>
> Add architectural error handling to the emulated command queue,
> updating the global error register and stopping command consumption
> on error.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/drivers/passthrough/arm/smmu-v3.h | 20 +++
> xen/drivers/passthrough/arm/vsmmu-v3.c | 233 +++++++++++++++++++++++--
> xen/include/public/arch-arm.h | 2 +
> 3 files changed, 244 insertions(+), 11 deletions(-)
>
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.h b/xen/drivers/passthrough/arm/smmu-v3.h
> index 4a912be740..44ecb5d89c 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.h
> +++ b/xen/drivers/passthrough/arm/smmu-v3.h
> @@ -358,6 +358,26 @@
>
> #define EVTQ_0_ID GENMASK_ULL(7, 0)
>
> +#define EVT_ID_BAD_STREAMID 0x02
> +#define EVT_ID_BAD_STE 0x04
> +#define EVT_ID_TRANSLATION_FAULT 0x10
> +#define EVT_ID_ADDR_SIZE_FAULT 0x11
> +#define EVT_ID_ACCESS_FAULT 0x12
> +#define EVT_ID_PERMISSION_FAULT 0x13
> +
> +#define EVTQ_0_SSV (1UL << 11)
> +#define EVTQ_0_SSID GENMASK_ULL(31, 12)
> +#define EVTQ_0_SID GENMASK_ULL(63, 32)
> +#define EVTQ_1_STAG GENMASK_ULL(15, 0)
> +#define EVTQ_1_STALL (1UL << 31)
> +#define EVTQ_1_PnU (1UL << 33)
> +#define EVTQ_1_InD (1UL << 34)
> +#define EVTQ_1_RnW (1UL << 35)
> +#define EVTQ_1_S2 (1UL << 39)
> +#define EVTQ_1_CLASS GENMASK_ULL(41, 40)
> +#define EVTQ_1_TT_READ (1UL << 44)
> +#define EVTQ_2_ADDR GENMASK_ULL(63, 0)
> +#define EVTQ_3_IPA GENMASK_ULL(51, 12)
> /* PRI queue */
> #define PRIQ_ENT_SZ_SHIFT 4
> #define PRIQ_ENT_DWORDS ((1 << PRIQ_ENT_SZ_SHIFT) >> 3)
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
> index 3f557d1c4d..87342fdf69 100644
> --- a/xen/drivers/passthrough/arm/vsmmu-v3.c
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
> @@ -85,6 +85,35 @@
> #define smmu_get_ste_s1ctxptr(x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \
> FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x))
>
> +/* event queue entry */
> +struct arm_smmu_evtq_ent {
> + /* Common fields */
> + uint8_t opcode;
> + uint32_t sid;
> +
> + /* Event-specific fields */
> + union {
> + struct {
> + uint32_t ssid;
> + bool ssv;
> + } c_bad_ste_streamid;
> +
> + struct {
> + bool stall;
> + uint16_t stag;
> + uint32_t ssid;
> + bool ssv;
> + bool s2;
> + uint64_t addr;
> + bool rnw;
> + bool pnu;
> + bool ind;
> + uint8_t class;
> + uint64_t addr2;
> + } f_translation;
> + };
> +};
> +
> /* stage-1 translation configuration */
> struct arm_vsmmu_s1_trans_cfg {
> paddr_t s1ctxptr;
> @@ -115,6 +144,7 @@ struct virt_smmu {
> uint32_t strtab_base_cfg;
> uint64_t strtab_base;
> uint32_t irq_ctrl;
> + uint32_t virq;
> uint64_t gerror_irq_cfg0;
> uint64_t evtq_irq_cfg0;
> struct arm_vsmmu_queue evtq, cmdq;
> @@ -160,9 +190,15 @@ static inline bool smmu_get_evtq_enabled(struct virt_smmu *smmu)
> spin_unlock(&smmu->cr0_lock);
>
> return enabled;
> -};
> +}
>
> /* Queue manipulation functions */
> +static bool queue_full(struct arm_vsmmu_queue *q)
> +{
> + return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
> + Q_WRP(q, q->prod) != Q_WRP(q, q->cons);
> +}
> +
> static bool queue_empty(struct arm_vsmmu_queue *q)
> {
> return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
> @@ -175,12 +211,133 @@ static void queue_inc_cons(struct arm_vsmmu_queue *q)
> q->cons = Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons);
> }
>
> +static void queue_inc_prod(struct arm_vsmmu_queue *q)
> +{
> + u32 prod = (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + 1;
> + q->prod = Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod);
> +}
> +
> static void dump_smmu_command(uint64_t *command)
> {
> gprintk(XENLOG_ERR, "cmd 0x%02llx: %016"PRIx64" %016"PRIx64"\n",
> smmu_cmd_get_command(command[0]), command[0], command[1]);
> }
>
> +static void arm_vsmmu_update_irq(struct virt_smmu *smmu)
> +{
> + bool evt_pending = false;
> + bool gerr_pending = false;
> + uint32_t irq_ctrl;
> +
> + spin_lock(&smmu->irq_cfg_lock);
> + irq_ctrl = smmu->irq_ctrl;
> + spin_unlock(&smmu->irq_cfg_lock);
> +
> + spin_lock(&smmu->evt_queue_lock);
> + evt_pending = !queue_empty(&smmu->evtq);
> + spin_unlock(&smmu->evt_queue_lock);
> +
> + spin_lock(&smmu->gerror_lock);
> + gerr_pending = (smmu->gerror ^ smmu->gerrorn) != 0;
> + spin_unlock(&smmu->gerror_lock);
> +
> + vgic_inject_irq(smmu->d, NULL, smmu->virq,
> + ((irq_ctrl & IRQ_CTRL_EVTQ_IRQEN) && evt_pending) ||
> + ((irq_ctrl & IRQ_CTRL_GERROR_IRQEN) && gerr_pending));
> +}
> +
> +static void arm_vsmmu_set_gerror(struct virt_smmu *smmu, uint32_t gerror_err)
> +{
> + uint32_t pending, new_gerrors;
> +
> + spin_lock(&smmu->gerror_lock);
> +
> + pending = smmu->gerror ^ smmu->gerrorn;
> + new_gerrors = ~pending & gerror_err;
> +
> + if ( new_gerrors )
> + smmu->gerror ^= new_gerrors;
> +
> + spin_unlock(&smmu->gerror_lock);
> +
> + arm_vsmmu_update_irq(smmu);
> +}
> +
> +static int arm_vsmmu_write_evtq(struct virt_smmu *smmu, uint64_t *evt)
> +{
> + struct arm_vsmmu_queue *q = &smmu->evtq;
> + struct domain *d = smmu->d;
> + paddr_t addr;
> + int ret;
> +
> + if ( !smmu_get_evtq_enabled(smmu) )
> + return -EINVAL;
If the queue is disabled, the event is simply discarded and nothing is
wrong. Why do we then raise GERROR.EVTQ_ABT_ERR via arm_vsmmu_set_gerror()?
> +
> + spin_lock(&smmu->evt_queue_lock);
> +
> + if ( queue_full(q) )
> + {
> + spin_unlock(&smmu->evt_queue_lock);
> + return -EINVAL;
This does not look like a global error either. From what I got from the
SMMU spec (ARM IHI 0070/H.a 7.4 Event queue overflow), when the queue is
full the event should be discarded and the overflow flag should be set
in EVTQ_PROD.
"The Event queue overflow condition is present when:
SMMU_EVENTQ_PROD.OVFLG != SMMU_EVENTQ_CONS.OVACKFLG
When an overflow occurs, the SMMU toggles the SMMU_EVENTQ_PROD.OVFLG
flag if the overflow condition is not already present."
Otherwise the guest might show a misleading message (an event queue
memory-write error instead of an overflow).
Or I really missed something?
> + }
> +
> + addr = Q_PROD_ENT(q);
> +
> + ret = access_guest_memory_by_gpa(d, addr, evt,
> + sizeof(*evt) * EVTQ_ENT_DWORDS, true);
> + if ( ret )
> + {
> + spin_unlock(&smmu->evt_queue_lock);
> + return ret;
> + }
> +
> + queue_inc_prod(q);
> +
> + spin_unlock(&smmu->evt_queue_lock);
> +
> + arm_vsmmu_update_irq(smmu);
> +
> + return 0;
> +}
> +
> +static void arm_vsmmu_send_event(struct virt_smmu *smmu,
> + struct arm_smmu_evtq_ent *ent)
> +{
> + uint64_t evt[EVTQ_ENT_DWORDS];
> + int ret;
> +
> + memset(evt, 0, 1 << EVTQ_ENT_SZ_SHIFT);
> +
> + if ( !smmu_get_evtq_enabled(smmu) )
> + return;
> +
> + evt[0] |= FIELD_PREP(EVTQ_0_ID, ent->opcode);
> + evt[0] |= FIELD_PREP(EVTQ_0_SID, ent->sid);
> +
> + switch ( ent->opcode )
> + {
> + case EVT_ID_BAD_STREAMID:
> + case EVT_ID_BAD_STE:
> + evt[0] |= FIELD_PREP(EVTQ_0_SSID, ent->c_bad_ste_streamid.ssid);
> + evt[0] |= FIELD_PREP(EVTQ_0_SSV, ent->c_bad_ste_streamid.ssv);
> + break;
> + case EVT_ID_TRANSLATION_FAULT:
> + case EVT_ID_ADDR_SIZE_FAULT:
> + case EVT_ID_ACCESS_FAULT:
> + case EVT_ID_PERMISSION_FAULT:
> + break;
> + default:
> + gdprintk(XENLOG_WARNING, "vSMMUv3: event opcode is bad\n");
> + break;
> + }
> +
> + ret = arm_vsmmu_write_evtq(smmu, evt);
> + if ( ret )
> + arm_vsmmu_set_gerror(smmu, GERROR_EVTQ_ABT_ERR);
> +
> + return;
> +}
> +
> static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
> uint64_t *ste)
> {
> @@ -189,6 +346,14 @@ static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
> uint32_t log2size;
> int strtab_size_shift;
> int ret;
> + struct arm_smmu_evtq_ent ent = {
> + .opcode = EVT_ID_BAD_STREAMID,
> + .sid = sid,
> + .c_bad_ste_streamid = {
> + .ssid = 0,
> + .ssv = false,
> + },
> + };
>
> spin_lock(&smmu->strtab_cfg_lock);
> log2size = MIN(FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, smmu->strtab_base_cfg),
> @@ -196,7 +361,10 @@ static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
> spin_unlock(&smmu->strtab_cfg_lock);
>
> if ( sid >= (1U << log2size) )
> + {
> + arm_vsmmu_send_event(smmu, &ent);
> return -EINVAL;
> + }
>
> spin_lock(&smmu->strtab_cfg_lock);
> if ( FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->strtab_base_cfg) ==
> @@ -249,6 +417,7 @@ static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
> {
> gdprintk(XENLOG_ERR, "idx=%d > max_l2_ste=%d\n",
> idx, max_l2_ste);
> + arm_vsmmu_send_event(smmu, &ent);
> return -EINVAL;
> }
> addr = l2ptr + idx * sizeof(*ste) * STRTAB_STE_DWORDS;
> @@ -277,6 +446,14 @@ static int arm_vsmmu_decode_ste(struct virt_smmu *smmu, uint32_t sid,
> uint64_t *ste)
> {
> uint64_t val = ste[0];
> + struct arm_smmu_evtq_ent ent = {
> + .opcode = EVT_ID_BAD_STE,
> + .sid = sid,
> + .c_bad_ste_streamid = {
> + .ssid = 0,
> + .ssv = false,
> + },
> + };
>
> if ( !(val & STRTAB_STE_0_V) )
> return -EAGAIN;
> @@ -312,6 +489,7 @@ static int arm_vsmmu_decode_ste(struct virt_smmu *smmu, uint32_t sid,
> return 0;
>
> bad_ste:
> + arm_vsmmu_send_event(smmu, &ent);
> return -EINVAL;
> }
>
> @@ -362,16 +540,19 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
> if ( !smmu_get_cmdq_enabled(smmu) )
> return 0;
>
> - while ( !queue_empty(q) )
> + while ( !queue_empty(q) &&
> + ((smmu->gerror ^ smmu->gerrorn) & (GERROR_CMDQ_ERR)) == 0 )
> {
> - int ret;
> -
> addr = Q_CONS_ENT(q);
> ret = access_guest_memory_by_gpa(d, addr, command,
> sizeof(command), false);
> - if ( ret ) {
> - queue_inc_cons(q);
> - return ret;
> + if ( ret )
> + {
> + smmu->cmdq.cons &= ~CMDQ_CONS_ERR;
> + smmu->cmdq.cons |= FIELD_PREP(CMDQ_CONS_ERR,
> + CMDQ_ERR_CERROR_ABT_IDX);
> + arm_vsmmu_set_gerror(smmu, GERROR_CMDQ_ERR);
> + break;
> }
>
> switch ( smmu_cmd_get_command(command[0]) )
> @@ -395,6 +576,10 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
> default:
> gdprintk(XENLOG_ERR, "vSMMUv3: unhandled command\n");
> dump_smmu_command(command);
> + smmu->cmdq.cons &= ~CMDQ_CONS_ERR;
> + smmu->cmdq.cons |= FIELD_PREP(CMDQ_CONS_ERR,
> + CMDQ_ERR_CERROR_ILL_IDX);
> + arm_vsmmu_set_gerror(smmu, GERROR_CMDQ_ERR);
> break;
> }
>
> @@ -405,6 +590,15 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)
> ret);
> dump_smmu_command(command);
> }
> +
> + /*
> + * Stop consuming the command queue if a command error occurred.
> + * CMDQ_CONS continues to point at the offending command until the
> + * error is acknowledged.
> + */
> + if ( (smmu->gerror ^ smmu->gerrorn) & GERROR_CMDQ_ERR )
> + break;
> +
> queue_inc_cons(q);
> }
>
> @@ -443,6 +637,7 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
> vreg_reg32_update(®32, r, info);
> smmu->evtq.cons = reg32;
> spin_unlock(&smmu->evt_queue_lock);
> + arm_vsmmu_update_irq(smmu);
> break;
> default:
> printk(XENLOG_G_ERR
> @@ -573,6 +768,7 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
> vreg_reg32_update(®32, r, info);
> smmu->irq_ctrl = reg32;
> spin_unlock(&smmu->irq_cfg_lock);
> + arm_vsmmu_update_irq(smmu);
> break;
>
> case VREG64(ARM_SMMU_GERROR_IRQ_CFG0):
> @@ -602,6 +798,10 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,
> vreg_reg32_update(®32, r, info);
> smmu->gerrorn = reg32;
> spin_unlock(&smmu->gerror_lock);
> + arm_vsmmu_update_irq(smmu);
> + spin_lock(&smmu->cmd_queue_lock);
> + arm_vsmmu_handle_cmds(smmu);
> + spin_unlock(&smmu->cmd_queue_lock);
> break;
>
> default:
> @@ -905,9 +1105,10 @@ static const struct mmio_handler_ops vsmmuv3_mmio_handler = {
> .write = vsmmuv3_mmio_write,
> };
>
> -static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
> - uint32_t features)
> +static int vsmmuv3_init_single(struct domain *d, paddr_t addr,
> + paddr_t size, uint32_t virq, uint32_t features)
> {
> + int ret;
> struct virt_smmu *smmu;
>
> smmu = xzalloc(struct virt_smmu);
> @@ -916,6 +1117,7 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
>
> smmu->d = d;
> smmu->features = features;
> + smmu->virq = virq;
> smmu->cmdq.ent_size = CMDQ_ENT_DWORDS * DWORDS_BYTES;
> smmu->evtq.ent_size = EVTQ_ENT_DWORDS * DWORDS_BYTES;
>
> @@ -928,6 +1130,15 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size,
> spin_lock_init(&smmu->strtab_cfg_lock);
> spin_lock_init(&smmu->irq_cfg_lock);
>
> + ret = vgic_reserve_virq(d, virq);
> + if ( !ret )
> + {
> + printk(XENLOG_G_ERR
> + "vSMMUv3: Failed to reserve domain vIRQ %u\n", virq);
> + xfree(smmu);
> + return -EINVAL;
> + }
> +
> register_mmio_handler(d, &vsmmuv3_mmio_handler, addr, size, smmu);
>
> /* Register the vIOMMU to be able to clean it up later. */
> @@ -953,7 +1164,7 @@ static int domain_vsmmuv3_init(struct domain *d)
> {
> smmu = (const struct arm_smmu_device *) hw_iommu->iommu;
> ret = vsmmuv3_init_single(d, hw_iommu->addr, hw_iommu->size,
> - smmu->features);
> + hw_iommu->irq, smmu->features);
> if ( ret )
> return ret;
> }
> @@ -983,7 +1194,7 @@ static int domain_vsmmuv3_init(struct domain *d)
> }
>
> ret = vsmmuv3_init_single(d, GUEST_VSMMUV3_BASE, GUEST_VSMMUV3_SIZE,
> - features);
> + GUEST_VSMMU_SPI, features);
> if ( ret )
> return ret;
> }
> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
> index 8863e2e457..c43b8b7b71 100644
> --- a/xen/include/public/arch-arm.h
> +++ b/xen/include/public/arch-arm.h
> @@ -538,6 +538,8 @@ typedef uint64_t xen_callback_t;
> #define GUEST_VIRTIO_MMIO_SPI_FIRST 33
> #define GUEST_VIRTIO_MMIO_SPI_LAST 43
>
> +#define GUEST_VSMMU_SPI 44
So this is the same emulated SPI as the one used for vPL011. You likely
need to update d_cfg->arch.nr_spis if vSMMU is enabled for the domain,
to ensure that we allocate enough SPIs for it (refer to
arch_parse_dom0less_node(), when "nr_spis" is not explicitly specified
we update nr_spis if vPL011 is requested). Moreover, you already do this
in libxl.
> +
> /*
> * SGI is the preferred delivery mechanism of FF-A pending notifications or
> * schedule recveive interrupt. SGIs 8-15 are normally not used by a guest
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 13/22] xen/arm: vsmmuv3: IOMMU device tree node for dom0
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (11 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 12/22] xen/arm: vsmmuv3: Add support for event queue and global error Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-10 15:30 ` Oleksandr Tyshchenko
2026-10-07 0:07 ` [PATCH v4 14/22] xen/arm: vsmmuv3: Emulated SMMUv3 device tree node for dom0less Milan Djokic
` (8 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
XEN will create an IOMMU device tree node in the device tree
to enable the dom0 to discover the virtual SMMUv3 during dom0 boot.
IOMMU device tree node will only be created when viommu is enabled
via Kconfig build options.
"iommus" property will be added for dom0 devices to virtual
IOMMU node to enable the dom0 linux kernel to configure the IOMMU.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/arch/arm/domain_build.c | 102 +++++++++++++++++++++++++++++-
xen/arch/arm/include/asm/viommu.h | 1 +
2 files changed, 101 insertions(+), 2 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index cd2e09f005..6513140133 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -557,9 +557,12 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
continue;
}
- if ( iommu_node )
+ /*
+ * Expose IOMMU specific properties to hwdom when vIOMMU is
+ * enabled.
+ */
+ if ( iommu_node && viommu_get_type() == XEN_DOMCTL_CONFIG_VIOMMU_NONE )
{
- /* Don't expose IOMMU specific properties to hwdom */
if ( dt_property_name_is_equal(prop, "iommus") )
continue;
@@ -1457,6 +1460,95 @@ int __init make_timer_node(const struct kernel_info *kinfo)
return res;
}
+#ifdef CONFIG_ARM_VIRTUAL_IOMMU
+static int __init make_hwdom_viommu_node(const struct kernel_info *kinfo)
+{
+ uint32_t len;
+ int res;
+ char buf[24];
+ void *fdt = kinfo->fdt;
+ const void *prop = NULL;
+ const struct dt_device_node *iommu = NULL;
+ struct host_iommu *iommu_data;
+ gic_interrupt_t intr;
+
+ if ( list_empty(&host_iommu_list) )
+ return 0;
+
+ list_for_each_entry( iommu_data, &host_iommu_list, entry )
+ {
+ if ( iommu_data->hwdom_node_created )
+ return 0;
+
+ iommu = iommu_data->dt_node;
+
+ snprintf(buf, sizeof(buf), "iommu@%"PRIx64, iommu_data->addr);
+
+ res = fdt_begin_node(fdt, buf);
+ if ( res )
+ return res;
+
+ prop = dt_get_property(iommu, "compatible", &len);
+ if ( !prop )
+ {
+ res = -FDT_ERR_XEN(ENOENT);
+ return res;
+ }
+
+ res = fdt_property(fdt, "compatible", prop, len);
+ if ( res )
+ return res;
+
+ if ( iommu->phandle )
+ {
+ res = fdt_property_cell(fdt, "phandle", iommu->phandle);
+ if ( res )
+ return res;
+ }
+
+ /* Use the same reg regions as the IOMMU node in host DTB. */
+ prop = dt_get_property(iommu, "reg", &len);
+ if ( !prop )
+ {
+ printk(XENLOG_ERR "vIOMMU: Can't find IOMMU reg property.\n");
+ res = -FDT_ERR_XEN(ENOENT);
+ return res;
+ }
+
+ res = fdt_property(fdt, "reg", prop, len);
+ if ( res )
+ return res;
+
+ prop = dt_get_property(iommu, "#iommu-cells", &len);
+ if ( !prop )
+ {
+ res = -FDT_ERR_XEN(ENOENT);
+ return res;
+ }
+
+ res = fdt_property(fdt, "#iommu-cells", prop, len);
+ if ( res )
+ return res;
+
+ res = fdt_property_string(fdt, "interrupt-names", "combined");
+ if ( res )
+ return res;
+
+ set_interrupt(intr, iommu_data->irq, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
+
+ res = fdt_property_interrupts(kinfo, &intr, 1);
+ if ( res )
+ return res;
+
+ iommu_data->hwdom_node_created = true;
+
+ fdt_end_node(fdt);
+ }
+
+ return res;
+}
+#endif
+
static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
struct dt_device_node *node,
p2m_type_t p2mt)
@@ -1525,6 +1617,12 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
if ( dt_match_node(timer_matches, node) )
return make_timer_node(kinfo);
+#ifdef CONFIG_ARM_VIRTUAL_IOMMU
+ if ( device_get_class(node) == DEVICE_IOMMU &&
+ viommu_get_type() != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
+ return make_hwdom_viommu_node(kinfo);
+#endif
+
/* Skip nodes used by Xen */
if ( dt_device_used_by(node) == DOMID_XEN )
{
diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
index 844cf34d56..5ae21042f0 100644
--- a/xen/arch/arm/include/asm/viommu.h
+++ b/xen/arch/arm/include/asm/viommu.h
@@ -20,6 +20,7 @@ struct host_iommu {
paddr_t size;
/* Used for dom0 EVTQ and GERROR interrupts (1-1 IRQ mapping) */
uint32_t irq;
+ bool hwdom_node_created;
};
struct viommu_ops {
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 13/22] xen/arm: vsmmuv3: IOMMU device tree node for dom0
2026-10-07 0:07 ` [PATCH v4 13/22] xen/arm: vsmmuv3: IOMMU device tree node for dom0 Milan Djokic
@ 2026-10-10 15:30 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-10 15:30 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk
On 10/7/26 03:07, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> XEN will create an IOMMU device tree node in the device tree
> to enable the dom0 to discover the virtual SMMUv3 during dom0 boot.
> IOMMU device tree node will only be created when viommu is enabled
> via Kconfig build options.
>
> "iommus" property will be added for dom0 devices to virtual
> IOMMU node to enable the dom0 linux kernel to configure the IOMMU.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/arch/arm/domain_build.c | 102 +++++++++++++++++++++++++++++-
> xen/arch/arm/include/asm/viommu.h | 1 +
> 2 files changed, 101 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index cd2e09f005..6513140133 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -557,9 +557,12 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,
> continue;
> }
>
> - if ( iommu_node )
> + /*
> + * Expose IOMMU specific properties to hwdom when vIOMMU is
> + * enabled.
> + */
> + if ( iommu_node && viommu_get_type() == XEN_DOMCTL_CONFIG_VIOMMU_NONE )
> {
> - /* Don't expose IOMMU specific properties to hwdom */
> if ( dt_property_name_is_equal(prop, "iommus") )
> continue;
>
> @@ -1457,6 +1460,95 @@ int __init make_timer_node(const struct kernel_info *kinfo)
> return res;
> }
>
> +#ifdef CONFIG_ARM_VIRTUAL_IOMMU
> +static int __init make_hwdom_viommu_node(const struct kernel_info *kinfo)
> +{
> + uint32_t len;
> + int res;
> + char buf[24];
> + void *fdt = kinfo->fdt;
> + const void *prop = NULL;
> + const struct dt_device_node *iommu = NULL;
> + struct host_iommu *iommu_data;
> + gic_interrupt_t intr;
> +
> + if ( list_empty(&host_iommu_list) )
> + return 0;
> +
> + list_for_each_entry( iommu_data, &host_iommu_list, entry )
> + {
> + if ( iommu_data->hwdom_node_created )
> + return 0;
> +
> + iommu = iommu_data->dt_node;
> +
> + snprintf(buf, sizeof(buf), "iommu@%"PRIx64, iommu_data->addr);
> +
> + res = fdt_begin_node(fdt, buf);
> + if ( res )
> + return res;
> +
> + prop = dt_get_property(iommu, "compatible", &len);
> + if ( !prop )
> + {
> + res = -FDT_ERR_XEN(ENOENT);
> + return res;
> + }
> +
> + res = fdt_property(fdt, "compatible", prop, len);
> + if ( res )
> + return res;
> +
> + if ( iommu->phandle )
> + {
> + res = fdt_property_cell(fdt, "phandle", iommu->phandle);
> + if ( res )
> + return res;
> + }
> +
> + /* Use the same reg regions as the IOMMU node in host DTB. */
> + prop = dt_get_property(iommu, "reg", &len);
> + if ( !prop )
> + {
> + printk(XENLOG_ERR "vIOMMU: Can't find IOMMU reg property.\n");
> + res = -FDT_ERR_XEN(ENOENT);
> + return res;
> + }
> +
> + res = fdt_property(fdt, "reg", prop, len);
> + if ( res )
> + return res;
> +
> + prop = dt_get_property(iommu, "#iommu-cells", &len);
> + if ( !prop )
> + {
> + res = -FDT_ERR_XEN(ENOENT);
> + return res;
> + }
> +
> + res = fdt_property(fdt, "#iommu-cells", prop, len);
> + if ( res )
> + return res;
> +
> + res = fdt_property_string(fdt, "interrupt-names", "combined");
> + if ( res )
> + return res;
> +
> + set_interrupt(intr, iommu_data->irq, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
> +
> + res = fdt_property_interrupts(kinfo, &intr, 1);
> + if ( res )
> + return res;
> +
> + iommu_data->hwdom_node_created = true;
> +
> + fdt_end_node(fdt);
> + }
> +
> + return res;
> +}
> +#endif
> +
> static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
> struct dt_device_node *node,
> p2m_type_t p2mt)
> @@ -1525,6 +1617,12 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
> if ( dt_match_node(timer_matches, node) )
> return make_timer_node(kinfo);
>
> +#ifdef CONFIG_ARM_VIRTUAL_IOMMU
> + if ( device_get_class(node) == DEVICE_IOMMU &&
> + viommu_get_type() != XEN_DOMCTL_CONFIG_VIOMMU_NONE )
> + return make_hwdom_viommu_node(kinfo);
So all vIOMMU nodes are created at the position of the first pIOMMU
node, right? If so, could it be possible in real life for IOMMUs to sit
under different buses (under different parent nodes)?
> +#endif
> +
> /* Skip nodes used by Xen */
> if ( dt_device_used_by(node) == DOMID_XEN )
> {
> diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
> index 844cf34d56..5ae21042f0 100644
> --- a/xen/arch/arm/include/asm/viommu.h
> +++ b/xen/arch/arm/include/asm/viommu.h
> @@ -20,6 +20,7 @@ struct host_iommu {
> paddr_t size;
> /* Used for dom0 EVTQ and GERROR interrupts (1-1 IRQ mapping) */
> uint32_t irq;
> + bool hwdom_node_created;
> };
>
> struct viommu_ops {
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 14/22] xen/arm: vsmmuv3: Emulated SMMUv3 device tree node for dom0less
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (12 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 13/22] xen/arm: vsmmuv3: IOMMU device tree node for dom0 Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-07 0:07 ` [PATCH v4 15/22] arm/libxl: vsmmuv3: Emulated SMMUv3 device tree node in libxl Milan Djokic
` (7 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Andrew Cooper, Anthony PERARD,
Jan Beulich, Roger Pau Monné, Milan Djokic, Luca Fancellu
From: Rahul Singh <rahul.singh@arm.com>
XEN will create an Emulated SMMUv3 device tree node in the device tree
to enable the dom0less domains to discover the virtual SMMUv3 during
boot. Emulated SMMUv3 device tree node will only be created when cmdline
option vsmmuv3 is enabled.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
---
xen/arch/arm/dom0less-build.c | 53 +++++++++++++++++++++++++++
xen/include/public/device_tree_defs.h | 1 +
2 files changed, 54 insertions(+)
diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
index f87437915e..fce9cdbef5 100644
--- a/xen/arch/arm/dom0less-build.c
+++ b/xen/arch/arm/dom0less-build.c
@@ -218,10 +218,63 @@ static int __init make_vpl011_uart_node(struct kernel_info *kinfo)
}
#endif
+#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3
+static int __init make_vsmmuv3_node(const struct kernel_info *kinfo)
+{
+ int res;
+ char buf[24];
+ __be32 reg[GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS];
+ __be32 *cells;
+ void *fdt = kinfo->fdt;
+
+ snprintf(buf, sizeof(buf), "iommu@%llx", GUEST_VSMMUV3_BASE);
+
+ res = fdt_begin_node(fdt, buf);
+ if ( res )
+ return res;
+
+ res = fdt_property_string(fdt, "compatible", "arm,smmu-v3");
+ if ( res )
+ return res;
+
+ /* Create reg property */
+ cells = ®[0];
+ dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
+ GUEST_VSMMUV3_BASE, GUEST_VSMMUV3_SIZE);
+ res = fdt_property(fdt, "reg", reg,
+ (GUEST_ROOT_ADDRESS_CELLS +
+ GUEST_ROOT_SIZE_CELLS) * sizeof(*reg));
+ if ( res )
+ return res;
+
+ res = fdt_property_cell(fdt, "phandle", GUEST_PHANDLE_VSMMUV3);
+ if ( res )
+ return res;
+
+ res = fdt_property_cell(fdt, "#iommu-cells", 1);
+ if ( res )
+ return res;
+
+ res = fdt_end_node(fdt);
+
+ return res;
+}
+#endif
+
int __init make_arch_nodes(struct kernel_info *kinfo)
{
int ret;
+
+#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3
+ if ( viommu_get_type() == XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 )
+ {
+ ret = make_vsmmuv3_node(kinfo);
+ if ( ret )
+ return -EINVAL;
+ }
+#endif
+
ret = make_psci_node(kinfo->fdt);
if ( ret )
return -EINVAL;
diff --git a/xen/include/public/device_tree_defs.h b/xen/include/public/device_tree_defs.h
index 9e80d0499d..7846a0425c 100644
--- a/xen/include/public/device_tree_defs.h
+++ b/xen/include/public/device_tree_defs.h
@@ -14,6 +14,7 @@
*/
#define GUEST_PHANDLE_GIC (65000)
#define GUEST_PHANDLE_IOMMU (GUEST_PHANDLE_GIC + 1)
+#define GUEST_PHANDLE_VSMMUV3 (GUEST_PHANDLE_IOMMU + 1)
#define GUEST_ROOT_ADDRESS_CELLS 2
#define GUEST_ROOT_SIZE_CELLS 2
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 15/22] arm/libxl: vsmmuv3: Emulated SMMUv3 device tree node in libxl
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (13 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 14/22] xen/arm: vsmmuv3: Emulated SMMUv3 device tree node for dom0less Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-07 0:07 ` [PATCH v4 16/22] xen/arm: vsmmuv3: Insert vIRQ properties into vIOMMU device tree node Milan Djokic
` (6 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Anthony PERARD, Juergen Gross, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
libxl will create an Emulated SMMUv3 device tree node in the device
tree to enable the guest OS to discover the virtual SMMUv3 during guest
boot.
Emulated SMMUv3 device tree node will only be created when
"viommu=smmuv3" is set in xl domain configuration.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
tools/libs/light/libxl_arm.c | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index a248793588..1207b620fa 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -901,6 +901,36 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
return 0;
}
+static int make_vsmmuv3_node(libxl__gc *gc, void *fdt,
+ const struct arch_info *ainfo,
+ struct xc_dom_image *dom)
+{
+ int res;
+ const char *name = GCSPRINTF("iommu@%llx", GUEST_VSMMUV3_BASE);
+
+ res = fdt_begin_node(fdt, name);
+ if (res) return res;
+
+ res = fdt_property_compat(gc, fdt, 1, "arm,smmu-v3");
+ if (res) return res;
+
+ res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+ GUEST_ROOT_SIZE_CELLS, 1, GUEST_VSMMUV3_BASE,
+ GUEST_VSMMUV3_SIZE);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "phandle", GUEST_PHANDLE_VSMMUV3);
+ if (res) return res;
+
+ res = fdt_property_cell(fdt, "#iommu-cells", 1);
+ if (res) return res;
+
+ res = fdt_end_node(fdt);
+ if (res) return res;
+
+ return 0;
+}
+
static int make_vpci_node(libxl__gc *gc, void *fdt,
const struct arch_info *ainfo,
struct xc_dom_image *dom)
@@ -1408,6 +1438,9 @@ next_resize:
if (d_config->num_pcidevs)
FDT( make_vpci_node(gc, fdt, ainfo, dom) );
+ if (info->arch_arm.viommu_type == LIBXL_VIOMMU_TYPE_SMMUV3)
+ FDT( make_vsmmuv3_node(gc, fdt, ainfo, dom) );
+
for (i = 0; i < d_config->num_disks; i++) {
libxl_device_disk *disk = &d_config->disks[i];
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 16/22] xen/arm: vsmmuv3: Insert vIRQ properties into vIOMMU device tree node
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (14 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 15/22] arm/libxl: vsmmuv3: Emulated SMMUv3 device tree node in libxl Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-07 0:07 ` [PATCH v4 17/22] xen/arm: vsmmuv3: Add support to send stage-1 event to guest Milan Djokic
` (5 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Anthony PERARD, Juergen Gross, Stefano Stabellini,
Julien Grall, Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Insert vIRQ properties for event queue and GERROR handling into the
emulated IOMMU device tree node for dom0less and libxl guests.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
tools/libs/light/libxl_arm.c | 30 +++++++++++++++++++++++++++---
xen/arch/arm/dom0less-build.c | 17 +++++++++++++++++
2 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index 1207b620fa..410dc98d09 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -86,8 +86,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
{
uint32_t nr_spis = 0, cfg_nr_spis = d_config->b_info.arch_arm.nr_spis;
unsigned int i;
- uint32_t vuart_irq, virtio_irq = 0;
- bool vuart_enabled = false, virtio_enabled = false;
+ uint32_t vuart_irq, virtio_irq = 0, vsmmu_irq = 0;
+ bool vuart_enabled = false, virtio_enabled = false, vsmmu_enabled = false;
uint64_t virtio_mmio_base = GUEST_VIRTIO_MMIO_BASE;
uint32_t virtio_mmio_irq = GUEST_VIRTIO_MMIO_SPI_FIRST;
int rc;
@@ -97,11 +97,21 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
* of SPI VIRQ for pl011.
*/
if (d_config->b_info.arch_arm.vuart == LIBXL_VUART_TYPE_SBSA_UART) {
- nr_spis += (GUEST_VPL011_SPI - 32) + 1;
vuart_irq = GUEST_VPL011_SPI;
+ nr_spis = max(nr_spis, vuart_irq - 32 + 1);
vuart_enabled = true;
}
+ /*
+ * If smmuv3 viommu is enabled then increment the nr_spis to allow
+ * allocation of SPI VIRQ for VSMMU.
+ */
+ if (d_config->b_info.arch_arm.viommu_type == LIBXL_VIOMMU_TYPE_SMMUV3) {
+ vsmmu_irq = GUEST_VSMMU_SPI;
+ nr_spis = max(nr_spis, vsmmu_irq - 32 + 1);
+ vsmmu_enabled = true;
+ }
+
for (i = 0; i < d_config->num_disks; i++) {
libxl_device_disk *disk = &d_config->disks[i];
@@ -170,6 +180,11 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
return ERROR_FAIL;
}
+ if (vsmmu_enabled && irq == vsmmu_irq) {
+ LOG(ERROR, "Physical IRQ %u conflicting with vSMMUv3 SPI\n", irq);
+ return ERROR_FAIL;
+ }
+
if (irq < 32)
continue;
@@ -907,6 +922,7 @@ static int make_vsmmuv3_node(libxl__gc *gc, void *fdt,
{
int res;
const char *name = GCSPRINTF("iommu@%llx", GUEST_VSMMUV3_BASE);
+ gic_interrupt intr;
res = fdt_begin_node(fdt, name);
if (res) return res;
@@ -925,6 +941,14 @@ static int make_vsmmuv3_node(libxl__gc *gc, void *fdt,
res = fdt_property_cell(fdt, "#iommu-cells", 1);
if (res) return res;
+ res = fdt_property_string(fdt, "interrupt-names", "combined");
+ if (res) return res;
+
+ set_interrupt(intr, GUEST_VSMMU_SPI, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
+
+ res = fdt_property_interrupts(gc, fdt, &intr, 1);
+ if (res) return res;
+
res = fdt_end_node(fdt);
if (res) return res;
diff --git a/xen/arch/arm/dom0less-build.c b/xen/arch/arm/dom0less-build.c
index fce9cdbef5..dce2ec7f49 100644
--- a/xen/arch/arm/dom0less-build.c
+++ b/xen/arch/arm/dom0less-build.c
@@ -225,6 +225,7 @@ static int __init make_vsmmuv3_node(const struct kernel_info *kinfo)
char buf[24];
__be32 reg[GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS];
__be32 *cells;
+ gic_interrupt_t intr;
void *fdt = kinfo->fdt;
snprintf(buf, sizeof(buf), "iommu@%llx", GUEST_VSMMUV3_BASE);
@@ -255,6 +256,22 @@ static int __init make_vsmmuv3_node(const struct kernel_info *kinfo)
if ( res )
return res;
+ res = fdt_property_string(fdt, "interrupt-names", "combined");
+ if ( res )
+ return res;
+
+ set_interrupt(intr, GUEST_VSMMU_SPI, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
+
+ res = fdt_property(kinfo->fdt, "interrupts",
+ intr, sizeof(intr));
+ if ( res )
+ return res;
+
+ res = fdt_property_cell(kinfo->fdt, "interrupt-parent",
+ kinfo->phandle_intc);
+ if ( res )
+ return res;
+
res = fdt_end_node(fdt);
return res;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 17/22] xen/arm: vsmmuv3: Add support to send stage-1 event to guest
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (15 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 16/22] xen/arm: vsmmuv3: Insert vIRQ properties into vIOMMU device tree node Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-09 15:12 ` Oleksandr Tyshchenko
2026-10-07 0:07 ` [PATCH v4 18/22] libxl/arm: vIOMMU: Modify the partial device tree for iommus Milan Djokic
` (4 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Bertrand Marquis, Stefano Stabellini, Julien Grall,
Michal Orzel, Volodymyr Babchuk, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
Stage-1 translation is handled by guest, therefore stage-1 fault has to
be forwarded to guest.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/drivers/passthrough/arm/smmu-v3.c | 48 ++++++++++++++++++++++++--
xen/drivers/passthrough/arm/vsmmu-v3.c | 44 +++++++++++++++++++++++
xen/drivers/passthrough/arm/vsmmu-v3.h | 8 +++++
3 files changed, 98 insertions(+), 2 deletions(-)
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index 196ac660ef..e44f872d2d 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -871,7 +871,6 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid)
return 0;
}
-__maybe_unused
static struct arm_smmu_master *
arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid)
{
@@ -892,10 +891,51 @@ arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid)
return NULL;
}
+static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt)
+{
+ int ret;
+ struct arm_smmu_master *master;
+ u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]);
+
+ switch (FIELD_GET(EVTQ_0_ID, evt[0])) {
+ case EVT_ID_TRANSLATION_FAULT:
+ break;
+ case EVT_ID_ADDR_SIZE_FAULT:
+ break;
+ case EVT_ID_ACCESS_FAULT:
+ break;
+ case EVT_ID_PERMISSION_FAULT:
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ /* Stage-2 event */
+ if (evt[1] & EVTQ_1_S2)
+ return -EFAULT;
+
+ mutex_lock(&smmu->streams_mutex);
+ master = arm_smmu_find_master(smmu, sid);
+ if (!master) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+
+ ret = arm_vsmmu_handle_evt(master->domain->d, smmu->dev, evt);
+ if (ret) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+
+out_unlock:
+ mutex_unlock(&smmu->streams_mutex);
+ return ret;
+}
+
/* IRQ and event handlers */
static void arm_smmu_evtq_tasklet(void *dev)
{
- int i;
+ int i, ret;
struct arm_smmu_device *smmu = dev;
struct arm_smmu_queue *q = &smmu->evtq.q;
struct arm_smmu_ll_queue *llq = &q->llq;
@@ -905,6 +945,10 @@ static void arm_smmu_evtq_tasklet(void *dev)
while (!queue_remove_raw(q, evt)) {
u8 id = FIELD_GET(EVTQ_0_ID, evt[0]);
+ ret = arm_smmu_handle_evt(smmu, evt);
+ if (!ret)
+ continue;
+
dev_info(smmu->dev, "event 0x%02x received:\n", id);
for (i = 0; i < ARRAY_SIZE(evt); ++i)
dev_info(smmu->dev, "\t0x%016llx\n",
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index 87342fdf69..b71511c2eb 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -338,6 +338,49 @@ static void arm_vsmmu_send_event(struct virt_smmu *smmu,
return;
}
+static struct virt_smmu *vsmmuv3_find_by_addr(struct domain *d, paddr_t paddr)
+{
+ struct virt_smmu *smmu;
+
+ list_for_each_entry( smmu, &d->arch.viommu_list, viommu_list )
+ {
+ if ( smmu->addr == paddr )
+ return smmu;
+ }
+
+ return NULL;
+}
+
+int arm_vsmmu_handle_evt(struct domain *d, struct device *dev, uint64_t *evt)
+{
+ int ret;
+ struct virt_smmu *smmu;
+
+ if ( is_hardware_domain(d) )
+ {
+ paddr_t paddr;
+ /* Base address */
+ ret = dt_device_get_address(dev_to_dt(dev), 0, &paddr, NULL);
+ if ( ret )
+ return -EINVAL;
+
+ smmu = vsmmuv3_find_by_addr(d, paddr);
+ if ( !smmu )
+ return -ENODEV;
+ }
+ else
+ {
+ smmu = list_entry(d->arch.viommu_list.next,
+ struct virt_smmu, viommu_list);
+ }
+
+ ret = arm_vsmmu_write_evtq(smmu, evt);
+ if ( ret )
+ arm_vsmmu_set_gerror(smmu, GERROR_EVTQ_ABT_ERR);
+
+ return 0;
+}
+
static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
uint64_t *ste)
{
@@ -1118,6 +1161,7 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr,
smmu->d = d;
smmu->features = features;
smmu->virq = virq;
+ smmu->addr = addr;
smmu->cmdq.ent_size = CMDQ_ENT_DWORDS * DWORDS_BYTES;
smmu->evtq.ent_size = EVTQ_ENT_DWORDS * DWORDS_BYTES;
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.h b/xen/drivers/passthrough/arm/vsmmu-v3.h
index 9f4c86abd5..d4b7cf805f 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.h
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.h
@@ -8,6 +8,8 @@
void vsmmuv3_set_type(void);
+int arm_vsmmu_handle_evt(struct domain *d, struct device *dev, uint64_t *evt);
+
#else
static inline void __init vsmmuv3_set_type(void)
@@ -15,6 +17,12 @@ static inline void __init vsmmuv3_set_type(void)
return;
}
+static inline int arm_vsmmu_handle_evt(struct domain *d,
+ struct device *dev, uint64_t *evt)
+{
+ return -EINVAL;
+}
+
#endif /* CONFIG_VIRTUAL_ARM_SMMU_V3 */
#endif /* __ARCH_ARM_VSMMU_V3_H__ */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 17/22] xen/arm: vsmmuv3: Add support to send stage-1 event to guest
2026-10-07 0:07 ` [PATCH v4 17/22] xen/arm: vsmmuv3: Add support to send stage-1 event to guest Milan Djokic
@ 2026-10-09 15:12 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-09 15:12 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Bertrand Marquis, Stefano Stabellini, Julien Grall,
Michal Orzel, Volodymyr Babchuk
On 10/7/26 03:07, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> Stage-1 translation is handled by guest, therefore stage-1 fault has to
> be forwarded to guest.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/drivers/passthrough/arm/smmu-v3.c | 48 ++++++++++++++++++++++++--
> xen/drivers/passthrough/arm/vsmmu-v3.c | 44 +++++++++++++++++++++++
> xen/drivers/passthrough/arm/vsmmu-v3.h | 8 +++++
> 3 files changed, 98 insertions(+), 2 deletions(-)
>
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
> index 196ac660ef..e44f872d2d 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.c
> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
> @@ -871,7 +871,6 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid)
> return 0;
> }
>
> -__maybe_unused
> static struct arm_smmu_master *
> arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid)
> {
> @@ -892,10 +891,51 @@ arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid)
> return NULL;
> }
>
> +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt)
> +{
> + int ret;
> + struct arm_smmu_master *master;
> + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]);
Maybe my comment should better be suited for commit #20/22 (where the
vSID->pSID mapping is introduced)...
Look like the event is copied to the guest unchanged (with patch 20
applied). It carries the pSID instead of the vSID, even though patch
20’s table could translate it. So the domain might not find the device,
or might blame a different one whose vSID matches that number. Or I
missed something?
> +
> + switch (FIELD_GET(EVTQ_0_ID, evt[0])) {
> + case EVT_ID_TRANSLATION_FAULT:
> + break;
> + case EVT_ID_ADDR_SIZE_FAULT:
> + break;
> + case EVT_ID_ACCESS_FAULT:
> + break;
> + case EVT_ID_PERMISSION_FAULT:
> + break;
> + default:
> + return -EOPNOTSUPP;
> + }
> +
> + /* Stage-2 event */
> + if (evt[1] & EVTQ_1_S2)
> + return -EFAULT;
> +
> + mutex_lock(&smmu->streams_mutex);
> + master = arm_smmu_find_master(smmu, sid);
> + if (!master) {
> + ret = -EINVAL;
> + goto out_unlock;
> + }
> +
> + ret = arm_vsmmu_handle_evt(master->domain->d, smmu->dev, evt);
^^^ dereferences master->domain->d with no NULL check. Would it be
possible that a fault event queued earlier can still arrive after the
device is detached (on domain destruction), so master->domain is NULL?
> + if (ret) {
> + ret = -EINVAL;
> + goto out_unlock;
> + }
> +
> +out_unlock:
> + mutex_unlock(&smmu->streams_mutex);
> + return ret;
> +}
> +
> /* IRQ and event handlers */
> static void arm_smmu_evtq_tasklet(void *dev)
> {
> - int i;
> + int i, ret;
> struct arm_smmu_device *smmu = dev;
> struct arm_smmu_queue *q = &smmu->evtq.q;
> struct arm_smmu_ll_queue *llq = &q->llq;
> @@ -905,6 +945,10 @@ static void arm_smmu_evtq_tasklet(void *dev)
> while (!queue_remove_raw(q, evt)) {
> u8 id = FIELD_GET(EVTQ_0_ID, evt[0]);
>
> + ret = arm_smmu_handle_evt(smmu, evt);
> + if (!ret)
> + continue;
> +
> dev_info(smmu->dev, "event 0x%02x received:\n", id);
> for (i = 0; i < ARRAY_SIZE(evt); ++i)
> dev_info(smmu->dev, "\t0x%016llx\n",
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
> index 87342fdf69..b71511c2eb 100644
> --- a/xen/drivers/passthrough/arm/vsmmu-v3.c
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
> @@ -338,6 +338,49 @@ static void arm_vsmmu_send_event(struct virt_smmu *smmu,
> return;
> }
>
> +static struct virt_smmu *vsmmuv3_find_by_addr(struct domain *d, paddr_t paddr)
> +{
> + struct virt_smmu *smmu;
> +
> + list_for_each_entry( smmu, &d->arch.viommu_list, viommu_list )
> + {
> + if ( smmu->addr == paddr )
> + return smmu;
> + }
> +
> + return NULL;
> +}
> +
> +int arm_vsmmu_handle_evt(struct domain *d, struct device *dev, uint64_t *evt)
> +{
> + int ret;
> + struct virt_smmu *smmu;
> +
> + if ( is_hardware_domain(d) )
> + {
> + paddr_t paddr;
> + /* Base address */
> + ret = dt_device_get_address(dev_to_dt(dev), 0, &paddr, NULL);
> + if ( ret )
> + return -EINVAL;
> +
> + smmu = vsmmuv3_find_by_addr(d, paddr);
> + if ( !smmu )
> + return -ENODEV;
> + }
> + else
> + {
> + smmu = list_entry(d->arch.viommu_list.next,
> + struct virt_smmu, viommu_list);
> + }
> +
> + ret = arm_vsmmu_write_evtq(smmu, evt);
> + if ( ret )
> + arm_vsmmu_set_gerror(smmu, GERROR_EVTQ_ABT_ERR);
> +
> + return 0;
> +}
> +
> static int arm_vsmmu_find_ste(struct virt_smmu *smmu, uint32_t sid,
> uint64_t *ste)
> {
> @@ -1118,6 +1161,7 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr,
> smmu->d = d;
> smmu->features = features;
> smmu->virq = virq;
> + smmu->addr = addr;
> smmu->cmdq.ent_size = CMDQ_ENT_DWORDS * DWORDS_BYTES;
> smmu->evtq.ent_size = EVTQ_ENT_DWORDS * DWORDS_BYTES;
>
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.h b/xen/drivers/passthrough/arm/vsmmu-v3.h
> index 9f4c86abd5..d4b7cf805f 100644
> --- a/xen/drivers/passthrough/arm/vsmmu-v3.h
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.h
> @@ -8,6 +8,8 @@
>
> void vsmmuv3_set_type(void);
>
> +int arm_vsmmu_handle_evt(struct domain *d, struct device *dev, uint64_t *evt);
> +
> #else
>
> static inline void __init vsmmuv3_set_type(void)
> @@ -15,6 +17,12 @@ static inline void __init vsmmuv3_set_type(void)
> return;
> }
>
> +static inline int arm_vsmmu_handle_evt(struct domain *d,
> + struct device *dev, uint64_t *evt)
> +{
> + return -EINVAL;
> +}
> +
> #endif /* CONFIG_VIRTUAL_ARM_SMMU_V3 */
>
> #endif /* __ARCH_ARM_VSMMU_V3_H__ */
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 18/22] libxl/arm: vIOMMU: Modify the partial device tree for iommus
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (16 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 17/22] xen/arm: vsmmuv3: Add support to send stage-1 event to guest Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-07 0:07 ` [PATCH v4 19/22] xen/arm: vIOMMU: Modify the partial device tree for dom0less Milan Djokic
` (3 subsequent siblings)
21 siblings, 0 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Anthony PERARD, Juergen Gross, Milan Djokic,
Luca Fancellu
From: Rahul Singh <rahul.singh@arm.com>
To configure IOMMU in guest for passthrough devices, user will need to
copy the unmodified "iommus" property from host device tree to partial
device tree. To enable the dom0 linux kernel to configure the IOMMU
correctly replace the phandle in partial device tree with virtual
IOMMU phandle when "iommus" property is set.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Singed-off-by: Milan Djokic <milan_djokic@epam.com>
Reviewed-by: Luca Fancellu <luca.fancellu@arm.com>
---
tools/libs/light/libxl_arm.c | 47 +++++++++++++++++++++++++++++++++++-
1 file changed, 46 insertions(+), 1 deletion(-)
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index 410dc98d09..daaa81f2aa 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -1322,6 +1322,41 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt)
return 0;
}
+static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
+{
+ int nodeoff, proplen, i, r;
+ const fdt32_t *prop;
+ fdt32_t *prop_c;
+
+ nodeoff = fdt_path_offset(pfdt, "/passthrough");
+ if (nodeoff < 0)
+ return nodeoff;
+
+ for (nodeoff = fdt_first_subnode(pfdt, nodeoff);
+ nodeoff >= 0;
+ nodeoff = fdt_next_subnode(pfdt, nodeoff)) {
+
+ prop = fdt_getprop(pfdt, nodeoff, "iommus", &proplen);
+ if (!prop)
+ continue;
+
+ prop_c = libxl__zalloc(gc, proplen);
+
+ for (i = 0; i < proplen / 8; ++i) {
+ prop_c[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
+ prop_c[i * 2 + 1] = prop[i * 2 + 1];
+ }
+
+ r = fdt_setprop(pfdt, nodeoff, "iommus", prop_c, proplen);
+ if (r) {
+ LOG(ERROR, "Can't set the iommus property in partial FDT");
+ return r;
+ }
+ }
+
+ return 0;
+}
+
#else
static int check_partial_fdt(libxl__gc *gc, void *fdt, size_t size)
@@ -1340,6 +1375,13 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt)
return -FDT_ERR_INTERNAL;
}
+static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
+{
+ LOG(ERROR, "partial device tree not supported");
+
+ return ERROR_FAIL;
+}
+
#endif /* ENABLE_PARTIAL_DEVICE_TREE */
#define FDT_MAX_SIZE (1<<20)
@@ -1462,8 +1504,11 @@ next_resize:
if (d_config->num_pcidevs)
FDT( make_vpci_node(gc, fdt, ainfo, dom) );
- if (info->arch_arm.viommu_type == LIBXL_VIOMMU_TYPE_SMMUV3)
+ if (info->arch_arm.viommu_type == LIBXL_VIOMMU_TYPE_SMMUV3) {
FDT( make_vsmmuv3_node(gc, fdt, ainfo, dom) );
+ if (pfdt)
+ FDT( modify_partial_fdt(gc, pfdt) );
+ }
for (i = 0; i < d_config->num_disks; i++) {
libxl_device_disk *disk = &d_config->disks[i];
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v4 19/22] xen/arm: vIOMMU: Modify the partial device tree for dom0less
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (17 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 18/22] libxl/arm: vIOMMU: Modify the partial device tree for iommus Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-09 12:34 ` Oleksandr Tyshchenko
2026-10-07 0:07 ` [PATCH v4 20/22] xen/arm: vIOMMU vSID->pSID mapping layer Milan Djokic
` (2 subsequent siblings)
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Milan Djokic
From: Rahul Singh <rahul.singh@arm.com>
To configure IOMMU in guest for passthrough devices, user will need to
copy the unmodified "iommus" property from host device tree to partial
device tree. To enable the dom0 linux kernel to configure the IOMMU
correctly replace the phandle in partial device tree with virtual
IOMMU phandle when "iommus" property is set.
Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/common/device-tree/dom0less-build.c | 40 ++++++++++++++++++++++++-
1 file changed, 39 insertions(+), 1 deletion(-)
diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
index fcbeb8adbd..7280bcf72b 100644
--- a/xen/common/device-tree/dom0less-build.c
+++ b/xen/common/device-tree/dom0less-build.c
@@ -331,7 +331,44 @@ static int __init handle_prop_pfdt(struct kernel_info *kinfo,
return ( propoff != -FDT_ERR_NOTFOUND ) ? propoff : 0;
}
-static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt,
+#ifdef CONFIG_ARM_VIRTUAL_IOMMU
+static void __init modify_pfdt_node(void *pfdt, int nodeoff)
+{
+ int proplen, i, rc;
+ const fdt32_t *prop;
+ fdt32_t *prop_c;
+
+ prop = fdt_getprop(pfdt, nodeoff, "iommus", &proplen);
+ if ( !prop )
+ return;
+
+ prop_c = xzalloc_bytes(proplen);
+ if ( !prop_c )
+ return;
+
+ for ( i = 0; i < proplen / 8; ++i )
+ {
+ prop_c[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
+ prop_c[i * 2 + 1] = prop[i * 2 + 1];
+ }
+
+ rc = fdt_setprop(pfdt, nodeoff, "iommus", prop_c, proplen);
+ if ( rc )
+ {
+ dprintk(XENLOG_ERR, "Can't set the iommus property in partial FDT");
+ return;
+ }
+
+ return;
+}
+#else
+ static inline void __init modify_pfdt_node(void *pfdt, int nodeoff)
+ {
+ return;
+ }
+#endif
+
+static int __init scan_pfdt_node(struct kernel_info *kinfo, void *pfdt,
int nodeoff,
uint32_t address_cells, uint32_t size_cells,
bool scan_passthrough_prop)
@@ -357,6 +394,7 @@ static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt,
node_next = fdt_first_subnode(pfdt, nodeoff);
while ( node_next > 0 )
{
+ modify_pfdt_node(pfdt, node_next);
rc = scan_pfdt_node(kinfo, pfdt, node_next, address_cells, size_cells,
scan_passthrough_prop);
if ( rc )
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 19/22] xen/arm: vIOMMU: Modify the partial device tree for dom0less
2026-10-07 0:07 ` [PATCH v4 19/22] xen/arm: vIOMMU: Modify the partial device tree for dom0less Milan Djokic
@ 2026-10-09 12:34 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-09 12:34 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel
On 10/7/26 03:07, Milan Djokic wrote:
Hello Milan
> From: Rahul Singh <rahul.singh@arm.com>
>
> To configure IOMMU in guest for passthrough devices, user will need to
> copy the unmodified "iommus" property from host device tree to partial
> device tree. To enable the dom0 linux kernel to configure the IOMMU
> correctly replace the phandle in partial device tree with virtual
> IOMMU phandle when "iommus" property is set.
>
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/common/device-tree/dom0less-build.c | 40 ++++++++++++++++++++++++-
> 1 file changed, 39 insertions(+), 1 deletion(-)
>
> diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
> index fcbeb8adbd..7280bcf72b 100644
> --- a/xen/common/device-tree/dom0less-build.c
> +++ b/xen/common/device-tree/dom0less-build.c
> @@ -331,7 +331,44 @@ static int __init handle_prop_pfdt(struct kernel_info *kinfo,
> return ( propoff != -FDT_ERR_NOTFOUND ) ? propoff : 0;
> }
>
> -static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt,
> +#ifdef CONFIG_ARM_VIRTUAL_IOMMU
> +static void __init modify_pfdt_node(void *pfdt, int nodeoff)
> +{
> + int proplen, i, rc;
> + const fdt32_t *prop;
> + fdt32_t *prop_c;
> +
> + prop = fdt_getprop(pfdt, nodeoff, "iommus", &proplen);
> + if ( !prop )
> + return;
> +
> + prop_c = xzalloc_bytes(proplen);
> + if ( !prop_c )
> + return;
> +
> + for ( i = 0; i < proplen / 8; ++i )
> + {
> + prop_c[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
> + prop_c[i * 2 + 1] = prop[i * 2 + 1];
> + }
> +
> + rc = fdt_setprop(pfdt, nodeoff, "iommus", prop_c, proplen);
> + if ( rc )
> + {
> + dprintk(XENLOG_ERR, "Can't set the iommus property in partial FDT");
> + return;
> + }
> +
> + return;
> +}
> +#else
> + static inline void __init modify_pfdt_node(void *pfdt, int nodeoff)
> + {
> + return;
> + }
> +#endif
> +
> +static int __init scan_pfdt_node(struct kernel_info *kinfo, void *pfdt,
> int nodeoff,
> uint32_t address_cells, uint32_t size_cells,
> bool scan_passthrough_prop)
> @@ -357,6 +394,7 @@ static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt,
> node_next = fdt_first_subnode(pfdt, nodeoff);
> while ( node_next > 0 )
> {
> + modify_pfdt_node(pfdt, node_next);
I might miss something, but ^^^ returns void, so allocation or vSID
assigment failures (viommu_allocate_vsid() is added by next commit
#20/22) are ignored. So the guest might be created with the host phandle
and SID in its device tree or with a half-updated device tree. The IOMMU
may (will?) not work and nobody is told. Should not we return the error
here?
Also looks like the each rewritten node also leaks prop_c.
> rc = scan_pfdt_node(kinfo, pfdt, node_next, address_cells, size_cells,
> scan_passthrough_prop);
> if ( rc )
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 20/22] xen/arm: vIOMMU vSID->pSID mapping layer
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (18 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 19/22] xen/arm: vIOMMU: Modify the partial device tree for dom0less Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-09 14:15 ` Oleksandr Tyshchenko
2026-10-07 0:07 ` [PATCH v4 21/22] libxl/arm: Introduce domctl command for IOMMU vSID mapping Milan Djokic
2026-10-07 0:08 ` [PATCH v4 22/22] doc/arm: vIOMMU design document Milan Djokic
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Milan Djokic, Stefano Stabellini, Julien Grall, Bertrand Marquis,
Michal Orzel, Volodymyr Babchuk, Rahul Singh, Jan Beulich,
Roger Pau Monné
Introduce a vIOMMU mapping layer to support passthrough of IOMMU devices
attached to different physical IOMMUs, including devices with the same
Stream ID.
Add a new generic vIOMMU API, viommu_allocate_vsid(), which allocates a
guest vSID and maps it to the input pSID and the backing physical SMMU.
Once the mapping is established, the guest uses the vSID for stage-1
commands, and Xen translates the vSID to the corresponding pSID before
processing the command on the physical SMMU.
Implement usage of the new API for dom0less guests. Allocate vSIDs when
creating the guest device tree and replace the original pSIDs with vSIDs
that are subsequently used by the guest driver.
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
xen/arch/arm/include/asm/viommu.h | 12 +++
xen/common/device-tree/dom0less-build.c | 33 +++++--
xen/drivers/passthrough/arm/smmu-v3.c | 10 ++-
xen/drivers/passthrough/arm/viommu.c | 9 ++
xen/drivers/passthrough/arm/vsmmu-v3.c | 109 +++++++++++++++++++++++-
xen/include/xen/iommu.h | 6 +-
6 files changed, 165 insertions(+), 14 deletions(-)
diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
index 5ae21042f0..3d0eb92655 100644
--- a/xen/arch/arm/include/asm/viommu.h
+++ b/xen/arch/arm/include/asm/viommu.h
@@ -34,6 +34,16 @@ struct viommu_ops {
* Called during domain destruction to free resources used by vIOMMU.
*/
int (*relinquish_resources)(struct domain *d);
+
+ /*
+ * Allocate free vSID/vRID for the guest device and establish vID->pID mapping
+ * Called during domain device assignment.
+ * Returns 0 on success and sets vsid argument to newly allocated vSID/vRID
+ * mapped to physical ID (id argument).
+ * Negative error code returned if allocation fails.
+ */
+ int (*allocate_vsid)(struct domain *d, uint32_t id, uint32_t phandle,
+ uint32_t *vsid);
};
struct viommu_desc {
@@ -54,6 +64,8 @@ extern struct list_head host_iommu_list;
int domain_viommu_init(struct domain *d, uint8_t viommu_type);
int viommu_relinquish_resources(struct domain *d);
uint8_t viommu_get_type(void);
+int viommu_allocate_vsid(struct domain *d, uint32_t id, uint32_t phandle,
+ uint32_t *vsid);
void add_to_host_iommu_list(paddr_t addr, paddr_t size,
const struct dt_device_node *node,
const void *iommu);
diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
index 7280bcf72b..673bcbbd1c 100644
--- a/xen/common/device-tree/dom0less-build.c
+++ b/xen/common/device-tree/dom0less-build.c
@@ -31,6 +31,8 @@
#include <xen/static-memory.h>
#include <xen/static-shmem.h>
+#include <asm/viommu.h>
+
#define XENSTORE_PFN_LATE_ALLOC UINT64_MAX
static domid_t __initdata xs_domid = DOMID_INVALID;
@@ -332,11 +334,12 @@ static int __init handle_prop_pfdt(struct kernel_info *kinfo,
}
#ifdef CONFIG_ARM_VIRTUAL_IOMMU
-static void __init modify_pfdt_node(void *pfdt, int nodeoff)
+static void __init modify_pfdt_node(void *pfdt, int nodeoff, struct domain *d)
{
int proplen, i, rc;
const fdt32_t *prop;
fdt32_t *prop_c;
+ uint32_t psid, phandle, vsid;
prop = fdt_getprop(pfdt, nodeoff, "iommus", &proplen);
if ( !prop )
@@ -346,10 +349,23 @@ static void __init modify_pfdt_node(void *pfdt, int nodeoff)
if ( !prop_c )
return;
+ /*
+ * Assign <vIOMMU vSID> pairs to iommus property and establish
+ * vSID->pSID mappings
+ */
for ( i = 0; i < proplen / 8; ++i )
{
+ phandle = fdt32_to_cpu(prop[i * 2]);
+ psid = fdt32_to_cpu(prop[i * 2 + 1]);
+ rc = viommu_allocate_vsid(d, psid, phandle, &vsid);
+ if ( rc )
+ {
+ dprintk(XENLOG_ERR, "(DOMID=%u) Failed to allocate vSID for "
+ "pSID: %u, pIOMMU: %u", d->domain_id, psid, phandle);
+ return;
+ }
prop_c[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
- prop_c[i * 2 + 1] = prop[i * 2 + 1];
+ prop_c[i * 2 + 1] = cpu_to_fdt32(vsid);
}
rc = fdt_setprop(pfdt, nodeoff, "iommus", prop_c, proplen);
@@ -362,7 +378,8 @@ static void __init modify_pfdt_node(void *pfdt, int nodeoff)
return;
}
#else
- static inline void __init modify_pfdt_node(void *pfdt, int nodeoff)
+ static inline void __init modify_pfdt_node(void *pfdt, int nodeoff,
+ struct domain *d)
{
return;
}
@@ -371,7 +388,7 @@ static void __init modify_pfdt_node(void *pfdt, int nodeoff)
static int __init scan_pfdt_node(struct kernel_info *kinfo, void *pfdt,
int nodeoff,
uint32_t address_cells, uint32_t size_cells,
- bool scan_passthrough_prop)
+ bool scan_passthrough_prop, struct domain *d)
{
int rc = 0;
void *fdt = kinfo->fdt;
@@ -394,9 +411,9 @@ static int __init scan_pfdt_node(struct kernel_info *kinfo, void *pfdt,
node_next = fdt_first_subnode(pfdt, nodeoff);
while ( node_next > 0 )
{
- modify_pfdt_node(pfdt, node_next);
+ modify_pfdt_node(pfdt, node_next, d);
rc = scan_pfdt_node(kinfo, pfdt, node_next, address_cells, size_cells,
- scan_passthrough_prop);
+ scan_passthrough_prop, d);
if ( rc )
return rc;
@@ -483,7 +500,7 @@ static int __init domain_handle_dtb_boot_module(struct domain *d,
res = scan_pfdt_node(kinfo, pfdt, node_next,
DT_ROOT_NODE_ADDR_CELLS_DEFAULT,
DT_ROOT_NODE_SIZE_CELLS_DEFAULT,
- false);
+ false, d);
if ( res )
goto out;
continue;
@@ -493,7 +510,7 @@ static int __init domain_handle_dtb_boot_module(struct domain *d,
res = scan_pfdt_node(kinfo, pfdt, node_next,
DT_ROOT_NODE_ADDR_CELLS_DEFAULT,
DT_ROOT_NODE_SIZE_CELLS_DEFAULT,
- true);
+ true, d);
if ( res )
goto out;
continue;
diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index e44f872d2d..297d2e5e7b 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -3048,7 +3048,7 @@ static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev)
}
static struct iommu_domain *arm_smmu_get_domain_by_sid(struct domain *d,
- u32 sid)
+ u32 sid, const struct arm_smmu_device *smmu)
{
int i;
unsigned long flags;
@@ -3064,6 +3064,9 @@ static struct iommu_domain *arm_smmu_get_domain_by_sid(struct domain *d,
list_for_each_entry(io_domain, &xen_domain->contexts, list) {
smmu_domain = to_smmu_domain(io_domain);
+ if (smmu_domain->smmu != smmu)
+ continue;
+
spin_lock_irqsave(&smmu_domain->devices_lock, flags);
list_for_each_entry(master, &smmu_domain->devices, domain_head) {
for (i = 0; i < master->num_streams; i++) {
@@ -3292,14 +3295,15 @@ static void arm_smmu_iommu_xen_domain_teardown(struct domain *d)
}
static int arm_smmu_attach_guest_config(struct domain *d, uint32_t sid,
- const struct iommu_guest_config *cfg)
+ const struct iommu_guest_config *cfg,
+ const struct arm_smmu_device *smmu)
{
int ret = -EINVAL;
unsigned long flags;
struct arm_smmu_master *master;
struct arm_smmu_domain *smmu_domain;
struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
- struct iommu_domain *io_domain = arm_smmu_get_domain_by_sid(d, sid);
+ struct iommu_domain *io_domain = arm_smmu_get_domain_by_sid(d, sid, smmu);
if (!io_domain)
return -ENODEV;
diff --git a/xen/drivers/passthrough/arm/viommu.c b/xen/drivers/passthrough/arm/viommu.c
index 407e68e62d..6c97acb049 100644
--- a/xen/drivers/passthrough/arm/viommu.c
+++ b/xen/drivers/passthrough/arm/viommu.c
@@ -75,6 +75,15 @@ int viommu_relinquish_resources(struct domain *d)
return cur_viommu->ops->relinquish_resources(d);
}
+int viommu_allocate_vsid(struct domain *d, uint32_t id, uint32_t phandle,
+ uint32_t *vsid)
+{
+ if ( !cur_viommu )
+ return -ENODEV;
+
+ return cur_viommu->ops->allocate_vsid(d, id, phandle, vsid);
+}
+
uint8_t viommu_get_type(void)
{
if ( !cur_viommu || host_iommu_list_incomplete ||
diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
index b71511c2eb..53fd7f2d63 100644
--- a/xen/drivers/passthrough/arm/vsmmu-v3.c
+++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
@@ -85,6 +85,8 @@
#define smmu_get_ste_s1ctxptr(x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \
FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x))
+#define MAX_VSID (1 << SMMU_IDR1_SIDSIZE)
+
/* event queue entry */
struct arm_smmu_evtq_ent {
/* Common fields */
@@ -132,6 +134,14 @@ struct arm_vsmmu_queue {
uint8_t max_n_shift;
};
+/* vSID->pSID mapping entry */
+struct vsid_entry {
+ bool valid;
+ uint32_t vsid;
+ uint32_t psid;
+ struct arm_smmu_device *smmu;
+};
+
struct virt_smmu {
struct domain *d;
struct list_head viommu_list;
@@ -156,6 +166,7 @@ struct virt_smmu {
spinlock_t cr2_lock;
spinlock_t strtab_cfg_lock;
spinlock_t irq_cfg_lock;
+ struct vsid_entry *vsids;
};
/* Helper functions */
@@ -545,6 +556,49 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
struct arm_vsmmu_s1_trans_cfg s1_cfg = {0};
uint32_t sid = smmu_cmd_get_sid(cmdptr[0]);
struct iommu_guest_config guest_cfg = {0};
+ struct arm_smmu_device *phys_smmu = NULL;
+ struct host_iommu *host_iommu;
+ uint32_t psid;
+ struct arm_smmu_evtq_ent ent = {
+ .opcode = EVT_ID_BAD_STREAMID,
+ .sid = sid,
+ .c_bad_ste_streamid = {
+ .ssid = 0,
+ .ssv = false,
+ },
+ };
+
+ /* vIOMMU->pIOMMU identity mapped for HW domain */
+ if ( is_hardware_domain(d) )
+ {
+ psid = sid;
+ list_for_each_entry(host_iommu, &host_iommu_list, entry)
+ {
+ if ( host_iommu->addr == smmu->addr )
+ {
+ phys_smmu = (struct arm_smmu_device *) host_iommu->iommu;
+ break;
+ }
+ }
+ }
+ else {
+ /* vSID out of range or not mapped to pSID */
+ if ( sid >= MAX_VSID || !smmu->vsids[sid].valid )
+ {
+ arm_vsmmu_send_event(smmu, &ent);
+ return -EINVAL;
+ }
+
+ psid = smmu->vsids[sid].psid;
+ phys_smmu = smmu->vsids[sid].smmu;
+ }
+
+ /* vSID mapped to invalid physical SMMU */
+ if (!phys_smmu)
+ {
+ arm_vsmmu_send_event(smmu, &ent);
+ return -EINVAL;
+ }
ret = arm_vsmmu_find_ste(smmu, sid, ste);
if ( ret )
@@ -565,7 +619,7 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
else
guest_cfg.config = ARM_SMMU_DOMAIN_S1;
- ret = hd->platform_ops->attach_guest_config(d, sid, &guest_cfg);
+ ret = hd->platform_ops->attach_guest_config(d, psid, &guest_cfg, phys_smmu);
if ( ret )
return ret;
@@ -1164,6 +1218,9 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr,
smmu->addr = addr;
smmu->cmdq.ent_size = CMDQ_ENT_DWORDS * DWORDS_BYTES;
smmu->evtq.ent_size = EVTQ_ENT_DWORDS * DWORDS_BYTES;
+ smmu->vsids = xzalloc_array(struct vsid_entry, MAX_VSID);
+ if ( !smmu->vsids )
+ return -ENOMEM;
spin_lock_init(&smmu->cmd_queue_lock);
spin_lock_init(&smmu->evt_queue_lock);
@@ -1254,8 +1311,9 @@ static int vsmmuv3_relinquish_resources(struct domain *d)
if ( list_head_is_null(&d->arch.viommu_list) )
return 0;
- list_for_each_entry_safe(pos, temp, &d->arch.viommu_list, viommu_list )
+ list_for_each_entry_safe(pos, temp, &d->arch.viommu_list, viommu_list)
{
+ xfree(pos->vsids);
list_del(&pos->viommu_list);
xfree(pos);
}
@@ -1263,8 +1321,55 @@ static int vsmmuv3_relinquish_resources(struct domain *d)
return 0;
}
+int vsmmuv3_allocate_vsid(struct domain *d, uint32_t id, uint32_t phandle,
+ uint32_t *vsid)
+{
+ uint32_t i = 0;
+ struct virt_smmu *smmu;
+ struct host_iommu *host_iommu;
+ struct arm_smmu_device *phys_smmu = NULL;
+
+ if ( list_head_is_null(&d->arch.viommu_list) )
+ return -ENODEV;
+
+ smmu = list_first_entry(&d->arch.viommu_list, struct virt_smmu, viommu_list);
+
+ /*
+ * Find the physical SMMU corresponding to the device-tree phandle.
+ */
+ list_for_each_entry(host_iommu, &host_iommu_list, entry)
+ {
+ if ( host_iommu->dt_node->phandle == phandle )
+ {
+ phys_smmu = (struct arm_smmu_device *) host_iommu->iommu;
+ break;
+ }
+ }
+
+ if ( !phys_smmu )
+ return -ENODEV;
+
+ /* Get first free vSID index */
+ while ( i < MAX_VSID && smmu->vsids[i].valid )
+ i++;
+
+ /* Max number of vSIDs already allocated? */
+ if ( i == MAX_VSID )
+ return -ENOMEM;
+
+ /* Establish vSID->pSID mapping */
+ smmu->vsids[i].valid = true;
+ smmu->vsids[i].vsid = i;
+ smmu->vsids[i].psid = id;
+ smmu->vsids[i].smmu = phys_smmu;
+ *vsid = smmu->vsids[i].vsid;
+
+ return 0;
+}
+
static const struct viommu_ops vsmmuv3_ops = {
.domain_init = domain_vsmmuv3_init,
+ .allocate_vsid = vsmmuv3_allocate_vsid,
.relinquish_resources = vsmmuv3_relinquish_resources,
};
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 253f8bb6cb..c94ee4f08e 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -312,7 +312,10 @@ static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
#endif /* HAS_DEVICE_TREE_DISCOVERY */
struct page_info;
+#ifdef CONFIG_ARM
struct iommu_guest_config;
+struct arm_smmu_device;
+#endif
/*
* Any non-zero value returned from callbacks of this type will cause the
@@ -391,7 +394,8 @@ struct iommu_ops {
#ifdef CONFIG_ARM
int (*attach_guest_config)(struct domain *d, uint32_t sid,
- const struct iommu_guest_config *cfg);
+ const struct iommu_guest_config *cfg,
+ const struct arm_smmu_device *smmu);
#endif
};
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 20/22] xen/arm: vIOMMU vSID->pSID mapping layer
2026-10-07 0:07 ` [PATCH v4 20/22] xen/arm: vIOMMU vSID->pSID mapping layer Milan Djokic
@ 2026-10-09 14:15 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-09 14:15 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Stefano Stabellini, Julien Grall, Bertrand Marquis, Michal Orzel,
Volodymyr Babchuk, Rahul Singh, Jan Beulich, Roger Pau Monné
On 10/7/26 03:07, Milan Djokic wrote:
Hello Milan
> Introduce a vIOMMU mapping layer to support passthrough of IOMMU devices
> attached to different physical IOMMUs, including devices with the same
> Stream ID.
>
> Add a new generic vIOMMU API, viommu_allocate_vsid(), which allocates a
> guest vSID and maps it to the input pSID and the backing physical SMMU.
>
> Once the mapping is established, the guest uses the vSID for stage-1
> commands, and Xen translates the vSID to the corresponding pSID before
> processing the command on the physical SMMU.
>
> Implement usage of the new API for dom0less guests. Allocate vSIDs when
> creating the guest device tree and replace the original pSIDs with vSIDs
> that are subsequently used by the guest driver.
>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> xen/arch/arm/include/asm/viommu.h | 12 +++
> xen/common/device-tree/dom0less-build.c | 33 +++++--
> xen/drivers/passthrough/arm/smmu-v3.c | 10 ++-
> xen/drivers/passthrough/arm/viommu.c | 9 ++
> xen/drivers/passthrough/arm/vsmmu-v3.c | 109 +++++++++++++++++++++++-
> xen/include/xen/iommu.h | 6 +-
> 6 files changed, 165 insertions(+), 14 deletions(-)
>
> diff --git a/xen/arch/arm/include/asm/viommu.h b/xen/arch/arm/include/asm/viommu.h
> index 5ae21042f0..3d0eb92655 100644
> --- a/xen/arch/arm/include/asm/viommu.h
> +++ b/xen/arch/arm/include/asm/viommu.h
> @@ -34,6 +34,16 @@ struct viommu_ops {
> * Called during domain destruction to free resources used by vIOMMU.
> */
> int (*relinquish_resources)(struct domain *d);
> +
> + /*
> + * Allocate free vSID/vRID for the guest device and establish vID->pID mapping
> + * Called during domain device assignment.
> + * Returns 0 on success and sets vsid argument to newly allocated vSID/vRID
> + * mapped to physical ID (id argument).
> + * Negative error code returned if allocation fails.
> + */
> + int (*allocate_vsid)(struct domain *d, uint32_t id, uint32_t phandle,
> + uint32_t *vsid);
> };
>
> struct viommu_desc {
> @@ -54,6 +64,8 @@ extern struct list_head host_iommu_list;
> int domain_viommu_init(struct domain *d, uint8_t viommu_type);
> int viommu_relinquish_resources(struct domain *d);
> uint8_t viommu_get_type(void);
> +int viommu_allocate_vsid(struct domain *d, uint32_t id, uint32_t phandle,
> + uint32_t *vsid);
> void add_to_host_iommu_list(paddr_t addr, paddr_t size,
> const struct dt_device_node *node,
> const void *iommu);
> diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/device-tree/dom0less-build.c
> index 7280bcf72b..673bcbbd1c 100644
> --- a/xen/common/device-tree/dom0less-build.c
> +++ b/xen/common/device-tree/dom0less-build.c
> @@ -31,6 +31,8 @@
> #include <xen/static-memory.h>
> #include <xen/static-shmem.h>
>
> +#include <asm/viommu.h>
> +
> #define XENSTORE_PFN_LATE_ALLOC UINT64_MAX
>
> static domid_t __initdata xs_domid = DOMID_INVALID;
> @@ -332,11 +334,12 @@ static int __init handle_prop_pfdt(struct kernel_info *kinfo,
> }
>
> #ifdef CONFIG_ARM_VIRTUAL_IOMMU
> -static void __init modify_pfdt_node(void *pfdt, int nodeoff)
> +static void __init modify_pfdt_node(void *pfdt, int nodeoff, struct domain *d)
> {
> int proplen, i, rc;
> const fdt32_t *prop;
> fdt32_t *prop_c;
> + uint32_t psid, phandle, vsid;
>
> prop = fdt_getprop(pfdt, nodeoff, "iommus", &proplen);
> if ( !prop )
> @@ -346,10 +349,23 @@ static void __init modify_pfdt_node(void *pfdt, int nodeoff)
> if ( !prop_c )
> return;
>
> + /*
> + * Assign <vIOMMU vSID> pairs to iommus property and establish
> + * vSID->pSID mappings
> + */
> for ( i = 0; i < proplen / 8; ++i )
> {
> + phandle = fdt32_to_cpu(prop[i * 2]);
> + psid = fdt32_to_cpu(prop[i * 2 + 1]);
> + rc = viommu_allocate_vsid(d, psid, phandle, &vsid);
> + if ( rc )
> + {
> + dprintk(XENLOG_ERR, "(DOMID=%u) Failed to allocate vSID for "
> + "pSID: %u, pIOMMU: %u", d->domain_id, psid, phandle);
> + return;
As I mentioned in prev commit #19/22, in that case the guest DT keeps
the host phandle and pSID while the domain creates normally since the
failure is ignored. I think, we need to propagate the error and fail
domain creation. What do you think?
> + }
> prop_c[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
> - prop_c[i * 2 + 1] = prop[i * 2 + 1];
> + prop_c[i * 2 + 1] = cpu_to_fdt32(vsid);
> }
>
> rc = fdt_setprop(pfdt, nodeoff, "iommus", prop_c, proplen);
> @@ -362,7 +378,8 @@ static void __init modify_pfdt_node(void *pfdt, int nodeoff)
> return;
> }
> #else
> - static inline void __init modify_pfdt_node(void *pfdt, int nodeoff)
> + static inline void __init modify_pfdt_node(void *pfdt, int nodeoff,
> + struct domain *d)
> {
> return;
> }
> @@ -371,7 +388,7 @@ static void __init modify_pfdt_node(void *pfdt, int nodeoff)
> static int __init scan_pfdt_node(struct kernel_info *kinfo, void *pfdt,
> int nodeoff,
> uint32_t address_cells, uint32_t size_cells,
> - bool scan_passthrough_prop)
> + bool scan_passthrough_prop, struct domain *d)
> {
> int rc = 0;
> void *fdt = kinfo->fdt;
> @@ -394,9 +411,9 @@ static int __init scan_pfdt_node(struct kernel_info *kinfo, void *pfdt,
> node_next = fdt_first_subnode(pfdt, nodeoff);
> while ( node_next > 0 )
> {
> - modify_pfdt_node(pfdt, node_next);
> + modify_pfdt_node(pfdt, node_next, d);
> rc = scan_pfdt_node(kinfo, pfdt, node_next, address_cells, size_cells,
> - scan_passthrough_prop);
> + scan_passthrough_prop, d);
> if ( rc )
> return rc;
>
> @@ -483,7 +500,7 @@ static int __init domain_handle_dtb_boot_module(struct domain *d,
> res = scan_pfdt_node(kinfo, pfdt, node_next,
> DT_ROOT_NODE_ADDR_CELLS_DEFAULT,
> DT_ROOT_NODE_SIZE_CELLS_DEFAULT,
> - false);
> + false, d);
> if ( res )
> goto out;
> continue;
> @@ -493,7 +510,7 @@ static int __init domain_handle_dtb_boot_module(struct domain *d,
> res = scan_pfdt_node(kinfo, pfdt, node_next,
> DT_ROOT_NODE_ADDR_CELLS_DEFAULT,
> DT_ROOT_NODE_SIZE_CELLS_DEFAULT,
> - true);
> + true, d);
> if ( res )
> goto out;
> continue;
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
> index e44f872d2d..297d2e5e7b 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.c
> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
> @@ -3048,7 +3048,7 @@ static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev)
> }
>
> static struct iommu_domain *arm_smmu_get_domain_by_sid(struct domain *d,
> - u32 sid)
> + u32 sid, const struct arm_smmu_device *smmu)
> {
> int i;
> unsigned long flags;
> @@ -3064,6 +3064,9 @@ static struct iommu_domain *arm_smmu_get_domain_by_sid(struct domain *d,
> list_for_each_entry(io_domain, &xen_domain->contexts, list) {
> smmu_domain = to_smmu_domain(io_domain);
>
> + if (smmu_domain->smmu != smmu)
> + continue;
> +
> spin_lock_irqsave(&smmu_domain->devices_lock, flags);
> list_for_each_entry(master, &smmu_domain->devices, domain_head) {
> for (i = 0; i < master->num_streams; i++) {
> @@ -3292,14 +3295,15 @@ static void arm_smmu_iommu_xen_domain_teardown(struct domain *d)
> }
>
> static int arm_smmu_attach_guest_config(struct domain *d, uint32_t sid,
> - const struct iommu_guest_config *cfg)
> + const struct iommu_guest_config *cfg,
> + const struct arm_smmu_device *smmu)
> {
> int ret = -EINVAL;
> unsigned long flags;
> struct arm_smmu_master *master;
> struct arm_smmu_domain *smmu_domain;
> struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> - struct iommu_domain *io_domain = arm_smmu_get_domain_by_sid(d, sid);
> + struct iommu_domain *io_domain = arm_smmu_get_domain_by_sid(d, sid, smmu);
>
> if (!io_domain)
> return -ENODEV;
> diff --git a/xen/drivers/passthrough/arm/viommu.c b/xen/drivers/passthrough/arm/viommu.c
> index 407e68e62d..6c97acb049 100644
> --- a/xen/drivers/passthrough/arm/viommu.c
> +++ b/xen/drivers/passthrough/arm/viommu.c
> @@ -75,6 +75,15 @@ int viommu_relinquish_resources(struct domain *d)
> return cur_viommu->ops->relinquish_resources(d);
> }
>
> +int viommu_allocate_vsid(struct domain *d, uint32_t id, uint32_t phandle,
> + uint32_t *vsid)
> +{
> + if ( !cur_viommu )
> + return -ENODEV;
> +
> + return cur_viommu->ops->allocate_vsid(d, id, phandle, vsid);
> +}
> +
> uint8_t viommu_get_type(void)
> {
> if ( !cur_viommu || host_iommu_list_incomplete ||
> diff --git a/xen/drivers/passthrough/arm/vsmmu-v3.c b/xen/drivers/passthrough/arm/vsmmu-v3.c
> index b71511c2eb..53fd7f2d63 100644
> --- a/xen/drivers/passthrough/arm/vsmmu-v3.c
> +++ b/xen/drivers/passthrough/arm/vsmmu-v3.c
> @@ -85,6 +85,8 @@
> #define smmu_get_ste_s1ctxptr(x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \
> FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x))
>
> +#define MAX_VSID (1 << SMMU_IDR1_SIDSIZE)
> +
> /* event queue entry */
> struct arm_smmu_evtq_ent {
> /* Common fields */
> @@ -132,6 +134,14 @@ struct arm_vsmmu_queue {
> uint8_t max_n_shift;
> };
>
> +/* vSID->pSID mapping entry */
> +struct vsid_entry {
> + bool valid;
> + uint32_t vsid;
> + uint32_t psid;
> + struct arm_smmu_device *smmu;
> +};
> +
> struct virt_smmu {
> struct domain *d;
> struct list_head viommu_list;
> @@ -156,6 +166,7 @@ struct virt_smmu {
> spinlock_t cr2_lock;
> spinlock_t strtab_cfg_lock;
> spinlock_t irq_cfg_lock;
> + struct vsid_entry *vsids;
> };
>
> /* Helper functions */
> @@ -545,6 +556,49 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
> struct arm_vsmmu_s1_trans_cfg s1_cfg = {0};
> uint32_t sid = smmu_cmd_get_sid(cmdptr[0]);
> struct iommu_guest_config guest_cfg = {0};
> + struct arm_smmu_device *phys_smmu = NULL;
> + struct host_iommu *host_iommu;
> + uint32_t psid;
> + struct arm_smmu_evtq_ent ent = {
> + .opcode = EVT_ID_BAD_STREAMID,
> + .sid = sid,
> + .c_bad_ste_streamid = {
> + .ssid = 0,
> + .ssv = false,
> + },
> + };
> +
> + /* vIOMMU->pIOMMU identity mapped for HW domain */
> + if ( is_hardware_domain(d) )
> + {
> + psid = sid;
> + list_for_each_entry(host_iommu, &host_iommu_list, entry)
> + {
> + if ( host_iommu->addr == smmu->addr )
> + {
> + phys_smmu = (struct arm_smmu_device *) host_iommu->iommu;
> + break;
> + }
> + }
> + }
> + else {
> + /* vSID out of range or not mapped to pSID */
> + if ( sid >= MAX_VSID || !smmu->vsids[sid].valid )
> + {
> + arm_vsmmu_send_event(smmu, &ent);
> + return -EINVAL;
> + }
> +
> + psid = smmu->vsids[sid].psid;
> + phys_smmu = smmu->vsids[sid].smmu;
> + }
> +
> + /* vSID mapped to invalid physical SMMU */
> + if (!phys_smmu)
> + {
> + arm_vsmmu_send_event(smmu, &ent);
> + return -EINVAL;
> + }
>
> ret = arm_vsmmu_find_ste(smmu, sid, ste);
> if ( ret )
> @@ -565,7 +619,7 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)
> else
> guest_cfg.config = ARM_SMMU_DOMAIN_S1;
>
> - ret = hd->platform_ops->attach_guest_config(d, sid, &guest_cfg);
> + ret = hd->platform_ops->attach_guest_config(d, psid, &guest_cfg, phys_smmu);
> if ( ret )
> return ret;
>
> @@ -1164,6 +1218,9 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr,
> smmu->addr = addr;
> smmu->cmdq.ent_size = CMDQ_ENT_DWORDS * DWORDS_BYTES;
> smmu->evtq.ent_size = EVTQ_ENT_DWORDS * DWORDS_BYTES;
> + smmu->vsids = xzalloc_array(struct vsid_entry, MAX_VSID);
^^^ is 65536 × 24 bytes = 1.5 MiB per vSMMU allocated contiguously even
if, lets say, the domain has two devices?
And the hwdom allocates vsids[MAX_VSID] per pSMMU even though the hwdom
never uses it (vIOMMU->pIOMMU identity mapped for HW domain) if I got
the code right.
> + if ( !smmu->vsids )
> + return -ENOMEM;
Leaks smmu when the vsids allocation fails.
and failure of vgic_reserve_virq() (non-visible in the context) leaks vsids.
>
> spin_lock_init(&smmu->cmd_queue_lock);
> spin_lock_init(&smmu->evt_queue_lock);
> @@ -1254,8 +1311,9 @@ static int vsmmuv3_relinquish_resources(struct domain *d)
> if ( list_head_is_null(&d->arch.viommu_list) )
> return 0;
>
> - list_for_each_entry_safe(pos, temp, &d->arch.viommu_list, viommu_list )
> + list_for_each_entry_safe(pos, temp, &d->arch.viommu_list, viommu_list)
> {
> + xfree(pos->vsids);
> list_del(&pos->viommu_list);
> xfree(pos);
> }
> @@ -1263,8 +1321,55 @@ static int vsmmuv3_relinquish_resources(struct domain *d)
> return 0;
> }
>
> +int vsmmuv3_allocate_vsid(struct domain *d, uint32_t id, uint32_t phandle,
> + uint32_t *vsid)
> +{
> + uint32_t i = 0;
> + struct virt_smmu *smmu;
> + struct host_iommu *host_iommu;
> + struct arm_smmu_device *phys_smmu = NULL;
> +
> + if ( list_head_is_null(&d->arch.viommu_list) )
> + return -ENODEV;
> +
> + smmu = list_first_entry(&d->arch.viommu_list, struct virt_smmu, viommu_list);
> +
> + /*
> + * Find the physical SMMU corresponding to the device-tree phandle.
> + */
> + list_for_each_entry(host_iommu, &host_iommu_list, entry)
> + {
> + if ( host_iommu->dt_node->phandle == phandle )
> + {
> + phys_smmu = (struct arm_smmu_device *) host_iommu->iommu;
> + break;
> + }
> + }
> +
> + if ( !phys_smmu )
> + return -ENODEV;
> +
> + /* Get first free vSID index */
> + while ( i < MAX_VSID && smmu->vsids[i].valid )
> + i++;
> +
> + /* Max number of vSIDs already allocated? */
> + if ( i == MAX_VSID )
> + return -ENOMEM;
> +
> + /* Establish vSID->pSID mapping */
> + smmu->vsids[i].valid = true;
> + smmu->vsids[i].vsid = i;
> + smmu->vsids[i].psid = id;
> + smmu->vsids[i].smmu = phys_smmu;
> + *vsid = smmu->vsids[i].vsid;
> +
> + return 0;
> +}
> +
> static const struct viommu_ops vsmmuv3_ops = {
> .domain_init = domain_vsmmuv3_init,
> + .allocate_vsid = vsmmuv3_allocate_vsid,
> .relinquish_resources = vsmmuv3_relinquish_resources,
> };
>
> diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
> index 253f8bb6cb..c94ee4f08e 100644
> --- a/xen/include/xen/iommu.h
> +++ b/xen/include/xen/iommu.h
> @@ -312,7 +312,10 @@ static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
> #endif /* HAS_DEVICE_TREE_DISCOVERY */
>
> struct page_info;
> +#ifdef CONFIG_ARM
> struct iommu_guest_config;
> +struct arm_smmu_device;
> +#endif
>
> /*
> * Any non-zero value returned from callbacks of this type will cause the
> @@ -391,7 +394,8 @@ struct iommu_ops {
>
> #ifdef CONFIG_ARM
> int (*attach_guest_config)(struct domain *d, uint32_t sid,
> - const struct iommu_guest_config *cfg);
> + const struct iommu_guest_config *cfg,
> + const struct arm_smmu_device *smmu);
> #endif
> };
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 21/22] libxl/arm: Introduce domctl command for IOMMU vSID mapping
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (19 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 20/22] xen/arm: vIOMMU vSID->pSID mapping layer Milan Djokic
@ 2026-10-07 0:07 ` Milan Djokic
2026-10-09 13:07 ` Oleksandr Tyshchenko
2026-10-07 0:08 ` [PATCH v4 22/22] doc/arm: vIOMMU design document Milan Djokic
21 siblings, 1 reply; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:07 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Milan Djokic, Anthony PERARD, Juergen Gross, Andrew Cooper,
Michal Orzel, Jan Beulich, Julien Grall, Roger Pau Monné,
Stefano Stabellini, Bertrand Marquis, Volodymyr Babchuk,
Daniel P. Smith
For guests created via the control domain (xl, Zephyr xenlib), a partial
device tree is parsed and loaded on the control domain side.
SIDs in the guest device tree have to be replaced with virtual SIDs that
are mapped to the corresponding physical SIDs. To do this, the control
domain requests Xen to allocate a new vSID and map it to the original
pSID for each IOMMU stream ID associated with a guest device.
Introduce a new domctl command (XEN_DOMCTL_viommu_allocate_vid) that
allows the control domain to request a new vSID mapping and replace the
original pSID with the allocated vSID in the guest device tree.
Implement vSID allocation in libxl using this interface for devices
configured behind an IOMMU.
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
tools/include/xenctrl.h | 13 ++++
tools/libs/ctrl/xc_domain.c | 25 +++++++
tools/libs/light/libxl_arm.c | 107 +++++++++++++++++++++++++---
xen/arch/arm/domctl.c | 38 ++++++++++
xen/include/public/domctl.h | 22 ++++++
xen/xsm/flask/hooks.c | 4 ++
xen/xsm/flask/policy/access_vectors | 2 +
7 files changed, 202 insertions(+), 9 deletions(-)
diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
index 9f00d4a19d..d38b365078 100644
--- a/tools/include/xenctrl.h
+++ b/tools/include/xenctrl.h
@@ -2664,6 +2664,19 @@ int xc_domain_set_llc_colors(xc_interface *xch, uint32_t domid,
const uint32_t *llc_colors,
uint32_t num_llc_colors);
+/*
+ * Allocate guest IOMMU vSID and establish its mapping to pSID.
+ * It can only be used on domain DT creation.
+ * Currently used for ARM only, possibly for RISC-V in the
+ * future. Function has no effect for x86.
+ */
+int xc_domain_viommu_allocate_vsid_range(xc_interface *xch,
+ uint32_t domid,
+ uint16_t nr_sids,
+ uint32_t first_psid,
+ uint32_t phandle,
+ uint32_t *first_vsid);
+
#if defined(__arm__) || defined(__aarch64__)
int xc_dt_overlay(xc_interface *xch, void *overlay_fdt,
uint32_t overlay_fdt_size, uint8_t overlay_op);
diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
index 94cfab0fa1..0bd3e45fbd 100644
--- a/tools/libs/ctrl/xc_domain.c
+++ b/tools/libs/ctrl/xc_domain.c
@@ -2222,6 +2222,31 @@ out:
return ret;
}
+
+int xc_domain_viommu_allocate_vsid_range(xc_interface *xch,
+ uint32_t domid,
+ uint16_t nr_sids,
+ uint32_t first_psid,
+ uint32_t phandle,
+ uint32_t *first_vsid)
+{
+ int err;
+ struct xen_domctl domctl = {};
+
+ domctl.cmd = XEN_DOMCTL_viommu_alloc_vsid_range;
+ domctl.domain = domid;
+ domctl.u.viommu_alloc_vsid_range.first_psid = first_psid;
+ domctl.u.viommu_alloc_vsid_range.phandle = phandle;
+ domctl.u.viommu_alloc_vsid_range.nr_sids = nr_sids;
+
+ if ( (err = do_domctl(xch, &domctl)) != 0 )
+ return err;
+
+ *first_vsid = domctl.u.viommu_alloc_vsid_range.first_vsid;
+
+ return 0;
+}
+
/*
* Local variables:
* mode: C
diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
index daaa81f2aa..75c6f8efc5 100644
--- a/tools/libs/light/libxl_arm.c
+++ b/tools/libs/light/libxl_arm.c
@@ -1322,11 +1322,96 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt)
return 0;
}
-static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
+/*
+ * Store virtualized 'iommus' properties for every node attached to IOMMU
+ * and passthroughed to guest.
+ * Used as a lookup table for mapping <phandle pSID> -> <vhandle vSID>
+ */
+struct viommu_stream {
+ XEN_LIST_ENTRY(struct viommu_stream) entry;
+ char path[128]; /* DT path, stable across resizes */
+ fdt32_t *iommus; /* fully virtualized iommus property */
+};
+
+static XEN_LIST_HEAD(, struct viommu_stream) viommu_stream_list;
+
+/*
+ * Helper function which creates mapping of dt node to
+ * to virtualized 'iommus' property
+ * Mappings stored in a global 'viommu_stream_list' to
+ * make it reusable for every fdt resize
+ */
+static int viommu_get_stream(libxl__gc *gc,
+ uint32_t domid,
+ const fdt32_t *prop,
+ int proplen,
+ const char* path, fdt32_t **iommus)
{
- int nodeoff, proplen, i, r;
+ int i, r;
+ uint32_t vsid, psid, phandle;
+ struct viommu_stream *viommu_stream;
+
+ /* Lookup if stream for target device is already allocated */
+ XEN_LIST_FOREACH(viommu_stream, &viommu_stream_list, entry)
+ {
+ if (!strcmp(viommu_stream->path, path)) {
+ *iommus = viommu_stream->iommus;
+ return 0;
+ }
+ }
+
+ /* Allocate new viommu stream */
+ viommu_stream = malloc(sizeof(struct viommu_stream));
+ if (!viommu_stream)
+ return ERROR_NOMEM;
+ memset(viommu_stream, 0, sizeof(struct viommu_stream));
+ viommu_stream->iommus = malloc(proplen);
+ if (!viommu_stream->iommus) {
+ free(viommu_stream);
+ return ERROR_NOMEM;
+ }
+ memset(viommu_stream->iommus, 0, proplen);
+
+ LOG(DEBUG, "Creating vIOMMU stream for device %s",
+ path);
+
+ /*
+ * Virtualize device "iommus" property
+ * (replace pIOMMU with vIOMMU phandle and pSIDs with mapped vSIDs)
+ */
+ for (i = 0; i < proplen / 8; ++i) {
+ /* Allocate new vSID mapped to pSID and physical IOMMU */
+ phandle = fdt32_to_cpu(prop[i * 2]);
+ psid = fdt32_to_cpu(prop[i * 2 + 1]);
+ r = xc_domain_viommu_allocate_vsid_range(CTX->xch, domid, 1, psid,
+ phandle, &vsid);
+ if (r) {
+ LOG(ERROR, "Can't allocate new vSID/vRID for guest IOMMU device");
+ return r;
+ }
+ viommu_stream->iommus[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
+ viommu_stream->iommus[i * 2 + 1] = cpu_to_fdt32(vsid);
+ LOG(DEBUG, "Mapped vSID: %u to pSID: %u", vsid, psid);
+ }
+
+ strcpy(viommu_stream->path, path);
+ *iommus = viommu_stream->iommus;
+
+ XEN_LIST_INSERT_HEAD(&viommu_stream_list, viommu_stream, entry);
+
+ return 0;
+}
+
+/*
+ * Used to update partial fdt when vIOMMU is enabled
+ * Maps dt properties of IOMMU devices to virtual IOMMU
+ */
+static int viommu_modify_partial_fdt(libxl__gc *gc, void *pfdt, uint32_t domid)
+{
+ int nodeoff, proplen, r;
const fdt32_t *prop;
fdt32_t *prop_c;
+ char path[128];
nodeoff = fdt_path_offset(pfdt, "/passthrough");
if (nodeoff < 0)
@@ -1340,11 +1425,16 @@ static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
if (!prop)
continue;
- prop_c = libxl__zalloc(gc, proplen);
+ r = fdt_get_path(pfdt, nodeoff, path, sizeof(path));
+ if ( r < 0 ) {
+ LOG(ERROR, "Can't get passthrough node path");
+ return r;
+ }
- for (i = 0; i < proplen / 8; ++i) {
- prop_c[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
- prop_c[i * 2 + 1] = prop[i * 2 + 1];
+ r = viommu_get_stream(gc, domid, prop, proplen, path, &prop_c);
+ if (r) {
+ LOG(ERROR, "Can't get viommu stream");
+ return r;
}
r = fdt_setprop(pfdt, nodeoff, "iommus", prop_c, proplen);
@@ -1356,7 +1446,6 @@ static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
return 0;
}
-
#else
static int check_partial_fdt(libxl__gc *gc, void *fdt, size_t size)
@@ -1375,7 +1464,7 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt)
return -FDT_ERR_INTERNAL;
}
-static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
+static int viommu_modify_partial_fdt(libxl__gc *gc, void *pfdt, uint32_t domid)
{
LOG(ERROR, "partial device tree not supported");
@@ -1507,7 +1596,7 @@ next_resize:
if (info->arch_arm.viommu_type == LIBXL_VIOMMU_TYPE_SMMUV3) {
FDT( make_vsmmuv3_node(gc, fdt, ainfo, dom) );
if (pfdt)
- FDT( modify_partial_fdt(gc, pfdt) );
+ FDT( viommu_modify_partial_fdt(gc, pfdt, dom->guest_domid) );
}
for (i = 0; i < d_config->num_disks; i++) {
diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
index b76af56fad..dc21dc9fca 100644
--- a/xen/arch/arm/domctl.c
+++ b/xen/arch/arm/domctl.c
@@ -16,6 +16,7 @@
#include <xen/types.h>
#include <xsm/xsm.h>
#include <public/domctl.h>
+#include <asm/viommu.h>
void arch_get_domain_info(const struct domain *d,
struct xen_domctl_getdomaininfo *info)
@@ -186,6 +187,43 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
}
case XEN_DOMCTL_dt_overlay:
return dt_overlay_domctl(d, &domctl->u.dt_overlay);
+
+#ifdef CONFIG_ARM_VIRTUAL_IOMMU
+ case XEN_DOMCTL_viommu_alloc_vsid_range:
+ {
+ int rc = 0;
+ uint16_t i;
+ uint32_t vsid;
+ struct xen_domctl_viommu_alloc_vsid_range *vsid_range =
+ &domctl->u.viommu_alloc_vsid_range;
+
+ if ( vsid_range->pad )
+ return -EINVAL;
+
+ for ( i = 0; i < vsid_range->nr_sids; i++ )
+ {
+ rc = viommu_allocate_vsid(d, vsid_range->first_psid
+ + i,
+ vsid_range->phandle,
+ &vsid);
+ if( rc )
+ return rc;
+ }
+
+ if ( !rc )
+ {
+ /* Calculate first vSID from allocated range */
+ vsid_range->first_vsid = vsid -
+ vsid_range->nr_sids + 1;
+ rc = copy_to_guest(u_domctl, domctl, 1);
+ if ( rc )
+ rc = -EFAULT;
+ }
+
+ return rc;
+ }
+#endif
+
default:
return subarch_do_domctl(domctl, d, u_domctl);
}
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 510300bb67..2708215d5b 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1279,6 +1279,26 @@ struct xen_domctl_get_domain_state {
uint64_t unique_id; /* Unique domain identifier. */
};
+/*
+ * XEN_DOMCTL_viommu_alloc_vsid_range
+ *
+ * Allocate guest vSID range and
+ * establish pSID->vSID mapping for target range and physical IOMMU (phandle).
+ * Allocated range is continous
+ */
+struct xen_domctl_viommu_alloc_vsid_range {
+ /* IN: Range first pSID */
+ uint32_t first_psid;
+ /* IN: Physical IOMMU phandle behind which the target device streams are configured */
+ uint32_t phandle;
+ /* IN: Number of vSIDs to allocate */
+ uint16_t nr_sids;
+ /* padding, must be 0 */
+ uint16_t pad;
+ /* OUT: Mapped range first vSID */
+ uint32_t first_vsid;
+};
+
struct xen_domctl {
/* Stable domctl ops: interface_version is required to be 0. */
uint32_t cmd;
@@ -1371,6 +1391,7 @@ struct xen_domctl {
#define XEN_DOMCTL_gsi_permission 88
#define XEN_DOMCTL_set_llc_colors 89
#define XEN_DOMCTL_get_domain_state 90 /* stable interface */
+#define XEN_DOMCTL_viommu_alloc_vsid_range 91
#define XEN_DOMCTL_gdbsx_guestmemio 1000
#define XEN_DOMCTL_gdbsx_pausevcpu 1001
#define XEN_DOMCTL_gdbsx_unpausevcpu 1002
@@ -1439,6 +1460,7 @@ struct xen_domctl {
#endif
struct xen_domctl_set_llc_colors set_llc_colors;
struct xen_domctl_get_domain_state get_domain_state;
+ struct xen_domctl_viommu_alloc_vsid_range viommu_alloc_vsid_range;
uint8_t pad[128];
} u;
};
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index d65ba0aeae..6321b4308b 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -881,6 +881,10 @@ static int cf_check flask_domctl(struct domain *d, struct xen_domctl *op)
case XEN_DOMCTL_set_llc_colors:
return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__SET_LLC_COLORS);
+ case XEN_DOMCTL_viommu_alloc_vsid_range:
+ return current_has_perm(d, SECCLASS_DOMAIN2,
+ DOMAIN2__VIOMMU_ALLOC_VSID_RANGE);
+
default:
return avc_unknown_permission("domctl", op->cmd);
}
diff --git a/xen/xsm/flask/policy/access_vectors b/xen/xsm/flask/policy/access_vectors
index bbb9c117ec..35acbefab2 100644
--- a/xen/xsm/flask/policy/access_vectors
+++ b/xen/xsm/flask/policy/access_vectors
@@ -253,6 +253,8 @@ class domain2
set_llc_colors
# XEN_DOMCTL_get_domain_state
get_domain_state
+# XEN_DOMCTL_viommu_alloc_vsid_range
+ viommu_alloc_vsid_range
}
# Similar to class domain, but primarily contains domctls related to HVM domains
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 21/22] libxl/arm: Introduce domctl command for IOMMU vSID mapping
2026-10-07 0:07 ` [PATCH v4 21/22] libxl/arm: Introduce domctl command for IOMMU vSID mapping Milan Djokic
@ 2026-10-09 13:07 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-09 13:07 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Anthony PERARD, Juergen Gross, Andrew Cooper, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné,
Stefano Stabellini, Bertrand Marquis, Volodymyr Babchuk,
Daniel P. Smith
On 10/7/26 03:07, Milan Djokic wrote:
Hello Milan
> For guests created via the control domain (xl, Zephyr xenlib), a partial
> device tree is parsed and loaded on the control domain side.
>
> SIDs in the guest device tree have to be replaced with virtual SIDs that
> are mapped to the corresponding physical SIDs. To do this, the control
> domain requests Xen to allocate a new vSID and map it to the original
> pSID for each IOMMU stream ID associated with a guest device.
>
> Introduce a new domctl command (XEN_DOMCTL_viommu_allocate_vid) that
> allows the control domain to request a new vSID mapping and replace the
> original pSID with the allocated vSID in the guest device tree.
>
> Implement vSID allocation in libxl using this interface for devices
> configured behind an IOMMU.
>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> tools/include/xenctrl.h | 13 ++++
> tools/libs/ctrl/xc_domain.c | 25 +++++++
> tools/libs/light/libxl_arm.c | 107 +++++++++++++++++++++++++---
> xen/arch/arm/domctl.c | 38 ++++++++++
> xen/include/public/domctl.h | 22 ++++++
> xen/xsm/flask/hooks.c | 4 ++
> xen/xsm/flask/policy/access_vectors | 2 +
> 7 files changed, 202 insertions(+), 9 deletions(-)
>
> diff --git a/tools/include/xenctrl.h b/tools/include/xenctrl.h
> index 9f00d4a19d..d38b365078 100644
> --- a/tools/include/xenctrl.h
> +++ b/tools/include/xenctrl.h
> @@ -2664,6 +2664,19 @@ int xc_domain_set_llc_colors(xc_interface *xch, uint32_t domid,
> const uint32_t *llc_colors,
> uint32_t num_llc_colors);
>
> +/*
> + * Allocate guest IOMMU vSID and establish its mapping to pSID.
> + * It can only be used on domain DT creation.
> + * Currently used for ARM only, possibly for RISC-V in the
> + * future. Function has no effect for x86.
> + */
> +int xc_domain_viommu_allocate_vsid_range(xc_interface *xch,
> + uint32_t domid,
> + uint16_t nr_sids,
> + uint32_t first_psid,
> + uint32_t phandle,
> + uint32_t *first_vsid);
> +
> #if defined(__arm__) || defined(__aarch64__)
> int xc_dt_overlay(xc_interface *xch, void *overlay_fdt,
> uint32_t overlay_fdt_size, uint8_t overlay_op);
> diff --git a/tools/libs/ctrl/xc_domain.c b/tools/libs/ctrl/xc_domain.c
> index 94cfab0fa1..0bd3e45fbd 100644
> --- a/tools/libs/ctrl/xc_domain.c
> +++ b/tools/libs/ctrl/xc_domain.c
> @@ -2222,6 +2222,31 @@ out:
>
> return ret;
> }
> +
> +int xc_domain_viommu_allocate_vsid_range(xc_interface *xch,
> + uint32_t domid,
> + uint16_t nr_sids,
> + uint32_t first_psid,
> + uint32_t phandle,
> + uint32_t *first_vsid)
> +{
> + int err;
> + struct xen_domctl domctl = {};
> +
> + domctl.cmd = XEN_DOMCTL_viommu_alloc_vsid_range;
> + domctl.domain = domid;
> + domctl.u.viommu_alloc_vsid_range.first_psid = first_psid;
> + domctl.u.viommu_alloc_vsid_range.phandle = phandle;
> + domctl.u.viommu_alloc_vsid_range.nr_sids = nr_sids;
> +
> + if ( (err = do_domctl(xch, &domctl)) != 0 )
> + return err;
> +
> + *first_vsid = domctl.u.viommu_alloc_vsid_range.first_vsid;
> +
> + return 0;
> +}
> +
> /*
> * Local variables:
> * mode: C
> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
> index daaa81f2aa..75c6f8efc5 100644
> --- a/tools/libs/light/libxl_arm.c
> +++ b/tools/libs/light/libxl_arm.c
> @@ -1322,11 +1322,96 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt)
> return 0;
> }
>
> -static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
> +/*
> + * Store virtualized 'iommus' properties for every node attached to IOMMU
> + * and passthroughed to guest.
> + * Used as a lookup table for mapping <phandle pSID> -> <vhandle vSID>
> + */
> +struct viommu_stream {
> + XEN_LIST_ENTRY(struct viommu_stream) entry;
> + char path[128]; /* DT path, stable across resizes */
> + fdt32_t *iommus; /* fully virtualized iommus property */
> +};
> +
> +static XEN_LIST_HEAD(, struct viommu_stream) viommu_stream_list;
Looks like ^^^ is keyed only by DT path ...
> +
> +/*
> + * Helper function which creates mapping of dt node to
> + * to virtualized 'iommus' property
> + * Mappings stored in a global 'viommu_stream_list' to
> + * make it reusable for every fdt resize
> + */
> +static int viommu_get_stream(libxl__gc *gc,
> + uint32_t domid,
> + const fdt32_t *prop,
> + int proplen,
> + const char* path, fdt32_t **iommus)
> {
> - int nodeoff, proplen, i, r;
> + int i, r;
> + uint32_t vsid, psid, phandle;
> + struct viommu_stream *viommu_stream;
> +
> + /* Lookup if stream for target device is already allocated */
> + XEN_LIST_FOREACH(viommu_stream, &viommu_stream_list, entry)
> + {
> + if (!strcmp(viommu_stream->path, path)) {
> + *iommus = viommu_stream->iommus;
> + return 0;
> + }
> + }
... a second domain with the same DT path gets the first domain’s
cached vSIDs without any mapping being created for it. Is it ok?
> +
> + /* Allocate new viommu stream */
> + viommu_stream = malloc(sizeof(struct viommu_stream));
> + if (!viommu_stream)
> + return ERROR_NOMEM;
> + memset(viommu_stream, 0, sizeof(struct viommu_stream));
> + viommu_stream->iommus = malloc(proplen);
> + if (!viommu_stream->iommus) {
> + free(viommu_stream);
> + return ERROR_NOMEM;
> + }
> + memset(viommu_stream->iommus, 0, proplen);
> +
> + LOG(DEBUG, "Creating vIOMMU stream for device %s",
> + path);
> +
> + /*
> + * Virtualize device "iommus" property
> + * (replace pIOMMU with vIOMMU phandle and pSIDs with mapped vSIDs)
> + */
> + for (i = 0; i < proplen / 8; ++i) {
> + /* Allocate new vSID mapped to pSID and physical IOMMU */
> + phandle = fdt32_to_cpu(prop[i * 2]);
> + psid = fdt32_to_cpu(prop[i * 2 + 1]);
> + r = xc_domain_viommu_allocate_vsid_range(CTX->xch, domid, 1, psid,
> + phandle, &vsid);
> + if (r) {
> + LOG(ERROR, "Can't allocate new vSID/vRID for guest IOMMU device");
> + return r;
> + }
> + viommu_stream->iommus[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
> + viommu_stream->iommus[i * 2 + 1] = cpu_to_fdt32(vsid);
> + LOG(DEBUG, "Mapped vSID: %u to pSID: %u", vsid, psid);
> + }
> +
> + strcpy(viommu_stream->path, path);
> + *iommus = viommu_stream->iommus;
> +
> + XEN_LIST_INSERT_HEAD(&viommu_stream_list, viommu_stream, entry);
> +
> + return 0;
> +}
> +
> +/*
> + * Used to update partial fdt when vIOMMU is enabled
> + * Maps dt properties of IOMMU devices to virtual IOMMU
> + */
> +static int viommu_modify_partial_fdt(libxl__gc *gc, void *pfdt, uint32_t domid)
> +{
> + int nodeoff, proplen, r;
> const fdt32_t *prop;
> fdt32_t *prop_c;
> + char path[128];
>
> nodeoff = fdt_path_offset(pfdt, "/passthrough");
> if (nodeoff < 0)
> @@ -1340,11 +1425,16 @@ static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
> if (!prop)
> continue;
>
> - prop_c = libxl__zalloc(gc, proplen);
> + r = fdt_get_path(pfdt, nodeoff, path, sizeof(path));
> + if ( r < 0 ) {
> + LOG(ERROR, "Can't get passthrough node path");
> + return r;
> + }
>
> - for (i = 0; i < proplen / 8; ++i) {
> - prop_c[i * 2] = cpu_to_fdt32(GUEST_PHANDLE_VSMMUV3);
> - prop_c[i * 2 + 1] = prop[i * 2 + 1];
> + r = viommu_get_stream(gc, domid, prop, proplen, path, &prop_c);
> + if (r) {
> + LOG(ERROR, "Can't get viommu stream");
> + return r;
> }
>
> r = fdt_setprop(pfdt, nodeoff, "iommus", prop_c, proplen);
> @@ -1356,7 +1446,6 @@ static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
>
> return 0;
> }
> -
> #else
>
> static int check_partial_fdt(libxl__gc *gc, void *fdt, size_t size)
> @@ -1375,7 +1464,7 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt)
> return -FDT_ERR_INTERNAL;
> }
>
> -static int modify_partial_fdt(libxl__gc *gc, void *pfdt)
> +static int viommu_modify_partial_fdt(libxl__gc *gc, void *pfdt, uint32_t domid)
> {
> LOG(ERROR, "partial device tree not supported");
>
> @@ -1507,7 +1596,7 @@ next_resize:
> if (info->arch_arm.viommu_type == LIBXL_VIOMMU_TYPE_SMMUV3) {
> FDT( make_vsmmuv3_node(gc, fdt, ainfo, dom) );
> if (pfdt)
> - FDT( modify_partial_fdt(gc, pfdt) );
> + FDT( viommu_modify_partial_fdt(gc, pfdt, dom->guest_domid) );
> }
>
> for (i = 0; i < d_config->num_disks; i++) {
> diff --git a/xen/arch/arm/domctl.c b/xen/arch/arm/domctl.c
> index b76af56fad..dc21dc9fca 100644
> --- a/xen/arch/arm/domctl.c
> +++ b/xen/arch/arm/domctl.c
> @@ -16,6 +16,7 @@
> #include <xen/types.h>
> #include <xsm/xsm.h>
> #include <public/domctl.h>
> +#include <asm/viommu.h>
>
> void arch_get_domain_info(const struct domain *d,
> struct xen_domctl_getdomaininfo *info)
> @@ -186,6 +187,43 @@ long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
> }
> case XEN_DOMCTL_dt_overlay:
> return dt_overlay_domctl(d, &domctl->u.dt_overlay);
> +
> +#ifdef CONFIG_ARM_VIRTUAL_IOMMU
> + case XEN_DOMCTL_viommu_alloc_vsid_range:
> + {
> + int rc = 0;
> + uint16_t i;
> + uint32_t vsid;
> + struct xen_domctl_viommu_alloc_vsid_range *vsid_range =
> + &domctl->u.viommu_alloc_vsid_range;
> +
> + if ( vsid_range->pad )
> + return -EINVAL;
> +
> + for ( i = 0; i < vsid_range->nr_sids; i++ )
> + {
> + rc = viommu_allocate_vsid(d, vsid_range->first_psid
> + + i,
> + vsid_range->phandle,
> + &vsid);
> + if( rc )
> + return rc;
> + }
> +
> + if ( !rc )
> + {
> + /* Calculate first vSID from allocated range */
> + vsid_range->first_vsid = vsid -
> + vsid_range->nr_sids + 1;
With vsid_range->nr_sids == 0, vsid stays uninitialised and garbage is
copied back to the caller.
> + rc = copy_to_guest(u_domctl, domctl, 1);
> + if ( rc )
> + rc = -EFAULT;
> + }
> +
> + return rc;
> + }
> +#endif
> +
> default:
> return subarch_do_domctl(domctl, d, u_domctl);
> }
> diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> index 510300bb67..2708215d5b 100644
> --- a/xen/include/public/domctl.h
> +++ b/xen/include/public/domctl.h
> @@ -1279,6 +1279,26 @@ struct xen_domctl_get_domain_state {
> uint64_t unique_id; /* Unique domain identifier. */
> };
>
> +/*
> + * XEN_DOMCTL_viommu_alloc_vsid_range
> + *
> + * Allocate guest vSID range and
> + * establish pSID->vSID mapping for target range and physical IOMMU (phandle).
> + * Allocated range is continous
> + */
> +struct xen_domctl_viommu_alloc_vsid_range {
> + /* IN: Range first pSID */
> + uint32_t first_psid;
> + /* IN: Physical IOMMU phandle behind which the target device streams are configured */
> + uint32_t phandle;
> + /* IN: Number of vSIDs to allocate */
> + uint16_t nr_sids;
> + /* padding, must be 0 */
> + uint16_t pad;
> + /* OUT: Mapped range first vSID */
> + uint32_t first_vsid;
> +};
> +
> struct xen_domctl {
> /* Stable domctl ops: interface_version is required to be 0. */
> uint32_t cmd;
> @@ -1371,6 +1391,7 @@ struct xen_domctl {
> #define XEN_DOMCTL_gsi_permission 88
> #define XEN_DOMCTL_set_llc_colors 89
> #define XEN_DOMCTL_get_domain_state 90 /* stable interface */
> +#define XEN_DOMCTL_viommu_alloc_vsid_range 91
> #define XEN_DOMCTL_gdbsx_guestmemio 1000
> #define XEN_DOMCTL_gdbsx_pausevcpu 1001
> #define XEN_DOMCTL_gdbsx_unpausevcpu 1002
> @@ -1439,6 +1460,7 @@ struct xen_domctl {
> #endif
> struct xen_domctl_set_llc_colors set_llc_colors;
> struct xen_domctl_get_domain_state get_domain_state;
> + struct xen_domctl_viommu_alloc_vsid_range viommu_alloc_vsid_range;
> uint8_t pad[128];
> } u;
> };
> diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
> index d65ba0aeae..6321b4308b 100644
> --- a/xen/xsm/flask/hooks.c
> +++ b/xen/xsm/flask/hooks.c
> @@ -881,6 +881,10 @@ static int cf_check flask_domctl(struct domain *d, struct xen_domctl *op)
> case XEN_DOMCTL_set_llc_colors:
> return current_has_perm(d, SECCLASS_DOMAIN2, DOMAIN2__SET_LLC_COLORS);
>
> + case XEN_DOMCTL_viommu_alloc_vsid_range:
> + return current_has_perm(d, SECCLASS_DOMAIN2,
> + DOMAIN2__VIOMMU_ALLOC_VSID_RANGE);
> +
> default:
> return avc_unknown_permission("domctl", op->cmd);
> }
> diff --git a/xen/xsm/flask/policy/access_vectors b/xen/xsm/flask/policy/access_vectors
> index bbb9c117ec..35acbefab2 100644
> --- a/xen/xsm/flask/policy/access_vectors
> +++ b/xen/xsm/flask/policy/access_vectors
> @@ -253,6 +253,8 @@ class domain2
> set_llc_colors
> # XEN_DOMCTL_get_domain_state
> get_domain_state
> +# XEN_DOMCTL_viommu_alloc_vsid_range
> + viommu_alloc_vsid_range
> }
>
> # Similar to class domain, but primarily contains domctls related to HVM domains
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v4 22/22] doc/arm: vIOMMU design document
2026-10-07 0:06 [PATCH v4 00/22] Add SMMUv3 Stage 1 Support for Xen guests Milan Djokic
` (20 preceding siblings ...)
2026-10-07 0:07 ` [PATCH v4 21/22] libxl/arm: Introduce domctl command for IOMMU vSID mapping Milan Djokic
@ 2026-10-07 0:08 ` Milan Djokic
2026-10-09 15:40 ` Dmytro Prokopchuk1
2026-10-10 15:16 ` Oleksandr Tyshchenko
21 siblings, 2 replies; 37+ messages in thread
From: Milan Djokic @ 2026-10-07 0:08 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Cc: Milan Djokic, Andrew Cooper, Anthony PERARD, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné,
Stefano Stabellini
This document outlines the design of the emulated IOMMU,
including supported features, security considerations
and future improvements.
Signed-off-by: Milan Djokic <milan_djokic@epam.com>
---
docs/designs/arm-viommu.rst | 502 ++++++++++++++++++++++++++++++++++++
1 file changed, 502 insertions(+)
create mode 100644 docs/designs/arm-viommu.rst
diff --git a/docs/designs/arm-viommu.rst b/docs/designs/arm-viommu.rst
new file mode 100644
index 0000000000..c68e771711
--- /dev/null
+++ b/docs/designs/arm-viommu.rst
@@ -0,0 +1,502 @@
+==========================================================
+Design Proposal: Add SMMUv3 Stage-1 Support for XEN Guests
+==========================================================
+
+:Author: Milan Djokic <milan_djokic@epam.com>
+:Date: 2026-02-13
+:Status: Draft
+
+Introduction
+============
+
+The SMMUv3 supports two stages of translation. Each stage of translation
+can be
+independently enabled. An incoming address is logically translated from
+VA to
+IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
+the output PA. Stage 1 translation support is required to provide
+isolation between different
+devices within OS. XEN already supports Stage 2 translation but there is no
+support for Stage 1 translation.
+This design proposal outlines the introduction of Stage-1 SMMUv3 support
+in Xen for ARM guests.
+
+Motivation
+==========
+
+ARM systems utilizing SMMUv3 require stage-1 address translation to
+ensure secure DMA and
+guest managed I/O memory mappings.
+With stage-1 enabled, guest manages IOVA to IPA mappings through its own
+IOMMU driver.
+
+This feature enables:
+
+- Stage-1 translation for the guest domain
+- Device passthrough with per-device I/O address space
+
+Design Overview
+===============
+
+These changes provide emulated SMMUv3 support:
+
+- **SMMUv3 Stage-1 Translation**: stage-1 and nested translation support
+ in SMMUv3 driver.
+- **vIOMMU Abstraction**: Virtual IOMMU framework for guest stage-1
+ handling.
+- **Register/Command Emulation**: SMMUv3 register emulation and command
+ queue handling.
+- **Device Tree Extensions**: Adds `iommus` and virtual SMMUv3 nodes to
+ device trees for dom0 and dom0less scenarios.
+- **Build Configuration**: Introduces a Kconfig arguments
+ (CONFIG_ARM_VIRTUAL_IOMMU, CONFIG_VIRTUAL_ARM_SMMU_V3) for build-time
+ enablement.
+
+Depending on the domain type, the emulated IOMMU model is the following:
+
+- For the hardware domain, the host layout is re-used. The number of exposed
+ IOMMUs to hardware domain is equal to number of physical IOMMUs
+ present (1-1 mapping).
+- For non-hardware domain, a single vIOMMU device is exposed to the guest and
+ mapped to one or more physical IOMMUs through a Xen-managed translation layer
+
+The vIOMMU feature provides a generic framework together with a backend
+implementation specific to the target IOMMU type. The backend is responsible
+for implementing the hardware-specific data structures and command handling
+logic (currently only SMMUv3 is supported).
+
+This modular design allows the stage-1 support to be reused
+for other IOMMU architectures in the future.
+
+vIOMMU architecture
+===================
+
+Responsibilities:
+
+Guest:
+ - Configures stage-1 via vIOMMU commands.
+ - Handles stage-1 faults received from Xen.
+
+Xen:
+ - Emulates the IOMMU interface (registers, commands, events).
+ - Provides vSID->pSID mappings.
+ - Programs stage-1/stage-2 configuration in the physical IOMMU.
+ - Propagate stage-1 faults to guest.
+
+The stage-1 context is shared between streams assigned to the same device.
+This follows the Linux SMMUv3 driver model, where the IOMMU domain is associated
+with the master/device, while the individual SIDs are represented by separate
+STEs pointing to the same translation context. This is considered sufficient for
+the current SMMUv3 driver model, although the SMMUv3 architecture does not
+forbid having separate translation contexts for different SIDs.
+
+vIOMMU commands and faults are transmitted between guest and Xen via
+command and event queues (one command/event queue created per guest).
+
+vIOMMU command Flow:
+
+::
+
+ Guest:
+ smmu_cmd(vSID, IOVA -> IPA)
+
+ Xen:
+ trap MMIO read/write
+ translate vSID->pSID
+ store stage-1 state
+ program pIOMMU for (pSID, IPA -> PA)
+
+All hardware programming of the physical IOMMU is performed exclusively by Xen.
+
+vIOMMU Stage-1 fault handling flow:
+
+::
+
+ Xen:
+ receives stage-1 fault
+ triggers vIOMMU callback
+ injects virtual fault
+
+ Guest:
+ receives and handles fault
+
+Register access
+---------------
+
+Guest accesses to SMMUV3 registers are trapped via MMIO regions. vIOMMU emulates
+a subset of SMMUv3 registers relevant to currently supported features.
+Unsupported registers are handled as RAZ (Read-As-Zero) and WI (Write-Ignored)
+for the register read and write operations, respectively.
+
+vSMMUv3 is a partially emulated resource for the guest domain. Some
+feature register bits (SMMU_IDR) are reflecting what the underlying HW actually
+support while the rest is preset in the emulation layer to limit guest from
+using features which are not supported by the emulation. Also, implementer
+identification register is fully emulated, providing generic ARM implementer
+ID values, to avoid activating errata/quirk paths in the guest SMMUv3 driver.
+
+Supported SMMUv3 Commands
+-------------------------
+
+The vIOMMU implementation currently supports a subset of the SMMUv3
+architectural command set required for stage-1 translation management.
+
+Commands are submitted by the guest through the virtual Command Queue
+(CMDQ) and consumed by Xen, which translates them into operations on the
+physical SMMUv3 hardware.
+
+Some commands are fully handled by the emulation layer, while others are
+currently treated as architectural no-ops because the required behavior
+is either implicitly guaranteed by Xen or not yet modeled explicitly.
+
+Emulation handled commands:
+
+- CMD_CFGI_STE
+- CMD_TLBI_NH_ASID
+- CMD_TLBI_NSNH_ALL
+- CMD_TLBI_NH_VA
+
+No-op/Implicitly handled commands:
+
+- CMD_SYNC
+- CMD_PREFETCH_CFG
+- CMD_CFGI_CD
+- CMD_CFGI_CD_ALL
+- CMD_CFGI_ALL
+
+Command Handling Model
+----------------------
+
+The guest submits commands through the virtual CMDQ interface.
+
+Xen:
+
+- Validates commands
+- Translates virtual identifiers (vSID->pSID) where required
+- Programs the physical SMMU
+- Performs synchronization
+
+The physical SMMU executes the resulting invalidation or configuration
+operations.
+
+Unsupported Commands
+--------------------
+
+Commands associated with currently unsupported architectural features
+are not implemented.
+
+This includes commands related to:
+
+- PRI
+- ATS
+- Address/Range TLB invalidation (currently, all TLBI commands perform complete
+ domain TLB invalidation)
+- Advanced stall/retry semantics
+- CD runtime update/sync
+
+vSID Mapping Layer
+------------------
+
+Each non-hardware domain-visible Stream ID (vSID) is mapped by Xen to a physical
+Stream ID (pSID). The mapping is maintained per-domain. The allocation policy
+guarantees vSID uniqueness within a domain while allowing reuse of pSIDs for
+different pIOMMUs.
+
+* Platform devices receive individually allocated vSIDs.
+* PCI devices receive a contiguous vSID range derived from RID space.
+
+
+Supported Device Model
+======================
+
+Currently, the vIOMMU framework supports only devices described via the
+Device Tree (DT) model. This includes platform devices and basic PCI
+devices support instantiated through the vPCI DT node. ACPI-described
+devices are not supported.
+
+Guest assigned platform devices are mapped via `iommus` property:
+
+::
+
+ <&pIOMMU pSID> -> <&vIOMMU vSID>
+
+PCI devices use RID-based mapping via the root complex `iommu-map`:
+
+::
+
+ <RID-base &viommu vSID-base length>
+
+PCI Topology Assumptions and Constraints:
+
+- RID space must be contiguous
+- Pre-defined continuous pSID space (0-0x1000)
+- No runtime PCI reconfiguration
+- Single root complex assumed
+- Mapping is fixed at guest DT construction
+
+Constraints for PCI devices will be addressed as part of the future work on
+this feature.
+
+Security Considerations
+=======================
+
+Stage-1 translation provides isolation between guest devices by
+enforcing a per-device I/O address space, preventing unauthorized DMA.
+With the introduction of emulated IOMMU, additional protection
+mechanisms are required to minimize security risks.
+
+1. Observation:
+---------------
+Support for Stage-1 translation in SMMUv3 introduces new data structures
+(`s1_cfg` alongside `s2_cfg`)
+and logic to write both Stage-1 and Stage-2 entries in the Stream Table
+Entry (STE), including an `abort`
+field to handle partial configuration states.
+
+**Risk:**
+Without proper handling, a partially applied configuration
+might leave guest DMA mappings in an inconsistent state, potentially
+enabling unauthorized access or causing cross-domain interference.
+
+**Mitigation:** *(Handled by design)*
+This feature introduces logic that writes both `s1_cfg` and `s2_cfg` to
+STE and manages the `abort` field - only considering
+configuration if fully attached. This ensures incomplete or invalid
+device configurations are safely ignored by the hypervisor.
+
+2. Observation:
+---------------
+Guests can now invalidate Stage-1 caches; invalidation needs forwarding
+to SMMUv3 hardware to maintain coherence.
+
+**Risk:**
+Failing to propagate cache invalidation could allow stale mappings,
+enabling access to old mappings and possibly
+data leakage or misrouting between devices assigned to the same guest.
+
+**Mitigation:**
+The guest must issue appropriate invalidation commands whenever
+its stage-1 I/O mappings are modified to ensure that translation caches
+remain coherent.
+
+3. Observation:
+---------------
+Introducing optional per-guest enabled features (`viommu` argument in xl
+guest config) means some guests
+may opt-out.
+
+**Risk:**
+Guests without vIOMMU enabled (stage-2 only) could potentially dominate
+access to the physical command and event queues, since they bypass the
+emulation layer and processing is faster comparing to vIOMMU-enabled guests.
+
+**Mitigation:**
+Audit the impact of emulation overhead effect on IOMMU processing fairness
+in a multi-guest environment.
+Consider enabling/disabling stage-1 on a system level, instead of per-domain.
+
+4. Observation:
+---------------
+Guests have the ability to issue Stage-1 IOMMU commands like cache
+invalidation, stream table entries
+configuration, etc. An adversarial guest may issue a high volume of
+commands in rapid succession.
+
+**Risk:**
+Excessive commands requests can cause high hypervisor CPU consumption
+and disrupt scheduling,
+leading to degraded system responsiveness and potential
+denial-of-service scenarios.
+
+**Mitigation:**
+
+- Implement vIOMMU commands execution restart and continuation support:
+
+ - Introduce processing budget with only a limited amount of commands
+ handled per invocation.
+ - If additional commands remain pending after the budget is exhausted,
+ defer further processing and resume it asynchronously, e.g. via a
+ per-domain tasklet.
+
+- Batch multiple commands of same type to reduce emulation overhead:
+
+ - Inspect the command queue and group commands that can be processed
+ together (e.g. multiple successive invalidation requests or STE
+ updates for the same SID).
+ - Execute the entire batch in one go, reducing repeated accesses to
+ guest memory and emulation overhead per command.
+ - This reduces CPU time spent in the vIOMMU command processing loop.
+ The optimization is applicable only when consecutive commands of the
+ same type operate on the same SID/context.
+
+5. Observation:
+---------------
+Some guest commands issued towards vIOMMU are propagated to pIOMMU
+command queue (e.g. TLB invalidate).
+
+**Risk:**
+Excessive commands requests from abusive guest can cause flooding of
+physical IOMMU command queue,
+leading to degraded pIOMMU responsiveness on commands issued from other
+guests.
+
+**Mitigation:**
+
+- Batch commands that are propagated to the pIOMMU command queue and
+ implement batch execution pause/continuation.
+ Rely on the same mechanisms as in the previous observation
+ (command continuation and batching of pIOMMU-related commands of the same
+ type and context).
+- If possible, implement domain penalization by adding a per-domain budget
+ for vIOMMU/pIOMMU usage:
+
+ - Apply per-domain dynamic budgeting of allowed IOMMU commands to
+ execute per invocation, reducing the budget for guests with
+ excessive command requests over a longer period of time
+ - Combine with command continuation mechanism
+
+6. Observation:
+---------------
+The vIOMMU feature includes an event queue used to forward IOMMU events
+to the guest (e.g. translation faults, invalid Stream IDs, permission errors).
+A malicious guest may misconfigure its IOMMU state or intentionally trigger
+faults at a high rate.
+
+**Risk:**
+Occurrence of IOMMU events with high frequency can cause Xen to flood the
+event queue and disrupt scheduling with
+high hypervisor CPU load for events handling.
+
+**Mitigation:**
+
+- Implement fail-safe state by disabling events forwarding when faults
+ are occurred with high frequency and
+ not processed by guest:
+
+ - Introduce a per-domain pending event counter.
+ - Stop forwarding events to the guest once the number of unprocessed
+ events reaches a predefined threshold.
+
+- Consider disabling the emulated event queue for untrusted guests.
+- Note that this risk is more general and may also apply to stage-2-only
+ guests. This section addresses mitigations in the emulated IOMMU layer
+ only. Mitigation of physical event queue flooding should also be considered
+ in the target pIOMMU driver.
+
+7. Observation:
+---------------
+vIOMMU `CFGI_STE` commands can trigger serialized STE programming on the pIOMMU
+while `xen_domain->lock`, `devices_lock` and `cmd_queue_lock` are held, with
+local interrupts disabled. STE programming may issue multiple polling CMD_SYNC
+operations, each with a one-second timeout.
+
+**Risk:**
+An abusive guest can issue CFGI_STE commands that cause multiple CMD_SYNC
+operations to reach their polling timeout. This can keep a pCPU with interrupts
+disabled for a significant period of time, delaying interrupt handling and
+potentially degrading the responsiveness of Xen and other domains running on
+the same pCPU.
+
+**Mitigation:**
+
+- Implement pinning or otherwise maintaining the lifetime of the required
+ objects before releasing the relevant locks, allowing the
+ serialized STE programming to be performed without holding locks and with
+ local interrupts enabled.
+- Consider redesigning the SMMUv3 command processing and locking model in
+ future improvements to avoid performing potentially long-running pIOMMU
+ operations while holding locks and with local interrupts disabled.
+
+Performance Impact
+==================
+
+With iommu stage-1 and nested translation inclusion, performance
+overhead is introduced comparing to existing,
+stage-2 only usage in Xen. Once mappings are established, translations
+should not introduce significant overhead.
+Emulated paths may introduce moderate overhead, primarily affecting
+device initialization and event/command handling.
+Testing is performed on Renesas R-Car platform.
+Performance is mostly impacted by emulated vIOMMU operations, results
+shown in the following table.
+
++-------------------------------+---------------------------------+
+| vIOMMU Operation | Execution time in guest |
++===============================+=================================+
+| Reg read | median: 645ns, worst-case: 2us |
++-------------------------------+---------------------------------+
+| Reg write | median: 630ns, worst-case: 1us |
++-------------------------------+---------------------------------+
+| Invalidate TLB | median: 2us, worst-case: 10us |
++-------------------------------+---------------------------------+
+| Invalidate STE | median: 5us worst_case: 100us |
++-------------------------------+---------------------------------+
+
+With vIOMMU exposed to guest, guest OS has to initialize IOMMU device
+and configure stage-1 mappings for the devices
+attached to it.
+Following table shows initialization stages which impact stage-1 enabled
+guest boot time and compares it with
+stage-1 disabled guest.
+
+NOTE: Device probe execution time varies depending on device complexity.
+A USB host controller was selected as the test device in this case.
+
++---------------------+-----------------------+------------------------+
+| Stage | Stage-1 Enabled Guest | Stage-1 Disabled Guest |
++=====================+=======================+========================+
+| IOMMU Init | ~10ms | / |
++---------------------+-----------------------+------------------------+
+| Dev Attach / Mapping| ~100ms | ~90ms |
++---------------------+-----------------------+------------------------+
+
+For devices configured with dynamic DMA mappings, DMA allocate/map/unmap
+operations performance is
+also impacted on stage-1 enabled guests.
+Dynamic DMA mapping operation trigger emulated IOMMU functions like mmio
+write/read and TLB invalidations.
+
++---------------+---------------------------+--------------------------+
+| DMA Op | Stage-1 Enabled Guest | Stage-1 Disabled Guest |
++===============+===========================+==========================+
+| dma_alloc | median: 20us, worst: 5ms | median: 8us, worst: 60us |
++---------------+---------------------------+--------------------------+
+| dma_free | median: 500us, worst: 10ms| median: 6us, worst: 30us |
++---------------+---------------------------+--------------------------+
+| dma_map | median: 12us, worst: 60us | median: 3us, worst: 20us |
++---------------+---------------------------+--------------------------+
+| dma_unmap | median: 400us, worst: 5ms | median: 5us, worst: 20us |
++---------------+---------------------------+--------------------------+
+
+Testing
+=======
+
+- QEMU-based ARM system tests for Stage-1 translation.
+- Actual hardware validation to ensure compatibility with real SMMUv3
+ implementations.
+- Unit/Functional tests validating correct translations (not implemented).
+
+Migration and Compatibility
+===========================
+
+This optional feature defaults to disabled (CONFIG_ARM_VIRTUAL_IOMMU=n,
+CONFIG_VIRTUAL_ARM_SMMU_V3=n).
+
+Future improvements
+===================
+
+- Implement the proposed mitigations to address security risks that are
+ not covered by the current design
+ (events batching, commands execution continuation)
+- PCI support
+- Support for other IOMMU HW (Renesas, RISC-V, etc.)
+
+References
+==========
+
+- Original feature implemented by Rahul Singh:
+
+https://patchwork.kernel.org/project/xen-devel/cover/cover.1669888522.git.rahul.singh@arm.com/
+
+- SMMUv3 architecture documentation
+- Existing vIOMMU code patterns (KVM, QEMU)
\ No newline at end of file
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v4 22/22] doc/arm: vIOMMU design document
2026-10-07 0:08 ` [PATCH v4 22/22] doc/arm: vIOMMU design document Milan Djokic
@ 2026-10-09 15:40 ` Dmytro Prokopchuk1
2026-10-10 15:16 ` Oleksandr Tyshchenko
1 sibling, 0 replies; 37+ messages in thread
From: Dmytro Prokopchuk1 @ 2026-10-09 15:40 UTC (permalink / raw)
To: xen-devel@lists.xenproject.org
Hi Milan,
On 10/7/26 03:08, Milan Djokic wrote:
> This document outlines the design of the emulated IOMMU,
> including supported features, security considerations
> and future improvements.
>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> docs/designs/arm-viommu.rst | 502 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 502 insertions(+)
> create mode 100644 docs/designs/arm-viommu.rst
>
This file has many trailing whitespaces. Please, remove them.
E.g.:
docs/designs/arm-viommu.rst:12: trailing whitespace.
+The SMMUv3 supports two stages of translation. Each stage of translation
docs/designs/arm-viommu.rst:14: trailing whitespace.
+independently enabled. An incoming address is logically translated from
docs/designs/arm-viommu.rst:17: trailing whitespace.
+the output PA. Stage 1 translation support is required to provide
docs/designs/arm-viommu.rst:21: trailing whitespace.
BR, Dmytro.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v4 22/22] doc/arm: vIOMMU design document
2026-10-07 0:08 ` [PATCH v4 22/22] doc/arm: vIOMMU design document Milan Djokic
2026-10-09 15:40 ` Dmytro Prokopchuk1
@ 2026-10-10 15:16 ` Oleksandr Tyshchenko
1 sibling, 0 replies; 37+ messages in thread
From: Oleksandr Tyshchenko @ 2026-10-10 15:16 UTC (permalink / raw)
To: Milan Djokic, xen-devel@lists.xenproject.org
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné, Stefano Stabellini
On 10/7/26 03:08, Milan Djokic wrote:
Hello Milan
> This document outlines the design of the emulated IOMMU,
> including supported features, security considerations
> and future improvements.
>
> Signed-off-by: Milan Djokic <milan_djokic@epam.com>
> ---
> docs/designs/arm-viommu.rst | 502 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 502 insertions(+)
> create mode 100644 docs/designs/arm-viommu.rst
>
> diff --git a/docs/designs/arm-viommu.rst b/docs/designs/arm-viommu.rst
> new file mode 100644
> index 0000000000..c68e771711
> --- /dev/null
> +++ b/docs/designs/arm-viommu.rst
> @@ -0,0 +1,502 @@
> +==========================================================
> +Design Proposal: Add SMMUv3 Stage-1 Support for XEN Guests
> +==========================================================
> +
> +:Author: Milan Djokic <milan_djokic@epam.com>
> +:Date: 2026-02-13
> +:Status: Draft
> +
> +Introduction
> +============
> +
> +The SMMUv3 supports two stages of translation. Each stage of translation
> +can be
> +independently enabled. An incoming address is logically translated from
> +VA to
> +IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
> +the output PA. Stage 1 translation support is required to provide
> +isolation between different
> +devices within OS. XEN already supports Stage 2 translation but there is no
> +support for Stage 1 translation.
> +This design proposal outlines the introduction of Stage-1 SMMUv3 support
> +in Xen for ARM guests.
Look like this doc does not mention the stage-1-only fallback.
> +
> +Motivation
> +==========
> +
> +ARM systems utilizing SMMUv3 require stage-1 address translation to
> +ensure secure DMA and
> +guest managed I/O memory mappings.
> +With stage-1 enabled, guest manages IOVA to IPA mappings through its own
> +IOMMU driver.
> +
> +This feature enables:
> +
> +- Stage-1 translation for the guest domain
> +- Device passthrough with per-device I/O address space
> +
> +Design Overview
> +===============
> +
> +These changes provide emulated SMMUv3 support:
> +
> +- **SMMUv3 Stage-1 Translation**: stage-1 and nested translation support
> + in SMMUv3 driver.
> +- **vIOMMU Abstraction**: Virtual IOMMU framework for guest stage-1
> + handling.
> +- **Register/Command Emulation**: SMMUv3 register emulation and command
> + queue handling.
> +- **Device Tree Extensions**: Adds `iommus` and virtual SMMUv3 nodes to
> + device trees for dom0 and dom0less scenarios.
> +- **Build Configuration**: Introduces a Kconfig arguments
> + (CONFIG_ARM_VIRTUAL_IOMMU, CONFIG_VIRTUAL_ARM_SMMU_V3) for build-time
> + enablement.
> +
> +Depending on the domain type, the emulated IOMMU model is the following:
> +
> +- For the hardware domain, the host layout is re-used. The number of exposed
> + IOMMUs to hardware domain is equal to number of physical IOMMUs
> + present (1-1 mapping).
> +- For non-hardware domain, a single vIOMMU device is exposed to the guest and
> + mapped to one or more physical IOMMUs through a Xen-managed translation layer
> +
> +The vIOMMU feature provides a generic framework together with a backend
> +implementation specific to the target IOMMU type. The backend is responsible
> +for implementing the hardware-specific data structures and command handling
> +logic (currently only SMMUv3 is supported).
> +
> +This modular design allows the stage-1 support to be reused
> +for other IOMMU architectures in the future.
> +
> +vIOMMU architecture
> +===================
> +
> +Responsibilities:
> +
> +Guest:
> + - Configures stage-1 via vIOMMU commands.
> + - Handles stage-1 faults received from Xen.
> +
> +Xen:
> + - Emulates the IOMMU interface (registers, commands, events).
> + - Provides vSID->pSID mappings.
> + - Programs stage-1/stage-2 configuration in the physical IOMMU.
> + - Propagate stage-1 faults to guest.
> +
> +The stage-1 context is shared between streams assigned to the same device.
> +This follows the Linux SMMUv3 driver model, where the IOMMU domain is associated
> +with the master/device, while the individual SIDs are represented by separate
> +STEs pointing to the same translation context. This is considered sufficient for
> +the current SMMUv3 driver model, although the SMMUv3 architecture does not
> +forbid having separate translation contexts for different SIDs.
> +
> +vIOMMU commands and faults are transmitted between guest and Xen via
> +command and event queues (one command/event queue created per guest).
> +
> +vIOMMU command Flow:
> +
> +::
> +
> + Guest:
> + smmu_cmd(vSID, IOVA -> IPA)
> +
> + Xen:
> + trap MMIO read/write
> + translate vSID->pSID
> + store stage-1 state
> + program pIOMMU for (pSID, IPA -> PA)
> +
> +All hardware programming of the physical IOMMU is performed exclusively by Xen.
> +
> +vIOMMU Stage-1 fault handling flow:
> +
> +::
> +
> + Xen:
> + receives stage-1 fault
> + triggers vIOMMU callback
> + injects virtual fault
> +
> + Guest:
> + receives and handles fault
> +
> +Register access
> +---------------
> +
> +Guest accesses to SMMUV3 registers are trapped via MMIO regions. vIOMMU emulates
> +a subset of SMMUv3 registers relevant to currently supported features.
> +Unsupported registers are handled as RAZ (Read-As-Zero) and WI (Write-Ignored)
> +for the register read and write operations, respectively.
> +
> +vSMMUv3 is a partially emulated resource for the guest domain. Some
> +feature register bits (SMMU_IDR) are reflecting what the underlying HW actually
> +support while the rest is preset in the emulation layer to limit guest from
> +using features which are not supported by the emulation. Also, implementer
> +identification register is fully emulated, providing generic ARM implementer
> +ID values, to avoid activating errata/quirk paths in the guest SMMUv3 driver.
> +
> +Supported SMMUv3 Commands
> +-------------------------
> +
> +The vIOMMU implementation currently supports a subset of the SMMUv3
> +architectural command set required for stage-1 translation management.
> +
> +Commands are submitted by the guest through the virtual Command Queue
> +(CMDQ) and consumed by Xen, which translates them into operations on the
> +physical SMMUv3 hardware.
> +
> +Some commands are fully handled by the emulation layer, while others are
> +currently treated as architectural no-ops because the required behavior
> +is either implicitly guaranteed by Xen or not yet modeled explicitly.
> +
> +Emulation handled commands:
> +
> +- CMD_CFGI_STE
> +- CMD_TLBI_NH_ASID
> +- CMD_TLBI_NSNH_ALL
> +- CMD_TLBI_NH_VA
> +
> +No-op/Implicitly handled commands:
> +
> +- CMD_SYNC
> +- CMD_PREFETCH_CFG
> +- CMD_CFGI_CD
> +- CMD_CFGI_CD_ALL
> +- CMD_CFGI_ALL
> +
> +Command Handling Model
> +----------------------
> +
> +The guest submits commands through the virtual CMDQ interface.
> +
> +Xen:
> +
> +- Validates commands
> +- Translates virtual identifiers (vSID->pSID) where required
> +- Programs the physical SMMU
> +- Performs synchronization
> +
> +The physical SMMU executes the resulting invalidation or configuration
> +operations.
> +
> +Unsupported Commands
> +--------------------
> +
> +Commands associated with currently unsupported architectural features
> +are not implemented.
> +
> +This includes commands related to:
> +
> +- PRI
> +- ATS
> +- Address/Range TLB invalidation (currently, all TLBI commands perform complete
> + domain TLB invalidation)
> +- Advanced stall/retry semantics
> +- CD runtime update/sync
> +
> +vSID Mapping Layer
> +------------------
> +
> +Each non-hardware domain-visible Stream ID (vSID) is mapped by Xen to a physical
> +Stream ID (pSID). The mapping is maintained per-domain. The allocation policy
> +guarantees vSID uniqueness within a domain while allowing reuse of pSIDs for
> +different pIOMMUs.
> +
> +* Platform devices receive individually allocated vSIDs.
> +* PCI devices receive a contiguous vSID range derived from RID space.
> +
> +
> +Supported Device Model
> +======================
> +
> +Currently, the vIOMMU framework supports only devices described via the
> +Device Tree (DT) model. This includes platform devices and basic PCI
> +devices support instantiated through the vPCI DT node. ACPI-described
> +devices are not supported.
> +
> +Guest assigned platform devices are mapped via `iommus` property:
> +
> +::
> +
> + <&pIOMMU pSID> -> <&vIOMMU vSID>
> +
> +PCI devices use RID-based mapping via the root complex `iommu-map`:
I failed to find iommu-map generation in libxl or dom0less.
> +
> +::
> +
> + <RID-base &viommu vSID-base length>
> +
> +PCI Topology Assumptions and Constraints:
> +
> +- RID space must be contiguous
> +- Pre-defined continuous pSID space (0-0x1000)
> +- No runtime PCI reconfiguration
> +- Single root complex assumed
> +- Mapping is fixed at guest DT construction
> +
> +Constraints for PCI devices will be addressed as part of the future work on
> +this feature.
> +
> +Security Considerations
> +=======================
> +
> +Stage-1 translation provides isolation between guest devices by
> +enforcing a per-device I/O address space, preventing unauthorized DMA.
> +With the introduction of emulated IOMMU, additional protection
> +mechanisms are required to minimize security risks.
> +
> +1. Observation:
> +---------------
> +Support for Stage-1 translation in SMMUv3 introduces new data structures
> +(`s1_cfg` alongside `s2_cfg`)
> +and logic to write both Stage-1 and Stage-2 entries in the Stream Table
> +Entry (STE), including an `abort`
> +field to handle partial configuration states.
> +
> +**Risk:**
> +Without proper handling, a partially applied configuration
> +might leave guest DMA mappings in an inconsistent state, potentially
> +enabling unauthorized access or causing cross-domain interference.
> +
> +**Mitigation:** *(Handled by design)*
> +This feature introduces logic that writes both `s1_cfg` and `s2_cfg` to
> +STE and manages the `abort` field - only considering
> +configuration if fully attached. This ensures incomplete or invalid
> +device configurations are safely ignored by the hypervisor.
Please see a potential bypass issue in commit #11/22
> +
> +2. Observation:
> +---------------
> +Guests can now invalidate Stage-1 caches; invalidation needs forwarding
> +to SMMUv3 hardware to maintain coherence.
> +
> +**Risk:**
> +Failing to propagate cache invalidation could allow stale mappings,
> +enabling access to old mappings and possibly
> +data leakage or misrouting between devices assigned to the same guest.
> +
> +**Mitigation:**
> +The guest must issue appropriate invalidation commands whenever
> +its stage-1 I/O mappings are modified to ensure that translation caches
> +remain coherent.
> +
> +3. Observation:
> +---------------
> +Introducing optional per-guest enabled features (`viommu` argument in xl
> +guest config) means some guests
> +may opt-out.
> +
> +**Risk:**
> +Guests without vIOMMU enabled (stage-2 only) could potentially dominate
> +access to the physical command and event queues, since they bypass the
> +emulation layer and processing is faster comparing to vIOMMU-enabled guests.
> +
> +**Mitigation:**
> +Audit the impact of emulation overhead effect on IOMMU processing fairness
> +in a multi-guest environment.
> +Consider enabling/disabling stage-1 on a system level, instead of per-domain.
> +
> +4. Observation:
> +---------------
> +Guests have the ability to issue Stage-1 IOMMU commands like cache
> +invalidation, stream table entries
> +configuration, etc. An adversarial guest may issue a high volume of
> +commands in rapid succession.
> +
> +**Risk:**
> +Excessive commands requests can cause high hypervisor CPU consumption
> +and disrupt scheduling,
> +leading to degraded system responsiveness and potential
> +denial-of-service scenarios.
> +
> +**Mitigation:**
> +
> +- Implement vIOMMU commands execution restart and continuation support:
> +
> + - Introduce processing budget with only a limited amount of commands
> + handled per invocation.
> + - If additional commands remain pending after the budget is exhausted,
> + defer further processing and resume it asynchronously, e.g. via a
> + per-domain tasklet.
> +
> +- Batch multiple commands of same type to reduce emulation overhead:
> +
> + - Inspect the command queue and group commands that can be processed
> + together (e.g. multiple successive invalidation requests or STE
> + updates for the same SID).
> + - Execute the entire batch in one go, reducing repeated accesses to
> + guest memory and emulation overhead per command.
> + - This reduces CPU time spent in the vIOMMU command processing loop.
> + The optimization is applicable only when consecutive commands of the
> + same type operate on the same SID/context.
> +
> +5. Observation:
> +---------------
> +Some guest commands issued towards vIOMMU are propagated to pIOMMU
> +command queue (e.g. TLB invalidate).
> +
> +**Risk:**
> +Excessive commands requests from abusive guest can cause flooding of
> +physical IOMMU command queue,
> +leading to degraded pIOMMU responsiveness on commands issued from other
> +guests.
> +
> +**Mitigation:**
> +
> +- Batch commands that are propagated to the pIOMMU command queue and
> + implement batch execution pause/continuation.
> + Rely on the same mechanisms as in the previous observation
> + (command continuation and batching of pIOMMU-related commands of the same
> + type and context).
> +- If possible, implement domain penalization by adding a per-domain budget
> + for vIOMMU/pIOMMU usage:
> +
> + - Apply per-domain dynamic budgeting of allowed IOMMU commands to
> + execute per invocation, reducing the budget for guests with
> + excessive command requests over a longer period of time
> + - Combine with command continuation mechanism
> +
> +6. Observation:
> +---------------
> +The vIOMMU feature includes an event queue used to forward IOMMU events
> +to the guest (e.g. translation faults, invalid Stream IDs, permission errors).
> +A malicious guest may misconfigure its IOMMU state or intentionally trigger
> +faults at a high rate.
> +
> +**Risk:**
> +Occurrence of IOMMU events with high frequency can cause Xen to flood the
> +event queue and disrupt scheduling with
> +high hypervisor CPU load for events handling.
> +
> +**Mitigation:**
> +
> +- Implement fail-safe state by disabling events forwarding when faults
> + are occurred with high frequency and
> + not processed by guest:
> +
> + - Introduce a per-domain pending event counter.
> + - Stop forwarding events to the guest once the number of unprocessed
> + events reaches a predefined threshold.
> +
> +- Consider disabling the emulated event queue for untrusted guests.
> +- Note that this risk is more general and may also apply to stage-2-only
> + guests. This section addresses mitigations in the emulated IOMMU layer
> + only. Mitigation of physical event queue flooding should also be considered
> + in the target pIOMMU driver.
> +
> +7. Observation:
> +---------------
> +vIOMMU `CFGI_STE` commands can trigger serialized STE programming on the pIOMMU
> +while `xen_domain->lock`, `devices_lock` and `cmd_queue_lock` are held, with
> +local interrupts disabled. STE programming may issue multiple polling CMD_SYNC
> +operations, each with a one-second timeout.
> +
> +**Risk:**
> +An abusive guest can issue CFGI_STE commands that cause multiple CMD_SYNC
> +operations to reach their polling timeout. This can keep a pCPU with interrupts
> +disabled for a significant period of time, delaying interrupt handling and
> +potentially degrading the responsiveness of Xen and other domains running on
> +the same pCPU.
> +
> +**Mitigation:**
> +
> +- Implement pinning or otherwise maintaining the lifetime of the required
> + objects before releasing the relevant locks, allowing the
> + serialized STE programming to be performed without holding locks and with
> + local interrupts enabled.
> +- Consider redesigning the SMMUv3 command processing and locking model in
> + future improvements to avoid performing potentially long-running pIOMMU
> + operations while holding locks and with local interrupts disabled.
> +
> +Performance Impact
> +==================
> +
> +With iommu stage-1 and nested translation inclusion, performance
> +overhead is introduced comparing to existing,
> +stage-2 only usage in Xen. Once mappings are established, translations
> +should not introduce significant overhead.
> +Emulated paths may introduce moderate overhead, primarily affecting
> +device initialization and event/command handling.
> +Testing is performed on Renesas R-Car platform.
> +Performance is mostly impacted by emulated vIOMMU operations, results
> +shown in the following table.
> +
> ++-------------------------------+---------------------------------+
> +| vIOMMU Operation | Execution time in guest |
> ++===============================+=================================+
> +| Reg read | median: 645ns, worst-case: 2us |
> ++-------------------------------+---------------------------------+
> +| Reg write | median: 630ns, worst-case: 1us |
> ++-------------------------------+---------------------------------+
> +| Invalidate TLB | median: 2us, worst-case: 10us |
> ++-------------------------------+---------------------------------+
> +| Invalidate STE | median: 5us worst_case: 100us |
> ++-------------------------------+---------------------------------+
> +
> +With vIOMMU exposed to guest, guest OS has to initialize IOMMU device
> +and configure stage-1 mappings for the devices
> +attached to it.
> +Following table shows initialization stages which impact stage-1 enabled
> +guest boot time and compares it with
> +stage-1 disabled guest.
> +
> +NOTE: Device probe execution time varies depending on device complexity.
> +A USB host controller was selected as the test device in this case.
> +
> ++---------------------+-----------------------+------------------------+
> +| Stage | Stage-1 Enabled Guest | Stage-1 Disabled Guest |
> ++=====================+=======================+========================+
> +| IOMMU Init | ~10ms | / |
> ++---------------------+-----------------------+------------------------+
> +| Dev Attach / Mapping| ~100ms | ~90ms |
> ++---------------------+-----------------------+------------------------+
> +
> +For devices configured with dynamic DMA mappings, DMA allocate/map/unmap
> +operations performance is
> +also impacted on stage-1 enabled guests.
> +Dynamic DMA mapping operation trigger emulated IOMMU functions like mmio
> +write/read and TLB invalidations.
> +
> ++---------------+---------------------------+--------------------------+
> +| DMA Op | Stage-1 Enabled Guest | Stage-1 Disabled Guest |
> ++===============+===========================+==========================+
> +| dma_alloc | median: 20us, worst: 5ms | median: 8us, worst: 60us |
> ++---------------+---------------------------+--------------------------+
> +| dma_free | median: 500us, worst: 10ms| median: 6us, worst: 30us |
> ++---------------+---------------------------+--------------------------+
> +| dma_map | median: 12us, worst: 60us | median: 3us, worst: 20us |
> ++---------------+---------------------------+--------------------------+
> +| dma_unmap | median: 400us, worst: 5ms | median: 5us, worst: 20us |
> ++---------------+---------------------------+--------------------------+
> +
> +Testing
> +=======
> +
> +- QEMU-based ARM system tests for Stage-1 translation.
> +- Actual hardware validation to ensure compatibility with real SMMUv3
> + implementations.
> +- Unit/Functional tests validating correct translations (not implemented).
> +
> +Migration and Compatibility
> +===========================
> +
> +This optional feature defaults to disabled (CONFIG_ARM_VIRTUAL_IOMMU=n,
> +CONFIG_VIRTUAL_ARM_SMMU_V3=n).
> +
> +Future improvements
> +===================
> +
> +- Implement the proposed mitigations to address security risks that are
> + not covered by the current design
> + (events batching, commands execution continuation)
> +- PCI support
> +- Support for other IOMMU HW (Renesas, RISC-V, etc.)
> +
> +References
> +==========
> +
> +- Original feature implemented by Rahul Singh:
> +
> +https://patchwork.kernel.org/project/xen-devel/cover/cover.1669888522.git.rahul.singh@arm.com/
> +
> +- SMMUv3 architecture documentation
> +- Existing vIOMMU code patterns (KVM, QEMU)
> \ No newline at end of file
^ permalink raw reply [flat|nested] 37+ messages in thread