[PATCH v1 0/9] iommu/arm-smmu-v3: Share domain across SMMU/vSMMU instances

[PATCH v1 0/9] iommu/arm-smmu-v3: Share domain across SMMU/vSMMU instances

[Nicolin Chen]

In a system with multiple physical SMMU instances, multiple devices can be
passed through to a VM. Currently, a VM would allocate one domain per SMMU
instance that might be shared across devices that sit behind the same SMMU
instance. However, the gPA->PA mappings (either an S1 unmanaged domain or
an S2 nesting parent domain) can be shared across all the devices that sit
behind different SMMU instances as well, provided that the shared I/O page
table is compatible with all the SMMU instances.

The major difficulty in sharing the domain has been invalidation, since a
change to the shared I/O page table results in an invalidation on all SMMU
instances. A traditional approach involves building a linked list of SMMUs
within the domain, which is very inefficient for the invalidation path as
the linked list has to be locked.

To address this, the SMMUv3 driver now uses an RCU-protected invalidation
array. Any new device (and its SMMU) is preloaded into the array during a
device attachment. This array maintains all necessary information, such as
ASID/VMID and which SMMU instance (CMDQ) to issue the command to.

The second issue concerns the lifecycle of the iotlb tag. Currently, ASID
or VMID is allocated per domain and kept in the domain structure (cd->asid
or s2_cfg->vmid). This does not work ideally when the domain (e.g. S2) is
shared, as the VMID will have to be global across all SMMU instances, even
if a VM is not using all of them. This results in wasted VMID resources in
the bitmaps of unused SMMU instances.

Instead, an iotlb tag should be allocated per SMMU instance. Consequently,
these tags must be allocated and maintained separately. Since ASID or VMID
is only used when a CD or STE is installed to the HW (which happens during
device attachment), and the invalidation array is built right before that,
arm_smmu_invs_merge() is the ideal place to allocate a new iotlb tag:
 - when a device attaches, the driver first searches for an existing iotlb
   tag for the SMMU the device sits behind
 - If a match is found, the "users" counter is incremented
 - otherwise, a new tag is allocated.

As ASID/VMID are programmed to CD/STE that belong to a device, it's natural
to store the ASID/VMID in the master structure.

Given the above, this series reworks the driver further:
 - Add ASID/VMID allocation/free ops to the arm_smmu_invs data structure
 - Store the allocated ASID/VMID in the arm_smmu_master structure
 - Replace cd->asid and s2_cfg->vmid with the tags stored in the master
   structure, when installing them to the CD and STE
 - Deprecate cd->asid and s2_cfg->vmid.

Finally, allow sharing a domain across the SMMU instances, so long as they
passes a compatibility test.

This is on Github:
https://github.com/nicolinc/iommufd/commits/smmuv3_share_domain-v1

This is based on the series "Introduce an RCU-protected invalidation array"
https://lore.kernel.org/all/cover.1766013662.git.nicolinc@nvidia.com/
So the whole implementation follows the path Jason envisioned initially.

A earlier effort to share S2 domain can be found:
https://lore.kernel.org/all/cover.1744692494.git.nicolinc@nvidia.com/

Thanks
Nicolin

Nicolin Chen (9):
  iommu/arm-smmu-v3: Pass in ssid to arm_smmu_make_s1_cd()
  iommu/arm-smmu-v3: Add alloc_id/free_id functions to arm_smmu_invs
  iommu/arm-smmu-v3: Store ASIDs and VMID in arm_smmu_master
  iommu/arm-smmu-v3: Use alloc_id/free_id ops in
    arm_smmu_invs_merge/unref
  iommu/arm-smmu-v3: Install to CD/STE the ASID/VMID stored in the
    master
  iommu/arm-smmu-v3: Use dummy ASID/VMID in arm_smmu_master_build_invs()
  iommu/arm-smmu-v3: Remove free_fn argument from arm_smmu_invs_unref()
  iommu/arm-smmu-v3: Remove ASID/VMID from arm_smmu_domain
  iommu/arm-smmu-v3: Allow sharing domain across SMMUs

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  47 ++-
 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     |  17 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |  27 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  |  24 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 307 +++++++++++-------
 5 files changed, 265 insertions(+), 157 deletions(-)

-- 
2.43.0

[PATCH v1 1/9] iommu/arm-smmu-v3: Pass in ssid to arm_smmu_make_s1_cd()

[Nicolin Chen]

An S1 domain that holds the mappings of guest VM's RAM space can be shared
across the passthrough devices, as long as the S1 page table is compatible
with all the SMMU instances that the devices physically sit behind.

On the other hand, ASID is per CD, which is further per STE (i.e. device).
Thus, it should be decoupled from a domain structure and ideally stored in
the master structure instead.

There will be an ASID array stored in the arm_smmu_master structure, so it
needs an SSID/PASID to index a specific ASID to program the CD. To prepare
for that, pass in an SSID/PASID to arm_smmu_make_s1_cd() from its callers.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h      | 2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c  | 3 ++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c      | 7 ++++---
 4 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 4f104c1baa67..0a5aead300b6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -1070,7 +1070,7 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master,
 					u32 ssid);
 void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
 			 struct arm_smmu_master *master,
-			 struct arm_smmu_domain *smmu_domain);
+			 struct arm_smmu_domain *smmu_domain, ioasid_t ssid);
 void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
 			     struct arm_smmu_cd *cdptr,
 			     const struct arm_smmu_cd *target);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index f1f8e01a7e91..adf802f165d1 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -29,7 +29,8 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain)
 		if (WARN_ON(!cdptr))
 			continue;
 
-		arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+		arm_smmu_make_s1_cd(&target_cd, master, smmu_domain,
+				    master_domain->ssid);
 		arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
 					&target_cd);
 	}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
index 238bfd328b5b..e4bdb4cfdacd 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
@@ -471,7 +471,7 @@ static void arm_smmu_test_make_s1_cd(struct arm_smmu_cd *cd, unsigned int asid)
 	io_pgtable.cfg.arm_lpae_s1_cfg.tcr.tsz = 4;
 	io_pgtable.cfg.arm_lpae_s1_cfg.mair = 0xabcdef012345678ULL;
 
-	arm_smmu_make_s1_cd(cd, &master, &smmu_domain);
+	arm_smmu_make_s1_cd(cd, &master, &smmu_domain, IOMMU_NO_PASID);
 }
 
 static void arm_smmu_v3_write_cd_test_s1_clear(struct kunit *test)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index d7c492ee0936..bf0df16cec45 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1613,7 +1613,7 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid,
 
 void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
 			 struct arm_smmu_master *master,
-			 struct arm_smmu_domain *smmu_domain)
+			 struct arm_smmu_domain *smmu_domain, ioasid_t ssid)
 {
 	struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
 	const struct io_pgtable_cfg *pgtbl_cfg =
@@ -3636,7 +3636,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev,
 	case ARM_SMMU_DOMAIN_S1: {
 		struct arm_smmu_cd target_cd;
 
-		arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+		arm_smmu_make_s1_cd(&target_cd, master, smmu_domain,
+				    IOMMU_NO_PASID);
 		arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr,
 					&target_cd);
 		arm_smmu_make_cdtable_ste(&target, master, state.ats_enabled,
@@ -3679,7 +3680,7 @@ static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
 	 * We can read cd.asid outside the lock because arm_smmu_set_pasid()
 	 * will fix it
 	 */
-	arm_smmu_make_s1_cd(&target_cd, master, smmu_domain);
+	arm_smmu_make_s1_cd(&target_cd, master, smmu_domain, id);
 	return arm_smmu_set_pasid(master, to_smmu_domain(domain), id,
 				  &target_cd, old);
 }
-- 
2.43.0

[PATCH v1 2/9] iommu/arm-smmu-v3: Add alloc_id/free_id functions to arm_smmu_invs

[Nicolin Chen]

An iotlb tag (ASID/VMID) will not be used:
1) Before being installed to CD/STE during a device attachment
2) After being removed from CD/STE during a device detachment

Both (1) and (2) exactly align with the lifecyle of the domain->invs. So,
it becomes very nature to use domain->invs to allocate/free an ASID/VMID.

Add a pair of function ops in struct arm_smmu_invs, to manage iotlb tag.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  6 ++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 93 +++++++++++++++++++++
 2 files changed, 99 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 0a5aead300b6..b275673c03ce 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -691,6 +691,9 @@ static inline bool arm_smmu_inv_is_ats(const struct arm_smmu_inv *inv)
  * @rwlock: optional rwlock to fench ATS operations
  * @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FULL
  * @rcu: rcu head for kfree_rcu()
+ * @smmu_domain: owner domain of the array
+ * @alloc_id: a callback to allocate a new iotlb tag
+ * @free_id: a callback to free an iotlb tag when its user counter reaches 0
  * @inv: flexible invalidation array
  *
  * The arm_smmu_invs is an RCU data structure. During a ->attach_dev callback,
@@ -720,6 +723,9 @@ struct arm_smmu_invs {
 	rwlock_t rwlock;
 	bool has_ats;
 	struct rcu_head rcu;
+	struct arm_smmu_domain *smmu_domain;
+	int (*alloc_id)(struct arm_smmu_inv *inv, void *data);
+	void (*free_id)(struct arm_smmu_inv *inv, bool flush);
 	struct arm_smmu_inv inv[] __counted_by(max_invs);
 };
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index bf0df16cec45..8a2b7064d29b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3117,6 +3117,94 @@ static void arm_smmu_disable_iopf(struct arm_smmu_master *master,
 		iopf_queue_remove_device(master->smmu->evtq.iopf, master->dev);
 }
 
+/*
+ * When an array entry's users count reaches zero, it means the ASID/VMID is no
+ * longer being invalidated by map/unmap and must be cleaned. The rule is that
+ * all ASIDs/VMIDs not in an invalidation array are left cleared in the IOTLB.
+ */
+static void arm_smmu_inv_free_asid(struct arm_smmu_inv *inv, bool flush)
+{
+	lockdep_assert_held(&arm_smmu_asid_lock);
+
+	if (inv->type != INV_TYPE_S1_ASID)
+		return;
+	if (refcount_read(&inv->users))
+		return;
+
+	if (flush) {
+		struct arm_smmu_cmdq_ent cmd = {
+			.opcode = inv->nsize_opcode,
+			.tlbi.asid = inv->id,
+		};
+
+		arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, &cmd);
+	}
+
+	/* Lastly, free the ASID as the last user detached */
+	xa_erase(&arm_smmu_asid_xa, inv->id);
+}
+
+static void arm_smmu_inv_free_vmid(struct arm_smmu_inv *inv, bool flush)
+{
+	lockdep_assert_held(&arm_smmu_asid_lock);
+
+	/* Note S2_VMID using nsize_opcode covers S2_VMID_S1_CLEAR already */
+	if (inv->type != INV_TYPE_S2_VMID)
+		return;
+	if (refcount_read(&inv->users))
+		return;
+
+	if (flush) {
+		struct arm_smmu_cmdq_ent cmd = {
+			.opcode = inv->nsize_opcode,
+			.tlbi.vmid = inv->id,
+		};
+
+		arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, &cmd);
+	}
+
+	/* Lastly, free the VMID as the last user detached */
+	ida_free(&inv->smmu->vmid_map, inv->id);
+}
+
+static int arm_smmu_inv_alloc_asid(struct arm_smmu_inv *inv, void *data)
+{
+	struct arm_smmu_domain *smmu_domain = data;
+	struct arm_smmu_device *smmu = inv->smmu;
+	u32 asid;
+	int ret;
+
+	lockdep_assert_held(&arm_smmu_asid_lock);
+
+	/* Allocate a new iotlb_tag.id */
+	WARN_ON(inv->type != INV_TYPE_S1_ASID);
+
+	ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+		       XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
+	if (ret)
+		return ret;
+	inv->id = asid;
+	return 0;
+}
+
+static int arm_smmu_inv_alloc_vmid(struct arm_smmu_inv *inv, void *data)
+{
+	struct arm_smmu_device *smmu = inv->smmu;
+	int vmid;
+
+	lockdep_assert_held(&arm_smmu_asid_lock);
+
+	WARN_ON(inv->type != INV_TYPE_S2_VMID);
+
+	/* Reserve VMID 0 for stage-2 bypass STEs */
+	vmid = ida_alloc_range(&smmu->vmid_map, 1, (1 << smmu->vmid_bits) - 1,
+			       GFP_KERNEL);
+	if (vmid < 0)
+		return vmid;
+	inv->id = vmid;
+	return 0;
+}
+
 static struct arm_smmu_inv *
 arm_smmu_master_build_inv(struct arm_smmu_master *master,
 			  enum arm_smmu_inv_type type, u32 id, ioasid_t ssid,
@@ -3191,12 +3279,17 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
 					       smmu_domain->cd.asid,
 					       IOMMU_NO_PASID, pgsize))
 			return NULL;
+		master->build_invs->alloc_id = arm_smmu_inv_alloc_asid;
+		master->build_invs->free_id = arm_smmu_inv_free_asid;
+		master->build_invs->smmu_domain = smmu_domain;
 		break;
 	case ARM_SMMU_DOMAIN_S2:
 		if (!arm_smmu_master_build_inv(master, INV_TYPE_S2_VMID,
 					       smmu_domain->s2_cfg.vmid,
 					       IOMMU_NO_PASID, pgsize))
 			return NULL;
+		master->build_invs->alloc_id = arm_smmu_inv_alloc_vmid;
+		master->build_invs->free_id = arm_smmu_inv_free_vmid;
 		break;
 	default:
 		WARN_ON(true);
-- 
2.43.0

[PATCH v1 3/9] iommu/arm-smmu-v3: Store ASIDs and VMID in arm_smmu_master

[Nicolin Chen]

Currently, ASID is allocated per smmu_domain, stored in the domain, and
freed with the domain.

Practically, ASID is only used in a CD as an iotlb tag. Therefore, ASID
doesn't really follow the life cycle of a domain but domain attachment.

On the other hand, the CD carrying ASID is installed to a device's STE.

This applies to the VMID as well, which is installed in an STE directly.

Since a device can only have one ASID per SSID and one VMID per SID, add
an ASID array and VMID in the arm_smmu_master structure, to decouple the
ASID/VMID from the domain structure.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  4 +++
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  | 13 ++++++---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 28 +++++++++++++++++++
 3 files changed, 41 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index b275673c03ce..e21e95936b05 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -953,6 +953,8 @@ struct arm_smmu_master {
 	bool				stall_enabled;
 	unsigned int			ssid_bits;
 	unsigned int			iopf_refcount;
+	/* Store allocated ASID[1 << ssid_bits] and VMID */
+	u16				*asid, vmid;
 };
 
 /* SMMU private data for an IOMMU domain */
@@ -1117,11 +1119,13 @@ static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master)
  * @new_invs: for new domain, this is the new invs array to update domain->invs;
  *            for old domain, this is the master->build_invs to pass in as the
  *            to_unref argument to an arm_smmu_invs_unref() call
+ * @iotlb_tag: copy of the first entry in the build_invs for the domain
  */
 struct arm_smmu_inv_state {
 	struct arm_smmu_invs __rcu **invs_ptr;
 	struct arm_smmu_invs *old_invs;
 	struct arm_smmu_invs *new_invs;
+	struct arm_smmu_inv iotlb_tag;
 };
 
 struct arm_smmu_attach_state {
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
index e4bdb4cfdacd..ead0d84cc9a0 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
@@ -449,7 +449,8 @@ static void arm_smmu_v3_test_cd_expect_hitless_transition(
 					      num_syncs_expected, true);
 }
 
-static void arm_smmu_test_make_s1_cd(struct arm_smmu_cd *cd, unsigned int asid)
+static void arm_smmu_test_make_s1_cd(struct kunit *test, struct arm_smmu_cd *cd,
+				     unsigned int asid)
 {
 	struct arm_smmu_master master = {
 		.smmu = &smmu,
@@ -471,6 +472,10 @@ static void arm_smmu_test_make_s1_cd(struct arm_smmu_cd *cd, unsigned int asid)
 	io_pgtable.cfg.arm_lpae_s1_cfg.tcr.tsz = 4;
 	io_pgtable.cfg.arm_lpae_s1_cfg.mair = 0xabcdef012345678ULL;
 
+	master.asid = kunit_kzalloc(test, sizeof(*master.asid), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, master.asid);
+
+	master.asid[IOMMU_NO_PASID] = asid;
 	arm_smmu_make_s1_cd(cd, &master, &smmu_domain, IOMMU_NO_PASID);
 }
 
@@ -479,7 +484,7 @@ static void arm_smmu_v3_write_cd_test_s1_clear(struct kunit *test)
 	struct arm_smmu_cd cd = {};
 	struct arm_smmu_cd cd_2;
 
-	arm_smmu_test_make_s1_cd(&cd_2, 1997);
+	arm_smmu_test_make_s1_cd(test, &cd_2, 1997);
 	arm_smmu_v3_test_cd_expect_non_hitless_transition(
 		test, &cd, &cd_2, NUM_EXPECTED_SYNCS(2));
 	arm_smmu_v3_test_cd_expect_non_hitless_transition(
@@ -491,8 +496,8 @@ static void arm_smmu_v3_write_cd_test_s1_change_asid(struct kunit *test)
 	struct arm_smmu_cd cd = {};
 	struct arm_smmu_cd cd_2;
 
-	arm_smmu_test_make_s1_cd(&cd, 778);
-	arm_smmu_test_make_s1_cd(&cd_2, 1997);
+	arm_smmu_test_make_s1_cd(test, &cd, 778);
+	arm_smmu_test_make_s1_cd(test, &cd_2, 1997);
 	arm_smmu_v3_test_cd_expect_hitless_transition(test, &cd, &cd_2,
 						      NUM_EXPECTED_SYNCS(1));
 	arm_smmu_v3_test_cd_expect_hitless_transition(test, &cd_2, &cd,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 8a2b7064d29b..1bf7b7233109 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3391,6 +3391,7 @@ static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *state,
 			arm_smmu_invs_merge(invst->old_invs, build_invs);
 		if (IS_ERR(invst->new_invs))
 			return PTR_ERR(invst->new_invs);
+		invst->iotlb_tag = build_invs->inv[0];
 	}
 
 	if (old_smmu_domain) {
@@ -3440,6 +3441,11 @@ arm_smmu_install_new_domain_invs(struct arm_smmu_attach_state *state)
 	 */
 	smp_mb();
 	kfree_rcu(invst->old_invs, rcu);
+
+	if (invst->iotlb_tag.type == INV_TYPE_S1_ASID)
+		state->master->asid[state->ssid] = invst->iotlb_tag.id;
+	else
+		state->master->vmid = invst->iotlb_tag.id;
 }
 
 /*
@@ -3471,8 +3477,11 @@ static void arm_smmu_inv_flush_iotlb_tag(struct arm_smmu_inv *inv)
 static void
 arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state)
 {
+	struct arm_smmu_inv *new_iotlb_tag = &state->new_domain_invst.iotlb_tag;
+	struct arm_smmu_inv *old_iotlb_tag = &state->old_domain_invst.iotlb_tag;
 	struct arm_smmu_inv_state *invst = &state->old_domain_invst;
 	struct arm_smmu_invs *old_invs = invst->old_invs;
+	struct arm_smmu_master *master = state->master;
 	struct arm_smmu_invs *new_invs;
 
 	lockdep_assert_held(&arm_smmu_asid_lock);
@@ -3482,6 +3491,7 @@ arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state)
 
 	arm_smmu_invs_unref(old_invs, invst->new_invs,
 			    arm_smmu_inv_flush_iotlb_tag);
+	*old_iotlb_tag = invst->new_invs->inv[0];
 
 	new_invs = arm_smmu_invs_purge(old_invs);
 	if (!new_invs)
@@ -3506,6 +3516,14 @@ arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state)
 	 */
 	smp_mb();
 	kfree_rcu(old_invs, rcu);
+
+	/* Make sure we don't clear the stored new iotlb tag */
+	if (!new_iotlb_tag->id) {
+		if (old_iotlb_tag->type == INV_TYPE_S1_ASID)
+			cmpxchg(&master->asid[state->ssid], old_iotlb_tag->id, 0);
+		else
+			cmpxchg(&master->vmid, old_iotlb_tag->id, 0);
+	}
 }
 
 /*
@@ -4286,6 +4304,13 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
 		master->ssid_bits = min_t(u8, master->ssid_bits,
 					  CTXDESC_LINEAR_CDMAX);
 
+	master->asid = kcalloc(1 << master->ssid_bits, sizeof(*master->asid),
+			       GFP_KERNEL);
+	if (!master->asid) {
+		ret = -ENOMEM;
+		goto err_disable_pasid;
+	}
+
 	if ((smmu->features & ARM_SMMU_FEAT_STALLS &&
 	     device_property_read_bool(dev, "dma-can-stall")) ||
 	    smmu->features & ARM_SMMU_FEAT_STALL_FORCE)
@@ -4299,6 +4324,9 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
 
 	return &smmu->iommu;
 
+err_disable_pasid:
+	arm_smmu_disable_pasid(master);
+	arm_smmu_remove_master(master);
 err_free_master:
 	kfree(master);
 	return ERR_PTR(ret);
-- 
2.43.0

[PATCH v1 4/9] iommu/arm-smmu-v3: Use alloc_id/free_id ops in arm_smmu_invs_merge/unref

[Nicolin Chen]

If a domain->invs has an iotlb tag (ASID/VMID) for an SMMU, all the devices
behind the SMMU can reuse the same iotlb tag. This exactly matches with the
existing case where attaching multiple devices to the same SMMU domain has
a shared iotlb tag stored in the domain (cd->asid or s2_cfg->vmid).

If a domain->invs doesn't have an iotlb tag for another SMMU, there can be
two cases:
1) This is a new domain so not yet attached to any devices
2) This is a shareable domain that is attached to a device behind one SMMU
   but not yet to the other SMMU.

In either case, a new iotlb tag is required. Call the ->alloc_id op in the
arm_smmu_invs_merge(). The domain->invs arrary will keep it. The allocated
new iotlb tag will be returned to the caller via to_merge arrary.

Relax the arm_smmu_inv_cmp(), to allow sharing an iotlb tag across devices
behind the same SMMU instance.

Similarly, call the ->free_id op in the arm_smmu_invs_unref() when there's
no device using it any more.

Lastly, add a free helper for the revert path of arm_smmu_attach_prepare()
to use.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  3 +
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 70 +++++++++++++++++++--
 2 files changed, 69 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index e21e95936b05..230ab902a9b6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -680,6 +680,9 @@ static inline bool arm_smmu_inv_is_ats(const struct arm_smmu_inv *inv)
 	return inv->type == INV_TYPE_ATS || inv->type == INV_TYPE_ATS_FULL;
 }
 
+/* S1_ASID/S2_VMID(S1_CLEAR) types */
+#define arm_smmu_inv_is_iotlb_tag(inv) !arm_smmu_inv_is_ats(inv)
+
 /**
  * struct arm_smmu_invs - Per-domain invalidation array
  * @max_invs: maximum capacity of the flexible array
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 1bf7b7233109..ec370e54b1bc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1051,6 +1051,9 @@ static int arm_smmu_inv_cmp(const struct arm_smmu_inv *inv_l,
 		return cmp_int((uintptr_t)inv_l->smmu, (uintptr_t)inv_r->smmu);
 	if (inv_l->type != inv_r->type)
 		return cmp_int(inv_l->type, inv_r->type);
+	/* Each SMMU shares a single iotlb tag on a domain, so it is a match */
+	if (arm_smmu_inv_is_iotlb_tag(inv_l))
+		return 0;
 	return cmp_int(inv_l->id, inv_r->id);
 }
 
@@ -1127,9 +1130,52 @@ struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
 	size_t i, j;
 	int cmp;
 
-	arm_smmu_invs_for_each_cmp(invs, i, to_merge, j, cmp)
+	arm_smmu_invs_for_each_cmp(invs, i, to_merge, j, cmp) {
+		struct arm_smmu_inv *cur = &to_merge->inv[j];
+
 		num_invs++;
 
+		if (!arm_smmu_inv_is_iotlb_tag(cur))
+			continue;
+
+		/* A matching iotlb tag owned by the same SMMU can be shared */
+		if (cmp == 0) {
+			*cur = invs->inv[i];
+			continue;
+		}
+
+		/* Iterate the base invs array to find if next is a match */
+		if (cmp < 0 && i < invs->num_invs)
+			continue;
+
+		/*
+		 * Currently the @to_merge array always carries an id (> 0) that
+		 * is also installed in the CD/STE. So, we cannot allocate a new
+		 * ID at this moment, because that would misalign with what's in
+		 * the CD/STE. To not break the existing flow, bypass the new ID
+		 * allocating code. We will lift this bypass line once rework is
+		 * done.
+		 */
+		if (cur->id)
+			continue;
+
+		/* No found. Allocate a new one */
+		if (j == 0) {
+			/* KUNIT test doesn't pass in an alloc_id function */
+			if (to_merge->alloc_id) {
+				int ret;
+
+				ret = to_merge->alloc_id(cur,
+							 invs->smmu_domain);
+				if (ret)
+					return ERR_PTR(ret);
+			}
+		} else {
+			/* Copy the allocated iotlb tag from the previous inv */
+			cur->id = cur[-1].id;
+		}
+	}
+
 	new_invs = arm_smmu_invs_alloc(num_invs);
 	if (!new_invs)
 		return ERR_PTR(-ENOMEM);
@@ -1207,9 +1253,9 @@ void arm_smmu_invs_unref(struct arm_smmu_invs *invs,
 				continue;
 			}
 
-			/* KUNIT test doesn't pass in a free_fn */
-			if (free_fn)
-				free_fn(&invs->inv[i]);
+			/* KUNIT test doesn't pass in a free_id function */
+			if (to_unref->free_id)
+				to_unref->free_id(&invs->inv[i], true);
 			invs->num_trashes++;
 		} else {
 			/* item in to_unref is not in invs or already a trash */
@@ -3167,6 +3213,21 @@ static void arm_smmu_inv_free_vmid(struct arm_smmu_inv *inv, bool flush)
 	ida_free(&inv->smmu->vmid_map, inv->id);
 }
 
+static void arm_smmu_inv_free_iotlb_tag(struct arm_smmu_inv *inv)
+{
+	switch (inv->type) {
+	case INV_TYPE_S1_ASID:
+		arm_smmu_inv_free_asid(inv, false);
+		return;
+	case INV_TYPE_S2_VMID:
+		arm_smmu_inv_free_vmid(inv, false);
+		return;
+	default:
+		WARN_ON(true);
+		return;
+	}
+}
+
 static int arm_smmu_inv_alloc_asid(struct arm_smmu_inv *inv, void *data)
 {
 	struct arm_smmu_domain *smmu_domain = data;
@@ -3662,6 +3723,7 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state,
 err_free_vmaster:
 	kfree(state->vmaster);
 err_unprepare_invs:
+	arm_smmu_inv_free_iotlb_tag(&state->new_domain_invst.iotlb_tag);
 	kfree(state->new_domain_invst.new_invs);
 	return ret;
 }
-- 
2.43.0

[PATCH v1 5/9] iommu/arm-smmu-v3: Install to CD/STE the ASID/VMID stored in the master

[Nicolin Chen]

Now the iotlb tag (ASID/VMID) allocated in arm_smmu_invs_merge() is stored
in the master, with a proper release and revert pathways. Replace the old
smmu_domain->cd.asid and smmu_domain->s2_cfg.vmid with the master ones.

The old asid/vmid will be depracated.

Note that the nested pathway needs vsmmu->vmid to be updated to align with
the VMID programmed in the STE as its invalidation code will overwrite the
VMID field using vsmmu->vmid.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h       |  1 +
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c   | 15 +++++++++++++--
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |  4 ++--
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c       | 10 ++++------
 4 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 230ab902a9b6..dac412ff0d71 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -1186,6 +1186,7 @@ struct arm_vsmmu {
 	struct iommufd_viommu core;
 	struct arm_smmu_device *smmu;
 	struct arm_smmu_domain *s2_parent;
+	int nr_vmasters;
 	u16 vmid;
 };
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
index 93fdadd07431..1c877d30f86e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
@@ -136,6 +136,19 @@ void arm_smmu_attach_commit_vmaster(struct arm_smmu_attach_state *state)
 	struct arm_smmu_master *master = state->master;
 
 	mutex_lock(&master->smmu->streams_mutex);
+
+	/* Clear the old vsmmu's VMID and set the new vsmmu's VMID */
+	if (master->vmaster && master->vmaster->vsmmu) {
+		if (--master->vmaster->vsmmu->nr_vmasters == 0)
+			master->vmaster->vsmmu->vmid = 0;
+		WARN_ON(master->vmaster->vsmmu->nr_vmasters < 0);
+	}
+	if (state->vmaster && state->vmaster->vsmmu) {
+		cmpxchg(&state->vmaster->vsmmu->vmid, 0, master->vmid);
+		WARN_ON(state->vmaster->vsmmu->vmid != master->vmid);
+		state->vmaster->vsmmu->nr_vmasters++;
+	}
+
 	kfree(master->vmaster);
 	master->vmaster = state->vmaster;
 	mutex_unlock(&master->smmu->streams_mutex);
@@ -454,8 +467,6 @@ int arm_vsmmu_init(struct iommufd_viommu *viommu,
 
 	vsmmu->smmu = smmu;
 	vsmmu->s2_parent = s2_parent;
-	/* FIXME Move VMID allocation from the S2 domain allocation to here */
-	vsmmu->vmid = s2_parent->s2_cfg.vmid;
 
 	if (viommu->type == IOMMU_VIOMMU_TYPE_ARM_SMMUV3) {
 		viommu->ops = &arm_vsmmu_ops;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index adf802f165d1..0e534f2b72e0 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -164,7 +164,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
 		if (WARN_ON(!cdptr))
 			continue;
 		arm_smmu_make_sva_cd(&target, master, NULL,
-				     smmu_domain->cd.asid);
+				     master->asid[master_domain->ssid]);
 		arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr,
 					&target);
 	}
@@ -266,7 +266,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain,
 	 * This does not need the arm_smmu_asid_lock because SVA domains never
 	 * get reassigned
 	 */
-	arm_smmu_make_sva_cd(&target, master, domain->mm, smmu_domain->cd.asid);
+	arm_smmu_make_sva_cd(&target, master, domain->mm, master->asid[id]);
 	ret = arm_smmu_set_pasid(master, smmu_domain, id, &target, old);
 
 	mmput(domain->mm);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index ec370e54b1bc..d6ae630d0de3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1661,7 +1661,6 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
 			 struct arm_smmu_master *master,
 			 struct arm_smmu_domain *smmu_domain, ioasid_t ssid)
 {
-	struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
 	const struct io_pgtable_cfg *pgtbl_cfg =
 		&io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
 	typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr =
@@ -1686,7 +1685,7 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target,
 		CTXDESC_CD_0_R |
 		CTXDESC_CD_0_A |
 		CTXDESC_CD_0_ASET |
-		FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid)
+		FIELD_PREP(CTXDESC_CD_0_ASID, master->asid[ssid])
 		);
 
 	/* To enable dirty flag update, set both Access flag and dirty state update */
@@ -1945,7 +1944,6 @@ void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
 				 struct arm_smmu_domain *smmu_domain,
 				 bool ats_enabled)
 {
-	struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg;
 	const struct io_pgtable_cfg *pgtbl_cfg =
 		&io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg;
 	typeof(&pgtbl_cfg->arm_lpae_s2_cfg.vtcr) vtcr =
@@ -1976,7 +1974,7 @@ void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
 		   FIELD_PREP(STRTAB_STE_2_VTCR_S2TG, vtcr->tg) |
 		   FIELD_PREP(STRTAB_STE_2_VTCR_S2PS, vtcr->ps);
 	target->data[2] = cpu_to_le64(
-		FIELD_PREP(STRTAB_STE_2_S2VMID, s2_cfg->vmid) |
+		FIELD_PREP(STRTAB_STE_2_S2VMID, master->vmid) |
 		FIELD_PREP(STRTAB_STE_2_VTCR, vtcr_val) |
 		STRTAB_STE_2_S2AA64 |
 #ifdef __BIG_ENDIAN
@@ -3850,7 +3848,7 @@ static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain,
 		return -EINVAL;
 
 	/*
-	 * We can read cd.asid outside the lock because arm_smmu_set_pasid()
+	 * We can read master.asid outside the lock because arm_smmu_set_pasid()
 	 * will fix it
 	 */
 	arm_smmu_make_s1_cd(&target_cd, master, smmu_domain, id);
@@ -3921,7 +3919,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
 	 */
 	cd->data[0] &= ~cpu_to_le64(CTXDESC_CD_0_ASID);
 	cd->data[0] |= cpu_to_le64(
-		FIELD_PREP(CTXDESC_CD_0_ASID, smmu_domain->cd.asid));
+		FIELD_PREP(CTXDESC_CD_0_ASID, master->asid[pasid]));
 
 	arm_smmu_write_cd_entry(master, pasid, cdptr, cd);
 	arm_smmu_update_ste(master, sid_domain, state.ats_enabled);
-- 
2.43.0

[PATCH v1 6/9] iommu/arm-smmu-v3: Use dummy ASID/VMID in arm_smmu_master_build_invs()

[Nicolin Chen]

The arm_smmu_inv_cmp() function no longer enforces ASID/VMID values on an
iotlb tag entry, to allow sharing an existing tag in the domain->invs. If
no tag is found on the same SMMU, a new tag will be allocated.

Therefore, there is no point in passing in any id to the iotlb entries in
the master->build_invs array. Use a dummy ID=0 that is out of range of an
ID allocation, so as to set free cd->asid and s2_cfg->vmid. An ATS entry
must still have a specific ID.

Since CD/STE has the ASID/VMID coming from the master structure, lift the
cur->id check in arm_smmu_invs_merge(), to use the new ID allocation path.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 ++++++++-------------
 1 file changed, 11 insertions(+), 18 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index d6ae630d0de3..d51bad1002ff 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1148,17 +1148,6 @@ struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
 		if (cmp < 0 && i < invs->num_invs)
 			continue;
 
-		/*
-		 * Currently the @to_merge array always carries an id (> 0) that
-		 * is also installed in the CD/STE. So, we cannot allocate a new
-		 * ID at this moment, because that would misalign with what's in
-		 * the CD/STE. To not break the existing flow, bypass the new ID
-		 * allocating code. We will lift this bypass line once rework is
-		 * done.
-		 */
-		if (cur->id)
-			continue;
-
 		/* No found. Allocate a new one */
 		if (j == 0) {
 			/* KUNIT test doesn't pass in an alloc_id function */
@@ -3331,11 +3320,16 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
 	if (master->smmu->features & ARM_SMMU_FEAT_RANGE_INV)
 		pgsize = __ffs(smmu_domain->domain.pgsize_bitmap);
 
+	/*
+	 * Each SMMU has only one iotlb tag in the array, so leave the iotlb tag
+	 * ID to be 0. The merge() and unref() will find the existing one in the
+	 * array to refcount_inc/dec. In case of missing a match, merge() should
+	 * allocate a new ID while unref() should WARN_ON.
+	 */
 	switch (smmu_domain->stage) {
 	case ARM_SMMU_DOMAIN_SVA:
 	case ARM_SMMU_DOMAIN_S1:
-		if (!arm_smmu_master_build_inv(master, INV_TYPE_S1_ASID,
-					       smmu_domain->cd.asid,
+		if (!arm_smmu_master_build_inv(master, INV_TYPE_S1_ASID, 0,
 					       IOMMU_NO_PASID, pgsize))
 			return NULL;
 		master->build_invs->alloc_id = arm_smmu_inv_alloc_asid;
@@ -3343,8 +3337,7 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
 		master->build_invs->smmu_domain = smmu_domain;
 		break;
 	case ARM_SMMU_DOMAIN_S2:
-		if (!arm_smmu_master_build_inv(master, INV_TYPE_S2_VMID,
-					       smmu_domain->s2_cfg.vmid,
+		if (!arm_smmu_master_build_inv(master, INV_TYPE_S2_VMID, 0,
 					       IOMMU_NO_PASID, pgsize))
 			return NULL;
 		master->build_invs->alloc_id = arm_smmu_inv_alloc_vmid;
@@ -3357,9 +3350,9 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
 
 	/* All the nested S1 ASIDs have to be flushed when S2 parent changes */
 	if (nesting) {
-		if (!arm_smmu_master_build_inv(
-			    master, INV_TYPE_S2_VMID_S1_CLEAR,
-			    smmu_domain->s2_cfg.vmid, IOMMU_NO_PASID, 0))
+		if (!arm_smmu_master_build_inv(master,
+					       INV_TYPE_S2_VMID_S1_CLEAR, 0,
+					       IOMMU_NO_PASID, 0))
 			return NULL;
 	}
 
-- 
2.43.0

[PATCH v1 7/9] iommu/arm-smmu-v3: Remove free_fn argument from arm_smmu_invs_unref()

[Nicolin Chen]

This is dead code now. Remove it.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  3 +-
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  |  6 ++--
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 34 ++-----------------
 3 files changed, 6 insertions(+), 37 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index dac412ff0d71..e38d2394e3be 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -1035,8 +1035,7 @@ void arm_smmu_make_sva_cd(struct arm_smmu_cd *target,
 struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
 					  struct arm_smmu_invs *to_merge);
 void arm_smmu_invs_unref(struct arm_smmu_invs *invs,
-			 struct arm_smmu_invs *to_unref,
-			 void (*free_fn)(struct arm_smmu_inv *inv));
+			 struct arm_smmu_invs *to_unref);
 struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs);
 #endif
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
index ead0d84cc9a0..5c8cb43f849c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
@@ -632,7 +632,7 @@ static void arm_smmu_v3_invs_test(struct kunit *test)
 				     results2[0], results2[1]);
 
 	/* Test3: unref invs2 (same array) */
-	arm_smmu_invs_unref(test_a, &invs2, NULL);
+	arm_smmu_invs_unref(test_a, &invs2);
 	arm_smmu_v3_invs_test_verify(test, test_a, ARRAY_SIZE(results3[0]),
 				     results3[0], results3[1]);
 	KUNIT_EXPECT_EQ(test, test_a->num_trashes, 0);
@@ -644,7 +644,7 @@ static void arm_smmu_v3_invs_test(struct kunit *test)
 				     results4[0], results4[1]);
 
 	/* Test5: unref invs1 (same array) */
-	arm_smmu_invs_unref(test_b, &invs1, NULL);
+	arm_smmu_invs_unref(test_b, &invs1);
 	arm_smmu_v3_invs_test_verify(test, test_b, ARRAY_SIZE(results5[0]),
 				     results5[0], results5[1]);
 	KUNIT_EXPECT_EQ(test, test_b->num_trashes, 2);
@@ -656,7 +656,7 @@ static void arm_smmu_v3_invs_test(struct kunit *test)
 				     results6[0], results6[1]);
 
 	/* Test7: unref invs3 (same array) */
-	arm_smmu_invs_unref(test_a, &invs3, NULL);
+	arm_smmu_invs_unref(test_a, &invs3);
 	KUNIT_EXPECT_EQ(test, test_a->num_invs, 0);
 	KUNIT_EXPECT_EQ(test, test_a->num_trashes, 0);
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index d51bad1002ff..5052988b0e4e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1203,9 +1203,6 @@ EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_merge);
  *                         the user counts without deletions
  * @invs: the base invalidation array
  * @to_unref: an array of invlidations to decrease their user counts
- * @free_fn: A callback function to invoke, when an entry's user count reduces
- *           to 0
- *
  * Return: the number of trash entries in the array, for arm_smmu_invs_purge()
  *
  * This function will not fail. Any entry with users=0 will be marked as trash.
@@ -1223,8 +1220,7 @@ EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_merge);
  */
 VISIBLE_IF_KUNIT
 void arm_smmu_invs_unref(struct arm_smmu_invs *invs,
-			 struct arm_smmu_invs *to_unref,
-			 void (*free_fn)(struct arm_smmu_inv *inv))
+			 struct arm_smmu_invs *to_unref)
 {
 	unsigned long flags;
 	size_t num_invs = 0;
@@ -3500,31 +3496,6 @@ arm_smmu_install_new_domain_invs(struct arm_smmu_attach_state *state)
 		state->master->vmid = invst->iotlb_tag.id;
 }
 
-/*
- * When an array entry's users count reaches zero, it means the ASID/VMID is no
- * longer being invalidated by map/unmap and must be cleaned. The rule is that
- * all ASIDs/VMIDs not in an invalidation array are left cleared in the IOTLB.
- */
-static void arm_smmu_inv_flush_iotlb_tag(struct arm_smmu_inv *inv)
-{
-	struct arm_smmu_cmdq_ent cmd = {};
-
-	switch (inv->type) {
-	case INV_TYPE_S1_ASID:
-		cmd.tlbi.asid = inv->id;
-		break;
-	case INV_TYPE_S2_VMID:
-		/* S2_VMID using nsize_opcode covers S2_VMID_S1_CLEAR */
-		cmd.tlbi.vmid = inv->id;
-		break;
-	default:
-		return;
-	}
-
-	cmd.opcode = inv->nsize_opcode;
-	arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, &cmd);
-}
-
 /* Should be installed after arm_smmu_install_ste_for_dev() */
 static void
 arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state)
@@ -3541,8 +3512,7 @@ arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state)
 	if (!invst->invs_ptr)
 		return;
 
-	arm_smmu_invs_unref(old_invs, invst->new_invs,
-			    arm_smmu_inv_flush_iotlb_tag);
+	arm_smmu_invs_unref(old_invs, invst->new_invs);
 	*old_iotlb_tag = invst->new_invs->inv[0];
 
 	new_invs = arm_smmu_invs_purge(old_invs);
-- 
2.43.0

[PATCH v1 8/9] iommu/arm-smmu-v3: Remove ASID/VMID from arm_smmu_domain

[Nicolin Chen]

Now ASID/VMID are stored in the arm_smmu_master. These are dead code now.

Remove all.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  8 ---
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   | 20 +------
 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  |  3 -
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 58 -------------------
 4 files changed, 1 insertion(+), 88 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index e38d2394e3be..4d7b7eb52dfd 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -792,10 +792,6 @@ static inline bool arm_smmu_ssids_in_use(struct arm_smmu_ctx_desc_cfg *cd_table)
 	return cd_table->used_ssids;
 }
 
-struct arm_smmu_s2_cfg {
-	u16				vmid;
-};
-
 struct arm_smmu_strtab_cfg {
 	union {
 		struct {
@@ -974,10 +970,6 @@ struct arm_smmu_domain {
 	atomic_t			nr_ats_masters;
 
 	enum arm_smmu_domain_stage	stage;
-	union {
-		struct arm_smmu_ctx_desc	cd;
-		struct arm_smmu_s2_cfg		s2_cfg;
-	};
 
 	struct iommu_domain		domain;
 
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 0e534f2b72e0..ea5ce3e6514e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -282,14 +282,6 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain)
 	 */
 	arm_smmu_domain_inv(smmu_domain);
 
-	/*
-	 * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can
-	 * still be called/running at this point. We allow the ASID to be
-	 * reused, and if there is a race then it just suffers harmless
-	 * unnecessary invalidation.
-	 */
-	xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
-
 	/*
 	 * Actual free is defered to the SRCU callback
 	 * arm_smmu_mmu_notifier_free()
@@ -308,7 +300,6 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
 	struct arm_smmu_master *master = dev_iommu_priv_get(dev);
 	struct arm_smmu_device *smmu = master->smmu;
 	struct arm_smmu_domain *smmu_domain;
-	u32 asid;
 	int ret;
 
 	if (!(master->smmu->features & ARM_SMMU_FEAT_SVA))
@@ -327,22 +318,13 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
 	smmu_domain->domain.pgsize_bitmap = PAGE_SIZE;
 	smmu_domain->stage = ARM_SMMU_DOMAIN_SVA;
 	smmu_domain->smmu = smmu;
-
-	ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
-		       XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
-	if (ret)
-		goto err_free;
-
-	smmu_domain->cd.asid = asid;
 	smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops;
 	ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm);
 	if (ret)
-		goto err_asid;
+		goto err_free;
 
 	return &smmu_domain->domain;
 
-err_asid:
-	xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
 err_free:
 	arm_smmu_domain_free(smmu_domain);
 	return ERR_PTR(ret);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
index 5c8cb43f849c..b174ac2dfb4b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
@@ -458,9 +458,6 @@ static void arm_smmu_test_make_s1_cd(struct kunit *test, struct arm_smmu_cd *cd,
 	struct io_pgtable io_pgtable = {};
 	struct arm_smmu_domain smmu_domain = {
 		.pgtbl_ops = &io_pgtable.ops,
-		.cd = {
-			.asid = asid,
-		},
 	};
 
 	io_pgtable.cfg.arm_lpae_s1_cfg.ttbr = 0xdaedbeefdeadbeefULL;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 5052988b0e4e..04e21af9c578 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2812,66 +2812,17 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void)
 static void arm_smmu_domain_free_paging(struct iommu_domain *domain)
 {
 	struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
-	struct arm_smmu_device *smmu = smmu_domain->smmu;
 
 	free_io_pgtable_ops(smmu_domain->pgtbl_ops);
-
-	/* Free the ASID or VMID */
-	if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
-		/* Prevent SVA from touching the CD while we're freeing it */
-		mutex_lock(&arm_smmu_asid_lock);
-		xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid);
-		mutex_unlock(&arm_smmu_asid_lock);
-	} else {
-		struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
-		if (cfg->vmid)
-			ida_free(&smmu->vmid_map, cfg->vmid);
-	}
-
 	arm_smmu_domain_free(smmu_domain);
 }
 
-static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu,
-				       struct arm_smmu_domain *smmu_domain)
-{
-	int ret;
-	u32 asid = 0;
-	struct arm_smmu_ctx_desc *cd = &smmu_domain->cd;
-
-	/* Prevent SVA from modifying the ASID until it is written to the CD */
-	mutex_lock(&arm_smmu_asid_lock);
-	ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
-		       XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL);
-	cd->asid	= (u16)asid;
-	mutex_unlock(&arm_smmu_asid_lock);
-	return ret;
-}
-
-static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu,
-				       struct arm_smmu_domain *smmu_domain)
-{
-	int vmid;
-	struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg;
-
-	/* Reserve VMID 0 for stage-2 bypass STEs */
-	vmid = ida_alloc_range(&smmu->vmid_map, 1, (1 << smmu->vmid_bits) - 1,
-			       GFP_KERNEL);
-	if (vmid < 0)
-		return vmid;
-
-	cfg->vmid	= (u16)vmid;
-	return 0;
-}
-
 static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
 				    struct arm_smmu_device *smmu, u32 flags)
 {
-	int ret;
 	enum io_pgtable_fmt fmt;
 	struct io_pgtable_cfg pgtbl_cfg;
 	struct io_pgtable_ops *pgtbl_ops;
-	int (*finalise_stage_fn)(struct arm_smmu_device *smmu,
-				 struct arm_smmu_domain *smmu_domain);
 	bool enable_dirty = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING;
 
 	pgtbl_cfg = (struct io_pgtable_cfg) {
@@ -2891,7 +2842,6 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
 		if (enable_dirty)
 			pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_ARM_HD;
 		fmt = ARM_64_LPAE_S1;
-		finalise_stage_fn = arm_smmu_domain_finalise_s1;
 		break;
 	}
 	case ARM_SMMU_DOMAIN_S2:
@@ -2900,7 +2850,6 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
 		pgtbl_cfg.ias = smmu->ias;
 		pgtbl_cfg.oas = smmu->oas;
 		fmt = ARM_64_LPAE_S2;
-		finalise_stage_fn = arm_smmu_domain_finalise_s2;
 		if ((smmu->features & ARM_SMMU_FEAT_S2FWB) &&
 		    (flags & IOMMU_HWPT_ALLOC_NEST_PARENT))
 			pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_ARM_S2FWB;
@@ -2918,13 +2867,6 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
 	smmu_domain->domain.geometry.force_aperture = true;
 	if (enable_dirty && smmu_domain->stage == ARM_SMMU_DOMAIN_S1)
 		smmu_domain->domain.dirty_ops = &arm_smmu_dirty_ops;
-
-	ret = finalise_stage_fn(smmu, smmu_domain);
-	if (ret < 0) {
-		free_io_pgtable_ops(pgtbl_ops);
-		return ret;
-	}
-
 	smmu_domain->pgtbl_ops = pgtbl_ops;
 	smmu_domain->smmu = smmu;
 	return 0;
-- 
2.43.0

[PATCH v1 9/9] iommu/arm-smmu-v3: Allow sharing domain across SMMUs

[Nicolin Chen]

VMM needs a domain holding the mappings between gPA to hPA. It can be an S1
domain or an S2 nesting parent domain, depending on whether the VM is built
with a vSMMU or not.

Given that the IOAS for this gPA mapping is the same across SMMU instances,
this domain can be shared across devices even if they sit behind different
SMMUs, so long as the underlying page table is compatible between the SMMU
instances.

There is no direct information about the page table from the master device,
but a comparison can be done between the physical SMMU that the domain was
allocated for and the physical SMMU that the device is behind.

Replace the smmu test in arm_smmu_attach_dev() and arm_vsmmu_init() with a
compatibility test for the S1 and S2 cases respectively. The compatibility
test goes through the physical SMMU parameters that were used to decide the
page table formats.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   | 20 +++++++++++++++++++
 .../arm/arm-smmu-v3/arm-smmu-v3-iommufd.c     |  2 +-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  2 +-
 3 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 4d7b7eb52dfd..d64e4e7c162d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -992,6 +992,26 @@ struct arm_smmu_nested_domain {
 	__le64 ste[2];
 };
 
+static inline bool
+arm_smmu_domain_can_share(struct arm_smmu_domain *smmu_domain,
+			  struct arm_smmu_device *new_smmu)
+{
+	struct arm_smmu_device *base_smmu = smmu_domain->smmu;
+
+	if (base_smmu == new_smmu)
+		return true;
+	/* Only support identical SMMUs for now */
+	if (base_smmu->features != new_smmu->features)
+		return false;
+	if (base_smmu->iommu.ops != new_smmu->iommu.ops)
+		return false;
+	if (base_smmu->pgsize_bitmap != new_smmu->pgsize_bitmap)
+		return false;
+	if (base_smmu->ias > new_smmu->ias || base_smmu->oas > new_smmu->oas)
+		return false;
+	return true;
+}
+
 /* The following are exposed for testing purposes. */
 struct arm_smmu_entry_writer_ops;
 struct arm_smmu_entry_writer {
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
index 1c877d30f86e..f2318e31d875 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
@@ -462,7 +462,7 @@ int arm_vsmmu_init(struct iommufd_viommu *viommu,
 		container_of(viommu->iommu_dev, struct arm_smmu_device, iommu);
 	struct arm_smmu_domain *s2_parent = to_smmu_domain(parent_domain);
 
-	if (s2_parent->smmu != smmu)
+	if (!arm_smmu_domain_can_share(s2_parent, smmu))
 		return -EINVAL;
 
 	vsmmu->smmu = smmu;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 04e21af9c578..ab329614da1f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3684,7 +3684,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev,
 	state.master = master = dev_iommu_priv_get(dev);
 	smmu = master->smmu;
 
-	if (smmu_domain->smmu != smmu)
+	if (!arm_smmu_domain_can_share(smmu_domain, smmu))
 		return -EINVAL;
 
 	if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
-- 
2.43.0

Re: [PATCH v1 3/9] iommu/arm-smmu-v3: Store ASIDs and VMID in arm_smmu_master

[Jason Gunthorpe]

On Thu, Dec 18, 2025 at 12:26:49PM -0800, Nicolin Chen wrote:
> Currently, ASID is allocated per smmu_domain, stored in the domain, and
> freed with the domain.
> 
> Practically, ASID is only used in a CD as an iotlb tag. Therefore, ASID
> doesn't really follow the life cycle of a domain but domain attachment.
> 
> On the other hand, the CD carrying ASID is installed to a device's STE.
> 
> This applies to the VMID as well, which is installed in an STE directly.
> 
> Since a device can only have one ASID per SSID and one VMID per SID, add
> an ASID array and VMID in the arm_smmu_master structure, to decouple the
> ASID/VMID from the domain structure.

I don't think this is entirely right..

When a S1 is attached to a master the master needs to store the VMID
it is using, as the VMID is global to the STE and effectively becomes
global to the master.

But the ASID should be stored in the invalidation list of the domain.

Jus search the list for the (instance, vmid) pair of the master to get
back the right ASID.

We don't need to store it again in another list, that's confusing.

Jason

Re: [PATCH v1 2/9] iommu/arm-smmu-v3: Add alloc_id/free_id functions to arm_smmu_invs

[Jason Gunthorpe]

On Thu, Dec 18, 2025 at 12:26:48PM -0800, Nicolin Chen wrote:
> @@ -720,6 +723,9 @@ struct arm_smmu_invs {
>  	rwlock_t rwlock;
>  	bool has_ats;
>  	struct rcu_head rcu;
> +	struct arm_smmu_domain *smmu_domain;
> +	int (*alloc_id)(struct arm_smmu_inv *inv, void *data);
> +	void (*free_id)(struct arm_smmu_inv *inv, bool flush);
>  	struct arm_smmu_inv inv[] __counted_by(max_invs);
>  };

I'm not keen on this at all..

On the allocation side everything should be stored inside the
invalidation list, we don't need function callbacks, just get the
information from the list in the first place.

You could probably split it up a bit, introduce arm_smmu_iotlb_tag
first and spread it around, filling it from the domain, then the below
to use the invs array to fill it. But this whole series
should basically just be this:

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index d7c492ee093602..5cd71a9690e292 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1188,8 +1188,7 @@ EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_merge);
  */
 VISIBLE_IF_KUNIT
 void arm_smmu_invs_unref(struct arm_smmu_invs *invs,
-			 struct arm_smmu_invs *to_unref,
-			 void (*free_fn)(struct arm_smmu_inv *inv))
+			 struct arm_smmu_invs *to_unref)
 {
 	unsigned long flags;
 	size_t num_invs = 0;
@@ -1200,16 +1199,16 @@ void arm_smmu_invs_unref(struct arm_smmu_invs *invs,
 		if (cmp < 0) {
 			/* not found in to_unref, leave alone */
 			num_invs = i + 1;
+			refcount_set(&to_unref->inv[j].users, 1);
 		} else if (cmp == 0) {
 			/* same item */
 			if (!refcount_dec_and_test(&invs->inv[i].users)) {
 				num_invs = i + 1;
+				refcount_set(&to_unref->inv[j].users, 1);
 				continue;
 			}
 
-			/* KUNIT test doesn't pass in a free_fn */
-			if (free_fn)
-				free_fn(&invs->inv[i]);
+			refcount_set(&to_unref->inv[j].users, 0);
 			invs->num_trashes++;
 		} else {
 			/* item in to_unref is not in invs or already a trash */
@@ -3046,6 +3045,13 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain,
 	return NULL;
 }
 
+static struct arm_vsmmu *to_vsmmu(struct iommu_domain *domain)
+{
+	if (domain->type == IOMMU_DOMAIN_NESTED)
+		return to_smmu_nested_domain(domain)->vsmmu;
+	return NULL;
+}
+
 /*
  * If the domain uses the smmu_domain->devices list return the arm_smmu_domain
  * structure, otherwise NULL. These domains track attached devices so they can
@@ -3156,12 +3162,136 @@ arm_smmu_master_build_inv(struct arm_smmu_master *master,
 	case INV_TYPE_ATS:
 	case INV_TYPE_ATS_FULL:
 		cur->size_opcode = cur->nsize_opcode = CMDQ_OP_ATC_INV;
+		cur->ssid = ssid;
 		break;
 	}
 
 	return cur;
 }
 
+static int arm_smmu_get_id_from_invs(struct arm_smmu_domain *smmu_domain,
+				     struct arm_smmu_device *smmu,
+				     unsigned int type)
+{
+	struct arm_smmu_invs *invs = rcu_dereference_protected(
+		smmu_domain->invs, lockdep_is_held(&arm_smmu_asid_lock));
+	size_t i;
+
+	for (i = 0; i != invs->num_invs; i++) {
+		if (invs->inv[i].type == type &&
+		    invs->inv[i].smmu == smmu &&
+		    refcount_read(&invs->inv[i].users))
+		    return invs->inv[i].id;
+	}
+
+	return -ENOENT;
+}
+
+static int arm_smmu_get_tag(struct arm_smmu_domain *smmu_domain,
+			    struct arm_smmu_master *master,
+			    struct arm_vsmmu *vsmmu,
+			    struct arm_smmu_iotlb_tag *tag, bool no_alloc)
+{
+	struct arm_smmu_device *smmu = master->smmu;
+	int ret;
+	int id;
+
+	lockdep_assert_held(&arm_smmu_asid_lock);
+
+	switch (smmu_domain->stage) {
+	case ARM_SMMU_DOMAIN_SVA:
+	case ARM_SMMU_DOMAIN_S1: {
+		u32 asid;
+
+		if (WARN_ON(vsmmu))
+			return -EINVAL;
+
+		id = arm_smmu_get_id_from_invs(smmu_domain, smmu,
+					       INV_TYPE_S1_ASID);
+		if (id > 0) {
+			tag->asid = id;
+			return 0;
+		}
+
+		if (no_alloc)
+			return -ENOENT;
+
+		ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain,
+			       XA_LIMIT(1, (1 << smmu->asid_bits) - 1),
+			       GFP_KERNEL);
+		if (ret)
+			return ret;
+		tag->asid = asid;
+		tag->need_free = 1;
+		return 0;
+	}
+
+	case ARM_SMMU_DOMAIN_S2:
+		if (smmu_domain->nest_parent) {
+			/* FIXME we can support attaching a nest_parent without
+			 * a vsmmu, but to do that we need to fix
+			 * arm_smmu_get_id_from_invs() to never return the vmid
+			 * of a vsmmu. Probably by making a
+			 * INV_TYPE_S2_VMID_VSMMU */
+			id = vsmmu->vmid;
+			return 0;
+		}
+
+		id = arm_smmu_get_id_from_invs(smmu_domain, smmu,
+					       INV_TYPE_S2_VMID);
+		if (id > 0) {
+			tag->vmid = id;
+			break;
+		}
+
+		if (no_alloc)
+			return -ENOENT;
+
+		/* Reserve VMID 0 for stage-2 bypass STEs */
+		id = ida_alloc_range(&smmu->vmid_map, 1,
+				     (1 << smmu->vmid_bits) - 1, GFP_KERNEL);
+		if (id < 0)
+			return id;
+		tag->vmid = id;
+		tag->need_free = 1;
+		return 0;
+	}
+	return -EINVAL;
+}
+
+static void arm_smmu_free_tag(struct arm_smmu_master *master,
+			      struct arm_smmu_domain *smmu_domain,
+			      struct arm_smmu_iotlb_tag *tag,
+			      struct arm_smmu_inv *inv)
+{
+	struct arm_smmu_device *smmu = master->smmu;
+	struct arm_smmu_cmdq_ent cmd = {};
+
+	switch (smmu_domain->stage) {
+	case ARM_SMMU_DOMAIN_SVA:
+	case ARM_SMMU_DOMAIN_S1:
+		cmd.opcode = inv->nsize_opcode;
+		cmd.tlbi.asid = tag->asid;
+		arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
+		xa_erase(&arm_smmu_asid_xa, tag->asid);
+		return;
+
+	case ARM_SMMU_DOMAIN_S2:
+		cmd.opcode = inv->nsize_opcode;
+		cmd.tlbi.vmid = tag->vmid;
+		arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd);
+
+		/*
+		 * Even though the VMID remains with the viommu and is not freed
+		 * we are stopping invalidations of it so it has to be cleared.
+		 */
+		if (smmu_domain->nest_parent)
+			return;
+		ida_free(&smmu->vmid_map, tag->vmid);
+		return;
+	}
+}
+
 /*
  * Use the preallocated scratch array at master->build_invs, to build a to_merge
  * or to_unref array, to pass into a following arm_smmu_invs_merge/unref() call.
@@ -3171,7 +3301,8 @@ arm_smmu_master_build_inv(struct arm_smmu_master *master,
  */
 static struct arm_smmu_invs *
 arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
-			   ioasid_t ssid, struct arm_smmu_domain *smmu_domain)
+			   ioasid_t ssid, struct arm_smmu_domain *smmu_domain,
+			   struct arm_smmu_iotlb_tag *tag)
 {
 	const bool nesting = smmu_domain->nest_parent;
 	size_t pgsize = 0, i;
@@ -3188,14 +3319,14 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
 	case ARM_SMMU_DOMAIN_SVA:
 	case ARM_SMMU_DOMAIN_S1:
 		if (!arm_smmu_master_build_inv(master, INV_TYPE_S1_ASID,
-					       smmu_domain->cd.asid,
-					       IOMMU_NO_PASID, pgsize))
+					       tag->asid, IOMMU_NO_PASID,
+					       pgsize))
 			return NULL;
 		break;
 	case ARM_SMMU_DOMAIN_S2:
 		if (!arm_smmu_master_build_inv(master, INV_TYPE_S2_VMID,
-					       smmu_domain->s2_cfg.vmid,
-					       IOMMU_NO_PASID, pgsize))
+					       tag->vmid, IOMMU_NO_PASID,
+					       pgsize))
 			return NULL;
 		break;
 	default:
@@ -3205,9 +3336,9 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
 
 	/* All the nested S1 ASIDs have to be flushed when S2 parent changes */
 	if (nesting) {
-		if (!arm_smmu_master_build_inv(
-			    master, INV_TYPE_S2_VMID_S1_CLEAR,
-			    smmu_domain->s2_cfg.vmid, IOMMU_NO_PASID, 0))
+		if (!arm_smmu_master_build_inv(master,
+					       INV_TYPE_S2_VMID_S1_CLEAR,
+					       tag->vmid, IOMMU_NO_PASID, 0))
 			return NULL;
 	}
 
@@ -3270,12 +3401,14 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master,
  * we are sequencing to update them.
  */
 static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *state,
+					struct iommu_domain *new_domain,
 					struct arm_smmu_domain *new_smmu_domain)
 {
 	struct arm_smmu_domain *old_smmu_domain =
 		to_smmu_domain_devices(state->old_domain);
 	struct arm_smmu_master *master = state->master;
 	ioasid_t ssid = state->ssid;
+	int ret;
 
 	/*
 	 * At this point a NULL domain indicates the domain doesn't use the
@@ -3289,8 +3422,18 @@ static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *state,
 		invst->old_invs = rcu_dereference_protected(
 			new_smmu_domain->invs,
 			lockdep_is_held(&arm_smmu_asid_lock));
-		build_invs = arm_smmu_master_build_invs(
-			master, state->ats_enabled, ssid, new_smmu_domain);
+
+		/* Re-use or allocate an IOTLB cache tag */
+		ret = arm_smmu_get_tag(new_smmu_domain, master,
+				       to_vsmmu(new_domain), &invst->tag,
+				       false);
+		if (ret)
+			return ret;
+
+		build_invs = arm_smmu_master_build_invs(master,
+							state->ats_enabled,
+							ssid, new_smmu_domain,
+							&invst->tag);
 		if (!build_invs)
 			return -EINVAL;
 
@@ -3311,9 +3454,18 @@ static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *state,
 			invst->old_invs = rcu_dereference_protected(
 				old_smmu_domain->invs,
 				lockdep_is_held(&arm_smmu_asid_lock));
+
+		/* Find the IOTLB cache tag this master was using */
+		ret = arm_smmu_get_tag(old_smmu_domain, master,
+				       to_vsmmu(state->old_domain), &invst->tag,
+				       true);
+		if (WARN_ON(ret))
+			return ret;
+
 		/* For old_smmu_domain, new_invs points to master->build_invs */
 		invst->new_invs = arm_smmu_master_build_invs(
-			master, master->ats_enabled, ssid, old_smmu_domain);
+			master, master->ats_enabled, ssid, old_smmu_domain,
+			&invst->tag);
 	}
 
 	return 0;
@@ -3349,36 +3501,13 @@ arm_smmu_install_new_domain_invs(struct arm_smmu_attach_state *state)
 	kfree_rcu(invst->old_invs, rcu);
 }
 
-/*
- * When an array entry's users count reaches zero, it means the ASID/VMID is no
- * longer being invalidated by map/unmap and must be cleaned. The rule is that
- * all ASIDs/VMIDs not in an invalidation array are left cleared in the IOTLB.
- */
-static void arm_smmu_inv_flush_iotlb_tag(struct arm_smmu_inv *inv)
-{
-	struct arm_smmu_cmdq_ent cmd = {};
-
-	switch (inv->type) {
-	case INV_TYPE_S1_ASID:
-		cmd.tlbi.asid = inv->id;
-		break;
-	case INV_TYPE_S2_VMID:
-		/* S2_VMID using nsize_opcode covers S2_VMID_S1_CLEAR */
-		cmd.tlbi.vmid = inv->id;
-		break;
-	default:
-		return;
-	}
-
-	cmd.opcode = inv->nsize_opcode;
-	arm_smmu_cmdq_issue_cmd_with_sync(inv->smmu, &cmd);
-}
-
 /* Should be installed after arm_smmu_install_ste_for_dev() */
 static void
 arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state)
 {
 	struct arm_smmu_inv_state *invst = &state->old_domain_invst;
+	struct arm_smmu_domain *old_smmu_domain =
+		to_smmu_domain_devices(state->old_domain);
 	struct arm_smmu_invs *old_invs = invst->old_invs;
 	struct arm_smmu_invs *new_invs;
 
@@ -3387,8 +3516,10 @@ arm_smmu_install_old_domain_invs(struct arm_smmu_attach_state *state)
 	if (!invst->invs_ptr)
 		return;
 
-	arm_smmu_invs_unref(old_invs, invst->new_invs,
-			    arm_smmu_inv_flush_iotlb_tag);
+	arm_smmu_invs_unref(old_invs, invst->new_invs);
+	if (old_smmu_domain && !refcount_read(&invst->new_invs->inv[0].users))
+		arm_smmu_free_tag(state->master, old_smmu_domain, &invst->tag,
+				  invst->new_invs->inv);
 
 	new_invs = arm_smmu_invs_purge(old_invs);
 	if (!new_invs)
@@ -3472,9 +3603,9 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state,
 				     arm_smmu_ats_supported(master);
 	}
 
-	ret = arm_smmu_attach_prepare_invs(state, smmu_domain);
+	ret = arm_smmu_attach_prepare_invs(state, new_domain, smmu_domain);
 	if (ret)
-		return ret;
+		goto err_unprepare_invs;
 
 	if (smmu_domain) {
 		if (new_domain->type == IOMMU_DOMAIN_NESTED) {
@@ -3551,6 +3682,10 @@ int arm_smmu_attach_prepare(struct arm_smmu_attach_state *state,
 err_free_vmaster:
 	kfree(state->vmaster);
 err_unprepare_invs:
+	if (state->new_domain_invst.tag.need_free)
+		arm_smmu_free_tag(master, smmu_domain,
+				  &state->new_domain_invst.tag,
+				  state->new_domain_invst.new_invs.inv);
 	kfree(state->new_domain_invst.new_invs);
 	return ret;
 }
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 4f104c1baa67a2..689b5ca7d61525 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -1103,6 +1103,12 @@ static inline bool arm_smmu_master_canwbs(struct arm_smmu_master *master)
 	       IOMMU_FWSPEC_PCI_RC_CANWBS;
 }
 
+struct arm_smmu_iotlb_tag {
+	u16 need_free : 1;
+	u16 asid;
+	u16 vmid;
+};
+
 /**
  * struct arm_smmu_inv_state - Per-domain invalidation array state
  * @invs_ptr: points to the domain->invs (unwinding nesting/etc.) or is NULL if
@@ -1116,6 +1122,7 @@ struct arm_smmu_inv_state {
 	struct arm_smmu_invs __rcu **invs_ptr;
 	struct arm_smmu_invs *old_invs;
 	struct arm_smmu_invs *new_invs;
+	struct arm_smmu_iotlb_tag tag;
 };
 
 struct arm_smmu_attach_state {

Re: [PATCH v1 3/9] iommu/arm-smmu-v3: Store ASIDs and VMID in arm_smmu_master

[Nicolin Chen]

On Fri, Dec 19, 2025 at 11:16:08AM -0400, Jason Gunthorpe wrote:
> On Thu, Dec 18, 2025 at 12:26:49PM -0800, Nicolin Chen wrote:
> > Currently, ASID is allocated per smmu_domain, stored in the domain, and
> > freed with the domain.
> > 
> > Practically, ASID is only used in a CD as an iotlb tag. Therefore, ASID
> > doesn't really follow the life cycle of a domain but domain attachment.
> > 
> > On the other hand, the CD carrying ASID is installed to a device's STE.
> > 
> > This applies to the VMID as well, which is installed in an STE directly.
> > 
> > Since a device can only have one ASID per SSID and one VMID per SID, add
> > an ASID array and VMID in the arm_smmu_master structure, to decouple the
> > ASID/VMID from the domain structure.
> 
> I don't think this is entirely right..
> 
> When a S1 is attached to a master the master needs to store the VMID
> it is using, as the VMID is global to the STE and effectively becomes
> global to the master.
>
> But the ASID should be stored in the invalidation list of the domain.

Hmm, you mean the nested case (host VMID + guest ASID), right?

But in that case, the guest-level ASID isn't needed anywhere in
the host driver? The ASID should be configured in a guest-level
CD. And the invalidation command directly coming from the guest
is already filled with the guest-level ASID.

Also, the invalidation array stores either ASID or VMID, there
is no case of storing both, right?

> Jus search the list for the (instance, vmid) pair of the master to get
> back the right ASID.
> 
> We don't need to store it again in another list, that's confusing.

Okay. I was trying to get rid of the extra for loop searching
for an ID in the array. That's why I had the alloc_id/free_id
ops too, so everything would be done in existing two loops in
the merge/unref functions.

Nicolin

Re: [PATCH v1 2/9] iommu/arm-smmu-v3: Add alloc_id/free_id functions to arm_smmu_invs

[Nicolin Chen]

On Fri, Dec 19, 2025 at 01:05:51PM -0400, Jason Gunthorpe wrote:
> On Thu, Dec 18, 2025 at 12:26:48PM -0800, Nicolin Chen wrote:
> > @@ -720,6 +723,9 @@ struct arm_smmu_invs {
> >  	rwlock_t rwlock;
> >  	bool has_ats;
> >  	struct rcu_head rcu;
> > +	struct arm_smmu_domain *smmu_domain;
> > +	int (*alloc_id)(struct arm_smmu_inv *inv, void *data);
> > +	void (*free_id)(struct arm_smmu_inv *inv, bool flush);
> >  	struct arm_smmu_inv inv[] __counted_by(max_invs);
> >  };
> 
> I'm not keen on this at all..
> 
> On the allocation side everything should be stored inside the
> invalidation list, we don't need function callbacks, just get the
> information from the list in the first place.
[...]
> @@ -3289,8 +3422,18 @@ static int arm_smmu_attach_prepare_invs(struct arm_smmu_attach_state *state,
>  		invst->old_invs = rcu_dereference_protected(
>  			new_smmu_domain->invs,
>  			lockdep_is_held(&arm_smmu_asid_lock));
> -		build_invs = arm_smmu_master_build_invs(
> -			master, state->ats_enabled, ssid, new_smmu_domain);
> +
> +		/* Re-use or allocate an IOTLB cache tag */
> +		ret = arm_smmu_get_tag(new_smmu_domain, master,
> +				       to_vsmmu(new_domain), &invst->tag,
> +				       false);
> +		if (ret)
> +			return ret;
> +
> +		build_invs = arm_smmu_master_build_invs(master,
> +							state->ats_enabled,
> +							ssid, new_smmu_domain,
> +							&invst->tag);
>  		if (!build_invs)
>  			return -EINVAL;

I had this when I was drafting the series in the beginning, but
later changed to the callback functions :-/

Again, my intention is to avoid iterating the array since merge
and unref functions are already doing that in their first loops.

That being said, if this get-before-build is the preferred way,
let's do that.

Thanks
Nicolin

Re: [PATCH v1 2/9] iommu/arm-smmu-v3: Add alloc_id/free_id functions to arm_smmu_invs

[Nicolin Chen]

Hi Jason,

On Fri, Dec 19, 2025 at 01:05:51PM -0400, Jason Gunthorpe wrote:
> +static int arm_smmu_get_tag(struct arm_smmu_domain *smmu_domain,
> +			    struct arm_smmu_master *master,
> +			    struct arm_vsmmu *vsmmu,
> +			    struct arm_smmu_iotlb_tag *tag, bool no_alloc)
[...]
> +	case ARM_SMMU_DOMAIN_S2:
> +		if (smmu_domain->nest_parent) {
> +			/* FIXME we can support attaching a nest_parent without
> +			 * a vsmmu, but to do that we need to fix
> +			 * arm_smmu_get_id_from_invs() to never return the vmid
> +			 * of a vsmmu. Probably by making a
> +			 * INV_TYPE_S2_VMID_VSMMU */
> +			id = vsmmu->vmid;
> +			return 0;
> +		}

Would you mind elaborating why arm_smmu_get_id_from_invs() can't
return vsmmu->vmid to share with a naked S2 STE?

I'm having a bit trouble justifying this INV_TYPE_S2_VMID_VSMMU.

Since it's the same S2 domain/iopt, if anything that is attached
(whether nested or naked) changes the S2 iopt, we should always
flush the nested S1 domain too, right?

If so, a naked S2 STE should refer to the same VMID in order to
allow its following INV_TYPE_S2_VMID_S1_CLEAR to work properly?

Thanks
Nicolin