Re: [PATCH v5 3/7] iommu/arm-smmu-v3: Introduce a per-domain arm_smmu_invs array

[Will Deacon <will@kernel.org>]

On Sat, Nov 08, 2025 at 12:08:04AM -0800, Nicolin Chen wrote:
> From: Jason Gunthorpe <jgg@nvidia.com>
> 
> Create a new data structure to hold an array of invalidations that need to
> be performed for the domain based on what masters are attached, to replace
> the single smmu pointer and linked list of masters in the current design.
> 
> Each array entry holds one of the invalidation actions - S1_ASID, S2_VMID,
> ATS or their variant with information to feed invalidation commands to HW.
> It is structured so that multiple SMMUs can participate in the same array,
> removing one key limitation of the current system.
> 
> To maximize performance, a sorted array is used as the data structure. It
> allows grouping SYNCs together to parallelize invalidations. For instance,
> it will group all the ATS entries after the ASID/VMID entry, so they will
> all be pushed to the PCI devices in parallel with one SYNC.
> 
> To minimize the locking cost on the invalidation fast path (reader of the
> invalidation array), the array is managed with RCU.
> 
> Provide a set of APIs to add/delete entries to/from an array, which cover
> cannot-fail attach cases, e.g. attaching to arm_smmu_blocked_domain. Also
> add kunit coverage for those APIs.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> Co-developed-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  91 +++++++
>  .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c  |  93 +++++++
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 248 ++++++++++++++++++
>  3 files changed, 432 insertions(+)
> 
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 96a23ca633cb6..757158b9ea655 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -649,6 +649,85 @@ struct arm_smmu_cmdq_batch {
>  	int				num;
>  };
>  
> +/*
> + * The order here also determines the sequence in which commands are sent to the
> + * command queue. E.g. TLBI must be done before ATC_INV.
> + */
> +enum arm_smmu_inv_type {
> +	INV_TYPE_S1_ASID,
> +	INV_TYPE_S2_VMID,
> +	INV_TYPE_S2_VMID_S1_CLEAR,
> +	INV_TYPE_ATS,
> +	INV_TYPE_ATS_FULL,
> +};
> +
> +struct arm_smmu_inv {
> +	struct arm_smmu_device *smmu;
> +	u8 type;
> +	u8 size_opcode;
> +	u8 nsize_opcode;
> +	u32 id; /* ASID or VMID or SID */
> +	union {
> +		size_t pgsize; /* ARM_SMMU_FEAT_RANGE_INV */
> +		u32 ssid; /* INV_TYPE_ATS */
> +	};
> +
> +	refcount_t users; /* users=0 to mark as a trash to be purged */
> +};
> +
> +static inline bool arm_smmu_inv_is_ats(struct arm_smmu_inv *inv)
> +{
> +	return inv->type == INV_TYPE_ATS || inv->type == INV_TYPE_ATS_FULL;
> +}
> +
> +/**
> + * struct arm_smmu_invs - Per-domain invalidation array
> + * @num_invs: number of invalidations in the flexible array
> + * @rwlock: optional rwlock to fench ATS operations
> + * @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FULL
> + * @rcu: rcu head for kfree_rcu()
> + * @inv: flexible invalidation array
> + *
> + * The arm_smmu_invs is an RCU data structure. During a ->attach_dev callback,
> + * arm_smmu_invs_merge(), arm_smmu_invs_unref() and arm_smmu_invs_purge() will
> + * be used to allocate a new copy of an old array for addition and deletion in
> + * the old domain's and new domain's invs arrays.
> + *
> + * The arm_smmu_invs_unref() mutates a given array, by internally reducing the
> + * users counts of some given entries. This exists to support a no-fail routine
> + * like attaching to an IOMMU_DOMAIN_BLOCKED. And it could pair with a followup
> + * arm_smmu_invs_purge() call to generate a new clean array.
> + *
> + * Concurrent invalidation thread will push every invalidation described in the
> + * array into the command queue for each invalidation event. It is designed like
> + * this to optimize the invalidation fast path by avoiding locks.
> + *
> + * A domain can be shared across SMMU instances. When an instance gets removed,
> + * it would delete all the entries that belong to that SMMU instance. Then, a
> + * synchronize_rcu() would have to be called to sync the array, to prevent any
> + * concurrent invalidation thread accessing the old array from issuing commands
> + * to the command queue of a removed SMMU instance.
> + */
> +struct arm_smmu_invs {
> +	size_t num_invs;
> +	rwlock_t rwlock;
> +	bool has_ats;
> +	struct rcu_head rcu;
> +	struct arm_smmu_inv inv[];
> +};

Can you use __counted_by(num_invs) here?

> +
> +static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
> +{
> +	struct arm_smmu_invs *new_invs;
> +
> +	new_invs = kzalloc(struct_size(new_invs, inv, num_invs), GFP_KERNEL);
> +	if (!new_invs)
> +		return ERR_PTR(-ENOMEM);

Just return NULL on failure like most allocator functions?

> +	rwlock_init(&new_invs->rwlock);
> +	new_invs->num_invs = num_invs;
> +	return new_invs;
> +}
> +

[...]

> +VISIBLE_IF_KUNIT
> +struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
> +					  struct arm_smmu_invs *to_merge)
> +{
> +	struct arm_smmu_invs *new_invs;
> +	struct arm_smmu_inv *new;
> +	size_t num_trashes = 0;
> +	size_t num_adds = 0;
> +	size_t i, j;
> +
> +	for (i = j = 0; i < invs->num_invs || j < to_merge->num_invs;) {

Maybe worth having a simple iterator macro for this?

> +		int cmp;
> +
> +		/* Skip any trash entry */
> +		if (i < invs->num_invs && !refcount_read(&invs->inv[i].users)) {
> +			num_trashes++;
> +			i++;
> +			continue;
> +		}
> +
> +		cmp = arm_smmu_invs_cmp(invs, i, to_merge, j);
> +		if (cmp < 0) {
> +			/* not found in to_merge, leave alone */
> +			i++;
> +		} else if (cmp == 0) {
> +			/* same item */
> +			i++;
> +			j++;
> +		} else {
> +			/* unique to to_merge */
> +			num_adds++;
> +			j++;
> +		}
> +	}
> +
> +	new_invs = arm_smmu_invs_alloc(invs->num_invs - num_trashes + num_adds);
> +	if (IS_ERR(new_invs))
> +		return new_invs;
> +
> +	new = new_invs->inv;
> +	for (i = j = 0; i < invs->num_invs || j < to_merge->num_invs;) {
> +		int cmp;
> +
> +		if (i < invs->num_invs && !refcount_read(&invs->inv[i].users)) {
> +			i++;
> +			continue;
> +		}
> +
> +		cmp = arm_smmu_invs_cmp(invs, i, to_merge, j);
> +		if (cmp < 0) {
> +			*new = invs->inv[i];
> +			i++;
> +		} else if (cmp == 0) {
> +			*new = invs->inv[i];
> +			refcount_inc(&new->users);
> +			i++;
> +			j++;
> +		} else {
> +			*new = to_merge->inv[j];
> +			refcount_set(&new->users, 1);
> +			j++;
> +		}
> +
> +		/*
> +		 * Check that the new array is sorted. This also validates that
> +		 * to_merge is sorted.
> +		 */
> +		if (new != new_invs->inv)
> +			WARN_ON_ONCE(arm_smmu_inv_cmp(new - 1, new) == 1);
> +		new++;
> +	}
> +
> +	WARN_ON(new != new_invs->inv + new_invs->num_invs);
> +
> +	return new_invs;
> +}
> +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_merge);

There's nothing really SMMU-specific about this data structure manipulation.
Do you think we can abstract the invalidation array concept into a library
which other IOMMU drivers could use too?

Will