Re: [PATCH v5 4/7] iommu/arm-smmu-v3: Pre-allocate a per-master invalidation array

[Will Deacon <will@kernel.org>]

On Sat, Nov 08, 2025 at 12:08:05AM -0800, Nicolin Chen wrote:
> When a master is attached from an old domain to a new domain, it needs to
> build an invalidation array to delete and add the array entries from/onto
> the invalidation arrays of those two domains, passed via the to_merge and
> to_unref arguments into arm_smmu_invs_merge/unref() respectively.
> 
> Since the master->num_streams might differ across masters, a memory would
> have to be allocated when building an to_merge/to_unref array which might
> fail with -ENOMEM.
> 
> On the other hand, an attachment to arm_smmu_blocked_domain must not fail
> so it's the best to avoid any memory allocation in that path.
> 
> Pre-allocate a fixed size invalidation array for every master. This array
> will be used as a scratch to fill dynamically when building a to_merge or
> to_unref invs array. Sort fwspec->ids in an ascending order to fit to the
> arm_smmu_invs_merge() function.
> 
> Co-developed-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h |  8 ++++++
>  drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 27 +++++++++++++++++++++
>  2 files changed, 35 insertions(+)
> 
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 757158b9ea655..7b81a82c0dfe4 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -922,6 +922,14 @@ struct arm_smmu_master {
>  	struct arm_smmu_device		*smmu;
>  	struct device			*dev;
>  	struct arm_smmu_stream		*streams;
> +	/*
> +	 * Scratch memory for a to_merge or to_unref array to build a per-domain
> +	 * invalidation array. It'll be pre-allocated with enough enries for all
> +	 * possible build scenarios. It can be used by only one caller at a time
> +	 * until the arm_smmu_invs_merge/unref() finishes. Must be locked by the
> +	 * iommu_group mutex.
> +	 */
> +	struct arm_smmu_invs		*build_invs;
>  	struct arm_smmu_vmaster		*vmaster; /* use smmu->streams_mutex */
>  	/* Locked by the iommu core using the group mutex */
>  	struct arm_smmu_ctx_desc_cfg	cd_table;
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 8266d0839a927..26b8492a13f20 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -3693,12 +3693,22 @@ static int arm_smmu_init_sid_strtab(struct arm_smmu_device *smmu, u32 sid)
>  	return 0;
>  }
>  
> +static int arm_smmu_ids_cmp(const void *_l, const void *_r)
> +{
> +	const typeof_member(struct iommu_fwspec, ids[0]) *l = _l;
> +	const typeof_member(struct iommu_fwspec, ids[0]) *r = _r;
> +
> +	return cmp_int(*l, *r);
> +}
> +
>  static int arm_smmu_insert_master(struct arm_smmu_device *smmu,
>  				  struct arm_smmu_master *master)
>  {
>  	int i;
>  	int ret = 0;
>  	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev);
> +	bool ats_supported = dev_is_pci(master->dev) &&
> +			     pci_ats_supported(to_pci_dev(master->dev));
>  
>  	master->streams = kcalloc(fwspec->num_ids, sizeof(*master->streams),
>  				  GFP_KERNEL);
> @@ -3706,6 +3716,21 @@ static int arm_smmu_insert_master(struct arm_smmu_device *smmu,
>  		return -ENOMEM;
>  	master->num_streams = fwspec->num_ids;
>  
> +	if (!ats_supported) {
> +		/* Base case has 1 ASID entry or maximum 2 VMID entries */
> +		master->build_invs = arm_smmu_invs_alloc(2);
> +	} else {
> +		/* Put the ids into order for sorted to_merge/to_unref arrays */
> +		sort_nonatomic(fwspec->ids, fwspec->num_ids,
> +			       sizeof(fwspec->ids[0]), arm_smmu_ids_cmp, NULL);
> +		/* ATS case adds num_ids of entries, on top of the base case */
> +		master->build_invs = arm_smmu_invs_alloc(2 + fwspec->num_ids);

Although I can't point at a specific issue here, I'm nervous about mutating
the 'fwspec->ids' array from within the driver, The array isn't allocated
or populated directly by the driver and so I don't think we really have any
business sorting it. Could we hack iommu_fwspec_add_ids() to keep the array
ordered instead?

Will