Re: [PATCH 17/27] iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*() interface

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, Oct 11, 2023 at 08:25:53PM -0300, Jason Gunthorpe wrote:
> @@ -2609,26 +2613,27 @@ static void arm_smmu_attach_commit(struct arm_smmu_master *master,
>   * When an arm_smmu_master_domain is removed we have to turn off ATS as there is
>   * no longer any tracking of invalidations.
>   */
> -static void arm_smmu_attach_remove(struct arm_smmu_master *master)
> +static void arm_smmu_attach_remove(struct arm_smmu_master *master,
> +				   struct arm_smmu_domain *smmu_domain,
> +				   ioasid_t ssid)
>  {
> -	struct arm_smmu_domain *smmu_domain =
> -		to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev));
> -
>  	if (!smmu_domain)
>  		return;
>  
> -	if (master->ats_enabled) {
> +	if (ssid == IOMMU_NO_PASID && master->ats_enabled) {
>  		pci_disable_ats(to_pci_dev(master->dev));
>  		/*
>  		 * Ensure ATS is disabled at the endpoint before we issue the
>  		 * ATC invalidation via the SMMU.
>  		 */
>  		wmb();
> -		arm_smmu_atc_inv_master(master);
> +		arm_smmu_atc_inv_master(master, ssid);
>  	}
>  
> -	arm_smmu_remove_master_domain(master, smmu_domain);
> -	master->ats_enabled = false;
> +	arm_smmu_remove_master_domain(master, smmu_domain, ssid);
> +
> +	if (ssid == IOMMU_NO_PASID)
> +		master->ats_enabled = false;
>  }

This is missing the atc invalidation for the PASID case, it should have:

	/*
	 * The translation has already been changed in the STE/CD so flush the
	 * ATC. This must be before the removal of the master_domain to
	 * ensure the ATC does not become incoherent.
	 */
	if (master->ats_enabled)
		arm_smmu_atc_inv_master(master, ssid);

Moved out of the 'if ssid == NO_PASID'

Otherwise S2 domain detach will become incoherent later on.

Jason