Re: [PATCH v4 1/7] iommu/arm-smmu-v3: Add release_domain to attach prior to release_dev()

[Nicolin Chen <nicolinc@nvidia.com>]

On Mon, Sep 15, 2025 at 09:35:15AM -0300, Jason Gunthorpe wrote:
> On Fri, Sep 12, 2025 at 09:33:06AM +0000, Tian, Kevin wrote:
> > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > Sent: Monday, September 1, 2025 7:32 AM
> > > 
> > > +static int arm_smmu_attach_dev_release(struct iommu_domain *domain,
> > > +				       struct device *dev)
> > > +{
> > > +	struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> > > +
> > > +	WARN_ON(master->iopf_refcount);
> 
> This doesn't look right anymore..
> 
> Now that iopf is managed automatically it technically doesn't go to
> zero until the attaches below:

I will leave this WARN_ON in the arm_smmu_release_device(), while
having a release_domain to call arm_smmu_attach_dev_blocked():

-----------------------------------------------------------------
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 2a8b46b948f05..3b21790938d24 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3291,6 +3291,16 @@ static struct iommu_domain arm_smmu_blocked_domain = {
 	.ops = &arm_smmu_blocked_ops,
 };
 
+/* Same as arm_smmu_blocked_ops but less set_dev_pasid */
+static const struct iommu_domain_ops arm_smmu_release_ops = {
+	.attach_dev = arm_smmu_attach_dev_blocked,
+};
+
+static struct iommu_domain arm_smmu_release_domain = {
+	.type = IOMMU_DOMAIN_BLOCKED,
+	.ops = &arm_smmu_release_ops,
+};
+
 static struct iommu_domain *
 arm_smmu_domain_alloc_paging_flags(struct device *dev, u32 flags,
 				   const struct iommu_user_data *user_data)
@@ -3582,12 +3592,6 @@ static void arm_smmu_release_device(struct device *dev)
 
 	WARN_ON(master->iopf_refcount);
 
-	/* Put the STE back to what arm_smmu_init_strtab() sets */
-	if (dev->iommu->require_direct)
-		arm_smmu_attach_dev_identity(&arm_smmu_identity_domain, dev);
-	else
-		arm_smmu_attach_dev_blocked(&arm_smmu_blocked_domain, dev);
-
 	arm_smmu_disable_pasid(master);
 	arm_smmu_remove_master(master);
 	if (arm_smmu_cdtab_allocated(&master->cd_table))
@@ -3678,6 +3682,7 @@ static int arm_smmu_def_domain_type(struct device *dev)
 static const struct iommu_ops arm_smmu_ops = {
 	.identity_domain	= &arm_smmu_identity_domain,
 	.blocked_domain		= &arm_smmu_blocked_domain,
+	.release_domain		= &arm_smmu_release_domain,
 	.capable		= arm_smmu_capable,
 	.hw_info		= arm_smmu_hw_info,
 	.domain_alloc_sva       = arm_smmu_sva_domain_alloc,
-----------------------------------------------------------------

> > > +
> > > +	/* Put the STE back to what arm_smmu_init_strtab() sets */
> > > +	if (dev->iommu->require_direct)
> > > +
> > > 	arm_smmu_attach_dev_identity(&arm_smmu_identity_domain,
> > > dev);
> > > +	else
> > > +
> > > 	arm_smmu_attach_dev_blocked(&arm_smmu_blocked_domain,
> > > dev);
> 
> And I'd argue the attaches internally should have the assertion. If no
> pasids and blocked/identity the iopf == 0.

Ack. I will try a separate SMMU patch from this series.

> Also, I don't think this should be in the smmu driver, every driver
> should have this same logic, it is part of the definition of RMR
> Let's put it in the core code:

Ack. Adding this patch prior to the SMMU release_domain:

-----------------------------------------------------------------
From: Jason Gunthorpe <jgg@nvidia.com>
Date: Fri, 19 Sep 2025 22:26:45 +0000
Subject: [PATCH] iommu: Use identity_domain as release_domain for
 require_direct

If dev->iommu->require_direct is set, the core prevent attaching a BLOCKED
domains entirely in __iommu_device_set_domain():

	if (dev->iommu->require_direct &&
	    (new_domain->type == IOMMU_DOMAIN_BLOCKED ||
	     new_domain == group->blocking_domain)) {
		dev_warn(dev, "....");
		return -EINVAL;
	}

Thus, in most sane cases, the above will never convert BLOCKED to IDENTITY
in order to preserve the RMRs (direct mappings).

A similar situation would happen to the release_domain: while driver might
have set it to a BLOCKED domain, replace it with an IDENTITY for RMRs.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 drivers/iommu/iommu.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 08ba7b929580f..438458b465cac 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -516,8 +516,20 @@ static void iommu_deinit_device(struct device *dev)
 	 * Regardless, if a delayed attach never occurred, then the release
 	 * should still avoid touching any hardware configuration either.
 	 */
-	if (!dev->iommu->attach_deferred && ops->release_domain)
-		ops->release_domain->ops->attach_dev(ops->release_domain, dev);
+	if (!dev->iommu->attach_deferred && ops->release_domain) {
+		struct iommu_domain *release_domain = ops->release_domain;
+
+		/*
+		 * If the device requires direct mappings then it should not
+		 * be parked on a BLOCKED domain during release as that would
+		 * break the direct mappings.
+		 */
+		if (dev->iommu->require_direct && ops->identity_domain &&
+		    release_domain == ops->blocked_domain)
+			release_domain = ops->identity_domain;
+
+		release_domain->ops->attach_dev(release_domain, dev);
+	}
 
 	if (ops->release_device)
 		ops->release_device(dev);
-- 
2.43.0

-----------------------------------------------------------------

Thanks
Nicolin