[PATCH v3 00/10] Make the 32-bit ARM DMA API work with normal IOMMU DMA default domains (+ Tegra PoC)

[PATCH v3 00/10] Make the 32-bit ARM DMA API work with normal IOMMU DMA default domains (+ Tegra PoC)

[Mikko Perttunen]

Depending on the probe ordering, TegraDRM does not currently probe on
kernels with CONFIG_ARM_DMA_USE_IOMMU enabled, such as with
multi_v7_defconfig. The Tegra SMMU puts several display and 2D/3D
clients into one IOMMU group, and the legacy 32-bit ARM DMA API path
creates one UNMANAGED domain per device and attaches it with
iommu_attach_device(). That only works for one device per group, so
every client after the first fails to attach,
iommu_device_use_default_domain() rejects the group with -EBUSY, and the
driver goes down with it. tegra_defconfig is currently working by virtue
of not enabling CONFIG_ARM_DMA_USE_IOMMU.

Robin suggested [1] that instead of teaching the legacy path about
groups, we should explore modernizing the 32-bit ARM DMA code a bit by
making it use domain cookies and have the core code own the domains, as
is the case on other architectures. Still, ARM's own DMA code is kept to
support existing drivers and APIs that expect it, and because dma-iommu
is considered too big and unnecessary for 32-bit ARM.

The series is in four parts:

- Patches 1-2 are preparation in the IOMMU core:
  1. Add iommu_teardown_dma_ops, counterpart to iommu_setup_dma_ops.
  2. Add cookie type for ARM's DMA domains.
- Patches 3-6 implement the DMA API side on 32-bit ARM:
  3. Split up mapping alloc and init (bitmap alloc) code to separate
     functions.
  4. Implement parallel dma_ops / cookie APIs on ARM side.
  5. Avoid domain allocation when IOMMU core does it.
  6. Check upper DMA range limit during IOVA allocation.
- Patch 7 removes the forced IDENTITY default domain in core IOMMU
  code and adds explicit IDENTITY defaults to a few individual IOMMU
  drivers.
- Patches 8-10 convert Tegra:
  8. Fixes device parameter in calls to dma_get_sgtable.
  9. Allows deferring translation enable until driver handshake for
     selected devices.
  10. Switches to DMA default domain for selected Tegra devices.

This moves 32-bit ARM devices from the unmanaged IOMMU DMA domain to the
core-owned default DMA domain except for mtk_iommu_v1, ipmmu-vmsa,
omap-iommu, and msm_iommu. (tegra-smmu also stays on the legacy path
until the last patch in the series.) See commit 7 for further
information. Existing drivers are intended to continue working without
changes.

Patches 8-10 are provided for demonstration; we can deal with them 
separately if the rest of the series is accepted.

Patches 1,2,3,6 are applicable independently.
Patches 4,5 depend on the previous patches.
Patch 7 requires the prior patches to be in place.

Patches 1-7 only touch drivers/iommu/ and arch/arm/mm/ and associated 
headers.

Best regards,
Mikko

[1] https://lore.kernel.org/linux-tegra/910d1492-ad3b-4201-93c6-208be1dcde7d@arm.com/

---
Changes in v3:
- Removed the forced IDENTITY default domain workaround from IOMMU core.
  Explicit IDENTITY default kept for a few IOMMU drivers.
- Keep the arm_iommu_* APIs usable with DMA default domains. Drop
  related Host1x/TegraDRM cleanup patches. Detaching an explicitly
  created mapping restores the default domain.
- Install and clear DMA ops in iommu_setup/teardown_dma_ops for
  core-owned domains.
- Use iommu_is_dma_domain instead of custom flag to distinguish between
  core-owned domains and unmanaged DMA domains.
- Allocate IOVA bookkeeping in iommu_setup_dma_ops, after attachment
  has established domain geometry.
- Add an upper-bound check during IOVA allocation to replace setup-time
  dma-ranges based check.
- Remove change to arm_iommu_get_sgtable in favor of correct fix in
  TegraDRM.
- Simplify tegra_smmu_def_domain_type.
- Link to v2: https://patch.msgid.link/20261016-b4-arm32-iommu-dma-cookie-v2-0-b06842a01938@nvidia.com

Changes in v2:
- Addressed sashiko comments:
  - We still don't take device tree dma-ranges into account for the allocatable
    IOVA range. This is closer to arm64 and only some OMAP5 devices currently
    have dma-ranges where this would matter. A check is added to error out in
    such cases. As in v1, this code path is only active when enabled by the
    IOMMU driver so there is no breakage.
  - Added a check in tegra_smmu_enable_translation to return early if the IOMMU
    attached to the device is not a tegra-smmu.
- Added .release_domain in tegra-smmu to handle address space release for DMA
  domains.
- Link to v1: https://patch.msgid.link/20260915-b4-arm32-iommu-dma-cookie-v1-0-ba4f9d33a2b8@nvidia.com

---
Mikko Perttunen (10):
      iommu: Add iommu_teardown_dma_ops
      iommu: Add cookie type for ARM_DMA_USE_IOMMU
      ARM: dma-mapping: Split mapping initialization from allocation
      ARM: dma-mapping: Implement and expose DMA setup functions
      ARM: dma-mapping: Bypass the legacy path for core-owned DMA API domains
      ARM: dma-mapping: Bound IOVA allocation by device DMA limits
      iommu: Enable DMA default domains on (most) 32-bit ARM platforms
      drm/tegra: Fix device parameter to dma_get_sgtable
      iommu/tegra-smmu: Defer display client translation to driver handover
      iommu/tegra-smmu: Use a DMA API default domain per swgroup

 arch/arm/include/asm/dma-iommu.h |   9 ++
 arch/arm/mm/dma-mapping.c        | 297 +++++++++++++++++++++++++++++----------
 drivers/gpu/drm/tegra/dc.c       |   8 ++
 drivers/gpu/drm/tegra/gem.c      |   5 +-
 drivers/iommu/Kconfig            |   1 +
 drivers/iommu/dma-iommu.c        |   5 +
 drivers/iommu/dma-iommu.h        |  49 +++++--
 drivers/iommu/iommu.c            |  24 +---
 drivers/iommu/ipmmu-vmsa.c       |   6 +
 drivers/iommu/msm_iommu.c        |   6 +
 drivers/iommu/mtk_iommu_v1.c     |   6 +
 drivers/iommu/omap-iommu.c       |   6 +
 drivers/iommu/tegra-smmu.c       | 103 +++++++++++++-
 drivers/memory/tegra/tegra114.c  |  15 +-
 drivers/memory/tegra/tegra124.c  |  12 +-
 drivers/memory/tegra/tegra210.c  |   6 +-
 drivers/memory/tegra/tegra30.c   |  18 ++-
 include/linux/iommu.h            |   3 +
 include/soc/tegra/mc.h           |  20 +++
 19 files changed, 467 insertions(+), 132 deletions(-)
---
base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
change-id: 20260728-b4-arm32-iommu-dma-cookie-3170e07f9125

[PATCH v3 01/10] iommu: Add iommu_teardown_dma_ops

[Mikko Perttunen]

Add a counterpart to iommu_setup_dma_ops, tearing down state set by it.

Currently this teardown is open-coded in iommu_deinit_device; however
when 32-bit ARM gains a parallel implementation of iommu_setup_dma_ops
it becomes nicer to have a matching parallel iommu_teardown_dma_ops as
well.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 drivers/iommu/dma-iommu.c | 5 +++++
 drivers/iommu/dma-iommu.h | 5 +++++
 drivers/iommu/iommu.c     | 3 +--
 3 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 68ee57800fef..379019748a5d 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -2161,6 +2161,11 @@ void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain)
 	dev_clear_dma_iommu(dev);
 }
 
+void iommu_teardown_dma_ops(struct device *dev)
+{
+	dev_clear_dma_iommu(dev);
+}
+
 static bool has_msi_cookie(const struct iommu_domain *domain)
 {
 	return domain && (domain->cookie_type == IOMMU_COOKIE_DMA_IOVA ||
diff --git a/drivers/iommu/dma-iommu.h b/drivers/iommu/dma-iommu.h
index 040d00252563..349d6a57e5a0 100644
--- a/drivers/iommu/dma-iommu.h
+++ b/drivers/iommu/dma-iommu.h
@@ -10,6 +10,7 @@
 #ifdef CONFIG_IOMMU_DMA
 
 void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain);
+void iommu_teardown_dma_ops(struct device *dev);
 
 int iommu_get_dma_cookie(struct iommu_domain *domain);
 void iommu_put_dma_cookie(struct iommu_domain *domain);
@@ -31,6 +32,10 @@ static inline void iommu_setup_dma_ops(struct device *dev,
 {
 }
 
+static inline void iommu_teardown_dma_ops(struct device *dev)
+{
+}
+
 static inline int iommu_dma_init_fq(struct iommu_domain *domain)
 {
 	return -EINVAL;
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index cd1bca7ede9a..a5440ed5e235 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -611,8 +611,7 @@ static void iommu_deinit_device(struct device *dev)
 	dev->iommu_group = NULL;
 	module_put(ops->owner);
 	dev_iommu_free(dev);
-	if (IS_ENABLED(CONFIG_IOMMU_DMA))
-		dev_clear_dma_iommu(dev);
+	iommu_teardown_dma_ops(dev);
 }
 
 static struct iommu_domain *pasid_array_entry_to_domain(void *entry)

-- 
2.55.0

[PATCH v3 02/10] iommu: Add cookie type for ARM_DMA_USE_IOMMU

[Mikko Perttunen]

The ARM_DMA_USE_IOMMU DMA API implementation keeps its state in a struct
dma_iommu_mapping. Currently that struct owns an IOMMU domain -- to
allow inverting that relation, add a cookie type pointing to a struct
dma_iommu_mapping, more in line with how other architectures work.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 drivers/iommu/iommu.c | 1 +
 include/linux/iommu.h | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index a5440ed5e235..5fbb76d44a1e 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2127,6 +2127,7 @@ void iommu_domain_free(struct iommu_domain *domain)
 {
 	switch (domain->cookie_type) {
 	case IOMMU_COOKIE_DMA_IOVA:
+	case IOMMU_COOKIE_ARM_DMA:
 		iommu_put_dma_cookie(domain);
 		break;
 	case IOMMU_COOKIE_DMA_MSI:
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index ac43b8b93f14..4b87625892ac 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -43,6 +43,7 @@ struct notifier_block;
 struct iommu_sva;
 struct iommu_dma_cookie;
 struct iommu_dma_msi_cookie;
+struct dma_iommu_mapping;
 struct iommu_fault_param;
 struct iommufd_ctx;
 struct iommufd_viommu;
@@ -174,6 +175,7 @@ enum iommu_domain_cookie_type {
 	IOMMU_COOKIE_FAULT_HANDLER,
 	IOMMU_COOKIE_SVA,
 	IOMMU_COOKIE_IOMMUFD,
+	IOMMU_COOKIE_ARM_DMA,
 };
 
 /* Domain feature flags */
@@ -234,6 +236,7 @@ struct iommu_domain {
 	union { /* cookie */
 		struct iommu_dma_cookie *iova_cookie;
 		struct iommu_dma_msi_cookie *msi_cookie;
+		struct dma_iommu_mapping *arm_cookie;
 		struct iommufd_hw_pagetable *iommufd_hwpt;
 		struct {
 			iommu_fault_handler_t handler;

-- 
2.55.0

[PATCH v3 03/10] ARM: dma-mapping: Split mapping initialization from allocation

[Mikko Perttunen]

arm_iommu_create_mapping allocates the mapping object, its IOVA bitmap
and the corresponding IOMMU domain together. Separate the complicated
part of mapping initialization, the bitmap initialization, into a new
function, __arm_iommu_init_mapping. This allows the default domain path
added later in the series to initialize the mapping object before the
bitmap address range is known and later call _init_mapping to initialize
the bitmaps.

No functional change.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 arch/arm/mm/dma-mapping.c | 109 +++++++++++++++++++++++++++-------------------
 1 file changed, 63 insertions(+), 46 deletions(-)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 7761099dde9e..7ef7c268c583 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -1470,6 +1470,57 @@ static const struct dma_map_ops iommu_ops = {
 	.sync_sg_for_device	= arm_iommu_sync_sg_for_device,
 };
 
+static int __arm_iommu_init_mapping(struct dma_iommu_mapping *mapping,
+				    dma_addr_t base, u64 size)
+{
+	unsigned int bits, bitmap_size;
+	unsigned long **bitmaps;
+	int extensions = 1;
+
+	/* currently only 32-bit DMA address space is supported */
+	if (size > DMA_BIT_MASK(32) + 1)
+		return -ERANGE;
+
+	bits = size >> PAGE_SHIFT;
+	bitmap_size = BITS_TO_LONGS(bits) * sizeof(long);
+	if (!bitmap_size)
+		return -EINVAL;
+
+	if (bitmap_size > PAGE_SIZE) {
+		extensions = bitmap_size / PAGE_SIZE;
+		bitmap_size = PAGE_SIZE;
+	}
+
+	bitmaps = kcalloc(extensions, sizeof(*bitmaps), GFP_KERNEL);
+	if (!bitmaps)
+		return -ENOMEM;
+
+	bitmaps[0] = kzalloc(bitmap_size, GFP_KERNEL);
+	if (!bitmaps[0]) {
+		kfree(bitmaps);
+		return -ENOMEM;
+	}
+
+	mapping->bitmap_size = bitmap_size;
+	mapping->nr_bitmaps = 1;
+	mapping->extensions = extensions;
+	mapping->base = base;
+	mapping->bits = BITS_PER_BYTE * bitmap_size;
+	mapping->bitmaps = bitmaps;
+
+	return 0;
+}
+
+static void __arm_iommu_free_mapping(struct dma_iommu_mapping *mapping)
+{
+	int i;
+
+	for (i = 0; i < mapping->nr_bitmaps; i++)
+		kfree(mapping->bitmaps[i]);
+	kfree(mapping->bitmaps);
+	kfree(mapping);
+}
+
 /**
  * arm_iommu_create_mapping
  * @dev: pointer to the client device (for IOMMU calls)
@@ -1486,75 +1537,41 @@ static const struct dma_map_ops iommu_ops = {
 struct dma_iommu_mapping *
 arm_iommu_create_mapping(struct device *dev, dma_addr_t base, u64 size)
 {
-	unsigned int bits = size >> PAGE_SHIFT;
-	unsigned int bitmap_size = BITS_TO_LONGS(bits) * sizeof(long);
 	struct dma_iommu_mapping *mapping;
-	int extensions = 1;
-	int err = -ENOMEM;
-
-	/* currently only 32-bit DMA address space is supported */
-	if (size > DMA_BIT_MASK(32) + 1)
-		return ERR_PTR(-ERANGE);
-
-	if (!bitmap_size)
-		return ERR_PTR(-EINVAL);
-
-	if (bitmap_size > PAGE_SIZE) {
-		extensions = bitmap_size / PAGE_SIZE;
-		bitmap_size = PAGE_SIZE;
-	}
+	int err;
 
-	mapping = kzalloc_obj(struct dma_iommu_mapping);
+	mapping = kzalloc_obj(*mapping);
 	if (!mapping)
-		goto err;
+		return ERR_PTR(-ENOMEM);
 
-	mapping->bitmap_size = bitmap_size;
-	mapping->bitmaps = kcalloc(extensions, sizeof(unsigned long *),
-				   GFP_KERNEL);
-	if (!mapping->bitmaps)
-		goto err2;
-
-	mapping->bitmaps[0] = kzalloc(bitmap_size, GFP_KERNEL);
-	if (!mapping->bitmaps[0])
-		goto err3;
-
-	mapping->nr_bitmaps = 1;
-	mapping->extensions = extensions;
-	mapping->base = base;
-	mapping->bits = BITS_PER_BYTE * bitmap_size;
+	err = __arm_iommu_init_mapping(mapping, base, size);
+	if (err) {
+		kfree(mapping);
+		return ERR_PTR(err);
+	}
 
 	spin_lock_init(&mapping->lock);
 
 	mapping->domain = iommu_paging_domain_alloc(dev);
 	if (IS_ERR(mapping->domain)) {
 		err = PTR_ERR(mapping->domain);
-		goto err4;
+
+		__arm_iommu_free_mapping(mapping);
+		return ERR_PTR(err);
 	}
 
 	kref_init(&mapping->kref);
 	return mapping;
-err4:
-	kfree(mapping->bitmaps[0]);
-err3:
-	kfree(mapping->bitmaps);
-err2:
-	kfree(mapping);
-err:
-	return ERR_PTR(err);
 }
 EXPORT_SYMBOL_GPL(arm_iommu_create_mapping);
 
 static void release_iommu_mapping(struct kref *kref)
 {
-	int i;
 	struct dma_iommu_mapping *mapping =
 		container_of(kref, struct dma_iommu_mapping, kref);
 
 	iommu_domain_free(mapping->domain);
-	for (i = 0; i < mapping->nr_bitmaps; i++)
-		kfree(mapping->bitmaps[i]);
-	kfree(mapping->bitmaps);
-	kfree(mapping);
+	__arm_iommu_free_mapping(mapping);
 }
 
 static int extend_iommu_mapping(struct dma_iommu_mapping *mapping)

-- 
2.55.0

[PATCH v3 04/10] ARM: dma-mapping: Implement and expose DMA setup functions

[Mikko Perttunen]

ARM DMA domains are currently allocated by the ARM DMA code, with struct
dma_iommu_mapping owning the domain and the IOVA bookkeeping. To allow
the IOMMU core to own DMA domains as is the case with other
architectures, we have to allow that code to allocate the domain and get
the cookie from the ARM DMA code.

Implement ARM versions of iommu_get/put_dma_cookie that create a struct
dma_iommu_mapping that doesn't own its domain, as well as
iommu_setup/teardown_dma_ops that set the device's archdata.mapping,
allowing ARM DMA ops to seamlessly work on these devices.

For core-owned DMA domains, the IOVA bitmap is allocated at
iommu_setup_dma_ops, after the IOMMU driver has been able to establish
domain geometry, matching arm64.

The code is protected by CONFIG_ARM_DMA_USE_IOMMU and mutually exclusive
with the same-named functions in the dma-iommu code.

arm_iommu_* APIs stay functional with DMA default domains, restoring the
default domain when detaching an explicitly created mapping.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 arch/arm/include/asm/dma-iommu.h |   9 +++
 arch/arm/mm/dma-mapping.c        | 145 ++++++++++++++++++++++++++++++++++++---
 drivers/iommu/dma-iommu.h        |  50 +++++++++-----
 3 files changed, 177 insertions(+), 27 deletions(-)

diff --git a/arch/arm/include/asm/dma-iommu.h b/arch/arm/include/asm/dma-iommu.h
index 2ce4c5683e6d..d8d37980794f 100644
--- a/arch/arm/include/asm/dma-iommu.h
+++ b/arch/arm/include/asm/dma-iommu.h
@@ -32,5 +32,14 @@ int arm_iommu_attach_device(struct device *dev,
 					struct dma_iommu_mapping *mapping);
 void arm_iommu_detach_device(struct device *dev);
 
+#ifdef CONFIG_ARM_DMA_USE_IOMMU
+struct iommu_domain;
+
+void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain);
+void iommu_teardown_dma_ops(struct device *dev);
+int iommu_get_dma_cookie(struct iommu_domain *domain);
+void iommu_put_dma_cookie(struct iommu_domain *domain);
+#endif
+
 #endif /* __KERNEL__ */
 #endif
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 7ef7c268c583..48b9a9f26095 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -1521,6 +1521,96 @@ static void __arm_iommu_free_mapping(struct dma_iommu_mapping *mapping)
 	kfree(mapping);
 }
 
+/**
+ * iommu_get_dma_cookie - Allocate an ARM DMA domain cookie
+ * @domain: the domain
+ */
+int iommu_get_dma_cookie(struct iommu_domain *domain)
+{
+	struct dma_iommu_mapping *mapping;
+
+	if (domain->cookie_type != IOMMU_COOKIE_NONE)
+		return -EEXIST;
+
+	mapping = kzalloc_obj(*mapping);
+	if (!mapping)
+		return -ENOMEM;
+
+	mapping->domain = domain;
+	spin_lock_init(&mapping->lock);
+	domain->cookie_type = IOMMU_COOKIE_ARM_DMA;
+	domain->arm_cookie = mapping;
+
+	return 0;
+}
+
+/**
+ * iommu_put_dma_cookie - Free domain cookie for ARM IOMMU-based DMA API
+ * @domain: the domain being freed
+ */
+void iommu_put_dma_cookie(struct iommu_domain *domain)
+{
+	__arm_iommu_free_mapping(domain->arm_cookie);
+}
+
+/**
+ * iommu_setup_dma_ops - Initialize device's DMA domain cookie and ops
+ * @dev: the device
+ * @domain: the group's new default domain
+ *
+ * Allocates IOVA bookkeeping based on the domain's geometry, which must be
+ * initialized by attaching the domain first. The aperture is limited to within
+ * the 32-bit DMA address space.
+ */
+void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain)
+{
+	const struct iommu_domain_geometry *geo = &domain->geometry;
+	struct dma_iommu_mapping *mapping;
+	u64 end, size;
+
+	dev->archdata.dma_ops_setup = false;
+
+	/* Bail out for non-DMA default domains */
+	if (!iommu_is_dma_domain(domain)) {
+		to_dma_iommu_mapping(dev) = NULL;
+		set_dma_ops(dev, NULL);
+		return;
+	}
+
+	mapping = domain->arm_cookie;
+	to_dma_iommu_mapping(dev) = mapping;
+
+	end = min_t(u64, geo->aperture_end, DMA_BIT_MASK(32));
+
+	/* Are we the first in the group to allocate the bitmaps? */
+	if (!mapping->bitmaps) {
+		if (geo->aperture_start > end)
+			goto out_err;
+
+		size = end - geo->aperture_start + 1;
+		if (__arm_iommu_init_mapping(mapping, geo->aperture_start, size))
+			goto out_err;
+	}
+
+	set_dma_ops(dev, &iommu_ops);
+
+	return;
+
+out_err:
+	set_dma_ops(dev, NULL);
+	dev_warn(dev, "Failed to initialize IOMMU DMA mapping; retaining platform DMA ops\n");
+}
+
+/**
+ * iommu_teardown_dma_ops - Clear a device's DMA domain cookie and ops
+ * @dev: the device
+ */
+void iommu_teardown_dma_ops(struct device *dev)
+{
+	to_dma_iommu_mapping(dev) = NULL;
+	set_dma_ops(dev, NULL);
+}
+
 /**
  * arm_iommu_create_mapping
  * @dev: pointer to the client device (for IOMMU calls)
@@ -1531,6 +1621,9 @@ static void __arm_iommu_free_mapping(struct dma_iommu_mapping *mapping)
  * IO address ranges, which is required to perform memory allocation and
  * mapping with IOMMU aware functions.
  *
+ * Unlike a default domain set up by the IOMMU core, the returned mapping owns
+ * the domain it describes and releases it along with the last reference.
+ *
  * The client device need to be attached to the mapping with
  * arm_iommu_attach_device function.
  */
@@ -1594,8 +1687,10 @@ static int extend_iommu_mapping(struct dma_iommu_mapping *mapping)
 
 void arm_iommu_release_mapping(struct dma_iommu_mapping *mapping)
 {
-	if (mapping)
-		kref_put(&mapping->kref, release_iommu_mapping);
+	if (!mapping || iommu_is_dma_domain(mapping->domain))
+		return;
+
+	kref_put(&mapping->kref, release_iommu_mapping);
 }
 EXPORT_SYMBOL_GPL(arm_iommu_release_mapping);
 
@@ -1604,11 +1699,17 @@ static int __arm_iommu_attach_device(struct device *dev,
 {
 	int err;
 
+	/* Avoid attaching if IOVA bookkeeping initialization failed earlier. */
+	if (!mapping->bitmaps)
+		return -EINVAL;
+
 	err = iommu_attach_device(mapping->domain, dev);
 	if (err)
 		return err;
 
-	kref_get(&mapping->kref);
+	if (!iommu_is_dma_domain(mapping->domain))
+		kref_get(&mapping->kref);
+
 	to_dma_iommu_mapping(dev) = mapping;
 
 	pr_debug("Attached IOMMU controller to %s device.\n", dev_name(dev));
@@ -1646,12 +1747,14 @@ EXPORT_SYMBOL_GPL(arm_iommu_attach_device);
  * arm_iommu_detach_device
  * @dev: valid struct device pointer
  *
- * Detaches the provided device from a previously attached map.
- * This overwrites the dma_ops pointer with appropriate non-IOMMU ops.
+ * Detaches an explicitly attached ARM DMA domain and restores the default
+ * domain's DMA cookie and ops. Does nothing if already using a DMA default
+ * domain.
  */
 void arm_iommu_detach_device(struct device *dev)
 {
 	struct dma_iommu_mapping *mapping;
+	struct iommu_domain *def_domain;
 
 	mapping = to_dma_iommu_mapping(dev);
 	if (!mapping) {
@@ -1659,10 +1762,25 @@ void arm_iommu_detach_device(struct device *dev)
 		return;
 	}
 
+	/*
+	 * Allow callers who think they're detaching the legacy UNMANAGED
+	 * domain to work transparently.
+	 */
+	if (iommu_is_dma_domain(mapping->domain))
+		return;
+
 	iommu_detach_device(mapping->domain, dev);
 	kref_put(&mapping->kref, release_iommu_mapping);
-	to_dma_iommu_mapping(dev) = NULL;
-	set_dma_ops(dev, NULL);
+
+	def_domain = iommu_get_domain_for_dev(dev);
+	if (iommu_is_dma_domain(def_domain)) {
+		to_dma_iommu_mapping(dev) = def_domain->arm_cookie;
+		/* Restore NULL ops if default domain initialization failed. */
+		set_dma_ops(dev, def_domain->arm_cookie->bitmaps ? &iommu_ops : NULL);
+	} else {
+		to_dma_iommu_mapping(dev) = NULL;
+		set_dma_ops(dev, NULL);
+	}
 
 	pr_debug("Detached IOMMU controller from %s device.\n", dev_name(dev));
 }
@@ -1717,6 +1835,8 @@ static void arm_teardown_iommu_dma_ops(struct device *dev) { }
 
 void arch_setup_dma_ops(struct device *dev, bool coherent)
 {
+	struct dma_iommu_mapping *mapping;
+
 	/*
 	 * Due to legacy code that sets the dma_coherent flag from a bus
 	 * notifier we can't just assign coherent to the dma_coherent flag
@@ -1727,10 +1847,14 @@ void arch_setup_dma_ops(struct device *dev, bool coherent)
 		dev_set_dma_coherent(dev);
 
 	/*
-	 * Don't override the dma_ops if they have already been set. Ideally
-	 * this should be the only location where dma_ops are set, remove this
-	 * check when all other callers of set_dma_ops will have disappeared.
+	 * For DMA default domains, iommu_setup_dma_ops manages the DMA ops.
+	 * Don't set them here.
 	 */
+	mapping = to_dma_iommu_mapping(dev);
+	if (mapping && iommu_is_dma_domain(mapping->domain))
+		return;
+
+	/* Preserve DMA ops installed by another owner. */
 	if (dev->dma_ops)
 		return;
 
@@ -1749,6 +1873,7 @@ void arch_teardown_dma_ops(struct device *dev)
 	arm_teardown_iommu_dma_ops(dev);
 	/* Let arch_setup_dma_ops() start again from scratch upon re-probe */
 	set_dma_ops(dev, NULL);
+	dev->archdata.dma_ops_setup = false;
 }
 
 void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
diff --git a/drivers/iommu/dma-iommu.h b/drivers/iommu/dma-iommu.h
index 349d6a57e5a0..71b23cb698c0 100644
--- a/drivers/iommu/dma-iommu.h
+++ b/drivers/iommu/dma-iommu.h
@@ -7,25 +7,24 @@
 
 #include <linux/iommu.h>
 
-#ifdef CONFIG_IOMMU_DMA
+/*
+ * Both dma-iommu.c and arch/arm/mm/dma-mapping.c implement functions to set up
+ * IOMMU_DOMAIN_DMA default domains. Only one provider can be compiled in at a
+ * time.
+ */
+#if defined(CONFIG_ARM_DMA_USE_IOMMU)
+
+#include <asm/dma-iommu.h>
+
+#elif defined(CONFIG_IOMMU_DMA)
 
 void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain);
 void iommu_teardown_dma_ops(struct device *dev);
 
 int iommu_get_dma_cookie(struct iommu_domain *domain);
 void iommu_put_dma_cookie(struct iommu_domain *domain);
-void iommu_put_msi_cookie(struct iommu_domain *domain);
 
-int iommu_dma_init_fq(struct iommu_domain *domain);
-
-void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
-
-int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
-		     phys_addr_t msi_addr);
-
-extern bool iommu_dma_forcedac;
-
-#else /* CONFIG_IOMMU_DMA */
+#else
 
 static inline void iommu_setup_dma_ops(struct device *dev,
 				       struct iommu_domain *domain)
@@ -36,11 +35,6 @@ static inline void iommu_teardown_dma_ops(struct device *dev)
 {
 }
 
-static inline int iommu_dma_init_fq(struct iommu_domain *domain)
-{
-	return -EINVAL;
-}
-
 static inline int iommu_get_dma_cookie(struct iommu_domain *domain)
 {
 	return -ENODEV;
@@ -50,6 +44,28 @@ static inline void iommu_put_dma_cookie(struct iommu_domain *domain)
 {
 }
 
+#endif
+
+#ifdef CONFIG_IOMMU_DMA
+
+void iommu_put_msi_cookie(struct iommu_domain *domain);
+
+int iommu_dma_init_fq(struct iommu_domain *domain);
+
+void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
+
+int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
+		     phys_addr_t msi_addr);
+
+extern bool iommu_dma_forcedac;
+
+#else /* CONFIG_IOMMU_DMA */
+
+static inline int iommu_dma_init_fq(struct iommu_domain *domain)
+{
+	return -EINVAL;
+}
+
 static inline void iommu_put_msi_cookie(struct iommu_domain *domain)
 {
 }

-- 
2.55.0

[PATCH v3 05/10] ARM: dma-mapping: Bypass the legacy path for core-owned DMA API domains

[Mikko Perttunen]

ARM's arch_setup_dma_ops / arm_setup_iommu_dma_ops creates a custom
IOMMU domain for DMA API usage, and continues to do so when an IDENTITY
default domain type is selected.

When a DMA default domain is requested, however, the core owns the
domain and we need to skip our custom domain management. Add checks to
arm_setup/teardown_iommu_dma_ops for these cases.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 arch/arm/mm/dma-mapping.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 48b9a9f26095..b82cf58f8bc7 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -1788,9 +1788,12 @@ EXPORT_SYMBOL_GPL(arm_iommu_detach_device);
 
 static void arm_setup_iommu_dma_ops(struct device *dev)
 {
-	struct dma_iommu_mapping *mapping;
+	struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
 	u64 dma_base = 0, size = 1ULL << 32;
 
+	if (mapping)
+		return;
+
 	if (dev->dma_range_map) {
 		dma_base = dma_range_map_min(dev->dma_range_map);
 		size = dma_range_map_max(dev->dma_range_map) - dma_base;
@@ -1816,7 +1819,7 @@ static void arm_teardown_iommu_dma_ops(struct device *dev)
 {
 	struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
 
-	if (!mapping)
+	if (!mapping || iommu_is_dma_domain(mapping->domain))
 		return;
 
 	arm_iommu_detach_device(dev);

-- 
2.55.0

[PATCH v3 06/10] ARM: dma-mapping: Bound IOVA allocation by device DMA limits

[Mikko Perttunen]

The legacy unmanaged DMA domain path sets up the IOVA bookkeeping based
on the dma_range_map of the device. This property however is not
necessarily available during the time we create the bookkeeping with
core-owned default domains, and doesn't take into account the device's
DMA mask.

Resolve this by passing the device's DMA limit to __alloc_iova and
refusing allocations that would go above that limit. Allocation logic is
unchanged except for the check.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 arch/arm/mm/dma-mapping.c | 36 ++++++++++++++++++++----------------
 1 file changed, 20 insertions(+), 16 deletions(-)

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index b82cf58f8bc7..3703f9e2c9a2 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -752,7 +752,7 @@ static int __dma_info_to_prot(enum dma_data_direction dir, unsigned long attrs)
 static int extend_iommu_mapping(struct dma_iommu_mapping *mapping);
 
 static inline dma_addr_t __alloc_iova(struct dma_iommu_mapping *mapping,
-				      size_t size)
+				      size_t size, u64 dma_limit)
 {
 	unsigned int order = get_order(size);
 	unsigned int align = 0;
@@ -776,7 +776,6 @@ static inline dma_addr_t __alloc_iova(struct dma_iommu_mapping *mapping,
 		if (start >= mapping->bits)
 			continue;
 
-		bitmap_set(mapping->bitmaps[i], start, count);
 		break;
 	}
 
@@ -786,27 +785,28 @@ static inline dma_addr_t __alloc_iova(struct dma_iommu_mapping *mapping,
 	 * address range of size bytes.
 	 */
 	if (i == mapping->nr_bitmaps) {
-		if (extend_iommu_mapping(mapping)) {
-			spin_unlock_irqrestore(&mapping->lock, flags);
-			return DMA_MAPPING_ERROR;
-		}
+		if (extend_iommu_mapping(mapping))
+			goto fail;
 
 		start = bitmap_find_next_zero_area(mapping->bitmaps[i],
 				mapping->bits, 0, count, align);
 
-		if (start >= mapping->bits) {
-			spin_unlock_irqrestore(&mapping->lock, flags);
-			return DMA_MAPPING_ERROR;
-		}
-
-		bitmap_set(mapping->bitmaps[i], start, count);
+		if (start >= mapping->bits)
+			goto fail;
 	}
-	spin_unlock_irqrestore(&mapping->lock, flags);
 
 	iova = mapping->base + (mapping_size * i);
 	iova += start << PAGE_SHIFT;
+	if ((u64)iova + PAGE_ALIGN(size) - 1 > dma_limit)
+		goto fail;
 
+	bitmap_set(mapping->bitmaps[i], start, count);
+	spin_unlock_irqrestore(&mapping->lock, flags);
 	return iova;
+
+fail:
+	spin_unlock_irqrestore(&mapping->lock, flags);
+	return DMA_MAPPING_ERROR;
 }
 
 static inline void __free_iova(struct dma_iommu_mapping *mapping,
@@ -965,7 +965,8 @@ __iommu_create_mapping(struct device *dev, struct page **pages, size_t size,
 	dma_addr_t dma_addr, iova;
 	int i;
 
-	dma_addr = __alloc_iova(mapping, size);
+	dma_addr = __alloc_iova(mapping, size,
+				min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit));
 	if (dma_addr == DMA_MAPPING_ERROR)
 		return dma_addr;
 
@@ -1194,7 +1195,9 @@ static int __map_sg_chunk(struct device *dev, struct scatterlist *sg,
 	size = PAGE_ALIGN(size);
 	*handle = DMA_MAPPING_ERROR;
 
-	iova_base = iova = __alloc_iova(mapping, size);
+	iova = __alloc_iova(mapping, size,
+			    min_not_zero(dma_get_mask(dev), dev->bus_dma_limit));
+	iova_base = iova;
 	if (iova == DMA_MAPPING_ERROR)
 		return -ENOMEM;
 
@@ -1375,7 +1378,8 @@ static dma_addr_t arm_iommu_map_phys(struct device *dev, phys_addr_t phys,
 	    !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO)))
 		arch_sync_dma_for_device(phys, size, dir);
 
-	dma_addr = __alloc_iova(mapping, len);
+	dma_addr = __alloc_iova(mapping, len,
+				min_not_zero(dma_get_mask(dev), dev->bus_dma_limit));
 	if (dma_addr == DMA_MAPPING_ERROR)
 		return dma_addr;
 

-- 
2.55.0

[PATCH v3 07/10] iommu: Enable DMA default domains on (most) 32-bit ARM platforms

[Mikko Perttunen]

iommu_get_default_domain_type currently forces an IDENTITY domain type
with CONFIG_ARM_DMA_USE_IOMMU because the ARM DMA implementation
traditionally installs its own UNMANAGED domains.

The ARM DMA implementation now supports core-owned DMA default domains,
so remove that override and follow the same logic as CONFIG_IOMMU_DMA
platforms. The static assertion making CONFIG_ARM_DMA_USE_IOMMU and
CONFIG_IOMMU_DMA mutually exclusive is removed as both now implement the
same callbacks making the build fail anyway.

Based on LLM eyeballing, some platforms are still not ready to be
switched over. For these platforms, a def_domain_type callback returning
IDENTITY is added unless it was already there. Specifically,

- mtk_iommu_v1 only supports the domain it internally creates via the
  arm_iommu API
- ipmmu-vmsa creates its own shared ARM DMA domain. Also creating a
  default DMA domain would exhaust contexts on configurations with only
  one context
- omap-iommu among other things due to the omap remoteproc driver,
  which relies on detaching the legacy ARM DMA domain to restore direct
  DMA ops
- msm_iommu expects previous context assignments to be released
  before attaching another paging domain

These drivers can be migrated later.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 drivers/iommu/iommu.c        | 20 ++++----------------
 drivers/iommu/ipmmu-vmsa.c   |  6 ++++++
 drivers/iommu/msm_iommu.c    |  6 ++++++
 drivers/iommu/mtk_iommu_v1.c |  6 ++++++
 drivers/iommu/omap-iommu.c   |  6 ++++++
 5 files changed, 28 insertions(+), 16 deletions(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 5fbb76d44a1e..e47f95fea44d 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1884,18 +1884,6 @@ static int iommu_get_default_domain_type(struct iommu_group *group,
 
 	lockdep_assert_held(&group->mutex);
 
-	/*
-	 * ARM32 drivers supporting CONFIG_ARM_DMA_USE_IOMMU can declare an
-	 * identity_domain and it will automatically become their default
-	 * domain. Later on ARM_DMA_USE_IOMMU will install its UNMANAGED domain.
-	 * Override the selection to IDENTITY.
-	 */
-	if (IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)) {
-		static_assert(!(IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU) &&
-				IS_ENABLED(CONFIG_IOMMU_DMA)));
-		driver_type = IOMMU_DOMAIN_IDENTITY;
-	}
-
 	for_each_group_device(group, gdev) {
 		driver_type = iommu_get_def_domain_type(group, gdev->dev,
 							driver_type);
@@ -1912,11 +1900,11 @@ static int iommu_get_default_domain_type(struct iommu_group *group,
 	}
 
 	/*
-	 * If the common dma ops are not selected in kconfig then we cannot use
-	 * IOMMU_DOMAIN_DMA at all. Force IDENTITY if nothing else has been
-	 * selected.
+	 * Without an IOMMU DMA implementation, force IDENTITY if the driver
+	 * has not selected a domain type.
 	 */
-	if (!IS_ENABLED(CONFIG_IOMMU_DMA)) {
+	if (!IS_ENABLED(CONFIG_IOMMU_DMA) &&
+	    !IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)) {
 		if (WARN_ON(driver_type == IOMMU_DOMAIN_DMA))
 			return -1;
 		if (!driver_type)
diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c
index 9386b752dea2..39dddc3467a2 100644
--- a/drivers/iommu/ipmmu-vmsa.c
+++ b/drivers/iommu/ipmmu-vmsa.c
@@ -873,8 +873,14 @@ static void ipmmu_release_device(struct device *dev)
 	arm_iommu_release_mapping(mmu->mapping);
 }
 
+static int ipmmu_def_domain_type(struct device *dev)
+{
+	return IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU) ? IOMMU_DOMAIN_IDENTITY : 0;
+}
+
 static const struct iommu_ops ipmmu_ops = {
 	.identity_domain = &ipmmu_iommu_identity_domain,
+	.def_domain_type = ipmmu_def_domain_type,
 	.domain_alloc_paging = ipmmu_domain_alloc_paging,
 	.probe_device = ipmmu_probe_device,
 	.release_device = ipmmu_release_device,
diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c
index 4ef1e8c8244b..7d7e91ffbf68 100644
--- a/drivers/iommu/msm_iommu.c
+++ b/drivers/iommu/msm_iommu.c
@@ -691,8 +691,14 @@ irqreturn_t msm_iommu_fault_handler(int irq, void *dev_id)
 	return 0;
 }
 
+static int msm_iommu_def_domain_type(struct device *dev)
+{
+	return IOMMU_DOMAIN_IDENTITY;
+}
+
 static struct iommu_ops msm_iommu_ops = {
 	.identity_domain = &msm_iommu_identity_domain,
+	.def_domain_type = msm_iommu_def_domain_type,
 	.domain_alloc_paging = msm_iommu_domain_alloc_paging,
 	.probe_device = msm_iommu_probe_device,
 	.device_group = generic_device_group,
diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c
index e907c9953142..8d8235f55050 100644
--- a/drivers/iommu/mtk_iommu_v1.c
+++ b/drivers/iommu/mtk_iommu_v1.c
@@ -578,8 +578,14 @@ static int mtk_iommu_v1_hw_init(const struct mtk_iommu_v1_data *data)
 	return 0;
 }
 
+static int mtk_iommu_v1_def_domain_type(struct device *dev)
+{
+	return IOMMU_DOMAIN_IDENTITY;
+}
+
 static const struct iommu_ops mtk_iommu_v1_ops = {
 	.identity_domain = &mtk_iommu_v1_identity_domain,
+	.def_domain_type = mtk_iommu_v1_def_domain_type,
 	.domain_alloc_paging = mtk_iommu_v1_domain_alloc_paging,
 	.probe_device	= mtk_iommu_v1_probe_device,
 	.probe_finalize = mtk_iommu_v1_probe_finalize,
diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c
index 8231d7d6bb6a..56cdc4c670c0 100644
--- a/drivers/iommu/omap-iommu.c
+++ b/drivers/iommu/omap-iommu.c
@@ -1712,8 +1712,14 @@ static int omap_iommu_of_xlate(struct device *dev, const struct of_phandle_args
 	return 0;
 }
 
+static int omap_iommu_def_domain_type(struct device *dev)
+{
+	return IOMMU_DOMAIN_IDENTITY;
+}
+
 static const struct iommu_ops omap_iommu_ops = {
 	.identity_domain = &omap_iommu_identity_domain,
+	.def_domain_type = omap_iommu_def_domain_type,
 	.domain_alloc_paging = omap_iommu_domain_alloc_paging,
 	.probe_device	= omap_iommu_probe_device,
 	.release_device	= omap_iommu_release_device,

-- 
2.55.0

[PATCH v3 08/10] drm/tegra: Fix device parameter to dma_get_sgtable

[Mikko Perttunen]

The device parameter to dma_get_sgtable should match the allocating
device, not the consuming device. Fix the calls.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 drivers/gpu/drm/tegra/gem.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c
index f76af733ea79..c4241ce8f71c 100644
--- a/drivers/gpu/drm/tegra/gem.c
+++ b/drivers/gpu/drm/tegra/gem.c
@@ -125,7 +125,8 @@ static struct host1x_bo_mapping *tegra_bo_pin(struct device *dev, struct host1x_
 		 * not imported, it had to be allocated with the DMA API, so
 		 * the DMA API helper can be used.
 		 */
-		err = dma_get_sgtable(dev, map->sgt, obj->vaddr, obj->iova, gem->size);
+		err = dma_get_sgtable(gem->dev->dev, map->sgt, obj->vaddr,
+				      obj->iova, gem->size);
 		if (err < 0)
 			goto free;
 	}
@@ -643,7 +644,7 @@ tegra_gem_prime_map_dma_buf(struct dma_buf_attachment *attach,
 					      0, gem->size, GFP_KERNEL) < 0)
 			goto free;
 	} else {
-		if (dma_get_sgtable(attach->dev, sgt, bo->vaddr, bo->iova,
+		if (dma_get_sgtable(gem->dev->dev, sgt, bo->vaddr, bo->iova,
 				    gem->size) < 0)
 			goto free;
 	}

-- 
2.55.0

[PATCH v3 09/10] iommu/tegra-smmu: Defer display client translation to driver handover

[Mikko Perttunen]

Bootloaders may leave display controllers scanning out a boot splash
while the kernel boots. Enabling translation for them without special
care during that time would cause translation faults.

Because of this, tegra_smmu_def_domain_type has kept every device on an
identity domain at boot, and TegraDRM has parallel custom IOMMU domain
management code paths for these clients.

Mark the dc and dcb swgroups on Tegra30, Tegra114, Tegra124 and Tegra210
with a new defer_enable flag. tegra_smmu_enable() sets those swgroups up
as usual, ASID and per-client enables and all, but leaves
SMMU_ASID_ENABLE clear, so the clients stay in bypass. tegra_dc_probe()
then calls tegra_smmu_enable_translation() once it has asserted the
display controller's reset (and set up its final IOMMU domain), at which
point the hardware is guaranteed not to be fetching and translation can
be turned on.

If the display driver never probes, the clients simply stay in bypass,
which is what they do today.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 drivers/gpu/drm/tegra/dc.c      |  8 +++++++
 drivers/iommu/tegra-smmu.c      | 51 ++++++++++++++++++++++++++++++++++++++++-
 drivers/memory/tegra/tegra114.c |  6 +++--
 drivers/memory/tegra/tegra124.c |  6 +++--
 drivers/memory/tegra/tegra210.c |  6 +++--
 drivers/memory/tegra/tegra30.c  |  6 +++--
 include/soc/tegra/mc.h          | 13 +++++++++++
 7 files changed, 87 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/tegra/dc.c b/drivers/gpu/drm/tegra/dc.c
index fefc3761a4bc..caf898e3db00 100644
--- a/drivers/gpu/drm/tegra/dc.c
+++ b/drivers/gpu/drm/tegra/dc.c
@@ -19,6 +19,7 @@
 #include <linux/reset.h>
 
 #include <soc/tegra/common.h>
+#include <soc/tegra/mc.h>
 #include <soc/tegra/pmc.h>
 
 #include <drm/drm_atomic.h>
@@ -2757,6 +2758,13 @@ static int tegra_dc_init(struct host1x_client *client)
 		return err;
 	}
 
+	/*
+	 * Enable translation now that the final domain is attached.
+	 * The hardware was reset at probe and has not been programmed
+	 * since.
+	 */
+	tegra_smmu_enable_translation(client->dev);
+
 	if (dc->soc->wgrps)
 		primary = tegra_dc_add_shared_planes(drm, dc);
 	else
diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c
index 67e7a7b925f0..6779528a9e7c 100644
--- a/drivers/iommu/tegra-smmu.c
+++ b/drivers/iommu/tegra-smmu.c
@@ -368,7 +368,13 @@ static void tegra_smmu_enable(struct tegra_smmu *smmu, unsigned int swgroup,
 		value = smmu_readl(smmu, group->reg);
 		value &= ~SMMU_ASID_MASK;
 		value |= SMMU_ASID_VALUE(asid);
-		value |= SMMU_ASID_ENABLE;
+		/*
+		 * A deferred client is set up completely, but with translation
+		 * gated off, so it stays in bypass until its driver calls
+		 * tegra_smmu_enable_translation().
+		 */
+		if (!group->defer_enable)
+			value |= SMMU_ASID_ENABLE;
 		smmu_writel(smmu, value, group->reg);
 	} else {
 		pr_warn("%s group from swgroup %u not found\n", __func__,
@@ -1190,3 +1196,46 @@ void tegra_smmu_remove(struct tegra_smmu *smmu)
 	if (IS_ENABLED(CONFIG_DEBUG_FS))
 		tegra_smmu_debugfs_exit(smmu);
 }
+
+/**
+ * tegra_smmu_enable_translation - Release a client's swgroups to translation
+ * @dev: memory client device, already quiesced by its driver
+ *
+ * Enables translation for swgroups which were left in bypass at attach time
+ * because the client could still have been running from a pre-kernel
+ * configuration. Must be called after @dev has been attached to the domain it
+ * will use. Safe to call for any device, including ones behind other IOMMUs;
+ * clients which were not deferred are unaffected.
+ */
+void tegra_smmu_enable_translation(struct device *dev)
+{
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct iommu_domain *domain;
+	struct tegra_smmu *smmu;
+	unsigned int i;
+
+	domain = iommu_get_domain_for_dev(dev);
+	if (!fwspec || !domain || domain->owner != &tegra_smmu_ops ||
+	    !(domain->type & __IOMMU_DOMAIN_PAGING))
+		return;
+
+	smmu = dev_iommu_priv_get(dev);
+
+	mutex_lock(&smmu->lock);
+
+	for (i = 0; i < fwspec->num_ids; i++) {
+		const struct tegra_smmu_swgroup *group;
+		u32 value;
+
+		group = tegra_smmu_find_swgroup(smmu, fwspec->ids[i]);
+		if (!group || !group->defer_enable)
+			continue;
+
+		value = smmu_readl(smmu, group->reg);
+		value |= SMMU_ASID_ENABLE;
+		smmu_writel(smmu, value, group->reg);
+	}
+
+	mutex_unlock(&smmu->lock);
+}
+EXPORT_SYMBOL_GPL(tegra_smmu_enable_translation);
diff --git a/drivers/memory/tegra/tegra114.c b/drivers/memory/tegra/tegra114.c
index 2cb153091228..2883c4f91144 100644
--- a/drivers/memory/tegra/tegra114.c
+++ b/drivers/memory/tegra/tegra114.c
@@ -1028,8 +1028,10 @@ static const struct tegra_mc_client tegra114_mc_clients[] = {
 };
 
 static const struct tegra_smmu_swgroup tegra114_swgroups[] = {
-	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240 },
-	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244 },
+	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240,
+	  .defer_enable = true },
+	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244,
+	  .defer_enable = true },
 	{ .name = "epp",       .swgroup = TEGRA_SWGROUP_EPP,       .reg = 0x248 },
 	{ .name = "g2",        .swgroup = TEGRA_SWGROUP_G2,        .reg = 0x24c },
 	{ .name = "avpc",      .swgroup = TEGRA_SWGROUP_AVPC,      .reg = 0x23c },
diff --git a/drivers/memory/tegra/tegra124.c b/drivers/memory/tegra/tegra124.c
index df87c5038625..a38fbf2baf6c 100644
--- a/drivers/memory/tegra/tegra124.c
+++ b/drivers/memory/tegra/tegra124.c
@@ -1070,8 +1070,10 @@ static const struct tegra_mc_client tegra124_mc_clients[] = {
 };
 
 static const struct tegra_smmu_swgroup tegra124_swgroups[] = {
-	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240 },
-	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244 },
+	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240,
+	  .defer_enable = true },
+	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244,
+	  .defer_enable = true },
 	{ .name = "afi",       .swgroup = TEGRA_SWGROUP_AFI,       .reg = 0x238 },
 	{ .name = "avpc",      .swgroup = TEGRA_SWGROUP_AVPC,      .reg = 0x23c },
 	{ .name = "hda",       .swgroup = TEGRA_SWGROUP_HDA,       .reg = 0x254 },
diff --git a/drivers/memory/tegra/tegra210.c b/drivers/memory/tegra/tegra210.c
index f58f3ef6f681..92e809cd7c26 100644
--- a/drivers/memory/tegra/tegra210.c
+++ b/drivers/memory/tegra/tegra210.c
@@ -1166,8 +1166,10 @@ static const struct tegra_mc_client tegra210_mc_clients[] = {
 static const struct tegra_smmu_swgroup tegra210_swgroups[] = {
 	{ .name = "afi",       .swgroup = TEGRA_SWGROUP_AFI,       .reg = 0x238 },
 	{ .name = "avpc",      .swgroup = TEGRA_SWGROUP_AVPC,      .reg = 0x23c },
-	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240 },
-	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244 },
+	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240,
+	  .defer_enable = true },
+	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244,
+	  .defer_enable = true },
 	{ .name = "hc",        .swgroup = TEGRA_SWGROUP_HC,        .reg = 0x250 },
 	{ .name = "hda",       .swgroup = TEGRA_SWGROUP_HDA,       .reg = 0x254 },
 	{ .name = "isp2",      .swgroup = TEGRA_SWGROUP_ISP2,      .reg = 0x258 },
diff --git a/drivers/memory/tegra/tegra30.c b/drivers/memory/tegra/tegra30.c
index 8389e3af0121..0ee4715413ba 100644
--- a/drivers/memory/tegra/tegra30.c
+++ b/drivers/memory/tegra/tegra30.c
@@ -1139,8 +1139,10 @@ static const struct tegra_mc_client tegra30_mc_clients[] = {
 };
 
 static const struct tegra_smmu_swgroup tegra30_swgroups[] = {
-	{ .name = "dc",   .swgroup = TEGRA_SWGROUP_DC,   .reg = 0x240 },
-	{ .name = "dcb",  .swgroup = TEGRA_SWGROUP_DCB,  .reg = 0x244 },
+	{ .name = "dc",   .swgroup = TEGRA_SWGROUP_DC,   .reg = 0x240,
+	  .defer_enable = true },
+	{ .name = "dcb",  .swgroup = TEGRA_SWGROUP_DCB,  .reg = 0x244,
+	  .defer_enable = true },
 	{ .name = "epp",  .swgroup = TEGRA_SWGROUP_EPP,  .reg = 0x248 },
 	{ .name = "g2",   .swgroup = TEGRA_SWGROUP_G2,   .reg = 0x24c },
 	{ .name = "mpe",  .swgroup = TEGRA_SWGROUP_MPE,  .reg = 0x264 },
diff --git a/include/soc/tegra/mc.h b/include/soc/tegra/mc.h
index 25d465d70493..ea67972fd585 100644
--- a/include/soc/tegra/mc.h
+++ b/include/soc/tegra/mc.h
@@ -70,6 +70,14 @@ struct tegra_smmu_swgroup {
 	const char *name;
 	unsigned int swgroup;
 	unsigned int reg;
+
+	/*
+	 * Set for clients which may already be running when the kernel takes
+	 * over, e.g. display controllers scanning out a boot splash. They are
+	 * left in bypass until their driver calls
+	 * tegra_smmu_enable_translation().
+	 */
+	bool defer_enable;
 };
 
 struct tegra_smmu_group_soc {
@@ -103,6 +111,7 @@ struct tegra_smmu *tegra_smmu_probe(struct device *dev,
 				    const struct tegra_smmu_soc *soc,
 				    struct tegra_mc *mc);
 void tegra_smmu_remove(struct tegra_smmu *smmu);
+void tegra_smmu_enable_translation(struct device *dev);
 #else
 static inline struct tegra_smmu *
 tegra_smmu_probe(struct device *dev, const struct tegra_smmu_soc *soc,
@@ -114,6 +123,10 @@ tegra_smmu_probe(struct device *dev, const struct tegra_smmu_soc *soc,
 static inline void tegra_smmu_remove(struct tegra_smmu *smmu)
 {
 }
+
+static inline void tegra_smmu_enable_translation(struct device *dev)
+{
+}
 #endif
 
 struct tegra_mc_reset {

-- 
2.55.0

[PATCH v3 10/10] iommu/tegra-smmu: Use a DMA API default domain per swgroup

[Mikko Perttunen]

Everything is now in place for the Tegra SMMU to give devices a real
IOMMU_DOMAIN_DMA default domain.

Some device drivers still rely on explicit domains (and being able to
detach a legacy ARM DMA domain if any), so DMA default domains are
enabled on a per-swgroup basis via a use_dma_api flag. As drivers are
converted to DMA API usage we can eventually get rid of this. Initially
the flag is enabled for TegraDRM and Host1x devices.

On multi_v7_defconfig (where CONFIG_ARM_DMA_USE_IOMMU was always
enabled), this will fix TegraDRM probing with certain probe orderings
where the legacy implementation's lack of support for multiple devices
in one IOMMU group would cause breakage.

On tegra_defconfig, as this now enables CONFIG_ARM_DMA_USE_IOMMU, all
devices move to either the legacy ARM DMA path or the new one. This
should in principle not cause issues as the drivers would already have
been verified on multi_v7_defconfig.

Also set .release_domain to the identity domain to ensure the AS is
released appropriately now that the domain can be non-static.

Signed-off-by: Mikko Perttunen <mperttunen@nvidia.com>
---
 drivers/iommu/Kconfig           |  1 +
 drivers/iommu/tegra-smmu.c      | 52 ++++++++++++++++++++++++++++++++++++-----
 drivers/memory/tegra/tegra114.c | 13 +++++++----
 drivers/memory/tegra/tegra124.c | 10 ++++----
 drivers/memory/tegra/tegra30.c  | 16 ++++++++-----
 include/soc/tegra/mc.h          |  7 ++++++
 6 files changed, 78 insertions(+), 21 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 6e07bd69467a..817e84e0e6ab 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -252,6 +252,7 @@ config TEGRA_IOMMU_SMMU
 	depends on TEGRA_AHB
 	depends on TEGRA_MC
 	select IOMMU_API
+	select ARM_DMA_USE_IOMMU if ARM
 	help
 	  This driver supports the IOMMU hardware (SMMU) found on NVIDIA Tegra
 	  SoCs (Tegra30 up to Tegra210).
diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c
index 6779528a9e7c..73c6151db8f8 100644
--- a/drivers/iommu/tegra-smmu.c
+++ b/drivers/iommu/tegra-smmu.c
@@ -994,16 +994,20 @@ static int tegra_smmu_of_xlate(struct device *dev,
 
 static int tegra_smmu_def_domain_type(struct device *dev)
 {
-	/*
-	 * FIXME: For now we want to run all translation in IDENTITY mode, due
-	 * to some device quirks. Better would be to just quirk the troubled
-	 * devices.
-	 */
-	return IOMMU_DOMAIN_IDENTITY;
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct tegra_smmu *smmu = dev_iommu_priv_get(dev);
+	const struct tegra_smmu_swgroup *swgroup;
+
+	swgroup = tegra_smmu_find_swgroup(smmu, fwspec->ids[0]);
+	if (WARN_ON(!swgroup) || !swgroup->use_dma_api)
+		return IOMMU_DOMAIN_IDENTITY;
+
+	return IOMMU_DOMAIN_DMA;
 }
 
 static const struct iommu_ops tegra_smmu_ops = {
 	.identity_domain = &tegra_smmu_identity_domain,
+	.release_domain = &tegra_smmu_identity_domain,
 	.def_domain_type = &tegra_smmu_def_domain_type,
 	.domain_alloc_paging = tegra_smmu_domain_alloc_paging,
 	.probe_device = tegra_smmu_probe_device,
@@ -1108,6 +1112,40 @@ static void tegra_smmu_debugfs_exit(struct tegra_smmu *smmu)
 	debugfs_remove_recursive(smmu->debugfs);
 }
 
+/*
+ * The default domain type is a property of the IOMMU group, so all swgroups in
+ * one group must agree on use_dma_api. Otherwise the core sees conflicting
+ * def_domain_type and falls back to identity for the whole group.
+ */
+static void tegra_smmu_check_groups(struct tegra_smmu *smmu)
+{
+	const struct tegra_smmu_soc *soc = smmu->soc;
+	unsigned int i, j;
+
+	for (i = 0; i < soc->num_groups; i++) {
+		const struct tegra_smmu_group_soc *group = &soc->groups[i];
+		int dma_api = -1;
+
+		for (j = 0; j < group->num_swgroups; j++) {
+			const struct tegra_smmu_swgroup *swgrp;
+
+			swgrp = tegra_smmu_find_swgroup(smmu,
+							group->swgroups[j]);
+			if (!swgrp)
+				continue;
+
+			if (dma_api < 0)
+				dma_api = swgrp->use_dma_api;
+			else if (swgrp->use_dma_api != !!dma_api)
+				break;
+		}
+
+		WARN(j < group->num_swgroups,
+		     "SMMU group %s mixes DMA API and identity swgroups\n",
+		     group->name);
+	}
+}
+
 struct tegra_smmu *tegra_smmu_probe(struct device *dev,
 				    const struct tegra_smmu_soc *soc,
 				    struct tegra_mc *mc)
@@ -1142,6 +1180,8 @@ struct tegra_smmu *tegra_smmu_probe(struct device *dev,
 	smmu->dev = dev;
 	smmu->mc = mc;
 
+	tegra_smmu_check_groups(smmu);
+
 	smmu->pfn_mask =
 		BIT_MASK(mc->soc->num_address_bits - SMMU_PTE_SHIFT) - 1;
 	dev_dbg(dev, "address bits: %u, PFN mask: %#lx\n",
diff --git a/drivers/memory/tegra/tegra114.c b/drivers/memory/tegra/tegra114.c
index 2883c4f91144..024c63bdd480 100644
--- a/drivers/memory/tegra/tegra114.c
+++ b/drivers/memory/tegra/tegra114.c
@@ -1029,15 +1029,18 @@ static const struct tegra_mc_client tegra114_mc_clients[] = {
 
 static const struct tegra_smmu_swgroup tegra114_swgroups[] = {
 	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240,
-	  .defer_enable = true },
+	  .defer_enable = true, .use_dma_api = true },
 	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244,
-	  .defer_enable = true },
+	  .defer_enable = true, .use_dma_api = true },
 	{ .name = "epp",       .swgroup = TEGRA_SWGROUP_EPP,       .reg = 0x248 },
-	{ .name = "g2",        .swgroup = TEGRA_SWGROUP_G2,        .reg = 0x24c },
+	{ .name = "g2",        .swgroup = TEGRA_SWGROUP_G2,        .reg = 0x24c,
+	  .use_dma_api = true },
 	{ .name = "avpc",      .swgroup = TEGRA_SWGROUP_AVPC,      .reg = 0x23c },
-	{ .name = "nv",        .swgroup = TEGRA_SWGROUP_NV,        .reg = 0x268 },
+	{ .name = "nv",        .swgroup = TEGRA_SWGROUP_NV,        .reg = 0x268,
+	  .use_dma_api = true },
 	{ .name = "hda",       .swgroup = TEGRA_SWGROUP_HDA,       .reg = 0x254 },
-	{ .name = "hc",        .swgroup = TEGRA_SWGROUP_HC,        .reg = 0x250 },
+	{ .name = "hc",        .swgroup = TEGRA_SWGROUP_HC,        .reg = 0x250,
+	  .use_dma_api = true },
 	{ .name = "msenc",     .swgroup = TEGRA_SWGROUP_MSENC,     .reg = 0x264 },
 	{ .name = "ppcs",      .swgroup = TEGRA_SWGROUP_PPCS,      .reg = 0x270 },
 	{ .name = "vde",       .swgroup = TEGRA_SWGROUP_VDE,       .reg = 0x27c },
diff --git a/drivers/memory/tegra/tegra124.c b/drivers/memory/tegra/tegra124.c
index a38fbf2baf6c..17068aaf163f 100644
--- a/drivers/memory/tegra/tegra124.c
+++ b/drivers/memory/tegra/tegra124.c
@@ -1071,13 +1071,14 @@ static const struct tegra_mc_client tegra124_mc_clients[] = {
 
 static const struct tegra_smmu_swgroup tegra124_swgroups[] = {
 	{ .name = "dc",        .swgroup = TEGRA_SWGROUP_DC,        .reg = 0x240,
-	  .defer_enable = true },
+	  .defer_enable = true, .use_dma_api = true },
 	{ .name = "dcb",       .swgroup = TEGRA_SWGROUP_DCB,       .reg = 0x244,
-	  .defer_enable = true },
+	  .defer_enable = true, .use_dma_api = true },
 	{ .name = "afi",       .swgroup = TEGRA_SWGROUP_AFI,       .reg = 0x238 },
 	{ .name = "avpc",      .swgroup = TEGRA_SWGROUP_AVPC,      .reg = 0x23c },
 	{ .name = "hda",       .swgroup = TEGRA_SWGROUP_HDA,       .reg = 0x254 },
-	{ .name = "hc",        .swgroup = TEGRA_SWGROUP_HC,        .reg = 0x250 },
+	{ .name = "hc",        .swgroup = TEGRA_SWGROUP_HC,        .reg = 0x250,
+	  .use_dma_api = true },
 	{ .name = "msenc",     .swgroup = TEGRA_SWGROUP_MSENC,     .reg = 0x264 },
 	{ .name = "ppcs",      .swgroup = TEGRA_SWGROUP_PPCS,      .reg = 0x270 },
 	{ .name = "sata",      .swgroup = TEGRA_SWGROUP_SATA,      .reg = 0x274 },
@@ -1093,7 +1094,8 @@ static const struct tegra_smmu_swgroup tegra124_swgroups[] = {
 	{ .name = "sdmmc2a",   .swgroup = TEGRA_SWGROUP_SDMMC2A,   .reg = 0xa98 },
 	{ .name = "sdmmc3a",   .swgroup = TEGRA_SWGROUP_SDMMC3A,   .reg = 0xa9c },
 	{ .name = "sdmmc4a",   .swgroup = TEGRA_SWGROUP_SDMMC4A,   .reg = 0xaa0 },
-	{ .name = "vic",       .swgroup = TEGRA_SWGROUP_VIC,       .reg = 0x284 },
+	{ .name = "vic",       .swgroup = TEGRA_SWGROUP_VIC,       .reg = 0x284,
+	  .use_dma_api = true },
 	{ .name = "vi",        .swgroup = TEGRA_SWGROUP_VI,        .reg = 0x280 },
 };
 
diff --git a/drivers/memory/tegra/tegra30.c b/drivers/memory/tegra/tegra30.c
index 0ee4715413ba..df6b5a499ccc 100644
--- a/drivers/memory/tegra/tegra30.c
+++ b/drivers/memory/tegra/tegra30.c
@@ -1140,19 +1140,23 @@ static const struct tegra_mc_client tegra30_mc_clients[] = {
 
 static const struct tegra_smmu_swgroup tegra30_swgroups[] = {
 	{ .name = "dc",   .swgroup = TEGRA_SWGROUP_DC,   .reg = 0x240,
-	  .defer_enable = true },
+	  .defer_enable = true, .use_dma_api = true },
 	{ .name = "dcb",  .swgroup = TEGRA_SWGROUP_DCB,  .reg = 0x244,
-	  .defer_enable = true },
+	  .defer_enable = true, .use_dma_api = true },
 	{ .name = "epp",  .swgroup = TEGRA_SWGROUP_EPP,  .reg = 0x248 },
-	{ .name = "g2",   .swgroup = TEGRA_SWGROUP_G2,   .reg = 0x24c },
+	{ .name = "g2",   .swgroup = TEGRA_SWGROUP_G2,   .reg = 0x24c,
+	  .use_dma_api = true },
 	{ .name = "mpe",  .swgroup = TEGRA_SWGROUP_MPE,  .reg = 0x264 },
 	{ .name = "vi",   .swgroup = TEGRA_SWGROUP_VI,   .reg = 0x280 },
 	{ .name = "afi",  .swgroup = TEGRA_SWGROUP_AFI,  .reg = 0x238 },
 	{ .name = "avpc", .swgroup = TEGRA_SWGROUP_AVPC, .reg = 0x23c },
-	{ .name = "nv",   .swgroup = TEGRA_SWGROUP_NV,   .reg = 0x268 },
-	{ .name = "nv2",  .swgroup = TEGRA_SWGROUP_NV2,  .reg = 0x26c },
+	{ .name = "nv",   .swgroup = TEGRA_SWGROUP_NV,   .reg = 0x268,
+	  .use_dma_api = true },
+	{ .name = "nv2",  .swgroup = TEGRA_SWGROUP_NV2,  .reg = 0x26c,
+	  .use_dma_api = true },
 	{ .name = "hda",  .swgroup = TEGRA_SWGROUP_HDA,  .reg = 0x254 },
-	{ .name = "hc",   .swgroup = TEGRA_SWGROUP_HC,   .reg = 0x250 },
+	{ .name = "hc",   .swgroup = TEGRA_SWGROUP_HC,   .reg = 0x250,
+	  .use_dma_api = true },
 	{ .name = "ppcs", .swgroup = TEGRA_SWGROUP_PPCS, .reg = 0x270 },
 	{ .name = "sata", .swgroup = TEGRA_SWGROUP_SATA, .reg = 0x278 },
 	{ .name = "vde",  .swgroup = TEGRA_SWGROUP_VDE,  .reg = 0x27c },
diff --git a/include/soc/tegra/mc.h b/include/soc/tegra/mc.h
index ea67972fd585..1cbd990fc0f4 100644
--- a/include/soc/tegra/mc.h
+++ b/include/soc/tegra/mc.h
@@ -78,6 +78,13 @@ struct tegra_smmu_swgroup {
 	 * tegra_smmu_enable_translation().
 	 */
 	bool defer_enable;
+
+	/*
+	 * Set once the clients behind this swgroup are known to work with an
+	 * IOMMU_DOMAIN_DMA default domain. All swgroups of a
+	 * tegra_smmu_group_soc must agree.
+	 */
+	bool use_dma_api;
 };
 
 struct tegra_smmu_group_soc {

-- 
2.55.0

Re: [PATCH v3 07/10] iommu: Enable DMA default domains on (most) 32-bit ARM platforms

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] iommu: missing IDENTITY default domain override for qcom_iommu on ARM32
- [Critical] iommu/rockchip: use-after-free on device teardown with DMA default domain
--

commit b1b0900ebc55a49be25130b8187ffecfc2191b42
Author: Mikko Perttunen <mperttunen@nvidia.com>

iommu: Enable DMA default domains on (most) 32-bit ARM platforms

This commit enables core-owned DMA default domains on 32-bit ARM platforms
with CONFIG_ARM_DMA_USE_IOMMU by removing the IOMMU_DOMAIN_IDENTITY override
in iommu_get_default_domain_type(). For several IOMMU drivers that are not
ready to switch over, it adds a def_domain_type callback returning
IOMMU_DOMAIN_IDENTITY.

> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index 5fbb76d44a1e9..e47f95fea44d0 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -1884,18 +1884,6 @@ static int iommu_get_default_domain_type(struct iommu_group *group,
>  
>  	lockdep_assert_held(&group->mutex);
>  
> -	/*
> -	 * ARM32 drivers supporting CONFIG_ARM_DMA_USE_IOMMU can declare an
> -	 * identity_domain and it will automatically become their default
> -	 * domain. Later on ARM_DMA_USE_IOMMU will install its UNMANAGED domain.
> -	 * Override the selection to IDENTITY.
> -	 */
> -	if (IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)) {
> -		static_assert(!(IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU) &&
> -				IS_ENABLED(CONFIG_IOMMU_DMA)));
> -		driver_type = IOMMU_DOMAIN_IDENTITY;
> -	}
> -

[Severity: Critical]
Should qcom_iommu also keep an IOMMU_DOMAIN_IDENTITY default domain on 32-bit
ARM platforms?

CONFIG_QCOM_IOMMU selects CONFIG_ARM_DMA_USE_IOMMU, and qcom_iommu_ops in
drivers/iommu/arm/arm-smmu/qcom_iommu.c does not define a def_domain_type
callback or release_domain:

drivers/iommu/arm/arm-smmu/qcom_iommu.c:
    static const struct iommu_ops qcom_iommu_ops = {
        .identity_domain = &qcom_iommu_identity_domain,
        .capable	= qcom_iommu_capable,
        .domain_alloc_paging = qcom_iommu_domain_alloc_paging,

With this change, iommu_setup_default_domain() allocates and attaches a
paging IOMMU_DOMAIN_DMA group->default_domain via
qcom_iommu_attach_dev() -> qcom_iommu_init_domain(), which sets
qcom_domain->iommu, programs ARM_SMMU_CB_TTBR0 and ARM_SMMU_CB_SCTLR, and
sets ctx->domain = group->default_domain.

When a client driver such as drm/msm in msm_iommu_new()
(drivers/gpu/drm/msm/msm_iommu.c) or Venus in venus_firmware_init()
(drivers/media/platform/qcom/venus/firmware.c) attaches its own unmanaged
paging domain, arm_iommu_detach_device() and arm_iommu_release_mapping()
become no-ops because iommu_is_dma_domain(mapping->domain) is now true,
leaving dev->dma_ops set to &iommu_ops instead of restoring direct DMA ops.
Attaching the client's unmanaged domain then overwrites the context bank
registers and sets ctx->domain to the unmanaged domain.

When that unmanaged domain is later detached and freed (for example via
mdp5_kms_destroy() -> msm_iommu_detach() followed by drm_gpuvm_put() ->
msm_gem_vm_free() -> msm_iommu_destroy() in
drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c), __iommu_group_set_core_domain()
attempts to re-attach group->default_domain through qcom_iommu_attach_dev() ->
qcom_iommu_init_domain():

drivers/iommu/arm/arm-smmu/qcom_iommu.c:qcom_iommu_init_domain() {
        mutex_lock(&qcom_domain->init_mutex);
        if (qcom_domain->iommu)
                goto out_unlock;
        ...
}

Because qcom_domain->iommu is already non-NULL from the initial boot attach,
qcom_iommu_init_domain() returns early without reprogramming
ARM_SMMU_CB_TTBR0 or ARM_SMMU_CB_SCTLR and without updating ctx->domain.
This leaves ARM_SMMU_CB_TTBR0 pointing to the freed page tables with
translation still enabled in ARM_SMMU_CB_SCTLR, and leaves ctx->domain
pointing to the freed unmanaged domain where qcom_iommu_fault() will
dereference it on a context fault:

drivers/iommu/arm/arm-smmu/qcom_iommu.c:qcom_iommu_fault() {
        ...
        if (report_iommu_fault(ctx->domain, ctx->dev, iova, 0)) {
        ...
}

Similarly, during iommu_deinit_device(), because qcom_iommu_ops does not set
release_domain = &qcom_iommu_identity_domain, group->default_domain is freed
while the hardware context bank remains enabled and ctx->domain remains a
dangling pointer.

>  	for_each_group_device(group, gdev) {
>  		driver_type = iommu_get_def_domain_type(group, gdev->dev,
>  							driver_type);
> @@ -1912,11 +1900,11 @@ static int iommu_get_default_domain_type(struct iommu_group *group,
>  	}
>  
>  	/*
> -	 * If the common dma ops are not selected in kconfig then we cannot use
> -	 * IOMMU_DOMAIN_DMA at all. Force IDENTITY if nothing else has been
> -	 * selected.
> +	 * Without an IOMMU DMA implementation, force IDENTITY if the driver
> +	 * has not selected a domain type.
>  	 */
> -	if (!IS_ENABLED(CONFIG_IOMMU_DMA)) {
> +	if (!IS_ENABLED(CONFIG_IOMMU_DMA) &&
> +	    !IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)) {

[Severity: Critical]
Can switching rockchip-iommu to an IOMMU_DOMAIN_DMA default domain on 32-bit
ARM cause a use-after-free during device teardown?

CONFIG_ROCKCHIP_IOMMU selects CONFIG_ARM_DMA_USE_IOMMU, and rk_iommu_ops in
drivers/iommu/rockchip-iommu.c defines neither def_domain_type nor
release_domain = &rk_identity_domain:

drivers/iommu/rockchip-iommu.c:
    static void rk_iommu_release_device(struct device *dev)
    {
        struct rk_iommudata *data = dev_iommu_priv_get(dev);

        device_link_del(data->link);
    }
    ...
    static const struct iommu_ops rk_iommu_ops = {
        .identity_domain = &rk_identity_domain,
        .domain_alloc_paging = rk_iommu_domain_alloc_paging,
        .probe_device = rk_iommu_probe_device,
        .release_device = rk_iommu_release_device,

Previously on 32-bit ARM, group->default_domain was the static
&rk_identity_domain, and during device teardown arch_teardown_dma_ops() ->
arm_teardown_iommu_dma_ops() -> arm_iommu_detach_device() detached the legacy
UNMANAGED domain (re-attaching &rk_identity_domain and removing iommu->node
from rk_domain->iommus) before freeing it.

With this patch, group->default_domain is a dynamically allocated
IOMMU_DOMAIN_DMA paging domain attached via rk_iommu_attach_device(), which
sets iommu->domain = domain and adds iommu->node to rk_domain->iommus. When a
client device is removed or fails probe, arm_teardown_iommu_dma_ops() returns
early for DMA default domains, and iommu_deinit_device() sees
ops->release_domain == NULL, calls rk_iommu_release_device() without
detaching the domain, and then calls iommu_domain_free(group->default_domain)
-> rk_iommu_domain_free():

drivers/iommu/rockchip-iommu.c:rk_iommu_domain_free() {
        struct rk_iommu_domain *rk_domain = to_rk_domain(domain);
        int i;

        WARN_ON(!list_empty(&rk_domain->iommus));
        ...
        kfree(rk_domain);
}

This triggers the WARN_ON() and frees rk_domain and its page directory while
iommu->node is still linked on rk_domain->iommus and iommu->domain still
points to the freed rk_domain (which is later dereferenced in
rk_iommu_enable() and rk_iommu_irq()).

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261026-b4-arm32-iommu-dma-cookie-v3-0-3901d74b1ff2@nvidia.com?part=7

Re: [PATCH v3 10/10] iommu/tegra-smmu: Use a DMA API default domain per swgroup

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] iommu/tegra-smmu: tegra_smmu_attach_dev() leaks old paging domain resources
- [High] iommu/tegra-smmu: NULL dereference in tegra_smmu_identity_attach() on probe failure
--

commit 82c4953fbede4f90881980155ea701c19e530777
Author: Mikko Perttunen <mperttunen@nvidia.com>

iommu/tegra-smmu: Use a DMA API default domain per swgroup

This commit enables IOMMU_DOMAIN_DMA default domains on a per-swgroup
basis via a use_dma_api flag, initially enabled for TegraDRM and Host1x
devices, and sets .release_domain to tegra_smmu_identity_domain.

> diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c
> index 6779528a9e7cc..73c6151db8f8c 100644
> --- a/drivers/iommu/tegra-smmu.c
> +++ b/drivers/iommu/tegra-smmu.c
> @@ -994,16 +994,20 @@ static int tegra_smmu_of_xlate(struct device *dev,
>  
>  static int tegra_smmu_def_domain_type(struct device *dev)
>  {
> -	/*
> -	 * FIXME: For now we want to run all translation in IDENTITY mode, due
> -	 * to some device quirks. Better would be to just quirk the troubled
> -	 * devices.
> -	 */
> -	return IOMMU_DOMAIN_IDENTITY;
> +	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> +	struct tegra_smmu *smmu = dev_iommu_priv_get(dev);
> +	const struct tegra_smmu_swgroup *swgroup;
> +
> +	swgroup = tegra_smmu_find_swgroup(smmu, fwspec->ids[0]);
> +	if (WARN_ON(!swgroup) || !swgroup->use_dma_api)
> +		return IOMMU_DOMAIN_IDENTITY;
> +
> +	return IOMMU_DOMAIN_DMA;
>  }

[Severity: High]
Does transitioning between two paging domains leak the old domain's as->id
and as->pd_dma?

When tegra_smmu_def_domain_type() returns IOMMU_DOMAIN_DMA,
group->default_domain is a paging domain (struct tegra_smmu_as) rather than
tegra_smmu_identity_domain.

When switching between two paging domains (for example, attaching or
detaching an explicit unmanaged domain or a blocking domain over the
IOMMU_DOMAIN_DMA default domain, or changing the default domain type via
sysfs), the core invokes tegra_smmu_attach_dev(domain, dev, old). However,
tegra_smmu_attach_dev() ignores old and only prepares and enables the new
domain:

drivers/iommu/tegra-smmu.c:tegra_smmu_attach_dev() {
	...
	for (index = 0; index < fwspec->num_ids; index++) {
		err = tegra_smmu_as_prepare(smmu, as);
		if (err)
			goto disable;

		tegra_smmu_enable(smmu, fwspec->ids[index], as->id);
	}
	...
}

Because tegra_smmu_as_unprepare() is only called from
tegra_smmu_identity_attach(), the replaced paging domain's as->use_count is
never decremented. When that domain is later freed via
tegra_smmu_domain_free(), it triggers WARN_ON_ONCE(as->use_count) and
permanently leaks its hardware ASID in smmu->asids and its page directory
DMA mapping as->pd_dma.

>  
>  static const struct iommu_ops tegra_smmu_ops = {
>  	.identity_domain = &tegra_smmu_identity_domain,
> +	.release_domain = &tegra_smmu_identity_domain,

[Severity: High]
Can setting .release_domain to tegra_smmu_identity_domain cause a NULL
pointer dereference in tegra_smmu_identity_attach() if attaching the
default domain fails during probe?

During __iommu_probe_device() -> iommu_setup_default_domain(), a new
IOMMU_DOMAIN_DMA domain is allocated with as->smmu = NULL and
as->use_count = 0, and attached via:

  __iommu_group_set_domain_internal(group, dom, IOMMU_SET_DOMAIN_MUST_SUCCEED)

If tegra_smmu_attach_dev() fails inside tegra_smmu_as_prepare() (for
example, if tegra_smmu_alloc_asid() returns -ENOSPC when all 4 hardware
ASIDs on Tegra30/Tegra114 are in use, or dma_map_single() fails), as->smmu
remains NULL. Because IOMMU_SET_DOMAIN_MUST_SUCCEED is passed,
__iommu_group_set_domain_internal() still assigns group->domain = dom
before returning the error.

On the error unwind path in __iommu_probe_device(), iommu_deinit_device()
attaches ops->release_domain while passing group->domain as old, which
calls tegra_smmu_identity_attach():

drivers/iommu/tegra-smmu.c:tegra_smmu_identity_attach() {
	...
	if (old == identity_domain || !old)
		return 0;

	as = to_smmu_as(old);
	smmu = as->smmu;
	for (index = 0; index < fwspec->num_ids; index++) {
		tegra_smmu_disable(smmu, fwspec->ids[index], as->id);
		tegra_smmu_as_unprepare(smmu, as);
	}
	return 0;
}

Here, smmu is NULL, which is then dereferenced in tegra_smmu_disable() and
tegra_smmu_as_unprepare(). Similarly, if a second device joining an already
active shared group fails in __iommu_probe_device() before attaching to
group->domain, iommu_deinit_device() will call tegra_smmu_identity_attach()
and unprepare the shared group->domain even though the failing device never
prepared it.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261026-b4-arm32-iommu-dma-cookie-v3-0-3901d74b1ff2@nvidia.com?part=10