[PATCH v2 0/2] IOMMU Performance Optimization support

[PATCH v2 0/2] IOMMU Performance Optimization support

[Mario Limonciello]

Add support for the AMD IOMMU Performance Optimization (PerfOpt)
feature as defined in the AMD I/O Virtualization Technology (IOMMU)
Specification, Section 3.4.9 (MMIO Offset 016Ch).

This feature allows privileged integrated I/O devices (GPUs) to bypass
the IOMMU when directly accessing system memory.  The IOMMU only
enforces the IR/IW permission bits without GPA->SPA translations.

Drivers will opt into this behavior, and enable amdgpu to turn this
feature on when an integrated GPU is already in identity mode.
If a user prefers to keep it always off (even with identity mode)
there is also a module parameter to force it off.

v2:
 * Rebase on 7.3-rc2
 * Adjust amdgpu parameter behavior

Mario Limonciello (2):
  iommu/amd: Add PerfOpt IOMMU performance optimization support
  drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in
    identity domain

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  50 +++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  12 ++
 drivers/iommu/amd/amd_iommu.h              |   3 +
 drivers/iommu/amd/amd_iommu_types.h        |   7 +
 drivers/iommu/amd/init.c                   |  43 +++++
 drivers/iommu/amd/iommu.c                  | 214 +++++++++++++++++++++
 include/linux/amd-iommu.h                  |  11 ++
 8 files changed, 341 insertions(+)

-- 
2.43.0

[PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

Add support for the AMD IOMMU Performance Optimization (PerfOpt)
feature as defined in the AMD I/O Virtualization Technology (IOMMU)
Specification, Section 3.4.9 (MMIO Offset 016Ch).

This feature allows privileged integrated I/O devices (GPUs) to bypass
the IOMMU when directly accessing system memory.  The IOMMU only
enforces the IR/IW permission bits without GPA->SPA translations.

amd_iommu_enable_perfopt() performs a detach/reattach cycle to rehome
devices already on the identity domain with ATS/PRI/PASID/GCR3
disabled (skip_caps path).  amd_iommu_disable_perfopt() restores those
capabilities.  The per-device dev_data->perfopt flag tracks state.

PERF_OPT_EN is a single control bit per IOMMU, shared by every device
behind that IOMMU, while enablement is requested per device.  It is
therefore reference counted (amd_iommu->perfopt_refcount): armed on the
first requesting device and cleared on the last, so one device's
teardown never clears the bit while a peer behind the same IOMMU still
needs it.

The per-device flag is cleared on every teardown path
(blocked_domain_attach, release_device, and amd_iommu_disable_perfopt),
dropping the reference with it, so a reused dev_data never carries stale
PerfOpt state onto its next bind.

On suspend/resume the hardware is reprogrammed from scratch:
amd_iommu_perfopt_clear() forces the bit off without touching the
reference count, and amd_iommu_perfopt_restore() re-asserts it from the
count after early_enable_iommu(), so armed devices keep the optimization
across resume without relying on each consumer driver to re-arm.

The exported amd_iommu_enable_perfopt()/amd_iommu_disable_perfopt() run
only from a consumer driver's bind/unbind path.  group->mutex is not
exposed to drivers, but a device bound to its native driver cannot have
its IOMMU domain changed concurrently by the core, which serializes the
detach/attach pair against core-driven attach.

PerfOpt is opt-in -- only enabled when explicitly requested by a driver.

Tested-by: Derek J. Clark <derekjohn.clark@gmail.com>
Co-developed-by: Jatin Kataria <jkataria@netflix.com>
Signed-off-by: Jatin Kataria <jkataria@netflix.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
v2:
 * rebase on 7.3-rc2
 * add tag
---
 drivers/iommu/amd/amd_iommu.h       |   3 +
 drivers/iommu/amd/amd_iommu_types.h |   7 +
 drivers/iommu/amd/init.c            |  43 ++++++
 drivers/iommu/amd/iommu.c           | 214 ++++++++++++++++++++++++++++
 include/linux/amd-iommu.h           |  11 ++
 5 files changed, 278 insertions(+)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index a2fe804b038b6..1f8f9df8e6c24 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -48,6 +48,9 @@ extern u8 amd_iommu_hpt_vasize;
 extern unsigned long amd_iommu_pgsize_bitmap;
 extern bool amd_iommu_hatdis;
 
+int amd_iommu_perfopt_clear(struct amd_iommu *iommu);
+int amd_iommu_perfopt_restore(struct amd_iommu *iommu);
+
 /* Protection domain ops */
 void amd_iommu_init_identity_domain(void);
 struct protection_domain *protection_domain_alloc(void);
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 3dbe20023456b..755421e5cd757 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -65,6 +65,7 @@
 #define MMIO_MSI_ADDR_LO_OFFSET	0x015C
 #define MMIO_MSI_ADDR_HI_OFFSET	0x0160
 #define MMIO_MSI_DATA_OFFSET	0x0164
+#define MMIO_PERF_OPT_OFFSET	0x016C
 #define MMIO_INTCAPXT_EVT_OFFSET	0x0170
 #define MMIO_INTCAPXT_PPR_OFFSET	0x0178
 #define MMIO_INTCAPXT_GALOG_OFFSET	0x0180
@@ -99,6 +100,8 @@
 #define FEATURE_GLX		GENMASK_ULL(15, 14)
 #define FEATURE_GAM_VAPIC	BIT_ULL(21)
 #define FEATURE_PASMAX		GENMASK_ULL(36, 32)
+#define FEATURE_PERF_OPT	BIT_ULL(45)
+#define PERF_OPT_EN		BIT(13)
 #define FEATURE_GIOSUP		BIT_ULL(48)
 #define FEATURE_HASUP		BIT_ULL(49)
 #define FEATURE_EPHSUP		BIT_ULL(50)
@@ -670,6 +673,9 @@ struct amd_iommu {
 	/* Extended features 2 */
 	u64 features2;
 
+	/* Devices requesting PerfOpt; the shared PERF_OPT_EN bit is on while >0. Protected by @lock. */
+	int perfopt_refcount;
+
 	/* PCI device id of the IOMMU device */
 	u16 devid;
 
@@ -831,6 +837,7 @@ struct iommu_dev_data {
 	u8 ppr          :1;		  /* Enable device PPR support */
 	bool use_vapic;			  /* Enable device to use vapic mode */
 	bool defer_attach;
+	bool perfopt;
 
 	struct ratelimit_state rs;        /* Ratelimit IOPF messages */
 };
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 40726dfef2733..ddcf56f101675 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1942,6 +1942,9 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h,
 	if (!iommu->mmio_base)
 		return -ENOMEM;
 
+	if (amd_iommu_perfopt_clear(iommu))
+		pr_err("IOMMU%d: failed to clear PerfOpt\n", iommu->index);
+
 	return init_iommu_from_acpi(iommu, h);
 }
 
@@ -3032,10 +3035,46 @@ static void enable_iommus_vapic(void)
 #endif
 }
 
+static int clear_perfopt_all(void)
+{
+	struct amd_iommu *iommu;
+	int err, ret = 0;
+
+	for_each_iommu(iommu) {
+		err = amd_iommu_perfopt_clear(iommu);
+		if (err)
+			ret = err;
+	}
+
+	return ret;
+}
+
+static int restore_perfopt_all(void)
+{
+	struct amd_iommu *iommu;
+	int err, ret = 0;
+
+	for_each_iommu(iommu) {
+		err = amd_iommu_perfopt_restore(iommu);
+		if (err)
+			ret = err;
+	}
+
+	return ret;
+}
+
 static void disable_iommus(void)
 {
 	struct amd_iommu *iommu;
 
+	/*
+	 * PerfOpt is an optional performance bit, so a failure to clear it must
+	 * not skip the mandatory disable below. This also runs from the void
+	 * amd_iommu_disable() shutdown/kexec path, which cannot report an error.
+	 */
+	if (clear_perfopt_all())
+		pr_err("Failed to clear PerfOpt while disabling IOMMUs\n");
+
 	for_each_iommu(iommu)
 		iommu_disable(iommu);
 
@@ -3061,6 +3100,10 @@ static void amd_iommu_resume(void *data)
 	for_each_iommu(iommu)
 		early_enable_iommu(iommu);
 
+	/* early_enable_iommu() cleared PERF_OPT_EN; re-assert it from the refcount. */
+	if (restore_perfopt_all())
+		pr_err("Failed to restore PerfOpt after IOMMU resume\n");
+
 	iommu_enable_event_buffer();
 	amd_iommu_enable_interrupts();
 }
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 4dc306a4b5c62..fa60affdfc035 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2395,6 +2395,9 @@ static int attach_device(struct device *dev,
 	if (ret)
 		goto out;
 
+	if (dev_data->perfopt)
+		goto skip_caps;
+
 	/* Setup GCR3 table */
 	if (pdom_is_sva_capable(domain)) {
 		ret = init_gcr3_table(dev_data, domain);
@@ -2419,6 +2422,7 @@ static int attach_device(struct device *dev,
 		pdev_enable_cap_ats(pdev);
 	}
 
+skip_caps:
 	/* Update data structures */
 	dev_data->domain = domain;
 	spin_lock_irqsave(&domain->lock, flags);
@@ -2487,6 +2491,192 @@ static void detach_device(struct device *dev)
 	mutex_unlock(&dev_data->mutex);
 }
 
+/* Program the per-IOMMU PerfOpt enable bit. Caller must hold iommu->lock. */
+static int __perfopt_write(struct amd_iommu *iommu, bool enable)
+{
+	u32 old, val, readback;
+
+	if (!(readq(iommu->mmio_base + MMIO_EXT_FEATURES) & FEATURE_PERF_OPT))
+		return enable ? -ENODEV : 0;
+
+	old = readl(iommu->mmio_base + MMIO_PERF_OPT_OFFSET);
+	if (old == U32_MAX)
+		return -EIO;
+
+	val = enable ? old | PERF_OPT_EN : old & ~PERF_OPT_EN;
+	if (val != old)
+		writel(val, iommu->mmio_base + MMIO_PERF_OPT_OFFSET);
+	readback = readl(iommu->mmio_base + MMIO_PERF_OPT_OFFSET);
+	if (readback == U32_MAX ||
+	    (readback & PERF_OPT_EN) != (val & PERF_OPT_EN))
+		return -EIO;
+	return 0;
+}
+
+/*
+ * PERF_OPT_EN is a single bit shared by every device behind @iommu, so it is
+ * reference counted: armed on the first requesting device, cleared on the last.
+ */
+static int perfopt_get(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	if (iommu->perfopt_refcount == 0) {
+		ret = __perfopt_write(iommu, true);
+		if (ret)
+			goto out;
+	}
+	iommu->perfopt_refcount++;
+out:
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+static int perfopt_put(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	if (iommu->perfopt_refcount > 0 && --iommu->perfopt_refcount == 0)
+		ret = __perfopt_write(iommu, false);
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+/*
+ * Force PERF_OPT_EN off without touching the refcount (used on init, shutdown,
+ * and suspend). The count is preserved so amd_iommu_perfopt_restore() can
+ * re-arm on resume.
+ */
+int amd_iommu_perfopt_clear(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	ret = __perfopt_write(iommu, false);
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+/*
+ * Re-assert PERF_OPT_EN from the refcount after the hardware was reprogrammed on
+ * resume, so devices armed before suspend keep the optimization without each
+ * consumer driver re-arming.
+ */
+int amd_iommu_perfopt_restore(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	ret = __perfopt_write(iommu, iommu->perfopt_refcount > 0);
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+int amd_iommu_enable_perfopt(struct pci_dev *pdev)
+{
+	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
+	struct amd_iommu *iommu = rlookup_amd_iommu(&pdev->dev);
+	struct protection_domain *domain;
+	int ret;
+
+	if (!iommu || !dev_data)
+		return -ENODEV;
+
+	if (!(iommu->features & FEATURE_PERF_OPT))
+		return -ENODEV;
+
+	domain = dev_data->domain;
+	if (!domain)
+		return -ENODEV;
+
+	/* Already armed for this device (e.g. re-entry on resume). */
+	if (dev_data->perfopt)
+		return 0;
+
+	/*
+	 * The bit is only architecturally valid while the device is untranslated:
+	 * identity domain with ATS/PRI/PASID off. The identity domain is
+	 * SVA-capable so attach_device() enabled ATS/PRI/PASID and built a GCR3
+	 * table. Re-home the device onto the same identity domain with
+	 * perfopt set, so the attach_device() skip_caps path leaves
+	 * ATS/PRI/PASID off and no GCR3 table. This follows the detach/attach
+	 * pattern used by amd_iommu_attach_device().
+	 *
+	 * Locking: this and amd_iommu_disable_perfopt() run only from the
+	 * consumer driver's bind/unbind path. group->mutex is not exposed to
+	 * drivers, but a device bound to its native driver cannot have its domain
+	 * changed concurrently by the core (VFIO ownership is mutually exclusive;
+	 * sysfs domain changes require an unused group), so the detach/attach pair
+	 * is serialized without it.
+	 */
+	dev_data->perfopt = true;
+	detach_device(&pdev->dev);
+	ret = attach_device(&pdev->dev, domain);
+	if (ret)
+		goto err_restore;
+
+	ret = perfopt_get(iommu);
+	if (ret)
+		goto err_rearm;
+
+	dev_info_once(&pdev->dev, "PerfOpt armed on IOMMU%d\n", iommu->index);
+	return 0;
+
+err_rearm:
+	detach_device(&pdev->dev);
+err_restore:
+	dev_data->perfopt = false;
+	if (attach_device(&pdev->dev, domain))
+		pci_err(pdev, "failed to restore state after PerfOpt setup; device left detached\n");
+	dev_err_once(&pdev->dev, "PerfOpt failed to arm on IOMMU%d (%d)\n",
+		     iommu->index, ret);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(amd_iommu_enable_perfopt);
+
+void amd_iommu_disable_perfopt(struct pci_dev *pdev)
+{
+	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
+	struct amd_iommu *iommu = rlookup_amd_iommu(&pdev->dev);
+	struct protection_domain *domain;
+
+	if (!iommu || !dev_data || !dev_data->perfopt || !dev_data->domain)
+		return;
+
+	if (WARN_ON(perfopt_put(iommu)))
+		pci_err(pdev, "failed to clear PerfOpt\n");
+
+	/*
+	 * Restore ATS/PRI/PASID (and thus SVA) by re-homing the device onto its
+	 * identity domain with the flag cleared, so a later bind without PerfOpt
+	 * sees a normally-capable device. See the locking note in
+	 * amd_iommu_enable_perfopt().
+	 */
+	domain = dev_data->domain;
+	dev_data->perfopt = false;
+	detach_device(&pdev->dev);
+	if (attach_device(&pdev->dev, domain))
+		pci_err(pdev, "failed to restore caps after PerfOpt disable\n");
+}
+EXPORT_SYMBOL_GPL(amd_iommu_disable_perfopt);
 static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 {
 	struct iommu_device *iommu_dev;
@@ -2554,6 +2744,14 @@ static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 static void amd_iommu_release_device(struct device *dev)
 {
 	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
+	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+
+	if (dev_data->perfopt) {
+		if (WARN_ON(perfopt_put(iommu)))
+			dev_err(dev, "IOMMU%d: failed to clear PerfOpt on release\n",
+				iommu->index);
+		dev_data->perfopt = false;
+	}
 
 	WARN_ON(dev_data->domain);
 
@@ -2928,6 +3126,19 @@ static int blocked_domain_attach_device(struct iommu_domain *domain,
 					struct iommu_domain *old)
 {
 	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
+	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+
+	/*
+	 * blocked_domain is also the .release_domain, so this is the normal
+	 * teardown path: drop the reference and clear the flag here too, and
+	 * don't fail teardown if the WARN-guarded write doesn't stick.
+	 */
+	if (dev_data->perfopt) {
+		if (WARN_ON(perfopt_put(iommu)))
+			dev_err(dev, "IOMMU%d: failed to clear PerfOpt for blocked domain\n",
+				iommu->index);
+		dev_data->perfopt = false;
+	}
 
 	if (dev_data->domain)
 		detach_device(dev);
@@ -2996,6 +3207,9 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, struct device *dev,
 	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev);
 	int ret;
 
+	if (dev_data->perfopt && !pdom_is_in_pt_mode(domain))
+		return -EBUSY;
+
 	/*
 	 * Skip attach device to domain if new domain is same as
 	 * devices current domain
diff --git a/include/linux/amd-iommu.h b/include/linux/amd-iommu.h
index edcee9f5335a6..e03575cbc08c6 100644
--- a/include/linux/amd-iommu.h
+++ b/include/linux/amd-iommu.h
@@ -76,4 +76,15 @@ static inline int amd_iommu_snp_disable(void) { return 0; }
 static inline bool amd_iommu_sev_tio_supported(void) { return false; }
 #endif
 
+#ifdef CONFIG_AMD_IOMMU
+int amd_iommu_enable_perfopt(struct pci_dev *pdev);
+void amd_iommu_disable_perfopt(struct pci_dev *pdev);
+#else
+static inline int amd_iommu_enable_perfopt(struct pci_dev *pdev)
+{
+	return 0;
+}
+static inline void amd_iommu_disable_perfopt(struct pci_dev *pdev) { }
+#endif
+
 #endif /* _ASM_X86_AMD_IOMMU_H */
-- 
2.43.0

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Jason Gunthorpe]

On Mon, Sep 07, 2026 at 11:12:06PM -0500, Mario Limonciello wrote:
> +EXPORT_SYMBOL_GPL(amd_iommu_enable_perfopt);

Last time we hardwired the GPU and IOMMU drivers together it was a
huge mess to undo.

Why does the GPU driver have to request this? Why can't the iommu
driver know that this is a special device that can use this magic fast
path and then auto set it when identity is asked for?

How does the GPU driver even know it is magic special?

> +	/*
> +	 * Restore ATS/PRI/PASID (and thus SVA) by re-homing the device onto its
> +	 * identity domain with the flag cleared, so a later bind without PerfOpt
> +	 * sees a normally-capable device. See the locking note in
> +	 * amd_iommu_enable_perfopt().
> +	 */

And this is sort of a wrong thing in the driver, it shouldn't have ATS
turned on for identitiy mappings, and it certainly shouldn't have PRI
turned on until a PRI capable domain is attached.

Jason

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

On 9/24/26 5:50 PM, Jason Gunthorpe wrote:
> On Mon, Sep 07, 2026 at 11:12:06PM -0500, Mario Limonciello wrote:
>> +EXPORT_SYMBOL_GPL(amd_iommu_enable_perfopt);
> 
> Last time we hardwired the GPU and IOMMU drivers together it was a
> huge mess to undo.
> 
> Why does the GPU driver have to request this? Why can't the iommu
> driver know that this is a special device that can use this magic fast
> path and then auto set it when identity is asked for?

The reason was to allow an "opt-out" path on the GPU driver.

> 
> How does the GPU driver even know it is magic special?

See patch 2/2.  GPU driver looks if it's an APU and it's in identity 
mode.  The special mode is only for the GPU in an APU.  It shouldn't be 
applied to anything else.
> 
>> +	/*
>> +	 * Restore ATS/PRI/PASID (and thus SVA) by re-homing the device onto its
>> +	 * identity domain with the flag cleared, so a later bind without PerfOpt
>> +	 * sees a normally-capable device. See the locking note in
>> +	 * amd_iommu_enable_perfopt().
>> +	 */
> 
> And this is sort of a wrong thing in the driver, it shouldn't have ATS
> turned on for identitiy mappings, and it certainly shouldn't have PRI
> turned on until a PRI capable domain is attached.
> 
> Jason
> 

amd_iommu_disable_perfopt() only re-applies the policy that was 
previously set before PerfOpt.

If you would like I'll send a follow up patch to clarify this comment.

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Jason Gunthorpe]

On Thu, Sep 24, 2026 at 07:03:56PM -0500, Mario Limonciello wrote:
> 
> 
> On 9/24/26 5:50 PM, Jason Gunthorpe wrote:
> > On Mon, Sep 07, 2026 at 11:12:06PM -0500, Mario Limonciello wrote:
> > > +EXPORT_SYMBOL_GPL(amd_iommu_enable_perfopt);
> > 
> > Last time we hardwired the GPU and IOMMU drivers together it was a
> > huge mess to undo.
> > 
> > Why does the GPU driver have to request this? Why can't the iommu
> > driver know that this is a special device that can use this magic fast
> > path and then auto set it when identity is asked for?
> 
> The reason was to allow an "opt-out" path on the GPU driver.

This seems like a very poor reason for doing this given the past
negative experiances.

Jason

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

On 9/24/26 7:17 PM, Jason Gunthorpe wrote:
> On Thu, Sep 24, 2026 at 07:03:56PM -0500, Mario Limonciello wrote:
>>
>>
>> On 9/24/26 5:50 PM, Jason Gunthorpe wrote:
>>> On Mon, Sep 07, 2026 at 11:12:06PM -0500, Mario Limonciello wrote:
>>>> +EXPORT_SYMBOL_GPL(amd_iommu_enable_perfopt);
>>>
>>> Last time we hardwired the GPU and IOMMU drivers together it was a
>>> huge mess to undo.
>>>
>>> Why does the GPU driver have to request this? Why can't the iommu
>>> driver know that this is a special device that can use this magic fast
>>> path and then auto set it when identity is asked for?
>>
>> The reason was to allow an "opt-out" path on the GPU driver.
> 
> This seems like a very poor reason for doing this given the past
> negative experiances.
> 
> Jason

I guess I don't have the context of what happened in the past.  Could 
you fill me in?  I'm not against it being automatic when in identity 
mode.  I'll just need to come up with a way to check it's the GPU inside 
of an APU not a dGPU connected to the APU.

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Joerg Roedel]

On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
> Why does the GPU driver have to request this? Why can't the iommu
> driver know that this is a special device that can use this magic fast
> path and then auto set it when identity is asked for?

This is a layering violation either way. Either the IOMMU driver needs to poke
into device details and their relations or the device driver needs to poke into
IOMMU details.

This is not perfect but to some degree unavoidable with these highly integrated
SOCs.

-Joerg

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Jason Gunthorpe]

On Fri, Sep 25, 2026 at 09:20:02AM +0200, Joerg Roedel wrote:
> On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
> > Why does the GPU driver have to request this? Why can't the iommu
> > driver know that this is a special device that can use this magic fast
> > path and then auto set it when identity is asked for?
> 
> This is a layering violation either way. Either the IOMMU driver needs to poke
> into device details and their relations or the device driver needs to poke into
> IOMMU details.

There are no device details, this is just a module option in the gpu
driver.

> This is not perfect but to some degree unavoidable with these highly integrated
> SOCs.

This is exactly the sort of hacky thing ACPI should describe so the
drivers can just turn things on automatically.

Who is ever going to track down and figure out it is OK to set these
kinds of module options in the first place? There is general negative
sentiment in the kernel to doing things like this with modue options.

How does Windows do it? There certainly isn't a "module option" there?

Jason

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

On 9/25/26 07:28, Jason Gunthorpe wrote:
> On Fri, Sep 25, 2026 at 09:20:02AM +0200, Joerg Roedel wrote:
>> On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
>>> Why does the GPU driver have to request this? Why can't the iommu
>>> driver know that this is a special device that can use this magic fast
>>> path and then auto set it when identity is asked for?
>>
>> This is a layering violation either way. Either the IOMMU driver needs to poke
>> into device details and their relations or the device driver needs to poke into
>> IOMMU details.
> 
> There are no device details, this is just a module option in the gpu
> driver.

Well the GPU driver has knowledge whether it's a GPU inside of an APU to 
decide if it's valid.  That information is part of the discovery table 
setup.

> 
>> This is not perfect but to some degree unavoidable with these highly integrated
>> SOCs.
> 
> This is exactly the sort of hacky thing ACPI should describe so the
> drivers can just turn things on automatically.
> 
> Who is ever going to track down and figure out it is OK to set these
> kinds of module options in the first place? There is general negative
> sentiment in the kernel to doing things like this with modue options.
> 
> How does Windows do it? There certainly isn't a "module option" there?
> 
> Jason

This specific one I don't know; but these kinds of things usually end up 
in knobs in a UI that plumb some IPC deep into the driver stack.

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Jason Gunthorpe]

On Fri, Sep 25, 2026 at 08:11:41AM -0500, Mario Limonciello wrote:
> 
> 
> On 9/25/26 07:28, Jason Gunthorpe wrote:
> > On Fri, Sep 25, 2026 at 09:20:02AM +0200, Joerg Roedel wrote:
> > > On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
> > > > Why does the GPU driver have to request this? Why can't the iommu
> > > > driver know that this is a special device that can use this magic fast
> > > > path and then auto set it when identity is asked for?
> > > 
> > > This is a layering violation either way. Either the IOMMU driver needs to poke
> > > into device details and their relations or the device driver needs to poke into
> > > IOMMU details.
> > 
> > There are no device details, this is just a module option in the gpu
> > driver.
> 
> Well the GPU driver has knowledge whether it's a GPU inside of an APU to
> decide if it's valid.  That information is part of the discovery table
> setup.

Where was that in patch 2?

The right way for this to work is to quirk it through the iommu driver
so it can choose the fast mode, and not involve the GPU driver at
all. Like we've done for every other GPU weirdness. And AMD should be
making this work better by providing ACPI support so the iommu driver
can understand how it should work without inspecting PCI IDs.

> > > This is not perfect but to some degree unavoidable with these highly integrated
> > > SOCs.
> > 
> > This is exactly the sort of hacky thing ACPI should describe so the
> > drivers can just turn things on automatically.
> > 
> > Who is ever going to track down and figure out it is OK to set these
> > kinds of module options in the first place? There is general negative
> > sentiment in the kernel to doing things like this with modue options.
> > 
> > How does Windows do it? There certainly isn't a "module option" there?
> 
> This specific one I don't know; but these kinds of things usually end up in
> knobs in a UI that plumb some IPC deep into the driver stack.

Given the huge delta I somehow doubt that AMD would ship a windows
driver that defaults to "slow" ? Nor should linux...

Jason

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

On 9/25/26 5:39 PM, Jason Gunthorpe wrote:
> On Fri, Sep 25, 2026 at 08:11:41AM -0500, Mario Limonciello wrote:
>>
>>
>> On 9/25/26 07:28, Jason Gunthorpe wrote:
>>> On Fri, Sep 25, 2026 at 09:20:02AM +0200, Joerg Roedel wrote:
>>>> On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
>>>>> Why does the GPU driver have to request this? Why can't the iommu
>>>>> driver know that this is a special device that can use this magic fast
>>>>> path and then auto set it when identity is asked for?
>>>>
>>>> This is a layering violation either way. Either the IOMMU driver needs to poke
>>>> into device details and their relations or the device driver needs to poke into
>>>> IOMMU details.
>>>
>>> There are no device details, this is just a module option in the gpu
>>> driver.
>>
>> Well the GPU driver has knowledge whether it's a GPU inside of an APU to
>> decide if it's valid.  That information is part of the discovery table
>> setup.
> 
> Where was that in patch 2?

amdgpu_device_use_perfopt()

> 
> The right way for this to work is to quirk it through the iommu driver
> so it can choose the fast mode, and not involve the GPU driver at
> all. Like we've done for every other GPU weirdness. And AMD should be
> making this work better by providing ACPI support so the iommu driver
> can understand how it should work without inspecting PCI IDs.

There is a heuristic that Vasant added recently that we detect APU from 
the IOMMU driver.  We might be able to use that for now.

I'll take a look at what an incremental patch looks like on top of 
iommu/next that I think reworks all these paths the way you want.  I 
should be able to post something next week.

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Jason Gunthorpe]

On Fri, Sep 25, 2026 at 07:33:08PM -0500, Mario Limonciello wrote:
> 
> 
> On 9/25/26 5:39 PM, Jason Gunthorpe wrote:
> > On Fri, Sep 25, 2026 at 08:11:41AM -0500, Mario Limonciello wrote:
> > > 
> > > 
> > > On 9/25/26 07:28, Jason Gunthorpe wrote:
> > > > On Fri, Sep 25, 2026 at 09:20:02AM +0200, Joerg Roedel wrote:
> > > > > On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
> > > > > > Why does the GPU driver have to request this? Why can't the iommu
> > > > > > driver know that this is a special device that can use this magic fast
> > > > > > path and then auto set it when identity is asked for?
> > > > > 
> > > > > This is a layering violation either way. Either the IOMMU driver needs to poke
> > > > > into device details and their relations or the device driver needs to poke into
> > > > > IOMMU details.
> > > > 
> > > > There are no device details, this is just a module option in the gpu
> > > > driver.
> > > 
> > > Well the GPU driver has knowledge whether it's a GPU inside of an APU to
> > > decide if it's valid.  That information is part of the discovery table
> > > setup.
> > 
> > Where was that in patch 2?
> 
> amdgpu_device_use_perfopt()

Ah, sneaky, that looks like it just decodes from a a giant list of PCI IDs
 
> > The right way for this to work is to quirk it through the iommu driver
> > so it can choose the fast mode, and not involve the GPU driver at
> > all. Like we've done for every other GPU weirdness. And AMD should be
> > making this work better by providing ACPI support so the iommu driver
> > can understand how it should work without inspecting PCI IDs.
> 
> There is a heuristic that Vasant added recently that we detect APU from the
> IOMMU driver.  We might be able to use that for now.
> 
> I'll take a look at what an incremental patch looks like on top of
> iommu/next that I think reworks all these paths the way you want.  I should
> be able to post something next week.

ACPI is the right answer to these kinds of problems.

Jason

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

On 9/28/26 07:08, Jason Gunthorpe wrote:
> On Fri, Sep 25, 2026 at 07:33:08PM -0500, Mario Limonciello wrote:
>>
>>
>> On 9/25/26 5:39 PM, Jason Gunthorpe wrote:
>>> On Fri, Sep 25, 2026 at 08:11:41AM -0500, Mario Limonciello wrote:
>>>>
>>>>
>>>> On 9/25/26 07:28, Jason Gunthorpe wrote:
>>>>> On Fri, Sep 25, 2026 at 09:20:02AM +0200, Joerg Roedel wrote:
>>>>>> On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
>>>>>>> Why does the GPU driver have to request this? Why can't the iommu
>>>>>>> driver know that this is a special device that can use this magic fast
>>>>>>> path and then auto set it when identity is asked for?
>>>>>>
>>>>>> This is a layering violation either way. Either the IOMMU driver needs to poke
>>>>>> into device details and their relations or the device driver needs to poke into
>>>>>> IOMMU details.
>>>>>
>>>>> There are no device details, this is just a module option in the gpu
>>>>> driver.
>>>>
>>>> Well the GPU driver has knowledge whether it's a GPU inside of an APU to
>>>> decide if it's valid.  That information is part of the discovery table
>>>> setup.
>>>
>>> Where was that in patch 2?
>>
>> amdgpu_device_use_perfopt()
> 
> Ah, sneaky, that looks like it just decodes from a a giant list of PCI IDs

Actually; no.  amdgpu has migrated away from PCI IDs and probes based on 
the "ATI vendor" + "display class" instead.  As part of the probe 
sequence there is something called an IP discovery table pull from the 
platform that identifies all IP blocks in the hardware.

This IP discovery table will indicate graphics IP and that is mapped to 
relevant driver code.  So for example GC 11.5.1 is what you find in a 
Strix Halo APU.  If a future APU had exactly the same graphics IP but a 
different display IP it would still be GC 11.5.1.

This design was implemented about 4 years ago, and all SoCs (APU, dGPU 
and accelerator) since then use it.
>   
>>> The right way for this to work is to quirk it through the iommu driver
>>> so it can choose the fast mode, and not involve the GPU driver at
>>> all. Like we've done for every other GPU weirdness. And AMD should be
>>> making this work better by providing ACPI support so the iommu driver
>>> can understand how it should work without inspecting PCI IDs.
>>
>> There is a heuristic that Vasant added recently that we detect APU from the
>> IOMMU driver.  We might be able to use that for now.
>>
>> I'll take a look at what an incremental patch looks like on top of
>> iommu/next that I think reworks all these paths the way you want.  I should
>> be able to post something next week.
> 
> ACPI is the right answer to these kinds of problems.
> 
> Jason

I'll discuss this with architects for the future programs.

For now as promissed I've worked out a patch [1] that keeps it all in 
iommu/amd per your suggestions.  It is using the existing APU heuristic.

https://lore.kernel.org/linux-iommu/20260928045050.955165-1-superm1@kernel.org/ 
[1]

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Jason Gunthorpe]

On Mon, Sep 28, 2026 at 10:32:41AM -0500, Mario Limonciello wrote:

> Actually; no.  amdgpu has migrated away from PCI IDs and probes based on the
> "ATI vendor" + "display class" instead.  As part of the probe sequence there
> is something called an IP discovery table pull from the platform that
> identifies all IP blocks in the hardware.

Oh, so that big table is not processing pci ids.. OK

> This IP discovery table will indicate graphics IP and that is mapped to
> relevant driver code.  So for example GC 11.5.1 is what you find in a Strix
> Halo APU.  If a future APU had exactly the same graphics IP but a different
> display IP it would still be GC 11.5.1.

Move this stuff to drivers/firmware and iommu and GPU can access it
together?

"IP discovery table" sure sounds like a platform firmware datablob to
me at least?

Jason

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

On 9/28/26 12:11, Jason Gunthorpe wrote:
> On Mon, Sep 28, 2026 at 10:32:41AM -0500, Mario Limonciello wrote:
> 
>> Actually; no.  amdgpu has migrated away from PCI IDs and probes based on the
>> "ATI vendor" + "display class" instead.  As part of the probe sequence there
>> is something called an IP discovery table pull from the platform that
>> identifies all IP blocks in the hardware.
> 
> Oh, so that big table is not processing pci ids.. OK
> 
>> This IP discovery table will indicate graphics IP and that is mapped to
>> relevant driver code.  So for example GC 11.5.1 is what you find in a Strix
>> Halo APU.  If a future APU had exactly the same graphics IP but a different
>> display IP it would still be GC 11.5.1.
> 
> Move this stuff to drivers/firmware and iommu and GPU can access it
> together?
> 
> "IP discovery table" sure sounds like a platform firmware datablob to
> me at least?
> 
> Jason

Yes; it comes from platform firmware.

Alex, what do you think of doing this?

RE: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Deucher, Alexander]

Public

> -----Original Message-----
> From: Limonciello, Mario <Mario.Limonciello@amd.com>
> Sent: Monday, September 28, 2026 1:15 PM
> To: Jason Gunthorpe <jgg@ziepe.ca>; Deucher, Alexander
> <Alexander.Deucher@amd.com>
> Cc: Mario Limonciello <superm1@gmail.com>; Joerg Roedel
> <joro@8bytes.org>; amd-gfx@lists.freedesktop.org; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Hegde, Vasant
> <Vasant.Hegde@amd.com>; Will Deacon <will@kernel.org>; Robin Murphy
> <robin.murphy@arm.com>; open list:AMD IOMMU (AMD-VI)
> <iommu@lists.linux.dev>; Jatin Kataria <jkataria@netflix.com>; Boqun Feng
> <boqunf@netflix.com>; Derek J . Clark <derekjohn.clark@gmail.com>
> Subject: Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance
> optimization support
>
>
>
> On 9/28/26 12:11, Jason Gunthorpe wrote:
> > On Mon, Sep 28, 2026 at 10:32:41AM -0500, Mario Limonciello wrote:
> >
> >> Actually; no.  amdgpu has migrated away from PCI IDs and probes based
> >> on the "ATI vendor" + "display class" instead.  As part of the probe
> >> sequence there is something called an IP discovery table pull from
> >> the platform that identifies all IP blocks in the hardware.
> >
> > Oh, so that big table is not processing pci ids.. OK
> >
> >> This IP discovery table will indicate graphics IP and that is mapped
> >> to relevant driver code.  So for example GC 11.5.1 is what you find
> >> in a Strix Halo APU.  If a future APU had exactly the same graphics
> >> IP but a different display IP it would still be GC 11.5.1.
> >
> > Move this stuff to drivers/firmware and iommu and GPU can access it
> > together?
> >
> > "IP discovery table" sure sounds like a platform firmware datablob to
> > me at least?
> >
> > Jason
>
> Yes; it comes from platform firmware.
>
> Alex, what do you think of doing this?

It's not platform specific, it's AMD GPU specific.  It also exists on dGPUs which will work in any platform.  I guess we could put some sort of data table in there for this sort of thing but as it stands today, I don't really see it as being much different from PCI IDs with respect to the IOMMU.  I think IVRS tables would be a better fit for this since they are supposed to describe the behavior of the IOMMU with respect to the devices present on the system.

Alex

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Christoph Hellwig]

On Thu, Sep 24, 2026 at 07:50:18PM -0300, Jason Gunthorpe wrote:
> On Mon, Sep 07, 2026 at 11:12:06PM -0500, Mario Limonciello wrote:
> > +EXPORT_SYMBOL_GPL(amd_iommu_enable_perfopt);
> 
> Last time we hardwired the GPU and IOMMU drivers together it was a
> huge mess to undo.

Every time a driver depends on an exact iommu driver is a major design
problem.

> Why does the GPU driver have to request this? Why can't the iommu
> driver know that this is a special device that can use this magic fast
> path and then auto set it when identity is asked for?
>
> How does the GPU driver even know it is magic special?

Yes.  And WTF is "perfopt" supposed to mean.  Everything is a
performance optimization.

Re: [PATCH v2 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Joerg Roedel]

On Mon, Sep 07, 2026 at 11:12:06PM -0500, Mario Limonciello wrote:
> +#ifdef CONFIG_AMD_IOMMU
> +int amd_iommu_enable_perfopt(struct pci_dev *pdev);
> +void amd_iommu_disable_perfopt(struct pci_dev *pdev);
> +#else
> +static inline int amd_iommu_enable_perfopt(struct pci_dev *pdev)
> +{
> +	return 0;
> +}
> +static inline void amd_iommu_disable_perfopt(struct pci_dev *pdev) { }
> +#endif

This commit broke build on i386, I fixed it up but please compile test on
32-bit next time.

-Joerg

[PATCH v2 2/2] drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in identity domain

[Mario Limonciello]

Enable PerfOpt via amd_iommu_enable_perfopt() when the GPU's
iommu_perfopt module parameter is enabled (default 1) and the GPU
resides in the identity domain. The identity domain means the GPU is
already performing direct DMA with the IOMMU only enforcing IR/IW
permission bits -- no GPA->SPA translations.

amd_iommu_enable_perfopt() clears ATS, PRI, PASID and SVA for the
device. This is safe in identity domain because DTE[I]=0 means the
IOMMU already returns target abort for ATS requests from this
peripheral and the GPU manages its own TLB.

PerfOpt is a soft, optional latency optimization: failing to arm it (for
example on an IOMMU that does not implement the feature, which returns
-ENODEV) must not be fatal, so probe and resume warn and continue rather
than aborting.

PERF_OPT_EN is a per-IOMMU control shared by all devices behind that
IOMMU; the IOMMU driver reference counts it so that on systems where
multiple devices share one IOMMU, one GPU's teardown does not clear the
bit while a peer still requires it.

Arming PerfOpt trades IOMMU DMA containment for lower DMA latency. This
is enabled by default for GPUs in the identity domain as a deliberate,
documented policy and can be disabled with iommu_perfopt=0.

The AMD IOMMU spec indicates this is only supported on integrated GPUs
so check explicitly for AMD_IS_APU (which is set by
amdgpu_device_ip_early_init()).

PerfOpt is disabled during GPU init teardown and restored on resume.

Tested-by: Derek J. Clark <derekjohn.clark@gmail.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
v2:
 * Add tag (Derek)
 * Use -1 for default module option (Alex)
 * Add a function to check whether to use perf opt
   Allows one place to control all the places for policy.
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 50 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 12 ++++++
 3 files changed, 63 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 7974f9b7944f3..930a78db7132e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -156,6 +156,7 @@ struct amdgpu_watchdog_timer {
  * Modules parameters.
  */
 extern int amdgpu_modeset;
+extern int amdgpu_iommu_perfopt;
 extern unsigned int amdgpu_vram_limit;
 extern int amdgpu_vis_vram_limit;
 extern int amdgpu_gart_size;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 104d1d2cbad96..57dedf3cffdaa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -33,6 +33,7 @@
 #include <linux/console.h>
 #include <linux/slab.h>
 #include <linux/iommu.h>
+#include <linux/amd-iommu.h>
 #include <linux/pci.h>
 #include <linux/pci-p2pdma.h>
 #include <linux/apple-gmux.h>
@@ -3755,6 +3756,28 @@ amdgpu_device_should_register_switcheroo(struct amdgpu_device *adev, bool px)
 		       apple_gmux_detect(NULL, NULL)));
 }
 
+static inline bool amdgpu_device_identity(struct amdgpu_device *adev)
+{
+	struct pci_dev *pdev = adev->pdev;
+	struct iommu_domain *domain = iommu_get_domain_for_dev(&pdev->dev);
+
+	if (!domain)
+		return false;
+
+	return domain->type == IOMMU_DOMAIN_IDENTITY;
+}
+
+static bool amdgpu_device_use_perfopt(struct amdgpu_device *adev)
+{
+	if (amdgpu_iommu_perfopt == 0)
+		return false;
+
+	if (!(adev->flags & AMD_IS_APU))
+		return false;
+
+	return amdgpu_device_identity(adev);
+}
+
 /**
  * amdgpu_device_init - initialize the driver
  *
@@ -3961,6 +3984,16 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	if (r)
 		return r;
 
+	if (amdgpu_device_use_perfopt(adev)) {
+		int perfopt_ret = amd_iommu_enable_perfopt(pdev);
+
+		/* Optional optimization; a failure to arm it must not abort probe. */
+		if (perfopt_ret)
+			dev_warn(adev->dev,
+				 "Failed to enable IOMMU PerfOpt (%d); continuing without it\n",
+				 perfopt_ret);
+	}
+
 	/*
 	 * No need to remove conflicting FBs for non-display class devices.
 	 * This prevents the sysfb from being freed accidently.
@@ -4330,6 +4363,9 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev)
 
 	amdgpu_gart_dummy_page_fini(adev);
 
+	if (amdgpu_device_use_perfopt(adev))
+		amd_iommu_disable_perfopt(adev->pdev);
+
 	if (pci_dev_is_disconnected(adev->pdev))
 		amdgpu_device_unmap_mmio(adev);
 
@@ -4695,6 +4731,20 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients)
 	if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)
 		return 0;
 
+	if (amdgpu_device_use_perfopt(adev)) {
+		int perfopt_ret = amd_iommu_enable_perfopt(adev->pdev);
+
+		/*
+		 * Must not return on failure: a bare return would leak the
+		 * SR-IOV VF exclusive-mode acquisition taken above (released
+		 * via the exit: path).
+		 */
+		if (perfopt_ret)
+			dev_warn(adev->dev,
+				 "Failed to enable IOMMU PerfOpt (%d); continuing without it\n",
+				 perfopt_ret);
+	}
+
 	if (adev->in_s0ix)
 		amdgpu_dpm_gfx_state_change(adev, sGpuChangeState_D0Entry);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 5b08afde37baf..7b54ca8beeb4c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -185,6 +185,7 @@ char *amdgpu_disable_cu;
 char *amdgpu_virtual_display;
 int amdgpu_enforce_isolation = -1;
 int amdgpu_modeset = -1;
+int amdgpu_iommu_perfopt = -1;
 
 /* Specifies the default granularity for SVM, used in buffer
  * migration and restoration of backing memory when handling
@@ -392,6 +393,17 @@ module_param_named(fw_load_type, amdgpu_fw_load_type, int, 0444);
 MODULE_PARM_DESC(aspm, "ASPM support (1 = enable, 0 = disable, -1 = auto)");
 module_param_named(aspm, amdgpu_aspm, int, 0444);
 
+/**
+ * DOC: iommu_perfopt (int)
+ * Control the AMD IOMMU PerfOpt DMA-latency optimization
+ * (0 = disable; -1 = enable on supported devices).
+ * This arms the IOMMU PerfOpt control (IOMMU spec, MMIO Offset 016Ch, EFR PerfOptSup / PerfOptEn)
+ * Arming it disables ATS, PRI, PASID and SVA for the GPU and removes IOMMU DMA containment for it,
+ * trading isolation for lower DMA latency.
+ */
+MODULE_PARM_DESC(iommu_perfopt, "Control IOMMU PerfOpt DMA-latency optimization (-1 = enable on supported devices, 0 = disable)");
+module_param_named(iommu_perfopt, amdgpu_iommu_perfopt, int, 0444);
+
 /**
  * DOC: runpm (int)
  * Override for runtime power management control for dGPUs. The amdgpu driver can dynamically power down
-- 
2.43.0

Re: [PATCH v2 0/2] IOMMU Performance Optimization support

[Mario Limonciello]

On 9/7/26 11:12 PM, Mario Limonciello wrote:
> Add support for the AMD IOMMU Performance Optimization (PerfOpt)
> feature as defined in the AMD I/O Virtualization Technology (IOMMU)
> Specification, Section 3.4.9 (MMIO Offset 016Ch).
> 
> This feature allows privileged integrated I/O devices (GPUs) to bypass
> the IOMMU when directly accessing system memory.  The IOMMU only
> enforces the IR/IW permission bits without GPA->SPA translations.
> 
> Drivers will opt into this behavior, and enable amdgpu to turn this
> feature on when an integrated GPU is already in identity mode.
> If a user prefers to keep it always off (even with identity mode)
> there is also a module parameter to force it off.
> 
> v2:
>   * Rebase on 7.3-rc2
>   * Adjust amdgpu parameter behavior
> 
> Mario Limonciello (2):
>    iommu/amd: Add PerfOpt IOMMU performance optimization support
>    drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in
>      identity domain
> 
>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  50 +++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  12 ++
>   drivers/iommu/amd/amd_iommu.h              |   3 +
>   drivers/iommu/amd/amd_iommu_types.h        |   7 +
>   drivers/iommu/amd/init.c                   |  43 +++++
>   drivers/iommu/amd/iommu.c                  | 214 +++++++++++++++++++++
>   include/linux/amd-iommu.h                  |  11 ++
>   8 files changed, 341 insertions(+)
> 

Alex, Vasant, Joerg,

Gentle ping on this series.

Thanks,

Re: [PATCH v2 0/2] IOMMU Performance Optimization support

[Boqun Feng]

On Mon, Sep 07, 2026 at 11:12:05PM -0500, Mario Limonciello wrote:
> Add support for the AMD IOMMU Performance Optimization (PerfOpt)
> feature as defined in the AMD I/O Virtualization Technology (IOMMU)
> Specification, Section 3.4.9 (MMIO Offset 016Ch).
> 
> This feature allows privileged integrated I/O devices (GPUs) to bypass
> the IOMMU when directly accessing system memory.  The IOMMU only
> enforces the IR/IW permission bits without GPA->SPA translations.
> 
> Drivers will opt into this behavior, and enable amdgpu to turn this
> feature on when an integrated GPU is already in identity mode.
> If a user prefers to keep it always off (even with identity mode)
> there is also a module parameter to force it off.
> 

Our GPU memory accessing benchmark confirms the improvement: without the
patchset, GTT is 2x - 3x slower than VRAM, after this, the latency gap
is almost gone (within 10%). Nice work!

Tested-by: Boqun Feng (Netflix) <boqun@kernel.org>

Regards,
Boqun

> v2:
>  * Rebase on 7.3-rc2
>  * Adjust amdgpu parameter behavior
> 
> Mario Limonciello (2):
>   iommu/amd: Add PerfOpt IOMMU performance optimization support
>   drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in
>     identity domain
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  50 +++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  12 ++
>  drivers/iommu/amd/amd_iommu.h              |   3 +
>  drivers/iommu/amd/amd_iommu_types.h        |   7 +
>  drivers/iommu/amd/init.c                   |  43 +++++
>  drivers/iommu/amd/iommu.c                  | 214 +++++++++++++++++++++
>  include/linux/amd-iommu.h                  |  11 ++
>  8 files changed, 341 insertions(+)
> 
> -- 
> 2.43.0
> 

Re: [PATCH v2 0/2] IOMMU Performance Optimization support

[Joerg Roedel]

Hi Alex,

On Mon, Sep 07, 2026 at 11:12:05PM -0500, Mario Limonciello wrote:
> Mario Limonciello (2):
>   iommu/amd: Add PerfOpt IOMMU performance optimization support
>   drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in
>     identity domain
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  50 +++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  12 ++
>  drivers/iommu/amd/amd_iommu.h              |   3 +
>  drivers/iommu/amd/amd_iommu_types.h        |   7 +
>  drivers/iommu/amd/init.c                   |  43 +++++
>  drivers/iommu/amd/iommu.c                  | 214 +++++++++++++++++++++
>  include/linux/amd-iommu.h                  |  11 ++
>  8 files changed, 341 insertions(+)

What is your preference on what tree this should go through? The IOMMU changes
are bigger, so my preference is the iommu tree, but for that I need your Ack on
the GPU part. Or let me know whether you prefer this to go through your tree.

-Joerg

RE: [PATCH v2 0/2] IOMMU Performance Optimization support

[Deucher, Alexander]

Public

> -----Original Message-----
> From: Joerg Roedel <joro@8bytes.org>
> Sent: Thursday, September 24, 2026 7:43 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: Limonciello, Mario <Mario.Limonciello@amd.com>; amd-
> gfx@lists.freedesktop.org; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Hegde, Vasant
> <Vasant.Hegde@amd.com>; Will Deacon <will@kernel.org>; Robin Murphy
> <robin.murphy@arm.com>; open list:AMD IOMMU (AMD-VI)
> <iommu@lists.linux.dev>; Jatin Kataria <jkataria@netflix.com>; Boqun Feng
> <boqunf@netflix.com>
> Subject: Re: [PATCH v2 0/2] IOMMU Performance Optimization support
>
> Hi Alex,
>
> On Mon, Sep 07, 2026 at 11:12:05PM -0500, Mario Limonciello wrote:
> > Mario Limonciello (2):
> >   iommu/amd: Add PerfOpt IOMMU performance optimization support
> >   drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in
> >     identity domain
> >
> >  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  50 +++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  12 ++
> >  drivers/iommu/amd/amd_iommu.h              |   3 +
> >  drivers/iommu/amd/amd_iommu_types.h        |   7 +
> >  drivers/iommu/amd/init.c                   |  43 +++++
> >  drivers/iommu/amd/iommu.c                  | 214 +++++++++++++++++++++
> >  include/linux/amd-iommu.h                  |  11 ++
> >  8 files changed, 341 insertions(+)
>
> What is your preference on what tree this should go through? The IOMMU
> changes are bigger, so my preference is the iommu tree, but for that I need
> your Ack on the GPU part. Or let me know whether you prefer this to go
> through your tree.

IOMMU tree is fine with me:
Acked-by: Alex Deucher <alexander.deucher@amd.com>
For the series.

Alex

Re: [PATCH v2 0/2] IOMMU Performance Optimization support

[Joerg Roedel]

On Thu, Sep 24, 2026 at 01:01:59PM +0000, Deucher, Alexander wrote:
> IOMMU tree is fine with me:
> Acked-by: Alex Deucher <alexander.deucher@amd.com>
> For the series.

Alright, this is now applied, thanks!