[PATCH 0/2] IOMMU Performance Optimization support

[PATCH 0/2] IOMMU Performance Optimization support

[Mario Limonciello]

Add support for the AMD IOMMU Performance Optimization (PerfOpt)
feature as defined in the AMD I/O Virtualization Technology (IOMMU)
Specification, Section 3.4.9 (MMIO Offset 016Ch).

This feature allows privileged integrated I/O devices (GPUs) to bypass
the IOMMU when directly accessing system memory.  The IOMMU only
enforces the IR/IW permission bits without GPA->SPA translations.

Drivers will opt into this behavior, and enable amdgpu to turn this
feature on when an integrated GPU is already in identity mode.
If a user prefers to keep it always off (even with identity mode)
there is also a module parameter to force it off.

Mario Limonciello (2):
  iommu/amd: Add PerfOpt IOMMU performance optimization support
  drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in
    identity domain

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  41 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  12 ++
 drivers/iommu/amd/amd_iommu.h              |   3 +
 drivers/iommu/amd/amd_iommu_types.h        |   7 +
 drivers/iommu/amd/init.c                   |  43 +++++
 drivers/iommu/amd/iommu.c                  | 214 +++++++++++++++++++++
 include/linux/amd-iommu.h                  |  11 ++
 8 files changed, 332 insertions(+)

-- 
2.43.0

[PATCH 1/2] iommu/amd: Add PerfOpt IOMMU performance optimization support

[Mario Limonciello]

Add support for the AMD IOMMU Performance Optimization (PerfOpt)
feature as defined in the AMD I/O Virtualization Technology (IOMMU)
Specification, Section 3.4.9 (MMIO Offset 016Ch).

This feature allows privileged integrated I/O devices (GPUs) to bypass
the IOMMU when directly accessing system memory.  The IOMMU only
enforces the IR/IW permission bits without GPA->SPA translations.

amd_iommu_enable_perfopt() performs a detach/reattach cycle to rehome
devices already on the identity domain with ATS/PRI/PASID/GCR3
disabled (skip_caps path).  amd_iommu_disable_perfopt() restores those
capabilities.  The per-device dev_data->perfopt flag tracks state.

PERF_OPT_EN is a single control bit per IOMMU, shared by every device
behind that IOMMU, while enablement is requested per device.  It is
therefore reference counted (amd_iommu->perfopt_refcount): armed on the
first requesting device and cleared on the last, so one device's
teardown never clears the bit while a peer behind the same IOMMU still
needs it.

The per-device flag is cleared on every teardown path
(blocked_domain_attach, release_device, and amd_iommu_disable_perfopt),
dropping the reference with it, so a reused dev_data never carries stale
PerfOpt state onto its next bind.

On suspend/resume the hardware is reprogrammed from scratch:
amd_iommu_perfopt_clear() forces the bit off without touching the
reference count, and amd_iommu_perfopt_restore() re-asserts it from the
count after early_enable_iommu(), so armed devices keep the optimization
across resume without relying on each consumer driver to re-arm.

The exported amd_iommu_enable_perfopt()/amd_iommu_disable_perfopt() run
only from a consumer driver's bind/unbind path.  group->mutex is not
exposed to drivers, but a device bound to its native driver cannot have
its IOMMU domain changed concurrently by the core, which serializes the
detach/attach pair against core-driven attach.

PerfOpt is opt-in -- only enabled when explicitly requested by a driver.

Co-developed-by: Jatin Kataria <jkataria@netflix.com>
Signed-off-by: Jatin Kataria <jkataria@netflix.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/iommu/amd/amd_iommu.h       |   3 +
 drivers/iommu/amd/amd_iommu_types.h |   7 +
 drivers/iommu/amd/init.c            |  43 ++++++
 drivers/iommu/amd/iommu.c           | 214 ++++++++++++++++++++++++++++
 include/linux/amd-iommu.h           |  11 ++
 5 files changed, 278 insertions(+)

diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
index a2fe804b038b6..1f8f9df8e6c24 100644
--- a/drivers/iommu/amd/amd_iommu.h
+++ b/drivers/iommu/amd/amd_iommu.h
@@ -48,6 +48,9 @@ extern u8 amd_iommu_hpt_vasize;
 extern unsigned long amd_iommu_pgsize_bitmap;
 extern bool amd_iommu_hatdis;
 
+int amd_iommu_perfopt_clear(struct amd_iommu *iommu);
+int amd_iommu_perfopt_restore(struct amd_iommu *iommu);
+
 /* Protection domain ops */
 void amd_iommu_init_identity_domain(void);
 struct protection_domain *protection_domain_alloc(void);
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 3dbe20023456b..755421e5cd757 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -65,6 +65,7 @@
 #define MMIO_MSI_ADDR_LO_OFFSET	0x015C
 #define MMIO_MSI_ADDR_HI_OFFSET	0x0160
 #define MMIO_MSI_DATA_OFFSET	0x0164
+#define MMIO_PERF_OPT_OFFSET	0x016C
 #define MMIO_INTCAPXT_EVT_OFFSET	0x0170
 #define MMIO_INTCAPXT_PPR_OFFSET	0x0178
 #define MMIO_INTCAPXT_GALOG_OFFSET	0x0180
@@ -99,6 +100,8 @@
 #define FEATURE_GLX		GENMASK_ULL(15, 14)
 #define FEATURE_GAM_VAPIC	BIT_ULL(21)
 #define FEATURE_PASMAX		GENMASK_ULL(36, 32)
+#define FEATURE_PERF_OPT	BIT_ULL(45)
+#define PERF_OPT_EN		BIT(13)
 #define FEATURE_GIOSUP		BIT_ULL(48)
 #define FEATURE_HASUP		BIT_ULL(49)
 #define FEATURE_EPHSUP		BIT_ULL(50)
@@ -670,6 +673,9 @@ struct amd_iommu {
 	/* Extended features 2 */
 	u64 features2;
 
+	/* Devices requesting PerfOpt; the shared PERF_OPT_EN bit is on while >0. Protected by @lock. */
+	int perfopt_refcount;
+
 	/* PCI device id of the IOMMU device */
 	u16 devid;
 
@@ -831,6 +837,7 @@ struct iommu_dev_data {
 	u8 ppr          :1;		  /* Enable device PPR support */
 	bool use_vapic;			  /* Enable device to use vapic mode */
 	bool defer_attach;
+	bool perfopt;
 
 	struct ratelimit_state rs;        /* Ratelimit IOPF messages */
 };
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 40726dfef2733..ddcf56f101675 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1942,6 +1942,9 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h,
 	if (!iommu->mmio_base)
 		return -ENOMEM;
 
+	if (amd_iommu_perfopt_clear(iommu))
+		pr_err("IOMMU%d: failed to clear PerfOpt\n", iommu->index);
+
 	return init_iommu_from_acpi(iommu, h);
 }
 
@@ -3032,10 +3035,46 @@ static void enable_iommus_vapic(void)
 #endif
 }
 
+static int clear_perfopt_all(void)
+{
+	struct amd_iommu *iommu;
+	int err, ret = 0;
+
+	for_each_iommu(iommu) {
+		err = amd_iommu_perfopt_clear(iommu);
+		if (err)
+			ret = err;
+	}
+
+	return ret;
+}
+
+static int restore_perfopt_all(void)
+{
+	struct amd_iommu *iommu;
+	int err, ret = 0;
+
+	for_each_iommu(iommu) {
+		err = amd_iommu_perfopt_restore(iommu);
+		if (err)
+			ret = err;
+	}
+
+	return ret;
+}
+
 static void disable_iommus(void)
 {
 	struct amd_iommu *iommu;
 
+	/*
+	 * PerfOpt is an optional performance bit, so a failure to clear it must
+	 * not skip the mandatory disable below. This also runs from the void
+	 * amd_iommu_disable() shutdown/kexec path, which cannot report an error.
+	 */
+	if (clear_perfopt_all())
+		pr_err("Failed to clear PerfOpt while disabling IOMMUs\n");
+
 	for_each_iommu(iommu)
 		iommu_disable(iommu);
 
@@ -3061,6 +3100,10 @@ static void amd_iommu_resume(void *data)
 	for_each_iommu(iommu)
 		early_enable_iommu(iommu);
 
+	/* early_enable_iommu() cleared PERF_OPT_EN; re-assert it from the refcount. */
+	if (restore_perfopt_all())
+		pr_err("Failed to restore PerfOpt after IOMMU resume\n");
+
 	iommu_enable_event_buffer();
 	amd_iommu_enable_interrupts();
 }
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 4dc306a4b5c62..fa60affdfc035 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2395,6 +2395,9 @@ static int attach_device(struct device *dev,
 	if (ret)
 		goto out;
 
+	if (dev_data->perfopt)
+		goto skip_caps;
+
 	/* Setup GCR3 table */
 	if (pdom_is_sva_capable(domain)) {
 		ret = init_gcr3_table(dev_data, domain);
@@ -2419,6 +2422,7 @@ static int attach_device(struct device *dev,
 		pdev_enable_cap_ats(pdev);
 	}
 
+skip_caps:
 	/* Update data structures */
 	dev_data->domain = domain;
 	spin_lock_irqsave(&domain->lock, flags);
@@ -2487,6 +2491,192 @@ static void detach_device(struct device *dev)
 	mutex_unlock(&dev_data->mutex);
 }
 
+/* Program the per-IOMMU PerfOpt enable bit. Caller must hold iommu->lock. */
+static int __perfopt_write(struct amd_iommu *iommu, bool enable)
+{
+	u32 old, val, readback;
+
+	if (!(readq(iommu->mmio_base + MMIO_EXT_FEATURES) & FEATURE_PERF_OPT))
+		return enable ? -ENODEV : 0;
+
+	old = readl(iommu->mmio_base + MMIO_PERF_OPT_OFFSET);
+	if (old == U32_MAX)
+		return -EIO;
+
+	val = enable ? old | PERF_OPT_EN : old & ~PERF_OPT_EN;
+	if (val != old)
+		writel(val, iommu->mmio_base + MMIO_PERF_OPT_OFFSET);
+	readback = readl(iommu->mmio_base + MMIO_PERF_OPT_OFFSET);
+	if (readback == U32_MAX ||
+	    (readback & PERF_OPT_EN) != (val & PERF_OPT_EN))
+		return -EIO;
+	return 0;
+}
+
+/*
+ * PERF_OPT_EN is a single bit shared by every device behind @iommu, so it is
+ * reference counted: armed on the first requesting device, cleared on the last.
+ */
+static int perfopt_get(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	if (iommu->perfopt_refcount == 0) {
+		ret = __perfopt_write(iommu, true);
+		if (ret)
+			goto out;
+	}
+	iommu->perfopt_refcount++;
+out:
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+static int perfopt_put(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	if (iommu->perfopt_refcount > 0 && --iommu->perfopt_refcount == 0)
+		ret = __perfopt_write(iommu, false);
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+/*
+ * Force PERF_OPT_EN off without touching the refcount (used on init, shutdown,
+ * and suspend). The count is preserved so amd_iommu_perfopt_restore() can
+ * re-arm on resume.
+ */
+int amd_iommu_perfopt_clear(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	ret = __perfopt_write(iommu, false);
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+/*
+ * Re-assert PERF_OPT_EN from the refcount after the hardware was reprogrammed on
+ * resume, so devices armed before suspend keep the optimization without each
+ * consumer driver re-arming.
+ */
+int amd_iommu_perfopt_restore(struct amd_iommu *iommu)
+{
+	unsigned long flags;
+	int ret;
+
+	if (!iommu->mmio_base)
+		return 0;
+
+	raw_spin_lock_irqsave(&iommu->lock, flags);
+	ret = __perfopt_write(iommu, iommu->perfopt_refcount > 0);
+	raw_spin_unlock_irqrestore(&iommu->lock, flags);
+	return ret;
+}
+
+int amd_iommu_enable_perfopt(struct pci_dev *pdev)
+{
+	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
+	struct amd_iommu *iommu = rlookup_amd_iommu(&pdev->dev);
+	struct protection_domain *domain;
+	int ret;
+
+	if (!iommu || !dev_data)
+		return -ENODEV;
+
+	if (!(iommu->features & FEATURE_PERF_OPT))
+		return -ENODEV;
+
+	domain = dev_data->domain;
+	if (!domain)
+		return -ENODEV;
+
+	/* Already armed for this device (e.g. re-entry on resume). */
+	if (dev_data->perfopt)
+		return 0;
+
+	/*
+	 * The bit is only architecturally valid while the device is untranslated:
+	 * identity domain with ATS/PRI/PASID off. The identity domain is
+	 * SVA-capable so attach_device() enabled ATS/PRI/PASID and built a GCR3
+	 * table. Re-home the device onto the same identity domain with
+	 * perfopt set, so the attach_device() skip_caps path leaves
+	 * ATS/PRI/PASID off and no GCR3 table. This follows the detach/attach
+	 * pattern used by amd_iommu_attach_device().
+	 *
+	 * Locking: this and amd_iommu_disable_perfopt() run only from the
+	 * consumer driver's bind/unbind path. group->mutex is not exposed to
+	 * drivers, but a device bound to its native driver cannot have its domain
+	 * changed concurrently by the core (VFIO ownership is mutually exclusive;
+	 * sysfs domain changes require an unused group), so the detach/attach pair
+	 * is serialized without it.
+	 */
+	dev_data->perfopt = true;
+	detach_device(&pdev->dev);
+	ret = attach_device(&pdev->dev, domain);
+	if (ret)
+		goto err_restore;
+
+	ret = perfopt_get(iommu);
+	if (ret)
+		goto err_rearm;
+
+	dev_info_once(&pdev->dev, "PerfOpt armed on IOMMU%d\n", iommu->index);
+	return 0;
+
+err_rearm:
+	detach_device(&pdev->dev);
+err_restore:
+	dev_data->perfopt = false;
+	if (attach_device(&pdev->dev, domain))
+		pci_err(pdev, "failed to restore state after PerfOpt setup; device left detached\n");
+	dev_err_once(&pdev->dev, "PerfOpt failed to arm on IOMMU%d (%d)\n",
+		     iommu->index, ret);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(amd_iommu_enable_perfopt);
+
+void amd_iommu_disable_perfopt(struct pci_dev *pdev)
+{
+	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
+	struct amd_iommu *iommu = rlookup_amd_iommu(&pdev->dev);
+	struct protection_domain *domain;
+
+	if (!iommu || !dev_data || !dev_data->perfopt || !dev_data->domain)
+		return;
+
+	if (WARN_ON(perfopt_put(iommu)))
+		pci_err(pdev, "failed to clear PerfOpt\n");
+
+	/*
+	 * Restore ATS/PRI/PASID (and thus SVA) by re-homing the device onto its
+	 * identity domain with the flag cleared, so a later bind without PerfOpt
+	 * sees a normally-capable device. See the locking note in
+	 * amd_iommu_enable_perfopt().
+	 */
+	domain = dev_data->domain;
+	dev_data->perfopt = false;
+	detach_device(&pdev->dev);
+	if (attach_device(&pdev->dev, domain))
+		pci_err(pdev, "failed to restore caps after PerfOpt disable\n");
+}
+EXPORT_SYMBOL_GPL(amd_iommu_disable_perfopt);
 static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 {
 	struct iommu_device *iommu_dev;
@@ -2554,6 +2744,14 @@ static struct iommu_device *amd_iommu_probe_device(struct device *dev)
 static void amd_iommu_release_device(struct device *dev)
 {
 	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
+	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+
+	if (dev_data->perfopt) {
+		if (WARN_ON(perfopt_put(iommu)))
+			dev_err(dev, "IOMMU%d: failed to clear PerfOpt on release\n",
+				iommu->index);
+		dev_data->perfopt = false;
+	}
 
 	WARN_ON(dev_data->domain);
 
@@ -2928,6 +3126,19 @@ static int blocked_domain_attach_device(struct iommu_domain *domain,
 					struct iommu_domain *old)
 {
 	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
+	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
+
+	/*
+	 * blocked_domain is also the .release_domain, so this is the normal
+	 * teardown path: drop the reference and clear the flag here too, and
+	 * don't fail teardown if the WARN-guarded write doesn't stick.
+	 */
+	if (dev_data->perfopt) {
+		if (WARN_ON(perfopt_put(iommu)))
+			dev_err(dev, "IOMMU%d: failed to clear PerfOpt for blocked domain\n",
+				iommu->index);
+		dev_data->perfopt = false;
+	}
 
 	if (dev_data->domain)
 		detach_device(dev);
@@ -2996,6 +3207,9 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, struct device *dev,
 	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev);
 	int ret;
 
+	if (dev_data->perfopt && !pdom_is_in_pt_mode(domain))
+		return -EBUSY;
+
 	/*
 	 * Skip attach device to domain if new domain is same as
 	 * devices current domain
diff --git a/include/linux/amd-iommu.h b/include/linux/amd-iommu.h
index edcee9f5335a6..e03575cbc08c6 100644
--- a/include/linux/amd-iommu.h
+++ b/include/linux/amd-iommu.h
@@ -76,4 +76,15 @@ static inline int amd_iommu_snp_disable(void) { return 0; }
 static inline bool amd_iommu_sev_tio_supported(void) { return false; }
 #endif
 
+#ifdef CONFIG_AMD_IOMMU
+int amd_iommu_enable_perfopt(struct pci_dev *pdev);
+void amd_iommu_disable_perfopt(struct pci_dev *pdev);
+#else
+static inline int amd_iommu_enable_perfopt(struct pci_dev *pdev)
+{
+	return 0;
+}
+static inline void amd_iommu_disable_perfopt(struct pci_dev *pdev) { }
+#endif
+
 #endif /* _ASM_X86_AMD_IOMMU_H */
-- 
2.43.0

[PATCH 2/2] drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in identity domain

[Mario Limonciello]

Enable PerfOpt via amd_iommu_enable_perfopt() when the GPU's
iommu_perfopt module parameter is enabled (default 1) and the GPU
resides in the identity domain. The identity domain means the GPU is
already performing direct DMA with the IOMMU only enforcing IR/IW
permission bits -- no GPA->SPA translations.

amd_iommu_enable_perfopt() clears ATS, PRI, PASID and SVA for the
device. This is safe in identity domain because DTE[I]=0 means the
IOMMU already returns target abort for ATS requests from this
peripheral and the GPU manages its own TLB.

PerfOpt is a soft, optional latency optimization: failing to arm it (for
example on an IOMMU that does not implement the feature, which returns
-ENODEV) must not be fatal, so probe and resume warn and continue rather
than aborting.

PERF_OPT_EN is a per-IOMMU control shared by all devices behind that
IOMMU; the IOMMU driver reference counts it so that on systems where
multiple devices share one IOMMU, one GPU's teardown does not clear the
bit while a peer still requires it.

Arming PerfOpt trades IOMMU DMA containment for lower DMA latency. This
is enabled by default for GPUs in the identity domain as a deliberate,
documented policy and can be disabled with iommu_perfopt=0.

The AMD IOMMU spec indicates this is only supported on integrated GPUs
so check explicitly for AMD_IS_APU (which is set by
amdgpu_device_ip_early_init()).

PerfOpt is disabled during GPU init teardown and restored on resume.

Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 41 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 12 +++++++
 3 files changed, 54 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 7974f9b7944f3..930a78db7132e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -156,6 +156,7 @@ struct amdgpu_watchdog_timer {
  * Modules parameters.
  */
 extern int amdgpu_modeset;
+extern int amdgpu_iommu_perfopt;
 extern unsigned int amdgpu_vram_limit;
 extern int amdgpu_vis_vram_limit;
 extern int amdgpu_gart_size;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 44bed0ba64a37..75c74c72a52ea 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -33,6 +33,7 @@
 #include <linux/console.h>
 #include <linux/slab.h>
 #include <linux/iommu.h>
+#include <linux/amd-iommu.h>
 #include <linux/pci.h>
 #include <linux/pci-p2pdma.h>
 #include <linux/apple-gmux.h>
@@ -3752,6 +3753,17 @@ amdgpu_device_should_register_switcheroo(struct amdgpu_device *adev, bool px)
 		       apple_gmux_detect(NULL, NULL)));
 }
 
+static inline bool amdgpu_device_identity(struct amdgpu_device *adev)
+{
+	struct pci_dev *pdev = adev->pdev;
+	struct iommu_domain *domain = iommu_get_domain_for_dev(&pdev->dev);
+
+	if (!domain)
+		return false;
+
+	return domain->type == IOMMU_DOMAIN_IDENTITY;
+}
+
 /**
  * amdgpu_device_init - initialize the driver
  *
@@ -3958,6 +3970,18 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	if (r)
 		return r;
 
+	if (amdgpu_iommu_perfopt != 0 &&
+	    amdgpu_device_identity(adev) &&
+	    adev->flags & AMD_IS_APU) {
+		int perfopt_ret = amd_iommu_enable_perfopt(pdev);
+
+		/* Optional optimization; a failure to arm it must not abort probe. */
+		if (perfopt_ret)
+			dev_warn(adev->dev,
+				 "Failed to enable IOMMU PerfOpt (%d); continuing without it\n",
+				 perfopt_ret);
+	}
+
 	/*
 	 * No need to remove conflicting FBs for non-display class devices.
 	 * This prevents the sysfb from being freed accidently.
@@ -4327,6 +4351,9 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev)
 
 	amdgpu_gart_dummy_page_fini(adev);
 
+	if (amdgpu_iommu_perfopt != 0)
+		amd_iommu_disable_perfopt(adev->pdev);
+
 	if (pci_dev_is_disconnected(adev->pdev))
 		amdgpu_device_unmap_mmio(adev);
 
@@ -4692,6 +4719,20 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients)
 	if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)
 		return 0;
 
+	if (amdgpu_iommu_perfopt != 0 && amdgpu_device_identity(adev)) {
+		int perfopt_ret = amd_iommu_enable_perfopt(adev->pdev);
+
+		/*
+		 * Must not return on failure: a bare return would leak the
+		 * SR-IOV VF exclusive-mode acquisition taken above (released
+		 * via the exit: path).
+		 */
+		if (perfopt_ret)
+			dev_warn(adev->dev,
+				 "Failed to enable IOMMU PerfOpt (%d); continuing without it\n",
+				 perfopt_ret);
+	}
+
 	if (adev->in_s0ix)
 		amdgpu_dpm_gfx_state_change(adev, sGpuChangeState_D0Entry);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 5b08afde37baf..26ba7852e1ceb 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -185,6 +185,7 @@ char *amdgpu_disable_cu;
 char *amdgpu_virtual_display;
 int amdgpu_enforce_isolation = -1;
 int amdgpu_modeset = -1;
+int amdgpu_iommu_perfopt = 1;
 
 /* Specifies the default granularity for SVM, used in buffer
  * migration and restoration of backing memory when handling
@@ -392,6 +393,17 @@ module_param_named(fw_load_type, amdgpu_fw_load_type, int, 0444);
 MODULE_PARM_DESC(aspm, "ASPM support (1 = enable, 0 = disable, -1 = auto)");
 module_param_named(aspm, amdgpu_aspm, int, 0444);
 
+/**
+ * DOC: iommu_perfopt (int)
+ * Control the AMD IOMMU PerfOpt DMA-latency optimization
+ * (0 = disable; 1 = enable on APU devices in identity domain).
+ * This arms the IOMMU PerfOpt control (IOMMU spec, MMIO Offset 016Ch, EFR PerfOptSup / PerfOptEn)
+ * Arming it disables ATS, PRI, PASID and SVA for the GPU and removes IOMMU DMA containment for it,
+ * trading isolation for lower DMA latency.
+ */
+MODULE_PARM_DESC(iommu_perfopt, "Control IOMMU PerfOpt DMA-latency optimization (1 = enable on APU devices in identity domain, 0 = disable)");
+module_param_named(iommu_perfopt, amdgpu_iommu_perfopt, int, 0444);
+
 /**
  * DOC: runpm (int)
  * Override for runtime power management control for dGPUs. The amdgpu driver can dynamically power down
-- 
2.43.0

RE: [PATCH 2/2] drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in identity domain

[Deucher, Alexander]

Public

> -----Original Message-----
> From: Limonciello, Mario <Mario.Limonciello@amd.com>
> Sent: Monday, August 31, 2026 1:51 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>; Joerg Roedel
> <joro@8bytes.org>
> Cc: amd-gfx@lists.freedesktop.org; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Hegde, Vasant
> <Vasant.Hegde@amd.com>; Will Deacon <will@kernel.org>; Robin Murphy
> <robin.murphy@arm.com>; open list:AMD IOMMU (AMD-VI)
> <iommu@lists.linux.dev>; Jatin Kataria <jkataria@netflix.com>; Boqun Feng
> <boqunf@netflix.com>; Limonciello, Mario <Mario.Limonciello@amd.com>
> Subject: [PATCH 2/2] drm/amdgpu: Enable PerfOpt IOMMU perf optimization
> when GPU in identity domain
>
> Enable PerfOpt via amd_iommu_enable_perfopt() when the GPU's
> iommu_perfopt module parameter is enabled (default 1) and the GPU resides
> in the identity domain. The identity domain means the GPU is already
> performing direct DMA with the IOMMU only enforcing IR/IW permission bits
> -- no GPA->SPA translations.
>
> amd_iommu_enable_perfopt() clears ATS, PRI, PASID and SVA for the device.
> This is safe in identity domain because DTE[I]=0 means the IOMMU already
> returns target abort for ATS requests from this peripheral and the GPU
> manages its own TLB.
>
> PerfOpt is a soft, optional latency optimization: failing to arm it (for example on
> an IOMMU that does not implement the feature, which returns
> -ENODEV) must not be fatal, so probe and resume warn and continue rather
> than aborting.
>
> PERF_OPT_EN is a per-IOMMU control shared by all devices behind that
> IOMMU; the IOMMU driver reference counts it so that on systems where
> multiple devices share one IOMMU, one GPU's teardown does not clear the bit
> while a peer still requires it.
>
> Arming PerfOpt trades IOMMU DMA containment for lower DMA latency. This
> is enabled by default for GPUs in the identity domain as a deliberate,
> documented policy and can be disabled with iommu_perfopt=0.
>
> The AMD IOMMU spec indicates this is only supported on integrated GPUs so
> check explicitly for AMD_IS_APU (which is set by
> amdgpu_device_ip_early_init()).
>
> PerfOpt is disabled during GPU init teardown and restored on resume.
>
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 41
> ++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 12 +++++++
>  3 files changed, 54 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index 7974f9b7944f3..930a78db7132e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -156,6 +156,7 @@ struct amdgpu_watchdog_timer {
>   * Modules parameters.
>   */
>  extern int amdgpu_modeset;
> +extern int amdgpu_iommu_perfopt;
>  extern unsigned int amdgpu_vram_limit;
>  extern int amdgpu_vis_vram_limit;
>  extern int amdgpu_gart_size;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 44bed0ba64a37..75c74c72a52ea 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -33,6 +33,7 @@
>  #include <linux/console.h>
>  #include <linux/slab.h>
>  #include <linux/iommu.h>
> +#include <linux/amd-iommu.h>
>  #include <linux/pci.h>
>  #include <linux/pci-p2pdma.h>
>  #include <linux/apple-gmux.h>
> @@ -3752,6 +3753,17 @@
> amdgpu_device_should_register_switcheroo(struct amdgpu_device *adev,
> bool px)
>                      apple_gmux_detect(NULL, NULL)));  }
>
> +static inline bool amdgpu_device_identity(struct amdgpu_device *adev) {
> +     struct pci_dev *pdev = adev->pdev;
> +     struct iommu_domain *domain =
> iommu_get_domain_for_dev(&pdev->dev);
> +
> +     if (!domain)
> +             return false;
> +
> +     return domain->type == IOMMU_DOMAIN_IDENTITY; }
> +
>  /**
>   * amdgpu_device_init - initialize the driver
>   *
> @@ -3958,6 +3970,18 @@ int amdgpu_device_init(struct amdgpu_device
> *adev,
>       if (r)
>               return r;
>
> +     if (amdgpu_iommu_perfopt != 0 &&
> +         amdgpu_device_identity(adev) &&
> +         adev->flags & AMD_IS_APU) {
> +             int perfopt_ret = amd_iommu_enable_perfopt(pdev);
> +
> +             /* Optional optimization; a failure to arm it must not abort
> probe. */
> +             if (perfopt_ret)
> +                     dev_warn(adev->dev,
> +                              "Failed to enable IOMMU PerfOpt (%d);
> continuing without it\n",
> +                              perfopt_ret);
> +     }
> +
>       /*
>        * No need to remove conflicting FBs for non-display class devices.
>        * This prevents the sysfb from being freed accidently.
> @@ -4327,6 +4351,9 @@ void amdgpu_device_fini_hw(struct
> amdgpu_device *adev)
>
>       amdgpu_gart_dummy_page_fini(adev);
>
> +     if (amdgpu_iommu_perfopt != 0)
> +             amd_iommu_disable_perfopt(adev->pdev);
> +
>       if (pci_dev_is_disconnected(adev->pdev))
>               amdgpu_device_unmap_mmio(adev);
>
> @@ -4692,6 +4719,20 @@ int amdgpu_device_resume(struct drm_device
> *dev, bool notify_clients)
>       if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)
>               return 0;
>
> +     if (amdgpu_iommu_perfopt != 0 && amdgpu_device_identity(adev)) {
> +             int perfopt_ret = amd_iommu_enable_perfopt(adev->pdev);
> +
> +             /*
> +              * Must not return on failure: a bare return would leak the
> +              * SR-IOV VF exclusive-mode acquisition taken above (released
> +              * via the exit: path).
> +              */
> +             if (perfopt_ret)
> +                     dev_warn(adev->dev,
> +                              "Failed to enable IOMMU PerfOpt (%d);
> continuing without it\n",
> +                              perfopt_ret);
> +     }
> +
>       if (adev->in_s0ix)
>               amdgpu_dpm_gfx_state_change(adev,
> sGpuChangeState_D0Entry);
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 5b08afde37baf..26ba7852e1ceb 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -185,6 +185,7 @@ char *amdgpu_disable_cu;  char
> *amdgpu_virtual_display;  int amdgpu_enforce_isolation = -1;  int
> amdgpu_modeset = -1;
> +int amdgpu_iommu_perfopt = 1;
>
>  /* Specifies the default granularity for SVM, used in buffer
>   * migration and restoration of backing memory when handling @@ -392,6
> +393,17 @@ module_param_named(fw_load_type, amdgpu_fw_load_type,
> int, 0444);  MODULE_PARM_DESC(aspm, "ASPM support (1 = enable, 0 =
> disable, -1 = auto)");  module_param_named(aspm, amdgpu_aspm, int,
> 0444);
>
> +/**
> + * DOC: iommu_perfopt (int)
> + * Control the AMD IOMMU PerfOpt DMA-latency optimization
> + * (0 = disable; 1 = enable on APU devices in identity domain).
> + * This arms the IOMMU PerfOpt control (IOMMU spec, MMIO Offset
> 016Ch,
> +EFR PerfOptSup / PerfOptEn)
> + * Arming it disables ATS, PRI, PASID and SVA for the GPU and removes
> +IOMMU DMA containment for it,
> + * trading isolation for lower DMA latency.
> + */
> +MODULE_PARM_DESC(iommu_perfopt, "Control IOMMU PerfOpt DMA-
> latency
> +optimization (1 = enable on APU devices in identity domain, 0 =
> +disable)"); module_param_named(iommu_perfopt,
> amdgpu_iommu_perfopt,
> +int, 0444);

For consistency with the other driver parameters, I would make the default -1 (auto) and have auto enable for APU devices.  That way we can add 1 as an option later if we want to add a way to force it on in certain scenarios.

Akex

> +
>  /**
>   * DOC: runpm (int)
>   * Override for runtime power management control for dGPUs. The amdgpu
> driver can dynamically power down
> --
> 2.43.0

Re: [PATCH 0/2] IOMMU Performance Optimization support

[Derek J. Clark]

On 8/30/26 22:51, Mario Limonciello wrote:
> Add support for the AMD IOMMU Performance Optimization (PerfOpt)
> feature as defined in the AMD I/O Virtualization Technology (IOMMU)
> Specification, Section 3.4.9 (MMIO Offset 016Ch).
>
> This feature allows privileged integrated I/O devices (GPUs) to bypass
> the IOMMU when directly accessing system memory.  The IOMMU only
> enforces the IR/IW permission bits without GPA->SPA translations.
>
> Drivers will opt into this behavior, and enable amdgpu to turn this
> feature on when an integrated GPU is already in identity mode.
> If a user prefers to keep it always off (even with identity mode)
> there is also a module parameter to force it off.
>
> Mario Limonciello (2):
>    iommu/amd: Add PerfOpt IOMMU performance optimization support
>    drm/amdgpu: Enable PerfOpt IOMMU perf optimization when GPU in
>      identity domain
>
>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  41 ++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  12 ++
>   drivers/iommu/amd/amd_iommu.h              |   3 +
>   drivers/iommu/amd/amd_iommu_types.h        |   7 +
>   drivers/iommu/amd/init.c                   |  43 +++++
>   drivers/iommu/amd/iommu.c                  | 214 +++++++++++++++++++++
>   include/linux/amd-iommu.h                  |  11 ++
>   8 files changed, 332 insertions(+)

Hi Mario,


The Legion Go 2 has a significant performance degradation when IOMMU is 
ON and VRAM is set to auto in the BIOS. I applied these to linux-next 
and compared to 7.2 neptune on the Legion Go 2 and got the following 
results:
Game: Avowed
With patches:

Go 2| Vram: 2GB, IOMMU: Off, FPS: 35-40, Frame Time: consistent
Go 2| Vram: 2GB, IOMMU: On, FPS: 35-40, Frame Time: consistent
Go 2| Vram: Auto, IOMMU: Off, FPS: 35-40, Frame Time: consistent
Go 2| Vram: Auto, IOMMU: On, FPS: 35-40, Frame Time: consistent

Without Patches:

Go 2| Vram: 2GB, IOMMU: Off, FPS: 35-40, Frame Time: consistent
Go 2| Vram: 2GB, IOMMU: On, FPS: 35-40, Frame Time: consistent
Go 2| Vram: Auto, IOMMU: Off, FPS: 35-40, Frame Time: consistent
Go 2| Vram: Auto, IOMMU: On, FPS: 20, Frame Time: inconsistent

Overall, this seems to fix the performance issue with IOMMU when BIOS is 
in the factory default setting.

Tested-by: Derek J. Clark <derekjohn.clark@gmail.com>