[RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Matthew Brost]

The dma-map IOVA alloc, link, and sync APIs perform significantly better
than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
This difference is especially noticeable when mapping a 2MB region in
4KB pages.

Use dma-map IOVA alloc, link, and sync APIs for GPU SVM and DRM page,
which mappings between the CPU and GPU.

Initial results are promising.

Baseline CPU time during 2M / 64K fault with a migration:
Average migrate 2M cpu time (us, percentage): 552.36049107142857142857, .71943789893868318799
Average migrate 64K cpu time (us, percentage): 24.97767857142857142857, .34789908128526791960

After this series CPU time during 2M / 64K fault with a migration:
Average migrate 2M cpu time (us, percentage): 224.81808035714285714286, .51412827364772602557
Average migrate 64K cpu time (us, percentage): 14.65625000000000000000, .25659463050529524405

Sending as an RFC as there appears to be intermittent memory corruptions
when running xe_exec_system_allocator that need to be tracked down.

Matt

v2:
 - Include missing basline patch for CI

Francois Dugast (1):
  drm/pagemap: Add helper to access zone_device_data

Matthew Brost (2):
  drm/gpusvm: Use new dma-map IOVA alloc, link, and sync API in GPU SVM
  drm/pagemap: Use new dma-map IOVA alloc, link, and sync API for DRM
    pagemap

 drivers/gpu/drm/drm_gpusvm.c  |  70 ++++++++++-----
 drivers/gpu/drm/drm_pagemap.c | 156 +++++++++++++++++++++++++---------
 include/drm/drm_gpusvm.h      |   3 +
 include/drm/drm_pagemap.h     |  14 +++
 4 files changed, 183 insertions(+), 60 deletions(-)

-- 
2.34.1

[RFC PATCH v2 1/3] drm/pagemap: Add helper to access zone_device_data

[Matthew Brost]

From: Francois Dugast <francois.dugast@intel.com>

This new helper helps ensure all accesses to zone_device_data use the
correct API whether the page is part of a folio or not.

v2:
- Move to drm_pagemap.h, stick to folio_zone_device_data (Matthew Brost)
- Return struct drm_pagemap_zdd * (Matthew Brost)

Suggested-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
---
 drivers/gpu/drm/drm_gpusvm.c  |  7 +++++--
 drivers/gpu/drm/drm_pagemap.c | 21 ++++++++++++---------
 include/drm/drm_pagemap.h     | 14 ++++++++++++++
 3 files changed, 31 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
index aa9a0b60e727..585d913d3d19 100644
--- a/drivers/gpu/drm/drm_gpusvm.c
+++ b/drivers/gpu/drm/drm_gpusvm.c
@@ -1488,12 +1488,15 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 		order = drm_gpusvm_hmm_pfn_to_order(pfns[i], i, npages);
 		if (is_device_private_page(page) ||
 		    is_device_coherent_page(page)) {
+			struct drm_pagemap_zdd *__zdd =
+				drm_pagemap_page_zone_device_data(page);
+
 			if (!ctx->allow_mixed &&
-			    zdd != page->zone_device_data && i > 0) {
+			    zdd != __zdd && i > 0) {
 				err = -EOPNOTSUPP;
 				goto err_unmap;
 			}
-			zdd = page->zone_device_data;
+			zdd = __zdd;
 			if (pagemap != page_pgmap(page)) {
 				if (i > 0) {
 					err = -EOPNOTSUPP;
diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
index 03ee39a761a4..2c67aabd8d65 100644
--- a/drivers/gpu/drm/drm_pagemap.c
+++ b/drivers/gpu/drm/drm_pagemap.c
@@ -244,7 +244,7 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
 		order = folio_order(folio);
 
 		if (is_device_private_page(page)) {
-			struct drm_pagemap_zdd *zdd = page->zone_device_data;
+			struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
 			struct drm_pagemap *dpagemap = zdd->dpagemap;
 			struct drm_pagemap_addr addr;
 
@@ -315,7 +315,7 @@ static void drm_pagemap_migrate_unmap_pages(struct device *dev,
 			goto next;
 
 		if (is_zone_device_page(page)) {
-			struct drm_pagemap_zdd *zdd = page->zone_device_data;
+			struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
 			struct drm_pagemap *dpagemap = zdd->dpagemap;
 
 			dpagemap->ops->device_unmap(dpagemap, dev, pagemap_addr[i]);
@@ -603,7 +603,8 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
 
 		pages[i] = NULL;
 		if (src_page && is_device_private_page(src_page)) {
-			struct drm_pagemap_zdd *src_zdd = src_page->zone_device_data;
+			struct drm_pagemap_zdd *src_zdd =
+				drm_pagemap_page_zone_device_data(src_page);
 
 			if (page_pgmap(src_page) == pagemap &&
 			    !mdetails->can_migrate_same_pagemap) {
@@ -725,8 +726,8 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas,
 			goto next;
 
 		if (fault_page) {
-			if (src_page->zone_device_data !=
-			    fault_page->zone_device_data)
+			if (drm_pagemap_page_zone_device_data(src_page) !=
+			    drm_pagemap_page_zone_device_data(fault_page))
 				goto next;
 		}
 
@@ -1067,7 +1068,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 	void *buf;
 	int i, err = 0;
 
-	zdd = page->zone_device_data;
+	zdd = drm_pagemap_page_zone_device_data(page);
 	if (time_before64(get_jiffies_64(), zdd->devmem_allocation->timeslice_expiration))
 		return 0;
 
@@ -1150,7 +1151,9 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
  */
 static void drm_pagemap_folio_free(struct folio *folio)
 {
-	drm_pagemap_zdd_put(folio->page.zone_device_data);
+	struct page *page = folio_page(folio, 0);
+
+	drm_pagemap_zdd_put(drm_pagemap_page_zone_device_data(page));
 }
 
 /**
@@ -1166,7 +1169,7 @@ static void drm_pagemap_folio_free(struct folio *folio)
  */
 static vm_fault_t drm_pagemap_migrate_to_ram(struct vm_fault *vmf)
 {
-	struct drm_pagemap_zdd *zdd = vmf->page->zone_device_data;
+	struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(vmf->page);
 	int err;
 
 	err = __drm_pagemap_migrate_to_ram(vmf->vma,
@@ -1232,7 +1235,7 @@ EXPORT_SYMBOL_GPL(drm_pagemap_devmem_init);
  */
 struct drm_pagemap *drm_pagemap_page_to_dpagemap(struct page *page)
 {
-	struct drm_pagemap_zdd *zdd = page->zone_device_data;
+	struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
 
 	return zdd->devmem_allocation->dpagemap;
 }
diff --git a/include/drm/drm_pagemap.h b/include/drm/drm_pagemap.h
index 2baf0861f78f..c6459f4bde8b 100644
--- a/include/drm/drm_pagemap.h
+++ b/include/drm/drm_pagemap.h
@@ -4,6 +4,7 @@
 
 #include <linux/dma-direction.h>
 #include <linux/hmm.h>
+#include <linux/memremap.h>
 #include <linux/types.h>
 
 #define NR_PAGES(order) (1U << (order))
@@ -367,6 +368,19 @@ void drm_pagemap_destroy(struct drm_pagemap *dpagemap, bool is_atomic_or_reclaim
 
 int drm_pagemap_reinit(struct drm_pagemap *dpagemap);
 
+/**
+ * drm_pagemap_page_zone_device_data() - Page to zone_device_data
+ * @page: Pointer to the page
+ *
+ * Return: Page's zone_device_data
+ */
+static inline struct drm_pagemap_zdd *drm_pagemap_page_zone_device_data(struct page *page)
+{
+	struct folio *folio = page_folio(page);
+
+	return folio_zone_device_data(folio);
+}
+
 #endif /* IS_ENABLED(CONFIG_ZONE_DEVICE) */
 
 #endif
-- 
2.34.1

[RFC PATCH v2 2/3] drm/gpusvm: Use new dma-map IOVA alloc, link, and sync API in GPU SVM

[Matthew Brost]

The dma-map IOVA alloc, link, and sync APIs perform significantly better
than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
This difference is especially noticeable when mapping a 2MB region in
4KB pages.

Use the IOVA alloc, link, and sync APIs for GPU SVM, which create DMA
mappings between the CPU and GPU.

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/drm_gpusvm.c | 63 +++++++++++++++++++++++++-----------
 include/drm/drm_gpusvm.h     |  3 ++
 2 files changed, 47 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
index 585d913d3d19..eff764445108 100644
--- a/drivers/gpu/drm/drm_gpusvm.c
+++ b/drivers/gpu/drm/drm_gpusvm.c
@@ -1139,19 +1139,26 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 		struct drm_gpusvm_pages_flags flags = {
 			.__flags = svm_pages->flags.__flags,
 		};
+		struct dma_iova_state __state = {};
 
-		for (i = 0, j = 0; i < npages; j++) {
-			struct drm_pagemap_addr *addr = &svm_pages->dma_addr[j];
-
-			if (addr->proto == DRM_INTERCONNECT_SYSTEM)
-				dma_unmap_page(dev,
-					       addr->addr,
-					       PAGE_SIZE << addr->order,
-					       addr->dir);
-			else if (dpagemap && dpagemap->ops->device_unmap)
-				dpagemap->ops->device_unmap(dpagemap,
-							    dev, *addr);
-			i += 1 << addr->order;
+		if (dma_use_iova(&svm_pages->state)) {
+			dma_iova_destroy(dev, &svm_pages->state,
+					 npages * PAGE_SIZE,
+					 svm_pages->dma_addr[0].dir, 0);
+		} else {
+			for (i = 0, j = 0; i < npages; j++) {
+				struct drm_pagemap_addr *addr = &svm_pages->dma_addr[j];
+
+				if (addr->proto == DRM_INTERCONNECT_SYSTEM)
+					dma_unmap_page(dev,
+						       addr->addr,
+						       PAGE_SIZE << addr->order,
+						       addr->dir);
+				else if (dpagemap && dpagemap->ops->device_unmap)
+					dpagemap->ops->device_unmap(dpagemap,
+								    dev, *addr);
+				i += 1 << addr->order;
+			}
 		}
 
 		/* WRITE_ONCE pairs with READ_ONCE for opportunistic checks */
@@ -1161,6 +1168,7 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 
 		drm_pagemap_put(svm_pages->dpagemap);
 		svm_pages->dpagemap = NULL;
+		svm_pages->state = __state;
 	}
 }
 
@@ -1408,6 +1416,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 	struct drm_gpusvm_pages_flags flags;
 	enum dma_data_direction dma_dir = ctx->read_only ? DMA_TO_DEVICE :
 							   DMA_BIDIRECTIONAL;
+	struct dma_iova_state *state = &svm_pages->state;
 
 retry:
 	if (time_after(jiffies, timeout))
@@ -1539,13 +1548,29 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				goto err_unmap;
 			}
 
-			addr = dma_map_page(gpusvm->drm->dev,
-					    page, 0,
-					    PAGE_SIZE << order,
-					    dma_dir);
-			if (dma_mapping_error(gpusvm->drm->dev, addr)) {
-				err = -EFAULT;
-				goto err_unmap;
+			if (!i)
+				dma_iova_try_alloc(gpusvm->drm->dev, state, 0,
+						   npages * PAGE_SIZE);
+
+			if (dma_use_iova(state)) {
+				err = dma_iova_link(gpusvm->drm->dev, state,
+						    page_to_phys(page),
+						    i * PAGE_SIZE,
+						    PAGE_SIZE << order,
+						    dma_dir, 0);
+				if (err)
+					goto err_unmap;
+
+				addr = state->addr + i * PAGE_SIZE;
+			} else {
+				addr = dma_map_page(gpusvm->drm->dev,
+						    page, 0,
+						    PAGE_SIZE << order,
+						    dma_dir);
+				if (dma_mapping_error(gpusvm->drm->dev, addr)) {
+					err = -EFAULT;
+					goto err_unmap;
+				}
 			}
 
 			svm_pages->dma_addr[j] = drm_pagemap_addr_encode
diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h
index 2578ac92a8d4..6772d8a92788 100644
--- a/include/drm/drm_gpusvm.h
+++ b/include/drm/drm_gpusvm.h
@@ -6,6 +6,7 @@
 #ifndef __DRM_GPUSVM_H__
 #define __DRM_GPUSVM_H__
 
+#include <linux/dma-mapping.h>
 #include <linux/kref.h>
 #include <linux/interval_tree.h>
 #include <linux/mmu_notifier.h>
@@ -136,6 +137,7 @@ struct drm_gpusvm_pages_flags {
  * @dma_addr: Device address array
  * @dpagemap: The struct drm_pagemap of the device pages we're dma-mapping.
  *            Note this is assuming only one drm_pagemap per range is allowed.
+ * @state: DMA IOVA state for mapping.
  * @notifier_seq: Notifier sequence number of the range's pages
  * @flags: Flags for range
  * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
@@ -147,6 +149,7 @@ struct drm_gpusvm_pages_flags {
 struct drm_gpusvm_pages {
 	struct drm_pagemap_addr *dma_addr;
 	struct drm_pagemap *dpagemap;
+	struct dma_iova_state state;
 	unsigned long notifier_seq;
 	struct drm_gpusvm_pages_flags flags;
 };
-- 
2.34.1

[RFC PATCH v2 3/3] drm/pagemap: Use new dma-map IOVA alloc, link, and sync API for DRM pagemap

[Matthew Brost]

The dma-map IOVA alloc, link, and sync APIs perform significantly better
than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
This difference is especially noticeable when mapping a 2MB region in
4KB pages.

Use the IOVA alloc, link, and sync APIs for DRM pagemap, which create DMA
mappings between the CPU and GPU for copying data.

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/drm_pagemap.c | 135 ++++++++++++++++++++++++++--------
 1 file changed, 105 insertions(+), 30 deletions(-)

diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
index 2c67aabd8d65..a4749aa77964 100644
--- a/drivers/gpu/drm/drm_pagemap.c
+++ b/drivers/gpu/drm/drm_pagemap.c
@@ -213,6 +213,7 @@ static void drm_pagemap_get_devmem_page(struct page *page,
  * @npages: Number of system pages or peer pages to map.
  * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
  * @mdetails: Details governing the migration behaviour.
+ * @state: DMA IOVA state for mapping.
  *
  * This function maps pages of memory for migration usage in GPU SVM. It
  * iterates over each page frame number provided in @migrate_pfn, maps the
@@ -227,27 +228,46 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
 					 unsigned long *migrate_pfn,
 					 unsigned long npages,
 					 enum dma_data_direction dir,
-					 const struct drm_pagemap_migrate_details *mdetails)
+					 const struct drm_pagemap_migrate_details *mdetails,
+					 struct dma_iova_state *state)
 {
-	unsigned long num_peer_pages = 0, num_local_pages = 0, i;
+	unsigned long num_peer_pages = 0, num_local_pages = 0, psize, i;
+	struct page *dummy_page = NULL;
+	bool try_alloc = false, device_private = false;
+	int err = 0;
 
 	for (i = 0; i < npages;) {
 		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
-		dma_addr_t dma_addr;
-		struct folio *folio;
+		dma_addr_t dma_addr = -1;
 		unsigned int order = 0;
 
-		if (!page)
+		/*
+		 * This loop is a bit goofy, but if IOVA linking is used, the
+		 * entire IOVA range must be populated with physical addresses.
+		 * The first page we find, dummy_page, is therefore used to
+		 * ensure every address in the IOVA range is populated.
+		 */
+
+		if (!page && (device_private || !dummy_page))
 			goto next;
 
-		folio = page_folio(page);
-		order = folio_order(folio);
+		if (!page) {
+			page = dummy_page;
+			psize = PAGE_SIZE;
+		} else {
+			struct folio *folio;
+
+			folio = page_folio(page);
+			order = folio_order(folio);
+			psize = page_size(page);
+		}
 
 		if (is_device_private_page(page)) {
 			struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
 			struct drm_pagemap *dpagemap = zdd->dpagemap;
 			struct drm_pagemap_addr addr;
 
+			device_private = true;
 			if (dpagemap == local_dpagemap) {
 				if (!mdetails->can_migrate_same_pagemap)
 					goto next;
@@ -263,28 +283,68 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
 
 			pagemap_addr[i] = addr;
 		} else {
-			dma_addr = dma_map_page(dev, page, 0, page_size(page), dir);
-			if (dma_mapping_error(dev, dma_addr))
-				return -EFAULT;
+			if (!try_alloc) {
+				dma_iova_try_alloc(dev, state, 0,
+						   npages * PAGE_SIZE);
+				try_alloc = true;
+			}
+
+			if (dma_use_iova(state)) {
+				bool found_dummy = page && !dummy_page;
+
+				if (found_dummy) {
+					unsigned long j;
 
-			pagemap_addr[i] =
-				drm_pagemap_addr_encode(dma_addr,
-							DRM_INTERCONNECT_SYSTEM,
-							order, dir);
+					for (j = 0; j < i; ++j) {
+						err = dma_iova_link(dev, state,
+								    page_to_phys(page),
+								    j * PAGE_SIZE,
+								    PAGE_SIZE,
+								    dir, 0);
+						if (err)
+							return err;
+					}
+				}
+
+				err = dma_iova_link(dev, state, page_to_phys(page),
+						    i * PAGE_SIZE, psize,
+						    dir, 0);
+				if (err)
+					return err;
+
+				if (page != dummy_page)
+					dma_addr = state->addr + i * PAGE_SIZE;
+
+				if (found_dummy)
+					dummy_page = page;
+
+			} else {
+				dma_addr = dma_map_page(dev, page, 0, psize, dir);
+				if (dma_mapping_error(dev, dma_addr))
+					return -EFAULT;
+			}
+
+			if (dma_addr != -1)
+				pagemap_addr[i] =
+					drm_pagemap_addr_encode(dma_addr,
+								DRM_INTERCONNECT_SYSTEM,
+								order, dir);
 		}
 
 next:
 		i += NR_PAGES(order);
 	}
 
-	if (num_peer_pages)
+	if (dma_use_iova(state))
+		err = dma_iova_sync(dev, state, 0, npages * PAGE_SIZE);
+	if (!err && num_peer_pages)
 		drm_dbg(local_dpagemap->drm, "Migrating %lu peer pages over interconnect.\n",
 			num_peer_pages);
-	if (num_local_pages)
+	if (!err && num_local_pages)
 		drm_dbg(local_dpagemap->drm, "Migrating %lu local pages over interconnect.\n",
 			num_local_pages);
 
-	return 0;
+	return err;
 }
 
 /**
@@ -295,6 +355,7 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
  * @pagemap_addr: Array of DMA information corresponding to mapped pages
  * @npages: Number of pages to unmap
  * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ * @state: DMA IOVA state for mapping.
  *
  * This function unmaps previously mapped pages of memory for GPU Shared Virtual
  * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
@@ -304,10 +365,16 @@ static void drm_pagemap_migrate_unmap_pages(struct device *dev,
 					    struct drm_pagemap_addr *pagemap_addr,
 					    unsigned long *migrate_pfn,
 					    unsigned long npages,
-					    enum dma_data_direction dir)
+					    enum dma_data_direction dir,
+					    struct dma_iova_state *state)
 {
 	unsigned long i;
 
+	if (dma_use_iova(state)) {
+		dma_iova_destroy(dev, state, npages * PAGE_SIZE, dir, 0);
+		return;
+	}
+
 	for (i = 0; i < npages;) {
 		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
 
@@ -344,12 +411,14 @@ drm_pagemap_migrate_remote_to_local(struct drm_pagemap_devmem *devmem,
 				    struct drm_pagemap_addr pagemap_addr[],
 				    unsigned long npages,
 				    const struct drm_pagemap_devmem_ops *ops,
-				    const struct drm_pagemap_migrate_details *mdetails)
+				    const struct drm_pagemap_migrate_details *mdetails,
+				    struct dma_iova_state *state)
 
 {
 	int err = drm_pagemap_migrate_map_pages(remote_device, remote_dpagemap,
 						pagemap_addr, local_pfns,
-						npages, DMA_FROM_DEVICE, mdetails);
+						npages, DMA_FROM_DEVICE,
+						mdetails, state);
 
 	if (err)
 		goto out;
@@ -358,7 +427,7 @@ drm_pagemap_migrate_remote_to_local(struct drm_pagemap_devmem *devmem,
 			       devmem->pre_migrate_fence);
 out:
 	drm_pagemap_migrate_unmap_pages(remote_device, pagemap_addr, local_pfns,
-					npages, DMA_FROM_DEVICE);
+					npages, DMA_FROM_DEVICE, state);
 	return err;
 }
 
@@ -369,11 +438,12 @@ drm_pagemap_migrate_sys_to_dev(struct drm_pagemap_devmem *devmem,
 			       struct drm_pagemap_addr pagemap_addr[],
 			       unsigned long npages,
 			       const struct drm_pagemap_devmem_ops *ops,
-			       const struct drm_pagemap_migrate_details *mdetails)
+			       const struct drm_pagemap_migrate_details *mdetails,
+			       struct dma_iova_state *state)
 {
 	int err = drm_pagemap_migrate_map_pages(devmem->dev, devmem->dpagemap,
 						pagemap_addr, sys_pfns, npages,
-						DMA_TO_DEVICE, mdetails);
+						DMA_TO_DEVICE, mdetails, state);
 
 	if (err)
 		goto out;
@@ -382,7 +452,7 @@ drm_pagemap_migrate_sys_to_dev(struct drm_pagemap_devmem *devmem,
 				  devmem->pre_migrate_fence);
 out:
 	drm_pagemap_migrate_unmap_pages(devmem->dev, pagemap_addr, sys_pfns, npages,
-					DMA_TO_DEVICE);
+					DMA_TO_DEVICE, state);
 	return err;
 }
 
@@ -410,6 +480,7 @@ static int drm_pagemap_migrate_range(struct drm_pagemap_devmem *devmem,
 				     const struct migrate_range_loc *cur,
 				     const struct drm_pagemap_migrate_details *mdetails)
 {
+	struct dma_iova_state state = {};
 	int ret = 0;
 
 	if (cur->start == 0)
@@ -429,7 +500,8 @@ static int drm_pagemap_migrate_range(struct drm_pagemap_devmem *devmem,
 							  &pages[last->start],
 							  &pagemap_addr[last->start],
 							  cur->start - last->start,
-							  last->ops, mdetails);
+							  last->ops, mdetails,
+							  &state);
 
 	else
 		ret = drm_pagemap_migrate_sys_to_dev(devmem,
@@ -437,7 +509,7 @@ static int drm_pagemap_migrate_range(struct drm_pagemap_devmem *devmem,
 						     &pages[last->start],
 						     &pagemap_addr[last->start],
 						     cur->start - last->start,
-						     last->ops, mdetails);
+						     last->ops, mdetails, &state);
 
 out:
 	*last = *cur;
@@ -955,6 +1027,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 {
 	const struct drm_pagemap_devmem_ops *ops = devmem_allocation->ops;
 	struct drm_pagemap_migrate_details mdetails = {};
+	struct dma_iova_state state = {};
 	unsigned long npages, mpages = 0;
 	struct page **pages;
 	unsigned long *src, *dst;
@@ -996,7 +1069,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 	err = drm_pagemap_migrate_map_pages(devmem_allocation->dev,
 					    devmem_allocation->dpagemap, pagemap_addr,
 					    dst, npages, DMA_FROM_DEVICE,
-					    &mdetails);
+					    &mdetails, &state);
 	if (err)
 		goto err_finalize;
 
@@ -1013,7 +1086,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 	migrate_device_pages(src, dst, npages);
 	migrate_device_finalize(src, dst, npages);
 	drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, dst, npages,
-					DMA_FROM_DEVICE);
+					DMA_FROM_DEVICE, &state);
 
 err_free:
 	kvfree(buf);
@@ -1058,6 +1131,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 		.fault_page	= page,
 	};
 	struct drm_pagemap_migrate_details mdetails = {};
+	struct dma_iova_state state = {};
 	struct drm_pagemap_zdd *zdd;
 	const struct drm_pagemap_devmem_ops *ops;
 	struct device *dev = NULL;
@@ -1116,7 +1190,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 		goto err_finalize;
 
 	err = drm_pagemap_migrate_map_pages(dev, zdd->dpagemap, pagemap_addr, migrate.dst, npages,
-					    DMA_FROM_DEVICE, &mdetails);
+					    DMA_FROM_DEVICE, &mdetails, &state);
 	if (err)
 		goto err_finalize;
 
@@ -1134,7 +1208,8 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 	migrate_vma_finalize(&migrate);
 	if (dev)
 		drm_pagemap_migrate_unmap_pages(dev, pagemap_addr, migrate.dst,
-						npages, DMA_FROM_DEVICE);
+						npages, DMA_FROM_DEVICE,
+						&state);
 err_free:
 	kvfree(buf);
 err_out:
-- 
2.34.1

Re: [RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Jason Gunthorpe]

On Fri, Jan 23, 2026 at 01:59:04PM -0800, Matthew Brost wrote:
> The dma-map IOVA alloc, link, and sync APIs perform significantly better
> than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
> This difference is especially noticeable when mapping a 2MB region in
> 4KB pages.
> 
> Use dma-map IOVA alloc, link, and sync APIs for GPU SVM and DRM page,
> which mappings between the CPU and GPU.
> 
> Initial results are promising.
> 
> Baseline CPU time during 2M / 64K fault with a migration:
> Average migrate 2M cpu time (us, percentage): 552.36049107142857142857, .71943789893868318799
> Average migrate 64K cpu time (us, percentage): 24.97767857142857142857, .34789908128526791960
> 
> After this series CPU time during 2M / 64K fault with a migration:
> Average migrate 2M cpu time (us, percentage): 224.81808035714285714286, .51412827364772602557
> Average migrate 64K cpu time (us, percentage): 14.65625000000000000000, .25659463050529524405

Thats a 2x improvement in overall full operation? Wow!

Did you look at how non-iommu cases perform too?

I think we can do better still for the non-cached platforms as I have
a way in mind to batch up lines and flush the line instead of flushing
for every 8 byte IOPTE written. Some ARM folks have been talking about
this problem too..

Jason

Re: [RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Matthew Brost]

On Fri, Jan 23, 2026 at 06:41:46PM -0400, Jason Gunthorpe wrote:
> On Fri, Jan 23, 2026 at 01:59:04PM -0800, Matthew Brost wrote:
> > The dma-map IOVA alloc, link, and sync APIs perform significantly better
> > than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
> > This difference is especially noticeable when mapping a 2MB region in
> > 4KB pages.
> > 
> > Use dma-map IOVA alloc, link, and sync APIs for GPU SVM and DRM page,
> > which mappings between the CPU and GPU.
> > 
> > Initial results are promising.
> > 
> > Baseline CPU time during 2M / 64K fault with a migration:
> > Average migrate 2M cpu time (us, percentage): 552.36049107142857142857, .71943789893868318799
> > Average migrate 64K cpu time (us, percentage): 24.97767857142857142857, .34789908128526791960
> > 
> > After this series CPU time during 2M / 64K fault with a migration:
> > Average migrate 2M cpu time (us, percentage): 224.81808035714285714286, .51412827364772602557
> > Average migrate 64K cpu time (us, percentage): 14.65625000000000000000, .25659463050529524405
> 
> Thats a 2x improvement in overall full operation? Wow!
> 
> Did you look at how non-iommu cases perform too?
> 

Like intel_iommu=off kerenl command line? I haven't checked that but can.

> I think we can do better still for the non-cached platforms as I have
> a way in mind to batch up lines and flush the line instead of flushing
> for every 8 byte IOPTE written. Some ARM folks have been talking about
> this problem too..

Yes, prior to the IOMMU changes I believe the basline was ~330us so
dma-map/unmap are still way slower than before and if this affect
platforms other than Intel x86 there will be complaints everyone until
the entire kernel moves to the IOVA alloc model.

Also another question does IOVA alloc support modes similar to
dma_map_resource between per device? We also do that and I haven't
modified that code or check that for perf regressions.

Matt 

> 
> Jason

Re: [RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Jason Gunthorpe]

On Fri, Jan 23, 2026 at 02:53:59PM -0800, Matthew Brost wrote:
> > Thats a 2x improvement in overall full operation? Wow!
> > 
> > Did you look at how non-iommu cases perform too?
> > 
> 
> Like intel_iommu=off kerenl command line? I haven't checked that but can.

iommu.passthrough=1

This is generally what we recommend everyone who cares about
performance more than iommu protection should use by default. It
leaves the iommu HW turned on, which x86 requires for other reasons,
but eliminates the performance cost to DMA.

> > I think we can do better still for the non-cached platforms as I have
> > a way in mind to batch up lines and flush the line instead of flushing
> > for every 8 byte IOPTE written. Some ARM folks have been talking about
> > this problem too..
> 
> Yes, prior to the IOMMU changes I believe the basline was ~330us so
> dma-map/unmap are still way slower than before and if this affect
> platforms other than Intel x86 there will be complaints everyone until
> the entire kernel moves to the IOVA alloc model.

I have managed to get a test showing that when cache flushing is
turned on the new code is 50% slower. I'm investigating this..

map_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    331,249   ,     289,214     , -35.35
     2^21,    335,243   ,     306,222     , -37.37
     2^30,    226,238   ,     205,215     ,   4.04
    # test_map_unmap_benchmark:
unmap_pages
   pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
     2^12,    389,272   ,     347,237     , -46.46
     2^21,    321,261   ,     297,239     , -24.24
     2^30,    237,251   ,     214,228     ,   6.06

So it looks to me like this is isolated to Intel GPU for the moment
because it is the only device that would use the cache flushing flow
until we convert ARM.

FWIW, on my system enabling cache flushing goes from 60ns to 250ns, it
has a huge, huge cost to these flows.

> Also another question does IOVA alloc support modes similar to
> dma_map_resource between per device? We also do that and I haven't
> modified that code or check that for perf regressions.

Yes, and no.. The API does, but Christoph doesn't want to let arbitary
drivers use it. So you need to figure out some way to get there.

For reference Leon added dma_buf_phys_vec_to_sgt() which shows this
flow to create a sg_table.

There are also hmm helpers for the mapping too if this is in a hmm
context.

A PCI device calling map_resource is incorrect usage of the DMA API,
but it was the only option till now.

Jason

Re: [RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Matthew Brost]

On Fri, Jan 23, 2026 at 08:26:10PM -0400, Jason Gunthorpe wrote:
> On Fri, Jan 23, 2026 at 02:53:59PM -0800, Matthew Brost wrote:
> > > Thats a 2x improvement in overall full operation? Wow!
> > > 
> > > Did you look at how non-iommu cases perform too?
> > > 
> > 
> > Like intel_iommu=off kerenl command line? I haven't checked that but can.
> 
> iommu.passthrough=1
> 
> This is generally what we recommend everyone who cares about
> performance more than iommu protection should use by default. It

Yes, worked in HPC for a long time and we always set the IOMMU to
passthrough.

> leaves the iommu HW turned on, which x86 requires for other reasons,
> but eliminates the performance cost to DMA.
> 

iommu.passthrough=1 brings the 2M case to roughly 130us for 2M - this
stat includes migrate_vma_* functions btw, also for reference this time
drops to ~10us in any scenario with 2M device pages.

> > > I think we can do better still for the non-cached platforms as I have
> > > a way in mind to batch up lines and flush the line instead of flushing
> > > for every 8 byte IOPTE written. Some ARM folks have been talking about
> > > this problem too..
> > 
> > Yes, prior to the IOMMU changes I believe the basline was ~330us so
> > dma-map/unmap are still way slower than before and if this affect
> > platforms other than Intel x86 there will be complaints everyone until
> > the entire kernel moves to the IOVA alloc model.
> 
> I have managed to get a test showing that when cache flushing is
> turned on the new code is 50% slower. I'm investigating this..
> 
> map_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    331,249   ,     289,214     , -35.35
>      2^21,    335,243   ,     306,222     , -37.37
>      2^30,    226,238   ,     205,215     ,   4.04
>     # test_map_unmap_benchmark:
> unmap_pages
>    pgsz  ,avg new,old ns, min new,old ns  , min % (+ve is better)
>      2^12,    389,272   ,     347,237     , -46.46
>      2^21,    321,261   ,     297,239     , -24.24
>      2^30,    237,251   ,     214,228     ,   6.06
> 
> So it looks to me like this is isolated to Intel GPU for the moment
> because it is the only device that would use the cache flushing flow
> until we convert ARM.
> 
> FWIW, on my system enabling cache flushing goes from 60ns to 250ns, it
> has a huge, huge cost to these flows.

I see that you have fixed this one, we verfieid it, thanks!.

> 
> > Also another question does IOVA alloc support modes similar to
> > dma_map_resource between per device? We also do that and I haven't
> > modified that code or check that for perf regressions.
> 
> Yes, and no.. The API does, but Christoph doesn't want to let arbitary
> drivers use it. So you need to figure out some way to get there.
> 

Yes, I see that API allows this and it seems to work too.

> For reference Leon added dma_buf_phys_vec_to_sgt() which shows this
> flow to create a sg_table.
> 

That will likely work for dma-buf, let me see if I can convert our
dma-buf flows to use this helper. But it won't work for things like SVM,
so it would be desirable to figure out to have an API drivers can use to
iova alloc/link/sync/unlink/free for multi-device or just agree we trust
drivers enough to use the existing API.

Matt

> There are also hmm helpers for the mapping too if this is in a hmm
> context.
> 
> A PCI device calling map_resource is incorrect usage of the DMA API,
> but it was the only option till now.
> 
> Jason

Re: [RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Jason Gunthorpe]

On Tue, Jan 27, 2026 at 08:54:22AM -0800, Matthew Brost wrote:

> That will likely work for dma-buf, let me see if I can convert our
> dma-buf flows to use this helper. But it won't work for things like SVM,
> so it would be desirable to figure out to have an API drivers can use to
> iova alloc/link/sync/unlink/free for multi-device or just agree we trust
> drivers enough to use the existing API.

SVM should be driven with HMM and there is a helper in
hmm_dma_map_pfn() for this.

Yonatan posted a series to expand it to work with ZONE_DEVICE PRIVATE
pages but it needs a refresh

https://lore.kernel.org/linux-rdma/20250718115112.3881129-1-ymaman@nvidia.com/

If there are other cases it would be reasonable to discuss enhancing
hmm_dma_map_pfn().

Jason

Re: [RFC PATCH v2 0/3] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Matthew Brost]

On Tue, Jan 27, 2026 at 02:37:01PM -0400, Jason Gunthorpe wrote:
> On Tue, Jan 27, 2026 at 08:54:22AM -0800, Matthew Brost wrote:
> 
> > That will likely work for dma-buf, let me see if I can convert our
> > dma-buf flows to use this helper. But it won't work for things like SVM,
> > so it would be desirable to figure out to have an API drivers can use to
> > iova alloc/link/sync/unlink/free for multi-device or just agree we trust
> > drivers enough to use the existing API.
> 
> SVM should be driven with HMM and there is a helper in
> hmm_dma_map_pfn() for this.
> 

Ok, I'm not sure if that will exactly fit how our SVM code is structured.
> Yonatan posted a series to expand it to work with ZONE_DEVICE PRIVATE
> pages but it needs a refresh
> 
> https://lore.kernel.org/linux-rdma/20250718115112.3881129-1-ymaman@nvidia.com/
> 

A brief look, this isn't all that far off from ideas we have in DRM with
ops in the pagemap (DRM pagemap) though to handle P2P mappings. We are
also forward looking to not just DMA connections but high speed fabrics
too. The thinking there was make high speed fabric API look like dma-map
iova alloc/link/sync/unlink/free but let DRM pagemap op pick between the
dma-map API and high speed fabric API based on connection and wrap
everything into DRM common layer (GPU SVM) to map the pages.


> If there are other cases it would be reasonable to discuss enhancing
> hmm_dma_map_pfn().
> 

Let me wrap my head around this one and get back to you. Something to
think about.

Matt

> Jason