[PATCH v4 0/4] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[PATCH v4 0/4] Use new dma-map IOVA alloc, link, and sync API in GPU SVM and DRM pagemap

[Matthew Brost]

The dma-map IOVA alloc, link, and sync APIs perform significantly better
than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
This difference is especially noticeable when mapping a 2MB region in
4KB pages.

Use dma-map IOVA alloc, link, and sync APIs for GPU SVM and DRM page,
which mappings between the CPU and GPU.

Initial results are promising.

Baseline CPU time during 2M / 64K fault with a migration:
Average migrate 2M cpu time (us, percentage): 333.99665178571428571429, .61102853199282922865
Average migrate 64K cpu time (us, percentage): 18.62723214285714285714, .30127985269960467173

After this series CPU time during 2M / 64K fault with a migration:
Average migrate 2M cpu time (us, percentage): 224.81808035714285714286, .51412827364772602557
Average migrate 64K cpu time (us, percentage): 14.65625000000000000000, .25659463050529524405

Matt

v2:
 - Include missing basline patch for CI
v3:
 - Fix memory corruption
 - PoC IOVA alloc for multi-GPU
v4:
 - Pack IOVA / drop dummy pages
 - Drop multi-GPU IOVA alloc

Francois Dugast (1):
  drm/pagemap: Add helper to access zone_device_data

Matthew Brost (3):
  drm/gpusvm: Use dma-map IOVA alloc, link, and sync API in GPU SVM
  drm/pagemap: Split drm_pagemap_migrate_map_pages into device / system
  drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM
    pagemap

 drivers/gpu/drm/drm_gpusvm.c  |  62 +++++++--
 drivers/gpu/drm/drm_pagemap.c | 229 +++++++++++++++++++++++++---------
 include/drm/drm_gpusvm.h      |   5 +
 include/drm/drm_pagemap.h     |  14 +++
 4 files changed, 238 insertions(+), 72 deletions(-)

-- 
2.34.1

[PATCH v4 1/4] drm/pagemap: Add helper to access zone_device_data

[Matthew Brost]

From: Francois Dugast <francois.dugast@intel.com>

This new helper helps ensure all accesses to zone_device_data use the
correct API whether the page is part of a folio or not.

Suggested-by: Matthew Brost <matthew.brost@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
---
v2:
- Move to drm_pagemap.h, stick to folio_zone_device_data (Matthew Brost)
- Return struct drm_pagemap_zdd * (Matthew Brost)

 drivers/gpu/drm/drm_gpusvm.c  |  7 +++++--
 drivers/gpu/drm/drm_pagemap.c | 21 ++++++++++++---------
 include/drm/drm_pagemap.h     | 14 ++++++++++++++
 3 files changed, 31 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
index 871fcccd128a..4b8130a4ce95 100644
--- a/drivers/gpu/drm/drm_gpusvm.c
+++ b/drivers/gpu/drm/drm_gpusvm.c
@@ -1488,12 +1488,15 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 		order = drm_gpusvm_hmm_pfn_to_order(pfns[i], i, npages);
 		if (is_device_private_page(page) ||
 		    is_device_coherent_page(page)) {
+			struct drm_pagemap_zdd *__zdd =
+				drm_pagemap_page_zone_device_data(page);
+
 			if (!ctx->allow_mixed &&
-			    zdd != page->zone_device_data && i > 0) {
+			    zdd != __zdd && i > 0) {
 				err = -EOPNOTSUPP;
 				goto err_unmap;
 			}
-			zdd = page->zone_device_data;
+			zdd = __zdd;
 			if (pagemap != page_pgmap(page)) {
 				if (pagemap) {
 					err = -EOPNOTSUPP;
diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
index 38eca94f01a1..fbd69f383457 100644
--- a/drivers/gpu/drm/drm_pagemap.c
+++ b/drivers/gpu/drm/drm_pagemap.c
@@ -244,7 +244,7 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
 		order = folio_order(folio);
 
 		if (is_device_private_page(page)) {
-			struct drm_pagemap_zdd *zdd = page->zone_device_data;
+			struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
 			struct drm_pagemap *dpagemap = zdd->dpagemap;
 			struct drm_pagemap_addr addr;
 
@@ -315,7 +315,7 @@ static void drm_pagemap_migrate_unmap_pages(struct device *dev,
 			goto next;
 
 		if (is_zone_device_page(page)) {
-			struct drm_pagemap_zdd *zdd = page->zone_device_data;
+			struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
 			struct drm_pagemap *dpagemap = zdd->dpagemap;
 
 			dpagemap->ops->device_unmap(dpagemap, dev, pagemap_addr[i]);
@@ -603,7 +603,8 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
 
 		pages[i] = NULL;
 		if (src_page && is_device_private_page(src_page)) {
-			struct drm_pagemap_zdd *src_zdd = src_page->zone_device_data;
+			struct drm_pagemap_zdd *src_zdd =
+				drm_pagemap_page_zone_device_data(src_page);
 
 			if (page_pgmap(src_page) == pagemap &&
 			    !mdetails->can_migrate_same_pagemap) {
@@ -725,8 +726,8 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas,
 			goto next;
 
 		if (fault_page) {
-			if (src_page->zone_device_data !=
-			    fault_page->zone_device_data)
+			if (drm_pagemap_page_zone_device_data(src_page) !=
+			    drm_pagemap_page_zone_device_data(fault_page))
 				goto next;
 		}
 
@@ -1067,7 +1068,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 	void *buf;
 	int i, err = 0;
 
-	zdd = page->zone_device_data;
+	zdd = drm_pagemap_page_zone_device_data(page);
 	if (time_before64(get_jiffies_64(), zdd->devmem_allocation->timeslice_expiration))
 		return 0;
 
@@ -1150,7 +1151,9 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
  */
 static void drm_pagemap_folio_free(struct folio *folio)
 {
-	drm_pagemap_zdd_put(folio->page.zone_device_data);
+	struct page *page = folio_page(folio, 0);
+
+	drm_pagemap_zdd_put(drm_pagemap_page_zone_device_data(page));
 }
 
 /**
@@ -1166,7 +1169,7 @@ static void drm_pagemap_folio_free(struct folio *folio)
  */
 static vm_fault_t drm_pagemap_migrate_to_ram(struct vm_fault *vmf)
 {
-	struct drm_pagemap_zdd *zdd = vmf->page->zone_device_data;
+	struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(vmf->page);
 	int err;
 
 	err = __drm_pagemap_migrate_to_ram(vmf->vma,
@@ -1232,7 +1235,7 @@ EXPORT_SYMBOL_GPL(drm_pagemap_devmem_init);
  */
 struct drm_pagemap *drm_pagemap_page_to_dpagemap(struct page *page)
 {
-	struct drm_pagemap_zdd *zdd = page->zone_device_data;
+	struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
 
 	return zdd->devmem_allocation->dpagemap;
 }
diff --git a/include/drm/drm_pagemap.h b/include/drm/drm_pagemap.h
index 2baf0861f78f..14e1db564c25 100644
--- a/include/drm/drm_pagemap.h
+++ b/include/drm/drm_pagemap.h
@@ -4,6 +4,7 @@
 
 #include <linux/dma-direction.h>
 #include <linux/hmm.h>
+#include <linux/memremap.h>
 #include <linux/types.h>
 
 #define NR_PAGES(order) (1U << (order))
@@ -341,6 +342,19 @@ struct drm_pagemap_migrate_details {
 	u32 source_peer_migrates : 1;
 };
 
+/**
+ * drm_pagemap_page_zone_device_data() - Page to zone_device_data
+ * @page: Pointer to the page
+ *
+ * Return: Page's zone_device_data
+ */
+static inline struct drm_pagemap_zdd *drm_pagemap_page_zone_device_data(struct page *page)
+{
+	struct folio *folio = page_folio(page);
+
+	return folio_zone_device_data(folio);
+}
+
 #if IS_ENABLED(CONFIG_ZONE_DEVICE)
 
 int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
-- 
2.34.1

[PATCH v4 2/4] drm/gpusvm: Use dma-map IOVA alloc, link, and sync API in GPU SVM

[Matthew Brost]

The dma-map IOVA alloc, link, and sync APIs perform significantly better
than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
This difference is especially noticeable when mapping a 2MB region in
4KB pages.

Use the IOVA alloc, link, and sync APIs for GPU SVM, which create DMA
mappings between the CPU and GPU.

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
v3:
 - Always link IOVA in mixed mappings
 - Sync IOVA
v4:
 - Initialize IOVA state in get_pages
 - Use pack IOVA linking (Jason)
 - s/page_to_phys/hmm_pfn_to_phys (Leon)

 drivers/gpu/drm/drm_gpusvm.c | 55 ++++++++++++++++++++++++++++++------
 include/drm/drm_gpusvm.h     |  5 ++++
 2 files changed, 52 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
index 4b8130a4ce95..800caaf0a783 100644
--- a/drivers/gpu/drm/drm_gpusvm.c
+++ b/drivers/gpu/drm/drm_gpusvm.c
@@ -1139,11 +1139,19 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 		struct drm_gpusvm_pages_flags flags = {
 			.__flags = svm_pages->flags.__flags,
 		};
+		bool use_iova = dma_use_iova(&svm_pages->state);
+
+		if (use_iova) {
+			dma_iova_unlink(dev, &svm_pages->state, 0,
+					svm_pages->state_offset,
+					svm_pages->dma_addr[0].dir, 0);
+			dma_iova_free(dev, &svm_pages->state);
+		}
 
 		for (i = 0, j = 0; i < npages; j++) {
 			struct drm_pagemap_addr *addr = &svm_pages->dma_addr[j];
 
-			if (addr->proto == DRM_INTERCONNECT_SYSTEM)
+			if (!use_iova && addr->proto == DRM_INTERCONNECT_SYSTEM)
 				dma_unmap_page(dev,
 					       addr->addr,
 					       PAGE_SIZE << addr->order,
@@ -1408,6 +1416,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 	struct drm_gpusvm_pages_flags flags;
 	enum dma_data_direction dma_dir = ctx->read_only ? DMA_TO_DEVICE :
 							   DMA_BIDIRECTIONAL;
+	struct dma_iova_state *state = &svm_pages->state;
 
 retry:
 	if (time_after(jiffies, timeout))
@@ -1446,6 +1455,9 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 	if (err)
 		goto err_free;
 
+	*state = (struct dma_iova_state){};
+	svm_pages->state_offset = 0;
+
 map_pages:
 	/*
 	 * Perform all dma mappings under the notifier lock to not
@@ -1539,13 +1551,33 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				goto err_unmap;
 			}
 
-			addr = dma_map_page(gpusvm->drm->dev,
-					    page, 0,
-					    PAGE_SIZE << order,
-					    dma_dir);
-			if (dma_mapping_error(gpusvm->drm->dev, addr)) {
-				err = -EFAULT;
-				goto err_unmap;
+			if (!i)
+				dma_iova_try_alloc(gpusvm->drm->dev, state,
+						   npages * PAGE_SIZE >=
+						   HPAGE_PMD_SIZE ?
+						   HPAGE_PMD_SIZE : 0,
+						   npages * PAGE_SIZE);
+
+			if (dma_use_iova(state)) {
+				err = dma_iova_link(gpusvm->drm->dev, state,
+						    hmm_pfn_to_phys(pfns[i]),
+						    svm_pages->state_offset,
+						    PAGE_SIZE << order,
+						    dma_dir, 0);
+				if (err)
+					goto err_unmap;
+
+				addr = state->addr + svm_pages->state_offset;
+				svm_pages->state_offset += PAGE_SIZE << order;
+			} else {
+				addr = dma_map_page(gpusvm->drm->dev,
+						    page, 0,
+						    PAGE_SIZE << order,
+						    dma_dir);
+				if (dma_mapping_error(gpusvm->drm->dev, addr)) {
+					err = -EFAULT;
+					goto err_unmap;
+				}
 			}
 
 			svm_pages->dma_addr[j] = drm_pagemap_addr_encode
@@ -1557,6 +1589,13 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 		flags.has_dma_mapping = true;
 	}
 
+	if (dma_use_iova(state)) {
+		err = dma_iova_sync(gpusvm->drm->dev, state, 0,
+				    svm_pages->state_offset);
+		if (err)
+			goto err_unmap;
+	}
+
 	if (pagemap) {
 		flags.has_devmem_pages = true;
 		drm_pagemap_get(dpagemap);
diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h
index 2578ac92a8d4..cd94bb2ee6ee 100644
--- a/include/drm/drm_gpusvm.h
+++ b/include/drm/drm_gpusvm.h
@@ -6,6 +6,7 @@
 #ifndef __DRM_GPUSVM_H__
 #define __DRM_GPUSVM_H__
 
+#include <linux/dma-mapping.h>
 #include <linux/kref.h>
 #include <linux/interval_tree.h>
 #include <linux/mmu_notifier.h>
@@ -136,6 +137,8 @@ struct drm_gpusvm_pages_flags {
  * @dma_addr: Device address array
  * @dpagemap: The struct drm_pagemap of the device pages we're dma-mapping.
  *            Note this is assuming only one drm_pagemap per range is allowed.
+ * @state: DMA IOVA state for mapping.
+ * @state_offset: DMA IOVA offset for mapping.
  * @notifier_seq: Notifier sequence number of the range's pages
  * @flags: Flags for range
  * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
@@ -147,6 +150,8 @@ struct drm_gpusvm_pages_flags {
 struct drm_gpusvm_pages {
 	struct drm_pagemap_addr *dma_addr;
 	struct drm_pagemap *dpagemap;
+	struct dma_iova_state state;
+	unsigned long state_offset;
 	unsigned long notifier_seq;
 	struct drm_gpusvm_pages_flags flags;
 };
-- 
2.34.1

[PATCH v4 3/4] drm/pagemap: Split drm_pagemap_migrate_map_pages into device / system

[Matthew Brost]

Split drm_pagemap_migrate_map_pages into device / system helpers clearly
seperating these operations. Will help with upcoming changes to split
IOVA allocation steps.

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/drm_pagemap.c | 146 ++++++++++++++++++++++------------
 1 file changed, 96 insertions(+), 50 deletions(-)

diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
index fbd69f383457..29677b19bb69 100644
--- a/drivers/gpu/drm/drm_pagemap.c
+++ b/drivers/gpu/drm/drm_pagemap.c
@@ -205,7 +205,7 @@ static void drm_pagemap_get_devmem_page(struct page *page,
 }
 
 /**
- * drm_pagemap_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * drm_pagemap_migrate_map_device_pages() - Map device migration pages for GPU SVM migration
  * @dev: The device performing the migration.
  * @local_dpagemap: The drm_pagemap local to the migrating device.
  * @pagemap_addr: Array to store DMA information corresponding to mapped pages.
@@ -221,19 +221,22 @@ static void drm_pagemap_get_devmem_page(struct page *page,
  *
  * Returns: 0 on success, -EFAULT if an error occurs during mapping.
  */
-static int drm_pagemap_migrate_map_pages(struct device *dev,
-					 struct drm_pagemap *local_dpagemap,
-					 struct drm_pagemap_addr *pagemap_addr,
-					 unsigned long *migrate_pfn,
-					 unsigned long npages,
-					 enum dma_data_direction dir,
-					 const struct drm_pagemap_migrate_details *mdetails)
+static int
+drm_pagemap_migrate_map_device_pages(struct device *dev,
+				     struct drm_pagemap *local_dpagemap,
+				     struct drm_pagemap_addr *pagemap_addr,
+				     unsigned long *migrate_pfn,
+				     unsigned long npages,
+				     enum dma_data_direction dir,
+				     const struct drm_pagemap_migrate_details *mdetails)
 {
 	unsigned long num_peer_pages = 0, num_local_pages = 0, i;
 
 	for (i = 0; i < npages;) {
 		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
-		dma_addr_t dma_addr;
+		struct drm_pagemap_zdd *zdd;
+		struct drm_pagemap *dpagemap;
+		struct drm_pagemap_addr addr;
 		struct folio *folio;
 		unsigned int order = 0;
 
@@ -243,36 +246,26 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
 		folio = page_folio(page);
 		order = folio_order(folio);
 
-		if (is_device_private_page(page)) {
-			struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
-			struct drm_pagemap *dpagemap = zdd->dpagemap;
-			struct drm_pagemap_addr addr;
-
-			if (dpagemap == local_dpagemap) {
-				if (!mdetails->can_migrate_same_pagemap)
-					goto next;
+		WARN_ON_ONCE(!is_device_private_page(page));
 
-				num_local_pages += NR_PAGES(order);
-			} else {
-				num_peer_pages += NR_PAGES(order);
-			}
+		zdd = drm_pagemap_page_zone_device_data(page);
+		dpagemap = zdd->dpagemap;
 
-			addr = dpagemap->ops->device_map(dpagemap, dev, page, order, dir);
-			if (dma_mapping_error(dev, addr.addr))
-				return -EFAULT;
+		if (dpagemap == local_dpagemap) {
+			if (!mdetails->can_migrate_same_pagemap)
+				goto next;
 
-			pagemap_addr[i] = addr;
+			num_local_pages += NR_PAGES(order);
 		} else {
-			dma_addr = dma_map_page(dev, page, 0, page_size(page), dir);
-			if (dma_mapping_error(dev, dma_addr))
-				return -EFAULT;
-
-			pagemap_addr[i] =
-				drm_pagemap_addr_encode(dma_addr,
-							DRM_INTERCONNECT_SYSTEM,
-							order, dir);
+			num_peer_pages += NR_PAGES(order);
 		}
 
+		addr = dpagemap->ops->device_map(dpagemap, dev, page, order, dir);
+		if (dma_mapping_error(dev, addr.addr))
+			return -EFAULT;
+
+		pagemap_addr[i] = addr;
+
 next:
 		i += NR_PAGES(order);
 	}
@@ -287,6 +280,59 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
 	return 0;
 }
 
+/**
+ * drm_pagemap_migrate_map_system_pages() - Map system migration pages for GPU SVM migration
+ * @dev: The device performing the migration.
+ * @pagemap_addr: Array to store DMA information corresponding to mapped pages.
+ * @migrate_pfn: Array of page frame numbers of system pages or peer pages to map.
+ * @npages: Number of system pages or peer pages to map.
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Returns: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int
+drm_pagemap_migrate_map_system_pages(struct device *dev,
+				     struct drm_pagemap_addr *pagemap_addr,
+				     unsigned long *migrate_pfn,
+				     unsigned long npages,
+				     enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages;) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+		dma_addr_t dma_addr;
+		struct folio *folio;
+		unsigned int order = 0;
+
+		if (!page)
+			goto next;
+
+		WARN_ON_ONCE(is_device_private_page(page));
+		folio = page_folio(page);
+		order = folio_order(folio);
+
+		dma_addr = dma_map_page(dev, page, 0, page_size(page), dir);
+		if (dma_mapping_error(dev, dma_addr))
+			return -EFAULT;
+
+		pagemap_addr[i] =
+			drm_pagemap_addr_encode(dma_addr,
+						DRM_INTERCONNECT_SYSTEM,
+						order, dir);
+
+next:
+		i += NR_PAGES(order);
+	}
+
+	return 0;
+}
+
 /**
  * drm_pagemap_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
  * @dev: The device for which the pages were mapped
@@ -347,9 +393,11 @@ drm_pagemap_migrate_remote_to_local(struct drm_pagemap_devmem *devmem,
 				    const struct drm_pagemap_migrate_details *mdetails)
 
 {
-	int err = drm_pagemap_migrate_map_pages(remote_device, remote_dpagemap,
-						pagemap_addr, local_pfns,
-						npages, DMA_FROM_DEVICE, mdetails);
+	int err = drm_pagemap_migrate_map_device_pages(remote_device,
+						       remote_dpagemap,
+						       pagemap_addr, local_pfns,
+						       npages, DMA_FROM_DEVICE,
+						       mdetails);
 
 	if (err)
 		goto out;
@@ -368,12 +416,11 @@ drm_pagemap_migrate_sys_to_dev(struct drm_pagemap_devmem *devmem,
 			       struct page *local_pages[],
 			       struct drm_pagemap_addr pagemap_addr[],
 			       unsigned long npages,
-			       const struct drm_pagemap_devmem_ops *ops,
-			       const struct drm_pagemap_migrate_details *mdetails)
+			       const struct drm_pagemap_devmem_ops *ops)
 {
-	int err = drm_pagemap_migrate_map_pages(devmem->dev, devmem->dpagemap,
-						pagemap_addr, sys_pfns, npages,
-						DMA_TO_DEVICE, mdetails);
+	int err = drm_pagemap_migrate_map_system_pages(devmem->dev,
+						       pagemap_addr, sys_pfns,
+						       npages, DMA_TO_DEVICE);
 
 	if (err)
 		goto out;
@@ -437,7 +484,7 @@ static int drm_pagemap_migrate_range(struct drm_pagemap_devmem *devmem,
 						     &pages[last->start],
 						     &pagemap_addr[last->start],
 						     cur->start - last->start,
-						     last->ops, mdetails);
+						     last->ops);
 
 out:
 	*last = *cur;
@@ -954,7 +1001,6 @@ EXPORT_SYMBOL(drm_pagemap_put);
 int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 {
 	const struct drm_pagemap_devmem_ops *ops = devmem_allocation->ops;
-	struct drm_pagemap_migrate_details mdetails = {};
 	unsigned long npages, mpages = 0;
 	struct page **pages;
 	unsigned long *src, *dst;
@@ -993,10 +1039,10 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 	if (err || !mpages)
 		goto err_finalize;
 
-	err = drm_pagemap_migrate_map_pages(devmem_allocation->dev,
-					    devmem_allocation->dpagemap, pagemap_addr,
-					    dst, npages, DMA_FROM_DEVICE,
-					    &mdetails);
+	err = drm_pagemap_migrate_map_system_pages(devmem_allocation->dev,
+						   pagemap_addr,
+						   dst, npages,
+						   DMA_FROM_DEVICE);
 	if (err)
 		goto err_finalize;
 
@@ -1057,7 +1103,6 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 		MIGRATE_VMA_SELECT_DEVICE_COHERENT,
 		.fault_page	= page,
 	};
-	struct drm_pagemap_migrate_details mdetails = {};
 	struct drm_pagemap_zdd *zdd;
 	const struct drm_pagemap_devmem_ops *ops;
 	struct device *dev = NULL;
@@ -1115,8 +1160,9 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 	if (err)
 		goto err_finalize;
 
-	err = drm_pagemap_migrate_map_pages(dev, zdd->dpagemap, pagemap_addr, migrate.dst, npages,
-					    DMA_FROM_DEVICE, &mdetails);
+	err = drm_pagemap_migrate_map_system_pages(dev, pagemap_addr,
+						   migrate.dst, npages,
+						   DMA_FROM_DEVICE);
 	if (err)
 		goto err_finalize;
 
-- 
2.34.1

[PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Matthew Brost]

The dma-map IOVA alloc, link, and sync APIs perform significantly better
than dma-map / dma-unmap, as they avoid costly IOMMU synchronizations.
This difference is especially noticeable when mapping a 2MB region in
4KB pages.

Use the IOVA alloc, link, and sync APIs for DRM pagemap, which create DMA
mappings between the CPU and GPU for copying data.

Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
v4:
 - Pack IOVA and drop dummy page (Jason)

 drivers/gpu/drm/drm_pagemap.c | 84 +++++++++++++++++++++++++++++------
 1 file changed, 70 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
index 29677b19bb69..52a196bc8459 100644
--- a/drivers/gpu/drm/drm_pagemap.c
+++ b/drivers/gpu/drm/drm_pagemap.c
@@ -280,6 +280,20 @@ drm_pagemap_migrate_map_device_pages(struct device *dev,
 	return 0;
 }
 
+/**
+ * struct drm_pagemap_iova_state - DRM pagemap IOVA state
+ *
+ * @dma_state: DMA IOVA state.
+ * @offset: Current offset in IOVA.
+ *
+ * This structure acts as an iterator for packing all IOVA addresses within a
+ * contiguous range.
+ */
+struct drm_pagemap_iova_state {
+	struct dma_iova_state dma_state;
+	unsigned long offset;
+};
+
 /**
  * drm_pagemap_migrate_map_system_pages() - Map system migration pages for GPU SVM migration
  * @dev: The device performing the migration.
@@ -287,6 +301,7 @@ drm_pagemap_migrate_map_device_pages(struct device *dev,
  * @migrate_pfn: Array of page frame numbers of system pages or peer pages to map.
  * @npages: Number of system pages or peer pages to map.
  * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ * @state: DMA IOVA state for mapping.
  *
  * This function maps pages of memory for migration usage in GPU SVM. It
  * iterates over each page frame number provided in @migrate_pfn, maps the
@@ -300,9 +315,11 @@ drm_pagemap_migrate_map_system_pages(struct device *dev,
 				     struct drm_pagemap_addr *pagemap_addr,
 				     unsigned long *migrate_pfn,
 				     unsigned long npages,
-				     enum dma_data_direction dir)
+				     enum dma_data_direction dir,
+				     struct drm_pagemap_iova_state *state)
 {
 	unsigned long i;
+	bool try_alloc = false;
 
 	for (i = 0; i < npages;) {
 		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
@@ -317,9 +334,31 @@ drm_pagemap_migrate_map_system_pages(struct device *dev,
 		folio = page_folio(page);
 		order = folio_order(folio);
 
-		dma_addr = dma_map_page(dev, page, 0, page_size(page), dir);
-		if (dma_mapping_error(dev, dma_addr))
-			return -EFAULT;
+		if (!try_alloc) {
+			dma_iova_try_alloc(dev, &state->dma_state,
+					   npages * PAGE_SIZE >=
+					   HPAGE_PMD_SIZE ?
+					   HPAGE_PMD_SIZE : 0,
+					   npages * PAGE_SIZE);
+			try_alloc = true;
+		}
+
+		if (dma_use_iova(&state->dma_state)) {
+			int err = dma_iova_link(dev, &state->dma_state,
+						page_to_phys(page),
+						state->offset, page_size(page),
+						dir, 0);
+			if (err)
+				return err;
+
+			dma_addr = state->dma_state.addr + state->offset;
+			state->offset += page_size(page);
+		} else {
+			dma_addr = dma_map_page(dev, page, 0, page_size(page),
+						dir);
+			if (dma_mapping_error(dev, dma_addr))
+				return -EFAULT;
+		}
 
 		pagemap_addr[i] =
 			drm_pagemap_addr_encode(dma_addr,
@@ -330,6 +369,9 @@ drm_pagemap_migrate_map_system_pages(struct device *dev,
 		i += NR_PAGES(order);
 	}
 
+	if (dma_use_iova(&state->dma_state))
+		return dma_iova_sync(dev, &state->dma_state, 0, state->offset);
+
 	return 0;
 }
 
@@ -341,6 +383,7 @@ drm_pagemap_migrate_map_system_pages(struct device *dev,
  * @pagemap_addr: Array of DMA information corresponding to mapped pages
  * @npages: Number of pages to unmap
  * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ * @state: DMA IOVA state for mapping.
  *
  * This function unmaps previously mapped pages of memory for GPU Shared Virtual
  * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
@@ -350,10 +393,17 @@ static void drm_pagemap_migrate_unmap_pages(struct device *dev,
 					    struct drm_pagemap_addr *pagemap_addr,
 					    unsigned long *migrate_pfn,
 					    unsigned long npages,
-					    enum dma_data_direction dir)
+					    enum dma_data_direction dir,
+					    struct drm_pagemap_iova_state *state)
 {
 	unsigned long i;
 
+	if (state && dma_use_iova(&state->dma_state)) {
+		dma_iova_unlink(dev, &state->dma_state, 0, state->offset, dir, 0);
+		dma_iova_free(dev, &state->dma_state);
+		return;
+	}
+
 	for (i = 0; i < npages;) {
 		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
 
@@ -406,7 +456,7 @@ drm_pagemap_migrate_remote_to_local(struct drm_pagemap_devmem *devmem,
 			       devmem->pre_migrate_fence);
 out:
 	drm_pagemap_migrate_unmap_pages(remote_device, pagemap_addr, local_pfns,
-					npages, DMA_FROM_DEVICE);
+					npages, DMA_FROM_DEVICE, NULL);
 	return err;
 }
 
@@ -416,11 +466,13 @@ drm_pagemap_migrate_sys_to_dev(struct drm_pagemap_devmem *devmem,
 			       struct page *local_pages[],
 			       struct drm_pagemap_addr pagemap_addr[],
 			       unsigned long npages,
-			       const struct drm_pagemap_devmem_ops *ops)
+			       const struct drm_pagemap_devmem_ops *ops,
+			       struct drm_pagemap_iova_state *state)
 {
 	int err = drm_pagemap_migrate_map_system_pages(devmem->dev,
 						       pagemap_addr, sys_pfns,
-						       npages, DMA_TO_DEVICE);
+						       npages, DMA_TO_DEVICE,
+						       state);
 
 	if (err)
 		goto out;
@@ -429,7 +481,7 @@ drm_pagemap_migrate_sys_to_dev(struct drm_pagemap_devmem *devmem,
 				  devmem->pre_migrate_fence);
 out:
 	drm_pagemap_migrate_unmap_pages(devmem->dev, pagemap_addr, sys_pfns, npages,
-					DMA_TO_DEVICE);
+					DMA_TO_DEVICE, state);
 	return err;
 }
 
@@ -457,6 +509,7 @@ static int drm_pagemap_migrate_range(struct drm_pagemap_devmem *devmem,
 				     const struct migrate_range_loc *cur,
 				     const struct drm_pagemap_migrate_details *mdetails)
 {
+	struct drm_pagemap_iova_state state = {};
 	int ret = 0;
 
 	if (cur->start == 0)
@@ -484,7 +537,7 @@ static int drm_pagemap_migrate_range(struct drm_pagemap_devmem *devmem,
 						     &pages[last->start],
 						     &pagemap_addr[last->start],
 						     cur->start - last->start,
-						     last->ops);
+						     last->ops, &state);
 
 out:
 	*last = *cur;
@@ -1001,6 +1054,7 @@ EXPORT_SYMBOL(drm_pagemap_put);
 int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 {
 	const struct drm_pagemap_devmem_ops *ops = devmem_allocation->ops;
+	struct drm_pagemap_iova_state state = {};
 	unsigned long npages, mpages = 0;
 	struct page **pages;
 	unsigned long *src, *dst;
@@ -1042,7 +1096,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 	err = drm_pagemap_migrate_map_system_pages(devmem_allocation->dev,
 						   pagemap_addr,
 						   dst, npages,
-						   DMA_FROM_DEVICE);
+						   DMA_FROM_DEVICE, &state);
 	if (err)
 		goto err_finalize;
 
@@ -1059,7 +1113,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
 	migrate_device_pages(src, dst, npages);
 	migrate_device_finalize(src, dst, npages);
 	drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, dst, npages,
-					DMA_FROM_DEVICE);
+					DMA_FROM_DEVICE, &state);
 
 err_free:
 	kvfree(buf);
@@ -1103,6 +1157,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 		MIGRATE_VMA_SELECT_DEVICE_COHERENT,
 		.fault_page	= page,
 	};
+	struct drm_pagemap_iova_state state = {};
 	struct drm_pagemap_zdd *zdd;
 	const struct drm_pagemap_devmem_ops *ops;
 	struct device *dev = NULL;
@@ -1162,7 +1217,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 
 	err = drm_pagemap_migrate_map_system_pages(dev, pagemap_addr,
 						   migrate.dst, npages,
-						   DMA_FROM_DEVICE);
+						   DMA_FROM_DEVICE, &state);
 	if (err)
 		goto err_finalize;
 
@@ -1180,7 +1235,8 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 	migrate_vma_finalize(&migrate);
 	if (dev)
 		drm_pagemap_migrate_unmap_pages(dev, pagemap_addr, migrate.dst,
-						npages, DMA_FROM_DEVICE);
+						npages, DMA_FROM_DEVICE,
+						&state);
 err_free:
 	kvfree(buf);
 err_out:
-- 
2.34.1

Re: [PATCH v4 2/4] drm/gpusvm: Use dma-map IOVA alloc, link, and sync API in GPU SVM

[Thomas Hellström]

On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> The dma-map IOVA alloc, link, and sync APIs perform significantly
> better
> than dma-map / dma-unmap, as they avoid costly IOMMU
> synchronizations.
> This difference is especially noticeable when mapping a 2MB region in
> 4KB pages.
> 
> Use the IOVA alloc, link, and sync APIs for GPU SVM, which create DMA
> mappings between the CPU and GPU.
> 
> Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> ---
> v3:
>  - Always link IOVA in mixed mappings
>  - Sync IOVA
> v4:
>  - Initialize IOVA state in get_pages
>  - Use pack IOVA linking (Jason)
>  - s/page_to_phys/hmm_pfn_to_phys (Leon)
> 
>  drivers/gpu/drm/drm_gpusvm.c | 55 ++++++++++++++++++++++++++++++----
> --
>  include/drm/drm_gpusvm.h     |  5 ++++
>  2 files changed, 52 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/gpu/drm/drm_gpusvm.c
> b/drivers/gpu/drm/drm_gpusvm.c
> index 4b8130a4ce95..800caaf0a783 100644
> --- a/drivers/gpu/drm/drm_gpusvm.c
> +++ b/drivers/gpu/drm/drm_gpusvm.c
> @@ -1139,11 +1139,19 @@ static void __drm_gpusvm_unmap_pages(struct
> drm_gpusvm *gpusvm,
>  		struct drm_gpusvm_pages_flags flags = {
>  			.__flags = svm_pages->flags.__flags,
>  		};
> +		bool use_iova = dma_use_iova(&svm_pages->state);
> +
> +		if (use_iova) {
> +			dma_iova_unlink(dev, &svm_pages->state, 0,
> +					svm_pages->state_offset,
> +					svm_pages->dma_addr[0].dir,
> 0);
> +			dma_iova_free(dev, &svm_pages->state);
> +		}
>  
>  		for (i = 0, j = 0; i < npages; j++) {
>  			struct drm_pagemap_addr *addr = &svm_pages-
> >dma_addr[j];
>  
> -			if (addr->proto == DRM_INTERCONNECT_SYSTEM)
> +			if (!use_iova && addr->proto ==
> DRM_INTERCONNECT_SYSTEM)
>  				dma_unmap_page(dev,
>  					       addr->addr,
>  					       PAGE_SIZE << addr-
> >order,
> @@ -1408,6 +1416,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> *gpusvm,
>  	struct drm_gpusvm_pages_flags flags;
>  	enum dma_data_direction dma_dir = ctx->read_only ?
> DMA_TO_DEVICE :
>  							  
> DMA_BIDIRECTIONAL;
> +	struct dma_iova_state *state = &svm_pages->state;
>  
>  retry:
>  	if (time_after(jiffies, timeout))
> @@ -1446,6 +1455,9 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> *gpusvm,
>  	if (err)
>  		goto err_free;
>  
> +	*state = (struct dma_iova_state){};
> +	svm_pages->state_offset = 0;
> +
>  map_pages:
>  	/*
>  	 * Perform all dma mappings under the notifier lock to not
> @@ -1539,13 +1551,33 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> *gpusvm,
>  				goto err_unmap;
>  			}
>  
> -			addr = dma_map_page(gpusvm->drm->dev,
> -					    page, 0,
> -					    PAGE_SIZE << order,
> -					    dma_dir);
> -			if (dma_mapping_error(gpusvm->drm->dev,
> addr)) {
> -				err = -EFAULT;
> -				goto err_unmap;
> +			if (!i)
> +				dma_iova_try_alloc(gpusvm->drm->dev,
> state,
> +						   npages *
> PAGE_SIZE >=
> +						   HPAGE_PMD_SIZE ?
> +						   HPAGE_PMD_SIZE :
> 0,

Doc says "callers that always do PAGE_SIZE aligned transfers can always
pass 0 here", so can be simplified?


> +						   npages *
> PAGE_SIZE);
> +
> +			if (dma_use_iova(state)) {
> +				err = dma_iova_link(gpusvm->drm-
> >dev, state,
> +						   
> hmm_pfn_to_phys(pfns[i]),
> +						    svm_pages-
> >state_offset,
> +						    PAGE_SIZE <<
> order,
> +						    dma_dir, 0);
> +				if (err)
> +					goto err_unmap;
> +
> +				addr = state->addr + svm_pages-
> >state_offset;
> +				svm_pages->state_offset += PAGE_SIZE
> << order;
> +			} else {
> +				addr = dma_map_page(gpusvm->drm-
> >dev,
> +						    page, 0,
> +						    PAGE_SIZE <<
> order,
> +						    dma_dir);
> +				if (dma_mapping_error(gpusvm->drm-
> >dev, addr)) {
> +					err = -EFAULT;
> +					goto err_unmap;
> +				}
>  			}
>  
>  			svm_pages->dma_addr[j] =
> drm_pagemap_addr_encode
> @@ -1557,6 +1589,13 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> *gpusvm,
>  		flags.has_dma_mapping = true;
>  	}
>  
> +	if (dma_use_iova(state)) {
> +		err = dma_iova_sync(gpusvm->drm->dev, state, 0,
> +				    svm_pages->state_offset);
> +		if (err)
> +			goto err_unmap;
> +	}
> +
>  	if (pagemap) {
>  		flags.has_devmem_pages = true;
>  		drm_pagemap_get(dpagemap);
> diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h
> index 2578ac92a8d4..cd94bb2ee6ee 100644
> --- a/include/drm/drm_gpusvm.h
> +++ b/include/drm/drm_gpusvm.h
> @@ -6,6 +6,7 @@
>  #ifndef __DRM_GPUSVM_H__
>  #define __DRM_GPUSVM_H__
>  
> +#include <linux/dma-mapping.h>
>  #include <linux/kref.h>
>  #include <linux/interval_tree.h>
>  #include <linux/mmu_notifier.h>
> @@ -136,6 +137,8 @@ struct drm_gpusvm_pages_flags {
>   * @dma_addr: Device address array
>   * @dpagemap: The struct drm_pagemap of the device pages we're dma-
> mapping.
>   *            Note this is assuming only one drm_pagemap per range
> is allowed.
> + * @state: DMA IOVA state for mapping.
> + * @state_offset: DMA IOVA offset for mapping.
>   * @notifier_seq: Notifier sequence number of the range's pages
>   * @flags: Flags for range
>   * @flags.migrate_devmem: Flag indicating whether the range can be
> migrated to device memory
> @@ -147,6 +150,8 @@ struct drm_gpusvm_pages_flags {
>  struct drm_gpusvm_pages {
>  	struct drm_pagemap_addr *dma_addr;
>  	struct drm_pagemap *dpagemap;
> +	struct dma_iova_state state;
> +	unsigned long state_offset;
>  	unsigned long notifier_seq;
>  	struct drm_gpusvm_pages_flags flags;
>  };

Otherwise LGTM. 
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Re: [PATCH v4 3/4] drm/pagemap: Split drm_pagemap_migrate_map_pages into device / system

[Thomas Hellström]

On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> Split drm_pagemap_migrate_map_pages into device / system helpers
> clearly
> seperating these operations. Will help with upcoming changes to split
> IOVA allocation steps.
> 
> Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> ---
>  drivers/gpu/drm/drm_pagemap.c | 146 ++++++++++++++++++++++----------
> --
>  1 file changed, 96 insertions(+), 50 deletions(-)
> 
> diff --git a/drivers/gpu/drm/drm_pagemap.c
> b/drivers/gpu/drm/drm_pagemap.c
> index fbd69f383457..29677b19bb69 100644
> --- a/drivers/gpu/drm/drm_pagemap.c
> +++ b/drivers/gpu/drm/drm_pagemap.c
> @@ -205,7 +205,7 @@ static void drm_pagemap_get_devmem_page(struct
> page *page,
>  }
>  
>  /**
> - * drm_pagemap_migrate_map_pages() - Map migration pages for GPU SVM
> migration
> + * drm_pagemap_migrate_map_device_pages() - Map device migration
> pages for GPU SVM migration
>   * @dev: The device performing the migration.
>   * @local_dpagemap: The drm_pagemap local to the migrating device.
>   * @pagemap_addr: Array to store DMA information corresponding to
> mapped pages.
> @@ -221,19 +221,22 @@ static void drm_pagemap_get_devmem_page(struct
> page *page,
>   *
>   * Returns: 0 on success, -EFAULT if an error occurs during mapping.
>   */
> -static int drm_pagemap_migrate_map_pages(struct device *dev,
> -					 struct drm_pagemap
> *local_dpagemap,
> -					 struct drm_pagemap_addr
> *pagemap_addr,
> -					 unsigned long *migrate_pfn,
> -					 unsigned long npages,
> -					 enum dma_data_direction
> dir,
> -					 const struct
> drm_pagemap_migrate_details *mdetails)
> +static int
> +drm_pagemap_migrate_map_device_pages(struct device *dev,
> +				     struct drm_pagemap
> *local_dpagemap,
> +				     struct drm_pagemap_addr
> *pagemap_addr,
> +				     unsigned long *migrate_pfn,
> +				     unsigned long npages,
> +				     enum dma_data_direction dir,
> +				     const struct
> drm_pagemap_migrate_details *mdetails)

We might want to call this device_private pages. Device coherent pages
are treated like system pages here, but I figure those are known to the
dma subsystem and can be handled by the map_system_pages callback.

>  {
>  	unsigned long num_peer_pages = 0, num_local_pages = 0, i;
>  
>  	for (i = 0; i < npages;) {
>  		struct page *page =
> migrate_pfn_to_page(migrate_pfn[i]);
> -		dma_addr_t dma_addr;
> +		struct drm_pagemap_zdd *zdd;
> +		struct drm_pagemap *dpagemap;
> +		struct drm_pagemap_addr addr;
>  		struct folio *folio;
>  		unsigned int order = 0;
>  
> @@ -243,36 +246,26 @@ static int drm_pagemap_migrate_map_pages(struct
> device *dev,
>  		folio = page_folio(page);
>  		order = folio_order(folio);
>  
> -		if (is_device_private_page(page)) {
> -			struct drm_pagemap_zdd *zdd =
> drm_pagemap_page_zone_device_data(page);
> -			struct drm_pagemap *dpagemap = zdd-
> >dpagemap;
> -			struct drm_pagemap_addr addr;
> -
> -			if (dpagemap == local_dpagemap) {
> -				if (!mdetails-
> >can_migrate_same_pagemap)
> -					goto next;
> +		WARN_ON_ONCE(!is_device_private_page(page));
>  
> -				num_local_pages += NR_PAGES(order);
> -			} else {
> -				num_peer_pages += NR_PAGES(order);
> -			}
> +		zdd = drm_pagemap_page_zone_device_data(page);
> +		dpagemap = zdd->dpagemap;
>  
> -			addr = dpagemap->ops->device_map(dpagemap,
> dev, page, order, dir);
> -			if (dma_mapping_error(dev, addr.addr))
> -				return -EFAULT;
> +		if (dpagemap == local_dpagemap) {
> +			if (!mdetails->can_migrate_same_pagemap)
> +				goto next;
>  
> -			pagemap_addr[i] = addr;
> +			num_local_pages += NR_PAGES(order);
>  		} else {
> -			dma_addr = dma_map_page(dev, page, 0,
> page_size(page), dir);
> -			if (dma_mapping_error(dev, dma_addr))
> -				return -EFAULT;
> -
> -			pagemap_addr[i] =
> -				drm_pagemap_addr_encode(dma_addr,
> -
> 							DRM_INTERCONNECT_SYSTEM,
> -							order, dir);
> +			num_peer_pages += NR_PAGES(order);
>  		}
>  
> +		addr = dpagemap->ops->device_map(dpagemap, dev,
> page, order, dir);
> +		if (dma_mapping_error(dev, addr.addr))
> +			return -EFAULT;
> +
> +		pagemap_addr[i] = addr;
> +
>  next:
>  		i += NR_PAGES(order);
>  	}
> @@ -287,6 +280,59 @@ static int drm_pagemap_migrate_map_pages(struct
> device *dev,
>  	return 0;
>  }
>  
> +/**
> + * drm_pagemap_migrate_map_system_pages() - Map system migration
> pages for GPU SVM migration
> + * @dev: The device performing the migration.
> + * @pagemap_addr: Array to store DMA information corresponding to
> mapped pages.
> + * @migrate_pfn: Array of page frame numbers of system pages or peer
> pages to map.

system pages or device coherent pages? "Peer" pages would typically be
device-private pages with the same owner.

> + * @npages: Number of system pages or peer pages to map.

Same here.

> + * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> + *
> + * This function maps pages of memory for migration usage in GPU
> SVM. It
> + * iterates over each page frame number provided in @migrate_pfn,
> maps the
> + * corresponding page, and stores the DMA address in the provided
> @dma_addr
> + * array.
> + *
> + * Returns: 0 on success, -EFAULT if an error occurs during mapping.
> + */
> +static int
> +drm_pagemap_migrate_map_system_pages(struct device *dev,
> +				     struct drm_pagemap_addr
> *pagemap_addr,
> +				     unsigned long *migrate_pfn,
> +				     unsigned long npages,
> +				     enum dma_data_direction dir)
> +{
> +	unsigned long i;
> +
> +	for (i = 0; i < npages;) {
> +		struct page *page =
> migrate_pfn_to_page(migrate_pfn[i]);
> +		dma_addr_t dma_addr;
> +		struct folio *folio;
> +		unsigned int order = 0;
> +
> +		if (!page)
> +			goto next;
> +
> +		WARN_ON_ONCE(is_device_private_page(page));
> +		folio = page_folio(page);
> +		order = folio_order(folio);
> +
> +		dma_addr = dma_map_page(dev, page, 0,
> page_size(page), dir);
> +		if (dma_mapping_error(dev, dma_addr))
> +			return -EFAULT;
> +
> +		pagemap_addr[i] =
> +			drm_pagemap_addr_encode(dma_addr,
> +						DRM_INTERCONNECT_SYS
> TEM,
> +						order, dir);
> +
> +next:
> +		i += NR_PAGES(order);
> +	}
> +
> +	return 0;
> +}
> +
>  /**
>   * drm_pagemap_migrate_unmap_pages() - Unmap pages previously mapped
> for GPU SVM migration
>   * @dev: The device for which the pages were mapped
> @@ -347,9 +393,11 @@ drm_pagemap_migrate_remote_to_local(struct
> drm_pagemap_devmem *devmem,
>  				    const struct
> drm_pagemap_migrate_details *mdetails)
>  
>  {
> -	int err = drm_pagemap_migrate_map_pages(remote_device,
> remote_dpagemap,
> -						pagemap_addr,
> local_pfns,
> -						npages,
> DMA_FROM_DEVICE, mdetails);
> +	int err =
> drm_pagemap_migrate_map_device_pages(remote_device,
> +						      
> remote_dpagemap,
> +						       pagemap_addr,
> local_pfns,
> +						       npages,
> DMA_FROM_DEVICE,
> +						       mdetails);
>  
>  	if (err)
>  		goto out;
> @@ -368,12 +416,11 @@ drm_pagemap_migrate_sys_to_dev(struct
> drm_pagemap_devmem *devmem,
>  			       struct page *local_pages[],
>  			       struct drm_pagemap_addr
> pagemap_addr[],
>  			       unsigned long npages,
> -			       const struct drm_pagemap_devmem_ops
> *ops,
> -			       const struct
> drm_pagemap_migrate_details *mdetails)
> +			       const struct drm_pagemap_devmem_ops
> *ops)
>  {
> -	int err = drm_pagemap_migrate_map_pages(devmem->dev, devmem-
> >dpagemap,
> -						pagemap_addr,
> sys_pfns, npages,
> -						DMA_TO_DEVICE,
> mdetails);
> +	int err = drm_pagemap_migrate_map_system_pages(devmem->dev,
> +						       pagemap_addr,
> sys_pfns,
> +						       npages,
> DMA_TO_DEVICE);


Unfortunately it's a bit more complicated than this. If the destination
gpu migrates, the range to migrate could be a mix of system pages,
device coherent pages and also device private pages, and previously
drm_pagemap_migrate_map_pages() took care of that and did the correct
thing on a per-page basis. 

You can exercise this by setting mdetails::source_peer_migrates to
false on xe. That typically "works" but might generate some errors in
the atomic multi-device tests AFAICT because reading from the BAR does
not flush the L2 caches on BMG. But should be sufficient to exercise
this path.

/Thomas

Re: [PATCH v4 2/4] drm/gpusvm: Use dma-map IOVA alloc, link, and sync API in GPU SVM

[Matthew Brost]

On Mon, Feb 09, 2026 at 10:44:43AM +0100, Thomas Hellström wrote:
> On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> > The dma-map IOVA alloc, link, and sync APIs perform significantly
> > better
> > than dma-map / dma-unmap, as they avoid costly IOMMU
> > synchronizations.
> > This difference is especially noticeable when mapping a 2MB region in
> > 4KB pages.
> > 
> > Use the IOVA alloc, link, and sync APIs for GPU SVM, which create DMA
> > mappings between the CPU and GPU.
> > 
> > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > ---
> > v3:
> >  - Always link IOVA in mixed mappings
> >  - Sync IOVA
> > v4:
> >  - Initialize IOVA state in get_pages
> >  - Use pack IOVA linking (Jason)
> >  - s/page_to_phys/hmm_pfn_to_phys (Leon)
> > 
> >  drivers/gpu/drm/drm_gpusvm.c | 55 ++++++++++++++++++++++++++++++----
> > --
> >  include/drm/drm_gpusvm.h     |  5 ++++
> >  2 files changed, 52 insertions(+), 8 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/drm_gpusvm.c
> > b/drivers/gpu/drm/drm_gpusvm.c
> > index 4b8130a4ce95..800caaf0a783 100644
> > --- a/drivers/gpu/drm/drm_gpusvm.c
> > +++ b/drivers/gpu/drm/drm_gpusvm.c
> > @@ -1139,11 +1139,19 @@ static void __drm_gpusvm_unmap_pages(struct
> > drm_gpusvm *gpusvm,
> >  		struct drm_gpusvm_pages_flags flags = {
> >  			.__flags = svm_pages->flags.__flags,
> >  		};
> > +		bool use_iova = dma_use_iova(&svm_pages->state);
> > +
> > +		if (use_iova) {
> > +			dma_iova_unlink(dev, &svm_pages->state, 0,
> > +					svm_pages->state_offset,
> > +					svm_pages->dma_addr[0].dir,
> > 0);
> > +			dma_iova_free(dev, &svm_pages->state);
> > +		}
> >  
> >  		for (i = 0, j = 0; i < npages; j++) {
> >  			struct drm_pagemap_addr *addr = &svm_pages-
> > >dma_addr[j];
> >  
> > -			if (addr->proto == DRM_INTERCONNECT_SYSTEM)
> > +			if (!use_iova && addr->proto ==
> > DRM_INTERCONNECT_SYSTEM)
> >  				dma_unmap_page(dev,
> >  					       addr->addr,
> >  					       PAGE_SIZE << addr-
> > >order,
> > @@ -1408,6 +1416,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> > *gpusvm,
> >  	struct drm_gpusvm_pages_flags flags;
> >  	enum dma_data_direction dma_dir = ctx->read_only ?
> > DMA_TO_DEVICE :
> >  							  
> > DMA_BIDIRECTIONAL;
> > +	struct dma_iova_state *state = &svm_pages->state;
> >  
> >  retry:
> >  	if (time_after(jiffies, timeout))
> > @@ -1446,6 +1455,9 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> > *gpusvm,
> >  	if (err)
> >  		goto err_free;
> >  
> > +	*state = (struct dma_iova_state){};
> > +	svm_pages->state_offset = 0;
> > +
> >  map_pages:
> >  	/*
> >  	 * Perform all dma mappings under the notifier lock to not
> > @@ -1539,13 +1551,33 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> > *gpusvm,
> >  				goto err_unmap;
> >  			}
> >  
> > -			addr = dma_map_page(gpusvm->drm->dev,
> > -					    page, 0,
> > -					    PAGE_SIZE << order,
> > -					    dma_dir);
> > -			if (dma_mapping_error(gpusvm->drm->dev,
> > addr)) {
> > -				err = -EFAULT;
> > -				goto err_unmap;
> > +			if (!i)
> > +				dma_iova_try_alloc(gpusvm->drm->dev,
> > state,
> > +						   npages *
> > PAGE_SIZE >=
> > +						   HPAGE_PMD_SIZE ?
> > +						   HPAGE_PMD_SIZE :
> > 0,
> 
> Doc says "callers that always do PAGE_SIZE aligned transfers can always
> pass 0 here", so can be simplified?
> 

 * Note: @phys is only used to calculate the IOVA alignment. Callers that always
 * do PAGE_SIZE aligned transfers can safely pass 0 here.

So 0 would be safe but possibly suboptimal. For mapping greater than or
equal to 2M, we'd like 2M MB alignment so large GPU pages can used too.
I think passing in '0' could result in odd alignment.

I am assuming other vendors have 2M GPU pages here too but that seems
like somewhat safe assumption...

Matt

> 
> > +						   npages *
> > PAGE_SIZE);
> > +
> > +			if (dma_use_iova(state)) {
> > +				err = dma_iova_link(gpusvm->drm-
> > >dev, state,
> > +						   
> > hmm_pfn_to_phys(pfns[i]),
> > +						    svm_pages-
> > >state_offset,
> > +						    PAGE_SIZE <<
> > order,
> > +						    dma_dir, 0);
> > +				if (err)
> > +					goto err_unmap;
> > +
> > +				addr = state->addr + svm_pages-
> > >state_offset;
> > +				svm_pages->state_offset += PAGE_SIZE
> > << order;
> > +			} else {
> > +				addr = dma_map_page(gpusvm->drm-
> > >dev,
> > +						    page, 0,
> > +						    PAGE_SIZE <<
> > order,
> > +						    dma_dir);
> > +				if (dma_mapping_error(gpusvm->drm-
> > >dev, addr)) {
> > +					err = -EFAULT;
> > +					goto err_unmap;
> > +				}
> >  			}
> >  
> >  			svm_pages->dma_addr[j] =
> > drm_pagemap_addr_encode
> > @@ -1557,6 +1589,13 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> > *gpusvm,
> >  		flags.has_dma_mapping = true;
> >  	}
> >  
> > +	if (dma_use_iova(state)) {
> > +		err = dma_iova_sync(gpusvm->drm->dev, state, 0,
> > +				    svm_pages->state_offset);
> > +		if (err)
> > +			goto err_unmap;
> > +	}
> > +
> >  	if (pagemap) {
> >  		flags.has_devmem_pages = true;
> >  		drm_pagemap_get(dpagemap);
> > diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h
> > index 2578ac92a8d4..cd94bb2ee6ee 100644
> > --- a/include/drm/drm_gpusvm.h
> > +++ b/include/drm/drm_gpusvm.h
> > @@ -6,6 +6,7 @@
> >  #ifndef __DRM_GPUSVM_H__
> >  #define __DRM_GPUSVM_H__
> >  
> > +#include <linux/dma-mapping.h>
> >  #include <linux/kref.h>
> >  #include <linux/interval_tree.h>
> >  #include <linux/mmu_notifier.h>
> > @@ -136,6 +137,8 @@ struct drm_gpusvm_pages_flags {
> >   * @dma_addr: Device address array
> >   * @dpagemap: The struct drm_pagemap of the device pages we're dma-
> > mapping.
> >   *            Note this is assuming only one drm_pagemap per range
> > is allowed.
> > + * @state: DMA IOVA state for mapping.
> > + * @state_offset: DMA IOVA offset for mapping.
> >   * @notifier_seq: Notifier sequence number of the range's pages
> >   * @flags: Flags for range
> >   * @flags.migrate_devmem: Flag indicating whether the range can be
> > migrated to device memory
> > @@ -147,6 +150,8 @@ struct drm_gpusvm_pages_flags {
> >  struct drm_gpusvm_pages {
> >  	struct drm_pagemap_addr *dma_addr;
> >  	struct drm_pagemap *dpagemap;
> > +	struct dma_iova_state state;
> > +	unsigned long state_offset;
> >  	unsigned long notifier_seq;
> >  	struct drm_gpusvm_pages_flags flags;
> >  };
> 
> Otherwise LGTM. 
> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Re: [PATCH v4 2/4] drm/gpusvm: Use dma-map IOVA alloc, link, and sync API in GPU SVM

[Thomas Hellström]

On Mon, 2026-02-09 at 08:13 -0800, Matthew Brost wrote:
> On Mon, Feb 09, 2026 at 10:44:43AM +0100, Thomas Hellström wrote:
> > On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> > > The dma-map IOVA alloc, link, and sync APIs perform significantly
> > > better
> > > than dma-map / dma-unmap, as they avoid costly IOMMU
> > > synchronizations.
> > > This difference is especially noticeable when mapping a 2MB
> > > region in
> > > 4KB pages.
> > > 
> > > Use the IOVA alloc, link, and sync APIs for GPU SVM, which create
> > > DMA
> > > mappings between the CPU and GPU.
> > > 
> > > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > > ---
> > > v3:
> > >  - Always link IOVA in mixed mappings
> > >  - Sync IOVA
> > > v4:
> > >  - Initialize IOVA state in get_pages
> > >  - Use pack IOVA linking (Jason)
> > >  - s/page_to_phys/hmm_pfn_to_phys (Leon)
> > > 
> > >  drivers/gpu/drm/drm_gpusvm.c | 55
> > > ++++++++++++++++++++++++++++++----
> > > --
> > >  include/drm/drm_gpusvm.h     |  5 ++++
> > >  2 files changed, 52 insertions(+), 8 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/drm_gpusvm.c
> > > b/drivers/gpu/drm/drm_gpusvm.c
> > > index 4b8130a4ce95..800caaf0a783 100644
> > > --- a/drivers/gpu/drm/drm_gpusvm.c
> > > +++ b/drivers/gpu/drm/drm_gpusvm.c
> > > @@ -1139,11 +1139,19 @@ static void
> > > __drm_gpusvm_unmap_pages(struct
> > > drm_gpusvm *gpusvm,
> > >  		struct drm_gpusvm_pages_flags flags = {
> > >  			.__flags = svm_pages->flags.__flags,
> > >  		};
> > > +		bool use_iova = dma_use_iova(&svm_pages->state);
> > > +
> > > +		if (use_iova) {
> > > +			dma_iova_unlink(dev, &svm_pages->state,
> > > 0,
> > > +					svm_pages->state_offset,
> > > +					svm_pages-
> > > >dma_addr[0].dir,
> > > 0);
> > > +			dma_iova_free(dev, &svm_pages->state);
> > > +		}
> > >  
> > >  		for (i = 0, j = 0; i < npages; j++) {
> > >  			struct drm_pagemap_addr *addr =
> > > &svm_pages-
> > > > dma_addr[j];
> > >  
> > > -			if (addr->proto ==
> > > DRM_INTERCONNECT_SYSTEM)
> > > +			if (!use_iova && addr->proto ==
> > > DRM_INTERCONNECT_SYSTEM)
> > >  				dma_unmap_page(dev,
> > >  					       addr->addr,
> > >  					       PAGE_SIZE <<
> > > addr-
> > > > order,
> > > @@ -1408,6 +1416,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> > > *gpusvm,
> > >  	struct drm_gpusvm_pages_flags flags;
> > >  	enum dma_data_direction dma_dir = ctx->read_only ?
> > > DMA_TO_DEVICE :
> > >  							  
> > > DMA_BIDIRECTIONAL;
> > > +	struct dma_iova_state *state = &svm_pages->state;
> > >  
> > >  retry:
> > >  	if (time_after(jiffies, timeout))
> > > @@ -1446,6 +1455,9 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> > > *gpusvm,
> > >  	if (err)
> > >  		goto err_free;
> > >  
> > > +	*state = (struct dma_iova_state){};
> > > +	svm_pages->state_offset = 0;
> > > +
> > >  map_pages:
> > >  	/*
> > >  	 * Perform all dma mappings under the notifier lock to
> > > not
> > > @@ -1539,13 +1551,33 @@ int drm_gpusvm_get_pages(struct
> > > drm_gpusvm
> > > *gpusvm,
> > >  				goto err_unmap;
> > >  			}
> > >  
> > > -			addr = dma_map_page(gpusvm->drm->dev,
> > > -					    page, 0,
> > > -					    PAGE_SIZE << order,
> > > -					    dma_dir);
> > > -			if (dma_mapping_error(gpusvm->drm->dev,
> > > addr)) {
> > > -				err = -EFAULT;
> > > -				goto err_unmap;
> > > +			if (!i)
> > > +				dma_iova_try_alloc(gpusvm->drm-
> > > >dev,
> > > state,
> > > +						   npages *
> > > PAGE_SIZE >=
> > > +						  
> > > HPAGE_PMD_SIZE ?
> > > +						  
> > > HPAGE_PMD_SIZE :
> > > 0,
> > 
> > Doc says "callers that always do PAGE_SIZE aligned transfers can
> > always
> > pass 0 here", so can be simplified?
> > 
> 
>  * Note: @phys is only used to calculate the IOVA alignment. Callers
> that always
>  * do PAGE_SIZE aligned transfers can safely pass 0 here.
> 
> So 0 would be safe but possibly suboptimal. For mapping greater than
> or
> equal to 2M, we'd like 2M MB alignment so large GPU pages can used
> too.
> I think passing in '0' could result in odd alignment.
> 
> I am assuming other vendors have 2M GPU pages here too but that seems
> like somewhat safe assumption...

Ah, I interpreted that as beyond PAGE_SIZE the function would behave
the same.

Agree that if we can get 2M alignment that's much better.

Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

> 
> Matt
> 
> > 
> > > +						   npages *
> > > PAGE_SIZE);
> > > +
> > > +			if (dma_use_iova(state)) {
> > > +				err = dma_iova_link(gpusvm->drm-
> > > > dev, state,
> > > +						   
> > > hmm_pfn_to_phys(pfns[i]),
> > > +						    svm_pages-
> > > > state_offset,
> > > +						    PAGE_SIZE <<
> > > order,
> > > +						    dma_dir, 0);
> > > +				if (err)
> > > +					goto err_unmap;
> > > +
> > > +				addr = state->addr + svm_pages-
> > > > state_offset;
> > > +				svm_pages->state_offset +=
> > > PAGE_SIZE
> > > << order;
> > > +			} else {
> > > +				addr = dma_map_page(gpusvm->drm-
> > > > dev,
> > > +						    page, 0,
> > > +						    PAGE_SIZE <<
> > > order,
> > > +						    dma_dir);
> > > +				if (dma_mapping_error(gpusvm-
> > > >drm-
> > > > dev, addr)) {
> > > +					err = -EFAULT;
> > > +					goto err_unmap;
> > > +				}
> > >  			}
> > >  
> > >  			svm_pages->dma_addr[j] =
> > > drm_pagemap_addr_encode
> > > @@ -1557,6 +1589,13 @@ int drm_gpusvm_get_pages(struct drm_gpusvm
> > > *gpusvm,
> > >  		flags.has_dma_mapping = true;
> > >  	}
> > >  
> > > +	if (dma_use_iova(state)) {
> > > +		err = dma_iova_sync(gpusvm->drm->dev, state, 0,
> > > +				    svm_pages->state_offset);
> > > +		if (err)
> > > +			goto err_unmap;
> > > +	}
> > > +
> > >  	if (pagemap) {
> > >  		flags.has_devmem_pages = true;
> > >  		drm_pagemap_get(dpagemap);
> > > diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h
> > > index 2578ac92a8d4..cd94bb2ee6ee 100644
> > > --- a/include/drm/drm_gpusvm.h
> > > +++ b/include/drm/drm_gpusvm.h
> > > @@ -6,6 +6,7 @@
> > >  #ifndef __DRM_GPUSVM_H__
> > >  #define __DRM_GPUSVM_H__
> > >  
> > > +#include <linux/dma-mapping.h>
> > >  #include <linux/kref.h>
> > >  #include <linux/interval_tree.h>
> > >  #include <linux/mmu_notifier.h>
> > > @@ -136,6 +137,8 @@ struct drm_gpusvm_pages_flags {
> > >   * @dma_addr: Device address array
> > >   * @dpagemap: The struct drm_pagemap of the device pages we're
> > > dma-
> > > mapping.
> > >   *            Note this is assuming only one drm_pagemap per
> > > range
> > > is allowed.
> > > + * @state: DMA IOVA state for mapping.
> > > + * @state_offset: DMA IOVA offset for mapping.
> > >   * @notifier_seq: Notifier sequence number of the range's pages
> > >   * @flags: Flags for range
> > >   * @flags.migrate_devmem: Flag indicating whether the range can
> > > be
> > > migrated to device memory
> > > @@ -147,6 +150,8 @@ struct drm_gpusvm_pages_flags {
> > >  struct drm_gpusvm_pages {
> > >  	struct drm_pagemap_addr *dma_addr;
> > >  	struct drm_pagemap *dpagemap;
> > > +	struct dma_iova_state state;
> > > +	unsigned long state_offset;
> > >  	unsigned long notifier_seq;
> > >  	struct drm_gpusvm_pages_flags flags;
> > >  };
> > 
> > Otherwise LGTM. 
> > Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Re: [PATCH v4 3/4] drm/pagemap: Split drm_pagemap_migrate_map_pages into device / system

[Matthew Brost]

On Mon, Feb 09, 2026 at 04:49:03PM +0100, Thomas Hellström wrote:
> On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> > Split drm_pagemap_migrate_map_pages into device / system helpers
> > clearly
> > seperating these operations. Will help with upcoming changes to split
> > IOVA allocation steps.
> > 
> > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > ---
> >  drivers/gpu/drm/drm_pagemap.c | 146 ++++++++++++++++++++++----------
> > --
> >  1 file changed, 96 insertions(+), 50 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/drm_pagemap.c
> > b/drivers/gpu/drm/drm_pagemap.c
> > index fbd69f383457..29677b19bb69 100644
> > --- a/drivers/gpu/drm/drm_pagemap.c
> > +++ b/drivers/gpu/drm/drm_pagemap.c
> > @@ -205,7 +205,7 @@ static void drm_pagemap_get_devmem_page(struct
> > page *page,
> >  }
> >  
> >  /**
> > - * drm_pagemap_migrate_map_pages() - Map migration pages for GPU SVM
> > migration
> > + * drm_pagemap_migrate_map_device_pages() - Map device migration
> > pages for GPU SVM migration
> >   * @dev: The device performing the migration.
> >   * @local_dpagemap: The drm_pagemap local to the migrating device.
> >   * @pagemap_addr: Array to store DMA information corresponding to
> > mapped pages.
> > @@ -221,19 +221,22 @@ static void drm_pagemap_get_devmem_page(struct
> > page *page,
> >   *
> >   * Returns: 0 on success, -EFAULT if an error occurs during mapping.
> >   */
> > -static int drm_pagemap_migrate_map_pages(struct device *dev,
> > -					 struct drm_pagemap
> > *local_dpagemap,
> > -					 struct drm_pagemap_addr
> > *pagemap_addr,
> > -					 unsigned long *migrate_pfn,
> > -					 unsigned long npages,
> > -					 enum dma_data_direction
> > dir,
> > -					 const struct
> > drm_pagemap_migrate_details *mdetails)
> > +static int
> > +drm_pagemap_migrate_map_device_pages(struct device *dev,
> > +				     struct drm_pagemap
> > *local_dpagemap,
> > +				     struct drm_pagemap_addr
> > *pagemap_addr,
> > +				     unsigned long *migrate_pfn,
> > +				     unsigned long npages,
> > +				     enum dma_data_direction dir,
> > +				     const struct
> > drm_pagemap_migrate_details *mdetails)
> 
> We might want to call this device_private pages. Device coherent pages
> are treated like system pages here, but I figure those are known to the
> dma subsystem and can be handled by the map_system_pages callback.
> 

Yes.

Eventually we will have figure out we'd want to handle Device coherent
pages with a high speed fabric though.

> >  {
> >  	unsigned long num_peer_pages = 0, num_local_pages = 0, i;
> >  
> >  	for (i = 0; i < npages;) {
> >  		struct page *page =
> > migrate_pfn_to_page(migrate_pfn[i]);
> > -		dma_addr_t dma_addr;
> > +		struct drm_pagemap_zdd *zdd;
> > +		struct drm_pagemap *dpagemap;
> > +		struct drm_pagemap_addr addr;
> >  		struct folio *folio;
> >  		unsigned int order = 0;
> >  
> > @@ -243,36 +246,26 @@ static int drm_pagemap_migrate_map_pages(struct
> > device *dev,
> >  		folio = page_folio(page);
> >  		order = folio_order(folio);
> >  
> > -		if (is_device_private_page(page)) {
> > -			struct drm_pagemap_zdd *zdd =
> > drm_pagemap_page_zone_device_data(page);
> > -			struct drm_pagemap *dpagemap = zdd-
> > >dpagemap;
> > -			struct drm_pagemap_addr addr;
> > -
> > -			if (dpagemap == local_dpagemap) {
> > -				if (!mdetails-
> > >can_migrate_same_pagemap)
> > -					goto next;
> > +		WARN_ON_ONCE(!is_device_private_page(page));
> >  
> > -				num_local_pages += NR_PAGES(order);
> > -			} else {
> > -				num_peer_pages += NR_PAGES(order);
> > -			}
> > +		zdd = drm_pagemap_page_zone_device_data(page);
> > +		dpagemap = zdd->dpagemap;
> >  
> > -			addr = dpagemap->ops->device_map(dpagemap,
> > dev, page, order, dir);
> > -			if (dma_mapping_error(dev, addr.addr))
> > -				return -EFAULT;
> > +		if (dpagemap == local_dpagemap) {
> > +			if (!mdetails->can_migrate_same_pagemap)
> > +				goto next;
> >  
> > -			pagemap_addr[i] = addr;
> > +			num_local_pages += NR_PAGES(order);
> >  		} else {
> > -			dma_addr = dma_map_page(dev, page, 0,
> > page_size(page), dir);
> > -			if (dma_mapping_error(dev, dma_addr))
> > -				return -EFAULT;
> > -
> > -			pagemap_addr[i] =
> > -				drm_pagemap_addr_encode(dma_addr,
> > -
> > 							DRM_INTERCONNECT_SYSTEM,
> > -							order, dir);
> > +			num_peer_pages += NR_PAGES(order);
> >  		}
> >  
> > +		addr = dpagemap->ops->device_map(dpagemap, dev,
> > page, order, dir);
> > +		if (dma_mapping_error(dev, addr.addr))
> > +			return -EFAULT;
> > +
> > +		pagemap_addr[i] = addr;
> > +
> >  next:
> >  		i += NR_PAGES(order);
> >  	}
> > @@ -287,6 +280,59 @@ static int drm_pagemap_migrate_map_pages(struct
> > device *dev,
> >  	return 0;
> >  }
> >  
> > +/**
> > + * drm_pagemap_migrate_map_system_pages() - Map system migration
> > pages for GPU SVM migration
> > + * @dev: The device performing the migration.
> > + * @pagemap_addr: Array to store DMA information corresponding to
> > mapped pages.
> > + * @migrate_pfn: Array of page frame numbers of system pages or peer
> > pages to map.
> 
> system pages or device coherent pages? "Peer" pages would typically be
> device-private pages with the same owner.
> 
> > + * @npages: Number of system pages or peer pages to map.
> 
> Same here.

Yes, copy paste error.

> 
> > + * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > + *
> > + * This function maps pages of memory for migration usage in GPU
> > SVM. It
> > + * iterates over each page frame number provided in @migrate_pfn,
> > maps the
> > + * corresponding page, and stores the DMA address in the provided
> > @dma_addr
> > + * array.
> > + *
> > + * Returns: 0 on success, -EFAULT if an error occurs during mapping.
> > + */
> > +static int
> > +drm_pagemap_migrate_map_system_pages(struct device *dev,
> > +				     struct drm_pagemap_addr
> > *pagemap_addr,
> > +				     unsigned long *migrate_pfn,
> > +				     unsigned long npages,
> > +				     enum dma_data_direction dir)
> > +{
> > +	unsigned long i;
> > +
> > +	for (i = 0; i < npages;) {
> > +		struct page *page =
> > migrate_pfn_to_page(migrate_pfn[i]);
> > +		dma_addr_t dma_addr;
> > +		struct folio *folio;
> > +		unsigned int order = 0;
> > +
> > +		if (!page)
> > +			goto next;
> > +
> > +		WARN_ON_ONCE(is_device_private_page(page));
> > +		folio = page_folio(page);
> > +		order = folio_order(folio);
> > +
> > +		dma_addr = dma_map_page(dev, page, 0,
> > page_size(page), dir);
> > +		if (dma_mapping_error(dev, dma_addr))
> > +			return -EFAULT;
> > +
> > +		pagemap_addr[i] =
> > +			drm_pagemap_addr_encode(dma_addr,
> > +						DRM_INTERCONNECT_SYS
> > TEM,
> > +						order, dir);
> > +
> > +next:
> > +		i += NR_PAGES(order);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> >  /**
> >   * drm_pagemap_migrate_unmap_pages() - Unmap pages previously mapped
> > for GPU SVM migration
> >   * @dev: The device for which the pages were mapped
> > @@ -347,9 +393,11 @@ drm_pagemap_migrate_remote_to_local(struct
> > drm_pagemap_devmem *devmem,
> >  				    const struct
> > drm_pagemap_migrate_details *mdetails)
> >  
> >  {
> > -	int err = drm_pagemap_migrate_map_pages(remote_device,
> > remote_dpagemap,
> > -						pagemap_addr,
> > local_pfns,
> > -						npages,
> > DMA_FROM_DEVICE, mdetails);
> > +	int err =
> > drm_pagemap_migrate_map_device_pages(remote_device,
> > +						      
> > remote_dpagemap,
> > +						       pagemap_addr,
> > local_pfns,
> > +						       npages,
> > DMA_FROM_DEVICE,
> > +						       mdetails);
> >  
> >  	if (err)
> >  		goto out;
> > @@ -368,12 +416,11 @@ drm_pagemap_migrate_sys_to_dev(struct
> > drm_pagemap_devmem *devmem,
> >  			       struct page *local_pages[],
> >  			       struct drm_pagemap_addr
> > pagemap_addr[],
> >  			       unsigned long npages,
> > -			       const struct drm_pagemap_devmem_ops
> > *ops,
> > -			       const struct
> > drm_pagemap_migrate_details *mdetails)
> > +			       const struct drm_pagemap_devmem_ops
> > *ops)
> >  {
> > -	int err = drm_pagemap_migrate_map_pages(devmem->dev, devmem-
> > >dpagemap,
> > -						pagemap_addr,
> > sys_pfns, npages,
> > -						DMA_TO_DEVICE,
> > mdetails);
> > +	int err = drm_pagemap_migrate_map_system_pages(devmem->dev,
> > +						       pagemap_addr,
> > sys_pfns,
> > +						       npages,
> > DMA_TO_DEVICE);
> 
> 
> Unfortunately it's a bit more complicated than this. If the destination
> gpu migrates, the range to migrate could be a mix of system pages,
> device coherent pages and also device private pages, and previously
> drm_pagemap_migrate_map_pages() took care of that and did the correct
> thing on a per-page basis. 
> 
> You can exercise this by setting mdetails::source_peer_migrates to
> false on xe. That typically "works" but might generate some errors in
> the atomic multi-device tests AFAICT because reading from the BAR does
> not flush the L2 caches on BMG. But should be sufficient to exercise
> this path.

Ah, yes I see I missed this - this patch isn't strickly required I just
didn't want drm_pagemap_migrate_map_pages to have massive cascading if
statements... I can remove for now if that is preferred or should be
just remove source_peer_migrates and assume a value of '1'.

I suggest the later because looking forward source_peer_migrates == 0
would bb difficult to support a high speed fabric, which requires a IOVA
(think UAL with virtual NAs at the target device), plus multiple
different devices being found in the migration pages. Also, with p2p,
isn't source_peer_migrates == '1' (write over p2p) faster than
source_peer_migrates == '0' (read over p2p)?

Matt

> 
> /Thomas

Re: [PATCH v4 3/4] drm/pagemap: Split drm_pagemap_migrate_map_pages into device / system

[Thomas Hellström]

On Mon, 2026-02-09 at 08:58 -0800, Matthew Brost wrote:
> On Mon, Feb 09, 2026 at 04:49:03PM +0100, Thomas Hellström wrote:
> > On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> > > Split drm_pagemap_migrate_map_pages into device / system helpers
> > > clearly
> > > seperating these operations. Will help with upcoming changes to
> > > split
> > > IOVA allocation steps.
> > > 
> > > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > > ---
> > >  drivers/gpu/drm/drm_pagemap.c | 146 ++++++++++++++++++++++------
> > > ----
> > > --
> > >  1 file changed, 96 insertions(+), 50 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/drm_pagemap.c
> > > b/drivers/gpu/drm/drm_pagemap.c
> > > index fbd69f383457..29677b19bb69 100644
> > > --- a/drivers/gpu/drm/drm_pagemap.c
> > > +++ b/drivers/gpu/drm/drm_pagemap.c
> > > @@ -205,7 +205,7 @@ static void
> > > drm_pagemap_get_devmem_page(struct
> > > page *page,
> > >  }
> > >  
> > >  /**
> > > - * drm_pagemap_migrate_map_pages() - Map migration pages for GPU
> > > SVM
> > > migration
> > > + * drm_pagemap_migrate_map_device_pages() - Map device migration
> > > pages for GPU SVM migration
> > >   * @dev: The device performing the migration.
> > >   * @local_dpagemap: The drm_pagemap local to the migrating
> > > device.
> > >   * @pagemap_addr: Array to store DMA information corresponding
> > > to
> > > mapped pages.
> > > @@ -221,19 +221,22 @@ static void
> > > drm_pagemap_get_devmem_page(struct
> > > page *page,
> > >   *
> > >   * Returns: 0 on success, -EFAULT if an error occurs during
> > > mapping.
> > >   */
> > > -static int drm_pagemap_migrate_map_pages(struct device *dev,
> > > -					 struct drm_pagemap
> > > *local_dpagemap,
> > > -					 struct drm_pagemap_addr
> > > *pagemap_addr,
> > > -					 unsigned long
> > > *migrate_pfn,
> > > -					 unsigned long npages,
> > > -					 enum dma_data_direction
> > > dir,
> > > -					 const struct
> > > drm_pagemap_migrate_details *mdetails)
> > > +static int
> > > +drm_pagemap_migrate_map_device_pages(struct device *dev,
> > > +				     struct drm_pagemap
> > > *local_dpagemap,
> > > +				     struct drm_pagemap_addr
> > > *pagemap_addr,
> > > +				     unsigned long *migrate_pfn,
> > > +				     unsigned long npages,
> > > +				     enum dma_data_direction
> > > dir,
> > > +				     const struct
> > > drm_pagemap_migrate_details *mdetails)
> > 
> > We might want to call this device_private pages. Device coherent
> > pages
> > are treated like system pages here, but I figure those are known to
> > the
> > dma subsystem and can be handled by the map_system_pages callback.
> > 
> 
> Yes.
> 
> Eventually we will have figure out we'd want to handle Device
> coherent
> pages with a high speed fabric though.
> 
> > >  {
> > >  	unsigned long num_peer_pages = 0, num_local_pages = 0,
> > > i;
> > >  
> > >  	for (i = 0; i < npages;) {
> > >  		struct page *page =
> > > migrate_pfn_to_page(migrate_pfn[i]);
> > > -		dma_addr_t dma_addr;
> > > +		struct drm_pagemap_zdd *zdd;
> > > +		struct drm_pagemap *dpagemap;
> > > +		struct drm_pagemap_addr addr;
> > >  		struct folio *folio;
> > >  		unsigned int order = 0;
> > >  
> > > @@ -243,36 +246,26 @@ static int
> > > drm_pagemap_migrate_map_pages(struct
> > > device *dev,
> > >  		folio = page_folio(page);
> > >  		order = folio_order(folio);
> > >  
> > > -		if (is_device_private_page(page)) {
> > > -			struct drm_pagemap_zdd *zdd =
> > > drm_pagemap_page_zone_device_data(page);
> > > -			struct drm_pagemap *dpagemap = zdd-
> > > > dpagemap;
> > > -			struct drm_pagemap_addr addr;
> > > -
> > > -			if (dpagemap == local_dpagemap) {
> > > -				if (!mdetails-
> > > > can_migrate_same_pagemap)
> > > -					goto next;
> > > +		WARN_ON_ONCE(!is_device_private_page(page));
> > >  
> > > -				num_local_pages +=
> > > NR_PAGES(order);
> > > -			} else {
> > > -				num_peer_pages +=
> > > NR_PAGES(order);
> > > -			}
> > > +		zdd = drm_pagemap_page_zone_device_data(page);
> > > +		dpagemap = zdd->dpagemap;
> > >  
> > > -			addr = dpagemap->ops-
> > > >device_map(dpagemap,
> > > dev, page, order, dir);
> > > -			if (dma_mapping_error(dev, addr.addr))
> > > -				return -EFAULT;
> > > +		if (dpagemap == local_dpagemap) {
> > > +			if (!mdetails->can_migrate_same_pagemap)
> > > +				goto next;
> > >  
> > > -			pagemap_addr[i] = addr;
> > > +			num_local_pages += NR_PAGES(order);
> > >  		} else {
> > > -			dma_addr = dma_map_page(dev, page, 0,
> > > page_size(page), dir);
> > > -			if (dma_mapping_error(dev, dma_addr))
> > > -				return -EFAULT;
> > > -
> > > -			pagemap_addr[i] =
> > > -
> > > 				drm_pagemap_addr_encode(dma_addr,
> > > -
> > > 							DRM_INTE
> > > RCONNECT_SYSTEM,
> > > -							order,
> > > dir);
> > > +			num_peer_pages += NR_PAGES(order);
> > >  		}
> > >  
> > > +		addr = dpagemap->ops->device_map(dpagemap, dev,
> > > page, order, dir);
> > > +		if (dma_mapping_error(dev, addr.addr))
> > > +			return -EFAULT;
> > > +
> > > +		pagemap_addr[i] = addr;
> > > +
> > >  next:
> > >  		i += NR_PAGES(order);
> > >  	}
> > > @@ -287,6 +280,59 @@ static int
> > > drm_pagemap_migrate_map_pages(struct
> > > device *dev,
> > >  	return 0;
> > >  }
> > >  
> > > +/**
> > > + * drm_pagemap_migrate_map_system_pages() - Map system migration
> > > pages for GPU SVM migration
> > > + * @dev: The device performing the migration.
> > > + * @pagemap_addr: Array to store DMA information corresponding
> > > to
> > > mapped pages.
> > > + * @migrate_pfn: Array of page frame numbers of system pages or
> > > peer
> > > pages to map.
> > 
> > system pages or device coherent pages? "Peer" pages would typically
> > be
> > device-private pages with the same owner.
> > 
> > > + * @npages: Number of system pages or peer pages to map.
> > 
> > Same here.
> 
> Yes, copy paste error.
> 
> > 
> > > + * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > > + *
> > > + * This function maps pages of memory for migration usage in GPU
> > > SVM. It
> > > + * iterates over each page frame number provided in
> > > @migrate_pfn,
> > > maps the
> > > + * corresponding page, and stores the DMA address in the
> > > provided
> > > @dma_addr
> > > + * array.
> > > + *
> > > + * Returns: 0 on success, -EFAULT if an error occurs during
> > > mapping.
> > > + */
> > > +static int
> > > +drm_pagemap_migrate_map_system_pages(struct device *dev,
> > > +				     struct drm_pagemap_addr
> > > *pagemap_addr,
> > > +				     unsigned long *migrate_pfn,
> > > +				     unsigned long npages,
> > > +				     enum dma_data_direction
> > > dir)
> > > +{
> > > +	unsigned long i;
> > > +
> > > +	for (i = 0; i < npages;) {
> > > +		struct page *page =
> > > migrate_pfn_to_page(migrate_pfn[i]);
> > > +		dma_addr_t dma_addr;
> > > +		struct folio *folio;
> > > +		unsigned int order = 0;
> > > +
> > > +		if (!page)
> > > +			goto next;
> > > +
> > > +		WARN_ON_ONCE(is_device_private_page(page));
> > > +		folio = page_folio(page);
> > > +		order = folio_order(folio);
> > > +
> > > +		dma_addr = dma_map_page(dev, page, 0,
> > > page_size(page), dir);
> > > +		if (dma_mapping_error(dev, dma_addr))
> > > +			return -EFAULT;
> > > +
> > > +		pagemap_addr[i] =
> > > +			drm_pagemap_addr_encode(dma_addr,
> > > +						DRM_INTERCONNECT
> > > _SYS
> > > TEM,
> > > +						order, dir);
> > > +
> > > +next:
> > > +		i += NR_PAGES(order);
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > >  /**
> > >   * drm_pagemap_migrate_unmap_pages() - Unmap pages previously
> > > mapped
> > > for GPU SVM migration
> > >   * @dev: The device for which the pages were mapped
> > > @@ -347,9 +393,11 @@ drm_pagemap_migrate_remote_to_local(struct
> > > drm_pagemap_devmem *devmem,
> > >  				    const struct
> > > drm_pagemap_migrate_details *mdetails)
> > >  
> > >  {
> > > -	int err = drm_pagemap_migrate_map_pages(remote_device,
> > > remote_dpagemap,
> > > -						pagemap_addr,
> > > local_pfns,
> > > -						npages,
> > > DMA_FROM_DEVICE, mdetails);
> > > +	int err =
> > > drm_pagemap_migrate_map_device_pages(remote_device,
> > > +						      
> > > remote_dpagemap,
> > > +						      
> > > pagemap_addr,
> > > local_pfns,
> > > +						       npages,
> > > DMA_FROM_DEVICE,
> > > +						      
> > > mdetails);
> > >  
> > >  	if (err)
> > >  		goto out;
> > > @@ -368,12 +416,11 @@ drm_pagemap_migrate_sys_to_dev(struct
> > > drm_pagemap_devmem *devmem,
> > >  			       struct page *local_pages[],
> > >  			       struct drm_pagemap_addr
> > > pagemap_addr[],
> > >  			       unsigned long npages,
> > > -			       const struct
> > > drm_pagemap_devmem_ops
> > > *ops,
> > > -			       const struct
> > > drm_pagemap_migrate_details *mdetails)
> > > +			       const struct
> > > drm_pagemap_devmem_ops
> > > *ops)
> > >  {
> > > -	int err = drm_pagemap_migrate_map_pages(devmem->dev,
> > > devmem-
> > > > dpagemap,
> > > -						pagemap_addr,
> > > sys_pfns, npages,
> > > -						DMA_TO_DEVICE,
> > > mdetails);
> > > +	int err = drm_pagemap_migrate_map_system_pages(devmem-
> > > >dev,
> > > +						      
> > > pagemap_addr,
> > > sys_pfns,
> > > +						       npages,
> > > DMA_TO_DEVICE);
> > 
> > 
> > Unfortunately it's a bit more complicated than this. If the
> > destination
> > gpu migrates, the range to migrate could be a mix of system pages,
> > device coherent pages and also device private pages, and previously
> > drm_pagemap_migrate_map_pages() took care of that and did the
> > correct
> > thing on a per-page basis. 
> > 
> > You can exercise this by setting mdetails::source_peer_migrates to
> > false on xe. That typically "works" but might generate some errors
> > in
> > the atomic multi-device tests AFAICT because reading from the BAR
> > does
> > not flush the L2 caches on BMG. But should be sufficient to
> > exercise
> > this path.
> 
> Ah, yes I see I missed this - this patch isn't strickly required I
> just
> didn't want drm_pagemap_migrate_map_pages to have massive cascading
> if
> statements... I can remove for now if that is preferred or should be
> just remove source_peer_migrates and assume a value of '1'.
> 
> I suggest the later because looking forward source_peer_migrates == 0
> would bb difficult to support a high speed fabric, which requires a
> IOVA
> (think UAL with virtual NAs at the target device), plus multiple
> different devices being found in the migration pages. Also, with p2p,
> isn't source_peer_migrates == '1' (write over p2p) faster than
> source_peer_migrates == '0' (read over p2p)?

With source_peer_migrates == 0 we have the drawbacks of missed cache
flushing, can't be used with CCS compression, high speed fabric and
also the speed over PCIe as you say, so I don't see xe using it in the
near-term.

So I agree. Let's just let's assume source_peer_migrates == 1 for now. 

Thanks,
Thomas 

> 
> Matt
> 
> > 
> > /Thomas

Re: [PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Thomas Hellström]

On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> The dma-map IOVA alloc, link, and sync APIs perform significantly
> better
> than dma-map / dma-unmap, as they avoid costly IOMMU
> synchronizations.
> This difference is especially noticeable when mapping a 2MB region in
> 4KB pages.
> 
> Use the IOVA alloc, link, and sync APIs for DRM pagemap, which create
> DMA
> mappings between the CPU and GPU for copying data.
> 
> Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> ---
> v4:
>  - Pack IOVA and drop dummy page (Jason)
> 
>  drivers/gpu/drm/drm_pagemap.c | 84 +++++++++++++++++++++++++++++----
> --
>  1 file changed, 70 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/gpu/drm/drm_pagemap.c
> b/drivers/gpu/drm/drm_pagemap.c
> index 29677b19bb69..52a196bc8459 100644
> --- a/drivers/gpu/drm/drm_pagemap.c
> +++ b/drivers/gpu/drm/drm_pagemap.c
> @@ -280,6 +280,20 @@ drm_pagemap_migrate_map_device_pages(struct
> device *dev,
>  	return 0;
>  }
>  
> +/**
> + * struct drm_pagemap_iova_state - DRM pagemap IOVA state
> + *

No newline 

> + * @dma_state: DMA IOVA state.
> + * @offset: Current offset in IOVA.
> + *
> + * This structure acts as an iterator for packing all IOVA addresses
> within a
> + * contiguous range.
> + */
> +struct drm_pagemap_iova_state {
> +	struct dma_iova_state dma_state;
> +	unsigned long offset;
> +};
> +
>  /**
>   * drm_pagemap_migrate_map_system_pages() - Map system migration
> pages for GPU SVM migration
>   * @dev: The device performing the migration.
> @@ -287,6 +301,7 @@ drm_pagemap_migrate_map_device_pages(struct
> device *dev,
>   * @migrate_pfn: Array of page frame numbers of system pages or peer
> pages to map.
>   * @npages: Number of system pages or peer pages to map.
>   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> + * @state: DMA IOVA state for mapping.
>   *
>   * This function maps pages of memory for migration usage in GPU
> SVM. It
>   * iterates over each page frame number provided in @migrate_pfn,
> maps the
> @@ -300,9 +315,11 @@ drm_pagemap_migrate_map_system_pages(struct
> device *dev,
>  				     struct drm_pagemap_addr
> *pagemap_addr,
>  				     unsigned long *migrate_pfn,
>  				     unsigned long npages,
> -				     enum dma_data_direction dir)
> +				     enum dma_data_direction dir,
> +				     struct drm_pagemap_iova_state
> *state)
>  {
>  	unsigned long i;
> +	bool try_alloc = false;
>  
>  	for (i = 0; i < npages;) {
>  		struct page *page =
> migrate_pfn_to_page(migrate_pfn[i]);
> @@ -317,9 +334,31 @@ drm_pagemap_migrate_map_system_pages(struct
> device *dev,
>  		folio = page_folio(page);
>  		order = folio_order(folio);
>  
> -		dma_addr = dma_map_page(dev, page, 0,
> page_size(page), dir);
> -		if (dma_mapping_error(dev, dma_addr))
> -			return -EFAULT;
> +		if (!try_alloc) {
> +			dma_iova_try_alloc(dev, &state->dma_state,
> +					   npages * PAGE_SIZE >=
> +					   HPAGE_PMD_SIZE ?
> +					   HPAGE_PMD_SIZE : 0,
> +					   npages * PAGE_SIZE);
> +			try_alloc = true;
> +		}

What happens if dma_iova_try_alloc() fails for all i < some value x and
then suddenly succeeds for i == x? While the below code looks correct,
I figure we'd allocate a too large IOVA region and possibly get the
alignment wrong?

Otherwise LGTM.


> +
> +		if (dma_use_iova(&state->dma_state)) {
> +			int err = dma_iova_link(dev, &state-
> >dma_state,
> +						page_to_phys(page),
> +						state->offset,
> page_size(page),
> +						dir, 0);
> +			if (err)
> +				return err;
> +
> +			dma_addr = state->dma_state.addr + state-
> >offset;
> +			state->offset += page_size(page);
> +		} else {
> +			dma_addr = dma_map_page(dev, page, 0,
> page_size(page),
> +						dir);
> +			if (dma_mapping_error(dev, dma_addr))
> +				return -EFAULT;
> +		}
>  
>  		pagemap_addr[i] =
>  			drm_pagemap_addr_encode(dma_addr,
> @@ -330,6 +369,9 @@ drm_pagemap_migrate_map_system_pages(struct
> device *dev,
>  		i += NR_PAGES(order);
>  	}
>  
> +	if (dma_use_iova(&state->dma_state))
> +		return dma_iova_sync(dev, &state->dma_state, 0,
> state->offset);
> +
>  	return 0;
>  }
>  
> @@ -341,6 +383,7 @@ drm_pagemap_migrate_map_system_pages(struct
> device *dev,
>   * @pagemap_addr: Array of DMA information corresponding to mapped
> pages
>   * @npages: Number of pages to unmap
>   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> + * @state: DMA IOVA state for mapping.
>   *
>   * This function unmaps previously mapped pages of memory for GPU
> Shared Virtual
>   * Memory (SVM). It iterates over each DMA address provided in
> @dma_addr, checks
> @@ -350,10 +393,17 @@ static void
> drm_pagemap_migrate_unmap_pages(struct device *dev,
>  					    struct drm_pagemap_addr
> *pagemap_addr,
>  					    unsigned long
> *migrate_pfn,
>  					    unsigned long npages,
> -					    enum dma_data_direction
> dir)
> +					    enum dma_data_direction
> dir,
> +					    struct
> drm_pagemap_iova_state *state)
>  {
>  	unsigned long i;
>  
> +	if (state && dma_use_iova(&state->dma_state)) {
> +		dma_iova_unlink(dev, &state->dma_state, 0, state-
> >offset, dir, 0);
> +		dma_iova_free(dev, &state->dma_state);
> +		return;
> +	}
> +
>  	for (i = 0; i < npages;) {
>  		struct page *page =
> migrate_pfn_to_page(migrate_pfn[i]);
>  
> @@ -406,7 +456,7 @@ drm_pagemap_migrate_remote_to_local(struct
> drm_pagemap_devmem *devmem,
>  			       devmem->pre_migrate_fence);
>  out:
>  	drm_pagemap_migrate_unmap_pages(remote_device, pagemap_addr,
> local_pfns,
> -					npages, DMA_FROM_DEVICE);
> +					npages, DMA_FROM_DEVICE,
> NULL);
>  	return err;
>  }
>  
> @@ -416,11 +466,13 @@ drm_pagemap_migrate_sys_to_dev(struct
> drm_pagemap_devmem *devmem,
>  			       struct page *local_pages[],
>  			       struct drm_pagemap_addr
> pagemap_addr[],
>  			       unsigned long npages,
> -			       const struct drm_pagemap_devmem_ops
> *ops)
> +			       const struct drm_pagemap_devmem_ops
> *ops,
> +			       struct drm_pagemap_iova_state *state)
>  {
>  	int err = drm_pagemap_migrate_map_system_pages(devmem->dev,
>  						       pagemap_addr,
> sys_pfns,
> -						       npages,
> DMA_TO_DEVICE);
> +						       npages,
> DMA_TO_DEVICE,
> +						       state);
>  
>  	if (err)
>  		goto out;
> @@ -429,7 +481,7 @@ drm_pagemap_migrate_sys_to_dev(struct
> drm_pagemap_devmem *devmem,
>  				  devmem->pre_migrate_fence);
>  out:
>  	drm_pagemap_migrate_unmap_pages(devmem->dev, pagemap_addr,
> sys_pfns, npages,
> -					DMA_TO_DEVICE);
> +					DMA_TO_DEVICE, state);
>  	return err;
>  }
>  
> @@ -457,6 +509,7 @@ static int drm_pagemap_migrate_range(struct
> drm_pagemap_devmem *devmem,
>  				     const struct migrate_range_loc
> *cur,
>  				     const struct
> drm_pagemap_migrate_details *mdetails)
>  {
> +	struct drm_pagemap_iova_state state = {};
>  	int ret = 0;
>  
>  	if (cur->start == 0)
> @@ -484,7 +537,7 @@ static int drm_pagemap_migrate_range(struct
> drm_pagemap_devmem *devmem,
>  						     &pages[last-
> >start],
>  						    
> &pagemap_addr[last->start],
>  						     cur->start -
> last->start,
> -						     last->ops);
> +						     last->ops,
> &state);
>  
>  out:
>  	*last = *cur;
> @@ -1001,6 +1054,7 @@ EXPORT_SYMBOL(drm_pagemap_put);
>  int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem
> *devmem_allocation)
>  {
>  	const struct drm_pagemap_devmem_ops *ops =
> devmem_allocation->ops;
> +	struct drm_pagemap_iova_state state = {};
>  	unsigned long npages, mpages = 0;
>  	struct page **pages;
>  	unsigned long *src, *dst;
> @@ -1042,7 +1096,7 @@ int drm_pagemap_evict_to_ram(struct
> drm_pagemap_devmem *devmem_allocation)
>  	err =
> drm_pagemap_migrate_map_system_pages(devmem_allocation->dev,
>  						   pagemap_addr,
>  						   dst, npages,
> -						   DMA_FROM_DEVICE);
> +						   DMA_FROM_DEVICE,
> &state);
>  	if (err)
>  		goto err_finalize;
>  
> @@ -1059,7 +1113,7 @@ int drm_pagemap_evict_to_ram(struct
> drm_pagemap_devmem *devmem_allocation)
>  	migrate_device_pages(src, dst, npages);
>  	migrate_device_finalize(src, dst, npages);
>  	drm_pagemap_migrate_unmap_pages(devmem_allocation->dev,
> pagemap_addr, dst, npages,
> -					DMA_FROM_DEVICE);
> +					DMA_FROM_DEVICE, &state);
>  
>  err_free:
>  	kvfree(buf);
> @@ -1103,6 +1157,7 @@ static int __drm_pagemap_migrate_to_ram(struct
> vm_area_struct *vas,
>  		MIGRATE_VMA_SELECT_DEVICE_COHERENT,
>  		.fault_page	= page,
>  	};
> +	struct drm_pagemap_iova_state state = {};
>  	struct drm_pagemap_zdd *zdd;
>  	const struct drm_pagemap_devmem_ops *ops;
>  	struct device *dev = NULL;
> @@ -1162,7 +1217,7 @@ static int __drm_pagemap_migrate_to_ram(struct
> vm_area_struct *vas,
>  
>  	err = drm_pagemap_migrate_map_system_pages(dev,
> pagemap_addr,
>  						   migrate.dst,
> npages,
> -						   DMA_FROM_DEVICE);
> +						   DMA_FROM_DEVICE,
> &state);
>  	if (err)
>  		goto err_finalize;
>  
> @@ -1180,7 +1235,8 @@ static int __drm_pagemap_migrate_to_ram(struct
> vm_area_struct *vas,
>  	migrate_vma_finalize(&migrate);
>  	if (dev)
>  		drm_pagemap_migrate_unmap_pages(dev, pagemap_addr,
> migrate.dst,
> -						npages,
> DMA_FROM_DEVICE);
> +						npages,
> DMA_FROM_DEVICE,
> +						&state);
>  err_free:
>  	kvfree(buf);
>  err_out:

Re: [PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Matthew Brost]

On Wed, Feb 11, 2026 at 12:34:12PM +0100, Thomas Hellström wrote:
> On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> > The dma-map IOVA alloc, link, and sync APIs perform significantly
> > better
> > than dma-map / dma-unmap, as they avoid costly IOMMU
> > synchronizations.
> > This difference is especially noticeable when mapping a 2MB region in
> > 4KB pages.
> > 
> > Use the IOVA alloc, link, and sync APIs for DRM pagemap, which create
> > DMA
> > mappings between the CPU and GPU for copying data.
> > 
> > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > ---
> > v4:
> >  - Pack IOVA and drop dummy page (Jason)
> > 
> >  drivers/gpu/drm/drm_pagemap.c | 84 +++++++++++++++++++++++++++++----
> > --
> >  1 file changed, 70 insertions(+), 14 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/drm_pagemap.c
> > b/drivers/gpu/drm/drm_pagemap.c
> > index 29677b19bb69..52a196bc8459 100644
> > --- a/drivers/gpu/drm/drm_pagemap.c
> > +++ b/drivers/gpu/drm/drm_pagemap.c
> > @@ -280,6 +280,20 @@ drm_pagemap_migrate_map_device_pages(struct
> > device *dev,
> >  	return 0;
> >  }
> >  
> > +/**
> > + * struct drm_pagemap_iova_state - DRM pagemap IOVA state
> > + *
> 
> No newline 
> 

+1

> > + * @dma_state: DMA IOVA state.
> > + * @offset: Current offset in IOVA.
> > + *
> > + * This structure acts as an iterator for packing all IOVA addresses
> > within a
> > + * contiguous range.
> > + */
> > +struct drm_pagemap_iova_state {
> > +	struct dma_iova_state dma_state;
> > +	unsigned long offset;
> > +};
> > +
> >  /**
> >   * drm_pagemap_migrate_map_system_pages() - Map system migration
> > pages for GPU SVM migration
> >   * @dev: The device performing the migration.
> > @@ -287,6 +301,7 @@ drm_pagemap_migrate_map_device_pages(struct
> > device *dev,
> >   * @migrate_pfn: Array of page frame numbers of system pages or peer
> > pages to map.
> >   * @npages: Number of system pages or peer pages to map.
> >   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > + * @state: DMA IOVA state for mapping.
> >   *
> >   * This function maps pages of memory for migration usage in GPU
> > SVM. It
> >   * iterates over each page frame number provided in @migrate_pfn,
> > maps the
> > @@ -300,9 +315,11 @@ drm_pagemap_migrate_map_system_pages(struct
> > device *dev,
> >  				     struct drm_pagemap_addr
> > *pagemap_addr,
> >  				     unsigned long *migrate_pfn,
> >  				     unsigned long npages,
> > -				     enum dma_data_direction dir)
> > +				     enum dma_data_direction dir,
> > +				     struct drm_pagemap_iova_state
> > *state)
> >  {
> >  	unsigned long i;
> > +	bool try_alloc = false;
> >  
> >  	for (i = 0; i < npages;) {
> >  		struct page *page =
> > migrate_pfn_to_page(migrate_pfn[i]);
> > @@ -317,9 +334,31 @@ drm_pagemap_migrate_map_system_pages(struct
> > device *dev,
> >  		folio = page_folio(page);
> >  		order = folio_order(folio);
> >  
> > -		dma_addr = dma_map_page(dev, page, 0,
> > page_size(page), dir);
> > -		if (dma_mapping_error(dev, dma_addr))
> > -			return -EFAULT;
> > +		if (!try_alloc) {
> > +			dma_iova_try_alloc(dev, &state->dma_state,
> > +					   npages * PAGE_SIZE >=
> > +					   HPAGE_PMD_SIZE ?
> > +					   HPAGE_PMD_SIZE : 0,
> > +					   npages * PAGE_SIZE);
> > +			try_alloc = true;
> > +		}
> 
> What happens if dma_iova_try_alloc() fails for all i < some value x and
> then suddenly succeeds for i == x? While the below code looks correct,

We only try to alloc on the first valid page - 'i' may be any value
based on the first page found  or we may never alloc if the number of
pages found == 0 (possible, hence why it is inside the loop). This step
is done at most once. If the allocation fails, we use the map_page path
for the remaining loop iterations.

> I figure we'd allocate a too large IOVA region and possibly get the
> alignment wrong?

The first and only IOVA allocation attempts an aligned allocation. What
can happen is only a subset of the IOVA is used for the copy but we pack
in the pages starting at IOVA[0] and end at IOVA[number valid pages - 1].

Matt

> 
> Otherwise LGTM.
> 
> 
> > +
> > +		if (dma_use_iova(&state->dma_state)) {
> > +			int err = dma_iova_link(dev, &state-
> > >dma_state,
> > +						page_to_phys(page),
> > +						state->offset,
> > page_size(page),
> > +						dir, 0);
> > +			if (err)
> > +				return err;
> > +
> > +			dma_addr = state->dma_state.addr + state-
> > >offset;
> > +			state->offset += page_size(page);
> > +		} else {
> > +			dma_addr = dma_map_page(dev, page, 0,
> > page_size(page),
> > +						dir);
> > +			if (dma_mapping_error(dev, dma_addr))
> > +				return -EFAULT;
> > +		}
> >  
> >  		pagemap_addr[i] =
> >  			drm_pagemap_addr_encode(dma_addr,
> > @@ -330,6 +369,9 @@ drm_pagemap_migrate_map_system_pages(struct
> > device *dev,
> >  		i += NR_PAGES(order);
> >  	}
> >  
> > +	if (dma_use_iova(&state->dma_state))
> > +		return dma_iova_sync(dev, &state->dma_state, 0,
> > state->offset);
> > +
> >  	return 0;
> >  }
> >  
> > @@ -341,6 +383,7 @@ drm_pagemap_migrate_map_system_pages(struct
> > device *dev,
> >   * @pagemap_addr: Array of DMA information corresponding to mapped
> > pages
> >   * @npages: Number of pages to unmap
> >   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > + * @state: DMA IOVA state for mapping.
> >   *
> >   * This function unmaps previously mapped pages of memory for GPU
> > Shared Virtual
> >   * Memory (SVM). It iterates over each DMA address provided in
> > @dma_addr, checks
> > @@ -350,10 +393,17 @@ static void
> > drm_pagemap_migrate_unmap_pages(struct device *dev,
> >  					    struct drm_pagemap_addr
> > *pagemap_addr,
> >  					    unsigned long
> > *migrate_pfn,
> >  					    unsigned long npages,
> > -					    enum dma_data_direction
> > dir)
> > +					    enum dma_data_direction
> > dir,
> > +					    struct
> > drm_pagemap_iova_state *state)
> >  {
> >  	unsigned long i;
> >  
> > +	if (state && dma_use_iova(&state->dma_state)) {
> > +		dma_iova_unlink(dev, &state->dma_state, 0, state-
> > >offset, dir, 0);
> > +		dma_iova_free(dev, &state->dma_state);
> > +		return;
> > +	}
> > +
> >  	for (i = 0; i < npages;) {
> >  		struct page *page =
> > migrate_pfn_to_page(migrate_pfn[i]);
> >  
> > @@ -406,7 +456,7 @@ drm_pagemap_migrate_remote_to_local(struct
> > drm_pagemap_devmem *devmem,
> >  			       devmem->pre_migrate_fence);
> >  out:
> >  	drm_pagemap_migrate_unmap_pages(remote_device, pagemap_addr,
> > local_pfns,
> > -					npages, DMA_FROM_DEVICE);
> > +					npages, DMA_FROM_DEVICE,
> > NULL);
> >  	return err;
> >  }
> >  
> > @@ -416,11 +466,13 @@ drm_pagemap_migrate_sys_to_dev(struct
> > drm_pagemap_devmem *devmem,
> >  			       struct page *local_pages[],
> >  			       struct drm_pagemap_addr
> > pagemap_addr[],
> >  			       unsigned long npages,
> > -			       const struct drm_pagemap_devmem_ops
> > *ops)
> > +			       const struct drm_pagemap_devmem_ops
> > *ops,
> > +			       struct drm_pagemap_iova_state *state)
> >  {
> >  	int err = drm_pagemap_migrate_map_system_pages(devmem->dev,
> >  						       pagemap_addr,
> > sys_pfns,
> > -						       npages,
> > DMA_TO_DEVICE);
> > +						       npages,
> > DMA_TO_DEVICE,
> > +						       state);
> >  
> >  	if (err)
> >  		goto out;
> > @@ -429,7 +481,7 @@ drm_pagemap_migrate_sys_to_dev(struct
> > drm_pagemap_devmem *devmem,
> >  				  devmem->pre_migrate_fence);
> >  out:
> >  	drm_pagemap_migrate_unmap_pages(devmem->dev, pagemap_addr,
> > sys_pfns, npages,
> > -					DMA_TO_DEVICE);
> > +					DMA_TO_DEVICE, state);
> >  	return err;
> >  }
> >  
> > @@ -457,6 +509,7 @@ static int drm_pagemap_migrate_range(struct
> > drm_pagemap_devmem *devmem,
> >  				     const struct migrate_range_loc
> > *cur,
> >  				     const struct
> > drm_pagemap_migrate_details *mdetails)
> >  {
> > +	struct drm_pagemap_iova_state state = {};
> >  	int ret = 0;
> >  
> >  	if (cur->start == 0)
> > @@ -484,7 +537,7 @@ static int drm_pagemap_migrate_range(struct
> > drm_pagemap_devmem *devmem,
> >  						     &pages[last-
> > >start],
> >  						    
> > &pagemap_addr[last->start],
> >  						     cur->start -
> > last->start,
> > -						     last->ops);
> > +						     last->ops,
> > &state);
> >  
> >  out:
> >  	*last = *cur;
> > @@ -1001,6 +1054,7 @@ EXPORT_SYMBOL(drm_pagemap_put);
> >  int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem
> > *devmem_allocation)
> >  {
> >  	const struct drm_pagemap_devmem_ops *ops =
> > devmem_allocation->ops;
> > +	struct drm_pagemap_iova_state state = {};
> >  	unsigned long npages, mpages = 0;
> >  	struct page **pages;
> >  	unsigned long *src, *dst;
> > @@ -1042,7 +1096,7 @@ int drm_pagemap_evict_to_ram(struct
> > drm_pagemap_devmem *devmem_allocation)
> >  	err =
> > drm_pagemap_migrate_map_system_pages(devmem_allocation->dev,
> >  						   pagemap_addr,
> >  						   dst, npages,
> > -						   DMA_FROM_DEVICE);
> > +						   DMA_FROM_DEVICE,
> > &state);
> >  	if (err)
> >  		goto err_finalize;
> >  
> > @@ -1059,7 +1113,7 @@ int drm_pagemap_evict_to_ram(struct
> > drm_pagemap_devmem *devmem_allocation)
> >  	migrate_device_pages(src, dst, npages);
> >  	migrate_device_finalize(src, dst, npages);
> >  	drm_pagemap_migrate_unmap_pages(devmem_allocation->dev,
> > pagemap_addr, dst, npages,
> > -					DMA_FROM_DEVICE);
> > +					DMA_FROM_DEVICE, &state);
> >  
> >  err_free:
> >  	kvfree(buf);
> > @@ -1103,6 +1157,7 @@ static int __drm_pagemap_migrate_to_ram(struct
> > vm_area_struct *vas,
> >  		MIGRATE_VMA_SELECT_DEVICE_COHERENT,
> >  		.fault_page	= page,
> >  	};
> > +	struct drm_pagemap_iova_state state = {};
> >  	struct drm_pagemap_zdd *zdd;
> >  	const struct drm_pagemap_devmem_ops *ops;
> >  	struct device *dev = NULL;
> > @@ -1162,7 +1217,7 @@ static int __drm_pagemap_migrate_to_ram(struct
> > vm_area_struct *vas,
> >  
> >  	err = drm_pagemap_migrate_map_system_pages(dev,
> > pagemap_addr,
> >  						   migrate.dst,
> > npages,
> > -						   DMA_FROM_DEVICE);
> > +						   DMA_FROM_DEVICE,
> > &state);
> >  	if (err)
> >  		goto err_finalize;
> >  
> > @@ -1180,7 +1235,8 @@ static int __drm_pagemap_migrate_to_ram(struct
> > vm_area_struct *vas,
> >  	migrate_vma_finalize(&migrate);
> >  	if (dev)
> >  		drm_pagemap_migrate_unmap_pages(dev, pagemap_addr,
> > migrate.dst,
> > -						npages,
> > DMA_FROM_DEVICE);
> > +						npages,
> > DMA_FROM_DEVICE,
> > +						&state);
> >  err_free:
> >  	kvfree(buf);
> >  err_out:

Re: [PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Thomas Hellström]

On Wed, 2026-02-11 at 07:37 -0800, Matthew Brost wrote:
> On Wed, Feb 11, 2026 at 12:34:12PM +0100, Thomas Hellström wrote:
> > On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> > > The dma-map IOVA alloc, link, and sync APIs perform significantly
> > > better
> > > than dma-map / dma-unmap, as they avoid costly IOMMU
> > > synchronizations.
> > > This difference is especially noticeable when mapping a 2MB
> > > region in
> > > 4KB pages.
> > > 
> > > Use the IOVA alloc, link, and sync APIs for DRM pagemap, which
> > > create
> > > DMA
> > > mappings between the CPU and GPU for copying data.
> > > 
> > > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > > ---
> > > v4:
> > >  - Pack IOVA and drop dummy page (Jason)
> > > 
> > >  drivers/gpu/drm/drm_pagemap.c | 84
> > > +++++++++++++++++++++++++++++----
> > > --
> > >  1 file changed, 70 insertions(+), 14 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/drm_pagemap.c
> > > b/drivers/gpu/drm/drm_pagemap.c
> > > index 29677b19bb69..52a196bc8459 100644
> > > --- a/drivers/gpu/drm/drm_pagemap.c
> > > +++ b/drivers/gpu/drm/drm_pagemap.c
> > > @@ -280,6 +280,20 @@ drm_pagemap_migrate_map_device_pages(struct
> > > device *dev,
> > >  	return 0;
> > >  }
> > >  
> > > +/**
> > > + * struct drm_pagemap_iova_state - DRM pagemap IOVA state
> > > + *
> > 
> > No newline 
> > 
> 
> +1
> 
> > > + * @dma_state: DMA IOVA state.
> > > + * @offset: Current offset in IOVA.
> > > + *
> > > + * This structure acts as an iterator for packing all IOVA
> > > addresses
> > > within a
> > > + * contiguous range.
> > > + */
> > > +struct drm_pagemap_iova_state {
> > > +	struct dma_iova_state dma_state;
> > > +	unsigned long offset;
> > > +};
> > > +
> > >  /**
> > >   * drm_pagemap_migrate_map_system_pages() - Map system migration
> > > pages for GPU SVM migration
> > >   * @dev: The device performing the migration.
> > > @@ -287,6 +301,7 @@ drm_pagemap_migrate_map_device_pages(struct
> > > device *dev,
> > >   * @migrate_pfn: Array of page frame numbers of system pages or
> > > peer
> > > pages to map.
> > >   * @npages: Number of system pages or peer pages to map.
> > >   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > > + * @state: DMA IOVA state for mapping.
> > >   *
> > >   * This function maps pages of memory for migration usage in GPU
> > > SVM. It
> > >   * iterates over each page frame number provided in
> > > @migrate_pfn,
> > > maps the
> > > @@ -300,9 +315,11 @@ drm_pagemap_migrate_map_system_pages(struct
> > > device *dev,
> > >  				     struct drm_pagemap_addr
> > > *pagemap_addr,
> > >  				     unsigned long *migrate_pfn,
> > >  				     unsigned long npages,
> > > -				     enum dma_data_direction
> > > dir)
> > > +				     enum dma_data_direction
> > > dir,
> > > +				     struct
> > > drm_pagemap_iova_state
> > > *state)
> > >  {
> > >  	unsigned long i;
> > > +	bool try_alloc = false;
> > >  
> > >  	for (i = 0; i < npages;) {
> > >  		struct page *page =
> > > migrate_pfn_to_page(migrate_pfn[i]);
> > > @@ -317,9 +334,31 @@ drm_pagemap_migrate_map_system_pages(struct
> > > device *dev,
> > >  		folio = page_folio(page);
> > >  		order = folio_order(folio);
> > >  
> > > -		dma_addr = dma_map_page(dev, page, 0,
> > > page_size(page), dir);
> > > -		if (dma_mapping_error(dev, dma_addr))
> > > -			return -EFAULT;
> > > +		if (!try_alloc) {
> > > +			dma_iova_try_alloc(dev, &state-
> > > >dma_state,
> > > +					   npages * PAGE_SIZE >=
> > > +					   HPAGE_PMD_SIZE ?
> > > +					   HPAGE_PMD_SIZE : 0,
> > > +					   npages * PAGE_SIZE);
> > > +			try_alloc = true;
> > > +		}
> > 
> > What happens if dma_iova_try_alloc() fails for all i < some value x
> > and
> > then suddenly succeeds for i == x? While the below code looks
> > correct,
> 
> We only try to alloc on the first valid page - 'i' may be any value
> based on the first page found  or we may never alloc if the number of
> pages found == 0 (possible, hence why it is inside the loop). This
> step
> is done at most once. If the allocation fails, we use the map_page
> path
> for the remaining loop iterations.
> 
> > I figure we'd allocate a too large IOVA region and possibly get the
> > alignment wrong?
> 
> The first and only IOVA allocation attempts an aligned allocation.
> What
> can happen is only a subset of the IOVA is used for the copy but we
> pack
> in the pages starting at IOVA[0] and end at IOVA[number valid pages -
> 1].
> 
> Matt

So to be a little nicer on the IOVA allocator we could use the below?

		dma_iova_try_alloc(dev, &state->dma_state,
					   (npages - i) * PAGE_SIZE >=
					   HPAGE_PMD_SIZE ?
					   HPAGE_PMD_SIZE : 0,
					   (npages - i) * PAGE_SIZE);

Thanks,
Thomas

> 
> > 
> > Otherwise LGTM.
> > 
> > 
> > > +
> > > +		if (dma_use_iova(&state->dma_state)) {
> > > +			int err = dma_iova_link(dev, &state-
> > > > dma_state,
> > > +						page_to_phys(pag
> > > e),
> > > +						state->offset,
> > > page_size(page),
> > > +						dir, 0);
> > > +			if (err)
> > > +				return err;
> > > +
> > > +			dma_addr = state->dma_state.addr +
> > > state-
> > > > offset;
> > > +			state->offset += page_size(page);
> > > +		} else {
> > > +			dma_addr = dma_map_page(dev, page, 0,
> > > page_size(page),
> > > +						dir);
> > > +			if (dma_mapping_error(dev, dma_addr))
> > > +				return -EFAULT;
> > > +		}
> > >  
> > >  		pagemap_addr[i] =
> > >  			drm_pagemap_addr_encode(dma_addr,
> > > @@ -330,6 +369,9 @@ drm_pagemap_migrate_map_system_pages(struct
> > > device *dev,
> > >  		i += NR_PAGES(order);
> > >  	}
> > >  
> > > +	if (dma_use_iova(&state->dma_state))
> > > +		return dma_iova_sync(dev, &state->dma_state, 0,
> > > state->offset);
> > > +
> > >  	return 0;
> > >  }
> > >  
> > > @@ -341,6 +383,7 @@ drm_pagemap_migrate_map_system_pages(struct
> > > device *dev,
> > >   * @pagemap_addr: Array of DMA information corresponding to
> > > mapped
> > > pages
> > >   * @npages: Number of pages to unmap
> > >   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > > + * @state: DMA IOVA state for mapping.
> > >   *
> > >   * This function unmaps previously mapped pages of memory for
> > > GPU
> > > Shared Virtual
> > >   * Memory (SVM). It iterates over each DMA address provided in
> > > @dma_addr, checks
> > > @@ -350,10 +393,17 @@ static void
> > > drm_pagemap_migrate_unmap_pages(struct device *dev,
> > >  					    struct
> > > drm_pagemap_addr
> > > *pagemap_addr,
> > >  					    unsigned long
> > > *migrate_pfn,
> > >  					    unsigned long
> > > npages,
> > > -					    enum
> > > dma_data_direction
> > > dir)
> > > +					    enum
> > > dma_data_direction
> > > dir,
> > > +					    struct
> > > drm_pagemap_iova_state *state)
> > >  {
> > >  	unsigned long i;
> > >  
> > > +	if (state && dma_use_iova(&state->dma_state)) {
> > > +		dma_iova_unlink(dev, &state->dma_state, 0,
> > > state-
> > > > offset, dir, 0);
> > > +		dma_iova_free(dev, &state->dma_state);
> > > +		return;
> > > +	}
> > > +
> > >  	for (i = 0; i < npages;) {
> > >  		struct page *page =
> > > migrate_pfn_to_page(migrate_pfn[i]);
> > >  
> > > @@ -406,7 +456,7 @@ drm_pagemap_migrate_remote_to_local(struct
> > > drm_pagemap_devmem *devmem,
> > >  			       devmem->pre_migrate_fence);
> > >  out:
> > >  	drm_pagemap_migrate_unmap_pages(remote_device,
> > > pagemap_addr,
> > > local_pfns,
> > > -					npages,
> > > DMA_FROM_DEVICE);
> > > +					npages, DMA_FROM_DEVICE,
> > > NULL);
> > >  	return err;
> > >  }
> > >  
> > > @@ -416,11 +466,13 @@ drm_pagemap_migrate_sys_to_dev(struct
> > > drm_pagemap_devmem *devmem,
> > >  			       struct page *local_pages[],
> > >  			       struct drm_pagemap_addr
> > > pagemap_addr[],
> > >  			       unsigned long npages,
> > > -			       const struct
> > > drm_pagemap_devmem_ops
> > > *ops)
> > > +			       const struct
> > > drm_pagemap_devmem_ops
> > > *ops,
> > > +			       struct drm_pagemap_iova_state
> > > *state)
> > >  {
> > >  	int err = drm_pagemap_migrate_map_system_pages(devmem-
> > > >dev,
> > >  						      
> > > pagemap_addr,
> > > sys_pfns,
> > > -						       npages,
> > > DMA_TO_DEVICE);
> > > +						       npages,
> > > DMA_TO_DEVICE,
> > > +						       state);
> > >  
> > >  	if (err)
> > >  		goto out;
> > > @@ -429,7 +481,7 @@ drm_pagemap_migrate_sys_to_dev(struct
> > > drm_pagemap_devmem *devmem,
> > >  				  devmem->pre_migrate_fence);
> > >  out:
> > >  	drm_pagemap_migrate_unmap_pages(devmem->dev,
> > > pagemap_addr,
> > > sys_pfns, npages,
> > > -					DMA_TO_DEVICE);
> > > +					DMA_TO_DEVICE, state);
> > >  	return err;
> > >  }
> > >  
> > > @@ -457,6 +509,7 @@ static int drm_pagemap_migrate_range(struct
> > > drm_pagemap_devmem *devmem,
> > >  				     const struct
> > > migrate_range_loc
> > > *cur,
> > >  				     const struct
> > > drm_pagemap_migrate_details *mdetails)
> > >  {
> > > +	struct drm_pagemap_iova_state state = {};
> > >  	int ret = 0;
> > >  
> > >  	if (cur->start == 0)
> > > @@ -484,7 +537,7 @@ static int drm_pagemap_migrate_range(struct
> > > drm_pagemap_devmem *devmem,
> > >  						    
> > > &pages[last-
> > > > start],
> > >  						    
> > > &pagemap_addr[last->start],
> > >  						     cur->start
> > > -
> > > last->start,
> > > -						     last->ops);
> > > +						     last->ops,
> > > &state);
> > >  
> > >  out:
> > >  	*last = *cur;
> > > @@ -1001,6 +1054,7 @@ EXPORT_SYMBOL(drm_pagemap_put);
> > >  int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem
> > > *devmem_allocation)
> > >  {
> > >  	const struct drm_pagemap_devmem_ops *ops =
> > > devmem_allocation->ops;
> > > +	struct drm_pagemap_iova_state state = {};
> > >  	unsigned long npages, mpages = 0;
> > >  	struct page **pages;
> > >  	unsigned long *src, *dst;
> > > @@ -1042,7 +1096,7 @@ int drm_pagemap_evict_to_ram(struct
> > > drm_pagemap_devmem *devmem_allocation)
> > >  	err =
> > > drm_pagemap_migrate_map_system_pages(devmem_allocation->dev,
> > >  						   pagemap_addr,
> > >  						   dst, npages,
> > > -						  
> > > DMA_FROM_DEVICE);
> > > +						  
> > > DMA_FROM_DEVICE,
> > > &state);
> > >  	if (err)
> > >  		goto err_finalize;
> > >  
> > > @@ -1059,7 +1113,7 @@ int drm_pagemap_evict_to_ram(struct
> > > drm_pagemap_devmem *devmem_allocation)
> > >  	migrate_device_pages(src, dst, npages);
> > >  	migrate_device_finalize(src, dst, npages);
> > >  	drm_pagemap_migrate_unmap_pages(devmem_allocation->dev,
> > > pagemap_addr, dst, npages,
> > > -					DMA_FROM_DEVICE);
> > > +					DMA_FROM_DEVICE,
> > > &state);
> > >  
> > >  err_free:
> > >  	kvfree(buf);
> > > @@ -1103,6 +1157,7 @@ static int
> > > __drm_pagemap_migrate_to_ram(struct
> > > vm_area_struct *vas,
> > >  		MIGRATE_VMA_SELECT_DEVICE_COHERENT,
> > >  		.fault_page	= page,
> > >  	};
> > > +	struct drm_pagemap_iova_state state = {};
> > >  	struct drm_pagemap_zdd *zdd;
> > >  	const struct drm_pagemap_devmem_ops *ops;
> > >  	struct device *dev = NULL;
> > > @@ -1162,7 +1217,7 @@ static int
> > > __drm_pagemap_migrate_to_ram(struct
> > > vm_area_struct *vas,
> > >  
> > >  	err = drm_pagemap_migrate_map_system_pages(dev,
> > > pagemap_addr,
> > >  						   migrate.dst,
> > > npages,
> > > -						  
> > > DMA_FROM_DEVICE);
> > > +						  
> > > DMA_FROM_DEVICE,
> > > &state);
> > >  	if (err)
> > >  		goto err_finalize;
> > >  
> > > @@ -1180,7 +1235,8 @@ static int
> > > __drm_pagemap_migrate_to_ram(struct
> > > vm_area_struct *vas,
> > >  	migrate_vma_finalize(&migrate);
> > >  	if (dev)
> > >  		drm_pagemap_migrate_unmap_pages(dev,
> > > pagemap_addr,
> > > migrate.dst,
> > > -						npages,
> > > DMA_FROM_DEVICE);
> > > +						npages,
> > > DMA_FROM_DEVICE,
> > > +						&state);
> > >  err_free:
> > >  	kvfree(buf);
> > >  err_out:

Re: [PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Matthew Brost]

On Wed, Feb 11, 2026 at 07:48:59PM +0100, Thomas Hellström wrote:
> On Wed, 2026-02-11 at 07:37 -0800, Matthew Brost wrote:
> > On Wed, Feb 11, 2026 at 12:34:12PM +0100, Thomas Hellström wrote:
> > > On Wed, 2026-02-04 at 20:19 -0800, Matthew Brost wrote:
> > > > The dma-map IOVA alloc, link, and sync APIs perform significantly
> > > > better
> > > > than dma-map / dma-unmap, as they avoid costly IOMMU
> > > > synchronizations.
> > > > This difference is especially noticeable when mapping a 2MB
> > > > region in
> > > > 4KB pages.
> > > > 
> > > > Use the IOVA alloc, link, and sync APIs for DRM pagemap, which
> > > > create
> > > > DMA
> > > > mappings between the CPU and GPU for copying data.
> > > > 
> > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > > > ---
> > > > v4:
> > > >  - Pack IOVA and drop dummy page (Jason)
> > > > 
> > > >  drivers/gpu/drm/drm_pagemap.c | 84
> > > > +++++++++++++++++++++++++++++----
> > > > --
> > > >  1 file changed, 70 insertions(+), 14 deletions(-)
> > > > 
> > > > diff --git a/drivers/gpu/drm/drm_pagemap.c
> > > > b/drivers/gpu/drm/drm_pagemap.c
> > > > index 29677b19bb69..52a196bc8459 100644
> > > > --- a/drivers/gpu/drm/drm_pagemap.c
> > > > +++ b/drivers/gpu/drm/drm_pagemap.c
> > > > @@ -280,6 +280,20 @@ drm_pagemap_migrate_map_device_pages(struct
> > > > device *dev,
> > > >  	return 0;
> > > >  }
> > > >  
> > > > +/**
> > > > + * struct drm_pagemap_iova_state - DRM pagemap IOVA state
> > > > + *
> > > 
> > > No newline 
> > > 
> > 
> > +1
> > 
> > > > + * @dma_state: DMA IOVA state.
> > > > + * @offset: Current offset in IOVA.
> > > > + *
> > > > + * This structure acts as an iterator for packing all IOVA
> > > > addresses
> > > > within a
> > > > + * contiguous range.
> > > > + */
> > > > +struct drm_pagemap_iova_state {
> > > > +	struct dma_iova_state dma_state;
> > > > +	unsigned long offset;
> > > > +};
> > > > +
> > > >  /**
> > > >   * drm_pagemap_migrate_map_system_pages() - Map system migration
> > > > pages for GPU SVM migration
> > > >   * @dev: The device performing the migration.
> > > > @@ -287,6 +301,7 @@ drm_pagemap_migrate_map_device_pages(struct
> > > > device *dev,
> > > >   * @migrate_pfn: Array of page frame numbers of system pages or
> > > > peer
> > > > pages to map.
> > > >   * @npages: Number of system pages or peer pages to map.
> > > >   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > > > + * @state: DMA IOVA state for mapping.
> > > >   *
> > > >   * This function maps pages of memory for migration usage in GPU
> > > > SVM. It
> > > >   * iterates over each page frame number provided in
> > > > @migrate_pfn,
> > > > maps the
> > > > @@ -300,9 +315,11 @@ drm_pagemap_migrate_map_system_pages(struct
> > > > device *dev,
> > > >  				     struct drm_pagemap_addr
> > > > *pagemap_addr,
> > > >  				     unsigned long *migrate_pfn,
> > > >  				     unsigned long npages,
> > > > -				     enum dma_data_direction
> > > > dir)
> > > > +				     enum dma_data_direction
> > > > dir,
> > > > +				     struct
> > > > drm_pagemap_iova_state
> > > > *state)
> > > >  {
> > > >  	unsigned long i;
> > > > +	bool try_alloc = false;
> > > >  
> > > >  	for (i = 0; i < npages;) {
> > > >  		struct page *page =
> > > > migrate_pfn_to_page(migrate_pfn[i]);
> > > > @@ -317,9 +334,31 @@ drm_pagemap_migrate_map_system_pages(struct
> > > > device *dev,
> > > >  		folio = page_folio(page);
> > > >  		order = folio_order(folio);
> > > >  
> > > > -		dma_addr = dma_map_page(dev, page, 0,
> > > > page_size(page), dir);
> > > > -		if (dma_mapping_error(dev, dma_addr))
> > > > -			return -EFAULT;
> > > > +		if (!try_alloc) {
> > > > +			dma_iova_try_alloc(dev, &state-
> > > > >dma_state,
> > > > +					   npages * PAGE_SIZE >=
> > > > +					   HPAGE_PMD_SIZE ?
> > > > +					   HPAGE_PMD_SIZE : 0,
> > > > +					   npages * PAGE_SIZE);
> > > > +			try_alloc = true;
> > > > +		}
> > > 
> > > What happens if dma_iova_try_alloc() fails for all i < some value x
> > > and
> > > then suddenly succeeds for i == x? While the below code looks
> > > correct,
> > 
> > We only try to alloc on the first valid page - 'i' may be any value
> > based on the first page found  or we may never alloc if the number of
> > pages found == 0 (possible, hence why it is inside the loop). This
> > step
> > is done at most once. If the allocation fails, we use the map_page
> > path
> > for the remaining loop iterations.
> > 
> > > I figure we'd allocate a too large IOVA region and possibly get the
> > > alignment wrong?
> > 
> > The first and only IOVA allocation attempts an aligned allocation.
> > What
> > can happen is only a subset of the IOVA is used for the copy but we
> > pack
> > in the pages starting at IOVA[0] and end at IOVA[number valid pages -
> > 1].
> > 
> > Matt
> 
> So to be a little nicer on the IOVA allocator we could use the below?
> 
> 		dma_iova_try_alloc(dev, &state->dma_state,
> 					   (npages - i) * PAGE_SIZE >=
> 					   HPAGE_PMD_SIZE ?
> 					   HPAGE_PMD_SIZE : 0,
> 					   (npages - i) * PAGE_SIZE);
> 

Yes, we can do that. No reason to force alignment if our copy code isn't
going to try to use 2M GPU pages.

Matt 

> Thanks,
> Thomas
> 
> > 
> > > 
> > > Otherwise LGTM.
> > > 
> > > 
> > > > +
> > > > +		if (dma_use_iova(&state->dma_state)) {
> > > > +			int err = dma_iova_link(dev, &state-
> > > > > dma_state,
> > > > +						page_to_phys(pag
> > > > e),
> > > > +						state->offset,
> > > > page_size(page),
> > > > +						dir, 0);
> > > > +			if (err)
> > > > +				return err;
> > > > +
> > > > +			dma_addr = state->dma_state.addr +
> > > > state-
> > > > > offset;
> > > > +			state->offset += page_size(page);
> > > > +		} else {
> > > > +			dma_addr = dma_map_page(dev, page, 0,
> > > > page_size(page),
> > > > +						dir);
> > > > +			if (dma_mapping_error(dev, dma_addr))
> > > > +				return -EFAULT;
> > > > +		}
> > > >  
> > > >  		pagemap_addr[i] =
> > > >  			drm_pagemap_addr_encode(dma_addr,
> > > > @@ -330,6 +369,9 @@ drm_pagemap_migrate_map_system_pages(struct
> > > > device *dev,
> > > >  		i += NR_PAGES(order);
> > > >  	}
> > > >  
> > > > +	if (dma_use_iova(&state->dma_state))
> > > > +		return dma_iova_sync(dev, &state->dma_state, 0,
> > > > state->offset);
> > > > +
> > > >  	return 0;
> > > >  }
> > > >  
> > > > @@ -341,6 +383,7 @@ drm_pagemap_migrate_map_system_pages(struct
> > > > device *dev,
> > > >   * @pagemap_addr: Array of DMA information corresponding to
> > > > mapped
> > > > pages
> > > >   * @npages: Number of pages to unmap
> > > >   * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
> > > > + * @state: DMA IOVA state for mapping.
> > > >   *
> > > >   * This function unmaps previously mapped pages of memory for
> > > > GPU
> > > > Shared Virtual
> > > >   * Memory (SVM). It iterates over each DMA address provided in
> > > > @dma_addr, checks
> > > > @@ -350,10 +393,17 @@ static void
> > > > drm_pagemap_migrate_unmap_pages(struct device *dev,
> > > >  					    struct
> > > > drm_pagemap_addr
> > > > *pagemap_addr,
> > > >  					    unsigned long
> > > > *migrate_pfn,
> > > >  					    unsigned long
> > > > npages,
> > > > -					    enum
> > > > dma_data_direction
> > > > dir)
> > > > +					    enum
> > > > dma_data_direction
> > > > dir,
> > > > +					    struct
> > > > drm_pagemap_iova_state *state)
> > > >  {
> > > >  	unsigned long i;
> > > >  
> > > > +	if (state && dma_use_iova(&state->dma_state)) {
> > > > +		dma_iova_unlink(dev, &state->dma_state, 0,
> > > > state-
> > > > > offset, dir, 0);
> > > > +		dma_iova_free(dev, &state->dma_state);
> > > > +		return;
> > > > +	}
> > > > +
> > > >  	for (i = 0; i < npages;) {
> > > >  		struct page *page =
> > > > migrate_pfn_to_page(migrate_pfn[i]);
> > > >  
> > > > @@ -406,7 +456,7 @@ drm_pagemap_migrate_remote_to_local(struct
> > > > drm_pagemap_devmem *devmem,
> > > >  			       devmem->pre_migrate_fence);
> > > >  out:
> > > >  	drm_pagemap_migrate_unmap_pages(remote_device,
> > > > pagemap_addr,
> > > > local_pfns,
> > > > -					npages,
> > > > DMA_FROM_DEVICE);
> > > > +					npages, DMA_FROM_DEVICE,
> > > > NULL);
> > > >  	return err;
> > > >  }
> > > >  
> > > > @@ -416,11 +466,13 @@ drm_pagemap_migrate_sys_to_dev(struct
> > > > drm_pagemap_devmem *devmem,
> > > >  			       struct page *local_pages[],
> > > >  			       struct drm_pagemap_addr
> > > > pagemap_addr[],
> > > >  			       unsigned long npages,
> > > > -			       const struct
> > > > drm_pagemap_devmem_ops
> > > > *ops)
> > > > +			       const struct
> > > > drm_pagemap_devmem_ops
> > > > *ops,
> > > > +			       struct drm_pagemap_iova_state
> > > > *state)
> > > >  {
> > > >  	int err = drm_pagemap_migrate_map_system_pages(devmem-
> > > > >dev,
> > > >  						      
> > > > pagemap_addr,
> > > > sys_pfns,
> > > > -						       npages,
> > > > DMA_TO_DEVICE);
> > > > +						       npages,
> > > > DMA_TO_DEVICE,
> > > > +						       state);
> > > >  
> > > >  	if (err)
> > > >  		goto out;
> > > > @@ -429,7 +481,7 @@ drm_pagemap_migrate_sys_to_dev(struct
> > > > drm_pagemap_devmem *devmem,
> > > >  				  devmem->pre_migrate_fence);
> > > >  out:
> > > >  	drm_pagemap_migrate_unmap_pages(devmem->dev,
> > > > pagemap_addr,
> > > > sys_pfns, npages,
> > > > -					DMA_TO_DEVICE);
> > > > +					DMA_TO_DEVICE, state);
> > > >  	return err;
> > > >  }
> > > >  
> > > > @@ -457,6 +509,7 @@ static int drm_pagemap_migrate_range(struct
> > > > drm_pagemap_devmem *devmem,
> > > >  				     const struct
> > > > migrate_range_loc
> > > > *cur,
> > > >  				     const struct
> > > > drm_pagemap_migrate_details *mdetails)
> > > >  {
> > > > +	struct drm_pagemap_iova_state state = {};
> > > >  	int ret = 0;
> > > >  
> > > >  	if (cur->start == 0)
> > > > @@ -484,7 +537,7 @@ static int drm_pagemap_migrate_range(struct
> > > > drm_pagemap_devmem *devmem,
> > > >  						    
> > > > &pages[last-
> > > > > start],
> > > >  						    
> > > > &pagemap_addr[last->start],
> > > >  						     cur->start
> > > > -
> > > > last->start,
> > > > -						     last->ops);
> > > > +						     last->ops,
> > > > &state);
> > > >  
> > > >  out:
> > > >  	*last = *cur;
> > > > @@ -1001,6 +1054,7 @@ EXPORT_SYMBOL(drm_pagemap_put);
> > > >  int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem
> > > > *devmem_allocation)
> > > >  {
> > > >  	const struct drm_pagemap_devmem_ops *ops =
> > > > devmem_allocation->ops;
> > > > +	struct drm_pagemap_iova_state state = {};
> > > >  	unsigned long npages, mpages = 0;
> > > >  	struct page **pages;
> > > >  	unsigned long *src, *dst;
> > > > @@ -1042,7 +1096,7 @@ int drm_pagemap_evict_to_ram(struct
> > > > drm_pagemap_devmem *devmem_allocation)
> > > >  	err =
> > > > drm_pagemap_migrate_map_system_pages(devmem_allocation->dev,
> > > >  						   pagemap_addr,
> > > >  						   dst, npages,
> > > > -						  
> > > > DMA_FROM_DEVICE);
> > > > +						  
> > > > DMA_FROM_DEVICE,
> > > > &state);
> > > >  	if (err)
> > > >  		goto err_finalize;
> > > >  
> > > > @@ -1059,7 +1113,7 @@ int drm_pagemap_evict_to_ram(struct
> > > > drm_pagemap_devmem *devmem_allocation)
> > > >  	migrate_device_pages(src, dst, npages);
> > > >  	migrate_device_finalize(src, dst, npages);
> > > >  	drm_pagemap_migrate_unmap_pages(devmem_allocation->dev,
> > > > pagemap_addr, dst, npages,
> > > > -					DMA_FROM_DEVICE);
> > > > +					DMA_FROM_DEVICE,
> > > > &state);
> > > >  
> > > >  err_free:
> > > >  	kvfree(buf);
> > > > @@ -1103,6 +1157,7 @@ static int
> > > > __drm_pagemap_migrate_to_ram(struct
> > > > vm_area_struct *vas,
> > > >  		MIGRATE_VMA_SELECT_DEVICE_COHERENT,
> > > >  		.fault_page	= page,
> > > >  	};
> > > > +	struct drm_pagemap_iova_state state = {};
> > > >  	struct drm_pagemap_zdd *zdd;
> > > >  	const struct drm_pagemap_devmem_ops *ops;
> > > >  	struct device *dev = NULL;
> > > > @@ -1162,7 +1217,7 @@ static int
> > > > __drm_pagemap_migrate_to_ram(struct
> > > > vm_area_struct *vas,
> > > >  
> > > >  	err = drm_pagemap_migrate_map_system_pages(dev,
> > > > pagemap_addr,
> > > >  						   migrate.dst,
> > > > npages,
> > > > -						  
> > > > DMA_FROM_DEVICE);
> > > > +						  
> > > > DMA_FROM_DEVICE,
> > > > &state);
> > > >  	if (err)
> > > >  		goto err_finalize;
> > > >  
> > > > @@ -1180,7 +1235,8 @@ static int
> > > > __drm_pagemap_migrate_to_ram(struct
> > > > vm_area_struct *vas,
> > > >  	migrate_vma_finalize(&migrate);
> > > >  	if (dev)
> > > >  		drm_pagemap_migrate_unmap_pages(dev,
> > > > pagemap_addr,
> > > > migrate.dst,
> > > > -						npages,
> > > > DMA_FROM_DEVICE);
> > > > +						npages,
> > > > DMA_FROM_DEVICE,
> > > > +						&state);
> > > >  err_free:
> > > >  	kvfree(buf);
> > > >  err_out:

Re: [PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Jason Gunthorpe]

On Wed, Feb 11, 2026 at 10:51:32AM -0800, Matthew Brost wrote:
> > So to be a little nicer on the IOVA allocator we could use the below?
> > 
> > 		dma_iova_try_alloc(dev, &state->dma_state,
> > 					   (npages - i) * PAGE_SIZE >=
> > 					   HPAGE_PMD_SIZE ?
> > 					   HPAGE_PMD_SIZE : 0,
> > 					   (npages - i) * PAGE_SIZE);
> > 
> 
> Yes, we can do that. No reason to force alignment if our copy code isn't
> going to try to use 2M GPU pages.

When it comes to this I prefer we try to add alignment information
down to the iova allocator because I have other use cases for this
alignment optimization.

Jason

Re: [PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Matthew Brost]

On Fri, Feb 13, 2026 at 10:56:46AM -0400, Jason Gunthorpe wrote:
> On Wed, Feb 11, 2026 at 10:51:32AM -0800, Matthew Brost wrote:
> > > So to be a little nicer on the IOVA allocator we could use the below?
> > > 
> > > 		dma_iova_try_alloc(dev, &state->dma_state,
> > > 					   (npages - i) * PAGE_SIZE >=
> > > 					   HPAGE_PMD_SIZE ?
> > > 					   HPAGE_PMD_SIZE : 0,
> > > 					   (npages - i) * PAGE_SIZE);
> > > 
> > 
> > Yes, we can do that. No reason to force alignment if our copy code isn't
> > going to try to use 2M GPU pages.
> 
> When it comes to this I prefer we try to add alignment information
> down to the iova allocator because I have other use cases for this
> alignment optimization.

Trying to parse this - what exactly is your preference here in the
context of this patch?

i.e., Is original code ok, is Thomas's suggestion ok, or should we do
something entirely different?

Matt

> 
> Jason

Re: [PATCH v4 4/4] drm/pagemap: Use dma-map IOVA alloc, link, and sync API for DRM pagemap

[Thomas Hellström]

On Fri, 2026-02-13 at 12:00 -0800, Matthew Brost wrote:
> On Fri, Feb 13, 2026 at 10:56:46AM -0400, Jason Gunthorpe wrote:
> > On Wed, Feb 11, 2026 at 10:51:32AM -0800, Matthew Brost wrote:
> > > > So to be a little nicer on the IOVA allocator we could use the
> > > > below?
> > > > 
> > > > 		dma_iova_try_alloc(dev, &state->dma_state,
> > > > 					   (npages - i) *
> > > > PAGE_SIZE >=
> > > > 					   HPAGE_PMD_SIZE ?
> > > > 					   HPAGE_PMD_SIZE : 0,
> > > > 					   (npages - i) *
> > > > PAGE_SIZE);
> > > > 
> > > 
> > > Yes, we can do that. No reason to force alignment if our copy
> > > code isn't
> > > going to try to use 2M GPU pages.
> > 
> > When it comes to this I prefer we try to add alignment information
> > down to the iova allocator because I have other use cases for this
> > alignment optimization.
> 
> Trying to parse this - what exactly is your preference here in the
> context of this patch?
> 
> i.e., Is original code ok, is Thomas's suggestion ok, or should we do
> something entirely different?
> 
> Matt

Interpreting this as Jason would want an alignment parameter to the
IOVA alloctor.

Although that's already the case, albeit somewhat awkwardly named.

Thanks,
Thomas


> 
> > 
> > Jason