All of lore.kernel.org
 help / color / mirror / Atom feed
From: Philip Yang <yangp@amd.com>
To: amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver
Date: Fri, 13 Nov 2026 14:29:42 -0500	[thread overview]
Message-ID: <ecc35307-3332-4bee-bdd9-db66948caac9@amd.com> (raw)
In-Reply-To: <20261019133845.166916-4-xiaogang.chen@amd.com>


[-- Attachment #1.1: Type: text/plain, Size: 13713 bytes --]



On 2026-10-19 09:38, Xiaogang.Chen wrote:
> From: Xiaogang Chen<xiaogang.chen@amd.com>
>
> HMM migrates a PMD-mapped THP as one compound folio when
> migrate_vma_setup() is passed MIGRATE_VMA_SELECT_COMPOUND.
>
> Without that flag the collect path splits the source THP to 4K pages.
> If split_folio() fails because of extra refs or pins, those pages are
> skipped.
>
> Set SELECT_COMPOUND for device-private ranges whose migrate window is
> at least HPAGE_PMD_NR pages. The RAM<->VRAM copy path follows
> MIGRATE_PFN_COMPOUND in src[] (HPAGE_PMD_ORDER device folio, THP DMA)
> and keeps the existing 4K path otherwise.
>
> Coherent zone-device memory is unchanged.

The approach looks right - aligning the migrate window to the folio to 
handle both MIGRATE_PFN_COMPOUND THP migration and normal page migration.

I am attaching two cleanup patches on top of your patch, "drm/amdkfd: 
Simplify svm_migrate_copy_to_vram()" and "drm/amdkfd: Simplify 
svm_migrate_copy_to_ram()". They make no functional change, just making 
the code nicer, cleaner and simpler.

Please see the commit messages for the details of each patch, then 
squash, rework or drop them as you prefer.

Regards,
Philip
>
> Signed-off-by: Xiaogang Chen<xiaogang.chen@amd.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 253 ++++++++++++++++++-----
>   1 file changed, 203 insertions(+), 50 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> index 2febe3bb690e..5c98028bfe73 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
> @@ -324,7 +324,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
>   	u64 mpages = 0;
>   	dma_addr_t *src;
>   	u64 *dst;
> -	u64 i, j;
> +	u64 i, j, k, l, m;
>   	int r = 0;
>   
>   	pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start,
> @@ -335,59 +335,155 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
>   
>   	amdgpu_res_first(prange->ttm_res, ttm_res_offset,
>   			 npages << PAGE_SHIFT, &cursor);
> -	for (i = j = 0; (i < npages) && (mpages < migrate->cpages); i++) {
> +	for (i = j = m = 0; (i < npages) && (mpages < migrate->cpages);) {
>   		struct page *spage;
> +		unsigned long cur_dst_pfn;
>   
> -		if (migrate->src[i] & MIGRATE_PFN_MIGRATE) {
> -			dst[i] = cursor.start + (j << PAGE_SHIFT);
> -			migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
> -			svm_migrate_get_vram_page(prange, migrate->dst[i], 0);
> -			migrate->dst[i] = migrate_pfn(migrate->dst[i]);
> +		cur_dst_pfn = svm_migrate_addr_to_pfn(adev,
> +						      cursor.start + (m << PAGE_SHIFT));
> +
> +		/* when migrate->src[i] has MIGRATE_PFN_COMPOUND set the src page
> +		 * is compound THP; its vm address is HPAGE_PMD_SIZE aligned and
> +		 * its MIGRATE_PFN_MIGRATE is set
> +		 */
> +		if ((m + HPAGE_PMD_NR) <= (cursor.size >> PAGE_SHIFT) &&
> +		    (i + HPAGE_PMD_NR) <= npages &&
> +		    (migrate->src[i] & MIGRATE_PFN_COMPOUND) &&
> +		    IS_ALIGNED(cur_dst_pfn, HPAGE_PMD_NR)) {
> +			k = HPAGE_PMD_NR;
> +		} else
> +			k = 1;
> +
> +		/* for THP src[0] alwas MIGRATE_PFN_MIGRATE
> +		 * just the first migrate->dst need be setup, others are zero
> +		 */
> +		if (k > 1) {
> +
> +			dst[i] = cursor.start + (m << PAGE_SHIFT);
> +			svm_migrate_get_vram_page(prange, cur_dst_pfn,
> +						  HPAGE_PMD_ORDER);
> +
> +			migrate->dst[i] = migrate_pfn(cur_dst_pfn);
> +			migrate->dst[i] |= MIGRATE_PFN_COMPOUND;
> +
> +			for (l = 1; l < k; l++)
> +				migrate->dst[i+l] = 0;
> +
> +			mpages++;
> +
> +		} else if ((migrate->src[i] & MIGRATE_PFN_MIGRATE)) {
> +			dst[i] = cursor.start + (m << PAGE_SHIFT);
> +			svm_migrate_get_vram_page(prange, cur_dst_pfn, 0);
> +			migrate->dst[i] = migrate_pfn(cur_dst_pfn);
>   			mpages++;
>   		}
> -		spage = migrate_pfn_to_page(migrate->src[i]);
> -		if (spage && !is_zone_device_page(spage)) {
> -			src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
> -					      DMA_BIDIRECTIONAL);
> -			r = dma_mapping_error(dev, src[i]);
> -			if (r) {
> -				src[i] = 0;
> -				dev_err(dev, "%s: fail %d dma_map_page\n",
> -					__func__, r);
> -				goto out_free_vram_pages;
> +
> +		if (k > 1) {
> +			if (j) {
> +				/* migrate previous accumulated pages */
> +				r = svm_migrate_copy_memory_gart(
> +						adev, src + i - j,
> +						dst + i - j, j,
> +						FROM_RAM_TO_VRAM,
> +						mfence);
> +
> +				if (r)
> +					goto out_free_vram_pages;
> +
> +				j = 0;
> +			}
> +
> +			/* for THP check if the first src page is valid
> +			 * if not valid skip following HPAGE_PMD_NR - 1 pages
> +			 */
> +			spage = migrate_pfn_to_page(migrate->src[i]);
> +			if (spage && !is_zone_device_page(spage)) {
> +				/* dma_map continuous HPAGE_PMD_NR sys ram pages */
> +				src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE*HPAGE_PMD_NR,
> +						      DMA_BIDIRECTIONAL);
> +
> +				r = dma_mapping_error(dev, src[i]);
> +				if (r) {
> +					dev_err(dev, "%s: fail %d dma_map_page\n",
> +							__func__, r);
> +					goto out_free_vram_pages;
> +				}
> +
> +				/* get dma address for following HPAGE_PMD_NR-1 pages
> +				 * since src pages are continuous their dma addresses
> +				 * are continuous too.
> +				 */
> +				for (l = 1; l < k; l++)
> +					src[i + l] = src[i] + l*PAGE_SIZE;
> +
> +				/* migrate the HPAGE_PMD_NR pages above */
> +				r = svm_migrate_copy_memory_gart(
> +						adev, src + i,
> +						dst + i, HPAGE_PMD_NR,
> +						FROM_RAM_TO_VRAM,
> +						mfence);
> +
> +				/* mark head page dma mapping as THP, tail pages dma addr
> +				 * are set to 0 for following dma_unmap
> +				 */
> +				src[i] |= SVM_RANGE_DMA_THP;
> +				for (l = 1; l < k; l++)
> +					src[i + l] = 0;
> +
> +				if (r)
> +					goto out_free_vram_pages;
> +
> +				j = 0;
>   			}
>   		} else {
> -			if (j) {
> +			/* single normal page case */
> +			spage = migrate_pfn_to_page(migrate->src[i]);
> +			if (spage && !is_zone_device_page(spage)) {
> +				src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
> +						      DMA_BIDIRECTIONAL);
> +
> +				r = dma_mapping_error(dev, src[i]);
> +
> +				if (r) {
> +					dev_err(dev, "%s: fail %d dma_map_page\n",
> +							__func__, r);
> +					goto out_free_vram_pages;
> +				}
> +				j += 1;
> +
> +			} else if (j) {
>   				r = svm_migrate_copy_memory_gart(
>   						adev, src + i - j,
>   						dst + i - j, j,
>   						FROM_RAM_TO_VRAM,
>   						mfence);
> +
>   				if (r)
>   					goto out_free_vram_pages;
> -				amdgpu_res_next(&cursor, (j + 1) << PAGE_SHIFT);
> +
>   				j = 0;
> -			} else {
> -				amdgpu_res_next(&cursor, PAGE_SIZE);
>   			}
> -			continue;
>   		}
>   
> -		pr_debug_ratelimited("dma mapping src to 0x%llx, pfn 0x%lx\n",
> -				     src[i] >> PAGE_SHIFT, page_to_pfn(spage));
> +		pr_debug_ratelimited("dma mapping %lld pages, src to 0x%llx, pfn 0x%lx\n",
> +				     k, src[i] >> PAGE_SHIFT, migrate->src[i] >> MIGRATE_PFN_SHIFT);
> +		i += k;
> +		m += k;
> +
> +		if (m >= (cursor.size >> PAGE_SHIFT)) {
> +			if (j > 0) {
> +				r = svm_migrate_copy_memory_gart(adev, src + i - j,
> +								 dst + i - j, j,
> +								 FROM_RAM_TO_VRAM,
> +								 mfence);
> +				if (r)
> +					goto out_free_vram_pages;
> +			}
> +
> +			amdgpu_res_next(&cursor, m*PAGE_SIZE);
>   
> -		/* accumulated j + 1 pages reach end of current drm_buddy_block */
> -		if (j + 1 >= (cursor.size >> PAGE_SHIFT)) {
> -			r = svm_migrate_copy_memory_gart(adev, src + i - j,
> -							 dst + i - j, j + 1,
> -							 FROM_RAM_TO_VRAM,
> -							 mfence);
> -			if (r)
> -				goto out_free_vram_pages;
> -			amdgpu_res_next(&cursor, (j + 1) * PAGE_SIZE);
>   			j = 0;
> -		} else {
> -			j++;
> +			m = 0;
>   		}
>   	}
>   
> @@ -476,6 +572,7 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange,
>   	struct kfd_process_device *pdd;
>   	struct dma_fence *mfence = NULL;
>   	struct migrate_vma migrate = { 0 };
> +	bool is_private_device = false;
>   	unsigned long cpages = 0;
>   	unsigned long mpages = 0;
>   	dma_addr_t *scratch;
> @@ -488,11 +585,17 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange,
>   	if (svm_range_window_resident(node, prange, start, end))
>   		return 0;
>   
> +	is_private_device = svm_is_private_zone(adev);
> +
>   	memset(&migrate, 0, sizeof(migrate));
>   	migrate.vma = vma;
>   	migrate.start = start;
>   	migrate.end = end;
>   	migrate.flags = MIGRATE_VMA_SELECT_SYSTEM;
> +
> +	if (is_private_device && ((end - start) >> PAGE_SHIFT) >= HPAGE_PMD_NR)
> +		migrate.flags = migrate.flags | MIGRATE_VMA_SELECT_COMPOUND;
> +
>   	migrate.pgmap_owner = SVM_ADEV_PGMAP_OWNER(adev);
>   
>   	buf = kvcalloc(npages,
> @@ -686,6 +789,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>   	u64 i = 0, j;
>   	u64 addr;
>   	int r = 0;
> +	u64 l, k;
>   
>   	pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start,
>   		 prange->last);
> @@ -694,8 +798,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>   
>   	src = (u64 *)(scratch + npages);
>   	dst = scratch;
> -
> -	for (i = 0, j = 0; i < npages; i++, addr += PAGE_SIZE) {
> +	for (i = 0, j = 0, k = 1; i < npages; i += k, addr += k*PAGE_SIZE, k = 1) {
>   		struct page *spage;
>   
>   		spage = migrate_pfn_to_page(migrate->src[i]);
> @@ -711,6 +814,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>   					goto out_oom;
>   				j = 0;
>   			}
> +
>   			continue;
>   		}
>   		src[i] = svm_migrate_addr(adev, spage);
> @@ -724,7 +828,19 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>   			j = 0;
>   		}
>   
> -		dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0);
> +		if (IS_ALIGNED(page_to_pfn(spage), HPAGE_PMD_NR) &&
> +		   (addr + HPAGE_PMD_SIZE) <= migrate->end &&
> +		   IS_ALIGNED(addr, HPAGE_PMD_SIZE) &&
> +		   migrate->src[i] & MIGRATE_PFN_COMPOUND) {
> +
> +			k = HPAGE_PMD_NR;
> +			dpage = svm_migrate_get_sys_page(migrate->vma, addr,
> +							 HPAGE_PMD_ORDER);
> +		} else {
> +			k = 1;
> +			dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0);
> +		}
> +
>   		if (!dpage) {
>   			pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n",
>   				 prange->svms, prange->start, prange->last);
> @@ -732,7 +848,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>   			goto out_oom;
>   		}
>   
> -		dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE, DMA_BIDIRECTIONAL);
> +		dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE*k, DMA_BIDIRECTIONAL);
>   		r = dma_mapping_error(dev, dst[i]);
>   		if (r) {
>   			dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r);
> @@ -740,13 +856,47 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>   			goto out_oom;
>   		}
>   
> -		pr_debug_ratelimited("dma mapping dst to 0x%llx, pfn 0x%lx\n",
> -				     dst[i] >> PAGE_SHIFT, page_to_pfn(dpage));
> -
>   		migrate->dst[i] = migrate_pfn(page_to_pfn(dpage));
>   
> -		dpage = NULL;
> -		j++;
> +		if (k > 1) {
> +			migrate->dst[i] |= MIGRATE_PFN_COMPOUND;
> +			/* migrate previous accumulated pages */
> +			if (j) {
> +				r = svm_migrate_copy_memory_gart(adev, dst + i - j,
> +								 src + i - j, j, FROM_VRAM_TO_RAM, mfence);
> +				if (r)
> +					goto out_oom;
> +				j = 0;
> +			}
> +
> +			for (l = 1; l < k; l++) {
> +
> +				src[i + l] = src[i] + l*PAGE_SIZE;
> +				dst[i + l] = dst[i] + l*PAGE_SIZE;
> +				migrate->dst[i + l] = 0;
> +			}
> +
> +			/* migrate the HPAGE_PMD_NR pages above
> +			 * the migration is for 2MB THP
> +			 */
> +			r = svm_migrate_copy_memory_gart(
> +						adev, dst + i,
> +						src + i, HPAGE_PMD_NR,
> +						FROM_VRAM_TO_RAM,
> +						mfence);
> +
> +			/* mark head page dma mapping as THP, tail pages dma addr
> +			 * are set to 0 for following dma_unmap
> +			 */
> +			dst[i] |= SVM_RANGE_DMA_THP;
> +			for (l = 1; l < k; l++)
> +				dst[i + l] = 0;
> +
> +			if (r)
> +				goto out_oom;
> +
> +		} else
> +			j++;
>   	}
>   
>   	if (j > 0)
> @@ -765,12 +915,9 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
>   		/* release previous allocated sys pages and unmap dma address */
>   		while (i--) {
>   
> -			if (dst[i]) {
> -				dma_unmap_page(dev, dst[i], PAGE_SIZE,
> -					       DMA_BIDIRECTIONAL);
> -				dst[i] = 0;
> -			}
> -
> +			/* follwing svm_range_dma_unmap_dev will do dma unmap anyway
> +			 * not need do dma unmap here
> +			 */
>   			dpage = migrate_pfn_to_page(migrate->dst[i]);
>   			if (!dpage)
>   				continue;
> @@ -811,6 +958,7 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
>   	unsigned long cpages = 0;
>   	unsigned long mpages = 0;
>   	struct amdgpu_device *adev = node->adev;
> +	bool is_private_device = false;
>   	struct kfd_process_device *pdd;
>   	struct dma_fence *mfence = NULL;
>   	struct migrate_vma migrate = { 0 };
> @@ -818,6 +966,8 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
>   	void *buf;
>   	int r = -ENOMEM;
>   
> +	is_private_device = svm_is_private_zone(adev);
> +
>   	memset(&migrate, 0, sizeof(migrate));
>   	migrate.vma = vma;
>   	migrate.start = start;
> @@ -828,6 +978,9 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
>   	else
>   		migrate.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE;
>   
> +	if (is_private_device && ((end - start) >> PAGE_SHIFT) >= HPAGE_PMD_NR)
> +		migrate.flags = migrate.flags | MIGRATE_VMA_SELECT_COMPOUND;
> +
>   	buf = kvcalloc(npages,
>   		       2 * sizeof(*migrate.src) + sizeof(u64) + sizeof(dma_addr_t),
>   		       GFP_KERNEL);

[-- Attachment #1.2: Type: text/html, Size: 14152 bytes --]

[-- Attachment #2: 0002-drm-amdkfd-Simplify-svm_migrate_copy_to_ram.patch --]
[-- Type: text/x-patch, Size: 9637 bytes --]

From b214d0b19f89f674578e05b289f9eed03aa8b873 Mon Sep 17 00:00:00 2001
From: Philip Yang <Philip.Yang@amd.com>
Date: Fri, 13 Nov 2026 14:06:50 -0500
Subject: [PATCH 2/2] drm/amdkfd: Simplify svm_migrate_copy_to_ram()

Fold the THP and the single page paths into one flow, the same way
svm_migrate_copy_to_vram() does:

 - reuse svm_migrate_flush_batch() for the accumulated copy, it takes the
   direction now so both migration directions share it. It is a no-op
   when nothing is pending, so the callers no longer need to guard it and
   svm_migrate_copy_memory_gart() is called from there only.
 - rename the loop variables to i and batch, and derive the step nr from
   order so that order can be passed to svm_migrate_get_sys_page()
   directly.
 - fill the tail entries unconditionally, the loops are no-ops for an
   order 0 page.

The accumulated pages still have to stay contiguous in VRAM, and a THP is
still copied on its own, so both drain the batch first.

This also fixes a double free on the error paths. dpage is declared
outside the loop and the cleanup releases it before walking
migrate->dst[], which is only correct when dma_map_page() failed. On any
other error, a failing copy for instance, dpage still points at a page of
an earlier iteration which is already recorded in migrate->dst[], so the
walk unlocks and puts it a second time. Clear dpage once the page is
recorded, and advance i before the THP copy so that the walk covers the
entry the copy failed on.

No functional change otherwise.

Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 154 +++++++++++------------
 1 file changed, 71 insertions(+), 83 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index b2fe40bc7510..357ff82fbb64 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -314,16 +314,17 @@ static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate)
 
 /* Copy the pages accumulated in [i - *batch, i) and reset the accumulator. */
 static int
-svm_migrate_flush_batch(struct amdgpu_device *adev, dma_addr_t *src, u64 *dst,
-			u64 i, u64 *batch, struct dma_fence **mfence)
+svm_migrate_flush_batch(struct amdgpu_device *adev, dma_addr_t *sys, u64 *vram,
+			u64 i, u64 *batch, enum MIGRATION_COPY_DIR dir,
+			struct dma_fence **mfence)
 {
 	int r;
 
 	if (!*batch)
 		return 0;
 
-	r = svm_migrate_copy_memory_gart(adev, src + i - *batch, dst + i - *batch,
-					 *batch, FROM_RAM_TO_VRAM, mfence);
+	r = svm_migrate_copy_memory_gart(adev, sys + i - *batch, vram + i - *batch,
+					 *batch, dir, mfence);
 	*batch = 0;
 	return r;
 }
@@ -396,7 +397,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 
 		/* A THP is copied on its own, so drain whatever preceded it. */
 		if (order || !spage) {
-			r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence);
+			r = svm_migrate_flush_batch(adev, src, dst, i, &batch,
+						    FROM_RAM_TO_VRAM, mfence);
 			if (r)
 				goto out_free_vram_pages;
 		}
@@ -420,7 +422,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 
 		if (order && spage) {
 			r = svm_migrate_flush_batch(adev, src, dst, i + nr, &batch,
-						    mfence);
+						    FROM_RAM_TO_VRAM, mfence);
 
 			/* Mark the head so svm_range_dma_unmap() unmaps the whole
 			 * THP, and clear the tails.
@@ -441,7 +443,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 		off += nr;
 
 		if (off >= (cursor.size >> PAGE_SHIFT)) {
-			r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence);
+			r = svm_migrate_flush_batch(adev, src, dst, i, &batch,
+						    FROM_RAM_TO_VRAM, mfence);
 			if (r)
 				goto out_free_vram_pages;
 
@@ -450,7 +453,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 		}
 	}
 
-	r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence);
+	r = svm_migrate_flush_batch(adev, src, dst, i, &batch, FROM_RAM_TO_VRAM,
+				    mfence);
 
 out_free_vram_pages:
 	if (r) {
@@ -746,11 +750,8 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
 	struct page *dpage = NULL;
 	dma_addr_t *dst;
 	u64 *src;
-
-	u64 i = 0, j;
-	u64 addr;
+	u64 i, batch, addr;
 	int r = 0;
-	u64 l, k;
 
 	pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start,
 		 prange->last);
@@ -759,49 +760,46 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
 
 	src = (u64 *)(scratch + npages);
 	dst = scratch;
-	for (i = 0, j = 0, k = 1; i < npages; i += k, addr += k*PAGE_SIZE, k = 1) {
-		struct page *spage;
 
-		spage = migrate_pfn_to_page(migrate->src[i]);
+	for (i = batch = 0; i < npages;) {
+		struct page *spage = migrate_pfn_to_page(migrate->src[i]);
+		unsigned int order = 0;
+		u64 nr, n;
+
 		if (!spage || !is_zone_device_page(spage)) {
 			pr_debug("invalid page. Could be in CPU already svms 0x%p [0x%lx 0x%lx]\n",
 				 prange->svms, prange->start, prange->last);
-			if (j) {
-				r = svm_migrate_copy_memory_gart(adev, dst + i - j,
-								 src + i - j, j,
-								 FROM_VRAM_TO_RAM,
-								 mfence);
-				if (r)
-					goto out_oom;
-				j = 0;
-			}
+			r = svm_migrate_flush_batch(adev, dst, src, i, &batch,
+						    FROM_VRAM_TO_RAM, mfence);
+			if (r)
+				goto out_oom;
 
+			i++;
+			addr += PAGE_SIZE;
 			continue;
 		}
+
 		src[i] = svm_migrate_addr(adev, spage);
-		if (j > 0 && src[i] != src[i - 1] + PAGE_SIZE) {
-			r = svm_migrate_copy_memory_gart(adev, dst + i - j,
-							 src + i - j, j,
-							 FROM_VRAM_TO_RAM,
-							 mfence);
-			if (r)
-				goto out_oom;
-			j = 0;
-		}
 
-		if (IS_ALIGNED(page_to_pfn(spage), HPAGE_PMD_NR) &&
-		   (addr + HPAGE_PMD_SIZE) <= migrate->end &&
-		   IS_ALIGNED(addr, HPAGE_PMD_SIZE) &&
-		   migrate->src[i] & MIGRATE_PFN_COMPOUND) {
+		/* MIGRATE_PFN_COMPOUND means the source is a 2MB VRAM folio. */
+		if ((migrate->src[i] & MIGRATE_PFN_COMPOUND) &&
+		    IS_ALIGNED(page_to_pfn(spage), HPAGE_PMD_NR) &&
+		    IS_ALIGNED(addr, HPAGE_PMD_SIZE) &&
+		    addr + HPAGE_PMD_SIZE <= migrate->end)
+			order = HPAGE_PMD_ORDER;
+		nr = 1ULL << order;
 
-			k = HPAGE_PMD_NR;
-			dpage = svm_migrate_get_sys_page(migrate->vma, addr,
-							 HPAGE_PMD_ORDER);
-		} else {
-			k = 1;
-			dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0);
+		/* The accumulated pages have to stay contiguous in VRAM, and a
+		 * THP is copied on its own.
+		 */
+		if (order || (batch && src[i] != src[i - 1] + PAGE_SIZE)) {
+			r = svm_migrate_flush_batch(adev, dst, src, i, &batch,
+						    FROM_VRAM_TO_RAM, mfence);
+			if (r)
+				goto out_oom;
 		}
 
+		dpage = svm_migrate_get_sys_page(migrate->vma, addr, order);
 		if (!dpage) {
 			pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n",
 				 prange->svms, prange->start, prange->last);
@@ -809,65 +807,55 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
 			goto out_oom;
 		}
 
-		dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE*k, DMA_BIDIRECTIONAL);
+		dst[i] = dma_map_page(dev, dpage, 0, nr << PAGE_SHIFT,
+				      DMA_BIDIRECTIONAL);
 		r = dma_mapping_error(dev, dst[i]);
 		if (r) {
-			dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r);
+			dev_err(dev, "%s: fail %d dma_map_page\n", __func__, r);
 			dst[i] = 0;
 			goto out_oom;
 		}
 
-		migrate->dst[i] = migrate_pfn(page_to_pfn(dpage));
-
-		if (k > 1) {
-			migrate->dst[i] |= MIGRATE_PFN_COMPOUND;
-			/* migrate previous accumulated pages */
-			if (j) {
-				r = svm_migrate_copy_memory_gart(adev, dst + i - j,
-								 src + i - j, j, FROM_VRAM_TO_RAM, mfence);
-				if (r)
-					goto out_oom;
-				j = 0;
-			}
+		migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)) |
+				  (order ? MIGRATE_PFN_COMPOUND : 0);
 
-			for (l = 1; l < k; l++) {
+		/* Both sides are contiguous, so are their addresses. */
+		for (n = 1; n < nr; n++) {
+			src[i + n] = src[i] + (n << PAGE_SHIFT);
+			dst[i + n] = dst[i] + (n << PAGE_SHIFT);
+			migrate->dst[i + n] = 0;
+		}
+		batch += nr;
 
-				src[i + l] = src[i] + l*PAGE_SIZE;
-				dst[i + l] = dst[i] + l*PAGE_SIZE;
-				migrate->dst[i + l] = 0;
-			}
+		/* dpage is now owned by migrate->dst[], out_oom frees it from there. */
+		dpage = NULL;
 
-			/* migrate the HPAGE_PMD_NR pages above
-			 * the migration is for 2MB THP
-			 */
-			r = svm_migrate_copy_memory_gart(
-						adev, dst + i,
-						src + i, HPAGE_PMD_NR,
-						FROM_VRAM_TO_RAM,
-						mfence);
-
-			/* mark head page dma mapping as THP, tail pages dma addr
-			 * are set to 0 for following dma_unmap
+		i += nr;
+		addr += nr << PAGE_SHIFT;
+
+		if (order) {
+			r = svm_migrate_flush_batch(adev, dst, src, i, &batch,
+						    FROM_VRAM_TO_RAM, mfence);
+
+			/* Mark the head so svm_range_dma_unmap() unmaps the whole
+			 * THP, and clear the tails.
 			 */
-			dst[i] |= SVM_RANGE_DMA_THP;
-			for (l = 1; l < k; l++)
-				dst[i + l] = 0;
+			dst[i - nr] |= SVM_RANGE_DMA_THP;
+			for (n = 1; n < nr; n++)
+				dst[i - nr + n] = 0;
 
 			if (r)
 				goto out_oom;
-
-		} else
-			j++;
+		}
 	}
 
-	if (j > 0)
-		r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j,
-						 FROM_VRAM_TO_RAM, mfence);
+	r = svm_migrate_flush_batch(adev, dst, src, i, &batch, FROM_VRAM_TO_RAM,
+				    mfence);
 out_oom:
 	if (r) {
 		pr_debug("failed %d copy to ram\n", r);
 
-		/* first release current dpage when dma_map_page fail */
+		/* The page of the current iteration is not in migrate->dst[] yet. */
 		if (dpage) {
 			unlock_page(dpage);
 			put_page(dpage);
-- 
2.50.1


[-- Attachment #3: 0001-drm-amdkfd-Simplify-svm_migrate_copy_to_vram.patch --]
[-- Type: text/x-patch, Size: 10731 bytes --]

From 20ba86327c7c67749b027ceaccf32b7d59fe3401 Mon Sep 17 00:00:00 2001
From: Philip Yang <Philip.Yang@amd.com>
Date: Thu, 12 Nov 2026 17:53:36 -0500
Subject: [PATCH 1/2] drm/amdkfd: Simplify svm_migrate_copy_to_vram()

The THP and the single page paths were written as two separate flows
using five single letter loop variables, with the same "copy whatever
has been accumulated so far" block open coded four times.

Fold them into one flow:

 - add svm_migrate_flush_batch() for the accumulated copy, it is a no-op
   when nothing is pending so the callers no longer need to guard it.
   svm_migrate_copy_memory_gart() is now called from there only.
 - rename the loop variables to i, batch and off, and derive the step nr
   from order so that order can be passed to svm_migrate_get_vram_page()
   directly.
 - claim the destination in a single branch, the tail loops are no-ops
   for an order 0 page.
 - map the source with one dma_map_page() of nr pages.

A THP is still copied on its own because only its head fills dst[], so
it cannot share a transfer with the accumulated pages.

This also fixes a page that is left in system memory because its reused
svm_bo destination page is still referenced by a CPU fault. The old code
signalled the skip by clearing spage, but the single page path
recomputed spage right after and the page was dma mapped and copied with
dst[i] == 0. A source page without MIGRATE_PFN_MIGRATE took the same
path. spage is now computed once and every rejection clears it for good.

No functional change otherwise.

Signed-off-by: Philip Yang <Philip.Yang@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 242 +++++++++--------------
 1 file changed, 98 insertions(+), 144 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index 690599dea15d..b2fe40bc7510 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -312,6 +312,22 @@ static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate)
 	return mpages;
 }
 
+/* Copy the pages accumulated in [i - *batch, i) and reset the accumulator. */
+static int
+svm_migrate_flush_batch(struct amdgpu_device *adev, dma_addr_t *src, u64 *dst,
+			u64 i, u64 *batch, struct dma_fence **mfence)
+{
+	int r;
+
+	if (!*batch)
+		return 0;
+
+	r = svm_migrate_copy_memory_gart(adev, src + i - *batch, dst + i - *batch,
+					 *batch, FROM_RAM_TO_VRAM, mfence);
+	*batch = 0;
+	return r;
+}
+
 static int
 svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 			 struct migrate_vma *migrate, struct dma_fence **mfence,
@@ -324,7 +340,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 	u64 mpages = 0;
 	dma_addr_t *src;
 	u64 *dst;
-	u64 i, j, k, l, m;
+	u64 i, batch, off;
 	int r = 0;
 
 	pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start,
@@ -335,168 +351,106 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 
 	amdgpu_res_first(prange->ttm_res, ttm_res_offset,
 			 npages << PAGE_SHIFT, &cursor);
-	for (i = j = m = 0; (i < npages) && (mpages < migrate->cpages);) {
-		struct page *spage = migrate_pfn_to_page(migrate->src[i]);
-		unsigned long cur_dst_pfn;
-
-		cur_dst_pfn = svm_migrate_addr_to_pfn(adev,
-						      cursor.start + (m << PAGE_SHIFT));
-
-		/* when migrate->src[i] has MIGRATE_PFN_COMPOUND set the src page
-		 * is compound THP; its vm address is HPAGE_PMD_SIZE aligned and
-		 * its MIGRATE_PFN_MIGRATE is set
-		*/
-		if ((m + HPAGE_PMD_NR) <= (cursor.size >> PAGE_SHIFT) &&
-		    (i + HPAGE_PMD_NR) <= npages &&
-		    (migrate->src[i] & MIGRATE_PFN_COMPOUND) &&
-		    IS_ALIGNED(cur_dst_pfn, HPAGE_PMD_NR))
-			k = HPAGE_PMD_NR;
-		else
-			k = 1;
-
-		/* for THP src[0] alwas MIGRATE_PFN_MIGRATE
-		 * just the first migrate->dst need be setup, others are zero
-		 */
-		if (k > 1) {
-			dst[i] = cursor.start + (m << PAGE_SHIFT);
-			svm_migrate_get_vram_page(prange, cur_dst_pfn,
-						  HPAGE_PMD_ORDER);
 
-			migrate->dst[i] = migrate_pfn(cur_dst_pfn);
-			migrate->dst[i] |= MIGRATE_PFN_COMPOUND;
+	for (i = batch = off = 0; i < npages && mpages < migrate->cpages;) {
+		struct page *spage = migrate_pfn_to_page(migrate->src[i]);
+		u64 vram_addr = cursor.start + (off << PAGE_SHIFT);
+		unsigned long dst_pfn;
+		unsigned int order = 0;
+		u64 nr, n;
 
-			for (l = 1; l < k; l++)
-				migrate->dst[i+l] = 0;
+		dst_pfn = svm_migrate_addr_to_pfn(adev, vram_addr);
 
+		/* MIGRATE_PFN_COMPOUND means the source is a THP, its VA is
+		 * HPAGE_PMD_SIZE aligned and MIGRATE_PFN_MIGRATE is set.
+		 */
+		if ((migrate->src[i] & MIGRATE_PFN_COMPOUND) &&
+		    off + HPAGE_PMD_NR <= (cursor.size >> PAGE_SHIFT) &&
+		    i + HPAGE_PMD_NR <= npages &&
+		    IS_ALIGNED(dst_pfn, HPAGE_PMD_NR))
+			order = HPAGE_PMD_ORDER;
+		nr = 1ULL << order;
+
+		if (spage && is_zone_device_page(spage))
+			spage = NULL;
+
+		/* Claim the destination, unless the source cannot migrate or a
+		 * reused svm_bo page is still held by a CPU fault: reinitializing
+		 * it would free it under this migration.
+		 */
+		if (!(migrate->src[i] & MIGRATE_PFN_MIGRATE) ||
+		    (!order && page_ref_count(pfn_to_page(dst_pfn)))) {
+			dst[i] = 0;
+			migrate->dst[i] = 0;
+			spage = NULL;
+		} else {
+			/* A THP only fills the head entry, the tails stay zero. */
+			dst[i] = vram_addr;
+			svm_migrate_get_vram_page(prange, dst_pfn, order);
+			migrate->dst[i] = migrate_pfn(dst_pfn) |
+					  (order ? MIGRATE_PFN_COMPOUND : 0);
+			for (n = 1; n < nr; n++)
+				migrate->dst[i + n] = 0;
 			mpages++;
-		} else if (migrate->src[i] & MIGRATE_PFN_MIGRATE) {
-			dst[i] = cursor.start + (m << PAGE_SHIFT);
-			/*
-			 * A reused svm_bo page may still be held by a CPU fault.
-			 * Reinitializing it would free it under this migration,
-			 * so leave this page in system memory.
-			 */
-			if (page_ref_count(pfn_to_page(cur_dst_pfn))) {
-				dst[i] = 0;
-				migrate->dst[i] = 0;
-				spage = NULL;
-			} else {
-				svm_migrate_get_vram_page(prange, cur_dst_pfn, 0);
-				migrate->dst[i] = migrate_pfn(cur_dst_pfn);
-				mpages++;
-			}
 		}
 
-		if (k > 1) {
-			if (j) {
-				/* migrate previous accumulated pages */
-				r = svm_migrate_copy_memory_gart(
-						adev, src + i - j,
-						dst + i - j, j,
-						FROM_RAM_TO_VRAM,
-						mfence);
-
-				if (r)
-					goto out_free_vram_pages;
+		/* A THP is copied on its own, so drain whatever preceded it. */
+		if (order || !spage) {
+			r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence);
+			if (r)
+				goto out_free_vram_pages;
+		}
 
-				j = 0;
+		if (spage) {
+			src[i] = dma_map_page(dev, spage, 0, nr << PAGE_SHIFT,
+					      DMA_BIDIRECTIONAL);
+			r = dma_mapping_error(dev, src[i]);
+			if (r) {
+				dev_err(dev, "%s: fail %d dma_map_page\n", __func__, r);
+				goto out_free_vram_pages;
 			}
 
-			/* for THP check if the first src page is valid
-			 * if not valid skip following HPAGE_PMD_NR - 1 pages
+			/* The source pages are contiguous, so are their dma
+			 * addresses.
 			 */
-			spage = migrate_pfn_to_page(migrate->src[i]);
-			if (spage && !is_zone_device_page(spage)) {
-				/* dma_map continuous HPAGE_PMD_NR sys ram pages */
-				src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE*HPAGE_PMD_NR,
-						      DMA_BIDIRECTIONAL);
-
-				r = dma_mapping_error(dev, src[i]);
-				if (r) {
-					dev_err(dev, "%s: fail %d dma_map_page\n",
-							__func__, r);
-					goto out_free_vram_pages;
-				}
-
-				/* get dma address for following HPAGE_PMD_NR-1 pages
-				 * since src pages are continuous their dma addresses
-				 * are continuous too.
-				 */
-				for (l = 1; l < k; l++)
-					src[i + l] = src[i] + l*PAGE_SIZE;
-
-				/* migrate the HPAGE_PMD_NR pages above */
-				r = svm_migrate_copy_memory_gart(
-						adev, src + i,
-						dst + i, HPAGE_PMD_NR,
-						FROM_RAM_TO_VRAM,
-						mfence);
-
-				/* mark head page dma mapping as THP, tail pages dma addr
-				 * are set to 0 for following dma_unmap
-				 */
-				src[i] |= SVM_RANGE_DMA_THP;
-				for (l = 1; l < k; l++)
-					src[i + l] = 0;
-
-				if (r)
-					goto out_free_vram_pages;
-
-				j = 0;
-			}
-		} else {
-
-			/* single normal page case */
-			spage = migrate_pfn_to_page(migrate->src[i]);
-			if (spage && !is_zone_device_page(spage)) {
-				src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
-					      DMA_BIDIRECTIONAL);
+			for (n = 1; n < nr; n++)
+				src[i + n] = src[i] + (n << PAGE_SHIFT);
+			batch += nr;
+		}
 
-				r = dma_mapping_error(dev, src[i]);
+		if (order && spage) {
+			r = svm_migrate_flush_batch(adev, src, dst, i + nr, &batch,
+						    mfence);
 
-				if (r) {
-					dev_err(dev, "%s: fail %d dma_map_page\n",
-							__func__, r);
-					goto out_free_vram_pages;
-				}
-				j += 1;
+			/* Mark the head so svm_range_dma_unmap() unmaps the whole
+			 * THP, and clear the tails.
+			 */
+			src[i] |= SVM_RANGE_DMA_THP;
+			for (n = 1; n < nr; n++)
+				src[i + n] = 0;
 
-			} else if (j) {
-				r = svm_migrate_copy_memory_gart(
-						adev, src + i - j,
-						dst + i - j, j,
-						FROM_RAM_TO_VRAM,
-						mfence);
-				if (r)
-					goto out_free_vram_pages;
-				j = 0;
-			}
+			if (r)
+				goto out_free_vram_pages;
 		}
 
-		pr_debug_ratelimited("dma mapping %lld pages, src to 0x%llx, pfn 0x%lx\n",
-				     k, src[i] >> PAGE_SHIFT, migrate->src[i] >> MIGRATE_PFN_SHIFT);
+		pr_debug_ratelimited("dma mapping %llu pages, src to 0x%llx, pfn 0x%lx\n",
+				     nr, src[i] >> PAGE_SHIFT,
+				     migrate->src[i] >> MIGRATE_PFN_SHIFT);
 
-		i += k;
-		m += k;
+		i += nr;
+		off += nr;
 
-		if (m >= (cursor.size >> PAGE_SHIFT)) {
-			if (j > 0) {
-				r = svm_migrate_copy_memory_gart(adev, src + i - j,
-								 dst + i - j, j,
-								 FROM_RAM_TO_VRAM,
-								 mfence);
-				if (r)
-					goto out_free_vram_pages;
-			}
+		if (off >= (cursor.size >> PAGE_SHIFT)) {
+			r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence);
+			if (r)
+				goto out_free_vram_pages;
 
-			amdgpu_res_next(&cursor, m*PAGE_SIZE);
-			m = 0;
+			amdgpu_res_next(&cursor, off << PAGE_SHIFT);
+			off = 0;
 		}
 	}
 
-	if (j > 0)
-		r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j,
-					 FROM_RAM_TO_VRAM, mfence);
+	r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence);
 
 out_free_vram_pages:
 	if (r) {
@@ -511,8 +465,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
 	}
 
 #ifdef DEBUG_FORCE_MIXED_DOMAINS
-	for (i = 0, j = 0; i < npages; i += 4, j++) {
-		if (j & 1)
+	for (i = 0, off = 0; i < npages; i += 4, off++) {
+		if (off & 1)
 			continue;
 		svm_migrate_put_vram_page(adev, dst[i]);
 		migrate->dst[i] = 0;
-- 
2.50.1


      reply	other threads:[~2026-11-13 19:48 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-19 13:38 [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver Xiaogang.Chen
2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen
2026-11-13 23:28   ` Philip Yang
2026-11-16 15:54     ` Chen, Xiaogang
2026-10-19 13:38 ` [PATCH v3 2/3] drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault handler Xiaogang.Chen
2026-10-19 13:38 ` [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver Xiaogang.Chen
2026-11-13 19:29   ` Philip Yang [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ecc35307-3332-4bee-bdd9-db66948caac9@amd.com \
    --to=yangp@amd.com \
    --cc=amd-gfx@lists.freedesktop.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.