* [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver
@ 2026-10-19 13:38 Xiaogang.Chen
2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Xiaogang.Chen @ 2026-10-19 13:38 UTC (permalink / raw)
To: amd-gfx; +Cc: Xiaogang Chen
From: Xiaogang Chen <xiaogang.chen@amd.com>
Add device private THP support to migration in kfd driver by using following
work [1]. The implementation enables efficient migration of large folios
between system memory and device-private memory.
A benchmark test shows the migration speed got improved more than 5 times
comparing to current kfd/svm driver.
[1]https://lore.kernel.org/linux-mm/20251001065707.920170-1-balbirs@nvidia.com/
v2:
- Reset is_large each 4K page iteration
- Unmap the THP DMA mapping once as DMA mapping, at its real size
- Leave migrate->dst[i] zero when the source is not migrating
- Count THP migrations as HPAGE_PMD_NR pages in prange->vram_pages
- Implement folio_split and hold one svm_bo ref per subpage
v3:
- Do not migrate partial THP memory at CPU/GPU page fault handler
- re-configure compound device folio after it is release from core HMM.
So, it can be reused by ttm as 4K pages.
- Remove SDMA page entry FLAG(9) setting at GART page table since
not see noticelbe perofmance change.
Xiaogang Chen (3):
drm/amdkfd: Add awareness of THP of device and system RAM in kfd svm
driver
drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault
handler
drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 372 +++++++++++++++++++----
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 119 +++++++-
drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 13 +
3 files changed, 435 insertions(+), 69 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM in kfd svm driver 2026-10-19 13:38 [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver Xiaogang.Chen @ 2026-10-19 13:38 ` Xiaogang.Chen 2026-11-13 23:28 ` Philip Yang 2026-10-19 13:38 ` [PATCH v3 2/3] drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault handler Xiaogang.Chen 2026-10-19 13:38 ` [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver Xiaogang.Chen 2 siblings, 1 reply; 7+ messages in thread From: Xiaogang.Chen @ 2026-10-19 13:38 UTC (permalink / raw) To: amd-gfx; +Cc: Xiaogang Chen From: Xiaogang Chen <xiaogang.chen@amd.com> Extend kfd/svm functions to allocate HPAGE_PMD_SIZE based device memory by buddy allocator, each drm_buddy_block is HPAGE_PMD_SIZE aligned; and to allocate THP system ram by vma_alloc_folio. Introduce SVM_RANGE_DMA_THP flag that indicates dma map of THP. THP dma addresss will use this flag. Add dev_pagemap_ops->folio_split callback that is called by folio_split when core MM splits device memory folio. Reset compound folio of device memory after it got released from core HMM. So, TTM can reuse this VRAM as 4K pages on next range. These are preparations for following support for (THP) migration of zone device-private memory, no function change. Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com> --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 84 ++++++++++++++++++++---- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 37 +++++++++-- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 + 3 files changed, 104 insertions(+), 18 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 8b96e1fe741a..7a129c9954fc 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -218,17 +218,45 @@ svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr) return (addr + adev->kfd.pgmap.range.start) >> PAGE_SHIFT; } +/* + * TTM can reuse a 2MB VRAM block that was previously a device THP. + * Reset unused pages so the next 4K init does not see leftover head/tail. + */ +static void svm_migrate_clear_compound_pages(struct page *page, unsigned int nr, + struct dev_pagemap *pgmap) +{ + unsigned int i; + + for (i = 0; i < nr; i++, page++) { + struct folio *new_folio = (struct folio *)page; + + if (page_count(page)) + continue; + + page->flags.f &= ~0xffUL; +#ifdef NR_PAGES_IN_LARGE_FOLIO + ((struct folio *)(page - 1))->_nr_pages = 0; +#endif + new_folio->mapping = NULL; + new_folio->pgmap = pgmap; + new_folio->share = 0; + page->zone_device_data = NULL; + } +} + static void -svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn) +svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn, + int order) { - struct page *page; + struct page *page = pfn_to_page(pfn); - page = pfn_to_page(pfn); - svm_range_bo_ref(prange->svm_bo); + /* Init this pfn, not page_folio() — that follows a leftover tail to the old head. */ + zone_device_folio_init((struct folio *)page, page_pgmap(page), order); page->zone_device_data = prange->svm_bo; - zone_device_page_init(page, page_pgmap(page), 0); + svm_range_bo_ref(prange->svm_bo); } + static void svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long addr) { @@ -249,11 +277,17 @@ svm_migrate_addr(struct amdgpu_device *adev, struct page *page) } static struct page * -svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr) +svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr, + unsigned long order) { struct page *page; - page = alloc_page_vma(GFP_HIGHUSER, vma, addr); + if (order) + page = folio_page(vma_alloc_folio(GFP_HIGHUSER, + order, vma, addr), 0); + else + page = alloc_page_vma(GFP_HIGHUSER, vma, addr); + if (page) lock_page(page); @@ -267,8 +301,13 @@ static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate) for (i = 0; i < migrate->npages; i++) { if (migrate->dst[i] & MIGRATE_PFN_VALID && - migrate->src[i] & MIGRATE_PFN_MIGRATE) - mpages++; + migrate->src[i] & MIGRATE_PFN_MIGRATE) { + + if (migrate->dst[i] & MIGRATE_PFN_COMPOUND) + mpages += HPAGE_PMD_NR; + else + mpages++; + } } return mpages; } @@ -302,7 +341,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, if (migrate->src[i] & MIGRATE_PFN_MIGRATE) { dst[i] = cursor.start + (j << PAGE_SHIFT); migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); - svm_migrate_get_vram_page(prange, migrate->dst[i]); + svm_migrate_get_vram_page(prange, migrate->dst[i], 0); migrate->dst[i] = migrate_pfn(migrate->dst[i]); mpages++; } @@ -616,15 +655,22 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, return r < 0 ? r : 0; } +/* folio can be compound folio or single page */ static void svm_migrate_folio_free(struct folio *folio) { struct page *page = &folio->page; struct svm_range_bo *svm_bo = page->zone_device_data; + struct dev_pagemap *pgmap = folio->pgmap; + unsigned int order = folio_order(folio); if (svm_bo) { pr_debug_ratelimited("ref: %d\n", kref_read(&svm_bo->kref)); svm_range_bo_unref_async(svm_bo); } + + /* So TTM can reuse this VRAM as 4K pages on the next range. */ + if (order) + svm_migrate_clear_compound_pages(page, 1U << order, pgmap); } static int @@ -678,7 +724,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, j = 0; } - dpage = svm_migrate_get_sys_page(migrate->vma, addr); + dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); if (!dpage) { pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); @@ -1081,9 +1127,25 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) return r ? VM_FAULT_SIGBUS : 0; } +static void svm_migrate_folio_split(struct folio *head, struct folio *tail) +{ + struct svm_range_bo *svm_bo; + + if (tail == NULL) + return; + + tail->pgmap = head->pgmap; + tail->mapping = head->mapping; + + svm_bo = folio_zone_device_data(head); + folio_set_zone_device_data(tail, svm_bo); + svm_range_bo_ref(svm_bo); +} + static const struct dev_pagemap_ops svm_migrate_pgmap_ops = { .folio_free = svm_migrate_folio_free, .migrate_to_ram = svm_migrate_to_ram, + .folio_split = svm_migrate_folio_split, }; /* Each VRAM page uses sizeof(struct page) on system memory */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index a99dbedf3cd4..8e1fb5df8abd 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -245,7 +245,16 @@ void svm_range_dma_unmap_dev(struct device *dev, dma_addr_t *dma_addr, if (!svm_is_valid_dma_mapping_addr(dev, dma_addr[i])) continue; pr_debug_ratelimited("unmap 0x%llx\n", dma_addr[i] >> PAGE_SHIFT); - dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); + + /* dma unmap of THP */ + if (dma_addr[i] & SVM_RANGE_DMA_THP) { + dma_addr[i] &= ~SVM_RANGE_DMA_THP; + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE*HPAGE_PMD_NR, + DMA_BIDIRECTIONAL); + } else { + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); + } + dma_addr[i] = 0; } } @@ -578,10 +587,15 @@ svm_range_vram_node_new(struct kfd_node *node, struct svm_range *prange, } memset(&bp, 0, sizeof(bp)); - bp.size = prange->npages * PAGE_SIZE; + if (prange->npages >= HPAGE_PMD_NR) { + bp.size = ALIGN(prange->npages * PAGE_SIZE, HPAGE_PMD_SIZE); + bp.byte_align = HPAGE_PMD_SIZE; + } else { + bp.size = prange->npages * PAGE_SIZE; + bp.byte_align = PAGE_SIZE; + } bp.bo_ptr_size = sizeof(struct svm_range_bo); bp.destroy = svm_range_bo_destroy; - bp.byte_align = PAGE_SIZE; bp.domain = AMDGPU_GEM_DOMAIN_VRAM; bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS; bp.flags |= clear ? AMDGPU_GEM_CREATE_VRAM_CLEARED : 0; @@ -593,9 +607,17 @@ svm_range_vram_node_new(struct kfd_node *node, struct svm_range *prange, r = amdgpu_bo_create(node->adev, &bp, &bo); if (r) { - pr_debug("failed %d to create bo\n", r); - mmput(mm); - goto create_bo_failed; + if (prange->npages >= HPAGE_PMD_NR) { + /* fall back PAGE_SIZE granularity */ + bp.size = prange->npages * PAGE_SIZE; + bp.byte_align = PAGE_SIZE; + r = amdgpu_bo_create(node->adev, &bp, &bo); + } + if (r) { + pr_debug("failed %d to create bo\n", r); + mmput(mm); + goto create_bo_failed; + } } svm_bo = to_svm_range_bo(bo); @@ -956,6 +978,7 @@ svm_range_copy_array(void *psrc, size_t size, uint64_t num_elements, *vram_pages = 0; for (i = 0; i < num_elements; i++) { dma_addr_t *temp; + temp = (dma_addr_t *)dst + i; *temp = *((dma_addr_t *)src + i); if (*temp&SVM_RANGE_VRAM_DOMAIN) @@ -1628,7 +1651,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) uint32_t gpuidx; int r; - drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT: 0, 0); + drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT : 0, 0); drm_exec_until_all_locked(&ctx->exec) { for_each_set_bit(gpuidx, ctx->bitmap, MAX_GPU_INSTANCE) { pdd = kfd_process_device_from_gpuidx(ctx->process, gpuidx); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index c7d7adae4476..f2b3a05cd8cf 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -35,6 +35,7 @@ #include "kfd_priv.h" #define SVM_RANGE_VRAM_DOMAIN (1UL << 0) +#define SVM_RANGE_DMA_THP (1UL << 1) #define SVM_ADEV_PGMAP_OWNER(adev)\ ((adev)->hive ? (void *)(adev)->hive : (void *)(adev)) -- 2.34.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM in kfd svm driver 2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen @ 2026-11-13 23:28 ` Philip Yang 2026-11-16 15:54 ` Chen, Xiaogang 0 siblings, 1 reply; 7+ messages in thread From: Philip Yang @ 2026-11-13 23:28 UTC (permalink / raw) To: Xiaogang.Chen, amd-gfx [-- Attachment #1: Type: text/plain, Size: 10635 bytes --] On 2026-10-19 09:38, Xiaogang.Chen wrote: > From: Xiaogang Chen<xiaogang.chen@amd.com> > > Extend kfd/svm functions to allocate HPAGE_PMD_SIZE based device memory by > buddy allocator, each drm_buddy_block is HPAGE_PMD_SIZE aligned; and to > allocate THP system ram by vma_alloc_folio. > > Introduce SVM_RANGE_DMA_THP flag that indicates dma map of THP. THP dma > addresss will use this flag. > > Add dev_pagemap_ops->folio_split callback that is called by folio_split when > core MM splits device memory folio. > > Reset compound folio of device memory after it got released from core HMM. > So, TTM can reuse this VRAM as 4K pages on next range. > > These are preparations for following support for (THP) migration of zone > device-private memory, no function change. > > Signed-off-by: Xiaogang Chen<xiaogang.chen@amd.com> > --- > drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 84 ++++++++++++++++++++---- > drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 37 +++++++++-- > drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 + > 3 files changed, 104 insertions(+), 18 deletions(-) > > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c > index 8b96e1fe741a..7a129c9954fc 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c > @@ -218,17 +218,45 @@ svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr) > return (addr + adev->kfd.pgmap.range.start) >> PAGE_SHIFT; > } > > +/* > + * TTM can reuse a 2MB VRAM block that was previously a device THP. > + * Reset unused pages so the next 4K init does not see leftover head/tail. > + */ > +static void svm_migrate_clear_compound_pages(struct page *page, unsigned int nr, > + struct dev_pagemap *pgmap) > +{ > + unsigned int i; > + > + for (i = 0; i < nr; i++, page++) { > + struct folio *new_folio = (struct folio *)page; > + > + if (page_count(page)) > + continue; > + > + page->flags.f &= ~0xffUL; > +#ifdef NR_PAGES_IN_LARGE_FOLIO > + ((struct folio *)(page - 1))->_nr_pages = 0; > +#endif > + new_folio->mapping = NULL; > + new_folio->pgmap = pgmap; > + new_folio->share = 0; > + page->zone_device_data = NULL; > + } > +} > + This is called after the device page is freed. When migrating, svm_migrate_get_vram_page() uses the VRAM page and zone_device_folio_init() does the same initialization, so this looks duplicated. Is it? > static void > -svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn) > +svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn, > + int order) > { > - struct page *page; > + struct page *page = pfn_to_page(pfn); > > - page = pfn_to_page(pfn); > - svm_range_bo_ref(prange->svm_bo); > + /* Init this pfn, not page_folio() — that follows a leftover tail to the old head. */ > + zone_device_folio_init((struct folio *)page, page_pgmap(page), order); > page->zone_device_data = prange->svm_bo; > - zone_device_page_init(page, page_pgmap(page), 0); > + svm_range_bo_ref(prange->svm_bo); > } > > + > static void > svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long addr) > { > @@ -249,11 +277,17 @@ svm_migrate_addr(struct amdgpu_device *adev, struct page *page) > } > > static struct page * > -svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr) > +svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr, > + unsigned long order) > { > struct page *page; > > - page = alloc_page_vma(GFP_HIGHUSER, vma, addr); > + if (order) > + page = folio_page(vma_alloc_folio(GFP_HIGHUSER, > + order, vma, addr), 0); vma_alloc_folio could fail, then folio_page has NULL access, should check vma_alloc_folio return value. if vma_alloc_folio() fails for order 9, this fails the entire migration with -ENOMEM and the CPU fault becomes SIGBUS. migrate_vma_pages() already handles a compound source with a non-compound destination by splitting the source, so falling back to a 4K page here would be enough. Where is that handled? > + else > + page = alloc_page_vma(GFP_HIGHUSER, vma, addr); > + > if (page) > lock_page(page); > > @@ -267,8 +301,13 @@ static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate) > > for (i = 0; i < migrate->npages; i++) { > if (migrate->dst[i] & MIGRATE_PFN_VALID && > - migrate->src[i] & MIGRATE_PFN_MIGRATE) > - mpages++; > + migrate->src[i] & MIGRATE_PFN_MIGRATE) { > + > + if (migrate->dst[i] & MIGRATE_PFN_COMPOUND) > + mpages += HPAGE_PMD_NR; > + else > + mpages++; > + } > } > return mpages; > } > @@ -302,7 +341,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, > if (migrate->src[i] & MIGRATE_PFN_MIGRATE) { > dst[i] = cursor.start + (j << PAGE_SHIFT); > migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); > - svm_migrate_get_vram_page(prange, migrate->dst[i]); > + svm_migrate_get_vram_page(prange, migrate->dst[i], 0); > migrate->dst[i] = migrate_pfn(migrate->dst[i]); > mpages++; > } > @@ -616,15 +655,22 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, > return r < 0 ? r : 0; > } > > +/* folio can be compound folio or single page */ > static void svm_migrate_folio_free(struct folio *folio) > { > struct page *page = &folio->page; > struct svm_range_bo *svm_bo = page->zone_device_data; > + struct dev_pagemap *pgmap = folio->pgmap; > + unsigned int order = folio_order(folio); > > if (svm_bo) { > pr_debug_ratelimited("ref: %d\n", kref_read(&svm_bo->kref)); > svm_range_bo_unref_async(svm_bo); > } > + > + /* So TTM can reuse this VRAM as 4K pages on the next range. */ > + if (order) > + svm_migrate_clear_compound_pages(page, 1U << order, pgmap); > } > > static int > @@ -678,7 +724,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > j = 0; > } > > - dpage = svm_migrate_get_sys_page(migrate->vma, addr); > + dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); > if (!dpage) { > pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n", > prange->svms, prange->start, prange->last); > @@ -1081,9 +1127,25 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) > return r ? VM_FAULT_SIGBUS : 0; > } > > +static void svm_migrate_folio_split(struct folio *head, struct folio *tail) > +{ > + struct svm_range_bo *svm_bo; > + > + if (tail == NULL) > + return; > + > + tail->pgmap = head->pgmap; > + tail->mapping = head->mapping; > + > + svm_bo = folio_zone_device_data(head); > + folio_set_zone_device_data(tail, svm_bo); > + svm_range_bo_ref(svm_bo); > +} > + > static const struct dev_pagemap_ops svm_migrate_pgmap_ops = { > .folio_free = svm_migrate_folio_free, > .migrate_to_ram = svm_migrate_to_ram, > + .folio_split = svm_migrate_folio_split, > }; > > /* Each VRAM page uses sizeof(struct page) on system memory */ > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c > index a99dbedf3cd4..8e1fb5df8abd 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c > @@ -245,7 +245,16 @@ void svm_range_dma_unmap_dev(struct device *dev, dma_addr_t *dma_addr, > if (!svm_is_valid_dma_mapping_addr(dev, dma_addr[i])) > continue; > pr_debug_ratelimited("unmap 0x%llx\n", dma_addr[i] >> PAGE_SHIFT); > - dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); > + > + /* dma unmap of THP */ > + if (dma_addr[i] & SVM_RANGE_DMA_THP) { > + dma_addr[i] &= ~SVM_RANGE_DMA_THP; > + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE*HPAGE_PMD_NR, > + DMA_BIDIRECTIONAL); > + } else { > + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); > + } > + > dma_addr[i] = 0; > } > } > @@ -578,10 +587,15 @@ svm_range_vram_node_new(struct kfd_node *node, struct svm_range *prange, > } > > memset(&bp, 0, sizeof(bp)); > - bp.size = prange->npages * PAGE_SIZE; > + if (prange->npages >= HPAGE_PMD_NR) { > + bp.size = ALIGN(prange->npages * PAGE_SIZE, HPAGE_PMD_SIZE); > + bp.byte_align = HPAGE_PMD_SIZE; > + } else { > + bp.size = prange->npages * PAGE_SIZE; > + bp.byte_align = PAGE_SIZE; > + } seems don't need align size to 2MB, this will alloc vram that never used. For example size 3MB, byte_align to 2MB, gpu buddy allocator will return first block with 2MB, the second block 1MB. Regards, Philip > bp.bo_ptr_size = sizeof(struct svm_range_bo); > bp.destroy = svm_range_bo_destroy; > - bp.byte_align = PAGE_SIZE; > bp.domain = AMDGPU_GEM_DOMAIN_VRAM; > bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS; > bp.flags |= clear ? AMDGPU_GEM_CREATE_VRAM_CLEARED : 0; > @@ -593,9 +607,17 @@ svm_range_vram_node_new(struct kfd_node *node, struct svm_range *prange, > > r = amdgpu_bo_create(node->adev, &bp, &bo); > if (r) { > - pr_debug("failed %d to create bo\n", r); > - mmput(mm); > - goto create_bo_failed; > + if (prange->npages >= HPAGE_PMD_NR) { > + /* fall back PAGE_SIZE granularity */ > + bp.size = prange->npages * PAGE_SIZE; > + bp.byte_align = PAGE_SIZE; > + r = amdgpu_bo_create(node->adev, &bp, &bo); > + } > + if (r) { > + pr_debug("failed %d to create bo\n", r); > + mmput(mm); > + goto create_bo_failed; > + } > } > > svm_bo = to_svm_range_bo(bo); > @@ -956,6 +978,7 @@ svm_range_copy_array(void *psrc, size_t size, uint64_t num_elements, > *vram_pages = 0; > for (i = 0; i < num_elements; i++) { > dma_addr_t *temp; > + > temp = (dma_addr_t *)dst + i; > *temp = *((dma_addr_t *)src + i); > if (*temp&SVM_RANGE_VRAM_DOMAIN) > @@ -1628,7 +1651,7 @@ static int svm_range_reserve_bos(struct svm_validate_context *ctx, bool intr) > uint32_t gpuidx; > int r; > > - drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT: 0, 0); > + drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT : 0, 0); > drm_exec_until_all_locked(&ctx->exec) { > for_each_set_bit(gpuidx, ctx->bitmap, MAX_GPU_INSTANCE) { > pdd = kfd_process_device_from_gpuidx(ctx->process, gpuidx); > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h > index c7d7adae4476..f2b3a05cd8cf 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h > @@ -35,6 +35,7 @@ > #include "kfd_priv.h" > > #define SVM_RANGE_VRAM_DOMAIN (1UL << 0) > +#define SVM_RANGE_DMA_THP (1UL << 1) > #define SVM_ADEV_PGMAP_OWNER(adev)\ > ((adev)->hive ? (void *)(adev)->hive : (void *)(adev)) > [-- Attachment #2: Type: text/html, Size: 11613 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM in kfd svm driver 2026-11-13 23:28 ` Philip Yang @ 2026-11-16 15:54 ` Chen, Xiaogang 0 siblings, 0 replies; 7+ messages in thread From: Chen, Xiaogang @ 2026-11-16 15:54 UTC (permalink / raw) To: Philip Yang, amd-gfx On 11/13/2026 5:28 PM, Philip Yang wrote: > > > On 2026-10-19 09:38, Xiaogang.Chen wrote: >> From: Xiaogang Chen<xiaogang.chen@amd.com> >> >> Extend kfd/svm functions to allocate HPAGE_PMD_SIZE based device >> memory by >> buddy allocator, each drm_buddy_block is HPAGE_PMD_SIZE aligned; and to >> allocate THP system ram by vma_alloc_folio. >> >> Introduce SVM_RANGE_DMA_THP flag that indicates dma map of THP. THP dma >> addresss will use this flag. >> >> Add dev_pagemap_ops->folio_split callback that is called by >> folio_split when >> core MM splits device memory folio. >> >> Reset compound folio of device memory after it got released from core >> HMM. >> So, TTM can reuse this VRAM as 4K pages on next range. >> >> These are preparations for following support for (THP) migration of zone >> device-private memory, no function change. >> >> Signed-off-by: Xiaogang Chen<xiaogang.chen@amd.com> >> --- >> drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 84 ++++++++++++++++++++---- >> drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 37 +++++++++-- >> drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 + >> 3 files changed, 104 insertions(+), 18 deletions(-) >> >> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c >> b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c >> index 8b96e1fe741a..7a129c9954fc 100644 >> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c >> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c >> @@ -218,17 +218,45 @@ svm_migrate_addr_to_pfn(struct amdgpu_device >> *adev, unsigned long addr) >> return (addr + adev->kfd.pgmap.range.start) >> PAGE_SHIFT; >> } >> +/* >> + * TTM can reuse a 2MB VRAM block that was previously a device THP. >> + * Reset unused pages so the next 4K init does not see leftover >> head/tail. >> + */ >> +static void svm_migrate_clear_compound_pages(struct page *page, >> unsigned int nr, >> + struct dev_pagemap *pgmap) >> +{ >> + unsigned int i; >> + >> + for (i = 0; i < nr; i++, page++) { >> + struct folio *new_folio = (struct folio *)page; >> + >> + if (page_count(page)) >> + continue; >> + >> + page->flags.f &= ~0xffUL; >> +#ifdef NR_PAGES_IN_LARGE_FOLIO >> + ((struct folio *)(page - 1))->_nr_pages = 0; >> +#endif >> + new_folio->mapping = NULL; >> + new_folio->pgmap = pgmap; >> + new_folio->share = 0; >> + page->zone_device_data = NULL; >> + } >> +} >> + > This is called after the device page is freed. When migrating, > svm_migrate_get_vram_page() uses the VRAM page and > zone_device_folio_init() does the same initialization, so this looks > duplicated. Is it? svm_migrate_clear_compound_pages() runs because freeing the device THP only releases the head folio. The other 511 struct pages stay linked to that head, and TTM can hand those same VRAM pages out later as ordinary 4K pages. free_zone_device_folio() clears folio->mapping on the head and calls folio_free. It does not walk the tails. Each tail still has its compound-tail marker, so page_folio() follows it back to the old head, and page_pgmap() is implemented as page_folio(page)->pgmap. The tail's own pgmap slot held the head pointer, not the dev_pagemap. The next 4K migration initializes that exact pfn: svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn, int order) { struct page *page = pfn_to_page(pfn); /* Init this pfn, not page_folio() ? that follows a leftover tail to the old head. */ zone_device_folio_init((struct folio *)page, page_pgmap(page), order); page_pgmap() runs before that init. On a leftover tail it returns the old head's pgmap, or a stale one if that head was already reused. The clear walks every page of the freed folio, drops the head/tail flag bits and _nr_pages, and stores the real pgmap on each page. After that, each page is a standalone zone-device page and the next order-0 init does not follow the old THP. >> static void >> -svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn) >> +svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn, >> + int order) >> { >> - struct page *page; >> + struct page *page = pfn_to_page(pfn); >> - page = pfn_to_page(pfn); >> - svm_range_bo_ref(prange->svm_bo); >> + /* Init this pfn, not page_folio() — that follows a leftover >> tail to the old head. */ >> + zone_device_folio_init((struct folio *)page, page_pgmap(page), >> order); >> page->zone_device_data = prange->svm_bo; >> - zone_device_page_init(page, page_pgmap(page), 0); >> + svm_range_bo_ref(prange->svm_bo); >> } >> + >> static void >> svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long >> addr) >> { >> @@ -249,11 +277,17 @@ svm_migrate_addr(struct amdgpu_device *adev, >> struct page *page) >> } >> static struct page * >> -svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long >> addr) >> +svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long >> addr, >> + unsigned long order) >> { >> struct page *page; >> - page = alloc_page_vma(GFP_HIGHUSER, vma, addr); >> + if (order) >> + page = folio_page(vma_alloc_folio(GFP_HIGHUSER, >> + order, vma, addr), 0); > vma_alloc_folio could fail, then folio_page has NULL access, should > check vma_alloc_folio return value. if vma_alloc_folio() fails for > order 9, this fails the entire migration with -ENOMEM and the CPU > fault becomes SIGBUS. migrate_vma_pages() already handles a compound > source with a non-compound destination by splitting the source, so > falling back to a 4K page here would be enough. Where is that handled? yes, that needs fallback to alloc_page_vma if alloc sys THP fail. >> + else >> + page = alloc_page_vma(GFP_HIGHUSER, vma, addr); >> + >> if (page) >> lock_page(page); >> @@ -267,8 +301,13 @@ static unsigned long >> svm_migrate_successful_pages(struct migrate_vma *migrate) >> for (i = 0; i < migrate->npages; i++) { >> if (migrate->dst[i] & MIGRATE_PFN_VALID && >> - migrate->src[i] & MIGRATE_PFN_MIGRATE) >> - mpages++; >> + migrate->src[i] & MIGRATE_PFN_MIGRATE) { >> + >> + if (migrate->dst[i] & MIGRATE_PFN_COMPOUND) >> + mpages += HPAGE_PMD_NR; >> + else >> + mpages++; >> + } >> } >> return mpages; >> } >> @@ -302,7 +341,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, >> struct svm_range *prange, >> if (migrate->src[i] & MIGRATE_PFN_MIGRATE) { >> dst[i] = cursor.start + (j << PAGE_SHIFT); >> migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); >> - svm_migrate_get_vram_page(prange, migrate->dst[i]); >> + svm_migrate_get_vram_page(prange, migrate->dst[i], 0); >> migrate->dst[i] = migrate_pfn(migrate->dst[i]); >> mpages++; >> } >> @@ -616,15 +655,22 @@ svm_migrate_ram_to_vram(struct svm_range >> *prange, uint32_t best_loc, >> return r < 0 ? r : 0; >> } >> +/* folio can be compound folio or single page */ >> static void svm_migrate_folio_free(struct folio *folio) >> { >> struct page *page = &folio->page; >> struct svm_range_bo *svm_bo = page->zone_device_data; >> + struct dev_pagemap *pgmap = folio->pgmap; >> + unsigned int order = folio_order(folio); >> if (svm_bo) { >> pr_debug_ratelimited("ref: %d\n", kref_read(&svm_bo->kref)); >> svm_range_bo_unref_async(svm_bo); >> } >> + >> + /* So TTM can reuse this VRAM as 4K pages on the next range. */ >> + if (order) >> + svm_migrate_clear_compound_pages(page, 1U << order, pgmap); >> } >> static int >> @@ -678,7 +724,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device >> *adev, struct svm_range *prange, >> j = 0; >> } >> - dpage = svm_migrate_get_sys_page(migrate->vma, addr); >> + dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); >> if (!dpage) { >> pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n", >> prange->svms, prange->start, prange->last); >> @@ -1081,9 +1127,25 @@ static vm_fault_t svm_migrate_to_ram(struct >> vm_fault *vmf) >> return r ? VM_FAULT_SIGBUS : 0; >> } >> +static void svm_migrate_folio_split(struct folio *head, struct >> folio *tail) >> +{ >> + struct svm_range_bo *svm_bo; >> + >> + if (tail == NULL) >> + return; >> + >> + tail->pgmap = head->pgmap; >> + tail->mapping = head->mapping; >> + >> + svm_bo = folio_zone_device_data(head); >> + folio_set_zone_device_data(tail, svm_bo); >> + svm_range_bo_ref(svm_bo); >> +} >> + >> static const struct dev_pagemap_ops svm_migrate_pgmap_ops = { >> .folio_free = svm_migrate_folio_free, >> .migrate_to_ram = svm_migrate_to_ram, >> + .folio_split = svm_migrate_folio_split, >> }; >> /* Each VRAM page uses sizeof(struct page) on system memory */ >> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c >> b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c >> index a99dbedf3cd4..8e1fb5df8abd 100644 >> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c >> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c >> @@ -245,7 +245,16 @@ void svm_range_dma_unmap_dev(struct device *dev, >> dma_addr_t *dma_addr, >> if (!svm_is_valid_dma_mapping_addr(dev, dma_addr[i])) >> continue; >> pr_debug_ratelimited("unmap 0x%llx\n", dma_addr[i] >> >> PAGE_SHIFT); >> - dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); >> + >> + /* dma unmap of THP */ >> + if (dma_addr[i] & SVM_RANGE_DMA_THP) { >> + dma_addr[i] &= ~SVM_RANGE_DMA_THP; >> + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE*HPAGE_PMD_NR, >> + DMA_BIDIRECTIONAL); >> + } else { >> + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); >> + } >> + >> dma_addr[i] = 0; >> } >> } >> @@ -578,10 +587,15 @@ svm_range_vram_node_new(struct kfd_node *node, >> struct svm_range *prange, >> } >> memset(&bp, 0, sizeof(bp)); >> - bp.size = prange->npages * PAGE_SIZE; >> + if (prange->npages >= HPAGE_PMD_NR) { >> + bp.size = ALIGN(prange->npages * PAGE_SIZE, HPAGE_PMD_SIZE); >> + bp.byte_align = HPAGE_PMD_SIZE; >> + } else { >> + bp.size = prange->npages * PAGE_SIZE; >> + bp.byte_align = PAGE_SIZE; >> + } > seems don't need align size to 2MB, this will alloc vram that never > used. For example size 3MB, byte_align to 2MB, gpu buddy allocator > will return first block with 2MB, the second block 1MB The fallback is align to PAGA_SIZE. Do you mean the case above? That is aligned to 2MB. Regards Xiaogang > . Regards, Philip >> bp.bo_ptr_size = sizeof(struct svm_range_bo); >> bp.destroy = svm_range_bo_destroy; >> - bp.byte_align = PAGE_SIZE; >> bp.domain = AMDGPU_GEM_DOMAIN_VRAM; >> bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS; >> bp.flags |= clear ? AMDGPU_GEM_CREATE_VRAM_CLEARED : 0; >> @@ -593,9 +607,17 @@ svm_range_vram_node_new(struct kfd_node *node, >> struct svm_range *prange, >> r = amdgpu_bo_create(node->adev, &bp, &bo); >> if (r) { >> - pr_debug("failed %d to create bo\n", r); >> - mmput(mm); >> - goto create_bo_failed; >> + if (prange->npages >= HPAGE_PMD_NR) { >> + /* fall back PAGE_SIZE granularity */ >> + bp.size = prange->npages * PAGE_SIZE; >> + bp.byte_align = PAGE_SIZE; >> + r = amdgpu_bo_create(node->adev, &bp, &bo); >> + } >> + if (r) { >> + pr_debug("failed %d to create bo\n", r); >> + mmput(mm); >> + goto create_bo_failed; >> + } >> } >> svm_bo = to_svm_range_bo(bo); >> @@ -956,6 +978,7 @@ svm_range_copy_array(void *psrc, size_t size, >> uint64_t num_elements, >> *vram_pages = 0; >> for (i = 0; i < num_elements; i++) { >> dma_addr_t *temp; >> + >> temp = (dma_addr_t *)dst + i; >> *temp = *((dma_addr_t *)src + i); >> if (*temp&SVM_RANGE_VRAM_DOMAIN) >> @@ -1628,7 +1651,7 @@ static int svm_range_reserve_bos(struct >> svm_validate_context *ctx, bool intr) >> uint32_t gpuidx; >> int r; >> - drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT: >> 0, 0); >> + drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT : >> 0, 0); >> drm_exec_until_all_locked(&ctx->exec) { >> for_each_set_bit(gpuidx, ctx->bitmap, MAX_GPU_INSTANCE) { >> pdd = kfd_process_device_from_gpuidx(ctx->process, >> gpuidx); >> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h >> b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h >> index c7d7adae4476..f2b3a05cd8cf 100644 >> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h >> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h >> @@ -35,6 +35,7 @@ >> #include "kfd_priv.h" >> #define SVM_RANGE_VRAM_DOMAIN (1UL << 0) >> +#define SVM_RANGE_DMA_THP (1UL << 1) >> #define SVM_ADEV_PGMAP_OWNER(adev)\ >> ((adev)->hive ? (void *)(adev)->hive : (void *)(adev)) > ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v3 2/3] drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault handler 2026-10-19 13:38 [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver Xiaogang.Chen 2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen @ 2026-10-19 13:38 ` Xiaogang.Chen 2026-10-19 13:38 ` [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver Xiaogang.Chen 2 siblings, 0 replies; 7+ messages in thread From: Xiaogang.Chen @ 2026-10-19 13:38 UTC (permalink / raw) To: amd-gfx; +Cc: Xiaogang Chen From: Xiaogang Chen <xiaogang.chen@amd.com> After support device THP the page fault(either CPU or GPU) address can point to THP(device or anonymous THP). If the correspondent svm prange covers the whole compound folio migrate at folio order so HMM can collect MIGRATE_PFN_COMPOUND. Do not migrate partial THP memory. If do HMM will split THP folio. If the compound folio has extra ref/pin the split will fail, then skip migration, leave without page restore, then cpu/gpu keep retrying. If the prange was split and no longer covers the PMD, do not call split_folio(): it is not exported to modules. Walk only this prange and let migrate_vma_setup() split the PMD inside mm. If the fault address has PTE mapping memory use prange->granularity to decide migration size. Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com> --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 39 ++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 82 +++++++++++++++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 12 ++++ 3 files changed, 130 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 7a129c9954fc..2febe3bb690e 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -1059,6 +1059,8 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) struct svm_range *prange; struct kfd_process *p; struct mm_struct *mm; + struct folio *folio; + unsigned int order; int r = 0; svm_bo = vmf->page->zone_device_data; @@ -1104,8 +1106,41 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) if (!prange->actual_loc) goto out_unlock_prange; - /* Align migration range start and size to granularity size */ - size = 1UL << prange->granularity; + folio = page_folio(vmf->page); + order = folio_order(folio); + + /* + * A PMD device-private fault always passes the folio head in + * vmf->page. If this prange still covers the whole folio, migrate + * at folio order so HMM can collect MIGRATE_PFN_COMPOUND. + * + * Do not want migrate partial THP device memory. If do HMM will split + * THP folio. If the compound folio has extra ref/pin the split will fail, + * then skip migration, leave without page restore, then cpu keep retry. + * + * If the prange was split and no longer covers the PMD, do not call + * split_folio() here: it is not exported to modules. Walk only this + * prange and let migrate_vma_setup() split the PMD inside mm. + * + * If the fault address has PTE mapping device-private memory use + * prange->granularity to decide migration size. + */ + if (order && !vmf->pte) { + unsigned long folio_start = ALIGN_DOWN(addr, 1UL << order); + unsigned long folio_last = folio_start + (1UL << order) - 1; + + if (prange->start > folio_start || prange->last < folio_last) { + pr_debug("partial prange on device THP svms 0x%p prange [0x%lx 0x%lx] folio [0x%lx 0x%lx]\n", + prange->svms, prange->start, prange->last, + folio_start, folio_last); + size = 1UL << prange->granularity; + } else { + size = 1UL << max(prange->granularity, order); + } + } else { + size = 1UL << prange->granularity; + } + start = max(ALIGN_DOWN(addr, size), prange->start); last = min(ALIGN(addr + 1, size) - 1, prange->last); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 8e1fb5df8abd..7443beb5778c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -38,6 +38,8 @@ #include "kfd_svm.h" #include "kfd_migrate.h" #include "kfd_smi_events.h" +#include <linux/mm.h> +#include <linux/leafops.h> #ifdef dev_fmt #undef dev_fmt @@ -3078,6 +3080,53 @@ svm_fault_allowed(struct vm_area_struct *vma, bool write_fault) return (vma->vm_flags & requested) == requested; } +/* + * svm_range_cpu_huge_folio_order - folio order of a PMD-mapped CPU mapping + * @vma: VMA covering @va + * @va: CPU virtual address of the GPU retry fault + * + * GPU retry faults have no vmf->page. Walk to the PMD (mmap read lock is + * already held) and return the folio order when that mapping is a huge PMD: + * anonymous THP or a device-private huge PMD. PTE-mapped pages return 0 so + * the caller keeps prange->granularity, matching svm_migrate_to_ram() which + * only expands on !vmf->pte. + * + * Return: folio order, or 0 if the PMD is not a huge mapping + */ +static unsigned int +svm_range_cpu_huge_folio_order(struct vm_area_struct *vma, unsigned long va) +{ + struct mm_struct *mm = vma->vm_mm; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmdp; + pmd_t pmd; + + pgd = pgd_offset(mm, va); + if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) + return 0; + + p4d = p4d_offset(pgd, va); + if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) + return 0; + + pud = pud_offset(p4d, va); + if (pud_none(*pud) || unlikely(pud_bad(*pud))) + return 0; + + pmdp = pmd_offset(pud, va); + pmd = pmdp_get_lockless(pmdp); + + if (pmd_trans_huge(pmd)) + return folio_order(pmd_folio(pmd)); + + if (pmd_is_device_private_entry(pmd)) + return folio_order(softleaf_to_folio(softleaf_from_pmd(pmd))); + + return 0; +} + int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint32_t vmid, uint32_t node_id, @@ -3246,8 +3295,39 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, kfd_smi_event_page_fault_start(node, p->lead_thread, addr, write_fault, timestamp); - /* Align migration range start and size to granularity size */ + /* + * Same policy as svm_migrate_to_ram(): if the CPU maps a device or + * anonymous THP at this address and this prange still covers the + * whole folio, migrate at folio order so HMM can collect + * MIGRATE_PFN_COMPOUND. A split prange that not convers the THP keeps + * using granularity. migrate_vma_setup() splits the PMD inside mm. + * + * GPU faults have no vmf->page, so inspect the PMD. + * + * If the fault address has PTE mapping use prange->granularity to decide + * migration size. + * + */ size = 1UL << prange->granularity; + if (svm_is_private_zone(adev)) { + unsigned int order; + + order = svm_range_cpu_huge_folio_order(vma, addr << PAGE_SHIFT); + if (order) { + unsigned long folio_start = ALIGN_DOWN(addr, 1UL << order); + unsigned long folio_last = folio_start + (1UL << order) - 1; + + if (prange->start > folio_start || + prange->last < folio_last) { + pr_debug("partial prange on CPU THP svms 0x%p prange [0x%lx 0x%lx] folio [0x%lx 0x%lx]\n", + svms, prange->start, prange->last, + folio_start, folio_last); + } else { + size = 1UL << max_t(unsigned int, + prange->granularity, order); + } + } + } start = max_t(unsigned long, ALIGN_DOWN(addr, size), prange->start); last = min_t(unsigned long, ALIGN(addr + 1, size) - 1, prange->last); if (prange->actual_loc != 0 || best_loc != 0) { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index f2b3a05cd8cf..d6862a80a433 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -214,6 +214,13 @@ void svm_range_bo_unref_async(struct svm_range_bo *svm_bo); void svm_range_set_max_pages(struct amdgpu_device *adev); int svm_range_switch_xnack_reserve_mem(struct kfd_process *p, bool xnack_enabled); +/* check adev has device private zone memory */ +static inline bool svm_is_private_zone(struct amdgpu_device *adev) +{ + struct amdgpu_kfd_dev *kfddev = &adev->kfd; + + return (kfddev->pgmap.type == MEMORY_DEVICE_PRIVATE); +} #else @@ -277,6 +284,11 @@ static inline void svm_range_set_max_pages(struct amdgpu_device *adev) { } +static inline bool svm_is_private_zone(struct amdgpu_device *adev) +{ + return false; +} + #define KFD_IS_SVM_API_SUPPORTED(dev) false #endif /* IS_ENABLED(CONFIG_HSA_AMD_SVM) */ -- 2.34.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver 2026-10-19 13:38 [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver Xiaogang.Chen 2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen 2026-10-19 13:38 ` [PATCH v3 2/3] drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault handler Xiaogang.Chen @ 2026-10-19 13:38 ` Xiaogang.Chen 2026-11-13 19:29 ` Philip Yang 2 siblings, 1 reply; 7+ messages in thread From: Xiaogang.Chen @ 2026-10-19 13:38 UTC (permalink / raw) To: amd-gfx; +Cc: Xiaogang Chen From: Xiaogang Chen <xiaogang.chen@amd.com> HMM migrates a PMD-mapped THP as one compound folio when migrate_vma_setup() is passed MIGRATE_VMA_SELECT_COMPOUND. Without that flag the collect path splits the source THP to 4K pages. If split_folio() fails because of extra refs or pins, those pages are skipped. Set SELECT_COMPOUND for device-private ranges whose migrate window is at least HPAGE_PMD_NR pages. The RAM<->VRAM copy path follows MIGRATE_PFN_COMPOUND in src[] (HPAGE_PMD_ORDER device folio, THP DMA) and keeps the existing 4K path otherwise. Coherent zone-device memory is unchanged. Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com> --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 253 ++++++++++++++++++----- 1 file changed, 203 insertions(+), 50 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 2febe3bb690e..5c98028bfe73 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -324,7 +324,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, u64 mpages = 0; dma_addr_t *src; u64 *dst; - u64 i, j; + u64 i, j, k, l, m; int r = 0; pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start, @@ -335,59 +335,155 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, amdgpu_res_first(prange->ttm_res, ttm_res_offset, npages << PAGE_SHIFT, &cursor); - for (i = j = 0; (i < npages) && (mpages < migrate->cpages); i++) { + for (i = j = m = 0; (i < npages) && (mpages < migrate->cpages);) { struct page *spage; + unsigned long cur_dst_pfn; - if (migrate->src[i] & MIGRATE_PFN_MIGRATE) { - dst[i] = cursor.start + (j << PAGE_SHIFT); - migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); - svm_migrate_get_vram_page(prange, migrate->dst[i], 0); - migrate->dst[i] = migrate_pfn(migrate->dst[i]); + cur_dst_pfn = svm_migrate_addr_to_pfn(adev, + cursor.start + (m << PAGE_SHIFT)); + + /* when migrate->src[i] has MIGRATE_PFN_COMPOUND set the src page + * is compound THP; its vm address is HPAGE_PMD_SIZE aligned and + * its MIGRATE_PFN_MIGRATE is set + */ + if ((m + HPAGE_PMD_NR) <= (cursor.size >> PAGE_SHIFT) && + (i + HPAGE_PMD_NR) <= npages && + (migrate->src[i] & MIGRATE_PFN_COMPOUND) && + IS_ALIGNED(cur_dst_pfn, HPAGE_PMD_NR)) { + k = HPAGE_PMD_NR; + } else + k = 1; + + /* for THP src[0] alwas MIGRATE_PFN_MIGRATE + * just the first migrate->dst need be setup, others are zero + */ + if (k > 1) { + + dst[i] = cursor.start + (m << PAGE_SHIFT); + svm_migrate_get_vram_page(prange, cur_dst_pfn, + HPAGE_PMD_ORDER); + + migrate->dst[i] = migrate_pfn(cur_dst_pfn); + migrate->dst[i] |= MIGRATE_PFN_COMPOUND; + + for (l = 1; l < k; l++) + migrate->dst[i+l] = 0; + + mpages++; + + } else if ((migrate->src[i] & MIGRATE_PFN_MIGRATE)) { + dst[i] = cursor.start + (m << PAGE_SHIFT); + svm_migrate_get_vram_page(prange, cur_dst_pfn, 0); + migrate->dst[i] = migrate_pfn(cur_dst_pfn); mpages++; } - spage = migrate_pfn_to_page(migrate->src[i]); - if (spage && !is_zone_device_page(spage)) { - src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, - DMA_BIDIRECTIONAL); - r = dma_mapping_error(dev, src[i]); - if (r) { - src[i] = 0; - dev_err(dev, "%s: fail %d dma_map_page\n", - __func__, r); - goto out_free_vram_pages; + + if (k > 1) { + if (j) { + /* migrate previous accumulated pages */ + r = svm_migrate_copy_memory_gart( + adev, src + i - j, + dst + i - j, j, + FROM_RAM_TO_VRAM, + mfence); + + if (r) + goto out_free_vram_pages; + + j = 0; + } + + /* for THP check if the first src page is valid + * if not valid skip following HPAGE_PMD_NR - 1 pages + */ + spage = migrate_pfn_to_page(migrate->src[i]); + if (spage && !is_zone_device_page(spage)) { + /* dma_map continuous HPAGE_PMD_NR sys ram pages */ + src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE*HPAGE_PMD_NR, + DMA_BIDIRECTIONAL); + + r = dma_mapping_error(dev, src[i]); + if (r) { + dev_err(dev, "%s: fail %d dma_map_page\n", + __func__, r); + goto out_free_vram_pages; + } + + /* get dma address for following HPAGE_PMD_NR-1 pages + * since src pages are continuous their dma addresses + * are continuous too. + */ + for (l = 1; l < k; l++) + src[i + l] = src[i] + l*PAGE_SIZE; + + /* migrate the HPAGE_PMD_NR pages above */ + r = svm_migrate_copy_memory_gart( + adev, src + i, + dst + i, HPAGE_PMD_NR, + FROM_RAM_TO_VRAM, + mfence); + + /* mark head page dma mapping as THP, tail pages dma addr + * are set to 0 for following dma_unmap + */ + src[i] |= SVM_RANGE_DMA_THP; + for (l = 1; l < k; l++) + src[i + l] = 0; + + if (r) + goto out_free_vram_pages; + + j = 0; } } else { - if (j) { + /* single normal page case */ + spage = migrate_pfn_to_page(migrate->src[i]); + if (spage && !is_zone_device_page(spage)) { + src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, + DMA_BIDIRECTIONAL); + + r = dma_mapping_error(dev, src[i]); + + if (r) { + dev_err(dev, "%s: fail %d dma_map_page\n", + __func__, r); + goto out_free_vram_pages; + } + j += 1; + + } else if (j) { r = svm_migrate_copy_memory_gart( adev, src + i - j, dst + i - j, j, FROM_RAM_TO_VRAM, mfence); + if (r) goto out_free_vram_pages; - amdgpu_res_next(&cursor, (j + 1) << PAGE_SHIFT); + j = 0; - } else { - amdgpu_res_next(&cursor, PAGE_SIZE); } - continue; } - pr_debug_ratelimited("dma mapping src to 0x%llx, pfn 0x%lx\n", - src[i] >> PAGE_SHIFT, page_to_pfn(spage)); + pr_debug_ratelimited("dma mapping %lld pages, src to 0x%llx, pfn 0x%lx\n", + k, src[i] >> PAGE_SHIFT, migrate->src[i] >> MIGRATE_PFN_SHIFT); + i += k; + m += k; + + if (m >= (cursor.size >> PAGE_SHIFT)) { + if (j > 0) { + r = svm_migrate_copy_memory_gart(adev, src + i - j, + dst + i - j, j, + FROM_RAM_TO_VRAM, + mfence); + if (r) + goto out_free_vram_pages; + } + + amdgpu_res_next(&cursor, m*PAGE_SIZE); - /* accumulated j + 1 pages reach end of current drm_buddy_block */ - if (j + 1 >= (cursor.size >> PAGE_SHIFT)) { - r = svm_migrate_copy_memory_gart(adev, src + i - j, - dst + i - j, j + 1, - FROM_RAM_TO_VRAM, - mfence); - if (r) - goto out_free_vram_pages; - amdgpu_res_next(&cursor, (j + 1) * PAGE_SIZE); j = 0; - } else { - j++; + m = 0; } } @@ -476,6 +572,7 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange, struct kfd_process_device *pdd; struct dma_fence *mfence = NULL; struct migrate_vma migrate = { 0 }; + bool is_private_device = false; unsigned long cpages = 0; unsigned long mpages = 0; dma_addr_t *scratch; @@ -488,11 +585,17 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange, if (svm_range_window_resident(node, prange, start, end)) return 0; + is_private_device = svm_is_private_zone(adev); + memset(&migrate, 0, sizeof(migrate)); migrate.vma = vma; migrate.start = start; migrate.end = end; migrate.flags = MIGRATE_VMA_SELECT_SYSTEM; + + if (is_private_device && ((end - start) >> PAGE_SHIFT) >= HPAGE_PMD_NR) + migrate.flags = migrate.flags | MIGRATE_VMA_SELECT_COMPOUND; + migrate.pgmap_owner = SVM_ADEV_PGMAP_OWNER(adev); buf = kvcalloc(npages, @@ -686,6 +789,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, u64 i = 0, j; u64 addr; int r = 0; + u64 l, k; pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); @@ -694,8 +798,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, src = (u64 *)(scratch + npages); dst = scratch; - - for (i = 0, j = 0; i < npages; i++, addr += PAGE_SIZE) { + for (i = 0, j = 0, k = 1; i < npages; i += k, addr += k*PAGE_SIZE, k = 1) { struct page *spage; spage = migrate_pfn_to_page(migrate->src[i]); @@ -711,6 +814,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, goto out_oom; j = 0; } + continue; } src[i] = svm_migrate_addr(adev, spage); @@ -724,7 +828,19 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, j = 0; } - dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); + if (IS_ALIGNED(page_to_pfn(spage), HPAGE_PMD_NR) && + (addr + HPAGE_PMD_SIZE) <= migrate->end && + IS_ALIGNED(addr, HPAGE_PMD_SIZE) && + migrate->src[i] & MIGRATE_PFN_COMPOUND) { + + k = HPAGE_PMD_NR; + dpage = svm_migrate_get_sys_page(migrate->vma, addr, + HPAGE_PMD_ORDER); + } else { + k = 1; + dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); + } + if (!dpage) { pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); @@ -732,7 +848,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, goto out_oom; } - dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE, DMA_BIDIRECTIONAL); + dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE*k, DMA_BIDIRECTIONAL); r = dma_mapping_error(dev, dst[i]); if (r) { dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r); @@ -740,13 +856,47 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, goto out_oom; } - pr_debug_ratelimited("dma mapping dst to 0x%llx, pfn 0x%lx\n", - dst[i] >> PAGE_SHIFT, page_to_pfn(dpage)); - migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); - dpage = NULL; - j++; + if (k > 1) { + migrate->dst[i] |= MIGRATE_PFN_COMPOUND; + /* migrate previous accumulated pages */ + if (j) { + r = svm_migrate_copy_memory_gart(adev, dst + i - j, + src + i - j, j, FROM_VRAM_TO_RAM, mfence); + if (r) + goto out_oom; + j = 0; + } + + for (l = 1; l < k; l++) { + + src[i + l] = src[i] + l*PAGE_SIZE; + dst[i + l] = dst[i] + l*PAGE_SIZE; + migrate->dst[i + l] = 0; + } + + /* migrate the HPAGE_PMD_NR pages above + * the migration is for 2MB THP + */ + r = svm_migrate_copy_memory_gart( + adev, dst + i, + src + i, HPAGE_PMD_NR, + FROM_VRAM_TO_RAM, + mfence); + + /* mark head page dma mapping as THP, tail pages dma addr + * are set to 0 for following dma_unmap + */ + dst[i] |= SVM_RANGE_DMA_THP; + for (l = 1; l < k; l++) + dst[i + l] = 0; + + if (r) + goto out_oom; + + } else + j++; } if (j > 0) @@ -765,12 +915,9 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, /* release previous allocated sys pages and unmap dma address */ while (i--) { - if (dst[i]) { - dma_unmap_page(dev, dst[i], PAGE_SIZE, - DMA_BIDIRECTIONAL); - dst[i] = 0; - } - + /* follwing svm_range_dma_unmap_dev will do dma unmap anyway + * not need do dma unmap here + */ dpage = migrate_pfn_to_page(migrate->dst[i]); if (!dpage) continue; @@ -811,6 +958,7 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange, unsigned long cpages = 0; unsigned long mpages = 0; struct amdgpu_device *adev = node->adev; + bool is_private_device = false; struct kfd_process_device *pdd; struct dma_fence *mfence = NULL; struct migrate_vma migrate = { 0 }; @@ -818,6 +966,8 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange, void *buf; int r = -ENOMEM; + is_private_device = svm_is_private_zone(adev); + memset(&migrate, 0, sizeof(migrate)); migrate.vma = vma; migrate.start = start; @@ -828,6 +978,9 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange, else migrate.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE; + if (is_private_device && ((end - start) >> PAGE_SHIFT) >= HPAGE_PMD_NR) + migrate.flags = migrate.flags | MIGRATE_VMA_SELECT_COMPOUND; + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(u64) + sizeof(dma_addr_t), GFP_KERNEL); -- 2.34.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver 2026-10-19 13:38 ` [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver Xiaogang.Chen @ 2026-11-13 19:29 ` Philip Yang 0 siblings, 0 replies; 7+ messages in thread From: Philip Yang @ 2026-11-13 19:29 UTC (permalink / raw) To: amd-gfx [-- Attachment #1.1: Type: text/plain, Size: 13713 bytes --] On 2026-10-19 09:38, Xiaogang.Chen wrote: > From: Xiaogang Chen<xiaogang.chen@amd.com> > > HMM migrates a PMD-mapped THP as one compound folio when > migrate_vma_setup() is passed MIGRATE_VMA_SELECT_COMPOUND. > > Without that flag the collect path splits the source THP to 4K pages. > If split_folio() fails because of extra refs or pins, those pages are > skipped. > > Set SELECT_COMPOUND for device-private ranges whose migrate window is > at least HPAGE_PMD_NR pages. The RAM<->VRAM copy path follows > MIGRATE_PFN_COMPOUND in src[] (HPAGE_PMD_ORDER device folio, THP DMA) > and keeps the existing 4K path otherwise. > > Coherent zone-device memory is unchanged. The approach looks right - aligning the migrate window to the folio to handle both MIGRATE_PFN_COMPOUND THP migration and normal page migration. I am attaching two cleanup patches on top of your patch, "drm/amdkfd: Simplify svm_migrate_copy_to_vram()" and "drm/amdkfd: Simplify svm_migrate_copy_to_ram()". They make no functional change, just making the code nicer, cleaner and simpler. Please see the commit messages for the details of each patch, then squash, rework or drop them as you prefer. Regards, Philip > > Signed-off-by: Xiaogang Chen<xiaogang.chen@amd.com> > --- > drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 253 ++++++++++++++++++----- > 1 file changed, 203 insertions(+), 50 deletions(-) > > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c > index 2febe3bb690e..5c98028bfe73 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c > @@ -324,7 +324,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, > u64 mpages = 0; > dma_addr_t *src; > u64 *dst; > - u64 i, j; > + u64 i, j, k, l, m; > int r = 0; > > pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start, > @@ -335,59 +335,155 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, > > amdgpu_res_first(prange->ttm_res, ttm_res_offset, > npages << PAGE_SHIFT, &cursor); > - for (i = j = 0; (i < npages) && (mpages < migrate->cpages); i++) { > + for (i = j = m = 0; (i < npages) && (mpages < migrate->cpages);) { > struct page *spage; > + unsigned long cur_dst_pfn; > > - if (migrate->src[i] & MIGRATE_PFN_MIGRATE) { > - dst[i] = cursor.start + (j << PAGE_SHIFT); > - migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); > - svm_migrate_get_vram_page(prange, migrate->dst[i], 0); > - migrate->dst[i] = migrate_pfn(migrate->dst[i]); > + cur_dst_pfn = svm_migrate_addr_to_pfn(adev, > + cursor.start + (m << PAGE_SHIFT)); > + > + /* when migrate->src[i] has MIGRATE_PFN_COMPOUND set the src page > + * is compound THP; its vm address is HPAGE_PMD_SIZE aligned and > + * its MIGRATE_PFN_MIGRATE is set > + */ > + if ((m + HPAGE_PMD_NR) <= (cursor.size >> PAGE_SHIFT) && > + (i + HPAGE_PMD_NR) <= npages && > + (migrate->src[i] & MIGRATE_PFN_COMPOUND) && > + IS_ALIGNED(cur_dst_pfn, HPAGE_PMD_NR)) { > + k = HPAGE_PMD_NR; > + } else > + k = 1; > + > + /* for THP src[0] alwas MIGRATE_PFN_MIGRATE > + * just the first migrate->dst need be setup, others are zero > + */ > + if (k > 1) { > + > + dst[i] = cursor.start + (m << PAGE_SHIFT); > + svm_migrate_get_vram_page(prange, cur_dst_pfn, > + HPAGE_PMD_ORDER); > + > + migrate->dst[i] = migrate_pfn(cur_dst_pfn); > + migrate->dst[i] |= MIGRATE_PFN_COMPOUND; > + > + for (l = 1; l < k; l++) > + migrate->dst[i+l] = 0; > + > + mpages++; > + > + } else if ((migrate->src[i] & MIGRATE_PFN_MIGRATE)) { > + dst[i] = cursor.start + (m << PAGE_SHIFT); > + svm_migrate_get_vram_page(prange, cur_dst_pfn, 0); > + migrate->dst[i] = migrate_pfn(cur_dst_pfn); > mpages++; > } > - spage = migrate_pfn_to_page(migrate->src[i]); > - if (spage && !is_zone_device_page(spage)) { > - src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, > - DMA_BIDIRECTIONAL); > - r = dma_mapping_error(dev, src[i]); > - if (r) { > - src[i] = 0; > - dev_err(dev, "%s: fail %d dma_map_page\n", > - __func__, r); > - goto out_free_vram_pages; > + > + if (k > 1) { > + if (j) { > + /* migrate previous accumulated pages */ > + r = svm_migrate_copy_memory_gart( > + adev, src + i - j, > + dst + i - j, j, > + FROM_RAM_TO_VRAM, > + mfence); > + > + if (r) > + goto out_free_vram_pages; > + > + j = 0; > + } > + > + /* for THP check if the first src page is valid > + * if not valid skip following HPAGE_PMD_NR - 1 pages > + */ > + spage = migrate_pfn_to_page(migrate->src[i]); > + if (spage && !is_zone_device_page(spage)) { > + /* dma_map continuous HPAGE_PMD_NR sys ram pages */ > + src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE*HPAGE_PMD_NR, > + DMA_BIDIRECTIONAL); > + > + r = dma_mapping_error(dev, src[i]); > + if (r) { > + dev_err(dev, "%s: fail %d dma_map_page\n", > + __func__, r); > + goto out_free_vram_pages; > + } > + > + /* get dma address for following HPAGE_PMD_NR-1 pages > + * since src pages are continuous their dma addresses > + * are continuous too. > + */ > + for (l = 1; l < k; l++) > + src[i + l] = src[i] + l*PAGE_SIZE; > + > + /* migrate the HPAGE_PMD_NR pages above */ > + r = svm_migrate_copy_memory_gart( > + adev, src + i, > + dst + i, HPAGE_PMD_NR, > + FROM_RAM_TO_VRAM, > + mfence); > + > + /* mark head page dma mapping as THP, tail pages dma addr > + * are set to 0 for following dma_unmap > + */ > + src[i] |= SVM_RANGE_DMA_THP; > + for (l = 1; l < k; l++) > + src[i + l] = 0; > + > + if (r) > + goto out_free_vram_pages; > + > + j = 0; > } > } else { > - if (j) { > + /* single normal page case */ > + spage = migrate_pfn_to_page(migrate->src[i]); > + if (spage && !is_zone_device_page(spage)) { > + src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, > + DMA_BIDIRECTIONAL); > + > + r = dma_mapping_error(dev, src[i]); > + > + if (r) { > + dev_err(dev, "%s: fail %d dma_map_page\n", > + __func__, r); > + goto out_free_vram_pages; > + } > + j += 1; > + > + } else if (j) { > r = svm_migrate_copy_memory_gart( > adev, src + i - j, > dst + i - j, j, > FROM_RAM_TO_VRAM, > mfence); > + > if (r) > goto out_free_vram_pages; > - amdgpu_res_next(&cursor, (j + 1) << PAGE_SHIFT); > + > j = 0; > - } else { > - amdgpu_res_next(&cursor, PAGE_SIZE); > } > - continue; > } > > - pr_debug_ratelimited("dma mapping src to 0x%llx, pfn 0x%lx\n", > - src[i] >> PAGE_SHIFT, page_to_pfn(spage)); > + pr_debug_ratelimited("dma mapping %lld pages, src to 0x%llx, pfn 0x%lx\n", > + k, src[i] >> PAGE_SHIFT, migrate->src[i] >> MIGRATE_PFN_SHIFT); > + i += k; > + m += k; > + > + if (m >= (cursor.size >> PAGE_SHIFT)) { > + if (j > 0) { > + r = svm_migrate_copy_memory_gart(adev, src + i - j, > + dst + i - j, j, > + FROM_RAM_TO_VRAM, > + mfence); > + if (r) > + goto out_free_vram_pages; > + } > + > + amdgpu_res_next(&cursor, m*PAGE_SIZE); > > - /* accumulated j + 1 pages reach end of current drm_buddy_block */ > - if (j + 1 >= (cursor.size >> PAGE_SHIFT)) { > - r = svm_migrate_copy_memory_gart(adev, src + i - j, > - dst + i - j, j + 1, > - FROM_RAM_TO_VRAM, > - mfence); > - if (r) > - goto out_free_vram_pages; > - amdgpu_res_next(&cursor, (j + 1) * PAGE_SIZE); > j = 0; > - } else { > - j++; > + m = 0; > } > } > > @@ -476,6 +572,7 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange, > struct kfd_process_device *pdd; > struct dma_fence *mfence = NULL; > struct migrate_vma migrate = { 0 }; > + bool is_private_device = false; > unsigned long cpages = 0; > unsigned long mpages = 0; > dma_addr_t *scratch; > @@ -488,11 +585,17 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange, > if (svm_range_window_resident(node, prange, start, end)) > return 0; > > + is_private_device = svm_is_private_zone(adev); > + > memset(&migrate, 0, sizeof(migrate)); > migrate.vma = vma; > migrate.start = start; > migrate.end = end; > migrate.flags = MIGRATE_VMA_SELECT_SYSTEM; > + > + if (is_private_device && ((end - start) >> PAGE_SHIFT) >= HPAGE_PMD_NR) > + migrate.flags = migrate.flags | MIGRATE_VMA_SELECT_COMPOUND; > + > migrate.pgmap_owner = SVM_ADEV_PGMAP_OWNER(adev); > > buf = kvcalloc(npages, > @@ -686,6 +789,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > u64 i = 0, j; > u64 addr; > int r = 0; > + u64 l, k; > > pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, > prange->last); > @@ -694,8 +798,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > > src = (u64 *)(scratch + npages); > dst = scratch; > - > - for (i = 0, j = 0; i < npages; i++, addr += PAGE_SIZE) { > + for (i = 0, j = 0, k = 1; i < npages; i += k, addr += k*PAGE_SIZE, k = 1) { > struct page *spage; > > spage = migrate_pfn_to_page(migrate->src[i]); > @@ -711,6 +814,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > goto out_oom; > j = 0; > } > + > continue; > } > src[i] = svm_migrate_addr(adev, spage); > @@ -724,7 +828,19 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > j = 0; > } > > - dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); > + if (IS_ALIGNED(page_to_pfn(spage), HPAGE_PMD_NR) && > + (addr + HPAGE_PMD_SIZE) <= migrate->end && > + IS_ALIGNED(addr, HPAGE_PMD_SIZE) && > + migrate->src[i] & MIGRATE_PFN_COMPOUND) { > + > + k = HPAGE_PMD_NR; > + dpage = svm_migrate_get_sys_page(migrate->vma, addr, > + HPAGE_PMD_ORDER); > + } else { > + k = 1; > + dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); > + } > + > if (!dpage) { > pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n", > prange->svms, prange->start, prange->last); > @@ -732,7 +848,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > goto out_oom; > } > > - dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE, DMA_BIDIRECTIONAL); > + dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE*k, DMA_BIDIRECTIONAL); > r = dma_mapping_error(dev, dst[i]); > if (r) { > dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r); > @@ -740,13 +856,47 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > goto out_oom; > } > > - pr_debug_ratelimited("dma mapping dst to 0x%llx, pfn 0x%lx\n", > - dst[i] >> PAGE_SHIFT, page_to_pfn(dpage)); > - > migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); > > - dpage = NULL; > - j++; > + if (k > 1) { > + migrate->dst[i] |= MIGRATE_PFN_COMPOUND; > + /* migrate previous accumulated pages */ > + if (j) { > + r = svm_migrate_copy_memory_gart(adev, dst + i - j, > + src + i - j, j, FROM_VRAM_TO_RAM, mfence); > + if (r) > + goto out_oom; > + j = 0; > + } > + > + for (l = 1; l < k; l++) { > + > + src[i + l] = src[i] + l*PAGE_SIZE; > + dst[i + l] = dst[i] + l*PAGE_SIZE; > + migrate->dst[i + l] = 0; > + } > + > + /* migrate the HPAGE_PMD_NR pages above > + * the migration is for 2MB THP > + */ > + r = svm_migrate_copy_memory_gart( > + adev, dst + i, > + src + i, HPAGE_PMD_NR, > + FROM_VRAM_TO_RAM, > + mfence); > + > + /* mark head page dma mapping as THP, tail pages dma addr > + * are set to 0 for following dma_unmap > + */ > + dst[i] |= SVM_RANGE_DMA_THP; > + for (l = 1; l < k; l++) > + dst[i + l] = 0; > + > + if (r) > + goto out_oom; > + > + } else > + j++; > } > > if (j > 0) > @@ -765,12 +915,9 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, > /* release previous allocated sys pages and unmap dma address */ > while (i--) { > > - if (dst[i]) { > - dma_unmap_page(dev, dst[i], PAGE_SIZE, > - DMA_BIDIRECTIONAL); > - dst[i] = 0; > - } > - > + /* follwing svm_range_dma_unmap_dev will do dma unmap anyway > + * not need do dma unmap here > + */ > dpage = migrate_pfn_to_page(migrate->dst[i]); > if (!dpage) > continue; > @@ -811,6 +958,7 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange, > unsigned long cpages = 0; > unsigned long mpages = 0; > struct amdgpu_device *adev = node->adev; > + bool is_private_device = false; > struct kfd_process_device *pdd; > struct dma_fence *mfence = NULL; > struct migrate_vma migrate = { 0 }; > @@ -818,6 +966,8 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange, > void *buf; > int r = -ENOMEM; > > + is_private_device = svm_is_private_zone(adev); > + > memset(&migrate, 0, sizeof(migrate)); > migrate.vma = vma; > migrate.start = start; > @@ -828,6 +978,9 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange, > else > migrate.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE; > > + if (is_private_device && ((end - start) >> PAGE_SHIFT) >= HPAGE_PMD_NR) > + migrate.flags = migrate.flags | MIGRATE_VMA_SELECT_COMPOUND; > + > buf = kvcalloc(npages, > 2 * sizeof(*migrate.src) + sizeof(u64) + sizeof(dma_addr_t), > GFP_KERNEL); [-- Attachment #1.2: Type: text/html, Size: 14152 bytes --] [-- Attachment #2: 0002-drm-amdkfd-Simplify-svm_migrate_copy_to_ram.patch --] [-- Type: text/x-patch, Size: 9637 bytes --] From b214d0b19f89f674578e05b289f9eed03aa8b873 Mon Sep 17 00:00:00 2001 From: Philip Yang <Philip.Yang@amd.com> Date: Fri, 13 Nov 2026 14:06:50 -0500 Subject: [PATCH 2/2] drm/amdkfd: Simplify svm_migrate_copy_to_ram() Fold the THP and the single page paths into one flow, the same way svm_migrate_copy_to_vram() does: - reuse svm_migrate_flush_batch() for the accumulated copy, it takes the direction now so both migration directions share it. It is a no-op when nothing is pending, so the callers no longer need to guard it and svm_migrate_copy_memory_gart() is called from there only. - rename the loop variables to i and batch, and derive the step nr from order so that order can be passed to svm_migrate_get_sys_page() directly. - fill the tail entries unconditionally, the loops are no-ops for an order 0 page. The accumulated pages still have to stay contiguous in VRAM, and a THP is still copied on its own, so both drain the batch first. This also fixes a double free on the error paths. dpage is declared outside the loop and the cleanup releases it before walking migrate->dst[], which is only correct when dma_map_page() failed. On any other error, a failing copy for instance, dpage still points at a page of an earlier iteration which is already recorded in migrate->dst[], so the walk unlocks and puts it a second time. Clear dpage once the page is recorded, and advance i before the THP copy so that the walk covers the entry the copy failed on. No functional change otherwise. Signed-off-by: Philip Yang <Philip.Yang@amd.com> --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 154 +++++++++++------------ 1 file changed, 71 insertions(+), 83 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index b2fe40bc7510..357ff82fbb64 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -314,16 +314,17 @@ static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate) /* Copy the pages accumulated in [i - *batch, i) and reset the accumulator. */ static int -svm_migrate_flush_batch(struct amdgpu_device *adev, dma_addr_t *src, u64 *dst, - u64 i, u64 *batch, struct dma_fence **mfence) +svm_migrate_flush_batch(struct amdgpu_device *adev, dma_addr_t *sys, u64 *vram, + u64 i, u64 *batch, enum MIGRATION_COPY_DIR dir, + struct dma_fence **mfence) { int r; if (!*batch) return 0; - r = svm_migrate_copy_memory_gart(adev, src + i - *batch, dst + i - *batch, - *batch, FROM_RAM_TO_VRAM, mfence); + r = svm_migrate_copy_memory_gart(adev, sys + i - *batch, vram + i - *batch, + *batch, dir, mfence); *batch = 0; return r; } @@ -396,7 +397,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, /* A THP is copied on its own, so drain whatever preceded it. */ if (order || !spage) { - r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence); + r = svm_migrate_flush_batch(adev, src, dst, i, &batch, + FROM_RAM_TO_VRAM, mfence); if (r) goto out_free_vram_pages; } @@ -420,7 +422,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, if (order && spage) { r = svm_migrate_flush_batch(adev, src, dst, i + nr, &batch, - mfence); + FROM_RAM_TO_VRAM, mfence); /* Mark the head so svm_range_dma_unmap() unmaps the whole * THP, and clear the tails. @@ -441,7 +443,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, off += nr; if (off >= (cursor.size >> PAGE_SHIFT)) { - r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence); + r = svm_migrate_flush_batch(adev, src, dst, i, &batch, + FROM_RAM_TO_VRAM, mfence); if (r) goto out_free_vram_pages; @@ -450,7 +453,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, } } - r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence); + r = svm_migrate_flush_batch(adev, src, dst, i, &batch, FROM_RAM_TO_VRAM, + mfence); out_free_vram_pages: if (r) { @@ -746,11 +750,8 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, struct page *dpage = NULL; dma_addr_t *dst; u64 *src; - - u64 i = 0, j; - u64 addr; + u64 i, batch, addr; int r = 0; - u64 l, k; pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); @@ -759,49 +760,46 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, src = (u64 *)(scratch + npages); dst = scratch; - for (i = 0, j = 0, k = 1; i < npages; i += k, addr += k*PAGE_SIZE, k = 1) { - struct page *spage; - spage = migrate_pfn_to_page(migrate->src[i]); + for (i = batch = 0; i < npages;) { + struct page *spage = migrate_pfn_to_page(migrate->src[i]); + unsigned int order = 0; + u64 nr, n; + if (!spage || !is_zone_device_page(spage)) { pr_debug("invalid page. Could be in CPU already svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); - if (j) { - r = svm_migrate_copy_memory_gart(adev, dst + i - j, - src + i - j, j, - FROM_VRAM_TO_RAM, - mfence); - if (r) - goto out_oom; - j = 0; - } + r = svm_migrate_flush_batch(adev, dst, src, i, &batch, + FROM_VRAM_TO_RAM, mfence); + if (r) + goto out_oom; + i++; + addr += PAGE_SIZE; continue; } + src[i] = svm_migrate_addr(adev, spage); - if (j > 0 && src[i] != src[i - 1] + PAGE_SIZE) { - r = svm_migrate_copy_memory_gart(adev, dst + i - j, - src + i - j, j, - FROM_VRAM_TO_RAM, - mfence); - if (r) - goto out_oom; - j = 0; - } - if (IS_ALIGNED(page_to_pfn(spage), HPAGE_PMD_NR) && - (addr + HPAGE_PMD_SIZE) <= migrate->end && - IS_ALIGNED(addr, HPAGE_PMD_SIZE) && - migrate->src[i] & MIGRATE_PFN_COMPOUND) { + /* MIGRATE_PFN_COMPOUND means the source is a 2MB VRAM folio. */ + if ((migrate->src[i] & MIGRATE_PFN_COMPOUND) && + IS_ALIGNED(page_to_pfn(spage), HPAGE_PMD_NR) && + IS_ALIGNED(addr, HPAGE_PMD_SIZE) && + addr + HPAGE_PMD_SIZE <= migrate->end) + order = HPAGE_PMD_ORDER; + nr = 1ULL << order; - k = HPAGE_PMD_NR; - dpage = svm_migrate_get_sys_page(migrate->vma, addr, - HPAGE_PMD_ORDER); - } else { - k = 1; - dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0); + /* The accumulated pages have to stay contiguous in VRAM, and a + * THP is copied on its own. + */ + if (order || (batch && src[i] != src[i - 1] + PAGE_SIZE)) { + r = svm_migrate_flush_batch(adev, dst, src, i, &batch, + FROM_VRAM_TO_RAM, mfence); + if (r) + goto out_oom; } + dpage = svm_migrate_get_sys_page(migrate->vma, addr, order); if (!dpage) { pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); @@ -809,65 +807,55 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, goto out_oom; } - dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE*k, DMA_BIDIRECTIONAL); + dst[i] = dma_map_page(dev, dpage, 0, nr << PAGE_SHIFT, + DMA_BIDIRECTIONAL); r = dma_mapping_error(dev, dst[i]); if (r) { - dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r); + dev_err(dev, "%s: fail %d dma_map_page\n", __func__, r); dst[i] = 0; goto out_oom; } - migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); - - if (k > 1) { - migrate->dst[i] |= MIGRATE_PFN_COMPOUND; - /* migrate previous accumulated pages */ - if (j) { - r = svm_migrate_copy_memory_gart(adev, dst + i - j, - src + i - j, j, FROM_VRAM_TO_RAM, mfence); - if (r) - goto out_oom; - j = 0; - } + migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)) | + (order ? MIGRATE_PFN_COMPOUND : 0); - for (l = 1; l < k; l++) { + /* Both sides are contiguous, so are their addresses. */ + for (n = 1; n < nr; n++) { + src[i + n] = src[i] + (n << PAGE_SHIFT); + dst[i + n] = dst[i] + (n << PAGE_SHIFT); + migrate->dst[i + n] = 0; + } + batch += nr; - src[i + l] = src[i] + l*PAGE_SIZE; - dst[i + l] = dst[i] + l*PAGE_SIZE; - migrate->dst[i + l] = 0; - } + /* dpage is now owned by migrate->dst[], out_oom frees it from there. */ + dpage = NULL; - /* migrate the HPAGE_PMD_NR pages above - * the migration is for 2MB THP - */ - r = svm_migrate_copy_memory_gart( - adev, dst + i, - src + i, HPAGE_PMD_NR, - FROM_VRAM_TO_RAM, - mfence); - - /* mark head page dma mapping as THP, tail pages dma addr - * are set to 0 for following dma_unmap + i += nr; + addr += nr << PAGE_SHIFT; + + if (order) { + r = svm_migrate_flush_batch(adev, dst, src, i, &batch, + FROM_VRAM_TO_RAM, mfence); + + /* Mark the head so svm_range_dma_unmap() unmaps the whole + * THP, and clear the tails. */ - dst[i] |= SVM_RANGE_DMA_THP; - for (l = 1; l < k; l++) - dst[i + l] = 0; + dst[i - nr] |= SVM_RANGE_DMA_THP; + for (n = 1; n < nr; n++) + dst[i - nr + n] = 0; if (r) goto out_oom; - - } else - j++; + } } - if (j > 0) - r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j, - FROM_VRAM_TO_RAM, mfence); + r = svm_migrate_flush_batch(adev, dst, src, i, &batch, FROM_VRAM_TO_RAM, + mfence); out_oom: if (r) { pr_debug("failed %d copy to ram\n", r); - /* first release current dpage when dma_map_page fail */ + /* The page of the current iteration is not in migrate->dst[] yet. */ if (dpage) { unlock_page(dpage); put_page(dpage); -- 2.50.1 [-- Attachment #3: 0001-drm-amdkfd-Simplify-svm_migrate_copy_to_vram.patch --] [-- Type: text/x-patch, Size: 10731 bytes --] From 20ba86327c7c67749b027ceaccf32b7d59fe3401 Mon Sep 17 00:00:00 2001 From: Philip Yang <Philip.Yang@amd.com> Date: Thu, 12 Nov 2026 17:53:36 -0500 Subject: [PATCH 1/2] drm/amdkfd: Simplify svm_migrate_copy_to_vram() The THP and the single page paths were written as two separate flows using five single letter loop variables, with the same "copy whatever has been accumulated so far" block open coded four times. Fold them into one flow: - add svm_migrate_flush_batch() for the accumulated copy, it is a no-op when nothing is pending so the callers no longer need to guard it. svm_migrate_copy_memory_gart() is now called from there only. - rename the loop variables to i, batch and off, and derive the step nr from order so that order can be passed to svm_migrate_get_vram_page() directly. - claim the destination in a single branch, the tail loops are no-ops for an order 0 page. - map the source with one dma_map_page() of nr pages. A THP is still copied on its own because only its head fills dst[], so it cannot share a transfer with the accumulated pages. This also fixes a page that is left in system memory because its reused svm_bo destination page is still referenced by a CPU fault. The old code signalled the skip by clearing spage, but the single page path recomputed spage right after and the page was dma mapped and copied with dst[i] == 0. A source page without MIGRATE_PFN_MIGRATE took the same path. spage is now computed once and every rejection clears it for good. No functional change otherwise. Signed-off-by: Philip Yang <Philip.Yang@amd.com> --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 242 +++++++++-------------- 1 file changed, 98 insertions(+), 144 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 690599dea15d..b2fe40bc7510 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -312,6 +312,22 @@ static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate) return mpages; } +/* Copy the pages accumulated in [i - *batch, i) and reset the accumulator. */ +static int +svm_migrate_flush_batch(struct amdgpu_device *adev, dma_addr_t *src, u64 *dst, + u64 i, u64 *batch, struct dma_fence **mfence) +{ + int r; + + if (!*batch) + return 0; + + r = svm_migrate_copy_memory_gart(adev, src + i - *batch, dst + i - *batch, + *batch, FROM_RAM_TO_VRAM, mfence); + *batch = 0; + return r; +} + static int svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, struct migrate_vma *migrate, struct dma_fence **mfence, @@ -324,7 +340,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, u64 mpages = 0; dma_addr_t *src; u64 *dst; - u64 i, j, k, l, m; + u64 i, batch, off; int r = 0; pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start, @@ -335,168 +351,106 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, amdgpu_res_first(prange->ttm_res, ttm_res_offset, npages << PAGE_SHIFT, &cursor); - for (i = j = m = 0; (i < npages) && (mpages < migrate->cpages);) { - struct page *spage = migrate_pfn_to_page(migrate->src[i]); - unsigned long cur_dst_pfn; - - cur_dst_pfn = svm_migrate_addr_to_pfn(adev, - cursor.start + (m << PAGE_SHIFT)); - - /* when migrate->src[i] has MIGRATE_PFN_COMPOUND set the src page - * is compound THP; its vm address is HPAGE_PMD_SIZE aligned and - * its MIGRATE_PFN_MIGRATE is set - */ - if ((m + HPAGE_PMD_NR) <= (cursor.size >> PAGE_SHIFT) && - (i + HPAGE_PMD_NR) <= npages && - (migrate->src[i] & MIGRATE_PFN_COMPOUND) && - IS_ALIGNED(cur_dst_pfn, HPAGE_PMD_NR)) - k = HPAGE_PMD_NR; - else - k = 1; - - /* for THP src[0] alwas MIGRATE_PFN_MIGRATE - * just the first migrate->dst need be setup, others are zero - */ - if (k > 1) { - dst[i] = cursor.start + (m << PAGE_SHIFT); - svm_migrate_get_vram_page(prange, cur_dst_pfn, - HPAGE_PMD_ORDER); - migrate->dst[i] = migrate_pfn(cur_dst_pfn); - migrate->dst[i] |= MIGRATE_PFN_COMPOUND; + for (i = batch = off = 0; i < npages && mpages < migrate->cpages;) { + struct page *spage = migrate_pfn_to_page(migrate->src[i]); + u64 vram_addr = cursor.start + (off << PAGE_SHIFT); + unsigned long dst_pfn; + unsigned int order = 0; + u64 nr, n; - for (l = 1; l < k; l++) - migrate->dst[i+l] = 0; + dst_pfn = svm_migrate_addr_to_pfn(adev, vram_addr); + /* MIGRATE_PFN_COMPOUND means the source is a THP, its VA is + * HPAGE_PMD_SIZE aligned and MIGRATE_PFN_MIGRATE is set. + */ + if ((migrate->src[i] & MIGRATE_PFN_COMPOUND) && + off + HPAGE_PMD_NR <= (cursor.size >> PAGE_SHIFT) && + i + HPAGE_PMD_NR <= npages && + IS_ALIGNED(dst_pfn, HPAGE_PMD_NR)) + order = HPAGE_PMD_ORDER; + nr = 1ULL << order; + + if (spage && is_zone_device_page(spage)) + spage = NULL; + + /* Claim the destination, unless the source cannot migrate or a + * reused svm_bo page is still held by a CPU fault: reinitializing + * it would free it under this migration. + */ + if (!(migrate->src[i] & MIGRATE_PFN_MIGRATE) || + (!order && page_ref_count(pfn_to_page(dst_pfn)))) { + dst[i] = 0; + migrate->dst[i] = 0; + spage = NULL; + } else { + /* A THP only fills the head entry, the tails stay zero. */ + dst[i] = vram_addr; + svm_migrate_get_vram_page(prange, dst_pfn, order); + migrate->dst[i] = migrate_pfn(dst_pfn) | + (order ? MIGRATE_PFN_COMPOUND : 0); + for (n = 1; n < nr; n++) + migrate->dst[i + n] = 0; mpages++; - } else if (migrate->src[i] & MIGRATE_PFN_MIGRATE) { - dst[i] = cursor.start + (m << PAGE_SHIFT); - /* - * A reused svm_bo page may still be held by a CPU fault. - * Reinitializing it would free it under this migration, - * so leave this page in system memory. - */ - if (page_ref_count(pfn_to_page(cur_dst_pfn))) { - dst[i] = 0; - migrate->dst[i] = 0; - spage = NULL; - } else { - svm_migrate_get_vram_page(prange, cur_dst_pfn, 0); - migrate->dst[i] = migrate_pfn(cur_dst_pfn); - mpages++; - } } - if (k > 1) { - if (j) { - /* migrate previous accumulated pages */ - r = svm_migrate_copy_memory_gart( - adev, src + i - j, - dst + i - j, j, - FROM_RAM_TO_VRAM, - mfence); - - if (r) - goto out_free_vram_pages; + /* A THP is copied on its own, so drain whatever preceded it. */ + if (order || !spage) { + r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence); + if (r) + goto out_free_vram_pages; + } - j = 0; + if (spage) { + src[i] = dma_map_page(dev, spage, 0, nr << PAGE_SHIFT, + DMA_BIDIRECTIONAL); + r = dma_mapping_error(dev, src[i]); + if (r) { + dev_err(dev, "%s: fail %d dma_map_page\n", __func__, r); + goto out_free_vram_pages; } - /* for THP check if the first src page is valid - * if not valid skip following HPAGE_PMD_NR - 1 pages + /* The source pages are contiguous, so are their dma + * addresses. */ - spage = migrate_pfn_to_page(migrate->src[i]); - if (spage && !is_zone_device_page(spage)) { - /* dma_map continuous HPAGE_PMD_NR sys ram pages */ - src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE*HPAGE_PMD_NR, - DMA_BIDIRECTIONAL); - - r = dma_mapping_error(dev, src[i]); - if (r) { - dev_err(dev, "%s: fail %d dma_map_page\n", - __func__, r); - goto out_free_vram_pages; - } - - /* get dma address for following HPAGE_PMD_NR-1 pages - * since src pages are continuous their dma addresses - * are continuous too. - */ - for (l = 1; l < k; l++) - src[i + l] = src[i] + l*PAGE_SIZE; - - /* migrate the HPAGE_PMD_NR pages above */ - r = svm_migrate_copy_memory_gart( - adev, src + i, - dst + i, HPAGE_PMD_NR, - FROM_RAM_TO_VRAM, - mfence); - - /* mark head page dma mapping as THP, tail pages dma addr - * are set to 0 for following dma_unmap - */ - src[i] |= SVM_RANGE_DMA_THP; - for (l = 1; l < k; l++) - src[i + l] = 0; - - if (r) - goto out_free_vram_pages; - - j = 0; - } - } else { - - /* single normal page case */ - spage = migrate_pfn_to_page(migrate->src[i]); - if (spage && !is_zone_device_page(spage)) { - src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, - DMA_BIDIRECTIONAL); + for (n = 1; n < nr; n++) + src[i + n] = src[i] + (n << PAGE_SHIFT); + batch += nr; + } - r = dma_mapping_error(dev, src[i]); + if (order && spage) { + r = svm_migrate_flush_batch(adev, src, dst, i + nr, &batch, + mfence); - if (r) { - dev_err(dev, "%s: fail %d dma_map_page\n", - __func__, r); - goto out_free_vram_pages; - } - j += 1; + /* Mark the head so svm_range_dma_unmap() unmaps the whole + * THP, and clear the tails. + */ + src[i] |= SVM_RANGE_DMA_THP; + for (n = 1; n < nr; n++) + src[i + n] = 0; - } else if (j) { - r = svm_migrate_copy_memory_gart( - adev, src + i - j, - dst + i - j, j, - FROM_RAM_TO_VRAM, - mfence); - if (r) - goto out_free_vram_pages; - j = 0; - } + if (r) + goto out_free_vram_pages; } - pr_debug_ratelimited("dma mapping %lld pages, src to 0x%llx, pfn 0x%lx\n", - k, src[i] >> PAGE_SHIFT, migrate->src[i] >> MIGRATE_PFN_SHIFT); + pr_debug_ratelimited("dma mapping %llu pages, src to 0x%llx, pfn 0x%lx\n", + nr, src[i] >> PAGE_SHIFT, + migrate->src[i] >> MIGRATE_PFN_SHIFT); - i += k; - m += k; + i += nr; + off += nr; - if (m >= (cursor.size >> PAGE_SHIFT)) { - if (j > 0) { - r = svm_migrate_copy_memory_gart(adev, src + i - j, - dst + i - j, j, - FROM_RAM_TO_VRAM, - mfence); - if (r) - goto out_free_vram_pages; - } + if (off >= (cursor.size >> PAGE_SHIFT)) { + r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence); + if (r) + goto out_free_vram_pages; - amdgpu_res_next(&cursor, m*PAGE_SIZE); - m = 0; + amdgpu_res_next(&cursor, off << PAGE_SHIFT); + off = 0; } } - if (j > 0) - r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j, - FROM_RAM_TO_VRAM, mfence); + r = svm_migrate_flush_batch(adev, src, dst, i, &batch, mfence); out_free_vram_pages: if (r) { @@ -511,8 +465,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, } #ifdef DEBUG_FORCE_MIXED_DOMAINS - for (i = 0, j = 0; i < npages; i += 4, j++) { - if (j & 1) + for (i = 0, off = 0; i < npages; i += 4, off++) { + if (off & 1) continue; svm_migrate_put_vram_page(adev, dst[i]); migrate->dst[i] = 0; -- 2.50.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-11-16 15:54 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-10-19 13:38 [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver Xiaogang.Chen 2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen 2026-11-13 23:28 ` Philip Yang 2026-11-16 15:54 ` Chen, Xiaogang 2026-10-19 13:38 ` [PATCH v3 2/3] drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault handler Xiaogang.Chen 2026-10-19 13:38 ` [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver Xiaogang.Chen 2026-11-13 19:29 ` Philip Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.