From: "Chen, Xiaogang" <xiaogang.chen@amd.com>
To: Philip Yang <yangp@amd.com>, amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM in kfd svm driver
Date: Mon, 16 Nov 2026 09:54:20 -0600 [thread overview]
Message-ID: <b6c80e1b-e6ca-4eac-9fe7-8f67e766192c@amd.com> (raw)
In-Reply-To: <2740efce-be38-4a82-98cf-4ddb6cc1212f@amd.com>
On 11/13/2026 5:28 PM, Philip Yang wrote:
>
>
> On 2026-10-19 09:38, Xiaogang.Chen wrote:
>> From: Xiaogang Chen<xiaogang.chen@amd.com>
>>
>> Extend kfd/svm functions to allocate HPAGE_PMD_SIZE based device
>> memory by
>> buddy allocator, each drm_buddy_block is HPAGE_PMD_SIZE aligned; and to
>> allocate THP system ram by vma_alloc_folio.
>>
>> Introduce SVM_RANGE_DMA_THP flag that indicates dma map of THP. THP dma
>> addresss will use this flag.
>>
>> Add dev_pagemap_ops->folio_split callback that is called by
>> folio_split when
>> core MM splits device memory folio.
>>
>> Reset compound folio of device memory after it got released from core
>> HMM.
>> So, TTM can reuse this VRAM as 4K pages on next range.
>>
>> These are preparations for following support for (THP) migration of zone
>> device-private memory, no function change.
>>
>> Signed-off-by: Xiaogang Chen<xiaogang.chen@amd.com>
>> ---
>> drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 84 ++++++++++++++++++++----
>> drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 37 +++++++++--
>> drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 +
>> 3 files changed, 104 insertions(+), 18 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>> b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>> index 8b96e1fe741a..7a129c9954fc 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>> @@ -218,17 +218,45 @@ svm_migrate_addr_to_pfn(struct amdgpu_device
>> *adev, unsigned long addr)
>> return (addr + adev->kfd.pgmap.range.start) >> PAGE_SHIFT;
>> }
>> +/*
>> + * TTM can reuse a 2MB VRAM block that was previously a device THP.
>> + * Reset unused pages so the next 4K init does not see leftover
>> head/tail.
>> + */
>> +static void svm_migrate_clear_compound_pages(struct page *page,
>> unsigned int nr,
>> + struct dev_pagemap *pgmap)
>> +{
>> + unsigned int i;
>> +
>> + for (i = 0; i < nr; i++, page++) {
>> + struct folio *new_folio = (struct folio *)page;
>> +
>> + if (page_count(page))
>> + continue;
>> +
>> + page->flags.f &= ~0xffUL;
>> +#ifdef NR_PAGES_IN_LARGE_FOLIO
>> + ((struct folio *)(page - 1))->_nr_pages = 0;
>> +#endif
>> + new_folio->mapping = NULL;
>> + new_folio->pgmap = pgmap;
>> + new_folio->share = 0;
>> + page->zone_device_data = NULL;
>> + }
>> +}
>> +
> This is called after the device page is freed. When migrating,
> svm_migrate_get_vram_page() uses the VRAM page and
> zone_device_folio_init() does the same initialization, so this looks
> duplicated. Is it?
svm_migrate_clear_compound_pages() runs because freeing the device THP
only releases the head folio. The other 511 struct pages stay linked to
that head, and TTM can hand those same VRAM pages out later as ordinary
4K pages.
free_zone_device_folio() clears folio->mapping on the head and calls
folio_free. It does not walk the tails. Each tail still has its
compound-tail marker, so page_folio() follows it back to the old head,
and page_pgmap() is implemented as page_folio(page)->pgmap. The tail's
own pgmap slot held the head pointer, not the dev_pagemap.
The next 4K migration initializes that exact pfn:
svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn,
int order)
{
struct page *page = pfn_to_page(pfn);
/* Init this pfn, not page_folio() ? that follows a leftover tail
to the old head. */
zone_device_folio_init((struct folio *)page, page_pgmap(page), order);
page_pgmap() runs before that init. On a leftover tail it returns the
old head's pgmap, or a stale one if that head was already reused.
The clear walks every page of the freed folio, drops the head/tail flag
bits and _nr_pages, and stores the real pgmap on each page. After that,
each page is a standalone zone-device page and the next order-0 init
does not follow the old THP.
>> static void
>> -svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
>> +svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn,
>> + int order)
>> {
>> - struct page *page;
>> + struct page *page = pfn_to_page(pfn);
>> - page = pfn_to_page(pfn);
>> - svm_range_bo_ref(prange->svm_bo);
>> + /* Init this pfn, not page_folio() — that follows a leftover
>> tail to the old head. */
>> + zone_device_folio_init((struct folio *)page, page_pgmap(page),
>> order);
>> page->zone_device_data = prange->svm_bo;
>> - zone_device_page_init(page, page_pgmap(page), 0);
>> + svm_range_bo_ref(prange->svm_bo);
>> }
>> +
>> static void
>> svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long
>> addr)
>> {
>> @@ -249,11 +277,17 @@ svm_migrate_addr(struct amdgpu_device *adev,
>> struct page *page)
>> }
>> static struct page *
>> -svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long
>> addr)
>> +svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long
>> addr,
>> + unsigned long order)
>> {
>> struct page *page;
>> - page = alloc_page_vma(GFP_HIGHUSER, vma, addr);
>> + if (order)
>> + page = folio_page(vma_alloc_folio(GFP_HIGHUSER,
>> + order, vma, addr), 0);
> vma_alloc_folio could fail, then folio_page has NULL access, should
> check vma_alloc_folio return value. if vma_alloc_folio() fails for
> order 9, this fails the entire migration with -ENOMEM and the CPU
> fault becomes SIGBUS. migrate_vma_pages() already handles a compound
> source with a non-compound destination by splitting the source, so
> falling back to a 4K page here would be enough. Where is that handled?
yes, that needs fallback to alloc_page_vma if alloc sys THP fail.
>> + else
>> + page = alloc_page_vma(GFP_HIGHUSER, vma, addr);
>> +
>> if (page)
>> lock_page(page);
>> @@ -267,8 +301,13 @@ static unsigned long
>> svm_migrate_successful_pages(struct migrate_vma *migrate)
>> for (i = 0; i < migrate->npages; i++) {
>> if (migrate->dst[i] & MIGRATE_PFN_VALID &&
>> - migrate->src[i] & MIGRATE_PFN_MIGRATE)
>> - mpages++;
>> + migrate->src[i] & MIGRATE_PFN_MIGRATE) {
>> +
>> + if (migrate->dst[i] & MIGRATE_PFN_COMPOUND)
>> + mpages += HPAGE_PMD_NR;
>> + else
>> + mpages++;
>> + }
>> }
>> return mpages;
>> }
>> @@ -302,7 +341,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node,
>> struct svm_range *prange,
>> if (migrate->src[i] & MIGRATE_PFN_MIGRATE) {
>> dst[i] = cursor.start + (j << PAGE_SHIFT);
>> migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
>> - svm_migrate_get_vram_page(prange, migrate->dst[i]);
>> + svm_migrate_get_vram_page(prange, migrate->dst[i], 0);
>> migrate->dst[i] = migrate_pfn(migrate->dst[i]);
>> mpages++;
>> }
>> @@ -616,15 +655,22 @@ svm_migrate_ram_to_vram(struct svm_range
>> *prange, uint32_t best_loc,
>> return r < 0 ? r : 0;
>> }
>> +/* folio can be compound folio or single page */
>> static void svm_migrate_folio_free(struct folio *folio)
>> {
>> struct page *page = &folio->page;
>> struct svm_range_bo *svm_bo = page->zone_device_data;
>> + struct dev_pagemap *pgmap = folio->pgmap;
>> + unsigned int order = folio_order(folio);
>> if (svm_bo) {
>> pr_debug_ratelimited("ref: %d\n", kref_read(&svm_bo->kref));
>> svm_range_bo_unref_async(svm_bo);
>> }
>> +
>> + /* So TTM can reuse this VRAM as 4K pages on the next range. */
>> + if (order)
>> + svm_migrate_clear_compound_pages(page, 1U << order, pgmap);
>> }
>> static int
>> @@ -678,7 +724,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device
>> *adev, struct svm_range *prange,
>> j = 0;
>> }
>> - dpage = svm_migrate_get_sys_page(migrate->vma, addr);
>> + dpage = svm_migrate_get_sys_page(migrate->vma, addr, 0);
>> if (!dpage) {
>> pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n",
>> prange->svms, prange->start, prange->last);
>> @@ -1081,9 +1127,25 @@ static vm_fault_t svm_migrate_to_ram(struct
>> vm_fault *vmf)
>> return r ? VM_FAULT_SIGBUS : 0;
>> }
>> +static void svm_migrate_folio_split(struct folio *head, struct
>> folio *tail)
>> +{
>> + struct svm_range_bo *svm_bo;
>> +
>> + if (tail == NULL)
>> + return;
>> +
>> + tail->pgmap = head->pgmap;
>> + tail->mapping = head->mapping;
>> +
>> + svm_bo = folio_zone_device_data(head);
>> + folio_set_zone_device_data(tail, svm_bo);
>> + svm_range_bo_ref(svm_bo);
>> +}
>> +
>> static const struct dev_pagemap_ops svm_migrate_pgmap_ops = {
>> .folio_free = svm_migrate_folio_free,
>> .migrate_to_ram = svm_migrate_to_ram,
>> + .folio_split = svm_migrate_folio_split,
>> };
>> /* Each VRAM page uses sizeof(struct page) on system memory */
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> index a99dbedf3cd4..8e1fb5df8abd 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
>> @@ -245,7 +245,16 @@ void svm_range_dma_unmap_dev(struct device *dev,
>> dma_addr_t *dma_addr,
>> if (!svm_is_valid_dma_mapping_addr(dev, dma_addr[i]))
>> continue;
>> pr_debug_ratelimited("unmap 0x%llx\n", dma_addr[i] >>
>> PAGE_SHIFT);
>> - dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
>> +
>> + /* dma unmap of THP */
>> + if (dma_addr[i] & SVM_RANGE_DMA_THP) {
>> + dma_addr[i] &= ~SVM_RANGE_DMA_THP;
>> + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE*HPAGE_PMD_NR,
>> + DMA_BIDIRECTIONAL);
>> + } else {
>> + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
>> + }
>> +
>> dma_addr[i] = 0;
>> }
>> }
>> @@ -578,10 +587,15 @@ svm_range_vram_node_new(struct kfd_node *node,
>> struct svm_range *prange,
>> }
>> memset(&bp, 0, sizeof(bp));
>> - bp.size = prange->npages * PAGE_SIZE;
>> + if (prange->npages >= HPAGE_PMD_NR) {
>> + bp.size = ALIGN(prange->npages * PAGE_SIZE, HPAGE_PMD_SIZE);
>> + bp.byte_align = HPAGE_PMD_SIZE;
>> + } else {
>> + bp.size = prange->npages * PAGE_SIZE;
>> + bp.byte_align = PAGE_SIZE;
>> + }
> seems don't need align size to 2MB, this will alloc vram that never
> used. For example size 3MB, byte_align to 2MB, gpu buddy allocator
> will return first block with 2MB, the second block 1MB
The fallback is align to PAGA_SIZE. Do you mean the case above? That is
aligned to 2MB.
Regards
Xiaogang
> . Regards, Philip
>> bp.bo_ptr_size = sizeof(struct svm_range_bo);
>> bp.destroy = svm_range_bo_destroy;
>> - bp.byte_align = PAGE_SIZE;
>> bp.domain = AMDGPU_GEM_DOMAIN_VRAM;
>> bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
>> bp.flags |= clear ? AMDGPU_GEM_CREATE_VRAM_CLEARED : 0;
>> @@ -593,9 +607,17 @@ svm_range_vram_node_new(struct kfd_node *node,
>> struct svm_range *prange,
>> r = amdgpu_bo_create(node->adev, &bp, &bo);
>> if (r) {
>> - pr_debug("failed %d to create bo\n", r);
>> - mmput(mm);
>> - goto create_bo_failed;
>> + if (prange->npages >= HPAGE_PMD_NR) {
>> + /* fall back PAGE_SIZE granularity */
>> + bp.size = prange->npages * PAGE_SIZE;
>> + bp.byte_align = PAGE_SIZE;
>> + r = amdgpu_bo_create(node->adev, &bp, &bo);
>> + }
>> + if (r) {
>> + pr_debug("failed %d to create bo\n", r);
>> + mmput(mm);
>> + goto create_bo_failed;
>> + }
>> }
>> svm_bo = to_svm_range_bo(bo);
>> @@ -956,6 +978,7 @@ svm_range_copy_array(void *psrc, size_t size,
>> uint64_t num_elements,
>> *vram_pages = 0;
>> for (i = 0; i < num_elements; i++) {
>> dma_addr_t *temp;
>> +
>> temp = (dma_addr_t *)dst + i;
>> *temp = *((dma_addr_t *)src + i);
>> if (*temp&SVM_RANGE_VRAM_DOMAIN)
>> @@ -1628,7 +1651,7 @@ static int svm_range_reserve_bos(struct
>> svm_validate_context *ctx, bool intr)
>> uint32_t gpuidx;
>> int r;
>> - drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT:
>> 0, 0);
>> + drm_exec_init(&ctx->exec, intr ? DRM_EXEC_INTERRUPTIBLE_WAIT :
>> 0, 0);
>> drm_exec_until_all_locked(&ctx->exec) {
>> for_each_set_bit(gpuidx, ctx->bitmap, MAX_GPU_INSTANCE) {
>> pdd = kfd_process_device_from_gpuidx(ctx->process,
>> gpuidx);
>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h
>> b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h
>> index c7d7adae4476..f2b3a05cd8cf 100644
>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h
>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h
>> @@ -35,6 +35,7 @@
>> #include "kfd_priv.h"
>> #define SVM_RANGE_VRAM_DOMAIN (1UL << 0)
>> +#define SVM_RANGE_DMA_THP (1UL << 1)
>> #define SVM_ADEV_PGMAP_OWNER(adev)\
>> ((adev)->hive ? (void *)(adev)->hive : (void *)(adev))
>
next prev parent reply other threads:[~2026-11-16 15:54 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-19 13:38 [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver Xiaogang.Chen
2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen
2026-11-13 23:28 ` Philip Yang
2026-11-16 15:54 ` Chen, Xiaogang [this message]
2026-10-19 13:38 ` [PATCH v3 2/3] drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault handler Xiaogang.Chen
2026-10-19 13:38 ` [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver Xiaogang.Chen
2026-11-13 19:29 ` Philip Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b6c80e1b-e6ca-4eac-9fe7-8f67e766192c@amd.com \
--to=xiaogang.chen@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=yangp@amd.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.