From: Xiaogang.Chen <xiaogang.chen@amd.com>
To: <amd-gfx@lists.freedesktop.org>
Cc: Xiaogang Chen <xiaogang.chen@amd.com>
Subject: [PATCH v3 2/3] drm/amdkfd: Do not migrate partial THP memory at CPU/GPU page fault handler
Date: Mon, 19 Oct 2026 08:38:44 -0500 [thread overview]
Message-ID: <20261019133845.166916-3-xiaogang.chen@amd.com> (raw)
In-Reply-To: <20261019133845.166916-1-xiaogang.chen@amd.com>
From: Xiaogang Chen <xiaogang.chen@amd.com>
After support device THP the page fault(either CPU or GPU) address can point
to THP(device or anonymous THP).
If the correspondent svm prange covers the whole compound folio migrate at
folio order so HMM can collect MIGRATE_PFN_COMPOUND.
Do not migrate partial THP memory. If do HMM will split THP folio. If the
compound folio has extra ref/pin the split will fail, then skip migration,
leave without page restore, then cpu/gpu keep retrying.
If the prange was split and no longer covers the PMD, do not call
split_folio(): it is not exported to modules. Walk only this prange and let
migrate_vma_setup() split the PMD inside mm.
If the fault address has PTE mapping memory use prange->granularity to decide
migration size.
Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 39 ++++++++++-
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 82 +++++++++++++++++++++++-
drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 12 ++++
3 files changed, 130 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index 7a129c9954fc..2febe3bb690e 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -1059,6 +1059,8 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf)
struct svm_range *prange;
struct kfd_process *p;
struct mm_struct *mm;
+ struct folio *folio;
+ unsigned int order;
int r = 0;
svm_bo = vmf->page->zone_device_data;
@@ -1104,8 +1106,41 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf)
if (!prange->actual_loc)
goto out_unlock_prange;
- /* Align migration range start and size to granularity size */
- size = 1UL << prange->granularity;
+ folio = page_folio(vmf->page);
+ order = folio_order(folio);
+
+ /*
+ * A PMD device-private fault always passes the folio head in
+ * vmf->page. If this prange still covers the whole folio, migrate
+ * at folio order so HMM can collect MIGRATE_PFN_COMPOUND.
+ *
+ * Do not want migrate partial THP device memory. If do HMM will split
+ * THP folio. If the compound folio has extra ref/pin the split will fail,
+ * then skip migration, leave without page restore, then cpu keep retry.
+ *
+ * If the prange was split and no longer covers the PMD, do not call
+ * split_folio() here: it is not exported to modules. Walk only this
+ * prange and let migrate_vma_setup() split the PMD inside mm.
+ *
+ * If the fault address has PTE mapping device-private memory use
+ * prange->granularity to decide migration size.
+ */
+ if (order && !vmf->pte) {
+ unsigned long folio_start = ALIGN_DOWN(addr, 1UL << order);
+ unsigned long folio_last = folio_start + (1UL << order) - 1;
+
+ if (prange->start > folio_start || prange->last < folio_last) {
+ pr_debug("partial prange on device THP svms 0x%p prange [0x%lx 0x%lx] folio [0x%lx 0x%lx]\n",
+ prange->svms, prange->start, prange->last,
+ folio_start, folio_last);
+ size = 1UL << prange->granularity;
+ } else {
+ size = 1UL << max(prange->granularity, order);
+ }
+ } else {
+ size = 1UL << prange->granularity;
+ }
+
start = max(ALIGN_DOWN(addr, size), prange->start);
last = min(ALIGN(addr + 1, size) - 1, prange->last);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 8e1fb5df8abd..7443beb5778c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -38,6 +38,8 @@
#include "kfd_svm.h"
#include "kfd_migrate.h"
#include "kfd_smi_events.h"
+#include <linux/mm.h>
+#include <linux/leafops.h>
#ifdef dev_fmt
#undef dev_fmt
@@ -3078,6 +3080,53 @@ svm_fault_allowed(struct vm_area_struct *vma, bool write_fault)
return (vma->vm_flags & requested) == requested;
}
+/*
+ * svm_range_cpu_huge_folio_order - folio order of a PMD-mapped CPU mapping
+ * @vma: VMA covering @va
+ * @va: CPU virtual address of the GPU retry fault
+ *
+ * GPU retry faults have no vmf->page. Walk to the PMD (mmap read lock is
+ * already held) and return the folio order when that mapping is a huge PMD:
+ * anonymous THP or a device-private huge PMD. PTE-mapped pages return 0 so
+ * the caller keeps prange->granularity, matching svm_migrate_to_ram() which
+ * only expands on !vmf->pte.
+ *
+ * Return: folio order, or 0 if the PMD is not a huge mapping
+ */
+static unsigned int
+svm_range_cpu_huge_folio_order(struct vm_area_struct *vma, unsigned long va)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmdp;
+ pmd_t pmd;
+
+ pgd = pgd_offset(mm, va);
+ if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
+ return 0;
+
+ p4d = p4d_offset(pgd, va);
+ if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d)))
+ return 0;
+
+ pud = pud_offset(p4d, va);
+ if (pud_none(*pud) || unlikely(pud_bad(*pud)))
+ return 0;
+
+ pmdp = pmd_offset(pud, va);
+ pmd = pmdp_get_lockless(pmdp);
+
+ if (pmd_trans_huge(pmd))
+ return folio_order(pmd_folio(pmd));
+
+ if (pmd_is_device_private_entry(pmd))
+ return folio_order(softleaf_to_folio(softleaf_from_pmd(pmd)));
+
+ return 0;
+}
+
int
svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
uint32_t vmid, uint32_t node_id,
@@ -3246,8 +3295,39 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid,
kfd_smi_event_page_fault_start(node, p->lead_thread, addr,
write_fault, timestamp);
- /* Align migration range start and size to granularity size */
+ /*
+ * Same policy as svm_migrate_to_ram(): if the CPU maps a device or
+ * anonymous THP at this address and this prange still covers the
+ * whole folio, migrate at folio order so HMM can collect
+ * MIGRATE_PFN_COMPOUND. A split prange that not convers the THP keeps
+ * using granularity. migrate_vma_setup() splits the PMD inside mm.
+ *
+ * GPU faults have no vmf->page, so inspect the PMD.
+ *
+ * If the fault address has PTE mapping use prange->granularity to decide
+ * migration size.
+ *
+ */
size = 1UL << prange->granularity;
+ if (svm_is_private_zone(adev)) {
+ unsigned int order;
+
+ order = svm_range_cpu_huge_folio_order(vma, addr << PAGE_SHIFT);
+ if (order) {
+ unsigned long folio_start = ALIGN_DOWN(addr, 1UL << order);
+ unsigned long folio_last = folio_start + (1UL << order) - 1;
+
+ if (prange->start > folio_start ||
+ prange->last < folio_last) {
+ pr_debug("partial prange on CPU THP svms 0x%p prange [0x%lx 0x%lx] folio [0x%lx 0x%lx]\n",
+ svms, prange->start, prange->last,
+ folio_start, folio_last);
+ } else {
+ size = 1UL << max_t(unsigned int,
+ prange->granularity, order);
+ }
+ }
+ }
start = max_t(unsigned long, ALIGN_DOWN(addr, size), prange->start);
last = min_t(unsigned long, ALIGN(addr + 1, size) - 1, prange->last);
if (prange->actual_loc != 0 || best_loc != 0) {
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h
index f2b3a05cd8cf..d6862a80a433 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h
@@ -214,6 +214,13 @@ void svm_range_bo_unref_async(struct svm_range_bo *svm_bo);
void svm_range_set_max_pages(struct amdgpu_device *adev);
int svm_range_switch_xnack_reserve_mem(struct kfd_process *p, bool xnack_enabled);
+/* check adev has device private zone memory */
+static inline bool svm_is_private_zone(struct amdgpu_device *adev)
+{
+ struct amdgpu_kfd_dev *kfddev = &adev->kfd;
+
+ return (kfddev->pgmap.type == MEMORY_DEVICE_PRIVATE);
+}
#else
@@ -277,6 +284,11 @@ static inline void svm_range_set_max_pages(struct amdgpu_device *adev)
{
}
+static inline bool svm_is_private_zone(struct amdgpu_device *adev)
+{
+ return false;
+}
+
#define KFD_IS_SVM_API_SUPPORTED(dev) false
#endif /* IS_ENABLED(CONFIG_HSA_AMD_SVM) */
--
2.34.1
next prev parent reply other threads:[~2026-10-19 13:38 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-19 13:38 [PATCH v3 0/3] drm/amdkfd: Enable device private memory THP support in kfd svm driver Xiaogang.Chen
2026-10-19 13:38 ` [PATCH v3 1/3] drm/amdkfd: Add awareness of THP of device and system RAM " Xiaogang.Chen
2026-11-13 23:28 ` Philip Yang
2026-11-16 15:54 ` Chen, Xiaogang
2026-10-19 13:38 ` Xiaogang.Chen [this message]
2026-10-19 13:38 ` [PATCH v3 3/3] drm/amdkfd: Enable HMM device-private THP migration in KFD SVM driver Xiaogang.Chen
2026-11-13 19:29 ` Philip Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261019133845.166916-3-xiaogang.chen@amd.com \
--to=xiaogang.chen@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.