* [PATCH 1/4] drm/amdgpu: limit userptr sg segments to half of sg->length
2026-11-18 15:10 [PATCH 0/4] drm/amdkfd: don't stall other threads during large userptr registration William Palacek
@ 2026-11-18 15:10 ` William Palacek
2026-11-18 15:10 ` [PATCH 2/4] drm/amdkfd: don't take p->mutex in AMDKFD_IOC_GET_CLOCK_COUNTERS William Palacek
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: William Palacek @ 2026-11-18 15:10 UTC (permalink / raw)
To: amd-gfx
Cc: William.Palacek, Harish.Kasiviswanathan, Felix.Kuehling,
christian.koenig, philip.yang
Userptr sg tables are built with sg_alloc_table_from_pages(), which
merges physically contiguous pages into segments of up to
0xfffff000 bytes. iommu_dma_map_sg() pads a segment so that the next
one does not cross the DMA segment boundary, and adds the padding to
the 32-bit sg->length. With two large contiguous segments the padded
length reaches 4 GiB and wraps to 0.
On kernels without commit 1c8f6308e557 ("iommu/dma: Catch scatterlist
length overflows") the wrapped length makes iommu_map_sg() map less
than the allocated IOVA range. The IOVA is then freed without
unmapping it, and a later map at that address fails with -EADDRINUSE.
On kernels with that commit, iommu_dma_map_sg() returns -EOVERFLOW
instead. In both cases a large userptr allocation fails to pin. On
MI300X with the default DMA-FQ IOMMU domain, the first 32 GiB
hipHostMalloc after boot built 16 segments of up to 3.7 GiB, mapped
only 15.7 GiB and failed.
The padding is shorter than the next segment, so a padded segment
still fits sg->length when no segment exceeds half of it. Limit both
userptr sg tables to that, as xe does in commit 1b1d3710380d
("drm/xe: Apply upper limit to sg element size").
Signed-off-by: William Palacek <William.Palacek@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 8 ++++----
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 7 ++++---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 7 +++++++
3 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index f060b206c255..565ecd7e012a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -589,10 +589,10 @@ kfd_mem_dmamap_userptr(struct kgd_mem *mem,
return -ENOMEM;
/* Same sequence as in amdgpu_ttm_tt_pin_userptr */
- ret = sg_alloc_table_from_pages(ttm->sg, src_ttm->pages,
- ttm->num_pages, 0,
- (u64)ttm->num_pages << PAGE_SHIFT,
- GFP_KERNEL);
+ ret = sg_alloc_table_from_pages_segment(ttm->sg, src_ttm->pages,
+ ttm->num_pages, 0,
+ (u64)ttm->num_pages << PAGE_SHIFT,
+ AMDGPU_SG_MAX_SEGMENT, GFP_KERNEL);
if (unlikely(ret))
goto free_sg;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 8dfa92967b63..b0672f1747ee 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -837,9 +837,10 @@ static int amdgpu_ttm_tt_pin_userptr(struct ttm_device *bdev,
int r;
/* Allocate an SG array and squash pages into it */
- r = sg_alloc_table_from_pages(ttm->sg, ttm->pages, ttm->num_pages, 0,
- (u64)ttm->num_pages << PAGE_SHIFT,
- GFP_KERNEL);
+ r = sg_alloc_table_from_pages_segment(ttm->sg, ttm->pages,
+ ttm->num_pages, 0,
+ (u64)ttm->num_pages << PAGE_SHIFT,
+ AMDGPU_SG_MAX_SEGMENT, GFP_KERNEL);
if (r)
goto release_sg;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
index 9856a2fc2d08..da7d7621aad1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
@@ -44,6 +44,13 @@
#define AMDGPU_GTT_MAX_TRANSFER_SIZE (1ULL << 22)
+/*
+ * iommu_dma_map_sg() pads a segment so that the next one does not cross the
+ * DMA segment boundary. The padding is shorter than the next segment, so the
+ * padded length still fits sg->length if no segment exceeds half of it.
+ */
+#define AMDGPU_SG_MAX_SEGMENT round_down(UINT_MAX / 2, PAGE_SIZE)
+
extern const struct attribute_group amdgpu_vram_mgr_attr_group;
extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 2/4] drm/amdkfd: don't take p->mutex in AMDKFD_IOC_GET_CLOCK_COUNTERS
2026-11-18 15:10 [PATCH 0/4] drm/amdkfd: don't stall other threads during large userptr registration William Palacek
2026-11-18 15:10 ` [PATCH 1/4] drm/amdgpu: limit userptr sg segments to half of sg->length William Palacek
@ 2026-11-18 15:10 ` William Palacek
2026-11-18 15:10 ` [PATCH 3/4] drm/amdgpu: drop mmap_lock between chunks when faulting userptr pages William Palacek
2026-11-18 15:10 ` [PATCH 4/4] drm/amdkfd: register userptr pages without holding KFD process locks William Palacek
3 siblings, 0 replies; 5+ messages in thread
From: William Palacek @ 2026-11-18 15:10 UTC (permalink / raw)
To: amd-gfx
Cc: William.Palacek, Harish.Kasiviswanathan, Felix.Kuehling,
christian.koenig, philip.yang
The ioctl takes p->mutex only to look up the pdd. p->pdds[] is filled
in create_process() before the process is published and freed only at
teardown, and the clock read that follows already uses the pdd without
the lock, so the lock protects nothing.
It does serialize every clock counter query of the process behind
whatever holds p->mutex. Threads submitting GPU work call this ioctl
(seen in a PyTorch workload on MI300X), so while one thread registered
and mapped a 32 GiB userptr, the others waited in it (205 ms measured
for one call).
Signed-off-by: William Palacek <William.Palacek@amd.com>
---
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
index 3d2136adb62b..9886bbd28ea2 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
@@ -745,9 +745,8 @@ static int kfd_ioctl_get_clock_counters(struct file *filep,
struct kfd_ioctl_get_clock_counters_args *args = data;
struct kfd_process_device *pdd;
- mutex_lock(&p->mutex);
+ /* p->pdds[] is fixed from process creation to teardown, no lock needed */
pdd = kfd_process_device_data_by_id(p, args->gpu_id);
- mutex_unlock(&p->mutex);
if (pdd)
/* Reading GPU clock counter from KGD */
args->gpu_clock_counter = amdgpu_amdkfd_get_gpu_clock_counter(pdd->dev->adev);
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/4] drm/amdgpu: drop mmap_lock between chunks when faulting userptr pages
2026-11-18 15:10 [PATCH 0/4] drm/amdkfd: don't stall other threads during large userptr registration William Palacek
2026-11-18 15:10 ` [PATCH 1/4] drm/amdgpu: limit userptr sg segments to half of sg->length William Palacek
2026-11-18 15:10 ` [PATCH 2/4] drm/amdkfd: don't take p->mutex in AMDKFD_IOC_GET_CLOCK_COUNTERS William Palacek
@ 2026-11-18 15:10 ` William Palacek
2026-11-18 15:10 ` [PATCH 4/4] drm/amdkfd: register userptr pages without holding KFD process locks William Palacek
3 siblings, 0 replies; 5+ messages in thread
From: William Palacek @ 2026-11-18 15:10 UTC (permalink / raw)
To: amd-gfx
Cc: William.Palacek, Harish.Kasiviswanathan, Felix.Kuehling,
christian.koenig, philip.yang
amdgpu_ttm_tt_get_user_pages() holds mmap_read_lock across
hmm_range_fault() for the whole userptr range. Faulting in a 32 GiB
range takes seconds, and every mmap_lock writer of the process (mmap,
munmap, madvise, the KFD SVM flush at each alloc ioctl) waits that long.
Fault the range in 2 MiB chunks and drop mmap_read_lock between them
when the caller passes its mm. A change to the range while the lock is
dropped bumps the notifier sequence, which hmm_range_fault() checks
before each chunk and callers check before using the pages. kfd_svm
passes NULL and keeps holding the lock across the whole range.
Signed-off-by: William Palacek <William.Palacek@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c | 14 ++++++++++++--
drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.h | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 2 +-
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +-
4 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
index a7d13e337d84..9f80ffaa0607 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
@@ -165,12 +165,17 @@ void amdgpu_hmm_unregister(struct amdgpu_bo *bo)
bo->notifier.mm = NULL;
}
+/*
+ * The caller holds mmap_read_lock. If @mm is non-NULL it is dropped between
+ * 2 MiB chunks, so faulting a large range blocks mmap_lock writers for at
+ * most one chunk; the notifier sequence catches any change made meanwhile.
+ */
int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
uint64_t start, uint64_t npages, bool readonly,
- void *owner,
+ void *owner, struct mm_struct *mm,
struct amdgpu_hmm_range *range)
{
- const u64 max_bytes = SZ_2G;
+ const u64 max_bytes = mm ? SZ_2M : SZ_2G;
struct hmm_range *hmm_range = &range->hmm_range;
unsigned long *pfns;
@@ -205,6 +210,11 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
if (hmm_range->end == end)
break;
+ if (mm) {
+ mmap_read_unlock(mm);
+ cond_resched();
+ mmap_read_lock(mm);
+ }
hmm_range->hmm_pfns += max_bytes >> PAGE_SHIFT;
hmm_range->start = hmm_range->end;
} while (hmm_range->end < end);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.h
index 140bc9cd57b4..4d1c96ad2704 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.h
@@ -38,7 +38,7 @@ struct amdgpu_hmm_range {
int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
uint64_t start, uint64_t npages, bool readonly,
- void *owner,
+ void *owner, struct mm_struct *mm,
struct amdgpu_hmm_range *range);
#if defined(CONFIG_HMM_MIRROR)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index b0672f1747ee..fae845632169 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -793,7 +793,7 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo,
readonly = amdgpu_ttm_tt_is_readonly(ttm);
r = amdgpu_hmm_range_get_pages(&bo->notifier, start, ttm->num_pages,
- readonly, NULL, range);
+ readonly, NULL, mm, range);
out_unlock:
mmap_read_unlock(mm);
if (r)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 3f32507e2aae..8d7de7bb39e0 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1892,7 +1892,7 @@ static int svm_range_validate_and_map(struct mm_struct *mm,
range = amdgpu_hmm_range_alloc(NULL);
if (likely(range))
r = amdgpu_hmm_range_get_pages(&prange->notifier, addr, npages,
- readonly, owner, range);
+ readonly, owner, NULL, range);
else
r = -ENOMEM;
WRITE_ONCE(p->svms.faulting_task, NULL);
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 4/4] drm/amdkfd: register userptr pages without holding KFD process locks
2026-11-18 15:10 [PATCH 0/4] drm/amdkfd: don't stall other threads during large userptr registration William Palacek
` (2 preceding siblings ...)
2026-11-18 15:10 ` [PATCH 3/4] drm/amdgpu: drop mmap_lock between chunks when faulting userptr pages William Palacek
@ 2026-11-18 15:10 ` William Palacek
3 siblings, 0 replies; 5+ messages in thread
From: William Palacek @ 2026-11-18 15:10 UTC (permalink / raw)
To: amd-gfx
Cc: William.Palacek, Harish.Kasiviswanathan, Felix.Kuehling,
christian.koenig, philip.yang
Registering a userptr BO faults in its whole range while holding
p->mutex (alloc ioctl) and process_info->lock (init_user_pages). For a
32 GiB hipHostMalloc this takes seconds, and other threads of the
process block on both locks in their own alloc and map ioctls, stalling
GPU submission on every stream.
Allocate the BO without p->mutex; it has no handle until the ioctl
takes the lock again, so no other thread can reach it. In
init_user_pages, fault in and validate the pages without
process_info->lock, then add the BO to userptr_valid_list under
process_info->lock and notifier_lock only if the range was not
invalidated meanwhile. Otherwise return -EAGAIN, which user mode
retries; an invalidation after that point is handled by the restore
worker as before.
Signed-off-by: William Palacek <William.Palacek@amd.com>
---
.../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 45 ++++++++++++-------
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 7 +++
2 files changed, 35 insertions(+), 17 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 565ecd7e012a..01a34ee5f316 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1062,12 +1062,12 @@ static void remove_kgd_mem_from_kfd_bo_list(struct kgd_mem *mem,
/* Initializes user pages. It registers the MMU notifier and validates
* the userptr BO in the GTT domain.
*
- * The BO must already be on the userptr_valid_list. Otherwise an
- * eviction and restore may happen that leaves the new BO unmapped
- * with the user mode queues running.
- *
- * Takes the process_info->lock to protect against concurrent restore
- * workers.
+ * Pages are faulted in and validated without process_info->lock, so a
+ * large range does not block other threads of the process. The BO is
+ * added to the userptr_valid_list only if no invalidation happened since
+ * its pages were read; otherwise an eviction and restore could leave it
+ * stale with the user mode queues running. A lost race returns -EAGAIN
+ * and user mode retries the ioctl.
*
* Returns 0 for success, negative errno for errors.
*/
@@ -1080,19 +1080,17 @@ static int init_user_pages(struct kgd_mem *mem, uint64_t user_addr,
struct amdgpu_hmm_range *range;
int ret = 0;
- mutex_lock(&process_info->lock);
-
ret = amdgpu_ttm_tt_set_userptr(&bo->tbo, user_addr, 0);
if (ret) {
pr_err("%s: Failed to set userptr: %d\n", __func__, ret);
- goto out;
+ return ret;
}
ret = amdgpu_hmm_register(bo, user_addr);
if (ret) {
pr_err("%s: Failed to register MMU notifier: %d\n",
__func__, ret);
- goto out;
+ return ret;
}
if (criu_resume) {
@@ -1102,10 +1100,10 @@ static int init_user_pages(struct kgd_mem *mem, uint64_t user_addr,
* later stage when it is scheduled by another ioctl called by
* CRIU master process for the target pid for restore.
*/
+ add_kgd_mem_to_kfd_bo_list(mem, process_info, true);
mutex_lock(&process_info->notifier_lock);
mem->invalid++;
mutex_unlock(&process_info->notifier_lock);
- mutex_unlock(&process_info->lock);
return 0;
}
@@ -1117,12 +1115,11 @@ static int init_user_pages(struct kgd_mem *mem, uint64_t user_addr,
ret = amdgpu_ttm_tt_get_user_pages(bo, range);
if (ret) {
- amdgpu_hmm_range_free(range);
if (ret == -EAGAIN)
pr_debug("Failed to get user pages, try again\n");
else
pr_err("%s: Failed to get user pages: %d\n", __func__, ret);
- goto unregister_out;
+ goto release_out;
}
ret = amdgpu_bo_reserve(bo, true);
@@ -1138,14 +1135,27 @@ static int init_user_pages(struct kgd_mem *mem, uint64_t user_addr,
if (ret)
pr_err("%s: failed to validate BO\n", __func__);
amdgpu_bo_unreserve(bo);
+ if (ret)
+ goto release_out;
+
+ mutex_lock(&process_info->lock);
+ mutex_lock(&process_info->notifier_lock);
+ if (amdgpu_hmm_range_valid(range)) {
+ /* Earlier invalidations completed before the pages were read */
+ mem->invalid = 0;
+ list_add_tail(&mem->validate_list,
+ &process_info->userptr_valid_list);
+ } else {
+ ret = -EAGAIN;
+ }
+ mutex_unlock(&process_info->notifier_lock);
+ mutex_unlock(&process_info->lock);
release_out:
amdgpu_hmm_range_free(range);
unregister_out:
if (ret)
amdgpu_hmm_unregister(bo);
-out:
- mutex_unlock(&process_info->lock);
return ret;
}
@@ -1875,7 +1885,8 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
(*mem)->mapped_to_gpu_memory = 0;
(*mem)->process_info = avm->process_info;
- add_kgd_mem_to_kfd_bo_list(*mem, avm->process_info, user_addr);
+ if (!user_addr)
+ add_kgd_mem_to_kfd_bo_list(*mem, avm->process_info, false);
if (user_addr) {
pr_debug("creating userptr BO for user_addr = %llx\n", user_addr);
@@ -1907,10 +1918,10 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
return 0;
-allocate_init_user_pages_failed:
err_pin_bo:
err_validate_bo:
remove_kgd_mem_from_kfd_bo_list(*mem, avm->process_info);
+allocate_init_user_pages_failed:
drm_gem_handle_delete(adev->kfd.client.file, (*mem)->gem_handle);
err_gem_handle_create:
drm_vma_node_revoke(&gobj->vma_node, drm_priv);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
index 9886bbd28ea2..acd4dd0f2f06 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
@@ -1228,10 +1228,17 @@ static int kfd_ioctl_alloc_memory_of_gpu(struct file *filep,
}
}
+ /*
+ * Not under p->mutex: a userptr BO faults in its whole range here.
+ * The BO has no handle yet, so nothing else can reach it, and pdd
+ * lives until process teardown.
+ */
+ mutex_unlock(&p->mutex);
err = amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
dev->adev, args->va_addr, args->size,
pdd->drm_priv, (struct kgd_mem **) &mem, &offset,
flags, false);
+ mutex_lock(&p->mutex);
if (err)
goto err_unlock;
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread