[RFC V2 00/12] drm/amdgpu: SVM implementation based on drm_gpusvm

Honglei Huang
2026-04-20T12:08:13Z

From: Honglei Huang <honghuan@amd.com>

V2 of the SVM patch series for amdgpu based on the drm_gpusvm framework.
This revision incorporates feedback from V1, adds XNACK on GPU fault handling,
improves code organization, and removes the XNACK off (no GPU fault) implementation
to focus on the fault driven model that aligns with drm_gpusvm's design.
The implementation references extensively from xe_svm.

This patch series implements SVM support with the following design:

1. Attributes separated from physical page management:

- Attribute layer (amdgpu_svm_attr_tree): a driver-side interval
tree storing per-range SVM attributes. Managed through SET_ATTR
ioctl and preserved across range lifecycle events.

- Physical page layer (drm_gpusvm ranges): managed by the
drm_gpusvm framework, representing HMM-backed DMA mappings
and GPU page table entries.

This separation ensures attributes survive when GPU ranges are
destroyed (partial munmap, attribute split, GC). The fault
handler recreates GPU ranges from the attribute tree on demand.

2. GPU fault driven mapping (XNACK on):

The core mapping path is driven by GPU page faults instead of ioctls.
amdgpu_svm_handle_fault() looks up SVM by PASID, runs GC,
resolves attributes, then maps via find_or_insert -> get_pages
-> GPU PTE update. For unregistered addresses, default
attributes are derived from VMA properties automatically.

3. MMU notifier invalidation:

Two-phase callback: event_begin() zaps GPU PTEs and flushes
TLB, event_end() unmaps DMA pages. UNMAP events queue ranges
to GC for deferred cleanup. Non-UNMAP events (eviction) rely
on GPU fault to remap.

4. Garbage collector:

GC workqueue processes unmapped ranges: removes them
from drm_gpusvm and clears corresponding attributes. No
rebuild or restore logic, GPU fault handles recreation.

Changes since V1:
- Added GPU fault handler: amdgpu_svm_handle_fault with PASID-based
SVM lookup, following the standard flow: garbage collector ->
find or insert range -> check valid -> migrate (TODO) / get_pages
-> GPU bind/map.

- Removed the restore worker queue entirely. V1 had separate GC
and restore workers: restore workers were responsible for
synchronously restore in queue stop/start cause no GPU fault support.
With XNACK on fault driven model, synchronous restore is unnecessary,
the GPU fault handler recreates ranges on demand. The GC worker in
V2 is simplified to only discard ranges and clear their attributes,
with no rebuild or restore logic. AMDGPU_SVM_FLAG_GPU_ALWAYS_MAPPED
support is removed as no restore worker.

- Reworked MMU notifier callback (amdgpu_svm_range_invalidate):
V1 had a monolithic dispatcher with flag combinations and
queue ops (CLEAR_PTE/QUEUE_INTERVAL, UNMAP/RESTORE) plus
begin_restore() to quiesce KFD queues. V2 uses a two-phase
model: event_begin() zaps GPU PTEs and flushes TLB,
event_end() unmaps DMA pages and queues UNMAP ranges to GC.
Non-UNMAP events (eviction) just zap PTEs and let GPU fault
remap. Removed begin_restore/end_restore callbacks,
has_always_mapped_range() check, and NOTIFIER flag dispatch.
Added checkpoint timestamp capture on UNMAP for fault dedup.

- Added amdgpu_svm_range_invalidate_interval(): when userspace
sets new attributes on a sub region of an existing attribute
range, the attribute tree splits the old range and the new
sub region gets different attributes. However, existing
drm_gpusvm ranges may across the new attribute boundary
(e.g., a 2M GPU range covers both the old and new attribute
regions). This function walks all gpusvm ranges in the
affected interval, zaps GPU PTEs and flushes TLB. Ranges
that cross the new boundary and old boundary are removed
entirely so the GPU fault handler can recreate them with
boundaries aligned to the updated attribute layout.

- On MMU_NOTIFY_UNMAP events, discard all affected gpusvm ranges
entirely without synchronous rebuild in v1. The unmap may destroy
more ranges than strictly necessary (e.g., a partial munmap
hits a 2M range that extends beyond the unmapped region), but
the attribute layer preserves the still valid attributes for
the remaining address space. When the GPU next accesses those
addresses, the fault handler automatically recreates the
ranges with correct boundaries from the surviving attributes.
This avoids the synchronous rebuild logic that V1 required
(unmap -> rebuild in GC/restore worker).

- Add attribute creation for unregistered addresses:
amdgpu_svm_range_get_unregistered_attrs() derives default
SVM attributes from VMA properties and GPU IP capabilities
when the faulting address has no user attributes registered.
this feature is needed to pass ROCm user mode runtime tests:
kfd/rocr/hip. ROCm supports no registered virtual address access
with default SVM attributes before, so amdgpu svm needs to support.

- Explicitly returns -EOPNOTSUPP in amdgpu_svm_init when XNACK
is disabled. V1 attempted mixed XNACK on/off support with
complex KFD queue quiesce/resume callbacks and ioctl driven
mapping paths, which added substantial complexity. V2 drops
these implementations to focus on the fault driven model.

- Removed kgd2kfd_quiesce_mm()/resume_mm() dependency that V1
used for XNACK off queue control. For XNACK on, the GPU fault
handler is the enterance for SVM range mapping, so no quiesce/resume
is needed for this version.

- Added new change triggers: TRIGGER_RANGE_SPLIT, TRIGGER_PREFETCH.
for sub attr set and prefetch trigger support.

- Added helper functions: find_locked, get_bounds_locked,
set_default for GPU fault handling.

- Design questions section removed.

TODO:
- Add multi GPU support.
- Add XNACK off mode.
- Add migration or prefetch. This part work is ongoing in:
https://lore.kernel.org/amd-gfx/20260410113146.146212-1-Junhua.Shen@amd.com/

Test results:
Tested on gfx943 (MI300X) and gfx906 (MI60) with XNACK on:
- KFD test: 95%+ passed.
- ROCR test: all passed.
- HIP catch test: gfx943 (MI300X): 96% passed.
gfx906 (MI60):99% passed.

Patch overview:

01/12 UAPI: DRM_AMDGPU_GEM_SVM ioctl, SVM flags, SET_ATTR/GET_ATTR
operations, attribute types in amdgpu_drm.h.

02/12 Core header: amdgpu_svm wrapping drm_gpusvm with refcount,
attr_tree, GC struct, locks, and VM integration hooks.

03/12 Attribute types: amdgpu_svm_attrs, attr_range (interval tree
node), attr_tree, access enum, flag masks, change triggers.

04/12 Attribute tree ops: interval tree lookup, insert, remove,
find_locked, get_bounds_locked, set_default, and lifecycle.

05/12 Attribute set/get/clear: validate UAPI attributes, apply to
tree with head/tail splitting, change propagation, and query.

06/12 Range types: amdgpu_svm_range extending drm_gpusvm_range
with gpu_mapped state, pending ops, work queue linkage,
and op_ctx for batch processing.

07/12 Range GPU mapping: PTE flags computation with read_only
support, GPU page table update, range mapping loop.

08/12 Notifier and GC helpers: two-phase notifier events, range
removal, GC enqueue/add with dedicated workqueue.

09/12 Attribute change and invalidation: apply attribute triggers
to GPU ranges, invalidate_interval for boundary realignment,
work queue dequeue helpers, checkpoint timestamp.

10/12 Initialization and lifecycle: kmem_cache, drm_gpusvm_init
with chunk sizes (2M/64K/4K), XNACK detection, GC init,
PASID lookup, TLB flush, and init/close/fini lifecycle.

11/12 Ioctl, GC, and fault handler: ioctl dispatcher, GC worker,
and amdgpu_svm_fault.c/h with full fault path including
unregistered attribute derivation and retry logic.

12/12 Build integration: Kconfig (CONFIG_DRM_AMDGPU_SVM), Makefile
rules, ioctl registration, and amdgpu_vm fault dispatch.

Honglei Huang (12):
drm/amdgpu: add SVM UAPI definitions
drm/amdgpu: add SVM data structures and header
drm/amdgpu: add SVM attribute data structures
drm/amdgpu: implement SVM attribute tree operations
drm/amdgpu: implement SVM attribute set
drm/amdgpu: add SVM range data structures
drm/amdgpu: implement SVM range PTE flags and GPU mapping
drm/amdgpu: implement SVM range notifier and invalidation
drm/amdgpu: implement SVM range workers
drm/amdgpu: implement SVM core initialization and fini
drm/amdgpu: implement SVM ioctl and fault handler
drm/amdgpu: wire up SVM build system and fault handler

drivers/gpu/drm/amd/amdgpu/Kconfig | 11 +
drivers/gpu/drm/amd/amdgpu/Makefile | 13 +
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 430 ++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 147 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 894 ++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h | 110 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 1196 +++++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 76 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 40 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 4 +
include/uapi/drm/amdgpu_drm.h | 39 +
12 files changed, 2958 insertions(+), 4 deletions(-)
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h

base-commit: 7d0a66e4bb9081d75c82ec4957c50034cb0ea449
--
2.34.1