[PATCH v9 00/18] drm/amdgpu: AMDGPU SVM support based on DRM (Phase 1: single GPU, XNACK on)

[PATCH v9 00/18] drm/amdgpu: AMDGPU SVM support based on DRM (Phase 1: single GPU, XNACK on)

[Huang Rui]

Hi Christian, Philip, Alex, Felix, and all,

These series introduce a new Shared Virtual Memory (SVM) implementation for
amdgpu that is built directly on top of the common DRM GPUSVM / drm_pagemap
core (drivers/gpu/drm/drm_gpusvm.c, drm_pagemap.c) rather than on the
existing KFD-private SVM code in amdkfd/kfd_svm.c.

The goal is to provide HMM-based unified memory through the same shared
infrastructure that the Xe driver already uses, so that amdgpu and xe
converge on one well-reviewed SVM/page-migration core instead of
maintaining a separate amdgpu-specific stack.

Phased plan
===========

This work is being upstreamed in phases to keep each submission reviewable:

  * Phase 1 (this series): single GPU, XNACK on (retry faults).
     - v8 https://lore.kernel.org/amd-gfx/20260612090928.29682-1-ray.huang@amd.com/

  * Phase 2 (in progress): single GPU, XNACK off (no-retry / eager mapping).
     - https://lore.kernel.org/amd-gfx/20260529054928.596825-1-honglei1.huang@amd.com/
     - https://lore.kernel.org/amd-gfx/20260605075340.20199-1-Junhua.Shen@amd.com/

  * Phase 3 (in progress): multiple GPUs (XGMI mapping, P2P device-to-device migration).
     - https://lore.kernel.org/amd-gfx/20260603065620.2555316-1-honglei1.huang@amd.com

Accordingly, this first submission targets the simplest useful configuration:

 * Single GPU with local VRAM migration: transparent RAM <-> VRAM page
   migration on one GPU via SDMA, with TTM-based eviction for overcommit.
 * XNACK on (retry faults): GPU page faults are retried after the driver
   lazily populates PTEs on demand. The XNACK-off/eager-mapping path is
   deferred to Phase 2.
 * Compute VMs only (amdgpu_vm_make_compute()), matching the KFD SVM
   use case.

Everything is gated behind a new, default-n Kconfig option
(CONFIG_DRM_AMDGPU_SVM) and is therefore opt-in and isolated from existing
users until the feature matures.

UAPI
====

A new render-node ioctl, DRM_IOCTL_AMDGPU_GEM_SVM, lets userspace describe SVM
attributes over a CPU virtual-address interval (patch 1):

  - operations: SET_ATTR / GET_ATTR / RESET_ATTR
  - access modes: INACCESSIBLE / IN_PLACE / ALLOW_MIGRATE
  - location hints: SYSMEM / UNDEFINED (preferred_loc, prefetch_loc)
  - per-range flags: HOST_ACCESS, COHERENT, EXT_COHERENT, HIVE_LOCAL,
    GPU_RO, GPU_EXEC, GPU_READ_MOSTLY, plus a granularity hint

Attributes are stored in a per-VM interval tree; the GPU page tables are
populated lazily on demand by retry faults, or eagerly on a prefetch request.

User Space Work
===============

  - ROCm UMD interface adaptation for the new drm SVM API is being
    developed in:
    https://github.com/ROCm/rocm-systems/pull/4364

Design Overview
===============

The implementation is split into clearly layered modules:

  amdgpu_svm.c            Core context (struct amdgpu_svm embeds
                          struct drm_gpusvm), kref lifecycle, PASID lookup,
                          drm_gpusvm_ops, the GEM_SVM ioctl entry point, and
                          the GC workqueue.
  amdgpu_svm_attr.c       The attribute interval tree: validation, gap/overlap
                          split-merge, SET/GET/RESET, and change-trigger
                          classification that decides whether an attribute
                          change needs PTE invalidation or a remap.
  amdgpu_svm_range.c      Per-range GPU mapping: PTE-flag computation per GC IP
                          version, DMA-segment coalescing, MMU-notifier
                          begin/end handling, PTE zapping, and the garbage
                          collector.
  amdgpu_svm_fault.c      The retry-fault entry point amdgpu_svm_handle_fault()
                          and the fault_map_range() pipeline.
  amdgpu_migrate.c        drm_pagemap / ZONE_DEVICE VRAM migration backend.
  amdgpu_svm_range_migrate.c  Per-range RAM<->VRAM migration helpers.

Fault path (XNACK on):
amdgpu_vm_handle_fault() routes a compute-VM retry fault to
amdgpu_svm_handle_fault() when the VM has an SVM context. After PASID lookup
and a checkpoint-timestamp filter that drops stale retry faults left over from
a recent unmap, it looks up (or synthesizes a default) attribute range and runs
fault_map_range(): garbage-collect -> VMA permission check -> find/insert range
-> short-circuit if recently validated or already valid -> drm_gpusvm get_pages
(HMM) -> program GPU PTEs under the notifier lock with a notifier-sequence
re-check.

Invalidation / GC:
MMU-notifier callbacks zap the affected PTEs (batched, single heavyweight TLB
flush) and, for unmap events, queue the range to a high-priority GC workqueue
that removes it via drm_gpusvm_range_remove() outside notifier context.

VRAM migration (patches 13-18):
A drm_pagemap is registered over the GPU's VRAM as a ZONE_DEVICE region at
device-init / reset-restore time. struct amdgpu_bo_svm (a new BO subtype)
backs migrated ranges, with SDMA-based copy_to_devmem / copy_to_ram callbacks
using a GART window. TTM eviction of SVM BOs synchronously migrates pages back
to system memory, which keeps VRAM overcommit working. The fault and prefetch
paths call amdgpu_svm_range_migrate_to_vram() before mapping when migration is
requested, with a single -EBUSY retry that evicts conflicting pages first.

Test Results
============

Tested on gfx943 (MI300X) and gfx906 (MI60) with XNACK on:
 - KFD test: 95%+ passed.
 - ROCR test: all passed.
 - HIP catch test: gfx943 (MI300X): 99% passed. gfx906 (MI60): 99% passed.

Changes from Old Version
========================

In this V8 version, we have consolidated all implementations for the
single-GPU XNACK-on mode. This approach follows Christian's suggestion to
make the code review process more straightforward.

Previously, the XNACK-on implementation was split into two parts:

 - Basic V7
   https://lore.kernel.org/amd-gfx/20260529054804.596214-1-honglei1.huang@amd.com/
 - Migration V5
   https://lore.kernel.org/amd-gfx/20260605075340.20199-1-Junhua.Shen@amd.com/

Above two parts are no longer needed to be reviewed separately, please
focus on this series.

Changes from V8 to V9:
 - This series has been rebased on top of the amdgpu vm fix.
   - https://lore.kernel.org/amd-gfx/20260529114031.3714-1-christian.koenig@amd.com/
 - amdgpu_svm.h: dropped the AMDGPU_SVM_KMEM_CACHE_* macros; SVM ranges use
   plain kzalloc()/kfree().
 - amdgpu_svm.h: dropped the flush_tlb callback; amdgpu_svm_flush_tlb() is
   called directly.
 - amdgpu_svm_attr.h: added kerneldoc for the interval tree and list.
 - amdgpu_svm_attr.c: dropped the attr slab cache; uses kzalloc()/kfree().
 - amdgpu_svm_attr.c: squashed the types header into its implementation patch
 - amdgpu_svm_attr.c: renamed the local "flags" to "unsupported_vm_flags".
 - amdgpu_svm_attr.c: dropped amdgpu_svm_attr_validate_range_vma() from the
   fault path; hmm_range_fault() covers it. A lighter sync check is kept
   only at the set/get attr ioctl entry for fix regression.
 - amdgpu_svm_range.h: AMDGPU_SVM_RANGE_DEBUG removed, replaced by a tracepoint.
 - amdgpu_svm_range.h: dropped the UNMAP_WORK macro.
 - amdgpu_svm_range.c: the fast range valid check is now lock free.
 - amdgpu_svm_range.c: zap_ptes always drains the fence regardless of the
   update return value.
 - per IP PTE flags: moved the switch out of the SVM core
   into gmc_v9_0.c/gmc_v12_0.c behind a new gmc_funcs->get_svm_pte_flags()
   callback.
 - eviction_lock: drm_gpusvm framwork's limitation blocks using the
   eviction_lock kept as is.
 - SVM/BO overlap: dropped the scan-based amdgpu_svm_attr_check_vm_bo() helper.
 - Added drm_pagemap refcount-based deferred destroy for amdgpu_pagemap:
   the ZONE_DEVICE mapping is torn down from a destroy worker
   (amdgpu_pagemap_destroy_work) once the drm_pagemap refcount reaches zero,
   using drm_dev_enter()/exit() to guard against device teardown
 - Added pre_migrate_fence handling in the SDMA copy callbacks: wait for
   outstanding async VRAM operations (DMA_RESV_USAGE_KERNEL) to complete
   before the copy reads or overwrites VRAM, fixing a data race during
   migration
 - In TTM eviction, return -EBUSY directly when mmget_not_zero() fails
   (owning process is exiting), before entering drm_pagemap_evict_to_ram():
   this lets the TTM eviction LRU walk skip that victim and continue
   instead of fatally failing the allocation, while keeping drm_pagemap's
   dead-mm -EFAULT path unreachable.

Thanks,
Ray/Honglei/Junhua

Honglei Huang (12):
  drm/amdgpu: add SVM ioctl UAPI definitions
  drm/amdgpu: add SVM core header and VM integration
  drm/amdgpu: implement SVM attribute tree and helper functions
  drm/amdgpu: implement SVM attribute set/get/clear operations
  drm/amdgpu: add SVM range types and work queue interface
  drm/amdgpu/gmc: add get_svm_pte_flags callback
  drm/amdgpu: implement SVM range GPU mapping core
  drm/amdgpu: implement SVM range notifier and GC helpers
  drm/amdgpu: add SVM notifier invalidate callback and checkpoint
  drm/amdgpu: implement SVM initialization and lifecycle
  drm/amdgpu: add SVM ioctl entry and fault handler module
  drm/amdgpu: integrate SVM into build system and VM fault path

Junhua Shen (6):
  drm/amdgpu: add VRAM migration infrastructure for drm_pagemap
  drm/amdgpu: implement drm_pagemap SDMA migration callbacks
  drm/amdgpu: implement synchronous TTM eviction for SVM BOs
  drm/amdgpu: hook up ZONE_DEVICE registration in device init and reset
  drm/amdgpu: add SVM range migration helpers for drm_pagemap
  drm/amdgpu: integrate VRAM migration into SVM fault and prefetch paths

 drivers/gpu/drm/amd/amdgpu/Kconfig            |  10 +
 drivers/gpu/drm/amd/amdgpu/Makefile           |  11 +
 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |   8 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h       |  24 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c   | 948 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h   | 107 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |   4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c     |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c       | 826 +++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       | 205 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c  | 948 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h  | 202 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 459 +++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h |  39 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 818 +++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 191 ++++
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 120 +++
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.h |  35 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  30 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |  20 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |  23 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |   4 +
 drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c        |  69 ++
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c         | 119 +++
 include/uapi/drm/amdgpu_drm.h                 | 106 ++
 27 files changed, 5329 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h

-- 
2.53.0

[PATCH v9 01/18] drm/amdgpu: add SVM ioctl UAPI definitions

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Define the following ioctl structures and enums:
- DRM_AMDGPU_GEM_SVM ioctl command and DRM_IOCTL_AMDGPU_GEM_SVM macro
- enum amdgpu_ioctl_svm_op: SET_ATTR, GET_ATTR, RESET_ATTR operations
- enum amdgpu_ioctl_svm_access: INACCESSIBLE, IN_PLACE, ALLOW_MIGRATE
- enum amdgpu_ioctl_svm_location: SYSMEM, UNDEFINED
- enum amdgpu_ioctl_svm_attr_type: PREFERRED_LOC, PREFETCH_LOC, ACCESS,
  GRANULARITY, HOST_ACCESS, COHERENT, EXT_COHERENT, HIVE_LOCAL, GPU_RO,
  GPU_EXEC, GPU_READ_MOSTLY
- struct drm_amdgpu_svm_attribute: type and value pair
- struct drm_amdgpu_gem_svm: ioctl payload with start_addr, size,
  operation, nattr, and attrs_ptr

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 include/uapi/drm/amdgpu_drm.h | 106 ++++++++++++++++++++++++++++++++++
 1 file changed, 106 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index b32c72a662b61..4c49cd36f0e77 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -59,6 +59,7 @@ extern "C" {
 #define DRM_AMDGPU_USERQ_WAIT		0x18
 #define DRM_AMDGPU_GEM_LIST_HANDLES	0x19
 #define DRM_AMDGPU_PROC_OPTIONS		0x1A
+#define DRM_AMDGPU_GEM_SVM		0x1B
 
 #define DRM_IOCTL_AMDGPU_GEM_CREATE	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
 #define DRM_IOCTL_AMDGPU_GEM_MMAP	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)
@@ -81,6 +82,7 @@ extern "C" {
 #define DRM_IOCTL_AMDGPU_USERQ_WAIT	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_USERQ_WAIT, struct drm_amdgpu_userq_wait)
 #define DRM_IOCTL_AMDGPU_GEM_LIST_HANDLES DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_LIST_HANDLES, struct drm_amdgpu_gem_list_handles)
 #define DRM_IOCTL_AMDGPU_PROC_OPTIONS	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_PROC_OPTIONS, struct drm_amdgpu_proc_options)
+#define DRM_IOCTL_AMDGPU_GEM_SVM	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_SVM, struct drm_amdgpu_gem_svm)
 
 /**
  * DOC: memory domains
@@ -1694,6 +1696,110 @@ struct drm_amdgpu_proc_options {
 	} kfd_sigbus_delay;
 };
 
+/**
+ * enum amdgpu_ioctl_svm_op - operation selector for DRM_IOCTL_AMDGPU_GEM_SVM.
+ * @AMDGPU_SVM_OP_SET_ATTR: apply the attributes in @attrs_ptr to the VA range.
+ * @AMDGPU_SVM_OP_GET_ATTR: read back the current value of each attribute
+ *                          listed in @attrs_ptr for the given VA range.
+ * @AMDGPU_SVM_OP_RESET_ATTR: reset all attributes for the VA range to their
+ *                            default values. @attrs_ptr and @nattr are ignored.
+ */
+enum amdgpu_ioctl_svm_op {
+	AMDGPU_SVM_OP_SET_ATTR = 0,
+	AMDGPU_SVM_OP_GET_ATTR = 1,
+	AMDGPU_SVM_OP_RESET_ATTR = 2,
+};
+
+/**
+ * enum amdgpu_ioctl_svm_access - values for AMDGPU_SVM_ATTR_ACCESS.
+ * @AMDGPU_SVM_ACCESS_INACCESSIBLE: GPU must not access the range; any access
+ *                                  is a fault.
+ * @AMDGPU_SVM_ACCESS_IN_PLACE:     GPU may access the range only at its
+ *                                  current backing store; the driver will
+ *                                  never migrate pages to local VRAM.
+ * @AMDGPU_SVM_ACCESS_ALLOW_MIGRATE: GPU may access the range and the driver
+ *                                  is allowed (but not required) to migrate
+ *                                  pages between system memory and local
+ *                                  VRAM to satisfy the preferred/prefetch
+ *                                  location.
+ */
+enum amdgpu_ioctl_svm_access {
+	AMDGPU_SVM_ACCESS_INACCESSIBLE		= 0,
+	AMDGPU_SVM_ACCESS_IN_PLACE			= 1,
+	AMDGPU_SVM_ACCESS_ALLOW_MIGRATE		= 2,
+};
+
+/**
+ * enum amdgpu_svm_location - values for AMDGPU_SVM_ATTR_PREFERRED_LOC /
+ *                            AMDGPU_SVM_ATTR_PREFETCH_LOC.
+ * @AMDGPU_SVM_LOCATION_SYSMEM:    back the range with system memory.
+ * @AMDGPU_SVM_LOCATION_UNDEFINED: no preference; the driver chooses.
+ */
+enum amdgpu_ioctl_svm_location {
+	AMDGPU_SVM_LOCATION_SYSMEM	= 0,
+	AMDGPU_SVM_LOCATION_UNDEFINED	= 0xffffffffU,
+};
+
+/**
+ * enum amdgpu_ioctl_svm_attr_type - attribute selector for
+ *                                   &drm_amdgpu_svm_attribute.type.
+ *
+ * @AMDGPU_SVM_ATTR_PREFERRED_LOC: Preferred backing location for the range.
+ *	Value is one of &enum amdgpu_ioctl_svm_location.
+ * @AMDGPU_SVM_ATTR_PREFETCH_LOC: Prefetch target for the range. Value is
+ *	one of &enum amdgpu_ioctl_svm_location.
+ * @AMDGPU_SVM_ATTR_ACCESS: GPU access policy for the range. Value is one
+ *	of &enum amdgpu_ioctl_svm_access.
+ * @AMDGPU_SVM_ATTR_GRANULARITY: log2 of the migration granularity in pages.
+ * @AMDGPU_SVM_ATTR_HOST_ACCESS: Guarantee host access to memory.
+ * @AMDGPU_SVM_ATTR_COHERENT: Fine-grained coherency between all devices
+ *	with access.
+ * @AMDGPU_SVM_ATTR_EXT_COHERENT: Fine-grained coherency between all devices
+ *	using device-scope atomics.
+ * @AMDGPU_SVM_ATTR_HIVE_LOCAL: Use any GPU in the same XGMI hive as the
+ *	preferred device.
+ * @AMDGPU_SVM_ATTR_GPU_RO: GPUs only read the range, allowing replication.
+ * @AMDGPU_SVM_ATTR_GPU_EXEC: Allow execution on GPU.
+ * @AMDGPU_SVM_ATTR_GPU_READ_MOSTLY: GPUs mostly read the range; may allow
+ *	optimizations similar to GPU_RO, but writes still fault.
+ */
+enum amdgpu_ioctl_svm_attr_type {
+	AMDGPU_SVM_ATTR_PREFERRED_LOC		= 0,
+	AMDGPU_SVM_ATTR_PREFETCH_LOC		= 1,
+	AMDGPU_SVM_ATTR_ACCESS				= 2,
+	AMDGPU_SVM_ATTR_GRANULARITY			= 3,
+	/* Boolean attributes below: value must be 0 or 1. */
+	AMDGPU_SVM_ATTR_HOST_ACCESS			= 4,
+	AMDGPU_SVM_ATTR_COHERENT			= 5,
+	AMDGPU_SVM_ATTR_EXT_COHERENT		= 6,
+	AMDGPU_SVM_ATTR_HIVE_LOCAL			= 7,
+	AMDGPU_SVM_ATTR_GPU_RO				= 8,
+	AMDGPU_SVM_ATTR_GPU_EXEC			= 9,
+	AMDGPU_SVM_ATTR_GPU_READ_MOSTLY		= 10,
+};
+
+/* One (type, value) pair carried by DRM_IOCTL_AMDGPU_GEM_SVM. */
+struct drm_amdgpu_svm_attribute {
+	/** AMDGPU_SVM_ATTR_* */
+	__u32 type;
+	/** Attribute value; interpretation depends on @type */
+	__u32 value;
+};
+
+/* Argument for DRM_IOCTL_AMDGPU_GEM_SVM. */
+struct drm_amdgpu_gem_svm {
+	/** Start of the virtual address range */
+	__u64 start_addr;
+	/** Size of the range in bytes */
+	__u64 size;
+	/** AMDGPU_SVM_OP_* */
+	__u32 operation;
+	/** Number of struct drm_amdgpu_svm_attribute entries in @attrs_ptr */
+	__u32 nattr;
+	/** User pointer to an array of @nattr struct drm_amdgpu_svm_attribute */
+	__u64 attrs_ptr;
+};
+
 #if defined(__cplusplus)
 }
 #endif
-- 
2.53.0

[PATCH v9 02/18] drm/amdgpu: add SVM core header and VM integration

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add amdgpu_svm.h with SVM data structures and public API:
- enum amdgpu_svm_xnack_mode: OFF, ON and DEFAULT mode
- struct amdgpu_svm: core context with drm_gpusvm, kref lifecycle,
  attribute tree, rw_semaphore, GC workqueue, xnack state.
- struct amdgpu_svm_gc: garbage collector with workqueue and work_struct
- Debug/trace macros: AMDGPU_SVM_TRACE, AMDGPU_SVM_WARN, AMDGPU_SVM_ERR
- Kmem cache helpers: AMDGPU_SVM_KMEM_CACHE_CREATE/DESTROY
- Locking helpers: amdgpu_svm_lock/unlock/assert_locked
- Public API declarations (including clean_queue, sync_work)

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 204 ++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |   4 +
 2 files changed, 208 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
new file mode 100644
index 0000000000000..f3b4f228405c2
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
@@ -0,0 +1,204 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_H__
+#define __AMDGPU_SVM_H__
+
+#include <drm/amdgpu_drm.h>
+#include <drm/drm_gpusvm.h>
+#include <linux/atomic.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/printk.h>
+#include <linux/rwsem.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+struct amdgpu_device;
+struct amdgpu_vm;
+struct amdgpu_svm_attr_tree;
+struct amdgpu_svm_attrs;
+struct drm_device;
+struct drm_file;
+
+enum amdgpu_svm_xnack_mode {
+	AMDGPU_SVM_XNACK_OFF,
+	AMDGPU_SVM_XNACK_ON,
+	AMDGPU_SVM_XNACK_DEFAULT,
+};
+
+#define AMDGPU_SVM_DBG(fmt, ...) \
+	pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
+
+#define AMDGPU_SVM_WARN(fmt, ...) \
+	pr_warn("%s: " fmt, __func__, ##__VA_ARGS__)
+
+#define AMDGPU_SVM_ERR(fmt, ...) \
+	pr_err("%s: " fmt, __func__, ##__VA_ARGS__)
+
+#define amdgpu_svm_assert_in_notifier(svm__) \
+	lockdep_assert_held_write(&(svm__)->gpusvm.notifier_lock)
+
+/**
+ * struct amdgpu_svm_gc - SVM range garbage collector used for unmapping
+ * SVM ranges in response to mmu_notifier events.
+ * @wq: Workqueue where @work runs on.
+ * @list: Ranges list in garbage collector. Protected by amdgpu_svm.work_lock.
+ * @work: Drains @list under amdgpu_svm.svm_lock.
+ */
+struct amdgpu_svm_gc {
+	struct workqueue_struct *wq;
+	struct list_head list;
+	struct work_struct work;
+};
+
+/**
+ * struct amdgpu_svm - shared virtual memory context
+ * @gpusvm: Embedded drm_gpusvm base, manages the range/notifier trees.
+ * @refcount: Reference count guarding the lifetime of this instance.
+ * @adev: The amdgpu device this SVM context belongs to.
+ * @vm: Back pointer to the owning amdgpu_vm.
+ * @attr_tree: Interval tree of SVM attributes.
+ * @svm_lock: The driver_svm_lock described in the GPU SVM locking section.
+ *            registered by drm_gpusvm_driver_set_lock().
+ * @work_lock: Protects the deferred work queues such as @gc.list.
+ * @gc: garbage collector for ranges queued for destruction.
+ * @exiting: Set once when the context starts tearing down.
+ * @checkpoint_ts: IH ring timestamp captured on unmap, used to drop stale
+ *                 faults.
+ * @default_granularity: Default range granularity.
+ * @xnack_enabled: Whether retry (XNACK) faults are enabled.
+ * @invalidate_ranges: Callback of the MMU notifier invalidate event.
+ */
+struct amdgpu_svm {
+	struct drm_gpusvm gpusvm;
+	struct kref refcount;
+	struct amdgpu_device *adev;
+	struct amdgpu_vm *vm;
+	struct amdgpu_svm_attr_tree *attr_tree;
+	struct rw_semaphore svm_lock;
+	spinlock_t work_lock;
+	struct amdgpu_svm_gc gc;
+	atomic_t exiting;
+	uint64_t checkpoint_ts;
+	u8 default_granularity;
+	bool xnack_enabled;
+	void (*invalidate_ranges)(struct amdgpu_svm *svm,
+				  struct drm_gpusvm_notifier *notifier,
+				  const struct mmu_notifier_range *mmu_range,
+				  struct drm_gpusvm_range *first,
+				  uint64_t adj_start, uint64_t adj_end);
+};
+
+static inline struct amdgpu_svm *to_amdgpu_svm(struct drm_gpusvm *gpusvm)
+{
+	return container_of(gpusvm, struct amdgpu_svm, gpusvm);
+}
+
+/*
+ * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
+ * Hold it in write mode around structural GPU SVM updates, including
+ * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
+ */
+static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
+{
+	down_write(&svm->svm_lock);
+}
+
+static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
+{
+	up_write(&svm->svm_lock);
+}
+
+static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
+{
+	lockdep_assert_held_write(&svm->svm_lock);
+}
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
+
+int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
+void amdgpu_svm_close(struct amdgpu_vm *vm);
+void amdgpu_svm_fini(struct amdgpu_vm *vm);
+
+void amdgpu_svm_put(struct amdgpu_svm *svm);
+struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
+					       uint32_t pasid);
+int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
+			    uint64_t fault_page, uint64_t ts,
+			    bool write_fault);
+bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
+
+int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
+			 struct drm_file *filp);
+void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
+			    struct list_head *work_list);
+void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
+int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
+int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
+				 const struct amdgpu_svm_attrs *old_attrs,
+				 const struct amdgpu_svm_attrs *new_attrs,
+				 unsigned long start_page,
+				 unsigned long last_page);
+bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
+#else
+static inline int amdgpu_svm_init(struct amdgpu_device *adev,
+				  struct amdgpu_vm *vm)
+{
+	return 0;
+}
+
+static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
+{
+}
+
+static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
+{
+}
+
+static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
+					  uint32_t pasid,
+					  uint64_t fault_page,
+					  uint64_t ts,
+					  bool write_fault)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
+{
+	return false;
+}
+
+static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
+				       struct drm_file *filp)
+{
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_DRM_AMDGPU_SVM */
+
+#endif /* __AMDGPU_SVM_H__ */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index ec1196d390bb7..30463a83e2e60 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -43,6 +43,7 @@ struct amdgpu_bo_va;
 struct amdgpu_job;
 struct amdgpu_bo_list_entry;
 struct amdgpu_bo_vm;
+struct amdgpu_svm;
 
 /*
  * GPUVM handling
@@ -373,6 +374,9 @@ struct amdgpu_vm {
 
 	/* cached fault info */
 	struct amdgpu_vm_fault_info fault_info;
+
+	/* SVM experimental implementation */
+	struct amdgpu_svm *svm;
 };
 
 struct amdgpu_vm_manager {
-- 
2.53.0

[PATCH v9 03/18] drm/amdgpu: implement SVM attribute tree and helper functions

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add amdgpu_svm_attr.h with the SVM attribute types and tree
infrastructure together with the helper implementation in
amdgpu_svm_attr.c that uses them. The header is squashed into the
implementation patch so it does not stand alone as declarations
without an implementation.

amdgpu_svm_attr.h:
- Internal flag bitmask definitions mapped from UAPI attr types
- PTE_FLAG_MASK and MAPPING_FLAG_MASK for change detection
- struct amdgpu_svm_attrs: user set attribute range
- struct amdgpu_svm_attr_range: interval tree node with attrs
- struct amdgpu_svm_attr_tree: mutex protected RB tree for store and
  search
- enum amdgpu_svm_attr_change_trigger: change flags of user attribute
  changes

amdgpu_svm_attr.c:
- Default attribute initialization: amdgpu_svm_attr_set_default
- Device memory and VRAM preference helpers
- VMA validity checker: amdgpu_svm_check_vma
- Attribute equality comparison: attr_equal
- Interval tree CRUD operations: find, get_bounds,
  alloc, insert, and remove
- attr_set_interval helper for range boundary updates

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 235 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h | 183 +++++++++++++++
 2 files changed, 418 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
new file mode 100644
index 0000000000000..9d3519776c9c8
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
@@ -0,0 +1,235 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/gfp.h>
+#include <linux/lockdep.h>
+#include <linux/minmax.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+struct attr_set_ctx {
+	struct amdgpu_svm_attrs old_attrs;
+	struct amdgpu_svm_attrs new_attrs;
+	unsigned long start_page;
+	unsigned long last_page;
+};
+
+struct attr_get_ctx {
+	int32_t preferred_loc;
+	int32_t prefetch_loc;
+	enum amdgpu_ioctl_svm_access access;
+	uint32_t granularity;
+	uint32_t flags_and;
+	bool has_range;
+};
+
+bool amdgpu_svm_attr_prefer_vram(const struct amdgpu_svm_attrs *attrs)
+{
+	if (attrs->preferred_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
+	    attrs->preferred_loc != AMDGPU_SVM_LOCATION_SYSMEM)
+		return true;
+
+	if (attrs->prefetch_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
+	    attrs->prefetch_loc != AMDGPU_SVM_LOCATION_SYSMEM)
+		return true;
+
+	return false;
+}
+
+struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
+					unsigned long addr)
+{
+	const unsigned long unsupported_vm_flags = VM_IO | VM_PFNMAP |
+						   VM_MIXEDMAP;
+	struct vm_area_struct *vma = vma_lookup(mm, addr);
+
+	if (!vma)
+		return ERR_PTR(-EFAULT);
+
+	if (vma->vm_flags & unsupported_vm_flags)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	return vma;
+}
+
+static void attr_set_interval(struct amdgpu_svm_attr_range *range,
+				unsigned long start_page,
+				unsigned long last_page)
+{
+	range->it_node.start = start_page;
+	range->it_node.last = last_page;
+}
+
+void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
+				 struct amdgpu_svm_attrs *attrs)
+{
+	attrs->preferred_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	attrs->prefetch_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	attrs->granularity = svm->default_granularity;
+	attrs->flags = AMDGPU_SVM_ATTR_BIT_HOST_ACCESS | AMDGPU_SVM_ATTR_BIT_COHERENT;
+	attrs->access = svm->xnack_enabled ?
+		AMDGPU_SVM_ACCESS_ALLOW_MIGRATE : AMDGPU_SVM_ACCESS_INACCESSIBLE;
+}
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
+			   unsigned long page)
+{
+	struct interval_tree_node *node;
+
+	node = interval_tree_iter_first(&attr_tree->tree, page, page);
+	if (node)
+		return container_of(node, struct amdgpu_svm_attr_range, it_node);
+
+	return NULL;
+}
+
+/**
+ * amdgpu_svm_attr_get_bounds_locked() - Find attributes  or surrounding bounds
+ * @attr_tree: attribute tree to search.
+ * @page: page index to look up.
+ * @start_page: out, first page of the returned range or gap.
+ * @last_page: out, last page of the returned range or gap.
+ *
+ * If @page is covered by an attribute range, return that range and report its
+ * bounds. Otherwise return NULL and fill @start_page/@last_page with the bounds
+ * of the default attribute gap around @page, clamped by neighboring explicit
+ * ranges or [0, ULONG_MAX] when none.
+ *
+ * Return: the covering range, or NULL if @page falls in a gap.
+ */
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
+				  unsigned long page,
+				  unsigned long *start_page,
+				  unsigned long *last_page)
+{
+	struct amdgpu_svm_attr_range *attr_range;
+	struct interval_tree_node *node;
+	struct rb_node *rb;
+
+	attr_range = amdgpu_svm_attr_find_locked(attr_tree, page);
+	if (attr_range) {
+		*start_page = amdgpu_svm_attr_start_page(attr_range);
+		*last_page = amdgpu_svm_attr_last_page(attr_range);
+		return attr_range;
+	}
+
+	*start_page = 0;
+	*last_page = ULONG_MAX;
+
+	if (page == ULONG_MAX)
+		return NULL;
+
+	node = interval_tree_iter_first(&attr_tree->tree, page + 1, ULONG_MAX);
+	if (node) {
+		if (node->start > page)
+			*last_page = node->start - 1;
+
+		rb = rb_prev(&node->rb);
+		if (rb) {
+			node = container_of(rb, struct interval_tree_node, rb);
+			if (node->last < page)
+				*start_page = node->last + 1;
+		}
+	} else {
+		rb = rb_last(&attr_tree->tree.rb_root);
+
+		if (rb) {
+			node = container_of(rb, struct interval_tree_node, rb);
+			if (node->last < page)
+				*start_page = node->last + 1;
+		}
+	}
+
+	return NULL;
+}
+
+static bool attr_equal(const struct amdgpu_svm_attrs *a,
+				 const struct amdgpu_svm_attrs *b)
+{
+	return a->flags == b->flags &&
+	       a->preferred_loc == b->preferred_loc &&
+	       a->prefetch_loc == b->prefetch_loc &&
+		       a->granularity == b->granularity &&
+		       a->access == b->access;
+}
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_range_alloc(unsigned long start_page,
+			   unsigned long last_page,
+			   const struct amdgpu_svm_attrs *attrs)
+{
+	struct amdgpu_svm_attr_range *range;
+
+	range = kzalloc(sizeof(*range), GFP_KERNEL);
+	if (!range)
+		return NULL;
+
+	INIT_LIST_HEAD(&range->list);
+	attr_set_interval(range, start_page, last_page);
+	range->attrs = *attrs;
+	return range;
+}
+
+void amdgpu_svm_attr_range_insert_locked(struct amdgpu_svm_attr_tree *attr_tree,
+					 struct amdgpu_svm_attr_range *range)
+{
+	struct interval_tree_node *node;
+	struct amdgpu_svm_attr_range *next;
+
+	lockdep_assert_held(&attr_tree->lock);
+
+	/*
+	 * Keep @range_list ordered by start page: insert before the first range
+	 * starting at or after @range, then add to the interval tree.
+	 */
+	node = interval_tree_iter_first(&attr_tree->tree, amdgpu_svm_attr_start_page(range),
+					ULONG_MAX);
+	if (node) {
+		next = container_of(node, struct amdgpu_svm_attr_range, it_node);
+		list_add_tail(&range->list, &next->list);
+	} else {
+		list_add_tail(&range->list, &attr_tree->range_list);
+	}
+
+	interval_tree_insert(&range->it_node, &attr_tree->tree);
+}
+
+static void attr_remove_range_locked(struct amdgpu_svm_attr_tree *attr_tree,
+					  struct amdgpu_svm_attr_range *range,
+					  bool free_range)
+{
+	lockdep_assert_held(&attr_tree->lock);
+
+	interval_tree_remove(&range->it_node, &attr_tree->tree);
+	list_del_init(&range->list);
+	if (free_range)
+		kfree(range);
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
new file mode 100644
index 0000000000000..0f712536a5dc1
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
@@ -0,0 +1,183 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_ATTR_H__
+#define __AMDGPU_SVM_ATTR_H__
+
+#include <drm/amdgpu_drm.h>
+#include <linux/interval_tree.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/rbtree.h>
+#include <linux/types.h>
+
+/* Internal SVM attribute bitmask flags mapped from UAPI ioctl definitions */
+#define AMDGPU_SVM_ATTR_BIT_HOST_ACCESS		(1u << 0)
+#define AMDGPU_SVM_ATTR_BIT_COHERENT		(1u << 1)
+#define AMDGPU_SVM_ATTR_BIT_EXT_COHERENT	(1u << 2)
+#define AMDGPU_SVM_ATTR_BIT_HIVE_LOCAL		(1u << 3)
+#define AMDGPU_SVM_ATTR_BIT_GPU_RO		(1u << 4)
+#define AMDGPU_SVM_ATTR_BIT_GPU_EXEC		(1u << 5)
+#define AMDGPU_SVM_ATTR_BIT_GPU_READ_MOSTLY	(1u << 6)
+
+#define AMDGPU_SVM_PTE_FLAG_MASK \
+	(AMDGPU_SVM_ATTR_BIT_COHERENT | AMDGPU_SVM_ATTR_BIT_EXT_COHERENT | \
+	 AMDGPU_SVM_ATTR_BIT_GPU_RO | AMDGPU_SVM_ATTR_BIT_GPU_EXEC)
+
+#define AMDGPU_SVM_MAPPING_FLAG_MASK \
+	(AMDGPU_SVM_ATTR_BIT_HOST_ACCESS | AMDGPU_SVM_ATTR_BIT_HIVE_LOCAL | \
+	 AMDGPU_SVM_ATTR_BIT_GPU_READ_MOSTLY)
+
+/**
+ * struct amdgpu_svm_attrs - SVM attributes for an address range
+ * @preferred_loc: Preferred backing location, using AMDGPU_SVM_LOCATION_*.
+ * @prefetch_loc: Target location for prefetch requests, using
+ *                AMDGPU_SVM_LOCATION_*.
+ * @flags: Internal AMDGPU_SVM_ATTR_BIT_* flags mapped from the UAPI.
+ * @granularity: Mapping granularity encoded as a page order.
+ * @access: CPU/GPU access policy from the SVM UAPI.
+ */
+struct amdgpu_svm_attrs {
+	int32_t preferred_loc;
+	int32_t prefetch_loc;
+	uint32_t flags;
+	uint32_t granularity;
+	enum amdgpu_ioctl_svm_access access;
+};
+
+/**
+ * struct amdgpu_svm_attr_range - a range of user attributes
+ * @it_node: interval tree node keyed by [start, last] page index.
+ * @list: links the range into amdgpu_svm_attr_tree.range_list in address order.
+ * @attrs: the attributes applied to this range.
+ */
+struct amdgpu_svm_attr_range {
+	struct interval_tree_node it_node;
+	struct list_head list;
+	struct amdgpu_svm_attrs attrs;
+};
+
+static inline unsigned long
+amdgpu_svm_attr_start_page(const struct amdgpu_svm_attr_range *range)
+{
+	return range->it_node.start;
+}
+
+static inline unsigned long
+amdgpu_svm_attr_last_page(const struct amdgpu_svm_attr_range *range)
+{
+	return range->it_node.last;
+}
+
+static inline unsigned long
+amdgpu_svm_attr_start(const struct amdgpu_svm_attr_range *range)
+{
+	return range->it_node.start << PAGE_SHIFT;
+}
+
+static inline unsigned long
+amdgpu_svm_attr_end(const struct amdgpu_svm_attr_range *range)
+{
+	return (range->it_node.last + 1) << PAGE_SHIFT;
+}
+
+struct amdgpu_svm;
+struct mm_struct;
+struct vm_area_struct;
+
+static inline bool
+amdgpu_svm_attr_has_access(enum amdgpu_ioctl_svm_access access)
+{
+	return access == AMDGPU_SVM_ACCESS_ALLOW_MIGRATE ||
+	       access == AMDGPU_SVM_ACCESS_IN_PLACE;
+}
+
+/**
+ * struct amdgpu_svm_attr_tree - per-SVM store of user attribute ranges
+ * @lock: protects @tree and @range_list.
+ * @tree: interval tree of struct amdgpu_svm_attr_range for fast lookup.
+ * @range_list: address-ordered list of the same ranges, kept in sync with
+ *              @tree to allow ordered traversal during modification.
+ * @svm: back pointer to the owning SVM instance.
+ */
+struct amdgpu_svm_attr_tree {
+	struct mutex lock;
+	struct rb_root_cached tree;
+	struct list_head range_list;
+	struct amdgpu_svm *svm;
+};
+
+/**
+ * enum amdgpu_svm_attr_change_trigger - effects caused by an attribute change
+ * @AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE: Access policy changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE: GPU PTE permission/cache bits changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE: Mapping policy bits changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE: Preferred or prefetch location changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE: Range granularity changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_PREFETCH: New attributes request a VRAM prefetch.
+ *
+ * Bitmask describing what an attribute update touched. It drives whether the
+ * existing GPU mappings must be invalidated and/or remapped.
+ */
+enum amdgpu_svm_attr_change_trigger {
+	AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE = (1U << 0),
+	AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE = (1U << 1),
+	AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE = (1U << 2),
+	AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE = (1U << 3),
+	AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE = (1U << 4),
+	AMDGPU_SVM_ATTR_TRIGGER_PREFETCH = (1U << 5),
+};
+
+/*
+ * Attribute changes that require invalidating existing GPU mappings.
+ * A granularity-only change does not, so it is intentionally excluded.
+ */
+#define AMDGPU_SVM_ATTR_TRIGGER_NEED_INVALIDATE \
+	(AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE | \
+	 AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE | \
+	 AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE | \
+	 AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE)
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
+			   unsigned long page);
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
+				  unsigned long page,
+				  unsigned long *start_page,
+				  unsigned long *last_page);
+void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
+				 struct amdgpu_svm_attrs *attrs);
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_range_alloc(unsigned long start_page,
+			   unsigned long last_page,
+			   const struct amdgpu_svm_attrs *attrs);
+void amdgpu_svm_attr_range_insert_locked(struct amdgpu_svm_attr_tree *attr_tree,
+					 struct amdgpu_svm_attr_range *range);
+bool amdgpu_svm_attr_prefer_vram(const struct amdgpu_svm_attrs *attrs);
+struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
+					unsigned long addr);
+
+#endif /* __AMDGPU_SVM_ATTR_H__ */
-- 
2.53.0

[PATCH v9 04/18] drm/amdgpu: implement SVM attribute set/get/clear operations

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Complete attribute subsystem with validation, tree modification, and
public API:
- Attribute validation: per-type checks for preferred_loc, prefetch_loc,
  access, flags, granularity
- Flag application: attr_apply_flag, amdgpu_svm_attr_apply
- VMA range validation: amdgpu_svm_attr_validate_range_vma
- VM BO overlap detection: amdgpu_svm_attr_check_vm_bo
- Tree modification with gap/overlap handling:
  amdgpu_svm_attr_set_hole (new range in gap),
  amdgpu_svm_attr_set_existing (split/update existing range)
- amdgpu_svm_attr_set_range: cursor based iteration with per segment
- Lifecycle: amdgpu_svm_attr_tree_create/destroy
- Public API: amdgpu_svm_attr_set (validate, set_range with retry),
  amdgpu_svm_attr_get (aggregate attrs across interval),
  amdgpu_svm_attr_clear (interval removal with split),
  amdgpu_svm_attr_reset

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 713 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h |  19 +
 2 files changed, 732 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
index 9d3519776c9c8..0d54aa7f5826f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
@@ -78,6 +78,43 @@ struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
 	return vma;
 }
 
+/*
+ * amdgpu_svm_attr_range_is_valid - validate an attr ioctl address range
+ *
+ * Synchronous input validation for the SVM set/get-attr ioctls: walk the
+ * CPU VMAs covering [start_page, last_page] and reject the request if any
+ * part is unmapped or backed by an unsupported VMA (-EOPNOTSUPP).
+ *
+ */
+static int amdgpu_svm_attr_range_is_valid(struct amdgpu_svm *svm,
+					  unsigned long start_page,
+					  unsigned long last_page)
+{
+	struct mm_struct *mm = svm->gpusvm.mm;
+	struct vm_area_struct *vma;
+	unsigned long start, end;
+	int r = 0;
+
+	if (!mm)
+		return -EFAULT;
+
+	start = start_page << PAGE_SHIFT;
+	end = (last_page + 1) << PAGE_SHIFT;
+
+	mmap_read_lock(mm);
+	do {
+		vma = amdgpu_svm_check_vma(mm, start);
+		if (IS_ERR(vma)) {
+			r = PTR_ERR(vma);
+			break;
+		}
+		start = min(end, vma->vm_end);
+	} while (start < end);
+	mmap_read_unlock(mm);
+
+	return r;
+}
+
 static void attr_set_interval(struct amdgpu_svm_attr_range *range,
 				unsigned long start_page,
 				unsigned long last_page)
@@ -233,3 +270,679 @@ static void attr_remove_range_locked(struct amdgpu_svm_attr_tree *attr_tree,
 	if (free_range)
 		kfree(range);
 }
+
+static void attr_change_ctx_set(
+		struct attr_set_ctx *change,
+		const struct amdgpu_svm_attrs *old_attrs,
+		const struct amdgpu_svm_attrs *new_attrs,
+		unsigned long start_page,
+		unsigned long last_page)
+{
+	change->old_attrs = *old_attrs;
+	change->new_attrs = *new_attrs;
+	change->start_page = start_page;
+	change->last_page = last_page;
+}
+
+static inline int attr_check_preferred_loc(uint32_t value)
+{
+	/*
+	 * One SVM context maps to a single GPU, so any non-sysmem,
+	 * non-undefined value is interpreted as this GPU.
+	 */
+	if (value == AMDGPU_SVM_LOCATION_SYSMEM || value == AMDGPU_SVM_LOCATION_UNDEFINED)
+		return 0;
+
+	return 0;
+}
+
+static inline int attr_check_prefetch_loc(uint32_t value)
+{
+	/* cause one svm one gpu so value > 0 then means prefetch loc is this GPU
+	 * keep prefetch loc to adapt to KFD API
+	 */
+	if (value == AMDGPU_SVM_LOCATION_SYSMEM)
+		return 0;
+
+	if (value == AMDGPU_SVM_LOCATION_UNDEFINED)
+		return -EINVAL;
+
+	return 0;
+}
+
+static inline int attr_check_access(uint32_t value)
+{
+	switch (value) {
+	case AMDGPU_SVM_ACCESS_INACCESSIBLE:
+	case AMDGPU_SVM_ACCESS_IN_PLACE:
+	case AMDGPU_SVM_ACCESS_ALLOW_MIGRATE:
+		return 0;
+	default:
+		return -EINVAL;
+	}
+}
+
+static bool attr_flag_type_to_bit(uint32_t type, uint32_t *flag_bit)
+{
+	if (type < AMDGPU_SVM_ATTR_HOST_ACCESS ||
+	    type > AMDGPU_SVM_ATTR_GPU_READ_MOSTLY)
+		return false;
+
+	*flag_bit = 1u << (type - AMDGPU_SVM_ATTR_HOST_ACCESS);
+	return true;
+}
+
+static inline int attr_check_flag_value(uint32_t value)
+{
+	if (value > 1)
+		return -EINVAL;
+
+	return 0;
+}
+
+static inline int attr_check_flag_attr(uint32_t type, uint32_t value)
+{
+	uint32_t flag_bit;
+	int ret;
+
+	if (!attr_flag_type_to_bit(type, &flag_bit))
+		return -EINVAL;
+
+	ret = attr_check_flag_value(value);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static inline int attr_check_granularity(uint32_t value)
+{
+	return 0;
+}
+
+static int attr_set_validate(const struct drm_amdgpu_svm_attribute *attr)
+{
+	switch (attr->type) {
+	case AMDGPU_SVM_ATTR_PREFERRED_LOC:
+		return attr_check_preferred_loc(attr->value);
+	case AMDGPU_SVM_ATTR_PREFETCH_LOC:
+		return attr_check_prefetch_loc(attr->value);
+	case AMDGPU_SVM_ATTR_ACCESS:
+		return attr_check_access(attr->value);
+	case AMDGPU_SVM_ATTR_GRANULARITY:
+		return attr_check_granularity(attr->value);
+	case AMDGPU_SVM_ATTR_HOST_ACCESS:
+	case AMDGPU_SVM_ATTR_COHERENT:
+	case AMDGPU_SVM_ATTR_HIVE_LOCAL:
+	case AMDGPU_SVM_ATTR_GPU_RO:
+	case AMDGPU_SVM_ATTR_GPU_EXEC:
+	case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
+	case AMDGPU_SVM_ATTR_EXT_COHERENT:
+		return attr_check_flag_attr(attr->type, attr->value);
+	default:
+		return -EINVAL;
+	}
+}
+
+static void attr_apply_flag(struct amdgpu_svm_attrs *attrs,
+			    uint32_t type, uint32_t value)
+{
+	uint32_t flag_bit;
+
+	if (!attr_flag_type_to_bit(type, &flag_bit))
+		return;
+
+	if (value)
+		attrs->flags |= flag_bit;
+	else
+		attrs->flags &= ~flag_bit;
+}
+
+static void attr_apply(struct amdgpu_svm_attrs *attrs,
+					uint32_t nattr,
+					const struct drm_amdgpu_svm_attribute *pattrs)
+{
+	const struct drm_amdgpu_svm_attribute *attr;
+
+	for (attr = pattrs; nattr--; attr++) {
+		switch (attr->type) {
+		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
+			attrs->preferred_loc = (int32_t)attr->value;
+			break;
+		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
+			attrs->prefetch_loc = (int32_t)attr->value;
+			break;
+		case AMDGPU_SVM_ATTR_ACCESS:
+			attrs->access = (enum amdgpu_ioctl_svm_access)attr->value;
+			break;
+		case AMDGPU_SVM_ATTR_HOST_ACCESS:
+		case AMDGPU_SVM_ATTR_COHERENT:
+		case AMDGPU_SVM_ATTR_HIVE_LOCAL:
+		case AMDGPU_SVM_ATTR_GPU_RO:
+		case AMDGPU_SVM_ATTR_GPU_EXEC:
+		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
+		case AMDGPU_SVM_ATTR_EXT_COHERENT:
+			attr_apply_flag(attrs, attr->type, attr->value);
+			break;
+		case AMDGPU_SVM_ATTR_GRANULARITY:
+			attrs->granularity = min_t(uint32_t, attr->value, 0x3f);
+			break;
+		default:
+			break;
+		}
+	}
+}
+
+static bool attr_same_attrs(const struct amdgpu_svm_attr_range *range,
+			    uint32_t nattr,
+			    const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm_attrs target;
+
+	target = range->attrs;
+	attr_apply(&target, nattr, attrs);
+	return attr_equal(&range->attrs, &target);
+}
+
+static int
+amdgpu_svm_attr_set_hole(struct amdgpu_svm_attr_tree *attr_tree,
+			  const struct amdgpu_svm_attrs *default_attrs,
+			  unsigned long start_page, unsigned long last_page,
+			  uint32_t nattr,
+			  const struct drm_amdgpu_svm_attribute *attrs,
+			  struct attr_set_ctx *change)
+{
+	struct amdgpu_svm_attrs new_attrs;
+	struct amdgpu_svm_attr_range *range;
+
+	lockdep_assert_held(&attr_tree->lock);
+
+	if (start_page > last_page)
+		return 0;
+
+	new_attrs = *default_attrs;
+	attr_apply(&new_attrs, nattr, attrs);
+
+	/* Always create a range entry even when attrs equal defaults */
+	range = amdgpu_svm_attr_range_alloc(start_page, last_page, &new_attrs);
+	if (!range)
+		return -ENOMEM;
+
+	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+
+	attr_change_ctx_set(change, default_attrs,
+				       &new_attrs, start_page, last_page);
+	return 0;
+}
+
+/*
+ * amdgpu_svm_attr_set_existing() - Apply attrs to part of an existing range
+ *
+ * Update @range over [@start_page, @last_page]. If the request fully covers
+ * @range the attrs are updated in place, otherwise the range is split into the
+ * head/tail, carrying the old attrs, and the covered middle, carrying
+ * the new attrs. The resulting change is reported by @change.
+ */
+static int
+amdgpu_svm_attr_set_existing(struct amdgpu_svm_attr_tree *attr_tree,
+			     struct amdgpu_svm_attr_range *range,
+			     unsigned long start_page, unsigned long last_page,
+			     uint32_t nattr,
+			     const struct drm_amdgpu_svm_attribute *attrs,
+			     struct attr_set_ctx *change)
+{
+	unsigned long range_start = amdgpu_svm_attr_start_page(range);
+	unsigned long range_last = amdgpu_svm_attr_last_page(range);
+	struct amdgpu_svm_attr_range *left = NULL;
+	struct amdgpu_svm_attr_range *right = NULL;
+	struct amdgpu_svm_attrs old_attrs;
+	struct amdgpu_svm_attrs new_attrs;
+
+	lockdep_assert_held(&attr_tree->lock);
+
+	old_attrs = range->attrs;
+
+	if (attr_same_attrs(range, nattr, attrs)) {
+		/* Report old==new so apply_attr_change can decide */
+		attr_change_ctx_set(change, &old_attrs,
+					       &old_attrs,
+					       start_page, last_page);
+		return 0;
+	}
+
+	new_attrs = old_attrs;
+	attr_apply(&new_attrs, nattr, attrs);
+
+	/* only need to update attr */
+	if (start_page == range_start && last_page == range_last) {
+		range->attrs = new_attrs;
+		attr_change_ctx_set(change, &old_attrs,
+					       &new_attrs, start_page, last_page);
+		return 0;
+	}
+
+	/* split head */
+	if (start_page > range_start) {
+		left = amdgpu_svm_attr_range_alloc(range_start, start_page - 1, &old_attrs);
+		if (!left)
+			return -ENOMEM;
+	}
+
+	/* split tail */
+	if (last_page < range_last) {
+		right = amdgpu_svm_attr_range_alloc(last_page + 1, range_last, &old_attrs);
+		if (!right) {
+			kfree(left);
+			return -ENOMEM;
+		}
+	}
+
+	attr_remove_range_locked(attr_tree, range, false);
+	if (left)
+		amdgpu_svm_attr_range_insert_locked(attr_tree, left);
+	attr_set_interval(range, start_page, last_page);
+	range->attrs = new_attrs;
+	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+	if (right)
+		amdgpu_svm_attr_range_insert_locked(attr_tree, right);
+
+	attr_change_ctx_set(change, &old_attrs,
+				       &new_attrs, start_page, last_page);
+	return 0;
+}
+
+/*
+ * amdgpu_svm_attr_set_range() - Apply attr changes across [@start_page, @last_page]
+ *
+ * Walk the interval in segments, each segment being either an existing range
+ * or a hole.
+ */
+static int
+amdgpu_svm_attr_set_range(struct amdgpu_svm_attr_tree *attr_tree,
+			  const struct amdgpu_svm_attrs *default_attrs,
+			  unsigned long start_page, unsigned long last_page,
+			  uint32_t nattr,
+			  const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm *svm = attr_tree->svm;
+	unsigned long cursor = start_page;
+	bool need_retry = false;
+
+	while (cursor <= last_page) {
+		struct interval_tree_node *node;
+		unsigned long seg_last;
+		struct attr_set_ctx change = { 0 };
+		int ret;
+
+		amdgpu_svm_lock(svm);
+		mutex_lock(&attr_tree->lock);
+		node = interval_tree_iter_first(&attr_tree->tree, cursor, cursor);
+		if (node) {
+			struct amdgpu_svm_attr_range *range;
+
+			range = container_of(node, struct amdgpu_svm_attr_range, it_node);
+			seg_last = min(last_page, amdgpu_svm_attr_last_page(range));
+			ret = amdgpu_svm_attr_set_existing(attr_tree, range,
+								   cursor, seg_last,
+								   nattr, attrs, &change);
+		} else {
+			struct interval_tree_node *next;
+
+			seg_last = last_page;
+			if (cursor != ULONG_MAX) {
+				next = interval_tree_iter_first(&attr_tree->tree,
+								cursor + 1,
+								ULONG_MAX);
+				if (next) {
+					struct amdgpu_svm_attr_range *next_range;
+
+					next_range = container_of(next,
+						struct amdgpu_svm_attr_range,
+						it_node);
+					seg_last = min(last_page,
+						       amdgpu_svm_attr_start_page(next_range) - 1);
+				}
+			}
+			ret = amdgpu_svm_attr_set_hole(attr_tree,
+							       default_attrs,
+							       cursor, seg_last,
+							       nattr, attrs,
+							       &change);
+		}
+		mutex_unlock(&attr_tree->lock);
+
+		if (ret) {
+			amdgpu_svm_unlock(svm);
+			return ret;
+		}
+
+		ret = amdgpu_svm_apply_attr_change(svm,
+						   &change.old_attrs,
+						   &change.new_attrs,
+						   change.start_page,
+						   change.last_page);
+		amdgpu_svm_unlock(svm);
+
+		if (ret == -EAGAIN) {
+			need_retry = true;
+			ret = 0;
+		}
+
+		if (ret)
+			return ret;
+
+		if (seg_last == ULONG_MAX || seg_last == last_page)
+			break;
+
+		cursor = seg_last + 1;
+	}
+
+	return need_retry ? -EAGAIN : 0;
+}
+
+struct amdgpu_svm_attr_tree *
+amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm)
+{
+	struct amdgpu_svm_attr_tree *attr_tree;
+
+	attr_tree = kzalloc(sizeof(*attr_tree), GFP_KERNEL);
+	if (!attr_tree)
+		return NULL;
+
+	mutex_init(&attr_tree->lock);
+	attr_tree->tree = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&attr_tree->range_list);
+	attr_tree->svm = svm;
+	return attr_tree;
+}
+
+void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree)
+{
+	struct amdgpu_svm_attr_range *range, *tmp;
+
+	if (!attr_tree)
+		return;
+
+	mutex_lock(&attr_tree->lock);
+	list_for_each_entry_safe(range, tmp, &attr_tree->range_list, list) {
+		interval_tree_remove(&range->it_node, &attr_tree->tree);
+		list_del_init(&range->list);
+		kfree(range);
+	}
+	mutex_unlock(&attr_tree->lock);
+
+	mutex_destroy(&attr_tree->lock);
+	kfree(attr_tree);
+}
+
+int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
+			uint64_t start,
+			uint64_t size,
+			uint32_t nattr,
+			const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm *svm = attr_tree->svm;
+	struct amdgpu_svm_attrs default_attrs;
+	unsigned long start_page, last_page;
+	uint32_t i;
+	int r;
+
+	start_page = start >> PAGE_SHIFT;
+	last_page = (start + size - 1) >> PAGE_SHIFT;
+
+	for (i = 0; i < nattr; i++) {
+		AMDGPU_SVM_DBG("set attr type %u value 0x%08x range [0x%lx-0x%lx] xnack:%d",
+			 attrs[i].type, attrs[i].value, start_page, last_page,
+			 svm->xnack_enabled ? 1 : 0);
+		r = attr_set_validate(&attrs[i]);
+		if (r) {
+			AMDGPU_SVM_DBG("invalid attribute %u value 0x%08x",
+					 attrs[i].type, attrs[i].value);
+			return r;
+		}
+	}
+
+	r = amdgpu_svm_attr_range_is_valid(svm, start_page, last_page);
+	if (r)
+		return r;
+
+	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
+
+retry:
+	r = amdgpu_svm_attr_set_range(attr_tree, &default_attrs,
+					       start_page, last_page,
+					       nattr, attrs);
+	if (r == -EAGAIN) {
+		AMDGPU_SVM_DBG("attr_set retry [0x%lx-0x%lx]\n",
+				 start_page, last_page);
+		amdgpu_svm_sync_work(svm);
+		cond_resched();
+		goto retry;
+	}
+
+	return r;
+}
+
+int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page)
+{
+	struct interval_tree_node *node;
+	int r = 0;
+
+	if (start_page > last_page)
+		return -EINVAL;
+
+	mutex_lock(&attr_tree->lock);
+
+	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
+	while (node) {
+		struct interval_tree_node *next;
+		struct amdgpu_svm_attr_range *range;
+		unsigned long range_start;
+		unsigned long range_last;
+
+		range = container_of(node, struct amdgpu_svm_attr_range, it_node);
+		next = interval_tree_iter_next(node, start_page, last_page);
+		range_start = amdgpu_svm_attr_start_page(range);
+		range_last = amdgpu_svm_attr_last_page(range);
+
+		if (range_start < start_page && range_last > last_page) {
+			struct amdgpu_svm_attr_range *tail;
+
+			tail = amdgpu_svm_attr_range_alloc(last_page + 1, range_last,
+							   &range->attrs);
+			if (!tail) {
+				r = -ENOMEM;
+				break;
+			}
+
+			attr_remove_range_locked(attr_tree, range, false);
+			attr_set_interval(range, range_start, start_page - 1);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, tail);
+		} else if (range_start < start_page) {
+			attr_remove_range_locked(attr_tree, range, false);
+			attr_set_interval(range, range_start, start_page - 1);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+		} else if (range_last > last_page) {
+			attr_remove_range_locked(attr_tree, range, false);
+			attr_set_interval(range, last_page + 1, range_last);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+		} else {
+			attr_remove_range_locked(attr_tree, range, true);
+		}
+
+		node = next;
+	}
+
+	mutex_unlock(&attr_tree->lock);
+	return r;
+}
+
+int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page)
+{
+	/*
+	 * Range with no attr node in this implementation is treated the
+	 * same as one that explicitly stores default attrs. So clear
+	 * the attribute ranges when user wants to reset the attrs.
+	 *
+	 * - GET_ATTR: return default_attrs to userspace when there is no attr
+	 *   range.
+	 * - SET_ATTR: holes are treated as having default_attrs when
+	 *   computing the change trigger.
+	 * - Fault: attr lookup falls back to defaults when no attr range exists,
+	 *   so migration and PTE flag decisions are unchanged.
+	 *
+	 * This approach simplifies the implementation and avoids redundant
+	 * attribute maintenance. This ioctl operation is for attribute only,
+	 * so do not invalidate the GPU mapping here.
+	 */
+	return amdgpu_svm_attr_clear(attr_tree, start_page, last_page);
+}
+
+static void attr_get_ctx_add(struct attr_get_ctx *ctx,
+			       const struct amdgpu_svm_attrs *attrs)
+{
+	if (!ctx->has_range) {
+		ctx->preferred_loc = attrs->preferred_loc;
+		ctx->prefetch_loc = attrs->prefetch_loc;
+		ctx->granularity = attrs->granularity;
+		ctx->access = attrs->access;
+		ctx->flags_and = attrs->flags;
+		ctx->has_range = true;
+		return;
+	}
+
+	if (ctx->preferred_loc != attrs->preferred_loc)
+		ctx->preferred_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	if (ctx->prefetch_loc != attrs->prefetch_loc)
+		ctx->prefetch_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	if (attrs->granularity < ctx->granularity)
+		ctx->granularity = attrs->granularity;
+	if (ctx->access != attrs->access)
+		ctx->access = AMDGPU_SVM_ACCESS_INACCESSIBLE;
+	ctx->flags_and &= attrs->flags;
+}
+
+static int attr_get_ctx_to_result(const struct attr_get_ctx *ctx,
+				uint32_t nattr,
+				struct drm_amdgpu_svm_attribute *attrs)
+{
+	uint32_t i;
+
+	for (i = 0; i < nattr; i++) {
+		switch (attrs[i].type) {
+		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
+			attrs[i].value = ctx->preferred_loc;
+			break;
+		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
+			attrs[i].value = ctx->prefetch_loc;
+			break;
+		case AMDGPU_SVM_ATTR_ACCESS:
+			attrs[i].value = (uint32_t)ctx->access;
+			break;
+		case AMDGPU_SVM_ATTR_HOST_ACCESS:
+		case AMDGPU_SVM_ATTR_COHERENT:
+		case AMDGPU_SVM_ATTR_HIVE_LOCAL:
+		case AMDGPU_SVM_ATTR_GPU_RO:
+		case AMDGPU_SVM_ATTR_GPU_EXEC:
+		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
+		case AMDGPU_SVM_ATTR_EXT_COHERENT: {
+			uint32_t flag_bit;
+
+			if (!attr_flag_type_to_bit(attrs[i].type, &flag_bit))
+				return -EINVAL;
+
+			attrs[i].value = (ctx->flags_and & flag_bit) ? 1 : 0;
+			break;
+		}
+		case AMDGPU_SVM_ATTR_GRANULARITY:
+			attrs[i].value = ctx->granularity;
+			break;
+		default:
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
+			uint64_t start, uint64_t size,
+			uint32_t nattr,
+			struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm_attrs default_attrs;
+	struct attr_get_ctx ctx = { 0 };
+	struct interval_tree_node *node;
+	unsigned long start_page, last_page, cursor;
+	int r;
+
+	start_page = start >> PAGE_SHIFT;
+	last_page = (start + size - 1) >> PAGE_SHIFT;
+
+	r = amdgpu_svm_attr_range_is_valid(attr_tree->svm, start_page, last_page);
+	if (r)
+		return r;
+
+	mutex_lock(&attr_tree->lock);
+	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
+	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
+
+	if (!node) {
+		attr_get_ctx_add(&ctx, &default_attrs);
+		r = attr_get_ctx_to_result(&ctx, nattr, attrs);
+		mutex_unlock(&attr_tree->lock);
+		return r;
+	}
+
+	cursor = start_page;
+	while (cursor <= last_page) {
+		const struct amdgpu_svm_attrs *range_attrs;
+		unsigned long range_last = last_page;
+		struct amdgpu_svm_attr_range *range = NULL;
+		unsigned long next;
+
+		if (node) {
+			range = container_of(node, struct amdgpu_svm_attr_range,
+					     it_node);
+
+			if (amdgpu_svm_attr_last_page(range) < cursor) {
+				node = interval_tree_iter_next(node, start_page,
+							      last_page);
+				continue;
+			}
+
+			if (amdgpu_svm_attr_start_page(range) <= cursor) {
+				range_last = min(last_page, amdgpu_svm_attr_last_page(range));
+				node = interval_tree_iter_next(node, start_page,
+							      last_page);
+			} else {
+				range_last = min(last_page,
+						 amdgpu_svm_attr_start_page(range) - 1);
+				range = NULL;
+			}
+		}
+
+		range_attrs = range ? &range->attrs : &default_attrs;
+		attr_get_ctx_add(&ctx, range_attrs);
+
+		if (range_last == ULONG_MAX)
+			break;
+
+		next = range_last + 1;
+		if (next <= cursor)
+			break;
+		cursor = next;
+	}
+
+	if (!ctx.has_range)
+		attr_get_ctx_add(&ctx, &default_attrs);
+
+	r = attr_get_ctx_to_result(&ctx, nattr, attrs);
+	mutex_unlock(&attr_tree->lock);
+	return r;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
index 0f712536a5dc1..301babef79d5b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
@@ -159,6 +159,9 @@ enum amdgpu_svm_attr_change_trigger {
 	 AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE | \
 	 AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE)
 
+struct amdgpu_svm_attr_tree *
+amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm);
+void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree);
 struct amdgpu_svm_attr_range *
 amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
 			   unsigned long page);
@@ -170,6 +173,22 @@ amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
 void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
 				 struct amdgpu_svm_attrs *attrs);
 
+int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
+			   uint64_t start,
+			   uint64_t size,
+			   uint32_t nattr,
+			   const struct drm_amdgpu_svm_attribute *attrs);
+int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
+				       uint64_t start,
+				       uint64_t size,
+				       uint32_t nattr,
+				       struct drm_amdgpu_svm_attribute *attrs);
+int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page);
+int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page);
 struct amdgpu_svm_attr_range *
 amdgpu_svm_attr_range_alloc(unsigned long start_page,
 			   unsigned long last_page,
-- 
2.53.0

[PATCH v9 05/18] drm/amdgpu: add SVM range types and work queue interface

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add amdgpu_svm_range.h with GPU mapped range types and interfaces:
- enum amdgpu_svm_range_queue_state: NOT_QUEUED, IN_GC, PROCESSING
  states for queue work
- struct amdgpu_svm_range: extends drm_gpusvm_range with gpu_mapped
  state, queue_state, attribute flags, work queue node, pending
  ops/pages, and validation timestamp
- enum amdgpu_svm_range_op: NONE, UNMAP operation types
- struct amdgpu_svm_range_op_ctx: dequeue context for GC processing
- Inline helper: amdgpu_svm_range_invalidate_gpu_mapping
- Range operations API declarations: find_or_insert, get_pages,
  update_mapping, update_gpu_range, invalidate

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 129 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  30 ++++
 2 files changed, 159 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
new file mode 100644
index 0000000000000..39db3a18b5f2f
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -0,0 +1,129 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_RANGE_H__
+#define __AMDGPU_SVM_RANGE_H__
+
+#include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
+
+#include "amdgpu_svm.h"
+#include "amdgpu_vm.h"
+
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/types.h>
+
+struct amdgpu_svm;
+struct amdgpu_svm_attr_range;
+struct amdgpu_svm_attrs;
+struct dma_fence;
+struct drm_exec;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_range;
+struct mmu_notifier_range;
+
+/**
+ * enum amdgpu_svm_range_queue_state - Work queue state of an SVM range
+ * @AMDGPU_SVM_RANGE_NOT_QUEUED: Not on any work list.
+ * @AMDGPU_SVM_RANGE_IN_GC: Queued on the GC list.
+ * @AMDGPU_SVM_RANGE_PROCESSING: Being processed in workqueue.
+ */
+enum amdgpu_svm_range_queue_state {
+	AMDGPU_SVM_RANGE_NOT_QUEUED = 0,
+	AMDGPU_SVM_RANGE_IN_GC,
+	AMDGPU_SVM_RANGE_PROCESSING,
+};
+
+/**
+ * struct amdgpu_svm_range - amdgpu GPU mapped SVM range
+ * @base: Embedded drm_gpusvm_range base.
+ * @work_node: Links the range into the work list. Protected by amdgpu_svm.work_lock.
+ * @gpu_mapped: True while the range has a live GPU mapping. Accessed with
+ *              READ_ONCE/WRITE_ONCE.
+ * @queue_state: Current work queue state @amdgpu_svm_range_queue_state.
+ *               Protected by amdgpu_svm.work_lock.
+ * @pending_ops: Pending amdgpu_svm_range_op bitmask. Protected by work_lock.
+ * @pending_start_page: First page of the accumulated pending interval.
+ * @pending_last_page: Last page of the accumulated pending interval.
+ * @attr_flags: Attribute flags last applied to the GPU mapping.
+ * @validate_timestamp: Time of the last successful mapping validation.
+ */
+struct amdgpu_svm_range {
+	struct drm_gpusvm_range base;
+	struct list_head work_node;
+	bool gpu_mapped;
+	u8 queue_state;
+	u8 pending_ops;
+	unsigned long pending_start_page;
+	unsigned long pending_last_page;
+	uint32_t attr_flags;
+	ktime_t validate_timestamp;
+};
+
+static inline struct amdgpu_svm_range *
+to_amdgpu_svm_range(struct drm_gpusvm_range *range)
+{
+	return container_of(range, struct amdgpu_svm_range, base);
+}
+
+static inline void
+amdgpu_svm_range_invalidate_gpu_mapping(struct amdgpu_svm_range *range)
+{
+	WRITE_ONCE(range->gpu_mapped, false);
+}
+
+#define AMDGPU_SVM_RANGE_TRACE(r__, op__)				    \
+	trace_amdgpu_svm_range((op__),					    \
+		to_amdgpu_svm((r__)->base.gpusvm)->vm->pasid,		    \
+		(r__)->base.gpusvm, READ_ONCE((r__)->gpu_mapped),	    \
+		(r__)->base.pages.notifier_seq,				    \
+		drm_gpusvm_range_start(&(r__)->base) >> PAGE_SHIFT,	    \
+		drm_gpusvm_range_end(&(r__)->base) >> PAGE_SHIFT)
+
+/**
+ * enum amdgpu_svm_range_op - Pending operation bits for an SVM range
+ * @AMDGPU_SVM_RANGE_OP_NONE: No pending operation.
+ * @AMDGPU_SVM_RANGE_OP_UNMAP: Range needs to be unmapped/removed by the GC.
+ */
+enum amdgpu_svm_range_op {
+	AMDGPU_SVM_RANGE_OP_NONE    = 0,
+	AMDGPU_SVM_RANGE_OP_UNMAP   = BIT(0),
+};
+
+/**
+ * struct amdgpu_svm_range_op_ctx - context of a range workqueue operation
+ * @range: The dequeued range to process.
+ * @start_page: First page for workqueue operation.
+ * @last_page: Last page for workqueue operation.
+ * @pending_ops: amdgpu_svm_range_op bitmask.
+ */
+struct amdgpu_svm_range_op_ctx {
+	struct amdgpu_svm_range *range;
+	unsigned long start_page;
+	unsigned long last_page;
+	uint8_t pending_ops;
+};
+
+#endif /* __AMDGPU_SVM_RANGE_H__ */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
index 4580ab533ca98..88c160c55f1fa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
@@ -114,6 +114,36 @@ TRACE_EVENT(amdgpu_iv,
 		      __entry->src_data[2], __entry->src_data[3])
 );
 
+TRACE_EVENT(amdgpu_svm_range,
+	    TP_PROTO(const char *op, u32 pasid, void *gpusvm, bool mapped,
+		     unsigned long seqno, unsigned long start_page,
+		     unsigned long end_page),
+	    TP_ARGS(op, pasid, gpusvm, mapped, seqno, start_page, end_page),
+	    TP_STRUCT__entry(
+			     __string(op, op)
+			     __field(u32, pasid)
+			     __field(void *, gpusvm)
+			     __field(bool, mapped)
+			     __field(unsigned long, seqno)
+			     __field(unsigned long, start_page)
+			     __field(unsigned long, end_page)
+			    ),
+	    TP_fast_assign(
+			   __assign_str(op);
+			   __entry->pasid = pasid;
+			   __entry->gpusvm = gpusvm;
+			   __entry->mapped = mapped;
+			   __entry->seqno = seqno;
+			   __entry->start_page = start_page;
+			   __entry->end_page = end_page;
+			   ),
+	    TP_printk("%s: pasid=%u gpusvm=%p mapped=%d seqno=%lu range=[0x%lx-0x%lx]-0x%lx",
+		      __get_str(op), __entry->pasid, __entry->gpusvm,
+		      __entry->mapped, __entry->seqno,
+		      __entry->start_page, __entry->end_page,
+		      __entry->end_page - __entry->start_page)
+);
+
 
 TRACE_EVENT(amdgpu_bo_create,
 	    TP_PROTO(struct amdgpu_bo *bo),
-- 
2.53.0

[PATCH v9 06/18] drm/amdgpu/gmc: add get_svm_pte_flags callback

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add a gmc_funcs callback for computing the GPU PTE flags of an SVM
mapping, plus a HW-agnostic input struct (amdgpu_svm_pte_flags_params)
and the amdgpu_gmc_get_svm_pte_flags() wrapper. This lets the per-IP
MTYPE/PTE selection live in the HW IP specific gmc_vX_0.c files instead
of the SVM core, mirroring how get_vm_pte handles BO mappings.

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  24 +++++
 drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c  |  69 ++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 119 ++++++++++++++++++++++++
 3 files changed, 212 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
index ddb0d500e0faa..448cfda67719d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
@@ -147,6 +147,24 @@ struct amdgpu_vmhub {
 	const struct amdgpu_vmhub_funcs *vmhub_funcs;
 };
 
+/* Decoded inputs for the SVM PTE flag selection, gmc_funcs::get_svm_pte_flags */
+struct amdgpu_svm_pte_flags_params {
+	/* SVM attribute: coherent */
+	bool coherent;
+	/* SVM attribute: extended coherent */
+	bool ext_coherent;
+	/* SVM attribute: GPU read only */
+	bool gpu_ro;
+	/* SVM attribute: GPU executable */
+	bool gpu_exec;
+	/* mapping is in VRAM local device or remote P2P */
+	bool is_vram;
+	/* mapping is in this GPU's local VRAM */
+	bool is_local;
+	/* remote VRAM BO is on a GPU in the same XGMI hive as this GPU. */
+	bool same_hive;
+};
+
 /*
  * GPU MC structures, functions & helpers
  */
@@ -175,6 +193,10 @@ struct amdgpu_gmc_funcs {
 			   struct amdgpu_bo *bo,
 			   uint32_t vm_flags,
 			   uint64_t *pte_flags);
+	/* get the full pte flags for an SVM mapping */
+	uint64_t (*get_svm_pte_flags)(struct amdgpu_device *adev,
+				      struct amdgpu_vm *vm,
+				      const struct amdgpu_svm_pte_flags_params *params);
 	/* override per-page pte flags */
 	void (*override_vm_pte_flags)(struct amdgpu_device *dev,
 				      struct amdgpu_vm *vm,
@@ -376,6 +398,8 @@ struct amdgpu_gmc {
 #define amdgpu_gmc_get_vm_pte(adev, vm, bo, vm_flags, pte_flags) \
 	((adev)->gmc.gmc_funcs->get_vm_pte((adev), (vm), (bo), (vm_flags), \
 					   (pte_flags)))
+#define amdgpu_gmc_get_svm_pte_flags(adev, vm, params) \
+	((adev)->gmc.gmc_funcs->get_svm_pte_flags((adev), (vm), (params)))
 #define amdgpu_gmc_override_vm_pte_flags(adev, vm, addr, pte_flags)	\
 	(adev)->gmc.gmc_funcs->override_vm_pte_flags			\
 		((adev), (vm), (addr), (pte_flags))
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
index 84c93364d2201..2e599c0212748 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
@@ -548,6 +548,74 @@ static void gmc_v12_0_get_vm_pte(struct amdgpu_device *adev,
 		*flags = AMDGPU_PTE_MTYPE_GFX12(*flags, MTYPE_UC);
 }
 
+/*
+ * Mirror of the GFX12 cases of svm_range_get_pte_flags(), with the
+ * per IP logic kept here in the HW specific file.
+ */
+static uint64_t
+gmc_v12_0_get_svm_pte_flags(struct amdgpu_device *adev,
+			    struct amdgpu_vm *vm,
+			    const struct amdgpu_svm_pte_flags_params *params)
+{
+	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
+	bool coherent = params->coherent;
+	bool ext_coherent = params->ext_coherent;
+	bool is_local = params->is_local;
+	bool is_vram = params->is_vram;
+	bool snoop = !is_vram;
+	uint32_t mapping_flags = 0;
+	unsigned int mtype_local, mtype_remote;
+	bool is_aid_a1;
+	uint64_t pte_flags;
+
+	switch (gc_ip_version) {
+	case IP_VERSION(12, 0, 0):
+	case IP_VERSION(12, 0, 1):
+		mapping_flags |= AMDGPU_VM_MTYPE_NC;
+		break;
+	case IP_VERSION(12, 1, 0):
+		is_aid_a1 = (adev->rev_id & 0x10);
+		mtype_local = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_RW :
+				amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
+				is_aid_a1 ? AMDGPU_VM_MTYPE_RW : AMDGPU_VM_MTYPE_NC;
+		mtype_remote = is_aid_a1 ? AMDGPU_VM_MTYPE_NC : AMDGPU_VM_MTYPE_UC;
+		snoop = true;
+
+		if (is_local) {
+			mapping_flags |= mtype_local;
+		} else if (ext_coherent) {
+			mapping_flags |= AMDGPU_VM_MTYPE_UC;
+		} else {
+			/* system memory or remote VRAM */
+			mapping_flags |= mtype_remote;
+		}
+		break;
+	default:
+		mapping_flags |= coherent ?
+			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		break;
+	}
+
+	if (params->gpu_exec)
+		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
+
+	pte_flags = AMDGPU_PTE_VALID;
+	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
+	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
+	pte_flags |= AMDGPU_PTE_IS_PTE;
+
+	gmc_v12_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
+	pte_flags |= AMDGPU_PTE_READABLE;
+	if (!params->gpu_ro)
+		pte_flags |= AMDGPU_PTE_WRITEABLE;
+
+	if (gc_ip_version == IP_VERSION(12, 1, 0) &&
+	    adev->have_atomics_support)
+		pte_flags |= AMDGPU_PTE_BUS_ATOMICS;
+
+	return pte_flags;
+}
+
 static unsigned gmc_v12_0_get_vbios_fb_size(struct amdgpu_device *adev)
 {
 	return 0;
@@ -577,6 +645,7 @@ static const struct amdgpu_gmc_funcs gmc_v12_0_gmc_funcs = {
 	.emit_pasid_mapping = gmc_v12_0_emit_pasid_mapping,
 	.get_vm_pde = gmc_v12_0_get_vm_pde,
 	.get_vm_pte = gmc_v12_0_get_vm_pte,
+	.get_svm_pte_flags = gmc_v12_0_get_svm_pte_flags,
 	.get_vbios_fb_size = gmc_v12_0_get_vbios_fb_size,
 	.get_dcc_alignment = gmc_v12_0_get_dcc_alignment,
 };
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
index 8a5c44810ba1e..b85a83ce2ce02 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
@@ -1198,6 +1198,124 @@ static void gmc_v9_0_get_vm_pte(struct amdgpu_device *adev,
 		gmc_v9_0_get_coherence_flags(adev, vm, bo, vm_flags, flags);
 }
 
+/*
+ * Mirror of the GFX9 cases of svm_range_get_pte_flags(), with the
+ * per IP logic kept here in the HW specific file.
+ */
+static uint64_t
+gmc_v9_0_get_svm_pte_flags(struct amdgpu_device *adev,
+			   struct amdgpu_vm *vm,
+			   const struct amdgpu_svm_pte_flags_params *params)
+{
+	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
+	bool coherent = params->coherent;
+	bool ext_coherent = params->ext_coherent;
+	bool is_local = params->is_local;
+	bool is_vram = params->is_vram;
+	bool same_hive = params->same_hive;
+	bool snoop = !is_vram;
+	uint32_t mapping_flags = 0;
+	unsigned int mtype_local;
+	uint64_t pte_flags;
+
+	switch (gc_ip_version) {
+	case IP_VERSION(9, 4, 1):
+		if (is_vram) {
+			if (is_local) {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
+			} else {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+				if (same_hive)
+					snoop = true;
+			}
+		} else {
+			mapping_flags |= coherent ?
+				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		}
+		break;
+	case IP_VERSION(9, 4, 2):
+		if (is_vram) {
+			if (is_local) {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
+				if (adev->gmc.xgmi.connected_to_cpu)
+					snoop = true;
+			} else {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+				if (same_hive)
+					snoop = true;
+			}
+		} else {
+			mapping_flags |= coherent ?
+				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		}
+		break;
+	case IP_VERSION(9, 4, 3):
+	case IP_VERSION(9, 4, 4):
+	case IP_VERSION(9, 5, 0):
+		if (ext_coherent)
+			mtype_local = AMDGPU_VM_MTYPE_CC;
+		else
+			mtype_local = amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
+				amdgpu_mtype_local == 2 ? AMDGPU_VM_MTYPE_CC :
+				AMDGPU_VM_MTYPE_RW;
+		snoop = true;
+		if (is_vram) {
+			/* local HBM region close to partition */
+			if (is_local)
+				mapping_flags |= mtype_local;
+			/* local HBM region far from partition or remote XGMI GPU
+			 * with regular system scope coherence
+			 */
+			else if (same_hive && !ext_coherent)
+				mapping_flags |= AMDGPU_VM_MTYPE_NC;
+			/* PCIe P2P on GPUs pre-9.5.0 */
+			else if (gc_ip_version < IP_VERSION(9, 5, 0) && !same_hive)
+				mapping_flags |= AMDGPU_VM_MTYPE_UC;
+			/* Other remote memory */
+			else
+				mapping_flags |= ext_coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		} else if (adev->flags & AMD_IS_APU) {
+			/* On NUMA systems, locality is determined per-page
+			 * in gmc_v9_0_override_vm_pte_flags.
+			 */
+			if (num_possible_nodes() <= 1)
+				mapping_flags |= mtype_local;
+			else
+				mapping_flags |= ext_coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		} else {
+			if (gc_ip_version < IP_VERSION(9, 5, 0) || ext_coherent)
+				mapping_flags |= AMDGPU_VM_MTYPE_UC;
+			else
+				mapping_flags |= AMDGPU_VM_MTYPE_NC;
+		}
+		break;
+	default:
+		mapping_flags |= coherent ?
+			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		break;
+	}
+
+	if (params->gpu_exec)
+		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
+
+	pte_flags = AMDGPU_PTE_VALID;
+	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
+	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
+
+	gmc_v9_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
+	pte_flags |= AMDGPU_PTE_READABLE;
+	if (!params->gpu_ro)
+		pte_flags |= AMDGPU_PTE_WRITEABLE;
+
+	return pte_flags;
+}
+
 static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev,
 					   struct amdgpu_vm *vm,
 					   uint64_t addr, uint64_t *flags)
@@ -1315,6 +1433,7 @@ static const struct amdgpu_gmc_funcs gmc_v9_0_gmc_funcs = {
 	.emit_pasid_mapping = gmc_v9_0_emit_pasid_mapping,
 	.get_vm_pde = gmc_v9_0_get_vm_pde,
 	.get_vm_pte = gmc_v9_0_get_vm_pte,
+	.get_svm_pte_flags = gmc_v9_0_get_svm_pte_flags,
 	.override_vm_pte_flags = gmc_v9_0_override_vm_pte_flags,
 	.get_vbios_fb_size = gmc_v9_0_get_vbios_fb_size,
 	.query_mem_partition_mode = &amdgpu_gmc_query_memory_partition,
-- 
2.53.0

[PATCH v9 07/18] drm/amdgpu: implement SVM range GPU mapping core

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Implement GPU page table mapping in amdgpu_svm_range.c:
- amdgpu_svm_range_pages_valid: check notifier sequence validity
- amdgpu_svm_range_is_valid: validate gpu_mapped and attr_flags
  against the requested attribute set, under MMU notifier lock.
- amdgpu_svm_range_zap_ptes: clear GPU PTEs via amdgpu_vm_update_range
- amdgpu_svm_range_attr_pte_flags: compute PTE flags per GC IP version
- amdgpu_svm_range_lock_vm_pd: acquire VM page directory by drm_exec
- amdgpu_svm_range_update_gpu_range: DMA segment coalescing; recomputes
  PTE flags per segment from entry->proto via attr_pte_flags, then
  programs PTEs under notifier lock
- amdgpu_svm_range_find_or_insert: wrapper of drm gpu SVM find or insert
  API.
- amdgpu_svm_range_get_pages: wrapper of drm gpu SVM get pages.
- amdgpu_svm_range_evict: thin wrapper of drm_gpusvm_range_evict
- amdgpu_svm_range_update_mapping: full pipeline (lock PD, validate
  pages, program PTEs, update PDEs, flush TLB, record attr_flags
  and mapped state); takes attrs + read_only and forwards them to
  update_gpu_range
- amdgpu_svm_range_map_attrs: cursor based iteration across attr tree,
  creating/mapping ranges

- Add to amdgpu_svm.h driver private interconnect tags used in
drm_pagemap_addr.proto:
  AMDGPU_INTERCONNECT_VRAM = DRM_INTERCONNECT_DRIVER
  AMDGPU_INTERCONNECT_P2P  = AMDGPU_INTERCONNECT_VRAM + 1

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       |   4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 419 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  40 ++
 3 files changed, 463 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
index f3b4f228405c2..c67f01b3b5d3b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
@@ -27,6 +27,7 @@
 
 #include <drm/amdgpu_drm.h>
 #include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
 #include <linux/atomic.h>
 #include <linux/kref.h>
 #include <linux/list.h>
@@ -44,6 +45,9 @@ struct amdgpu_svm_attrs;
 struct drm_device;
 struct drm_file;
 
+#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
+#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
+
 enum amdgpu_svm_xnack_mode {
 	AMDGPU_SVM_XNACK_OFF,
 	AMDGPU_SVM_XNACK_ON,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
new file mode 100644
index 0000000000000..335fe5a551f0f
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -0,0 +1,419 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu_svm_fault.h"
+#include "amdgpu.h"
+#include "amdgpu_vm.h"
+#include "amdgpu_trace.h"
+
+#include <drm/drm_exec.h>
+#include <drm/drm_pagemap.h>
+
+#include <linux/mmu_notifier.h>
+#include <uapi/linux/kfd_ioctl.h>
+
+bool
+amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
+		  struct amdgpu_svm_range *range)
+{
+	struct drm_gpusvm_range *base = &range->base;
+
+	lockdep_assert_held(&svm->gpusvm.notifier_lock);
+
+	if (base->pages.flags.unmapped || base->pages.flags.partial_unmap)
+		return false;
+
+	return drm_gpusvm_range_pages_valid(&svm->gpusvm, base);
+}
+
+bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
+			       const struct amdgpu_svm_attrs *attrs)
+{
+	return READ_ONCE(range->gpu_mapped) &&
+	       READ_ONCE(range->attr_flags) == attrs->flags;
+}
+
+
+int
+amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
+			  struct amdgpu_svm_range *range,
+			  unsigned long start_page,
+			  unsigned long last_page)
+{
+	struct dma_fence *fence = NULL;
+	unsigned int flags;
+	int ret;
+
+	if (last_page < start_page)
+		return 0;
+
+	flags = memalloc_noreclaim_save();
+	ret = amdgpu_vm_unmap_range(svm->adev, svm->vm, NULL,
+				    start_page, last_page, 0, &fence);
+	memalloc_noreclaim_restore(flags);
+
+	if (fence) {
+		dma_fence_wait(fence, false);
+		dma_fence_put(fence);
+	}
+
+	return ret;
+}
+
+uint64_t
+amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
+			    const struct amdgpu_svm_attrs *attrs,
+			    bool read_only,
+			    enum drm_interconnect_protocol proto)
+{
+	struct amdgpu_device *adev = svm->adev;
+	uint32_t flags = attrs->flags;
+	bool is_local = (proto == AMDGPU_INTERCONNECT_VRAM);
+	struct amdgpu_svm_pte_flags_params params = {
+		.coherent = flags & (AMDGPU_SVM_ATTR_BIT_COHERENT |
+				     AMDGPU_SVM_ATTR_BIT_EXT_COHERENT),
+		.ext_coherent = flags & AMDGPU_SVM_ATTR_BIT_EXT_COHERENT,
+		.gpu_ro = flags & AMDGPU_SVM_ATTR_BIT_GPU_RO,
+		.gpu_exec = flags & AMDGPU_SVM_ATTR_BIT_GPU_EXEC,
+		.is_local = is_local,
+		.is_vram = is_local || (proto == AMDGPU_INTERCONNECT_P2P),
+		/* TODO: add hive check. */
+		.same_hive = false,
+	};
+	uint32_t gc_ip_version;
+	uint32_t mapping_flags = 0;
+	uint64_t pte_flags;
+	bool snoop;
+
+	/*
+	 * The per-IP MTYPE / PTE flag selection lives in the HW IP specific
+	 * gmc_vX_0.c files mirroring amdgpu_gmc_get_vm_pte for BO mappings.
+	 */
+	if (adev->gmc.gmc_funcs->get_svm_pte_flags) {
+		pte_flags = amdgpu_gmc_get_svm_pte_flags(adev, svm->vm, &params);
+	} else {
+		gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
+		snoop = !is_local;
+		mapping_flags |= params.coherent ?
+			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+
+		if (params.gpu_exec)
+			mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
+
+		pte_flags = AMDGPU_PTE_VALID;
+		pte_flags |= is_local ? 0 : AMDGPU_PTE_SYSTEM;
+		pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
+		if (gc_ip_version >= IP_VERSION(12, 0, 0))
+			pte_flags |= AMDGPU_PTE_IS_PTE;
+
+		amdgpu_gmc_get_vm_pte(adev, svm->vm, NULL, mapping_flags,
+				      &pte_flags);
+		pte_flags |= AMDGPU_PTE_READABLE;
+		if (!params.gpu_ro)
+			pte_flags |= AMDGPU_PTE_WRITEABLE;
+	}
+
+	if (read_only)
+		pte_flags &= ~AMDGPU_PTE_WRITEABLE;
+
+	return pte_flags;
+}
+
+
+
+int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm, struct drm_exec *exec,
+				bool intr)
+{
+	unsigned int exec_flags = DRM_EXEC_IGNORE_DUPLICATES;
+	int ret;
+
+	if (intr)
+		exec_flags |= DRM_EXEC_INTERRUPTIBLE_WAIT;
+
+	drm_exec_init(exec, exec_flags, 0);
+	drm_exec_until_all_locked(exec) {
+		ret = amdgpu_vm_lock_pd(svm->vm, exec, 1);
+		drm_exec_retry_on_contention(exec);
+		if (ret) {
+			drm_exec_fini(exec);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
+amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
+				  struct amdgpu_svm_range *range,
+				  const struct amdgpu_svm_attrs *attrs,
+				  bool read_only,
+				  bool flush_tlb,
+				  bool wait_fence,
+				  struct dma_fence **fence)
+{
+	struct drm_gpusvm_range *base = &range->base;
+
+	lockdep_assert_held(&svm->gpusvm.notifier_lock);
+
+	const unsigned long range_start_page = drm_gpusvm_range_start(base) >> PAGE_SHIFT;
+	const unsigned long range_end_page = drm_gpusvm_range_end(base) >> PAGE_SHIFT;
+	const unsigned long npages = range_end_page - range_start_page;
+	unsigned long mapped_pages = 0;
+	unsigned long dma_idx = 0;
+	int ret;
+
+	if (!base->pages.dma_addr || !npages)
+		return -EINVAL;
+
+	while (mapped_pages < npages) {
+		const struct drm_pagemap_addr *entry = &base->pages.dma_addr[dma_idx++];
+		unsigned long seg_pages = min_t(unsigned long, 1UL << entry->order,
+						npages - mapped_pages);
+		uint64_t pte_flags;
+		unsigned long start_page, last_page;
+		bool is_last_seg;
+
+		if (entry->proto != DRM_INTERCONNECT_SYSTEM)
+			return -EOPNOTSUPP;
+
+		pte_flags = amdgpu_svm_range_attr_pte_flags(svm, attrs,
+							    read_only,
+							    entry->proto);
+
+		start_page = range_start_page + mapped_pages;
+		last_page = start_page + seg_pages - 1;
+		mapped_pages += seg_pages;
+		is_last_seg = mapped_pages == npages;
+
+		/*
+		 * The struct hmm_range hmm_range field inside amdgpu_hmm_range
+		 * is drm_gpusvm internal state. It is not exposed to the
+		 * implementing driver and is managed entirely by the framework
+		 * as part of the framework logic. This amdgpu_vm_map_range call
+		 * always runs inside the drm_gpusvm notifier, so omitting
+		 * hmm_range here is safe.
+		 */
+		ret = amdgpu_vm_map_range(svm->adev, svm->vm,
+					  flush_tlb && is_last_seg, true, NULL,
+					  start_page, last_page, pte_flags,
+					  0, entry->addr, NULL, NULL, NULL,
+					  wait_fence && is_last_seg ? fence : NULL);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+struct amdgpu_svm_range *
+amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
+				unsigned long gpuva_start, unsigned long gpuva_end,
+				struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_range *r;
+
+retry:
+	r = drm_gpusvm_range_find_or_insert(&svm->gpusvm, addr,
+					    gpuva_start, gpuva_end, ctx);
+	/*
+	 * UMD doesn't set RO for some RO VMAs, but the drm gpu svm framework
+	 * denies no RO flag range insert for RO VMAs, so treat
+	 * -EPERM as an indication of RO and retry if not set.
+	 */
+	if (PTR_ERR_OR_ZERO(r) == -EPERM && !ctx->read_only) {
+		ctx->read_only = true;
+		goto retry;
+	}
+
+	if (IS_ERR(r))
+		return ERR_CAST(r);
+
+	return to_amdgpu_svm_range(r);
+}
+
+int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
+			       struct drm_gpusvm_range *range,
+			       struct drm_gpusvm_ctx *ctx)
+{
+	int ret;
+
+retry:
+	ret = drm_gpusvm_range_get_pages(&svm->gpusvm, range, ctx);
+	/*
+	 * HMM returns -EPERM when write access is requested for a read-only
+	 * VMA. Retry as read-only so the eventual GPU mapping follows the CPU
+	 * VMA permissions.
+	 */
+	if (ret == -EPERM && !ctx->read_only) {
+		ctx->read_only = true;
+		goto retry;
+	}
+
+	if (ret == -EOPNOTSUPP) {
+		AMDGPU_SVM_ERR(
+			"range get pages -EOPNOTSUPP, evict and retry: gpuva=[0x%lx-0x%lx) ret=%d\n",
+			drm_gpusvm_range_start(range),
+			drm_gpusvm_range_end(range), ret);
+		amdgpu_svm_range_evict(svm, range);
+	}
+
+	return ret;
+}
+
+void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
+			    struct drm_gpusvm_range *range)
+{
+	if (!range->pages.flags.has_devmem_pages)
+		return;
+
+	drm_gpusvm_range_evict(&svm->gpusvm, range);
+}
+
+int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
+				    struct amdgpu_svm_range *range,
+				    const struct amdgpu_svm_attrs *attrs,
+				    bool read_only,
+				    bool intr, bool wait,
+				    bool flush_tlb)
+{
+	struct drm_exec exec;
+	struct dma_fence *fence = NULL;
+	unsigned int flags;
+	int ret;
+
+	ret = amdgpu_svm_range_lock_vm_pd(svm, &exec, intr);
+	if (ret)
+		return ret;
+
+	flags = memalloc_noreclaim_save();
+	drm_gpusvm_notifier_lock(&svm->gpusvm);
+
+	if (!amdgpu_svm_range_pages_valid(svm, range)) {
+		amdgpu_svm_range_invalidate_gpu_mapping(range);
+		ret = -EAGAIN;
+	} else {
+		ret = amdgpu_svm_range_update_gpu_range(svm, range, attrs,
+							read_only, flush_tlb,
+							wait, wait ? &fence : NULL);
+	}
+
+	drm_gpusvm_notifier_unlock(&svm->gpusvm);
+	memalloc_noreclaim_restore(flags);
+
+	if (!ret && fence)
+		dma_fence_wait(fence, intr);
+	dma_fence_put(fence);
+
+	if (!ret)
+		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, false);
+
+	if (!ret) {
+		if (flush_tlb)
+			amdgpu_svm_flush_tlb(svm);
+		WRITE_ONCE(range->attr_flags, attrs->flags);
+		WRITE_ONCE(range->gpu_mapped, true);
+		range->validate_timestamp = ktime_get_boottime();
+	}
+
+	drm_exec_fini(&exec);
+	return ret;
+}
+
+int
+amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
+		       const struct amdgpu_svm_attrs *attrs,
+		       unsigned long start, unsigned long end)
+{
+	unsigned long addr = start;
+	int ret;
+	bool devmem_possible = amdgpu_svm_devmem_possible(svm);
+	bool need_vram_migration = devmem_possible &&
+				   amdgpu_svm_attr_prefer_vram(attrs);
+	struct drm_gpusvm_ctx map_ctx = {
+		.devmem_possible = devmem_possible,
+		.devmem_only = need_vram_migration,
+		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
+	};
+
+	while (addr < end) {
+		struct amdgpu_svm_range *range;
+		unsigned long next_addr;
+		/* reset read_only every iteration, amdgpu_svm_range_find_or_insert may change it */
+		map_ctx.read_only = !!(attrs->flags & AMDGPU_SVM_ATTR_BIT_GPU_RO);
+
+		range = amdgpu_svm_range_find_or_insert(svm, addr,
+							addr, end,
+							&map_ctx);
+		if (IS_ERR(range)) {
+			AMDGPU_SVM_ERR(
+				"failed to find/insert range for gpuva 0x%lx [0x%lx-0x%lx), ret=%ld\n",
+				addr, start, end, PTR_ERR(range));
+			return PTR_ERR(range);
+		}
+
+		next_addr = drm_gpusvm_range_end(&range->base);
+		if (next_addr <= addr)
+			return -EINVAL;
+
+		if (amdgpu_svm_range_is_valid(range, attrs)) {
+			addr = next_addr;
+			continue;
+		}
+
+		/* TODO: add migration */
+
+		AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
+
+		ret = amdgpu_svm_range_get_pages(svm, &range->base,
+						 &map_ctx);
+		if (ret) {
+			AMDGPU_SVM_ERR("failed to get pages for range [0x%lx-0x%lx), ret=%d\n",
+					drm_gpusvm_range_start(&range->base),
+					drm_gpusvm_range_end(&range->base), ret);
+			return ret;
+		}
+
+		AMDGPU_SVM_RANGE_TRACE(range, "UPDATE MAPPING");
+
+		ret = amdgpu_svm_range_update_mapping(svm, range, attrs,
+						      map_ctx.read_only,
+						      true, true,
+						      true);
+		if (ret) {
+			AMDGPU_SVM_ERR("failed to update gpu map for range [0x%lx-0x%lx), ret=%d\n",
+					drm_gpusvm_range_start(&range->base),
+					drm_gpusvm_range_end(&range->base), ret);
+			return ret;
+		}
+
+		addr = next_addr;
+	}
+
+	return 0;
+}
+
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
index 39db3a18b5f2f..e5a5f52166576 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -126,4 +126,44 @@ struct amdgpu_svm_range_op_ctx {
 	uint8_t pending_ops;
 };
 
+uint64_t amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
+					 const struct amdgpu_svm_attrs *attrs,
+					 bool read_only,
+					 enum drm_interconnect_protocol proto);
+int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm,
+				struct drm_exec *exec,
+				bool intr);
+bool amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
+				  struct amdgpu_svm_range *range);
+bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
+			       const struct amdgpu_svm_attrs *attrs);
+int amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
+				      struct amdgpu_svm_range *range,
+				      const struct amdgpu_svm_attrs *attrs,
+				      bool read_only,
+				      bool flush_tlb, bool wait,
+				      struct dma_fence **fence);
+int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
+				    struct amdgpu_svm_range *range,
+				    const struct amdgpu_svm_attrs *attrs,
+				    bool read_only,
+				    bool intr, bool wait,
+				    bool flush_tlb);
+int amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
+			       const struct amdgpu_svm_attrs *attrs,
+			       unsigned long start, unsigned long end);
+int amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
+			      struct amdgpu_svm_range *range,
+			      unsigned long start_page,
+			      unsigned long last_page);
+void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
+			    struct drm_gpusvm_range *range);
+struct amdgpu_svm_range *
+amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
+				unsigned long gpuva_start, unsigned long gpuva_end,
+				struct drm_gpusvm_ctx *ctx);
+int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
+			       struct drm_gpusvm_range *range,
+			       struct drm_gpusvm_ctx *ctx);
+
 #endif /* __AMDGPU_SVM_RANGE_H__ */
-- 
2.53.0

[PATCH v9 08/18] drm/amdgpu: implement SVM range notifier and GC helpers

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add MMU notifier event handling and garbage collection infrastructure:
- amdgpu_svm_range_remove: unmap pages and remove from gpusvm
- amdgpu_svm_range_notifier_event_begin: compute the [start_page,
  last_page] window from the notifier range and the gpusvm range,
  then zap PTEs via amdgpu_svm_range_zap_ptes and invalidate the
  GPU mapping
- amdgpu_svm_gc_enqueue: queue range for GC worker and set queue state
- amdgpu_svm_gc_add_range: mark range unmapped and enqueue for GC
- amdgpu_svm_range_notifier_event_end: DMA unmap and GC on MMU_UNMAP
- amdgpu_svm_range_invalidate_interval: cursor based PTE clearing
  across notifiers/ranges; clears PTEs through
  amdgpu_svm_range_zap_ptes
- amdgpu_svm_range_dequeue_locked: dequeue work item with atomic
  pending state transfer
- amdgpu_svm_range_put_if_dequeued: release range ref after dequeue,
  re-dispatch if new work was enqueued during processing
- amdgpu_svm_range_unqueue: locked context helper that pulls a range
  off the GC list

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 311 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  16 +
 2 files changed, 327 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index 335fe5a551f0f..9e348390795a6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -417,3 +417,314 @@ amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 	return 0;
 }
 
+void amdgpu_svm_range_remove(struct amdgpu_svm *svm,
+			     struct amdgpu_svm_range *range,
+			     struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_range *base = &range->base;
+
+	amdgpu_svm_assert_locked(svm);
+
+	if (!base->pages.flags.unmapped && !base->pages.flags.partial_unmap)
+		drm_gpusvm_range_unmap_pages(&svm->gpusvm, base, ctx);
+
+	amdgpu_svm_range_invalidate_gpu_mapping(range);
+	drm_gpusvm_range_remove(&svm->gpusvm, base);
+}
+
+/**
+ * amdgpu_svm_range_notifier_event_begin() - First MMU notifier phase
+ * @svm: The SVM context.
+ * @range: The GPU SVM range covered by the notifier event.
+ * @mmu_range: The MMU notifier range describing the invalidation window.
+ *
+ * Invalidate the GPU view of @range over the notifier window: zap the
+ * affected PTEs and clear the gpu_mapped flag. The range is not removed
+ * here; structural removal if needed happens in the event-end phase.
+ *
+ * Return: true if PTEs were zapped, false otherwise.
+ */
+bool
+amdgpu_svm_range_notifier_event_begin(struct amdgpu_svm *svm,
+				      struct drm_gpusvm_range *range,
+				      const struct mmu_notifier_range *mmu_range)
+{
+	struct amdgpu_svm_range *svm_range = to_amdgpu_svm_range(range);
+	unsigned long start_page, last_page;
+
+	amdgpu_svm_assert_in_notifier(svm);
+
+	AMDGPU_SVM_RANGE_TRACE(svm_range, "NOTIFIER");
+
+	if (range->pages.flags.unmapped || !svm_range->gpu_mapped)
+		return false;
+
+	AMDGPU_SVM_RANGE_TRACE(svm_range, "NOTIFIER - EXECUTE");
+
+	start_page = max(drm_gpusvm_range_start(range),
+			 mmu_range->start) >> PAGE_SHIFT;
+	last_page = (min(drm_gpusvm_range_end(range),
+			 mmu_range->end) >> PAGE_SHIFT) - 1;
+
+	amdgpu_svm_range_zap_ptes(svm, svm_range, start_page, last_page);
+	amdgpu_svm_range_invalidate_gpu_mapping(svm_range);
+
+	return true;
+}
+
+/**
+ * amdgpu_svm_gc_enqueue() - Queue a range for work queue processing
+ * @svm: The SVM context.
+ * @range: The SVM range to queue.
+ * @start_page: First page of the pending invalidation interval.
+ * @last_page: Last page of the pending invalidation interval.
+ *
+ * Queue @range on the GC list and kick the GC worker. A range reference
+ * is taken when it first enters IN_GC and dropped only when it returns
+ * to NOT_QUEUED, so the range stays alive while on the GC list or being
+ * processed. All transitions are serialized by svm->work_lock.
+ */
+static void
+amdgpu_svm_gc_enqueue(struct amdgpu_svm *svm,
+		      struct amdgpu_svm_range *range,
+		      unsigned long start_page, unsigned long last_page)
+{
+	if (atomic_read(&svm->exiting))
+		return;
+
+	spin_lock(&svm->work_lock);
+	if (range->queue_state == AMDGPU_SVM_RANGE_NOT_QUEUED) {
+		drm_gpusvm_range_get(&range->base);
+		range->queue_state = AMDGPU_SVM_RANGE_IN_GC;
+	}
+
+	range->pending_start_page = min(range->pending_start_page, start_page);
+	range->pending_last_page = max(range->pending_last_page, last_page);
+	if (range->pending_ops == AMDGPU_SVM_RANGE_OP_NONE)
+		list_add_tail(&range->work_node, &svm->gc.list);
+	range->pending_ops |= AMDGPU_SVM_RANGE_OP_UNMAP;
+	spin_unlock(&svm->work_lock);
+
+	queue_work(svm->gc.wq, &svm->gc.work);
+}
+
+static void
+amdgpu_svm_gc_add_range(struct amdgpu_svm *svm,
+			struct amdgpu_svm_range *svm_range,
+			const struct mmu_notifier_range *mmu_range)
+{
+	unsigned long start_page = max(drm_gpusvm_range_start(&svm_range->base),
+				       mmu_range->start) >> PAGE_SHIFT;
+	unsigned long last_page = (min(drm_gpusvm_range_end(&svm_range->base),
+				       mmu_range->end) >> PAGE_SHIFT) - 1;
+
+	AMDGPU_SVM_RANGE_TRACE(svm_range, "GARBAGE COLLECTOR ADD");
+
+	drm_gpusvm_range_set_unmapped(&svm_range->base, mmu_range);
+	amdgpu_svm_gc_enqueue(svm, svm_range, start_page, last_page);
+}
+
+/**
+ * amdgpu_svm_range_notifier_event_end() - Second MMU notifier phase
+ * @svm: The SVM context.
+ * @range: The GPU SVM range covered by the notifier event.
+ * @mmu_range: The MMU notifier range describing the event.
+ *
+ * DMA-unmap the pages for @range. On an actual unmap event, also queue the
+ * range to the GC so it gets structurally removed from the gpusvm trees.
+ */
+static void
+amdgpu_svm_range_notifier_event_end(struct amdgpu_svm *svm,
+				    struct drm_gpusvm_range *range,
+				    const struct mmu_notifier_range *mmu_range)
+{
+	struct drm_gpusvm_ctx ctx = { .in_notifier = true, };
+
+	amdgpu_svm_assert_in_notifier(svm);
+
+	drm_gpusvm_range_unmap_pages(&svm->gpusvm, range, &ctx);
+	if (mmu_range->event == MMU_NOTIFY_UNMAP)
+		amdgpu_svm_gc_add_range(svm, to_amdgpu_svm_range(range),
+					mmu_range);
+}
+
+int
+amdgpu_svm_range_invalidate_interval(struct amdgpu_svm *svm,
+				     unsigned long start_page,
+				     unsigned long last_page)
+{
+	unsigned long start = start_page << PAGE_SHIFT;
+	unsigned long end = (last_page + 1) << PAGE_SHIFT;
+	struct drm_gpusvm_notifier *notifier, *next_notifier;
+	struct drm_gpusvm_ctx ctx = { .in_notifier = false };
+	struct drm_exec exec;
+	bool needs_flush = false;
+	int ret;
+
+	amdgpu_svm_assert_locked(svm);
+
+	ret = amdgpu_svm_range_lock_vm_pd(svm, &exec, true);
+	if (ret)
+		return ret;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next_notifier, &svm->gpusvm,
+					  start, end) {
+		struct drm_gpusvm_range *range, *next_range;
+
+		drm_gpusvm_for_each_range_safe(range, next_range, notifier,
+					       start, end) {
+			struct amdgpu_svm_range *svm_range = to_amdgpu_svm_range(range);
+			bool crosses_boundary = start > drm_gpusvm_range_start(range) ||
+						end < drm_gpusvm_range_end(range);
+
+			if (svm_range->gpu_mapped) {
+				AMDGPU_SVM_RANGE_TRACE(svm_range,
+					crosses_boundary ? "ATTR DESTROY" :
+							   "ATTR ZAP PTE");
+
+				ret = amdgpu_svm_range_zap_ptes(svm, svm_range,
+						drm_gpusvm_range_start(range) >> PAGE_SHIFT,
+						(drm_gpusvm_range_end(range) >> PAGE_SHIFT) - 1);
+				if (ret < 0) {
+					drm_exec_fini(&exec);
+					return ret;
+				}
+
+				amdgpu_svm_range_invalidate_gpu_mapping(svm_range);
+				needs_flush = true;
+			}
+
+			if (crosses_boundary) {
+				/* remove ranges crossing the boundary so GPU fault
+				 * creates new ranges bounded by the updated
+				 * attr_range boundaries.
+				 * Evict devmem-backed pages back to sysmem first
+				 * so VRAM-resident data is not lost when the range
+				 * is destroyed. No-op for sysmem-only ranges.
+				 */
+				drm_gpusvm_range_get(range);
+
+				amdgpu_svm_range_evict(svm, range);
+				amdgpu_svm_range_remove(svm, svm_range, &ctx);
+
+				amdgpu_svm_range_unqueue(svm, svm_range);
+
+				drm_gpusvm_range_put(range);
+			}
+		}
+	}
+
+	drm_exec_fini(&exec);
+
+	if (needs_flush)
+		amdgpu_svm_flush_tlb(svm);
+
+	AMDGPU_SVM_DBG("attr invalidate done [0x%lx-0x%lx]-0x%lx needs_flush=%d\n",
+			 start_page, last_page, last_page - start_page + 1,
+			 needs_flush ? 1 : 0);
+
+	return 0;
+}
+
+/**
+ * amdgpu_svm_range_dequeue_locked() - Pull the next range off a work list
+ * @svm: The SVM context.
+ * @work_list: The work list to dequeue from.
+ * @op_ctx: Output context filled with the dequeued range's pending state.
+ *
+ * Move the first range on @work_list to PROCESSING and atomically snapshot
+ * its pending op/interval into @op_ctx while resetting them on the range, so
+ * invalidations that arrive during processing accumulate afresh. Caller must
+ * hold svm->work_lock.
+ *
+ * Return: true if a range was dequeued, false if @work_list was empty.
+ */
+bool
+amdgpu_svm_range_dequeue_locked(struct amdgpu_svm *svm,
+					struct list_head *work_list,
+					struct amdgpu_svm_range_op_ctx *op_ctx)
+{
+	struct amdgpu_svm_range *range;
+
+	lockdep_assert_held(&svm->work_lock);
+
+	range = list_first_entry_or_null(work_list, struct amdgpu_svm_range,
+					work_node);
+	if (!range)
+		return false;
+
+	list_del_init(&range->work_node);
+	range->queue_state = AMDGPU_SVM_RANGE_PROCESSING;
+
+	op_ctx->range = range;
+	op_ctx->start_page = range->pending_start_page;
+	op_ctx->last_page = range->pending_last_page;
+	op_ctx->pending_ops = range->pending_ops;
+
+	range->pending_start_page = ULONG_MAX;
+	range->pending_last_page = 0;
+	range->pending_ops = AMDGPU_SVM_RANGE_OP_NONE;
+
+	return true;
+}
+
+/**
+ * amdgpu_svm_range_put_if_dequeued() - Finish processing a dequeued range
+ * @svm: The SVM context.
+ * @range: The range previously moved to PROCESSING.
+ *
+ * Called after a PROCESSING range has been handled. If new UNMAP work was
+ * enqueued while it was being processed, re-queue it on the GC list and kick
+ * the worker again; otherwise return it to NOT_QUEUED and drop the reference
+ * taken at enqueue time.
+ */
+void
+amdgpu_svm_range_put_if_dequeued(struct amdgpu_svm *svm,
+				     struct amdgpu_svm_range *range)
+{
+	bool release_kref = false;
+	bool queue_gc = false;
+
+	spin_lock(&svm->work_lock);
+
+	if (range->queue_state != AMDGPU_SVM_RANGE_PROCESSING) {
+		spin_unlock(&svm->work_lock);
+		return;
+	}
+
+	if (range->pending_ops & AMDGPU_SVM_RANGE_OP_UNMAP) {
+		list_add_tail(&range->work_node, &svm->gc.list);
+		range->queue_state = AMDGPU_SVM_RANGE_IN_GC;
+		queue_gc = true;
+	} else {
+		range->queue_state = AMDGPU_SVM_RANGE_NOT_QUEUED;
+		release_kref = true;
+	}
+
+	spin_unlock(&svm->work_lock);
+
+	if (queue_gc)
+		queue_work(svm->gc.wq, &svm->gc.work);
+	if (release_kref)
+		drm_gpusvm_range_put(&range->base);
+}
+
+void amdgpu_svm_range_unqueue(struct amdgpu_svm *svm,
+					struct amdgpu_svm_range *range)
+{
+	bool put = false;
+
+	amdgpu_svm_assert_locked(svm);
+
+	spin_lock(&svm->work_lock);
+	if (range->queue_state == AMDGPU_SVM_RANGE_IN_GC) {
+		list_del_init(&range->work_node);
+		range->queue_state = AMDGPU_SVM_RANGE_NOT_QUEUED;
+		range->pending_ops = AMDGPU_SVM_RANGE_OP_NONE;
+		put = true;
+	}
+	spin_unlock(&svm->work_lock);
+
+	if (put)
+		drm_gpusvm_range_put(&range->base);
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
index e5a5f52166576..f423be277a85a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -149,6 +149,16 @@ int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
 				    bool read_only,
 				    bool intr, bool wait,
 				    bool flush_tlb);
+bool amdgpu_svm_range_dequeue_locked(struct amdgpu_svm *svm,
+				     struct list_head *work_list,
+				     struct amdgpu_svm_range_op_ctx *op_ctx);
+void amdgpu_svm_range_put_if_dequeued(struct amdgpu_svm *svm,
+				      struct amdgpu_svm_range *range);
+void amdgpu_svm_range_unqueue(struct amdgpu_svm *svm,
+					struct amdgpu_svm_range *range);
+void amdgpu_svm_range_remove(struct amdgpu_svm *svm,
+			     struct amdgpu_svm_range *range,
+			     struct drm_gpusvm_ctx *ctx);
 int amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 			       const struct amdgpu_svm_attrs *attrs,
 			       unsigned long start, unsigned long end);
@@ -165,5 +175,11 @@ amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
 int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
 			       struct drm_gpusvm_range *range,
 			       struct drm_gpusvm_ctx *ctx);
+bool amdgpu_svm_range_notifier_event_begin(struct amdgpu_svm *svm,
+					   struct drm_gpusvm_range *range,
+					   const struct mmu_notifier_range *mmu_range);
+int amdgpu_svm_range_invalidate_interval(struct amdgpu_svm *svm,
+					 unsigned long start_page,
+					 unsigned long last_page);
 
 #endif /* __AMDGPU_SVM_RANGE_H__ */
-- 
2.53.0

[PATCH v9 09/18] drm/amdgpu: add SVM notifier invalidate callback and checkpoint

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add invalidate_ranges callback and checkpoint timestamp:
- amdgpu_svm_capture_checkpoint_ts: capture interrupt handler write
  pointer timestamp for stale retry fault filtering
- amdgpu_svm_range_invalidate: invalidate_ranges callback dispatching
  per range notifier begin/end with TLB flush batching, checkpoint
  timestamp capture on MMU_NOTIFY_UNMAP events

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 71 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  6 ++
 2 files changed, 77 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index 9e348390795a6..ed057545d0964 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -728,3 +728,74 @@ void amdgpu_svm_range_unqueue(struct amdgpu_svm *svm,
 	if (put)
 		drm_gpusvm_range_put(&range->base);
 }
+
+/**
+ * amdgpu_svm_capture_checkpoint_ts() - Record the IH ring write pointer time
+ * @svm: The SVM context.
+ *
+ * Capture the timestamp of the most recent interrupt handler write pointer
+ * into @svm->checkpoint_ts. Retry faults whose timestamp predates this
+ * checkpoint are stale and are dropped by the fault handler.
+ */
+void amdgpu_svm_capture_checkpoint_ts(struct amdgpu_svm *svm)
+{
+	struct amdgpu_device *adev = svm->adev;
+	struct amdgpu_ih_ring *ih;
+	uint32_t checkpoint_wptr;
+
+	if (!adev->irq.retry_cam_enabled && adev->irq.ih1.ring_size) {
+		ih = &adev->irq.ih1;
+		checkpoint_wptr = amdgpu_ih_get_wptr(adev, ih);
+		if (ih->rptr != checkpoint_wptr) {
+			WRITE_ONCE(svm->checkpoint_ts,
+				   amdgpu_ih_decode_iv_ts(adev, ih,
+							  checkpoint_wptr, -1));
+			return;
+		}
+	}
+
+	ih = &adev->irq.ih_soft;
+	checkpoint_wptr = amdgpu_ih_get_wptr(adev, ih);
+	if (ih->rptr != checkpoint_wptr)
+		WRITE_ONCE(svm->checkpoint_ts,
+			   amdgpu_ih_decode_iv_ts(adev, ih,
+						  checkpoint_wptr, -1));
+}
+
+/**
+ * amdgpu_svm_range_invalidate() - MMU notifier invalidate_ranges callback
+ * @svm: The SVM context.
+ * @notifier: The GPU SVM notifier covering the invalidated ranges.
+ * @mmu_range: The MMU notifier range describing the event.
+ * @first: First GPU SVM range in the affected interval.
+ * @adj_start: Adjusted start address of the affected interval.
+ * @adj_end: Adjusted end address of the affected interval.
+ *
+ * Dispatch the two phase notifier handling across every range in
+ * [@adj_start, @adj_end): run the begin phase for all ranges, then
+ * the end phase. On an unmap event, also capture a checkpoint timestamp
+ * so stale retry faults can be filtered.
+ */
+void amdgpu_svm_range_invalidate(struct amdgpu_svm *svm,
+				 struct drm_gpusvm_notifier *notifier,
+				 const struct mmu_notifier_range *mmu_range,
+				 struct drm_gpusvm_range *first,
+				 uint64_t adj_start, uint64_t adj_end)
+{
+	struct drm_gpusvm_range *r;
+	bool needs_flush = false;
+
+	if (mmu_range->event == MMU_NOTIFY_UNMAP)
+		amdgpu_svm_capture_checkpoint_ts(svm);
+
+	r = first;
+	drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end)
+		needs_flush |= amdgpu_svm_range_notifier_event_begin(svm, r,
+								     mmu_range);
+	if (needs_flush)
+		amdgpu_svm_flush_tlb(svm);
+
+	r = first;
+	drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end)
+		amdgpu_svm_range_notifier_event_end(svm, r, mmu_range);
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
index f423be277a85a..5e49d774711ce 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -175,6 +175,12 @@ amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
 int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
 			       struct drm_gpusvm_range *range,
 			       struct drm_gpusvm_ctx *ctx);
+void amdgpu_svm_capture_checkpoint_ts(struct amdgpu_svm *svm);
+void amdgpu_svm_range_invalidate(struct amdgpu_svm *svm,
+				 struct drm_gpusvm_notifier *notifier,
+				 const struct mmu_notifier_range *mmu_range,
+				 struct drm_gpusvm_range *first,
+				 uint64_t adj_start, uint64_t adj_end);
 bool amdgpu_svm_range_notifier_event_begin(struct amdgpu_svm *svm,
 					   struct drm_gpusvm_range *range,
 					   const struct mmu_notifier_range *mmu_range);
-- 
2.53.0

[PATCH v9 10/18] drm/amdgpu: implement SVM initialization and lifecycle

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Implement amdgpu_svm.c core module:
- XNACK_OFF/ON helper macros for xnack state checks
- Static amdgpu_svm_cache_lock mutex for slab cache lifecycle
- drm_gpusvm_ops callbacks: range_alloc (kmem_cache), range_free,
  invalidate (dispatches via svm->invalidate_ranges callback)
- kref-based lifecycle: amdgpu_svm_release, amdgpu_svm_put
- PASID lookup: amdgpu_svm_lookup_by_pasid
- Slab cache management: amdgpu_svm_cache_init/fini
- Ioctl operation wrappers: op_set_attr, op_get_attr, op_reset_attr
- Attribute change detection and application:
  attr_change_trigger classifies changes into trigger types,
  amdgpu_svm_apply_attr_change dispatches invalidate or remap
  based on trigger flags and xnack state
- Hardware detection: amdgpu_svm_default_xnack_enabled per GC IP
- TLB flush: amdgpu_svm_flush_tlb_compute
- xnack mode: amdgpu_svm_init_xnack_mode validates requested mode
- Initialization: amdgpu_svm_init_with_ops (drm_gpusvm_init with
  2M/64K/4K chunk sizes, attr tree, invalidate_ranges/flush_tlb
  callbacks), amdgpu_svm_init_compute with xnack_mode parameter
- Teardown: amdgpu_svm_close (mark exiting, sync work),
  amdgpu_svm_fini (gpusvm_fini, destroy attr tree, release ref)

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 624 ++++++++++++++++++++++++
 1 file changed, 624 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
new file mode 100644
index 0000000000000..7dc43470037d2
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -0,0 +1,624 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include <linux/sched/mm.h>
+#include <linux/uaccess.h>
+#include <linux/xarray.h>
+
+#include <drm/drm_file.h>
+
+#include "amdgpu.h"
+#include "amdgpu_ih.h"
+#include "amdgpu_reset.h"
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+#include "amdgpu_svm_fault.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu_vm.h"
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+
+#define AMDGPU_SVM_MAX_ATTRS 64
+#define AMDGPU_SVM_DEFAULT_SVM_NOTIFIER_SIZE 512
+
+static const unsigned long amdgpu_svm_chunk_sizes[] = {
+	SZ_2M,
+	SZ_64K,
+	SZ_4K,
+};
+
+#define AMDGPU_SVM_GC_WQ_NAME "amdgpu_svm_gc"
+#define XNACK_OFF(svm)	((svm)->xnack_enabled == false)
+#define XNACK_ON(svm)	((svm)->xnack_enabled == true)
+
+/**
+ * amdgpu_svm_invalidate() - drm_gpusvm invalidate callback
+ * @gpusvm: The drm_gpusvm instance.
+ * @notifier: The GPU SVM notifier reporting the event.
+ * @mmu_range: The MMU notifier range describing the invalidation.
+ *
+ * Clamp the event to the notifier window, find the first affected range and
+ * dispatch to the driver's @invalidate_ranges handler.
+ */
+static void amdgpu_svm_invalidate(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_notifier *notifier,
+				  const struct mmu_notifier_range *mmu_range)
+{
+	struct amdgpu_svm *svm = to_amdgpu_svm(gpusvm);
+	struct drm_gpusvm_range *first;
+	uint64_t adj_start = mmu_range->start, adj_end = mmu_range->end;
+
+	amdgpu_svm_assert_in_notifier(svm);
+
+	AMDGPU_SVM_DBG(
+		"INVALIDATE: pasid=%u, gpusvm=%p, seqno=%lu, [0x%016lx-0x%016lx]-0x%lx, ev=%d\n",
+		svm->vm->pasid, &svm->gpusvm,
+		notifier->notifier.invalidate_seq,
+			 mmu_range->start, mmu_range->end,
+			 mmu_range->end - mmu_range->start, mmu_range->event);
+
+	if (mmu_range->event == MMU_NOTIFY_RELEASE)
+		return;
+	if (atomic_read(&svm->exiting))
+		return;
+
+	adj_start = max(drm_gpusvm_notifier_start(notifier), adj_start);
+	adj_end = min(drm_gpusvm_notifier_end(notifier), adj_end);
+
+	first = drm_gpusvm_range_find(notifier, adj_start, adj_end);
+	if (!first)
+		return;
+
+	svm->invalidate_ranges(svm, notifier, mmu_range, first,
+			       adj_start, adj_end);
+}
+
+static struct drm_gpusvm_range *amdgpu_svm_range_alloc(struct drm_gpusvm *gpusvm)
+{
+	struct amdgpu_svm_range *range;
+
+	range = kzalloc(sizeof(*range), GFP_KERNEL);
+	if (!range)
+		return NULL;
+
+	INIT_LIST_HEAD(&range->work_node);
+	range->pending_start_page = ULONG_MAX;
+	return &range->base;
+}
+
+static void amdgpu_svm_range_free(struct drm_gpusvm_range *range)
+{
+	kfree(to_amdgpu_svm_range(range));
+}
+
+static const struct drm_gpusvm_ops amdgpu_gpusvm_ops = {
+	.range_alloc = amdgpu_svm_range_alloc,
+	.range_free = amdgpu_svm_range_free,
+	.invalidate = amdgpu_svm_invalidate,
+};
+
+static void amdgpu_svm_release(struct kref *ref)
+{
+	kfree(container_of(ref, struct amdgpu_svm, refcount));
+}
+
+/**
+ * amdgpu_svm_put() - Drop a reference on an SVM context
+ * @svm: The SVM context.
+ *
+ * Release a reference taken on @svm and free it once the last reference is
+ * dropped.
+ */
+void amdgpu_svm_put(struct amdgpu_svm *svm)
+{
+	if (svm)
+		kref_put(&svm->refcount, amdgpu_svm_release);
+}
+
+/**
+ * amdgpu_svm_lookup_by_pasid() - Find the SVM context for a PASID
+ * @adev: The amdgpu device.
+ * @pasid: The PASID to look up.
+ *
+ * Look up the VM bound to @pasid and return its SVM context with a reference
+ * taken. The caller must drop it with amdgpu_svm_put().
+ *
+ * Return: The referenced SVM context, or %NULL if none is bound.
+ */
+struct amdgpu_svm *
+amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev, uint32_t pasid)
+{
+	struct amdgpu_svm *svm = NULL;
+	struct amdgpu_vm *vm;
+	unsigned long irqflags;
+
+	xa_lock_irqsave(&adev->vm_manager.pasids, irqflags);
+	vm = xa_load(&adev->vm_manager.pasids, pasid);
+	if (vm && vm->svm) {
+		svm = vm->svm;
+		kref_get(&svm->refcount);
+	}
+	xa_unlock_irqrestore(&adev->vm_manager.pasids, irqflags);
+
+	return svm;
+}
+
+static int amdgpu_svm_op_set_attr(struct amdgpu_vm *vm,
+				  uint64_t start,
+				  uint64_t size,
+				  uint32_t nattr,
+				  const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm *svm = vm->svm;
+
+	amdgpu_svm_sync_work(svm);
+
+	return amdgpu_svm_attr_set(svm->attr_tree, start, size, nattr,
+				   attrs);
+}
+
+static int amdgpu_svm_op_get_attr(struct amdgpu_vm *vm,
+				  uint64_t start,
+				  uint64_t size,
+				  uint32_t nattr,
+				  struct drm_amdgpu_svm_attribute *attrs)
+{
+	amdgpu_svm_sync_work(vm->svm);
+
+	return amdgpu_svm_attr_get(vm->svm->attr_tree, start, size, nattr, attrs);
+}
+
+static int amdgpu_svm_op_reset_attr(struct amdgpu_vm *vm,
+				    uint64_t start, uint64_t size)
+{
+	struct amdgpu_svm *svm = vm->svm;
+	unsigned long start_page = start >> PAGE_SHIFT;
+	unsigned long last_page = (start + size - 1) >> PAGE_SHIFT;
+
+	amdgpu_svm_sync_work(svm);
+
+	return amdgpu_svm_attr_reset(svm->attr_tree,
+				     start_page, last_page);
+}
+
+/**
+ * attr_change_trigger() - Classify what an attribute update changed
+ * @old_attrs: Attributes before the update.
+ * @new_attrs: Attributes after the update.
+ *
+ * Compare the two attribute sets and return an
+ * amdgpu_svm_attr_change_trigger bitmask describing which aspects changed:
+ * access, PTE flags, mapping flags, location, granularity, prefetch.
+ *
+ * Return: The trigger bitmask.
+ */
+static uint32_t
+attr_change_trigger(const struct amdgpu_svm_attrs *old_attrs,
+		    const struct amdgpu_svm_attrs *new_attrs)
+{
+	uint32_t trigger = 0;
+	uint32_t changed_flags = old_attrs->flags ^ new_attrs->flags;
+
+	if (old_attrs->access != new_attrs->access)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE;
+	if (changed_flags & AMDGPU_SVM_PTE_FLAG_MASK)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE;
+	if (changed_flags & AMDGPU_SVM_MAPPING_FLAG_MASK)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE;
+	if (old_attrs->preferred_loc != new_attrs->preferred_loc ||
+	    old_attrs->prefetch_loc != new_attrs->prefetch_loc)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE;
+	if (old_attrs->granularity != new_attrs->granularity)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE;
+	if (new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
+	    new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_SYSMEM)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_PREFETCH;
+
+	return trigger;
+}
+
+/**
+ * amdgpu_svm_apply_attr_change() - React to an attribute change on a range
+ * @svm: The SVM context.
+ * @old_attrs: Attributes before the change.
+ * @new_attrs: Attributes after the change.
+ * @start_page: First page of the affected interval.
+ * @last_page: Last page of the affected interval.
+ *
+ * Classify the change and act on it: when XNACK is on and the change affects
+ * existing GPU mappings, invalidate the interval; when the new attributes
+ * request a prefetch, map / remap the interval with the new attributes.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
+				 const struct amdgpu_svm_attrs *old_attrs,
+				 const struct amdgpu_svm_attrs *new_attrs,
+				 unsigned long start_page,
+				 unsigned long last_page)
+{
+	bool old_access, new_access;
+	bool needs_invalidate = false;
+	bool needs_mapping = false;
+	uint32_t trigger;
+	int ret;
+
+	amdgpu_svm_assert_locked(svm);
+
+	if (!start_page && !last_page)
+		return 0;
+
+	trigger = attr_change_trigger(old_attrs, new_attrs);
+	old_access = amdgpu_svm_attr_has_access(old_attrs->access);
+	new_access = amdgpu_svm_attr_has_access(new_attrs->access);
+	if (XNACK_ON(svm) &&
+	    (trigger & AMDGPU_SVM_ATTR_TRIGGER_NEED_INVALIDATE))
+		needs_invalidate = true;
+
+	if (trigger & AMDGPU_SVM_ATTR_TRIGGER_PREFETCH)
+		needs_mapping = true;
+
+	if (!trigger && !needs_mapping)
+		return 0;
+
+	AMDGPU_SVM_DBG("attr change trigger=0x%x old=%d new=%d [0x%lx-0x%lx]-0x%lx, xnack=%d\n",
+			 trigger, old_access, new_access, start_page, last_page,
+			 last_page - start_page + 1,
+			 svm->xnack_enabled ? 1 : 0);
+
+	if (needs_invalidate) {
+		AMDGPU_SVM_DBG("attr change invalidate [0x%lx-0x%lx]-0x%lx trigger=0x%x\n",
+				 start_page, last_page,
+				 last_page - start_page + 1, trigger);
+		ret = amdgpu_svm_range_invalidate_interval(svm, start_page,
+							   last_page);
+		if (ret) {
+			AMDGPU_SVM_ERR(
+				"failed to invalidate range for attr change: [0x%lx-0x%lx], ret=%d\n",
+				start_page, last_page, ret);
+			return ret;
+		}
+	}
+
+	if (!needs_mapping)
+		return 0;
+
+	return amdgpu_svm_range_map_attrs(svm, new_attrs,
+					  start_page << PAGE_SHIFT,
+					  (last_page + 1) << PAGE_SHIFT);
+}
+
+bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm)
+{
+	if (svm->adev->apu_prefer_gtt)
+		return false;
+
+	/* TODO: add amdgpu_pagemap_capable() */
+
+	return false;
+}
+
+/**
+ * amdgpu_svm_default_xnack_enabled() - Whether XNACK defaults to on for the HW
+ * @adev: The amdgpu device.
+ *
+ * Decide the default retry fault (XNACK) policy from the GC IP version and
+ * platform constraints.
+ *
+ * Return: true if XNACK should default to enabled.
+ */
+static bool amdgpu_svm_default_xnack_enabled(struct amdgpu_device *adev)
+{
+	uint32_t gc_ver = amdgpu_ip_version(adev, GC_HWIP, 0);
+
+	if (gc_ver < IP_VERSION(9, 0, 1))
+		return false;
+	if (!amdgpu_sriov_xnack_support(adev))
+		return false;
+
+	if (adev->gmc.noretry)
+		return false;
+
+	switch (gc_ver) {
+	case IP_VERSION(9, 4, 2):
+	case IP_VERSION(9, 4, 3):
+	case IP_VERSION(9, 4, 4):
+	case IP_VERSION(9, 5, 0):
+		return true;
+	default:
+		break;
+	}
+	if (gc_ver >= IP_VERSION(10, 1, 1))
+		return false;
+
+	return true;
+}
+
+void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm)
+{
+	amdgpu_vm_flush_compute_tlb(svm->adev, svm->vm, TLB_FLUSH_HEAVYWEIGHT,
+				    svm->adev->gfx.xcc_mask);
+}
+
+static int amdgpu_svm_work_init(struct amdgpu_svm *svm,
+				void (*gc_work_func)(struct work_struct *));
+static void amdgpu_svm_work_fini(struct amdgpu_svm *svm);
+
+/**
+ * amdgpu_svm_init_xnack_mode() - Resolve the requested XNACK mode
+ * @adev: The amdgpu device.
+ * @mode: The requested XNACK mode.
+ * @xnack_enabled: Output, set to the resolved enable state.
+ *
+ * Validate @mode against the hardware default: DEFAULT follows the HW policy,
+ * ON is rejected if the HW does not support it, OFF always disables.
+ *
+ * Return: 0 on success, -EOPNOTSUPP if ON is unavailable, -EINVAL on a bad
+ * mode.
+ */
+static int amdgpu_svm_init_xnack_mode(struct amdgpu_device *adev,
+				    enum amdgpu_svm_xnack_mode mode,
+				    bool *xnack_enabled)
+{
+	bool xnack_default = amdgpu_svm_default_xnack_enabled(adev);
+
+	switch (mode) {
+	case AMDGPU_SVM_XNACK_DEFAULT:
+		*xnack_enabled = xnack_default;
+		break;
+	case AMDGPU_SVM_XNACK_ON:
+		if (!xnack_default) {
+			AMDGPU_SVM_ERR("xnack on not available (mode=%d)\n",
+					mode);
+			*xnack_enabled = xnack_default;
+			return -EOPNOTSUPP;
+		}
+		*xnack_enabled = true;
+		break;
+	case AMDGPU_SVM_XNACK_OFF:
+		*xnack_enabled = false;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * amdgpu_svm_init_with_ops() - Initialize the SVM core with driver callbacks
+ * @svm: The SVM context to initialize.
+ * @invalidate_ranges: Callback invoked from the MMU notifier path.
+ * @gc_work_func: Work function draining the garbage collector.
+ *
+ * Set up the work queues, attribute tree and the embedded drm_gpusvm (with
+ * the 2M/64K/4K chunk sizes and the driver lock), wiring the supplied
+ * callbacks.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int amdgpu_svm_init_with_ops(struct amdgpu_svm *svm,
+				    void (*invalidate_ranges)(struct amdgpu_svm *,
+						struct drm_gpusvm_notifier *,
+						const struct mmu_notifier_range *,
+						struct drm_gpusvm_range *,
+						uint64_t, uint64_t),
+				    void (*gc_work_func)(struct work_struct *))
+{
+	struct amdgpu_device *adev = svm->adev;
+	int ret;
+
+	svm->invalidate_ranges = invalidate_ranges;
+
+	ret = amdgpu_svm_work_init(svm, gc_work_func);
+	if (ret)
+		return ret;
+
+	svm->attr_tree = amdgpu_svm_attr_tree_create(svm);
+	if (!svm->attr_tree) {
+		ret = -ENOMEM;
+		goto err_work_fini;
+	}
+
+	ret = drm_gpusvm_init(&svm->gpusvm, "AMDGPU SVM",
+						adev_to_drm(adev), current->mm, 0,
+						adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT,
+						AMDGPU_SVM_DEFAULT_SVM_NOTIFIER_SIZE * SZ_1M,
+						&amdgpu_gpusvm_ops,
+						amdgpu_svm_chunk_sizes,
+						ARRAY_SIZE(amdgpu_svm_chunk_sizes));
+
+	if (ret)
+		goto err_attr_tree_destroy;
+
+	drm_gpusvm_driver_set_lock(&svm->gpusvm, &svm->svm_lock);
+
+	return 0;
+
+err_attr_tree_destroy:
+	amdgpu_svm_attr_tree_destroy(svm->attr_tree);
+err_work_fini:
+	amdgpu_svm_work_fini(svm);
+	return ret;
+}
+
+static void amdgpu_svm_gc_work_func(struct work_struct *w);
+
+/**
+ * amdgpu_svm_init_compute() - Create the SVM context for a compute VM
+ * @adev: The amdgpu device.
+ * @vm: The VM to attach the SVM context to.
+ * @xnack_mode: The requested XNACK mode.
+ *
+ * Allocate and initialize an SVM context for @vm (idempotent if one already
+ * exists), resolving the XNACK mode and wiring the compute callbacks. XNACK
+ * off is not supported yet.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int amdgpu_svm_init_compute(struct amdgpu_device *adev,
+				   struct amdgpu_vm *vm,
+				   enum amdgpu_svm_xnack_mode xnack_mode)
+{
+	struct amdgpu_svm *svm;
+	int ret;
+
+	if (vm->svm)
+		return 0;
+
+	svm = kzalloc(sizeof(*svm), GFP_KERNEL);
+	if (!svm)
+		return -ENOMEM;
+
+	kref_init(&svm->refcount);
+	svm->adev = adev;
+	svm->vm = vm;
+	svm->default_granularity = min_t(u8, amdgpu_svm_default_granularity, 0x1B);
+	atomic_set(&svm->exiting, 0);
+
+	ret = amdgpu_svm_init_xnack_mode(adev, xnack_mode,
+					  &svm->xnack_enabled);
+	if (ret)
+		goto err_free;
+
+	if (svm->xnack_enabled) {
+		ret = amdgpu_svm_init_with_ops(svm,
+					       amdgpu_svm_range_invalidate,
+					       amdgpu_svm_gc_work_func);
+	} else {
+		AMDGPU_SVM_ERR("xnack off is not supported yet\n");
+		ret = -EOPNOTSUPP;
+	}
+
+	if (ret)
+		goto err_free;
+
+	AMDGPU_SVM_DBG("AMDGPU SVM initialized: default granularity 0x%lx bytes, xnack: %s\n",
+	       1UL << (svm->default_granularity + PAGE_SHIFT),
+	       svm->xnack_enabled ? "enabled" : "disabled");
+
+	vm->svm = svm;
+	return 0;
+
+err_free:
+	kfree(svm);
+	return ret;
+}
+
+/**
+ * amdgpu_svm_init() - Initialize SVM for a VM
+ * @adev: The amdgpu device.
+ * @vm: The VM to enable SVM on.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
+{
+	/* graphics svm init maybe different */
+
+	return amdgpu_svm_init_compute(adev, vm, AMDGPU_SVM_XNACK_DEFAULT);
+}
+
+/**
+ * amdgpu_svm_drain_retry_fault() - Wait for retry faults to drain
+ * @adev: The amdgpu device.
+ *
+ * Wait until the interrupt handler has processed up to the current checkpoint
+ * on the relevant IH rings, so no stale retry faults remain in flight. Skips
+ * draining during a GPU reset or if the reset domain cannot be entered.
+ */
+static void amdgpu_svm_drain_retry_fault(struct amdgpu_device *adev)
+{
+	if (!adev)
+		return;
+
+	if (amdgpu_in_reset(adev))
+		return;
+
+	if (!down_read_trylock(&adev->reset_domain->sem))
+		return;
+
+	amdgpu_ih_wait_on_checkpoint_process_ts(adev,
+		adev->irq.retry_cam_enabled ?
+			&adev->irq.ih : &adev->irq.ih1);
+	if (adev->irq.retry_cam_enabled)
+		amdgpu_ih_wait_on_checkpoint_process_ts(adev,
+			&adev->irq.ih_soft);
+
+	up_read(&adev->reset_domain->sem);
+}
+
+/**
+ * amdgpu_svm_close() - Begin SVM teardown for a VM
+ * @vm: The VM whose SVM context is closing.
+ *
+ * Mark the context as exiting (once), flush pending GC work and drain
+ * in-flight retry faults. Safe to call on a VM without an SVM context.
+ */
+void amdgpu_svm_close(struct amdgpu_vm *vm)
+{
+	struct amdgpu_svm *svm = vm->svm;
+
+	if (!svm)
+		return;
+
+	if (atomic_xchg(&svm->exiting, 1))
+		return;
+
+	amdgpu_svm_sync_work(svm);
+	amdgpu_svm_drain_retry_fault(svm->adev);
+}
+
+/**
+ * amdgpu_svm_fini() - Finalize and release a VM's SVM context
+ * @vm: The VM whose SVM context is being torn down.
+ *
+ * Close the context, tear down the embedded drm_gpusvm under the SVM lock,
+ * destroy the attribute tree and work queues, and drop the context
+ * reference. Safe to call on a VM without an SVM context.
+ */
+void amdgpu_svm_fini(struct amdgpu_vm *vm)
+{
+	struct amdgpu_svm *svm = vm->svm;
+
+	if (!svm)
+		return;
+
+	amdgpu_svm_close(vm);
+	amdgpu_svm_lock(svm);
+	drm_gpusvm_fini(&svm->gpusvm);
+	amdgpu_svm_unlock(svm);
+
+	amdgpu_svm_attr_tree_destroy(svm->attr_tree);
+	amdgpu_svm_work_fini(svm);
+	vm->svm = NULL;
+	amdgpu_svm_put(svm);
+}
+
+bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
+{
+	return vm->svm != NULL;
+}
+
+#endif /* CONFIG_DRM_AMDGPU_SVM */
-- 
2.53.0

[PATCH v9 11/18] drm/amdgpu: add SVM ioctl entry and fault handler module

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add ioctl entry point and fault handling:

amdgpu_svm.c additions:
- amdgpu_svm_copy_attrs: memdup_user for attribute array
- amdgpu_svm_garbage_collector: dequeue and remove unmapped ranges
- amdgpu_svm_clean_queue: drain work list releasing refs
- amdgpu_svm_gc_work_func: static GC work handler
- amdgpu_svm_gc_init/fini/flush: static workqueue lifecycle
- amdgpu_svm_work_init/work_fini: wrappers for GC lifecycle
- amdgpu_svm_sync_work: wrapper for GC flush
- amdgpu_gem_svm_ioctl: validate args, dispatch
  SET_ATTR/GET_ATTR/RESET_ATTR with copy_to_user for GET

amdgpu_svm_fault.c (new):
- fault_check_allowed: VMA permission check for fault address
- fault_get_unregistered_attrs: create default attr range for
  faulting address not covered by explicit attributes, bounded
  by VMA and granularity alignment
- fault_map_range: fault mapping pipeline with retry, GC,
  permission check, find/insert range, timestamp dedup,
  validity check, get_pages, update_mapping.
- amdgpu_svm_handle_fault: top level entry point from VM fault path

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c       | 200 ++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 450 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h |  39 ++
 3 files changed, 689 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
index 7dc43470037d2..ac101574bd61d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -621,4 +621,204 @@ bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
 	return vm->svm != NULL;
 }
 
+static int amdgpu_svm_copy_attrs(const struct drm_amdgpu_gem_svm *args,
+					   struct drm_amdgpu_svm_attribute **attrs,
+					   size_t *size)
+{
+	if (!args->nattr || args->nattr > AMDGPU_SVM_MAX_ATTRS)
+		return -EINVAL;
+	if (!args->attrs_ptr)
+		return -EINVAL;
+
+	*size = args->nattr * sizeof(**attrs);
+	*attrs = memdup_user(u64_to_user_ptr(args->attrs_ptr), *size);
+
+	return PTR_ERR_OR_ZERO(*attrs);
+}
+
+/**
+ * amdgpu_svm_garbage_collector() - Remove ranges queued for unmapping
+ * @svm: The SVM context.
+ *
+ * Drain the GC list: for each queued range, remove it from the gpusvm trees
+ * and drop the reference held while queued. Caller must hold the SVM lock.
+ *
+ * Return: 0 on success.
+ */
+int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm)
+{
+	struct amdgpu_svm_range_op_ctx op_ctx;
+
+	amdgpu_svm_assert_locked(svm);
+
+	spin_lock(&svm->work_lock);
+	while (amdgpu_svm_range_dequeue_locked(svm, &svm->gc.list, &op_ctx)) {
+		spin_unlock(&svm->work_lock);
+
+		WARN_ON(!(op_ctx.pending_ops & AMDGPU_SVM_RANGE_OP_UNMAP));
+
+		drm_gpusvm_range_remove(&svm->gpusvm,
+					&op_ctx.range->base);
+
+		amdgpu_svm_range_put_if_dequeued(svm, op_ctx.range);
+		spin_lock(&svm->work_lock);
+	}
+	spin_unlock(&svm->work_lock);
+	return 0;
+}
+
+/**
+ * amdgpu_svm_clean_queue() - Drain a work list, releasing range references
+ * @svm: The SVM context.
+ * @work_list: The work list to drain.
+ *
+ * Dequeue every range on @work_list and drop the reference held while it was
+ * queued, without performing any unmapping. Used during teardown.
+ */
+void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
+			   struct list_head *work_list)
+{
+	struct amdgpu_svm_range_op_ctx op_ctx;
+
+	spin_lock(&svm->work_lock);
+	while (amdgpu_svm_range_dequeue_locked(svm, work_list,
+						 &op_ctx)) {
+		spin_unlock(&svm->work_lock);
+		amdgpu_svm_range_put_if_dequeued(svm, op_ctx.range);
+		spin_lock(&svm->work_lock);
+	}
+	spin_unlock(&svm->work_lock);
+}
+
+static void amdgpu_svm_gc_work_func(struct work_struct *w)
+{
+	struct amdgpu_svm_gc *gc = container_of(w, struct amdgpu_svm_gc, work);
+	struct amdgpu_svm *svm = container_of(gc, struct amdgpu_svm, gc);
+
+	amdgpu_svm_lock(svm);
+	amdgpu_svm_garbage_collector(svm);
+	amdgpu_svm_unlock(svm);
+}
+
+static int amdgpu_svm_gc_init(struct amdgpu_svm *svm,
+			      void (*gc_work_func)(struct work_struct *))
+{
+	svm->gc.wq = alloc_workqueue(AMDGPU_SVM_GC_WQ_NAME,
+					WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM, 0);
+	if (!svm->gc.wq)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&svm->gc.list);
+	INIT_WORK(&svm->gc.work, gc_work_func);
+
+	return 0;
+}
+
+static void amdgpu_svm_gc_fini(struct amdgpu_svm *svm)
+{
+	flush_work(&svm->gc.work);
+	amdgpu_svm_clean_queue(svm, &svm->gc.list);
+	destroy_workqueue(svm->gc.wq);
+	svm->gc.wq = NULL;
+}
+
+static void amdgpu_svm_gc_flush(struct amdgpu_svm *svm)
+{
+	flush_work(&svm->gc.work);
+}
+
+static int amdgpu_svm_work_init(struct amdgpu_svm *svm,
+				void (*gc_work_func)(struct work_struct *))
+{
+	int ret;
+
+	init_rwsem(&svm->svm_lock);
+	spin_lock_init(&svm->work_lock);
+
+	ret = amdgpu_svm_gc_init(svm, gc_work_func);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static void amdgpu_svm_work_fini(struct amdgpu_svm *svm)
+{
+	amdgpu_svm_gc_fini(svm);
+}
+
+void amdgpu_svm_sync_work(struct amdgpu_svm *svm)
+{
+	amdgpu_svm_gc_flush(svm);
+}
+
+/**
+ * amdgpu_gem_svm_ioctl() - SVM GEM ioctl entry point
+ * @dev: The DRM device.
+ * @data: The ioctl argument (struct drm_amdgpu_gem_svm).
+ * @filp: The DRM file.
+ *
+ * Validate the arguments and dispatch to the SET_ATTR / GET_ATTR /
+ * RESET_ATTR operation, copying results back to user space for GET.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
+			 struct drm_file *filp)
+{
+	struct amdgpu_fpriv *fpriv = filp->driver_priv;
+	struct amdgpu_device *adev = drm_to_adev(dev);
+	struct drm_amdgpu_gem_svm *args = data;
+	struct drm_amdgpu_svm_attribute *attrs = NULL;
+	struct amdgpu_vm *vm;
+	size_t attrs_size = 0;
+	int ret = 0;
+
+	AMDGPU_SVM_DBG("ioctl op=%u va:[0x%llx-0x%llx)-0x%llx nattr=%u\n",
+			 args->operation, args->start_addr, args->start_addr + args->size,
+			 args->size, args->nattr);
+
+	vm = &fpriv->vm;
+	if (!amdgpu_svm_is_enabled(vm)) {
+		ret = amdgpu_svm_init(adev, vm);
+		if (ret)
+			return ret;
+	}
+
+	if ((args->start_addr & ~PAGE_MASK) || (args->size & ~PAGE_MASK))
+		return -EINVAL;
+
+	if (!args->start_addr || !args->size)
+		return -EINVAL;
+
+	if (args->operation != AMDGPU_SVM_OP_RESET_ATTR) {
+		ret = amdgpu_svm_copy_attrs(args, &attrs, &attrs_size);
+		if (ret)
+			return ret;
+	}
+
+	switch (args->operation) {
+	case AMDGPU_SVM_OP_SET_ATTR:
+		ret = amdgpu_svm_op_set_attr(vm, args->start_addr, args->size,
+					 args->nattr, attrs);
+		break;
+	case AMDGPU_SVM_OP_GET_ATTR:
+		ret = amdgpu_svm_op_get_attr(vm, args->start_addr, args->size,
+					 args->nattr, attrs);
+		if (!ret && copy_to_user(u64_to_user_ptr(args->attrs_ptr),
+					 attrs, attrs_size))
+			ret = -EFAULT;
+		break;
+	case AMDGPU_SVM_OP_RESET_ATTR:
+		ret = amdgpu_svm_op_reset_attr(vm, args->start_addr, args->size);
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+
+	kvfree(attrs);
+	return ret;
+}
+
 #endif /* CONFIG_DRM_AMDGPU_SVM */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
new file mode 100644
index 0000000000000..f28b3eb14500d
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
@@ -0,0 +1,450 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+#include "amdgpu_svm_fault.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu.h"
+#include "amdgpu_vm.h"
+#include "amdgpu_gmc.h"
+#include "amdgpu_ih.h"
+#include "amdgpu_trace.h"
+
+#include <drm/drm_exec.h>
+#include <drm/drm_gpusvm.h>
+
+#include <linux/delay.h>
+#include <linux/mm.h>
+#include <linux/sched/mm.h>
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+
+#define AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING	(2UL * NSEC_PER_MSEC)
+
+/**
+ * fault_get_unregistered_attrs() - Create a default attr range for a fault
+ * @svm: The SVM context.
+ * @fault_addr: The faulting address.
+ * @attr_start_page: Lower bound (first page) of the surrounding attribute gap.
+ * @attr_last_page: Upper bound (last page) of the surrounding attribute gap.
+ * @out: Output, the newly created attribute range.
+ *
+ * For a fault that lands in an address gap not covered by any explicit
+ * attribute range, create a default attribute range. Its bounds are the
+ * faulting page's granularity aligned window, clamped to the CPU VMA and to
+ * the surrounding attribute gap [@attr_start_page, @attr_last_page].
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int fault_get_unregistered_attrs(struct amdgpu_svm *svm,
+					    unsigned long fault_addr,
+					    unsigned long attr_start_page,
+					    unsigned long attr_last_page,
+					    struct amdgpu_svm_attr_range **out)
+{
+	struct amdgpu_svm_attr_tree *attr_tree = svm->attr_tree;
+	struct amdgpu_svm_attr_range *range;
+	struct amdgpu_svm_attrs attrs;
+	struct mm_struct *mm = svm->gpusvm.mm;
+	struct vm_area_struct *vma;
+	unsigned long fault_page = fault_addr >> PAGE_SHIFT;
+	unsigned long start_page, last_page;
+	unsigned long vma_start_page, vma_last_page;
+
+	amdgpu_svm_attr_set_default(svm, &attrs);
+
+	mmap_read_lock(mm);
+
+	vma = amdgpu_svm_check_vma(mm, fault_addr);
+	if (IS_ERR(vma)) {
+		mmap_read_unlock(mm);
+		AMDGPU_SVM_ERR("get_unregistered_attrs: invalid VMA for fault_addr=0x%lx\n",
+		       fault_addr);
+		return PTR_ERR(vma);
+	}
+	vma_start_page = vma->vm_start >> PAGE_SHIFT;
+	vma_last_page = (vma->vm_end >> PAGE_SHIFT) - 1;
+
+	if (vma_is_initial_heap(vma) || vma_is_initial_stack(vma))
+		attrs.preferred_loc = AMDGPU_SVM_LOCATION_SYSMEM;
+
+	mmap_read_unlock(mm);
+
+	start_page = max_t(unsigned long, vma_start_page,
+			   ALIGN_DOWN(fault_page, 1UL << attrs.granularity));
+	last_page = min_t(unsigned long, vma_last_page,
+			  ALIGN(fault_page + 1, 1UL << attrs.granularity) - 1);
+
+	start_page = max(start_page, attr_start_page);
+	last_page = min(last_page, attr_last_page);
+
+	mutex_lock(&attr_tree->lock);
+	range = amdgpu_svm_attr_range_alloc(start_page, last_page, &attrs);
+	if (!range) {
+		mutex_unlock(&attr_tree->lock);
+		return -ENOMEM;
+	}
+	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+	mutex_unlock(&attr_tree->lock);
+
+	AMDGPU_SVM_DBG(
+		"Created unregistered range for fault_addr=0x%lx: attr range=[0x%lx-0x%lx] size: 0x%lx attrs={preferred_loc=%d, prefetch_loc=%d, flags=0x%x, granularity=%u, access=%u}\n",
+		fault_addr, amdgpu_svm_attr_start_page(range),
+		amdgpu_svm_attr_last_page(range) + 1,
+		amdgpu_svm_attr_last_page(range) -
+			amdgpu_svm_attr_start_page(range) + 1,
+		range->attrs.preferred_loc, range->attrs.prefetch_loc,
+		range->attrs.flags, range->attrs.granularity,
+		range->attrs.access);
+
+	*out = range;
+	return 0;
+}
+
+/**
+ * fault_check_allowed() - Verify the CPU VMA
+ * @svm: The SVM context.
+ * @fault_addr: The faulting address.
+ * @write_fault: True if the fault was caused by a write.
+ *
+ * Check that the VMA covering @fault_addr grants the requested access
+ *
+ * Return: 0 if allowed, -EPERM if the required permission is missing.
+ */
+static int fault_check_allowed(struct amdgpu_svm *svm,
+				   unsigned long fault_addr, bool write_fault)
+{
+	struct mm_struct *mm = svm->gpusvm.mm;
+	struct vm_area_struct *vma;
+	unsigned long requested = VM_READ;
+	int ret = 0;
+
+	if (write_fault)
+		requested |= VM_WRITE;
+
+	mmap_read_lock(mm);
+	vma = vma_lookup(mm, fault_addr);
+	if (vma && (vma->vm_flags & requested) != requested) {
+		AMDGPU_SVM_ERR("fault addr 0x%lx no %s permission\n",
+			 fault_addr, write_fault ? "write" : "read");
+		ret = -EPERM;
+	}
+	mmap_read_unlock(mm);
+
+	return ret;
+}
+
+/**
+ * fault_map_range() - Map the faulting range into the GPU page tables
+ * @svm: The SVM context.
+ * @fault_addr: The faulting address.
+ * @attr_range: The attribute range covering @fault_addr.
+ * @write_fault: True if the fault was caused by a write.
+ *
+ * Run the fault mapping pipeline under the SVM lock: drain the garbage
+ * collector, check VMA permissions, find or insert the gpusvm range, skip
+ * work that a concurrent fault already handled, collect the pages and do
+ * the GPU mapping. Transient failures (HMM retry, changed CPU mappings)
+ * are retried a bounded number of times.
+ *
+ * Return: 0 on success or a benign drop, negative error code on failure.
+ */
+static int fault_map_range(struct amdgpu_svm *svm,
+			       unsigned long fault_addr,
+			       const struct amdgpu_svm_attr_range *attr_range,
+			       bool write_fault)
+{
+	const struct amdgpu_svm_attrs *attrs = &attr_range->attrs;
+	bool devmem_possible = amdgpu_svm_devmem_possible(svm);
+	bool need_vram_migration = devmem_possible &&
+				   amdgpu_svm_attr_prefer_vram(attrs);
+	struct drm_gpusvm_ctx map_ctx = {
+		.read_only = !!(attrs->flags & AMDGPU_SVM_ATTR_BIT_GPU_RO),
+		.devmem_possible = devmem_possible,
+		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
+		.devmem_only = need_vram_migration,
+		.timeslice_ms = need_vram_migration ? 5 : 0,
+	};
+	struct amdgpu_svm_range *range;
+	ktime_t timestamp = ktime_get_boottime();
+	int retry_count = 3;
+	int ret;
+
+	amdgpu_svm_assert_locked(svm);
+	WARN_ON(!svm->xnack_enabled);
+
+retry:
+	ret = amdgpu_svm_garbage_collector(svm);
+	if (ret) {
+		AMDGPU_SVM_ERR(
+			"fault garbage collector failed: ret=%d, fault_addr=0x%lx\n",
+			ret, fault_addr);
+		return ret;
+	}
+
+	ret = fault_check_allowed(svm, fault_addr, write_fault);
+	if (ret)
+		return ret;
+
+	range = amdgpu_svm_range_find_or_insert(
+		svm, fault_addr,
+		amdgpu_svm_attr_start(attr_range),
+		amdgpu_svm_attr_end(attr_range),
+		&map_ctx);
+	if (IS_ERR(range)) {
+		ret = PTR_ERR(range);
+		/*
+		 * -EINVAL: fault_addr out of gpusvm range, or no chunk size
+		 *          fits within VMA/notifier/attr_range bounds.
+		 * -EFAULT: mmget_not_zero failed (mm is dying).
+		 * -ENOENT: No VMA at fault_addr.
+		 */
+		if (ret == -EFAULT || ret == -ENOENT) {
+			AMDGPU_SVM_DBG("no vma or mm dying: 0x%lx ret=%d\n",
+					 fault_addr, ret);
+			ret = 0;
+			goto out;
+		}
+
+		AMDGPU_SVM_ERR("map_fault: range_find_or_insert failed: fault=0x%lx ret=%d\n",
+				 fault_addr, ret);
+		return ret;
+	}
+
+	if (write_fault && map_ctx.read_only) {
+		AMDGPU_SVM_WARN("write fault on read-only range: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				 fault_addr, drm_gpusvm_range_start(&range->base),
+				 drm_gpusvm_range_end(&range->base));
+	}
+
+	if (ktime_before(timestamp, ktime_add_ns(range->validate_timestamp,
+					 AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING))) {
+		AMDGPU_SVM_DBG("already restored, skip: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				 fault_addr, drm_gpusvm_range_start(&range->base),
+				 drm_gpusvm_range_end(&range->base));
+		goto out;
+	}
+
+	if (amdgpu_svm_range_is_valid(range, attrs)) {
+		AMDGPU_SVM_DBG("valid range, skip: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				 fault_addr, drm_gpusvm_range_start(&range->base),
+				 drm_gpusvm_range_end(&range->base));
+		goto out;
+	}
+
+	AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT");
+	/* TODO: add migration*/
+
+	AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
+	ret = amdgpu_svm_range_get_pages(svm, &range->base, &map_ctx);
+	if (ret == -EOPNOTSUPP || ret == -EFAULT || ret == -EBUSY) {
+		/*
+		 * -EOPNOTSUPP  Mixed page types within range.
+		 * -EFAULT      mm is dying，range was unmapped，DMA mapping failed，
+		 *              devmem_only requested but system page encountered，
+		 *              no VMA, page fault error, bad pte/pmd.
+		 * -EBUSY       HMM retry loop timed out.
+		 * -ENOMEM      PFN or DMA address array allocation failed.
+		 * -EINVAL      hmm_range_fault: invalid VMA type.
+		 */
+		map_ctx.timeslice_ms <<= 1;
+		if (!map_ctx.devmem_only && --retry_count > 0) {
+			AMDGPU_SVM_DBG(
+				"retry: get_pages failed %d, left=%d: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				ret, retry_count, fault_addr,
+				drm_gpusvm_range_start(&range->base),
+				drm_gpusvm_range_end(&range->base));
+			goto retry;
+		}
+		AMDGPU_SVM_ERR(
+			"map_fault: get_pages retry exhausted, ret=%d: fault=0x%lx range=[0x%lx-0x%lx)\n",
+			ret, fault_addr,
+			drm_gpusvm_range_start(&range->base),
+			drm_gpusvm_range_end(&range->base));
+	}
+
+	if (ret == -EPERM) {
+		/* According to AMD UMD contract, treat -EPERM as a error fault */
+		AMDGPU_SVM_ERR("get_pages -EPERM: fault=0x%lx range=[0x%lx-0x%lx)\n",
+			       fault_addr, drm_gpusvm_range_start(&range->base),
+				       drm_gpusvm_range_end(&range->base));
+		return ret;
+	}
+
+	if (ret == -ENOMEM) {
+		AMDGPU_SVM_ERR("get_pages -ENOMEM: fault=0x%lx range=[0x%lx-0x%lx)\n",
+			       fault_addr, drm_gpusvm_range_start(&range->base),
+			       drm_gpusvm_range_end(&range->base));
+		return ret;
+	}
+
+	if (ret) {
+		AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT - FAIL PAGE COLLECT");
+		goto out;
+	}
+
+	AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT - GPU MAP");
+
+	ret = amdgpu_svm_range_update_mapping(svm, range, attrs,
+					      map_ctx.read_only,
+					      false, false, false);
+
+	if (ret)
+		goto err_out;
+
+out:
+	return 0;
+
+err_out:
+	if (ret == -EAGAIN && --retry_count > 0) {
+		map_ctx.timeslice_ms <<= 1;
+		AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT - RETRY GPU MAP");
+		goto retry;
+	}
+
+	return ret;
+}
+
+/**
+ * amdgpu_svm_handle_fault() - Top-level SVM retry fault handler
+ * @adev: The amdgpu device.
+ * @pasid: PASID of the faulting context.
+ * @fault_page: Faulting page index.
+ * @ts: IH ring timestamp of the fault.
+ * @write_fault: True if the fault was caused by a write.
+ *
+ * Entry point from the VM fault path. Look up the SVM context, drop the
+ * fault if the context is gone/exiting or the mm is dying, filter stale
+ * retry faults using the checkpoint timestamp, then resolve the covering
+ * attribute range (creating a default one if needed) and map it.
+ *
+ * Return: 0 on success or a dropped fault, negative error code on failure.
+ */
+int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
+			    uint64_t fault_page, uint64_t ts,
+			    bool write_fault)
+{
+	struct amdgpu_svm *svm;
+	struct amdgpu_svm_attr_range *attr_range;
+	unsigned long attr_start_page, attr_last_page;
+	uint64_t fault_addr = fault_page << PAGE_SHIFT;
+	struct mm_struct *mm = NULL;
+	uint64_t ckpt;
+	int ret;
+
+	if (amdgpu_in_reset(adev))
+		return 0;
+
+	svm = amdgpu_svm_lookup_by_pasid(adev, pasid);
+	if (!svm) {
+		AMDGPU_SVM_DBG("no SVM context for pasid %u, drop fault 0x%llx\n",
+				 pasid, fault_page << PAGE_SHIFT);
+		return 0;
+	}
+
+	if (atomic_read(&svm->exiting)) {
+		AMDGPU_SVM_DBG("SVM context is exiting for pasid %u, drop fault 0x%llx\n",
+				 pasid, fault_page << PAGE_SHIFT);
+		ret = 0;
+		goto out_put;
+	}
+
+	mm = svm->gpusvm.mm;
+	if (!mm || !mmget_not_zero(mm)) {
+		AMDGPU_SVM_DBG("mm dying for pasid %u, drop fault 0x%llx\n",
+				 pasid, fault_page << PAGE_SHIFT);
+		mm = NULL;
+		ret = 0;
+		goto out_put;
+	}
+
+	if (!svm->xnack_enabled) {
+		AMDGPU_SVM_ERR("xnack not enabled for pasid %u\n",
+			       pasid);
+		ret = -EOPNOTSUPP;
+		goto out_put;
+	}
+
+	ckpt = READ_ONCE(svm->checkpoint_ts);
+	if (ckpt != 0) {
+		if (amdgpu_ih_ts_after_or_equal(ts, ckpt)) {
+			AMDGPU_SVM_DBG(
+				"draining stale retry fault, drop fault 0x%llx ts=%llu checkpoint=%llu\n",
+				fault_addr, ts, ckpt);
+			amdgpu_gmc_filter_faults_remove(
+				adev, fault_page, pasid);
+			ret = 0;
+			goto out_put;
+		} else {
+			WRITE_ONCE(svm->checkpoint_ts, 0);
+		}
+	}
+
+	amdgpu_svm_lock(svm);
+
+	mutex_lock(&svm->attr_tree->lock);
+	attr_range = amdgpu_svm_attr_get_bounds_locked(svm->attr_tree,
+						       fault_page,
+						       &attr_start_page, &attr_last_page);
+	mutex_unlock(&svm->attr_tree->lock);
+	if (!attr_range) {
+		ret = fault_get_unregistered_attrs(svm, fault_addr,
+							      attr_start_page,
+							      attr_last_page,
+							      &attr_range);
+		if (ret) {
+			if (ret == -EFAULT)
+				goto out_no_vma;
+			goto out_unlock;
+		}
+	}
+	ret = fault_map_range(svm, fault_addr, attr_range,
+					 write_fault);
+
+	if (ret == -EAGAIN) {
+		AMDGPU_SVM_DBG("retry fault=0x%llx\n",
+				 fault_addr);
+		amdgpu_gmc_filter_faults_remove(adev, fault_page, pasid);
+		ret = 0;
+	}
+
+	goto out_unlock;
+
+out_no_vma:
+	AMDGPU_SVM_ERR("no VMA for fault=0x%llx (stale retry or GPU NULL deref)\n",
+		 fault_addr);
+	ret = 0;
+
+out_unlock:
+	amdgpu_svm_unlock(svm);
+
+out_put:
+	if (mm)
+		mmput_async(mm);
+	amdgpu_svm_put(svm);
+	return ret;
+}
+
+#endif /* CONFIG_DRM_AMDGPU_SVM */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h
new file mode 100644
index 0000000000000..c169a96600de4
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_FAULT_H__
+#define __AMDGPU_SVM_FAULT_H__
+
+#include <linux/types.h>
+
+struct amdgpu_device;
+struct amdgpu_svm;
+struct amdgpu_svm_attr_range;
+struct amdgpu_svm_attrs;
+
+int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
+			    uint64_t fault_page, uint64_t ts,
+			    bool write_fault);
+
+#endif /* __AMDGPU_SVM_FAULT_H__ */
-- 
2.53.0

[PATCH v9 12/18] drm/amdgpu: integrate SVM into build system and VM fault path

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Wire up the SVM subsystem:

Kconfig:
- CONFIG_DRM_AMDGPU_SVM option, depends on DEVICE_PRIVATE, selects
  DRM_GPUSVM, HMM_MIRROR, MMU_NOTIFIER, default n

Makefile:
- Build amdgpu_svm.o, amdgpu_svm_attr.o, amdgpu_svm_fault.o,
  amdgpu_svm_range.o when CONFIG_DRM_AMDGPU_SVM=y
- KBUILD_EXTRA_SYMBOLS for drm Module.symvers
- clean-svm phony target for development convenience

amdgpu_drv.c:
- Register DRM_IOCTL_DEF_DRV in ioctl table

amdgpu_vm.c:
- Initialize vm->svm = NULL in amdgpu_vm_init
- Call amdgpu_svm_init in amdgpu_vm_make_compute
- Call amdgpu_svm_close + amdgpu_svm_fini in amdgpu_vm_fini
- Route GPU page faults through amdgpu_svm_handle_fault when SVM
  context exists, without falling back to svm_range_restore_pages

Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Kconfig      | 10 ++++++++++
 drivers/gpu/drm/amd/amdgpu/Makefile     |  9 +++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 23 +++++++++++++++++++++--
 4 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/Kconfig b/drivers/gpu/drm/amd/amdgpu/Kconfig
index 12e4a41bf1f04..68f519b3479db 100644
--- a/drivers/gpu/drm/amd/amdgpu/Kconfig
+++ b/drivers/gpu/drm/amd/amdgpu/Kconfig
@@ -81,6 +81,16 @@ config DRM_AMDGPU_USERPTR
 	  This option selects CONFIG_HMM and CONFIG_HMM_MIRROR if it
 	  isn't already selected to enabled full userptr support.
 
+config DRM_AMDGPU_SVM
+	bool "Enable AMDGPU SVM support (experimental)"
+	depends on DRM_AMDGPU
+	depends on DEVICE_PRIVATE
+	select DRM_GPUSVM
+	select HMM_MIRROR
+	select MMU_NOTIFIER
+	help
+	  Experimental SVM support based on DRM GPUSVM.
+
 config DRM_AMD_ISP
 	bool "Enable AMD Image Signal Processor IP support"
 	depends on DRM_AMDGPU && ACPI
diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index ba80542ead9d9..884cdc5c1e4af 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -322,6 +322,15 @@ amdgpu-$(CONFIG_VGA_SWITCHEROO) += amdgpu_atpx_handler.o
 amdgpu-$(CONFIG_ACPI) += amdgpu_acpi.o
 amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
 
+# svm support
+amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
+	amdgpu_svm_fault.o amdgpu_svm_range.o
+
+.PHONY: clean-svm
+clean-svm:
+	rm -f $(obj)/amdgpu_svm.o $(obj)/amdgpu_svm_attr.o $(obj)/amdgpu_svm_fault.o $(obj)/amdgpu_svm_range.o \
+	      $(obj)/.amdgpu_svm.o.cmd $(obj)/.amdgpu_svm_attr.o.cmd $(obj)/.amdgpu_svm_fault.o.cmd $(obj)/.amdgpu_svm_range.o.cmd
+
 include $(FULL_AMD_PATH)/pm/Makefile
 
 amdgpu-y += $(AMD_POWERPLAY_FILES)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 503bb64c1e55f..46a074caba53a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -51,6 +51,7 @@
 #include "amdgpu_ras.h"
 #include "amdgpu_reset.h"
 #include "amdgpu_sched.h"
+#include "amdgpu_svm.h"
 #include "amdgpu_xgmi.h"
 #include "amdgpu_userq.h"
 #include "amdgpu_userq_fence.h"
@@ -3077,6 +3078,7 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = {
 	DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(AMDGPU_GEM_LIST_HANDLES, amdgpu_gem_list_handles_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(AMDGPU_PROC_OPTIONS, amdgpu_proc_options_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
+	DRM_IOCTL_DEF_DRV(AMDGPU_GEM_SVM, amdgpu_gem_svm_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 };
 
 static const struct drm_driver amdgpu_kms_driver = {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 7a4d452a0ed9c..e91fa4cb5a16d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -44,6 +44,7 @@
 #include "amdgpu_xgmi.h"
 #include "amdgpu_dma_buf.h"
 #include "amdgpu_res_cursor.h"
+#include "amdgpu_svm.h"
 #include "kfd_svm.h"
 
 /**
@@ -2640,6 +2641,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	int r, i;
 
 	vm->va = RB_ROOT_CACHED;
+	vm->svm = NULL;
 	for (i = 0; i < AMDGPU_MAX_VMHUBS; i++)
 		vm->reserved_vmid[i] = NULL;
 
@@ -2798,6 +2800,10 @@ int amdgpu_vm_make_compute(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	vm->is_compute_context = true;
 	vm->need_tlb_fence = true;
 
+	r = amdgpu_svm_init(adev, vm);
+	if (r)
+		goto unreserve_bo;
+
 unreserve_bo:
 	amdgpu_bo_unreserve(vm->root.bo);
 	return r;
@@ -2830,6 +2836,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	unsigned long flags;
 	int i;
 
+	amdgpu_svm_close(vm);
+	amdgpu_svm_fini(vm);
+
 	amdgpu_amdkfd_gpuvm_destroy_cb(adev, vm);
 
 	root = amdgpu_bo_ref(vm->root.bo);
@@ -3053,6 +3062,7 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 {
 	struct amdgpu_vm_update_params params;
 	bool is_compute_context;
+	bool use_amdgpu_svm = false;
 	struct amdgpu_bo *root;
 	uint64_t value, flags;
 	struct amdgpu_vm *vm;
@@ -3064,14 +3074,23 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 		return false;
 
 	is_compute_context = vm->is_compute_context;
+	use_amdgpu_svm = !!vm->svm;
 
 	if (is_compute_context) {
 		/* Unreserve root since svm_range_restore_pages might try to reserve it. */
 		/* TODO: rework svm_range_restore_pages so that this isn't necessary. */
 		amdgpu_bo_unreserve(root);
 
-		if (!svm_range_restore_pages(adev, pasid, vmid,
-					     node_id, addr >> PAGE_SHIFT, ts, write_fault)) {
+		if (use_amdgpu_svm)
+			r = amdgpu_svm_handle_fault(adev, pasid,
+						    addr >> PAGE_SHIFT, ts,
+						    write_fault);
+		else
+			r = svm_range_restore_pages(adev, pasid, vmid, node_id,
+						    addr >> PAGE_SHIFT, ts,
+						    write_fault);
+
+		if (!r) {
 			amdgpu_bo_unref(&root);
 			return true;
 		}
-- 
2.53.0

[PATCH v9 13/18] drm/amdgpu: add VRAM migration infrastructure for drm_pagemap

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Add the drm_pagemap-based VRAM migration infrastructure:

- Define struct amdgpu_pagemap wrapping dev_pagemap + drm_pagemap
- Define AMDGPU_PGMAP_OWNER() and AMDGPU_INTERCONNECT_VRAM macros
- Implement amdgpu_svm_migration_init() to register ZONE_DEVICE via
  devm_memremap_pages() and initialize the drm_pagemap
- Add amdgpu_pagemap pointer (apagemap) to struct amdgpu_device

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Makefile         |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |   8 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 252 ++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h |  92 +++++++
 4 files changed, 356 insertions(+), 2 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index 884cdc5c1e4af..75047adbdd57b 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -324,12 +324,14 @@ amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
 
 # svm support
 amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
-	amdgpu_svm_fault.o amdgpu_svm_range.o
+	amdgpu_svm_fault.o amdgpu_svm_range.o amdgpu_migrate.o
 
 .PHONY: clean-svm
 clean-svm:
 	rm -f $(obj)/amdgpu_svm.o $(obj)/amdgpu_svm_attr.o $(obj)/amdgpu_svm_fault.o $(obj)/amdgpu_svm_range.o \
-	      $(obj)/.amdgpu_svm.o.cmd $(obj)/.amdgpu_svm_attr.o.cmd $(obj)/.amdgpu_svm_fault.o.cmd $(obj)/.amdgpu_svm_range.o.cmd
+	      $(obj)/amdgpu_migrate.o \
+	      $(obj)/.amdgpu_svm.o.cmd $(obj)/.amdgpu_svm_attr.o.cmd $(obj)/.amdgpu_svm_fault.o.cmd $(obj)/.amdgpu_svm_range.o.cmd \
+	      $(obj)/.amdgpu_migrate.o.cmd
 
 include $(FULL_AMD_PATH)/pm/Makefile
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 8326c18e07653..c36ffa5414977 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -328,6 +328,7 @@ struct amdgpu_fpriv;
 struct amdgpu_bo_va_mapping;
 struct kfd_vm_fault_info;
 struct amdgpu_hive_info;
+struct amdgpu_pagemap;
 struct amdgpu_reset_context;
 struct amdgpu_reset_control;
 struct amdgpu_coredump_info;
@@ -1129,6 +1130,13 @@ struct amdgpu_device {
 
 	struct amdgpu_uma_carveout_info uma_info;
 
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+	/* SVM VRAM migration via drm_pagemap (drm_gpusvm path).
+	 * Allocated in amdgpu_svm_migration_init(), NULL if SVM disabled.
+	 */
+	struct amdgpu_pagemap		*apagemap;
+#endif
+
 	/* KFD
 	 * Must be last --ends in a flexible-array member.
 	 */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
new file mode 100644
index 0000000000000..054520d3ff1f7
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -0,0 +1,252 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+/**
+ * DOC: AMDGPU SVM Migration
+ *
+ * This file implements the drm_pagemap-based migration infrastructure for
+ * AMDGPU SVM. It provides the callbacks that the DRM GPUSVM / drm_pagemap
+ * framework needs to:
+ *
+ * 1. Map ZONE_DEVICE pages to GPU-visible VRAM MC addresses (device_map)
+ * 2. Allocate VRAM and migrate pages from system memory (populate_mm)
+ * 3. Copy data between RAM and VRAM using SDMA (copy_to_devmem / copy_to_ram)
+ * 4. Release VRAM backing when pages migrate back to system memory (devmem_release)
+ *
+ * Architecture overview::
+ *
+ *   adev->apagemap->dpagemap   (struct drm_pagemap)
+ *       .ops = &amdgpu_svm_drm_pagemap_ops
+ *           |
+ *       +---+-------------------+
+ *       |                       |
+ *  .populate_mm            .device_map
+ *  (alloc BO + migrate)    (page -> VRAM MC addr)
+ *       |
+ *       v
+ *   drm_pagemap_devmem_ops  (per-BO migration mechanics)
+ *       .populate_devmem_pfn  -> BO buddy blocks -> PFN array
+ *       .copy_to_devmem       -> SDMA copy RAM -> VRAM
+ *       .copy_to_ram          -> SDMA copy VRAM -> RAM
+ *       .devmem_release       -> release BO reference
+ *
+ * The three address spaces involved:
+ *
+ *   VRAM offset  [0, real_vram_size)     - buddy allocator managed
+ *       + hpa_base
+ *   HPA / PFN    [hpa_base, hpa_base+..) - ZONE_DEVICE struct page management
+ *       + vm_manager.vram_base_offset
+ *   PTE address  [vram_base_offset, ..]  - GPU page table entries (from MMHUB FB_OFFSET)
+ */
+
+#include <drm/drm_drv.h>
+#include <drm/drm_pagemap.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+
+#include "amdgpu_migrate.h"
+#include "amdgpu.h"
+
+static inline struct amdgpu_pagemap *
+dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
+{
+	return container_of(dpagemap, struct amdgpu_pagemap, dpagemap);
+}
+
+static inline struct amdgpu_device *
+dpagemap_to_adev(struct drm_pagemap *dpagemap)
+{
+	return drm_to_adev(dpagemap->drm);
+}
+
+/**
+ * amdgpu_svm_page_to_apagemap - Get amdgpu_pagemap from a ZONE_DEVICE page
+ * @page: A ZONE_DEVICE page backed by VRAM
+ *
+ * Follows: page -> pgmap -> container_of(apagemap)
+ */
+static inline struct amdgpu_pagemap *
+amdgpu_svm_page_to_apagemap(struct page *page)
+{
+	struct dev_pagemap *pgmap = page_pgmap(page);
+
+	return container_of(pgmap, struct amdgpu_pagemap, pgmap);
+}
+
+/**
+ * amdgpu_pagemap_destroy_work - Deferred destruction of amdgpu_pagemap
+ * @work: work_struct embedded in amdgpu_pagemap
+ *
+ * Called when drm_pagemap refcount drops to zero.  Releases the
+ * ZONE_DEVICE mapping and frees the amdgpu_pagemap structure.
+ * Uses drm_dev_enter/exit to ensure device is still valid.
+ */
+static void amdgpu_pagemap_destroy_work(struct work_struct *work)
+{
+	struct amdgpu_pagemap *apagemap =
+		container_of(work, struct amdgpu_pagemap, destroy_work);
+	struct dev_pagemap *pgmap = &apagemap->pgmap;
+	struct drm_device *drm = apagemap->dpagemap.drm;
+	int idx;
+
+	/*
+	 * Only unmap/release if devm release hasn't run yet.
+	 * Otherwise the devm callbacks have already released, or
+	 * will do shortly.
+	 */
+	if (drm_dev_enter(drm, &idx)) {
+		devm_memunmap_pages(drm->dev, pgmap);
+		if (pgmap->type == MEMORY_DEVICE_PRIVATE)
+			devm_release_mem_region(drm->dev, pgmap->range.start,
+						pgmap->range.end - pgmap->range.start + 1);
+		drm_dev_exit(idx);
+	}
+
+	kfree(apagemap);
+}
+
+/**
+ * amdgpu_pagemap_destroy - drm_pagemap_ops.destroy callback
+ * @dpagemap: the drm_pagemap being destroyed
+ * @from_atomic_or_reclaim: true if called from atomic/reclaim context
+ *
+ * Called when the drm_pagemap refcount drops to zero.  If called from
+ * atomic or reclaim context, queues the destroy work to run later.
+ * Otherwise performs cleanup synchronously.
+ */
+static void amdgpu_pagemap_destroy(struct drm_pagemap *dpagemap,
+				   bool from_atomic_or_reclaim)
+{
+	struct amdgpu_pagemap *apagemap = dpagemap_to_apagemap(dpagemap);
+
+	if (from_atomic_or_reclaim)
+		schedule_work(&apagemap->destroy_work);
+	else
+		amdgpu_pagemap_destroy_work(&apagemap->destroy_work);
+}
+
+const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = {
+	.destroy = amdgpu_pagemap_destroy,
+};
+
+/**
+ * amdgpu_svm_migration_init - Register ZONE_DEVICE and initialize drm_pagemap
+ * @adev: AMDGPU device to set up VRAM migration for
+ *
+ * Allocates a ZONE_DEVICE region covering the GPU's VRAM, registers it
+ * via devm_memremap_pages() with drm_pagemap's generic dev_pagemap_ops,
+ * and then initializes the drm_pagemap (dpagemap) that provides the
+ * device_map / populate_mm callbacks for the DRM GPUSVM migration path.
+ *
+ * For XGMI-connected CPUs, uses the device's aperture directly
+ * (MEMORY_DEVICE_COHERENT).  For discrete GPUs, requests a free
+ * iomem region for MEMORY_DEVICE_PRIVATE pages.
+ *
+ * Return: 0 on success, -EINVAL if GPU IP too old, negative error on failure
+ */
+int amdgpu_svm_migration_init(struct amdgpu_device *adev)
+{
+	struct amdgpu_pagemap *apagemap;
+	struct drm_pagemap *dpagemap;
+	struct dev_pagemap *pgmap;
+	struct resource *res = NULL;
+	unsigned long size;
+	void *r;
+	int err;
+
+	if (amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(9, 0, 1))
+		return -EINVAL;
+
+	if (adev->apu_prefer_gtt)
+		return 0;
+
+	if (adev->apagemap && adev->apagemap->initialized)
+		return 0;
+
+	apagemap = kzalloc(sizeof(*apagemap), GFP_KERNEL);
+	if (!apagemap)
+		return -ENOMEM;
+
+	pgmap = &apagemap->pgmap;
+	dpagemap = &apagemap->dpagemap;
+	INIT_WORK(&apagemap->destroy_work, amdgpu_pagemap_destroy_work);
+	apagemap->adev = adev;
+
+	/* Initialize drm_pagemap first — takes refs on drm_device and module */
+	err = drm_pagemap_init(dpagemap, pgmap, adev_to_drm(adev),
+			       &amdgpu_svm_drm_pagemap_ops);
+	if (err) {
+		dev_err(adev->dev, "SVM: failed to init drm_pagemap\n");
+		goto err_free;
+	}
+
+	/* Now allocate ZONE_DEVICE resources */
+	size = ALIGN(adev->gmc.real_vram_size, 2ULL << 20);
+	if (adev->gmc.xgmi.connected_to_cpu) {
+		pgmap->range.start = adev->gmc.aper_base;
+		pgmap->range.end = adev->gmc.aper_base + adev->gmc.aper_size - 1;
+		pgmap->type = MEMORY_DEVICE_COHERENT;
+	} else {
+		res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
+		if (IS_ERR(res)) {
+			err = PTR_ERR(res);
+			goto err_pagemap;
+		}
+		pgmap->range.start = res->start;
+		pgmap->range.end = res->end;
+		pgmap->type = MEMORY_DEVICE_PRIVATE;
+	}
+
+	pgmap->nr_range = 1;
+	pgmap->flags = 0;
+	pgmap->ops = drm_pagemap_pagemap_ops_get();
+	pgmap->owner = AMDGPU_PGMAP_OWNER(adev);
+
+	r = devm_memremap_pages(adev->dev, pgmap);
+	if (IS_ERR(r)) {
+		dev_err(adev->dev, "SVM: failed to register HMM device memory\n");
+		err = PTR_ERR(r);
+		goto err_mem_region;
+	}
+
+	apagemap->hpa_base = pgmap->range.start;
+	apagemap->initialized = true;
+	adev->apagemap = apagemap;
+
+	dev_info(adev->dev, "SVM: registered %ldMB device memory, hpa_base=0x%llx\n",
+		 size >> 20, apagemap->hpa_base);
+	return 0;
+
+err_mem_region:
+	if (pgmap->type == MEMORY_DEVICE_PRIVATE && res)
+		devm_release_mem_region(adev->dev, res->start, resource_size(res));
+	pgmap->type = 0;
+err_pagemap:
+	drm_pagemap_put(dpagemap);
+	return err;
+
+err_free:
+	kfree(apagemap);
+	return err;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
new file mode 100644
index 0000000000000..f5f726a740469
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_MIGRATE_H__
+#define __AMDGPU_MIGRATE_H__
+
+#include <drm/drm_pagemap.h>
+#include <linux/memremap.h>
+#include <linux/workqueue.h>
+
+struct amdgpu_device;
+
+/*
+ * AMDGPU_PGMAP_OWNER - Unique owner token for dev_pagemap registration.
+ *
+ * migrate_vma_setup() uses pgmap->owner to distinguish "own" device pages
+ * from "foreign" device pages (e.g., another GPU in an XGMI hive).
+ * Pages whose page->pgmap->owner matches the migration source are skipped
+ * (they're already in the right place).
+ *
+ * For XGMI hive: all GPUs in the hive share the same owner (the hive pointer)
+ * so intra-hive pages are treated as local.
+ * For standalone GPU: use the adev pointer itself as a unique per-device token.
+ */
+#define AMDGPU_PGMAP_OWNER(adev) \
+	((adev)->hive ? (void *)(adev)->hive : (void *)(adev))
+
+/**
+ * struct amdgpu_pagemap - VRAM migration infrastructure for drm_pagemap
+ * @dpagemap: DRM pagemap wrapper providing device_map/populate_mm callbacks
+ * @adev: back-pointer to the owning amdgpu_device
+ * @hpa_base: host physical address base of the ZONE_DEVICE region
+ * @initialized: set to true after successful registration
+ * @destroy_work: work struct for deferred destruction when called from
+ *                atomic/reclaim context
+ * @pgmap: the dev_pagemap registered with devm_memremap_pages();
+ *         must be last — contains a flexible-array member (ranges[])
+ *
+ * Allocated with kzalloc() in amdgpu_svm_migration_init() and stored
+ * as adev->apagemap.  Freed via drm_pagemap_ops.destroy callback when
+ * dpagemap refcount drops to zero.
+ */
+struct amdgpu_pagemap {
+	struct drm_pagemap	dpagemap;
+	struct amdgpu_device	*adev;
+	resource_size_t		hpa_base;
+	bool			initialized;
+	struct work_struct	destroy_work;
+	struct dev_pagemap	pgmap;	/* must be last — flex-array */
+};
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+int amdgpu_svm_migration_init(struct amdgpu_device *adev);
+#else
+static inline
+int amdgpu_svm_migration_init(struct amdgpu_device *adev)
+{
+	return 0;
+}
+#endif
+
+/**
+ * amdgpu_svm_drm_pagemap_ops - drm_pagemap_ops for AMDGPU VRAM migration
+ *
+ * Provides:
+ *   .device_map  - Convert ZONE_DEVICE page to VRAM address
+ *   .populate_mm - Allocate VRAM BO and migrate pages from system memory
+ */
+extern const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops;
+
+#endif /* __AMDGPU_MIGRATE_H__ */
-- 
2.53.0

[PATCH v9 14/18] drm/amdgpu: implement drm_pagemap SDMA migration callbacks

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Implement the drm_pagemap_devmem_ops and drm_pagemap_ops callbacks
that the DRM GPUSVM migration framework requires.

Introduce struct amdgpu_bo_svm as a BO subtype (following the
amdgpu_bo_user/amdgpu_bo_vm pattern) that embeds struct amdgpu_bo
and carries a drm_pagemap_devmem allocation.

drm_pagemap_ops (top-level entry points):
  - device_map:   convert ZONE_DEVICE page to GPU PTE address
  - populate_mm:  allocate amdgpu_bo_svm and trigger migration

drm_pagemap_devmem_ops (per-BO migration mechanics):
  - populate_devmem_pfn: walk BO buddy blocks to build PFN array
  - copy_to_devmem:      SDMA copy system RAM -> VRAM via GART window
  - copy_to_ram:         SDMA copy VRAM -> system RAM via GART window
  - devmem_release:      drop BO ref (triggers destroy to free amdgpu_bo_svm)

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 652 ++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h |   3 +
 2 files changed, 655 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
index 054520d3ff1f7..6aeba9220ac41 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -67,6 +67,17 @@
 
 #include "amdgpu_migrate.h"
 #include "amdgpu.h"
+#include "amdgpu_ttm.h"
+#include "amdgpu_res_cursor.h"
+
+#define AMDGPU_MIGRATE_TRACE(fmt, ...) \
+	pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
+
+/* SDMA copy direction */
+#define FROM_RAM_TO_VRAM	0
+#define FROM_VRAM_TO_RAM	1
+
+static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops;
 
 static inline struct amdgpu_pagemap *
 dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
@@ -94,6 +105,645 @@ amdgpu_svm_page_to_apagemap(struct page *page)
 	return container_of(pgmap, struct amdgpu_pagemap, pgmap);
 }
 
+/**
+ * struct amdgpu_bo_svm - SVM BO subtype with drm_pagemap devmem allocation
+ *
+ * @bo: Embedded base amdgpu_bo
+ * @devmem: drm_pagemap device memory allocation (passed to framework)
+ *
+ * Lifecycle is managed by the drm_pagemap framework's internal zdd refcount:
+ *   - zdd->devmem_allocation points to &svm_bo->devmem
+ *   - When zdd refcount drops to zero, framework calls devmem_release()
+ *   - devmem_release() drops the BO reference (triggering destroy callback)
+ */
+struct amdgpu_bo_svm {
+	struct amdgpu_bo bo;
+	struct drm_pagemap_devmem devmem;
+};
+
+#define to_amdgpu_bo_svm(abo) container_of((abo), struct amdgpu_bo_svm, bo)
+
+static inline struct amdgpu_bo_svm *
+devmem_to_amdgpu_bo_svm(struct drm_pagemap_devmem *devmem_allocation)
+{
+	return container_of(devmem_allocation, struct amdgpu_bo_svm, devmem);
+}
+
+/**
+ * amdgpu_bo_svm_destroy - TTM destroy callback for SVM BO
+ *
+ * Called when the last reference to the BO is dropped.
+ * Follows amdgpu_bo_user_destroy pattern: delegates to amdgpu_bo_base_destroy.
+ */
+static void amdgpu_bo_svm_destroy(struct ttm_buffer_object *tbo)
+{
+	struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
+	struct amdgpu_bo_svm *svm_bo = to_amdgpu_bo_svm(bo);
+
+	amdgpu_bo_kunmap(bo);
+	drm_gem_object_release(&bo->tbo.base);
+	amdgpu_bo_unref(&bo->parent);
+	kvfree(svm_bo);
+}
+
+/**
+ * amdgpu_bo_svm_alloc - Allocate an amdgpu_bo_svm with VRAM backing
+ * @adev: AMDGPU device
+ * @dpagemap: The drm_pagemap for this device
+ * @mm: mm_struct of the owning process
+ * @size: Allocation size in bytes
+ *
+ * Uses bp->bo_ptr_size = sizeof(struct amdgpu_bo_svm) so that
+ * amdgpu_bo_create() allocates the full amdgpu_bo_svm structure,
+ * with the embedded amdgpu_bo as the base.
+ *
+ * Return: Pointer to allocated amdgpu_bo_svm on success, ERR_PTR on failure
+ */
+static struct amdgpu_bo_svm *
+amdgpu_bo_svm_alloc(struct amdgpu_device *adev,
+		     struct drm_pagemap *dpagemap,
+		     struct mm_struct *mm, unsigned long size)
+{
+	struct dma_fence *pre_migrate_fence = NULL;
+	struct amdgpu_bo_param bp = {};
+	struct amdgpu_bo_svm *svm_bo;
+	struct amdgpu_bo *bo;
+	int ret;
+
+	bp.size = size;
+	bp.bo_ptr_size = sizeof(struct amdgpu_bo_svm);
+	bp.destroy = &amdgpu_bo_svm_destroy;
+	bp.domain = AMDGPU_GEM_DOMAIN_VRAM;
+	bp.type = ttm_bo_type_device;
+	bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS |
+		   AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
+		   AMDGPU_GEM_CREATE_VRAM_CLEARED;
+
+	ret = amdgpu_bo_create(adev, &bp, &bo);
+	if (ret) {
+		AMDGPU_MIGRATE_TRACE("Failed to create SVM BO\n");
+		return ERR_PTR(ret);
+	}
+
+	svm_bo = to_amdgpu_bo_svm(bo);
+
+	/* Ensure any async VRAM operations complete before migration copy */
+	if (!dma_resv_test_signaled(bo->tbo.base.resv,
+					 DMA_RESV_USAGE_KERNEL)) {
+		ret = dma_resv_get_singleton(bo->tbo.base.resv,
+					      DMA_RESV_USAGE_KERNEL,
+					      &pre_migrate_fence);
+		if (ret) {
+			amdgpu_bo_unref(&bo);
+			return ERR_PTR(ret);
+		}
+	}
+	drm_pagemap_devmem_init(&svm_bo->devmem, adev->dev, mm,
+				&amdgpu_pagemap_ops, dpagemap, size,
+				pre_migrate_fence);
+
+	return svm_bo;
+}
+
+/**
+ * amdgpu_svm_devmem_release - Release BO when all device pages migrate back
+ *
+ * Called by the drm_pagemap framework (via drm_pagemap_zdd_destroy) when the
+ * last device-private page backed by this allocation has been migrated back
+ * to system memory (or the owning process exits).
+ *
+ * Drops the BO reference, which triggers the destroy callback to free
+ * the amdgpu_bo_svm structure.
+ */
+static void
+amdgpu_svm_devmem_release(struct drm_pagemap_devmem *devmem_allocation)
+{
+	struct amdgpu_bo_svm *svm_bo = devmem_to_amdgpu_bo_svm(devmem_allocation);
+	struct amdgpu_bo *bo = &svm_bo->bo;
+
+	AMDGPU_MIGRATE_TRACE("Release svm_bo=%px bo=%px\n", svm_bo, bo);
+	dma_fence_put(devmem_allocation->pre_migrate_fence);
+	devmem_allocation->pre_migrate_fence = NULL;
+	amdgpu_bo_unref(&bo);
+}
+
+/**
+ * amdgpu_svm_populate_devmem_pfn - Convert BO VRAM allocation to PFN array
+ * @devmem_allocation: The devmem allocation in the amdgpu_bo_svm wrapper
+ * @npages: Number of PFN entries to fill
+ * @pfn: Output PFN array
+ *
+ * Iterates over the BO's TTM vram_mgr buddy blocks and converts each
+ * block's VRAM offset to ZONE_DEVICE PFNs:
+ *
+ *   PFN = PHYS_PFN(block_offset + apagemap.hpa_base) + page_index
+ *
+ * This is called by drm_pagemap_migrate_to_devmem() to build the
+ * destination PFN array for migrate_vma_pages().
+ *
+ * Return: 0 on success
+ */
+static int
+amdgpu_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocation,
+				unsigned long npages, unsigned long *pfn)
+{
+	struct amdgpu_pagemap *svm_dm = dpagemap_to_apagemap(devmem_allocation->dpagemap);
+	struct amdgpu_bo_svm *svm_bo = devmem_to_amdgpu_bo_svm(devmem_allocation);
+	struct amdgpu_bo *bo = &svm_bo->bo;
+	struct amdgpu_res_cursor cursor;
+	unsigned long i = 0;
+
+	dma_resv_assert_held(bo->tbo.base.resv);
+
+	amdgpu_res_first(bo->tbo.resource, 0, npages << PAGE_SHIFT, &cursor);
+	while (cursor.remaining && i < npages) {
+		u64 pfn_base = PHYS_PFN(cursor.start + svm_dm->hpa_base);
+		u64 pages = cursor.size >> PAGE_SHIFT;
+		unsigned long j;
+
+		for (j = 0; j < pages && i < npages; j++, i++)
+			pfn[i] = pfn_base + j;
+
+		amdgpu_res_next(&cursor, cursor.size);
+	}
+
+	AMDGPU_MIGRATE_TRACE("populate_devmem_pfn: npages=%lu first_pfn=0x%lx\n",
+			  npages, npages > 0 ? pfn[0] : 0);
+
+	return 0;
+}
+
+/* SDMA copy helpers — GART window based data transfer */
+
+/**
+ * amdgpu_svm_direct_mapping_addr - Convert VRAM offset to MC address
+ * @adev: AMDGPU device
+ * @vram_offset: Byte offset within VRAM
+ *
+ * Return: MC address suitable for SDMA src/dst
+ */
+static u64
+amdgpu_svm_direct_mapping_addr(struct amdgpu_device *adev, u64 vram_offset)
+{
+	return vram_offset + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM);
+}
+
+/**
+ * amdgpu_svm_gart_map - Map system DMA addresses into GART window
+ * @ring: SDMA ring for the GART update job
+ * @npages: Number of pages to map
+ * @addr: Array of system memory DMA addresses
+ * @gart_addr: Output — GART base address to use in SDMA copy
+ * @flags: PTE flags (e.g. writeable for RAM-to-VRAM src)
+ *
+ * Builds GART PTEs pointing at the given DMA addresses, submits an
+ * SDMA job to update the GART entries, and returns the GART address
+ * that can be used as src or dst in a subsequent amdgpu_copy_buffer().
+ *
+ * Uses GART window 0, protected by gtt_window_lock.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_gart_map(struct amdgpu_ring *ring,
+		     struct amdgpu_ttm_buffer_entity *entity,
+		     u64 npages,
+		     dma_addr_t *addr, u64 *gart_addr, u64 flags)
+{
+	struct amdgpu_device *adev = ring->adev;
+	struct amdgpu_job *job;
+	unsigned int num_dw, num_bytes;
+	struct dma_fence *fence;
+	u64 src_addr, dst_addr;
+	u64 pte_flags;
+	void *cpu_addr;
+	int r;
+
+	/* Use entity's GART window 0 */
+	*gart_addr = amdgpu_compute_gart_address(&adev->gmc, entity, 0);
+
+	num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
+	num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+
+	r = amdgpu_job_alloc_with_ib(adev, &entity->base,
+				     AMDGPU_FENCE_OWNER_UNDEFINED,
+				     num_dw * 4 + num_bytes,
+				     AMDGPU_IB_POOL_DELAYED,
+				     AMDGPU_KERNEL_JOB_ID_KFD_GART_MAP,
+				     &job);
+	if (r)
+		return r;
+
+	src_addr = num_dw * 4;
+	src_addr += job->ibs[0].gpu_addr;
+
+	dst_addr = amdgpu_bo_gpu_offset(adev->gart.bo);
+	dst_addr += (entity->gart_window_offs[0] >> AMDGPU_GPU_PAGE_SHIFT) * 8;
+	amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr,
+				dst_addr, num_bytes, 0);
+
+	amdgpu_ring_pad_ib(ring, &job->ibs[0]);
+	WARN_ON(job->ibs[0].length_dw > num_dw);
+
+	pte_flags = AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE;
+	pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED;
+	if (flags & AMDGPU_PTE_WRITEABLE)
+		pte_flags |= AMDGPU_PTE_WRITEABLE;
+	pte_flags |= adev->gart.gart_pte_flags;
+
+	cpu_addr = &job->ibs[0].ptr[num_dw];
+
+	amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr);
+	fence = amdgpu_job_submit(job);
+	dma_fence_put(fence);
+
+	return 0;
+}
+
+/**
+ * amdgpu_svm_copy_memory_gart - SDMA copy between system RAM and VRAM
+ * @adev: AMDGPU device
+ * @sys: Array of DMA addresses for system memory pages
+ * @vram: Array of VRAM byte offsets (relative to start of VRAM)
+ * @npages: Number of pages to copy
+ * @direction: FROM_RAM_TO_VRAM or FROM_VRAM_TO_RAM
+ * @mfence: In/out — carries the last SDMA fence for serialization
+ *
+ * Maps system memory pages into the GART window and uses SDMA to copy
+ * data to/from VRAM. Handles splitting into AMDGPU_GTT_MAX_TRANSFER_SIZE
+ * chunks. Acquires entity->lock internally to protect the GART window,
+ * matching the KFD svm_migrate_copy_memory_gart() pattern.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys,
+			    u64 *vram, u64 npages, int direction,
+			    struct dma_fence **mfence)
+{
+	const u64 max_pages = AMDGPU_GTT_MAX_TRANSFER_SIZE;
+	struct amdgpu_ring *ring = to_amdgpu_ring(adev->mman.buffer_funcs_scheds[0]);
+	struct amdgpu_ttm_buffer_entity *entity = &adev->mman.move_entities[0];
+	u64 gart_s, gart_d;
+	struct dma_fence *next;
+	u64 size;
+	int r;
+
+	mutex_lock(&entity->lock);
+
+	while (npages) {
+		size = min(max_pages, npages);
+
+		if (direction == FROM_VRAM_TO_RAM) {
+			gart_s = amdgpu_svm_direct_mapping_addr(adev, *vram);
+			r = amdgpu_svm_gart_map(ring, entity, size, sys,
+						&gart_d, AMDGPU_PTE_WRITEABLE);
+		} else {
+			r = amdgpu_svm_gart_map(ring, entity, size, sys,
+						&gart_s, 0);
+			gart_d = amdgpu_svm_direct_mapping_addr(adev, *vram);
+		}
+		if (r) {
+			dev_err(adev->dev, "failed %d to map GART for SDMA\n", r);
+			goto out_unlock;
+		}
+
+		AMDGPU_MIGRATE_TRACE("SDMA_COPY: %s npages=%llu vram_off=0x%llx\n",
+				  direction == FROM_RAM_TO_VRAM ? "RAM->VRAM" : "VRAM->RAM",
+				  size, (u64)*vram);
+
+		r = amdgpu_copy_buffer(adev, entity, gart_s, gart_d,
+				       size * PAGE_SIZE,
+				       NULL, &next, true, 0);
+		if (r) {
+			dev_err(adev->dev, "failed %d to copy buffer\n", r);
+			goto out_unlock;
+		}
+
+		dma_fence_put(*mfence);
+		*mfence = next;
+		npages -= size;
+		if (npages) {
+			sys += size;
+			vram += size;
+		}
+	}
+
+out_unlock:
+	mutex_unlock(&entity->lock);
+
+	return r;
+}
+
+/**
+ * amdgpu_svm_copy_to_devmem - SDMA copy system memory -> VRAM
+ * @pages: Array of destination ZONE_DEVICE pages (VRAM-backed)
+ * @pagemap_addr: Array of source DMA addresses (system memory, already mapped)
+ * @npages: Number of pages to copy
+ *
+ * Builds parallel sys[] and vram[] arrays from the framework-provided
+ * pagemap_addr and device pages, then submits batched SDMA copies via
+ * the GART window.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_to_devmem(struct page **pages,
+			   struct drm_pagemap_addr *pagemap_addr,
+			   unsigned long npages,
+			   struct dma_fence *pre_migrate_fence)
+{
+	struct amdgpu_device *adev;
+	struct amdgpu_pagemap *svm_dm;
+	struct dma_fence *mfence = NULL;
+	dma_addr_t *sys;
+	u64 *vram;
+	unsigned long i, j;
+	int ret = 0;
+
+	if (!npages)
+		return 0;
+
+	/*
+	 * Find the first non-NULL page to derive the device.
+	 * The pages array may contain NULL entries for positions where
+	 * no valid device page exists.
+	 */
+	for (i = 0; i < npages; i++) {
+		if (pages[i])
+			break;
+	}
+	if (i == npages)
+		return 0;
+
+	svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
+	adev = svm_dm->adev;
+
+	/* Wait for async VRAM operations to complete before overwriting with data */
+	if (pre_migrate_fence)
+		dma_fence_wait(pre_migrate_fence, false);
+
+	sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
+	vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
+	if (!sys || !vram) {
+		ret = -ENOMEM;
+		goto out_free;
+	}
+
+	for (i = 0, j = 0; i < npages; i++) {
+		if (!pagemap_addr[i].addr || !pages[i])
+			goto flush;
+
+		sys[j] = pagemap_addr[i].addr;
+		vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+			  svm_dm->hpa_base;
+
+		/* Check if next vram page is contiguous with current */
+		if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
+			goto flush;
+
+		j++;
+		continue;
+flush:
+		if (j) {
+			ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+							  FROM_RAM_TO_VRAM,
+							  &mfence);
+			if (ret)
+				goto out_fence;
+			j = 0;
+		}
+		/* Re-process current page if it was valid but broke contiguity */
+		if (pagemap_addr[i].addr && pages[i]) {
+			sys[0] = pagemap_addr[i].addr;
+			vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+				  svm_dm->hpa_base;
+			j = 1;
+		}
+	}
+
+	/* Flush remaining batch */
+	if (j)
+		ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+						  FROM_RAM_TO_VRAM, &mfence);
+
+out_fence:
+	if (mfence) {
+		dma_fence_wait(mfence, false);
+		dma_fence_put(mfence);
+	}
+
+	AMDGPU_MIGRATE_TRACE("copy_to_devmem done: npages=%ld ret=%d\n",
+			  npages, ret);
+
+out_free:
+	kvfree(vram);
+	kvfree(sys);
+	return ret;
+}
+
+/**
+ * amdgpu_svm_copy_to_ram - SDMA copy VRAM -> system memory
+ * @pages: Array of source ZONE_DEVICE pages (VRAM-backed)
+ * @pagemap_addr: Array of destination DMA addresses (system memory, already mapped)
+ * @npages: Number of pages to copy
+ *
+ * Mirror of copy_to_devmem with src/dst swapped.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_to_ram(struct page **pages,
+			struct drm_pagemap_addr *pagemap_addr,
+			unsigned long npages,
+			struct dma_fence *pre_migrate_fence)
+{
+	struct amdgpu_device *adev;
+	struct amdgpu_pagemap *svm_dm;
+	struct dma_fence *mfence = NULL;
+	dma_addr_t *sys;
+	u64 *vram;
+	unsigned long i, j;
+	int ret = 0;
+
+	if (!npages)
+		return 0;
+
+	for (i = 0; i < npages; i++) {
+		if (pages[i])
+			break;
+	}
+	if (i == npages)
+		return 0;
+
+	svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
+	adev = svm_dm->adev;
+
+	/* Wait for async VRAM operations to complete before reading data */
+	if (pre_migrate_fence)
+		dma_fence_wait(pre_migrate_fence, false);
+
+	sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
+	vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
+	if (!sys || !vram) {
+		ret = -ENOMEM;
+		goto out_free;
+	}
+
+	for (i = 0, j = 0; i < npages; i++) {
+		if (!pagemap_addr[i].addr || !pages[i])
+			goto flush;
+
+		vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+			  svm_dm->hpa_base;
+		sys[j] = pagemap_addr[i].addr;
+
+		/* Check if next vram page is contiguous with current */
+		if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
+			goto flush;
+
+		j++;
+		continue;
+flush:
+		if (j) {
+			ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+							  FROM_VRAM_TO_RAM,
+							  &mfence);
+			if (ret)
+				goto out_fence;
+			j = 0;
+		}
+		/* Re-process current page if it was valid but broke contiguity */
+		if (pagemap_addr[i].addr && pages[i]) {
+			vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+				  svm_dm->hpa_base;
+			sys[0] = pagemap_addr[i].addr;
+			j = 1;
+		}
+	}
+
+	/* Flush remaining batch */
+	if (j)
+		ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+						  FROM_VRAM_TO_RAM, &mfence);
+
+out_fence:
+	if (mfence) {
+		dma_fence_wait(mfence, false);
+		dma_fence_put(mfence);
+	}
+
+	AMDGPU_MIGRATE_TRACE("copy_to_ram done: npages=%ld ret=%d\n", npages, ret);
+
+out_free:
+	kvfree(vram);
+	kvfree(sys);
+	return ret;
+}
+
+static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = {
+	.devmem_release      = amdgpu_svm_devmem_release,
+	.populate_devmem_pfn = amdgpu_svm_populate_devmem_pfn,
+	.copy_to_devmem      = amdgpu_svm_copy_to_devmem,
+	.copy_to_ram         = amdgpu_svm_copy_to_ram,
+};
+
+/* drm_pagemap_ops — top-level migration entry points */
+
+/**
+ * amdgpu_svm_device_map - Convert ZONE_DEVICE page to GPU PTE address
+ * @dpagemap: The drm_pagemap for this device
+ * @dev: Requesting device (for P2P check)
+ * @page: ZONE_DEVICE page backed by VRAM
+ * @order: Page order (0 = 4K, 9 = 2M, etc.)
+ * @dir: DMA direction (unused for local VRAM)
+ *
+ * Address conversion chain:
+ *   page -> PFN -> HPA -> VRAM offset -> PTE address
+ *
+ *   HPA = page_to_pfn(page) << PAGE_SHIFT
+ *   VRAM offset = HPA - apagemap.hpa_base
+ *   PTE address = VRAM offset + adev->vm_manager.vram_base_offset
+ *
+ * Return: drm_pagemap_addr with PTE address and AMDGPU_INTERCONNECT_VRAM protocol
+ */
+static struct drm_pagemap_addr
+amdgpu_svm_device_map(struct drm_pagemap *dpagemap,
+		       struct device *dev,
+		       struct page *page,
+		       unsigned int order,
+		       enum dma_data_direction dir)
+{
+	struct amdgpu_pagemap *svm_dm = dpagemap_to_apagemap(dpagemap);
+	struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
+	dma_addr_t addr;
+
+	if (dpagemap->drm->dev == dev) {
+		/* Same device: return VRAM PTE address */
+		u64 hpa = (u64)page_to_pfn(page) << PAGE_SHIFT;
+		u64 vram_offset = hpa - svm_dm->hpa_base;
+
+		addr = vram_offset + adev->vm_manager.vram_base_offset;
+	} else {
+		/* Cross-device P2P: not yet supported */
+		addr = DMA_MAPPING_ERROR;
+	}
+
+	return drm_pagemap_addr_encode(addr,
+				AMDGPU_INTERCONNECT_VRAM, order, dir);
+}
+
+/**
+ * amdgpu_svm_populate_mm - Allocate VRAM BO and migrate pages
+ * @dpagemap: The drm_pagemap for this device
+ * @start: Start virtual address of the range to migrate
+ * @end: End virtual address (exclusive)
+ * @mm: mm_struct of the owning process
+ * @timeslice_ms: Maximum time to spend migrating (for fairness)
+ *
+ * Core migration entry point called by drm_pagemap_populate_mm().
+ * Allocates an amdgpu_bo_svm via amdgpu_bo_svm_alloc(), then calls
+ * drm_pagemap_migrate_to_devmem() to execute the actual migration.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_populate_mm(struct drm_pagemap *dpagemap,
+			unsigned long start, unsigned long end,
+			struct mm_struct *mm,
+			unsigned long timeslice_ms)
+{
+	struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
+	struct drm_pagemap_migrate_details mdetails = {
+		.timeslice_ms = timeslice_ms,
+	};
+	struct amdgpu_bo_svm *svm_bo;
+	int ret;
+
+	svm_bo = amdgpu_bo_svm_alloc(adev, dpagemap, mm, end - start);
+	if (IS_ERR(svm_bo))
+		return PTR_ERR(svm_bo);
+
+	AMDGPU_MIGRATE_TRACE("populate_mm: [0x%lx-0x%lx] size=%lu\n",
+			  start, end, end - start);
+
+	ret = amdgpu_bo_reserve(&svm_bo->bo, true);
+	if (ret) {
+		struct amdgpu_bo *bo = &svm_bo->bo;
+
+		amdgpu_bo_unref(&bo);
+		return ret;
+	}
+
+	ret = drm_pagemap_migrate_to_devmem(&svm_bo->devmem,
+					     mm, start, end,
+					     &mdetails);
+
+	amdgpu_bo_unreserve(&svm_bo->bo);
+
+	return ret;
+}
+
 /**
  * amdgpu_pagemap_destroy_work - Deferred destruction of amdgpu_pagemap
  * @work: work_struct embedded in amdgpu_pagemap
@@ -148,6 +798,8 @@ static void amdgpu_pagemap_destroy(struct drm_pagemap *dpagemap,
 
 const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = {
 	.destroy = amdgpu_pagemap_destroy,
+	.device_map = amdgpu_svm_device_map,
+	.populate_mm = amdgpu_svm_populate_mm,
 };
 
 /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
index f5f726a740469..56c66851f0a47 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
@@ -31,6 +31,9 @@
 
 struct amdgpu_device;
 
+#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
+#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
+
 /*
  * AMDGPU_PGMAP_OWNER - Unique owner token for dev_pagemap registration.
  *
-- 
2.53.0

[PATCH v9 15/18] drm/amdgpu: implement synchronous TTM eviction for SVM BOs

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Implement the TTM eviction path for SVM (Shared Virtual Memory) BOs,
enabling VRAM overcommit scenarios where device-private pages must be
migrated back to system memory when VRAM is full.

- amdgpu_ttm.c: Add SVM BO detection in amdgpu_evict_flags() to evict
  directly to SYSTEM domain. Add SVM BO handling in amdgpu_bo_move()
  to call amdgpu_svm_bo_evict() for VRAM->SYSTEM transitions, which
  synchronously migrates device-private pages back to RAM via
  SDMA.

- amdgpu_migrate.c: Add amdgpu_svm_bo_evict() and
  amdgpu_svm_bo_is_svm() helpers.

- amdgpu_object.c: Register SVM BO destroy callback in
  amdgpu_bo_is_amdgpu_bo() so TTM eviction can identify and handle
  SVM BOs correctly.

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 44 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h | 12 ++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c     | 20 ++++++++++
 4 files changed, 79 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
index 6aeba9220ac41..26b99e3088f23 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -64,6 +64,7 @@
 #include <drm/drm_pagemap.h>
 #include <linux/memremap.h>
 #include <linux/migrate.h>
+#include <linux/sched/mm.h>
 
 #include "amdgpu_migrate.h"
 #include "amdgpu.h"
@@ -648,6 +649,49 @@ static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = {
 	.copy_to_ram         = amdgpu_svm_copy_to_ram,
 };
 
+/**
+ * amdgpu_svm_bo_evict - Evict SVM BO by migrating device pages back to RAM
+ * @bo: The amdgpu_bo to evict (must be an amdgpu_bo_svm)
+ *
+ * Called from amdgpu_bo_move() when TTM needs to evict a SVM BO from VRAM.
+ * Uses drm_pagemap_evict_to_ram() to synchronously migrate all device-private
+ * pages back to system memory via SDMA.
+ *
+ * If the owning process is exiting, there is nothing to migrate; return -EBUSY
+ * so the TTM eviction LRU walk skips this BO instead of aborting the whole walk
+ * (which would fail the allocation that triggered the eviction). Hold an mm
+ * reference across the call so the framework's own dead-mm path stays
+ * unreachable and any -EFAULT it returns is unambiguously a real error.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+int amdgpu_svm_bo_evict(struct amdgpu_bo *bo)
+{
+	struct amdgpu_bo_svm *sbo = to_amdgpu_bo_svm(bo);
+	struct mm_struct *mm = sbo->devmem.mm;
+	int r;
+
+	if (!mmget_not_zero(mm))
+		return -EBUSY;
+
+	r = drm_pagemap_evict_to_ram(&sbo->devmem);
+
+	mmput_async(mm);
+
+	return r;
+}
+
+/**
+ * amdgpu_svm_bo_is_svm - Check if a TTM BO is an SVM BO
+ * @tbo: TTM buffer object
+ *
+ * Return: true if this is an amdgpu_bo_svm (identified by destroy callback)
+ */
+bool amdgpu_svm_bo_is_svm(struct ttm_buffer_object *tbo)
+{
+	return tbo->destroy == &amdgpu_bo_svm_destroy;
+}
+
 /* drm_pagemap_ops — top-level migration entry points */
 
 /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
index 56c66851f0a47..e9159e8167dd7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
@@ -30,6 +30,8 @@
 #include <linux/workqueue.h>
 
 struct amdgpu_device;
+struct amdgpu_bo;
+struct ttm_buffer_object;
 
 #define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
 #define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
@@ -75,12 +77,22 @@ struct amdgpu_pagemap {
 
 #if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
 int amdgpu_svm_migration_init(struct amdgpu_device *adev);
+int amdgpu_svm_bo_evict(struct amdgpu_bo *bo);
+bool amdgpu_svm_bo_is_svm(struct ttm_buffer_object *tbo);
 #else
 static inline
 int amdgpu_svm_migration_init(struct amdgpu_device *adev)
 {
 	return 0;
 }
+static inline int amdgpu_svm_bo_evict(struct amdgpu_bo *bo)
+{
+	return -ENODEV;
+}
+static inline bool amdgpu_svm_bo_is_svm(struct ttm_buffer_object *tbo)
+{
+	return false;
+}
 #endif
 
 /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 4dd7c712b8c31..d6ea8737dedc5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -43,6 +43,7 @@
 #include "amdgpu_vram_mgr.h"
 #include "amdgpu_vm.h"
 #include "amdgpu_dma_buf.h"
+#include "amdgpu_migrate.h"
 
 /**
  * DOC: amdgpu_object
@@ -93,7 +94,8 @@ static void amdgpu_bo_user_destroy(struct ttm_buffer_object *tbo)
 bool amdgpu_bo_is_amdgpu_bo(struct ttm_buffer_object *bo)
 {
 	if (bo->destroy == &amdgpu_bo_destroy ||
-	    bo->destroy == &amdgpu_bo_user_destroy)
+	    bo->destroy == &amdgpu_bo_user_destroy ||
+	    amdgpu_svm_bo_is_svm(bo))
 		return true;
 
 	return false;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index cf78d70204941..6632abd92311b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -59,6 +59,7 @@
 #include "amdgpu_hmm.h"
 #include "amdgpu_atomfirmware.h"
 #include "amdgpu_res_cursor.h"
+#include "amdgpu_migrate.h"
 #include "bif/bif_4_1_d.h"
 
 MODULE_IMPORT_NS("DMA_BUF");
@@ -121,6 +122,15 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,
 		return;
 	}
 
+	/* SVM BOs must evict directly to system memory for drm_pagemap
+	 * migration back to RAM in amdgpu_bo_move().
+	 */
+	if (amdgpu_svm_bo_is_svm(bo)) {
+		amdgpu_bo_placement_from_domain(abo, AMDGPU_GEM_DOMAIN_CPU);
+		*placement = abo->placement;
+		return;
+	}
+
 	switch (bo->resource->mem_type) {
 	case AMDGPU_PL_GDS:
 	case AMDGPU_PL_GWS:
@@ -576,6 +586,16 @@ static int amdgpu_bo_move(struct ttm_buffer_object *bo, bool evict,
 		abo->flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
 	}
 
+	/* SVM BO eviction: migrate device-private pages back to RAM */
+	if (amdgpu_svm_bo_is_svm(bo) &&
+	    old_mem->mem_type == TTM_PL_VRAM &&
+	    new_mem->mem_type == TTM_PL_SYSTEM) {
+		r = amdgpu_svm_bo_evict(abo);
+		if (!r)
+			ttm_bo_move_null(bo, new_mem);
+		return r;
+	}
+
 	if (adev->mman.buffer_funcs_enabled &&
 	    ((old_mem->mem_type == TTM_PL_SYSTEM &&
 	      new_mem->mem_type == TTM_PL_VRAM) ||
-- 
2.53.0

[PATCH v9 16/18] drm/amdgpu: hook up ZONE_DEVICE registration in device init and reset

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Call amdgpu_svm_migration_init() in the device initialization and
XGMI reset-restore paths to register the GPU's VRAM as a ZONE_DEVICE
region before KFD initialization.

This activates the drm_pagemap migration infrastructure.

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c  | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index dc8c650fc3416..b5b6b2a1eb066 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -78,6 +78,7 @@
 #include "amdgpu_reset.h"
 #include "amdgpu_virt.h"
 #include "amdgpu_dev_coredump.h"
+#include "amdgpu_migrate.h"
 
 #include <linux/suspend.h>
 #include <drm/task_barrier.h>
@@ -4102,6 +4103,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 
 	/* Don't init kfd if whole hive need to be reset during init */
 	if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) {
+		amdgpu_svm_migration_init(adev);
 		kgd2kfd_init_zone_device(adev);
 		kfd_update_svm_support_properties(adev);
 	}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
index da77e0ef129ba..c2ea540e392b4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
@@ -25,6 +25,7 @@
 #include "aldebaran.h"
 #include "sienna_cichlid.h"
 #include "smu_v13_0_10.h"
+#include "amdgpu_migrate.h"
 
 static int amdgpu_reset_xgmi_reset_on_init_suspend(struct amdgpu_device *adev)
 {
@@ -87,6 +88,7 @@ static int amdgpu_reset_xgmi_reset_on_init_restore_hwctxt(
 		return r;
 	list_for_each_entry(tmp_adev, reset_device_list, reset_list) {
 		if (!tmp_adev->kfd.init_complete) {
+			amdgpu_svm_migration_init(tmp_adev);
 			kgd2kfd_init_zone_device(tmp_adev);
 			amdgpu_amdkfd_device_init(tmp_adev);
 			amdgpu_amdkfd_drm_client_create(tmp_adev);
-- 
2.53.0

[PATCH v9 17/18] drm/amdgpu: add SVM range migration helpers for drm_pagemap

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Add amdgpu_svm_range_migrate.c/.h implementing the per-range migration
helpers for SVM VRAM migration via drm_pagemap:

- Implement range_needs_migrate_to_vram() checking migrate_devmem
  capability and current backing location.
- Add amdgpu_svm_range_migrate_to_vram() wrapping
  drm_pagemap_populate_mm() for RAM-to-VRAM migration. On -EBUSY
  (partial migration due to races), evict conflicting pages back to
  system memory via drm_gpusvm_range_evict() and retry once.
- Add amdgpu_pagemap_capable() to check device memory support.

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Makefile           |   6 +-
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 120 ++++++++++++++++++
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.h |  35 +++++
 3 files changed, 158 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index 75047adbdd57b..0d08561b4922a 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -324,14 +324,14 @@ amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
 
 # svm support
 amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
-	amdgpu_svm_fault.o amdgpu_svm_range.o amdgpu_migrate.o
+	amdgpu_svm_fault.o amdgpu_svm_range.o amdgpu_svm_range_migrate.o amdgpu_migrate.o
 
 .PHONY: clean-svm
 clean-svm:
 	rm -f $(obj)/amdgpu_svm.o $(obj)/amdgpu_svm_attr.o $(obj)/amdgpu_svm_fault.o $(obj)/amdgpu_svm_range.o \
-	      $(obj)/amdgpu_migrate.o \
+	      $(obj)/amdgpu_svm_range_migrate.o $(obj)/amdgpu_migrate.o \
 	      $(obj)/.amdgpu_svm.o.cmd $(obj)/.amdgpu_svm_attr.o.cmd $(obj)/.amdgpu_svm_fault.o.cmd $(obj)/.amdgpu_svm_range.o.cmd \
-	      $(obj)/.amdgpu_migrate.o.cmd
+	      $(obj)/.amdgpu_svm_range_migrate.o.cmd $(obj)/.amdgpu_migrate.o.cmd
 
 include $(FULL_AMD_PATH)/pm/Makefile
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
new file mode 100644
index 0000000000000..a908989964f4b
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
@@ -0,0 +1,120 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu.h"
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu_migrate.h"
+#include "amdgpu_svm_range_migrate.h"
+
+static bool
+range_in_vram(struct drm_gpusvm_range *range)
+{
+	struct drm_gpusvm_pages_flags flags = {
+		/* Pairs with WRITE_ONCE in drm_gpusvm_get_pages() */
+		.__flags = READ_ONCE(range->pages.flags.__flags),
+	};
+
+	return flags.has_devmem_pages;
+}
+
+static bool
+range_needs_migrate_to_vram(struct drm_gpusvm_range *range)
+{
+	if (!range->pages.flags.migrate_devmem)
+		return false;
+	if (range_in_vram(range))
+		return false;
+
+	return true;
+}
+
+static struct drm_pagemap *
+amdgpu_svm_get_dpagemap(struct amdgpu_svm *svm)
+{
+	struct amdgpu_pagemap *apagemap = svm->adev->apagemap;
+
+	if (!apagemap || !apagemap->initialized)
+		return NULL;
+
+	return &apagemap->dpagemap;
+}
+
+bool
+amdgpu_pagemap_capable(struct amdgpu_svm *svm)
+{
+	if (svm->adev->gmc.is_app_apu)
+		return false;
+
+	if (!amdgpu_svm_get_dpagemap(svm))
+		return false;
+
+	return true;
+}
+
+/**
+ * amdgpu_svm_range_migrate_to_vram - Migrate range pages to VRAM
+ * @svm: Pointer to the AMDGPU SVM structure
+ * @range: The GPU SVM range to migrate
+ *
+ * Wraps drm_pagemap_populate_mm() to migrate system memory pages
+ * backing the given range into device VRAM. Skips migration if the
+ * range doesn't support devmem or is already in VRAM.
+ *
+ * Return: 0 on success or no-op, negative error code on failure
+ */
+int
+amdgpu_svm_range_migrate_to_vram(struct amdgpu_svm *svm,
+				 struct drm_gpusvm_range *range)
+{
+	struct drm_pagemap *dpagemap;
+	unsigned long start, end;
+	int ret, retries = 1;
+
+	if (!range_needs_migrate_to_vram(range))
+		return 0;
+
+	dpagemap = amdgpu_svm_get_dpagemap(svm);
+	if (!dpagemap)
+		return -ENODEV;
+
+	start = drm_gpusvm_range_start(range);
+	end = drm_gpusvm_range_end(range);
+
+	do {
+		ret = drm_pagemap_populate_mm(dpagemap, start, end,
+					      svm->gpusvm.mm, 0);
+
+		if (ret == -EBUSY && retries)
+			drm_gpusvm_range_evict(&svm->gpusvm, range);
+	} while (ret == -EBUSY && retries--);
+
+	if (ret) {
+		AMDGPU_SVM_ERR("migrate_to_vram failed: ret=%d [0x%lx-0x%lx]\n",
+				 ret, start, end);
+		return ret;
+	}
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
new file mode 100644
index 0000000000000..6e4b6e45512e7
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_RANGE_MIGRATE_H__
+#define __AMDGPU_SVM_RANGE_MIGRATE_H__
+
+struct amdgpu_svm;
+struct drm_gpusvm_range;
+
+bool amdgpu_pagemap_capable(struct amdgpu_svm *svm);
+int amdgpu_svm_range_migrate_to_vram(struct amdgpu_svm *svm,
+				     struct drm_gpusvm_range *range);
+
+#endif /* __AMDGPU_SVM_RANGE_MIGRATE_H__ */
-- 
2.53.0

[PATCH v9 18/18] drm/amdgpu: integrate VRAM migration into SVM fault and prefetch paths

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

- Call amdgpu_svm_range_migrate_to_vram() before GPU mapping in
  fault and prefetch paths when VRAM migration is needed
- Set device_private_page_owner in drm_gpusvm_ctx for devmem awareness
- Allow AMDGPU_INTERCONNECT_VRAM entries in update_gpu_range proto check
- Skip valid-mapping fast path in prefetch when VRAM migration is
  requested, so prefetch can actively move pages to VRAM
- Enable amdgpu_pagemap_capable() check in amdgpu_svm_devmem_possible()
- Move AMDGPU_INTERCONNECT_VRAM/P2P defines to migration header

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c       |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       |  3 ---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 11 ++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 23 ++++++++++++++++---
 4 files changed, 33 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
index ac101574bd61d..8b3747e444f02 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -35,6 +35,7 @@
 #include "amdgpu_svm_attr.h"
 #include "amdgpu_svm_fault.h"
 #include "amdgpu_svm_range.h"
+#include "amdgpu_svm_range_migrate.h"
 #include "amdgpu_vm.h"
 
 #if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
@@ -314,7 +315,8 @@ bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm)
 	if (svm->adev->apu_prefer_gtt)
 		return false;
 
-	/* TODO: add amdgpu_pagemap_capable() */
+	if (amdgpu_pagemap_capable(svm))
+		return true;
 
 	return false;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
index c67f01b3b5d3b..a28cce16cd9ac 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
@@ -45,9 +45,6 @@ struct amdgpu_svm_attrs;
 struct drm_device;
 struct drm_file;
 
-#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
-#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
-
 enum amdgpu_svm_xnack_mode {
 	AMDGPU_SVM_XNACK_OFF,
 	AMDGPU_SVM_XNACK_ON,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
index f28b3eb14500d..6ac7e583c9bc5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
@@ -26,6 +26,8 @@
 #include "amdgpu_svm_attr.h"
 #include "amdgpu_svm_fault.h"
 #include "amdgpu_svm_range.h"
+#include "amdgpu_svm_range_migrate.h"
+#include "amdgpu_migrate.h"
 #include "amdgpu.h"
 #include "amdgpu_vm.h"
 #include "amdgpu_gmc.h"
@@ -186,6 +188,8 @@ static int fault_map_range(struct amdgpu_svm *svm,
 		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
 		.devmem_only = need_vram_migration,
 		.timeslice_ms = need_vram_migration ? 5 : 0,
+		.device_private_page_owner = devmem_possible ?
+			AMDGPU_PGMAP_OWNER(svm->adev) : NULL,
 	};
 	struct amdgpu_svm_range *range;
 	ktime_t timestamp = ktime_get_boottime();
@@ -255,7 +259,12 @@ static int fault_map_range(struct amdgpu_svm *svm,
 	}
 
 	AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT");
-	/* TODO: add migration*/
+	if (need_vram_migration) {
+		AMDGPU_SVM_RANGE_TRACE(range, "MIGRATE PAGES");
+		ret = amdgpu_svm_range_migrate_to_vram(svm, &range->base);
+		if (ret)
+			return ret;
+	}
 
 	AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
 	ret = amdgpu_svm_range_get_pages(svm, &range->base, &map_ctx);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index ed057545d0964..8ee83904fd532 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -26,6 +26,8 @@
 #include "amdgpu_svm_attr.h"
 #include "amdgpu_svm_range.h"
 #include "amdgpu_svm_fault.h"
+#include "amdgpu_svm_range_migrate.h"
+#include "amdgpu_migrate.h"
 #include "amdgpu.h"
 #include "amdgpu_vm.h"
 #include "amdgpu_trace.h"
@@ -198,7 +200,8 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
 		unsigned long start_page, last_page;
 		bool is_last_seg;
 
-		if (entry->proto != DRM_INTERCONNECT_SYSTEM)
+		if (entry->proto != DRM_INTERCONNECT_SYSTEM &&
+		    entry->proto != AMDGPU_INTERCONNECT_VRAM)
 			return -EOPNOTSUPP;
 
 		pte_flags = amdgpu_svm_range_attr_pte_flags(svm, attrs,
@@ -358,6 +361,8 @@ amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 		.devmem_possible = devmem_possible,
 		.devmem_only = need_vram_migration,
 		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
+		.device_private_page_owner = devmem_possible ?
+			AMDGPU_PGMAP_OWNER(svm->adev) : NULL,
 	};
 
 	while (addr < end) {
@@ -380,12 +385,24 @@ amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 		if (next_addr <= addr)
 			return -EINVAL;
 
-		if (amdgpu_svm_range_is_valid(range, attrs)) {
+		/*
+		 * Prefetch to device memory should still run migration logic even if
+		 * the current GPU mapping is already valid. This allows prefetch
+		 * requests to actively move backing pages to VRAM instead of being
+		 * skipped by the valid-mapping fast path.
+		 */
+		if (amdgpu_svm_range_is_valid(range, attrs) &&
+		    !need_vram_migration) {
 			addr = next_addr;
 			continue;
 		}
 
-		/* TODO: add migration */
+		if (need_vram_migration) {
+			AMDGPU_SVM_RANGE_TRACE(range, "MIGRATE PAGES");
+			ret = amdgpu_svm_range_migrate_to_vram(svm, &range->base);
+			if (ret)
+				return ret;
+		}
 
 		AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
 
-- 
2.53.0

Re: [PATCH v9 01/18] drm/amdgpu: add SVM ioctl UAPI definitions

[Christian König]

On 8/4/26 11:42, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Define the following ioctl structures and enums:
> - DRM_AMDGPU_GEM_SVM ioctl command and DRM_IOCTL_AMDGPU_GEM_SVM macro
> - enum amdgpu_ioctl_svm_op: SET_ATTR, GET_ATTR, RESET_ATTR operations
> - enum amdgpu_ioctl_svm_access: INACCESSIBLE, IN_PLACE, ALLOW_MIGRATE
> - enum amdgpu_ioctl_svm_location: SYSMEM, UNDEFINED
> - enum amdgpu_ioctl_svm_attr_type: PREFERRED_LOC, PREFETCH_LOC, ACCESS,
>   GRANULARITY, HOST_ACCESS, COHERENT, EXT_COHERENT, HIVE_LOCAL, GPU_RO,
>   GPU_EXEC, GPU_READ_MOSTLY
> - struct drm_amdgpu_svm_attribute: type and value pair
> - struct drm_amdgpu_gem_svm: ioctl payload with start_addr, size,
>   operation, nattr, and attrs_ptr
> 
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  include/uapi/drm/amdgpu_drm.h | 106 ++++++++++++++++++++++++++++++++++
>  1 file changed, 106 insertions(+)
> 
> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
> index b32c72a662b61..4c49cd36f0e77 100644
> --- a/include/uapi/drm/amdgpu_drm.h
> +++ b/include/uapi/drm/amdgpu_drm.h
> @@ -59,6 +59,7 @@ extern "C" {
>  #define DRM_AMDGPU_USERQ_WAIT		0x18
>  #define DRM_AMDGPU_GEM_LIST_HANDLES	0x19
>  #define DRM_AMDGPU_PROC_OPTIONS		0x1A
> +#define DRM_AMDGPU_GEM_SVM		0x1B
>  
>  #define DRM_IOCTL_AMDGPU_GEM_CREATE	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
>  #define DRM_IOCTL_AMDGPU_GEM_MMAP	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)
> @@ -81,6 +82,7 @@ extern "C" {
>  #define DRM_IOCTL_AMDGPU_USERQ_WAIT	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_USERQ_WAIT, struct drm_amdgpu_userq_wait)
>  #define DRM_IOCTL_AMDGPU_GEM_LIST_HANDLES DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_LIST_HANDLES, struct drm_amdgpu_gem_list_handles)
>  #define DRM_IOCTL_AMDGPU_PROC_OPTIONS	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_PROC_OPTIONS, struct drm_amdgpu_proc_options)
> +#define DRM_IOCTL_AMDGPU_GEM_SVM	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_SVM, struct drm_amdgpu_gem_svm)
>  
>  /**
>   * DOC: memory domains
> @@ -1694,6 +1696,110 @@ struct drm_amdgpu_proc_options {
>  	} kfd_sigbus_delay;
>  };
>  
> +/**
> + * enum amdgpu_ioctl_svm_op - operation selector for DRM_IOCTL_AMDGPU_GEM_SVM.
> + * @AMDGPU_SVM_OP_SET_ATTR: apply the attributes in @attrs_ptr to the VA range.
> + * @AMDGPU_SVM_OP_GET_ATTR: read back the current value of each attribute
> + *                          listed in @attrs_ptr for the given VA range.

That this works on the given VA range is not correct.

To read out the attributes userspace needs to give the start addr and attributes and get the size the attributes are the same in return.

Only this way CRIU is able to read out the attributes from kernel to userspace.

> + * @AMDGPU_SVM_OP_RESET_ATTR: reset all attributes for the VA range to their
> + *                            default values. @attrs_ptr and @nattr are ignored.
> + */
> +enum amdgpu_ioctl_svm_op {
> +	AMDGPU_SVM_OP_SET_ATTR = 0,
> +	AMDGPU_SVM_OP_GET_ATTR = 1,
> +	AMDGPU_SVM_OP_RESET_ATTR = 2,
> +};
> +
> +/**
> + * enum amdgpu_ioctl_svm_access - values for AMDGPU_SVM_ATTR_ACCESS.
> + * @AMDGPU_SVM_ACCESS_INACCESSIBLE: GPU must not access the range; any access
> + *                                  is a fault.
> + * @AMDGPU_SVM_ACCESS_IN_PLACE:     GPU may access the range
This needs more.  only at its
> + *                                  current backing store; the driver will
> + *                                  never migrate pages to local VRAM.
> + * @AMDGPU_SVM_ACCESS_ALLOW_MIGRATE: GPU may access the range and the driver
> + *                                  is allowed (but not required) to migrate
> + *                                  pages between system memory and local
> + *                                  VRAM to satisfy the preferred/prefetch
> + *                                  location.
> + */
> +enum amdgpu_ioctl_svm_access {
> +	AMDGPU_SVM_ACCESS_INACCESSIBLE		= 0,
> +	AMDGPU_SVM_ACCESS_IN_PLACE			= 1,
> +	AMDGPU_SVM_ACCESS_ALLOW_MIGRATE		= 2,
> +};
> +
> +/**
> + * enum amdgpu_svm_location - values for AMDGPU_SVM_ATTR_PREFERRED_LOC /
> + *                            AMDGPU_SVM_ATTR_PREFETCH_LOC.
> + * @AMDGPU_SVM_LOCATION_SYSMEM:    back the range with system memory.
> + * @AMDGPU_SVM_LOCATION_UNDEFINED: no preference; the driver chooses.
> + */
> +enum amdgpu_ioctl_svm_location {
> +	AMDGPU_SVM_LOCATION_SYSMEM	= 0,
> +	AMDGPU_SVM_LOCATION_UNDEFINED	= 0xffffffffU,

Please make the AMDGPU_SVM_LOCATION_UNDEFINED 1 here and add a value AMDGPU_SVM_LOCATION_LOCAL and AMDGPU_SVM_LOCATION_HIVE.

The XE idea of giving the file descriptor of the device driver to prefetch to is a clear NAK from my side now. This whole concept breaks as soon as you add CRIU to the picture.

> +};
> +
> +/**
> + * enum amdgpu_ioctl_svm_attr_type - attribute selector for
> + *                                   &drm_amdgpu_svm_attribute.type.
> + *
> + * @AMDGPU_SVM_ATTR_PREFERRED_LOC: Preferred backing location for the range.
> + *	Value is one of &enum amdgpu_ioctl_svm_location.
> + * @AMDGPU_SVM_ATTR_PREFETCH_LOC: Prefetch target for the range. Value is
> + *	one of &enum amdgpu_ioctl_svm_location.
> + * @AMDGPU_SVM_ATTR_ACCESS: GPU access policy for the range. Value is one
> + *	of &enum amdgpu_ioctl_svm_access.
> + * @AMDGPU_SVM_ATTR_GRANULARITY: log2 of the migration granularity in pages.
> + * @AMDGPU_SVM_ATTR_HOST_ACCESS: Guarantee host access to memory.
> + * @AMDGPU_SVM_ATTR_COHERENT: Fine-grained coherency between all devices
> + *	with access.
> + * @AMDGPU_SVM_ATTR_EXT_COHERENT: Fine-grained coherency between all devices
> + *	using device-scope atomics.

> + * @AMDGPU_SVM_ATTR_HIVE_LOCAL: Use any GPU in the same XGMI hive as the
> + *	preferred device.

Please drop that one, it should be covered by AMDGPU_SVM_LOCATION_* above.

Regards,
Christian.

> + * @AMDGPU_SVM_ATTR_GPU_RO: GPUs only read the range, allowing replication.
> + * @AMDGPU_SVM_ATTR_GPU_EXEC: Allow execution on GPU.
> + * @AMDGPU_SVM_ATTR_GPU_READ_MOSTLY: GPUs mostly read the range; may allow
> + *	optimizations similar to GPU_RO, but writes still fault.
> + */
> +enum amdgpu_ioctl_svm_attr_type {
> +	AMDGPU_SVM_ATTR_PREFERRED_LOC		= 0,
> +	AMDGPU_SVM_ATTR_PREFETCH_LOC		= 1,
> +	AMDGPU_SVM_ATTR_ACCESS				= 2,
> +	AMDGPU_SVM_ATTR_GRANULARITY			= 3,
> +	/* Boolean attributes below: value must be 0 or 1. */
> +	AMDGPU_SVM_ATTR_HOST_ACCESS			= 4,
> +	AMDGPU_SVM_ATTR_COHERENT			= 5,
> +	AMDGPU_SVM_ATTR_EXT_COHERENT		= 6,
> +	AMDGPU_SVM_ATTR_HIVE_LOCAL			= 7,
> +	AMDGPU_SVM_ATTR_GPU_RO				= 8,
> +	AMDGPU_SVM_ATTR_GPU_EXEC			= 9,
> +	AMDGPU_SVM_ATTR_GPU_READ_MOSTLY		= 10,
> +};
> +
> +/* One (type, value) pair carried by DRM_IOCTL_AMDGPU_GEM_SVM. */
> +struct drm_amdgpu_svm_attribute {
> +	/** AMDGPU_SVM_ATTR_* */
> +	__u32 type;
> +	/** Attribute value; interpretation depends on @type */
> +	__u32 value;
> +};
> +
> +/* Argument for DRM_IOCTL_AMDGPU_GEM_SVM. */
> +struct drm_amdgpu_gem_svm {
> +	/** Start of the virtual address range */
> +	__u64 start_addr;
> +	/** Size of the range in bytes */
> +	__u64 size;
> +	/** AMDGPU_SVM_OP_* */
> +	__u32 operation;
> +	/** Number of struct drm_amdgpu_svm_attribute entries in @attrs_ptr */
> +	__u32 nattr;
> +	/** User pointer to an array of @nattr struct drm_amdgpu_svm_attribute */
> +	__u64 attrs_ptr;
> +};
> +
>  #if defined(__cplusplus)
>  }
>  #endif

Re: [PATCH v9 02/18] drm/amdgpu: add SVM core header and VM integration

[Christian König]

On 8/4/26 11:42, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Add amdgpu_svm.h with SVM data structures and public API:
> - enum amdgpu_svm_xnack_mode: OFF, ON and DEFAULT mode
> - struct amdgpu_svm: core context with drm_gpusvm, kref lifecycle,
>   attribute tree, rw_semaphore, GC workqueue, xnack state.
> - struct amdgpu_svm_gc: garbage collector with workqueue and work_struct

> - Debug/trace macros: AMDGPU_SVM_TRACE, AMDGPU_SVM_WARN, AMDGPU_SVM_ERR

Please drop those, use pr_debug/warn/err directly in the code.

> - Kmem cache helpers: AMDGPU_SVM_KMEM_CACHE_CREATE/DESTROY

Those where already dropped.

> - Locking helpers: amdgpu_svm_lock/unlock/assert_locked
> - Public API declarations (including clean_queue, sync_work)
> 
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 204 ++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |   4 +
>  2 files changed, 208 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> new file mode 100644
> index 0000000000000..f3b4f228405c2
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> @@ -0,0 +1,204 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright 2026 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + */
> +
> +#ifndef __AMDGPU_SVM_H__
> +#define __AMDGPU_SVM_H__
> +
> +#include <drm/amdgpu_drm.h>
> +#include <drm/drm_gpusvm.h>
> +#include <linux/atomic.h>
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/printk.h>
> +#include <linux/rwsem.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +struct amdgpu_device;
> +struct amdgpu_vm;
> +struct amdgpu_svm_attr_tree;
> +struct amdgpu_svm_attrs;
> +struct drm_device;
> +struct drm_file;
> +
> +enum amdgpu_svm_xnack_mode {
> +	AMDGPU_SVM_XNACK_OFF,
> +	AMDGPU_SVM_XNACK_ON,
> +	AMDGPU_SVM_XNACK_DEFAULT,
> +};
> +
> +#define AMDGPU_SVM_DBG(fmt, ...) \
> +	pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
> +
> +#define AMDGPU_SVM_WARN(fmt, ...) \
> +	pr_warn("%s: " fmt, __func__, ##__VA_ARGS__)
> +
> +#define AMDGPU_SVM_ERR(fmt, ...) \
> +	pr_err("%s: " fmt, __func__, ##__VA_ARGS__)
> +
> +#define amdgpu_svm_assert_in_notifier(svm__) \
> +	lockdep_assert_held_write(&(svm__)->gpusvm.notifier_lock)
> +
> +/**
> + * struct amdgpu_svm_gc - SVM range garbage collector used for unmapping
> + * SVM ranges in response to mmu_notifier events.
> + * @wq: Workqueue where @work runs on.
> + * @list: Ranges list in garbage collector. Protected by amdgpu_svm.work_lock.
> + * @work: Drains @list under amdgpu_svm.svm_lock.
> + */
> +struct amdgpu_svm_gc {
> +	struct workqueue_struct *wq;
> +	struct list_head list;
> +	struct work_struct work;
> +};
> +
> +/**
> + * struct amdgpu_svm - shared virtual memory context
> + * @gpusvm: Embedded drm_gpusvm base, manages the range/notifier trees.
> + * @refcount: Reference count guarding the lifetime of this instance.
> + * @adev: The amdgpu device this SVM context belongs to.
> + * @vm: Back pointer to the owning amdgpu_vm.
> + * @attr_tree: Interval tree of SVM attributes.
> + * @svm_lock: The driver_svm_lock described in the GPU SVM locking section.
> + *            registered by drm_gpusvm_driver_set_lock().
> + * @work_lock: Protects the deferred work queues such as @gc.list.
> + * @gc: garbage collector for ranges queued for destruction.
> + * @exiting: Set once when the context starts tearing down.
> + * @checkpoint_ts: IH ring timestamp captured on unmap, used to drop stale
> + *                 faults.
> + * @default_granularity: Default range granularity.
> + * @xnack_enabled: Whether retry (XNACK) faults are enabled.
> + * @invalidate_ranges: Callback of the MMU notifier invalidate event.
> + */
> +struct amdgpu_svm {
> +	struct drm_gpusvm gpusvm;
> +	struct kref refcount;
> +	struct amdgpu_device *adev;
> +	struct amdgpu_vm *vm;
> +	struct amdgpu_svm_attr_tree *attr_tree;
> +	struct rw_semaphore svm_lock;
> +	spinlock_t work_lock;
> +	struct amdgpu_svm_gc gc;
> +	atomic_t exiting;
> +	uint64_t checkpoint_ts;
> +	u8 default_granularity;
> +	bool xnack_enabled;
> +	void (*invalidate_ranges)(struct amdgpu_svm *svm,
> +				  struct drm_gpusvm_notifier *notifier,
> +				  const struct mmu_notifier_range *mmu_range,
> +				  struct drm_gpusvm_range *first,
> +				  uint64_t adj_start, uint64_t adj_end);
> +};
> +
> +static inline struct amdgpu_svm *to_amdgpu_svm(struct drm_gpusvm *gpusvm)
> +{
> +	return container_of(gpusvm, struct amdgpu_svm, gpusvm);
> +}
> +


> +/*
> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
> + * Hold it in write mode around structural GPU SVM updates, including
> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
> + */
> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
> +{
> +	down_write(&svm->svm_lock);
> +}
> +
> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
> +{
> +	up_write(&svm->svm_lock);
> +}
> +
> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
> +{
> +	lockdep_assert_held_write(&svm->svm_lock);
> +}

I'm starting to repeat myself, so once more: This stuff doesn't work like that!

The lock the SVM subsystem uses to serialize updates *must* be the amdgpu_vm->eviction_lock and *not* a separate one.

So clear NAK to having this functions here.

Regards,
Christian.

> +
> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
> +
> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
> +void amdgpu_svm_close(struct amdgpu_vm *vm);
> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
> +
> +void amdgpu_svm_put(struct amdgpu_svm *svm);
> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
> +					       uint32_t pasid);
> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
> +			    uint64_t fault_page, uint64_t ts,
> +			    bool write_fault);
> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
> +
> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
> +			 struct drm_file *filp);
> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
> +			    struct list_head *work_list);
> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
> +				 const struct amdgpu_svm_attrs *old_attrs,
> +				 const struct amdgpu_svm_attrs *new_attrs,
> +				 unsigned long start_page,
> +				 unsigned long last_page);
> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
> +#else
> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
> +				  struct amdgpu_vm *vm)
> +{
> +	return 0;
> +}
> +
> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
> +{
> +}
> +
> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
> +{
> +}
> +
> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
> +					  uint32_t pasid,
> +					  uint64_t fault_page,
> +					  uint64_t ts,
> +					  bool write_fault)
> +{
> +	return -EOPNOTSUPP;
> +}
> +
> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
> +{
> +	return false;
> +}
> +
> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
> +				       struct drm_file *filp)
> +{
> +	return -EOPNOTSUPP;
> +}
> +#endif /* CONFIG_DRM_AMDGPU_SVM */
> +
> +#endif /* __AMDGPU_SVM_H__ */
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index ec1196d390bb7..30463a83e2e60 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>  struct amdgpu_job;
>  struct amdgpu_bo_list_entry;
>  struct amdgpu_bo_vm;
> +struct amdgpu_svm;
>  
>  /*
>   * GPUVM handling
> @@ -373,6 +374,9 @@ struct amdgpu_vm {
>  
>  	/* cached fault info */
>  	struct amdgpu_vm_fault_info fault_info;
> +
> +	/* SVM experimental implementation */
> +	struct amdgpu_svm *svm;
>  };
>  
>  struct amdgpu_vm_manager {

Re: [PATCH v9 01/18] drm/amdgpu: add SVM ioctl UAPI definitions

[Huang, Honglei]

On 8/11/2026 6:58 PM, Christian König wrote:
> On 8/4/26 11:42, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Define the following ioctl structures and enums:
>> - DRM_AMDGPU_GEM_SVM ioctl command and DRM_IOCTL_AMDGPU_GEM_SVM macro
>> - enum amdgpu_ioctl_svm_op: SET_ATTR, GET_ATTR, RESET_ATTR operations
>> - enum amdgpu_ioctl_svm_access: INACCESSIBLE, IN_PLACE, ALLOW_MIGRATE
>> - enum amdgpu_ioctl_svm_location: SYSMEM, UNDEFINED
>> - enum amdgpu_ioctl_svm_attr_type: PREFERRED_LOC, PREFETCH_LOC, ACCESS,
>>    GRANULARITY, HOST_ACCESS, COHERENT, EXT_COHERENT, HIVE_LOCAL, GPU_RO,
>>    GPU_EXEC, GPU_READ_MOSTLY
>> - struct drm_amdgpu_svm_attribute: type and value pair
>> - struct drm_amdgpu_gem_svm: ioctl payload with start_addr, size,
>>    operation, nattr, and attrs_ptr
>>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   include/uapi/drm/amdgpu_drm.h | 106 ++++++++++++++++++++++++++++++++++
>>   1 file changed, 106 insertions(+)
>>
>> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
>> index b32c72a662b61..4c49cd36f0e77 100644
>> --- a/include/uapi/drm/amdgpu_drm.h
>> +++ b/include/uapi/drm/amdgpu_drm.h
>> @@ -59,6 +59,7 @@ extern "C" {
>>   #define DRM_AMDGPU_USERQ_WAIT		0x18
>>   #define DRM_AMDGPU_GEM_LIST_HANDLES	0x19
>>   #define DRM_AMDGPU_PROC_OPTIONS		0x1A
>> +#define DRM_AMDGPU_GEM_SVM		0x1B
>>   
>>   #define DRM_IOCTL_AMDGPU_GEM_CREATE	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
>>   #define DRM_IOCTL_AMDGPU_GEM_MMAP	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)
>> @@ -81,6 +82,7 @@ extern "C" {
>>   #define DRM_IOCTL_AMDGPU_USERQ_WAIT	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_USERQ_WAIT, struct drm_amdgpu_userq_wait)
>>   #define DRM_IOCTL_AMDGPU_GEM_LIST_HANDLES DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_LIST_HANDLES, struct drm_amdgpu_gem_list_handles)
>>   #define DRM_IOCTL_AMDGPU_PROC_OPTIONS	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_PROC_OPTIONS, struct drm_amdgpu_proc_options)
>> +#define DRM_IOCTL_AMDGPU_GEM_SVM	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_SVM, struct drm_amdgpu_gem_svm)
>>   
>>   /**
>>    * DOC: memory domains
>> @@ -1694,6 +1696,110 @@ struct drm_amdgpu_proc_options {
>>   	} kfd_sigbus_delay;
>>   };
>>   
>> +/**
>> + * enum amdgpu_ioctl_svm_op - operation selector for DRM_IOCTL_AMDGPU_GEM_SVM.
>> + * @AMDGPU_SVM_OP_SET_ATTR: apply the attributes in @attrs_ptr to the VA range.
>> + * @AMDGPU_SVM_OP_GET_ATTR: read back the current value of each attribute
>> + *                          listed in @attrs_ptr for the given VA range.
> 
> 32
> 
> To read out the attributes userspace needs to give the start addr and attributes and get the size the attributes are the same in return.
> 
> Only this way CRIU is able to read out the attributes from kernel to userspace.

Will modify the get attr accordgin to your method.


> 
>> + * @AMDGPU_SVM_OP_RESET_ATTR: reset all attributes for the VA range to their
>> + *                            default values. @attrs_ptr and @nattr are ignored.
>> + */
>> +enum amdgpu_ioctl_svm_op {
>> +	AMDGPU_SVM_OP_SET_ATTR = 0,
>> +	AMDGPU_SVM_OP_GET_ATTR = 1,
>> +	AMDGPU_SVM_OP_RESET_ATTR = 2,
>> +};
>> +
>> +/**
>> + * enum amdgpu_ioctl_svm_access - values for AMDGPU_SVM_ATTR_ACCESS.
>> + * @AMDGPU_SVM_ACCESS_INACCESSIBLE: GPU must not access the range; any access
>> + *                                  is a fault.
>> + * @AMDGPU_SVM_ACCESS_IN_PLACE:     GPU may access the range
> This needs more.  only at its
>> + *                                  current backing store; the driver will
>> + *                                  never migrate pages to local VRAM.
>> + * @AMDGPU_SVM_ACCESS_ALLOW_MIGRATE: GPU may access the range and the driver
>> + *                                  is allowed (but not required) to migrate
>> + *                                  pages between system memory and local
>> + *                                  VRAM to satisfy the preferred/prefetch
>> + *                                  location.
>> + */
>> +enum amdgpu_ioctl_svm_access {
>> +	AMDGPU_SVM_ACCESS_INACCESSIBLE		= 0,
>> +	AMDGPU_SVM_ACCESS_IN_PLACE			= 1,
>> +	AMDGPU_SVM_ACCESS_ALLOW_MIGRATE		= 2,
>> +};
>> +
>> +/**
>> + * enum amdgpu_svm_location - values for AMDGPU_SVM_ATTR_PREFERRED_LOC /
>> + *                            AMDGPU_SVM_ATTR_PREFETCH_LOC.
>> + * @AMDGPU_SVM_LOCATION_SYSMEM:    back the range with system memory.
>> + * @AMDGPU_SVM_LOCATION_UNDEFINED: no preference; the driver chooses.
>> + */
>> +enum amdgpu_ioctl_svm_location {
>> +	AMDGPU_SVM_LOCATION_SYSMEM	= 0,
>> +	AMDGPU_SVM_LOCATION_UNDEFINED	= 0xffffffffU,
> 
> Please make the AMDGPU_SVM_LOCATION_UNDEFINED 1 here and add a value AMDGPU_SVM_LOCATION_LOCAL and AMDGPU_SVM_LOCATION_HIVE.
> 
> The XE idea of giving the file descriptor of the device driver to prefetch to is a clear NAK from my side now. This whole concept breaks as soon as you add CRIU to the picture.

Will change the flags according to your comments.


> 
>> +};
>> +
>> +/**
>> + * enum amdgpu_ioctl_svm_attr_type - attribute selector for
>> + *                                   &drm_amdgpu_svm_attribute.type.
>> + *
>> + * @AMDGPU_SVM_ATTR_PREFERRED_LOC: Preferred backing location for the range.
>> + *	Value is one of &enum amdgpu_ioctl_svm_location.
>> + * @AMDGPU_SVM_ATTR_PREFETCH_LOC: Prefetch target for the range. Value is
>> + *	one of &enum amdgpu_ioctl_svm_location.
>> + * @AMDGPU_SVM_ATTR_ACCESS: GPU access policy for the range. Value is one
>> + *	of &enum amdgpu_ioctl_svm_access.
>> + * @AMDGPU_SVM_ATTR_GRANULARITY: log2 of the migration granularity in pages.
>> + * @AMDGPU_SVM_ATTR_HOST_ACCESS: Guarantee host access to memory.
>> + * @AMDGPU_SVM_ATTR_COHERENT: Fine-grained coherency between all devices
>> + *	with access.
>> + * @AMDGPU_SVM_ATTR_EXT_COHERENT: Fine-grained coherency between all devices
>> + *	using device-scope atomics.
> 
>> + * @AMDGPU_SVM_ATTR_HIVE_LOCAL: Use any GPU in the same XGMI hive as the
>> + *	preferred device.
> 
> Please drop that one, it should be covered by AMDGPU_SVM_LOCATION_* above.


will drop it.

> 
> Regards,
> Christian.
> 
>> + * @AMDGPU_SVM_ATTR_GPU_RO: GPUs only read the range, allowing replication.
>> + * @AMDGPU_SVM_ATTR_GPU_EXEC: Allow execution on GPU.
>> + * @AMDGPU_SVM_ATTR_GPU_READ_MOSTLY: GPUs mostly read the range; may allow
>> + *	optimizations similar to GPU_RO, but writes still fault.
>> + */
>> +enum amdgpu_ioctl_svm_attr_type {
>> +	AMDGPU_SVM_ATTR_PREFERRED_LOC		= 0,
>> +	AMDGPU_SVM_ATTR_PREFETCH_LOC		= 1,
>> +	AMDGPU_SVM_ATTR_ACCESS				= 2,
>> +	AMDGPU_SVM_ATTR_GRANULARITY			= 3,
>> +	/* Boolean attributes below: value must be 0 or 1. */
>> +	AMDGPU_SVM_ATTR_HOST_ACCESS			= 4,
>> +	AMDGPU_SVM_ATTR_COHERENT			= 5,
>> +	AMDGPU_SVM_ATTR_EXT_COHERENT		= 6,
>> +	AMDGPU_SVM_ATTR_HIVE_LOCAL			= 7,
>> +	AMDGPU_SVM_ATTR_GPU_RO				= 8,
>> +	AMDGPU_SVM_ATTR_GPU_EXEC			= 9,
>> +	AMDGPU_SVM_ATTR_GPU_READ_MOSTLY		= 10,
>> +};
>> +
>> +/* One (type, value) pair carried by DRM_IOCTL_AMDGPU_GEM_SVM. */
>> +struct drm_amdgpu_svm_attribute {
>> +	/** AMDGPU_SVM_ATTR_* */
>> +	__u32 type;
>> +	/** Attribute value; interpretation depends on @type */
>> +	__u32 value;
>> +};
>> +
>> +/* Argument for DRM_IOCTL_AMDGPU_GEM_SVM. */
>> +struct drm_amdgpu_gem_svm {
>> +	/** Start of the virtual address range */
>> +	__u64 start_addr;
>> +	/** Size of the range in bytes */
>> +	__u64 size;
>> +	/** AMDGPU_SVM_OP_* */
>> +	__u32 operation;
>> +	/** Number of struct drm_amdgpu_svm_attribute entries in @attrs_ptr */
>> +	__u32 nattr;
>> +	/** User pointer to an array of @nattr struct drm_amdgpu_svm_attribute */
>> +	__u64 attrs_ptr;
>> +};
>> +
>>   #if defined(__cplusplus)
>>   }
>>   #endif
> 

Re: [PATCH v9 02/18] drm/amdgpu: add SVM core header and VM integration

[Huang, Honglei]

On 8/11/2026 7:02 PM, Christian König wrote:
> 
> 
> On 8/4/26 11:42, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Add amdgpu_svm.h with SVM data structures and public API:
>> - enum amdgpu_svm_xnack_mode: OFF, ON and DEFAULT mode
>> - struct amdgpu_svm: core context with drm_gpusvm, kref lifecycle,
>>    attribute tree, rw_semaphore, GC workqueue, xnack state.
>> - struct amdgpu_svm_gc: garbage collector with workqueue and work_struct
> 
>> - Debug/trace macros: AMDGPU_SVM_TRACE, AMDGPU_SVM_WARN, AMDGPU_SVM_ERR
> 
> Please drop those, use pr_debug/warn/err directly in the code.

will drop it.

> 
>> - Kmem cache helpers: AMDGPU_SVM_KMEM_CACHE_CREATE/DESTROY

will remove this commit message.

> 
> Those where already dropped.
> 
>> - Locking helpers: amdgpu_svm_lock/unlock/assert_locked
>> - Public API declarations (including clean_queue, sync_work)
>>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 204 ++++++++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |   4 +
>>   2 files changed, 208 insertions(+)
>>   create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> new file mode 100644
>> index 0000000000000..f3b4f228405c2
>> --- /dev/null
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> @@ -0,0 +1,204 @@
>> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
>> +/*
>> + * Copyright 2026 Advanced Micro Devices, Inc.
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice shall be included in
>> + * all copies or substantial portions of the Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
>> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
>> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>> + * OTHER DEALINGS IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#ifndef __AMDGPU_SVM_H__
>> +#define __AMDGPU_SVM_H__
>> +
>> +#include <drm/amdgpu_drm.h>
>> +#include <drm/drm_gpusvm.h>
>> +#include <linux/atomic.h>
>> +#include <linux/kref.h>
>> +#include <linux/list.h>
>> +#include <linux/printk.h>
>> +#include <linux/rwsem.h>
>> +#include <linux/slab.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/types.h>
>> +#include <linux/workqueue.h>
>> +
>> +struct amdgpu_device;
>> +struct amdgpu_vm;
>> +struct amdgpu_svm_attr_tree;
>> +struct amdgpu_svm_attrs;
>> +struct drm_device;
>> +struct drm_file;
>> +
>> +enum amdgpu_svm_xnack_mode {
>> +	AMDGPU_SVM_XNACK_OFF,
>> +	AMDGPU_SVM_XNACK_ON,
>> +	AMDGPU_SVM_XNACK_DEFAULT,
>> +};
>> +
>> +#define AMDGPU_SVM_DBG(fmt, ...) \
>> +	pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
>> +
>> +#define AMDGPU_SVM_WARN(fmt, ...) \
>> +	pr_warn("%s: " fmt, __func__, ##__VA_ARGS__)
>> +
>> +#define AMDGPU_SVM_ERR(fmt, ...) \
>> +	pr_err("%s: " fmt, __func__, ##__VA_ARGS__)
>> +
>> +#define amdgpu_svm_assert_in_notifier(svm__) \
>> +	lockdep_assert_held_write(&(svm__)->gpusvm.notifier_lock)
>> +
>> +/**
>> + * struct amdgpu_svm_gc - SVM range garbage collector used for unmapping
>> + * SVM ranges in response to mmu_notifier events.
>> + * @wq: Workqueue where @work runs on.
>> + * @list: Ranges list in garbage collector. Protected by amdgpu_svm.work_lock.
>> + * @work: Drains @list under amdgpu_svm.svm_lock.
>> + */
>> +struct amdgpu_svm_gc {
>> +	struct workqueue_struct *wq;
>> +	struct list_head list;
>> +	struct work_struct work;
>> +};
>> +
>> +/**
>> + * struct amdgpu_svm - shared virtual memory context
>> + * @gpusvm: Embedded drm_gpusvm base, manages the range/notifier trees.
>> + * @refcount: Reference count guarding the lifetime of this instance.
>> + * @adev: The amdgpu device this SVM context belongs to.
>> + * @vm: Back pointer to the owning amdgpu_vm.
>> + * @attr_tree: Interval tree of SVM attributes.
>> + * @svm_lock: The driver_svm_lock described in the GPU SVM locking section.
>> + *            registered by drm_gpusvm_driver_set_lock().
>> + * @work_lock: Protects the deferred work queues such as @gc.list.
>> + * @gc: garbage collector for ranges queued for destruction.
>> + * @exiting: Set once when the context starts tearing down.
>> + * @checkpoint_ts: IH ring timestamp captured on unmap, used to drop stale
>> + *                 faults.
>> + * @default_granularity: Default range granularity.
>> + * @xnack_enabled: Whether retry (XNACK) faults are enabled.
>> + * @invalidate_ranges: Callback of the MMU notifier invalidate event.
>> + */
>> +struct amdgpu_svm {
>> +	struct drm_gpusvm gpusvm;
>> +	struct kref refcount;
>> +	struct amdgpu_device *adev;
>> +	struct amdgpu_vm *vm;
>> +	struct amdgpu_svm_attr_tree *attr_tree;
>> +	struct rw_semaphore svm_lock;
>> +	spinlock_t work_lock;
>> +	struct amdgpu_svm_gc gc;
>> +	atomic_t exiting;
>> +	uint64_t checkpoint_ts;
>> +	u8 default_granularity;
>> +	bool xnack_enabled;
>> +	void (*invalidate_ranges)(struct amdgpu_svm *svm,
>> +				  struct drm_gpusvm_notifier *notifier,
>> +				  const struct mmu_notifier_range *mmu_range,
>> +				  struct drm_gpusvm_range *first,
>> +				  uint64_t adj_start, uint64_t adj_end);
>> +};
>> +
>> +static inline struct amdgpu_svm *to_amdgpu_svm(struct drm_gpusvm *gpusvm)
>> +{
>> +	return container_of(gpusvm, struct amdgpu_svm, gpusvm);
>> +}
>> +
> 
> 
>> +/*
>> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
>> + * Hold it in write mode around structural GPU SVM updates, including
>> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
>> + */
>> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
>> +{
>> +	down_write(&svm->svm_lock);
>> +}
>> +
>> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
>> +{
>> +	up_write(&svm->svm_lock);
>> +}
>> +
>> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
>> +{
>> +	lockdep_assert_held_write(&svm->svm_lock);
>> +}
> 
> I'm starting to repeat myself, so once more: This stuff doesn't work like that!
> 
> The lock the SVM subsystem uses to serialize updates *must* be the amdgpu_vm->eviction_lock and *not* a separate one.
> 
> So clear NAK to having this functions here.


I have explained why eviction lock can not be used as svm lock in V8, 
previous version. And it seems like we have a big gap about it.

The eviction lock can not be used for svm lcok.

And the design of svm lock is the core locking design of drmsvm frame 
work, without this desgin, this framework lost its soul. So I have to 
explain how to use this lock.

I believe we are talking about two different locks. You may treat svm 
lock as notifier lock. drm_gpusvm has two of them, and the "no 
allocation while held in the MMU notifier" rule applies
to the other one, not to driver_svm_lock.

1) drm_gpusvm has two distinct locks: drivers/gpu/drm/drm_gpusvm.c

   - notifier_lock: safeguards the notifier's range RB tree and list, as
     well as the range's DMA mappings and sequence number. ... This lock
     corresponds to the driver->update lock mentioned in
     Documentation/mm/hmm.rst."

   - driver_svm_lock: In addition to the locking mentioned above, the
     driver should implement a lock to safeguard core GPU SVM function
     calls that modify state, such as drm_gpusvm_range_find_or_insert and
     drm_gpusvm_range_remove.

   Two locks, two jobs.

2) The lock held in the MMU notifier is notifier_lock, never driver_svm_lock

   drm_gpusvm_notifier_invalidate():
         down_write(&gpusvm->notifier_lock);
         ...
         gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);

   The driver invalidate callback runs under notifier_lock only. Per the
   framework's own notifier example it just unmaps pages
   and queues the range to the garbage collector no allocation, and it
   does not take driver_svm_lock:

         drm_gpusvm_range_unmap_pages(...);
         drm_gpusvm_range_set_unmapped(...);
         driver_garbage_collector_add(...);

3) driver_svm_lock is by design an allocating, process context lock

   drm_gpusvm_range_find_or_insert() asserts it and then allocates under it:

         drm_gpusvm_range_find_or_insert():
                 drm_gpusvm_driver_lock_held(gpusvm);
                 ...
                 range = drm_gpusvm_range_alloc(...);
                 ... mmu_interval_notifier_insert(), kzalloc

   drm_gpusvm_range_remove() asserts it and frees. This is only safe
   because driver_svm_lock is a sleepable, reclaim friendly lock that is
   never taken from the MMU notifier. Reference counting
    handles range *lifetime*, but it does not
   serialize tree insert/remove, which is exactly why the framework still
   asserts driver_svm_lock on those two entry points regardless of refcount.

Now the three concrete points:

A) Why the primary driver_svm_lock is required

   It is a framework requirement, not an amdgpu invention:
    - DOC: Locking says the driver "should implement" it.
    - drm_gpusvm lockdep-asserts it on every structural entry:
      drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove() both
      call drm_gpusvm_driver_lock_held().
    - The reference fault handler holds it across the whole fault:
      GC -> find_or_insert -> migrate -> get_pages -> bind.

   Xe does exactly this:
    - xe_svm.c:      drm_gpusvm_driver_set_lock(&vm->svm.gpusvm, &vm->lock);
    - xe_pagefault.c: down_write(&vm->lock); before dispatching the fault
    - __xe_svm_handle_pagefault(): lockdep_assert_held_write(&vm->lock);
      held across GC / find_or_insert / alloc_vram / get_pages / rebind
    - xe_svm_garbage_collector(): lockdep_assert_held_write(&vm->lock);

   amdgpu's svm_lock is the same driver_svm_lock, used the same way.

B) Why eviction_lock cannot be that lock

 > This lock eviction_lock can only be grabbed while updating the 
mapping range.

   and that is precisely why it cannot be driver_svm_lock.
   driver_svm_lock must wrap find_or_insert, migration, and
   drm_gpusvm_range_get_pages
   eviction_lock is the opposite by contract:

    - It is taken with memalloc_noreclaim_save() in
      amdgpu_vm_begin_critical(), specifically so no reclaim happens while
      held (to avoid the reclaim -> MMU-notifier deadlock). Holding it
      across get_pages/migration breaks that.
    - TTM eviction try-locks it: amdgpu_vm_evictable() does
      scoped_cond_guard(mutex_try, return false, &vm->eviction_lock) and
      sets vm->evicting. Long holds starve eviction.
    - It is a plain mutex that the SVM map path re-enters:
      amdgpu_svm_range_update_mapping() -> amdgpu_vm_map_range() ->
      amdgpu_vm_begin_critical() -> mutex_lock(&vm->eviction_lock). If
      eviction_lock were also the outer SVM lock, this is a self-deadlock.

   In short, eviction_lock has the contract of notifier_lock , not of
   driver_svm_lock. This is also why the current split is correct:
   svm_lock (outer) != eviction_lock (inner). Your own rule - "you can't
   call the VM code with the lock held, the VM code must take it itself" -
   is satisfied today only because they are separate: svm_lock is held
   while calling amdgpu_vm_map_range(), and amdgpu_vm_map_range() takes
   eviction_lock itself. Merging them is what would violate that rule.

 > No, they Xe vm->lock and eviction_lock are actually identical in the 
handling.

   They are not. Xe's vm->lock is a rw_semaphore, the "outer most lock" of
   the VM , held down_write across the whole fault. amdgpu's
   eviction_lock is a mutex taken only inside amdgpu_vm_begin_critical()
   during a PT update, under memalloc_noreclaim. Xe's eviction/reclaim
   handling is separate from vm->lock. The amdgpu analogue of Xe's vm->lock
   is svm_lock, not eviction_lock.

C) Reusing an existing amdgpu_vm lock as the primary lock needs refactor 
amdgpu VM

   Xe can register vm->lock because Xe's VM was designed with an outer
   rw_semaphore held across faults. amdgpu_vm has no such lock: only
   eviction_lock , the root PD dma_resv , and a few spinlocks.

   So do it like Xe means introducing a dedicated, outer, sleepable VM
   lock held across the fault. That lock is exactly svm_lock. Folding it
   into struct amdgpu_vm as a general vm->lock is a core amdgpu VM 
refactor.

Regards,
Honglei


> 
> Regards,
> Christian.
> 
>> +
>> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
>> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
>> +
>> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>> +void amdgpu_svm_close(struct amdgpu_vm *vm);
>> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
>> +
>> +void amdgpu_svm_put(struct amdgpu_svm *svm);
>> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
>> +					       uint32_t pasid);
>> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
>> +			    uint64_t fault_page, uint64_t ts,
>> +			    bool write_fault);
>> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
>> +
>> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>> +			 struct drm_file *filp);
>> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
>> +			    struct list_head *work_list);
>> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
>> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
>> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
>> +				 const struct amdgpu_svm_attrs *old_attrs,
>> +				 const struct amdgpu_svm_attrs *new_attrs,
>> +				 unsigned long start_page,
>> +				 unsigned long last_page);
>> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
>> +#else
>> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
>> +				  struct amdgpu_vm *vm)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
>> +{
>> +}
>> +
>> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
>> +{
>> +}
>> +
>> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
>> +					  uint32_t pasid,
>> +					  uint64_t fault_page,
>> +					  uint64_t ts,
>> +					  bool write_fault)
>> +{
>> +	return -EOPNOTSUPP;
>> +}
>> +
>> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
>> +{
>> +	return false;
>> +}
>> +
>> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>> +				       struct drm_file *filp)
>> +{
>> +	return -EOPNOTSUPP;
>> +}
>> +#endif /* CONFIG_DRM_AMDGPU_SVM */
>> +
>> +#endif /* __AMDGPU_SVM_H__ */
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> index ec1196d390bb7..30463a83e2e60 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>>   struct amdgpu_job;
>>   struct amdgpu_bo_list_entry;
>>   struct amdgpu_bo_vm;
>> +struct amdgpu_svm;
>>   
>>   /*
>>    * GPUVM handling
>> @@ -373,6 +374,9 @@ struct amdgpu_vm {
>>   
>>   	/* cached fault info */
>>   	struct amdgpu_vm_fault_info fault_info;
>> +
>> +	/* SVM experimental implementation */
>> +	struct amdgpu_svm *svm;
>>   };
>>   
>>   struct amdgpu_vm_manager {
> 

Re: [PATCH v9 02/18] drm/amdgpu: add SVM core header and VM integration

[Christian König]

On 8/11/26 16:06, Huang, Honglei wrote:
...
>>> +/*
>>> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
>>> + * Hold it in write mode around structural GPU SVM updates, including
>>> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
>>> + */
>>> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
>>> +{
>>> +    down_write(&svm->svm_lock);
>>> +}
>>> +
>>> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
>>> +{
>>> +    up_write(&svm->svm_lock);
>>> +}
>>> +
>>> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
>>> +{
>>> +    lockdep_assert_held_write(&svm->svm_lock);
>>> +}
>>
>> I'm starting to repeat myself, so once more: This stuff doesn't work like that!
>>
>> The lock the SVM subsystem uses to serialize updates *must* be the amdgpu_vm->eviction_lock and *not* a separate one.
>>
>> So clear NAK to having this functions here.
> 
> 
> I have explained why eviction lock can not be used as svm lock in V8, previous version. And it seems like we have a big gap about it.
> 
> The eviction lock can not be used for svm lcok.
> 
> And the design of svm lock is the core locking design of drmsvm frame work, without this desgin, this framework lost its soul. So I have to explain how to use this lock.
> 
> I believe we are talking about two different locks.

Yeah, that stuff is more than a bit complicated. The key point is that I still don't see any of the mandatory changes to amdgpu_vm.c in this patch set.

> You may treat svm lock as notifier lock. drm_gpusvm has two of them, and the "no allocation while held in the MMU notifier" rule applies
> to the other one, not to driver_svm_lock.
> 
> 1) drm_gpusvm has two distinct locks: drivers/gpu/drm/drm_gpusvm.c
> 
>   - notifier_lock: safeguards the notifier's range RB tree and list, as
>     well as the range's DMA mappings and sequence number. ... This lock
>     corresponds to the driver->update lock mentioned in
>     Documentation/mm/hmm.rst."
And that one here *MUST* be identical to the eviction lock in amdgpu_vm.c

The background is that XE uses a different page table allocation approach than amdgpu and we need to drop this lock in amdgpu to be able to allocate page tables. See function amdgpu_vm_pt_alloc().

With that design here that currently doesn't work at all.

We have two options, either use the drm_gpusvm notifier_lock as eviction_lock in amdgpu_vm.c or re-design amdgpu_vm.c to use the same approach for allocating page tables as XE.

Some engineer from Valve is working on re-designing amdgpu_vm.c, but that will potentially take month if not years.

So my take is that the new SVM code needs to modify amdgpu_vm.c so that the drm_gpusvm notifier_lock is used as eviction lock by the VM code.

Regards,
Christian.


> 
>   - driver_svm_lock: In addition to the locking mentioned above, the
>     driver should implement a lock to safeguard core GPU SVM function
>     calls that modify state, such as drm_gpusvm_range_find_or_insert and
>     drm_gpusvm_range_remove.
> 
>   Two locks, two jobs.
> 
> 2) The lock held in the MMU notifier is notifier_lock, never driver_svm_lock
> 
>   drm_gpusvm_notifier_invalidate():
>         down_write(&gpusvm->notifier_lock);
>         ...
>         gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
> 
>   The driver invalidate callback runs under notifier_lock only. Per the
>   framework's own notifier example it just unmaps pages
>   and queues the range to the garbage collector no allocation, and it
>   does not take driver_svm_lock:
> 
>         drm_gpusvm_range_unmap_pages(...);
>         drm_gpusvm_range_set_unmapped(...);
>         driver_garbage_collector_add(...);
> 
> 3) driver_svm_lock is by design an allocating, process context lock
> 
>   drm_gpusvm_range_find_or_insert() asserts it and then allocates under it:
> 
>         drm_gpusvm_range_find_or_insert():
>                 drm_gpusvm_driver_lock_held(gpusvm);
>                 ...
>                 range = drm_gpusvm_range_alloc(...);
>                 ... mmu_interval_notifier_insert(), kzalloc
> 
>   drm_gpusvm_range_remove() asserts it and frees. This is only safe
>   because driver_svm_lock is a sleepable, reclaim friendly lock that is
>   never taken from the MMU notifier. Reference counting
>    handles range *lifetime*, but it does not
>   serialize tree insert/remove, which is exactly why the framework still
>   asserts driver_svm_lock on those two entry points regardless of refcount.
> 
> Now the three concrete points:
> 
> A) Why the primary driver_svm_lock is required
> 
>   It is a framework requirement, not an amdgpu invention:
>    - DOC: Locking says the driver "should implement" it.
>    - drm_gpusvm lockdep-asserts it on every structural entry:
>      drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove() both
>      call drm_gpusvm_driver_lock_held().
>    - The reference fault handler holds it across the whole fault:
>      GC -> find_or_insert -> migrate -> get_pages -> bind.
> 
>   Xe does exactly this:
>    - xe_svm.c:      drm_gpusvm_driver_set_lock(&vm->svm.gpusvm, &vm->lock);
>    - xe_pagefault.c: down_write(&vm->lock); before dispatching the fault
>    - __xe_svm_handle_pagefault(): lockdep_assert_held_write(&vm->lock);
>      held across GC / find_or_insert / alloc_vram / get_pages / rebind
>    - xe_svm_garbage_collector(): lockdep_assert_held_write(&vm->lock);
> 
>   amdgpu's svm_lock is the same driver_svm_lock, used the same way.
> 
> B) Why eviction_lock cannot be that lock
> 
>> This lock eviction_lock can only be grabbed while updating the mapping range.
> 
>   and that is precisely why it cannot be driver_svm_lock.
>   driver_svm_lock must wrap find_or_insert, migration, and
>   drm_gpusvm_range_get_pages
>   eviction_lock is the opposite by contract:
> 
>    - It is taken with memalloc_noreclaim_save() in
>      amdgpu_vm_begin_critical(), specifically so no reclaim happens while
>      held (to avoid the reclaim -> MMU-notifier deadlock). Holding it
>      across get_pages/migration breaks that.
>    - TTM eviction try-locks it: amdgpu_vm_evictable() does
>      scoped_cond_guard(mutex_try, return false, &vm->eviction_lock) and
>      sets vm->evicting. Long holds starve eviction.
>    - It is a plain mutex that the SVM map path re-enters:
>      amdgpu_svm_range_update_mapping() -> amdgpu_vm_map_range() ->
>      amdgpu_vm_begin_critical() -> mutex_lock(&vm->eviction_lock). If
>      eviction_lock were also the outer SVM lock, this is a self-deadlock.
> 
>   In short, eviction_lock has the contract of notifier_lock , not of
>   driver_svm_lock. This is also why the current split is correct:
>   svm_lock (outer) != eviction_lock (inner). Your own rule - "you can't
>   call the VM code with the lock held, the VM code must take it itself" -
>   is satisfied today only because they are separate: svm_lock is held
>   while calling amdgpu_vm_map_range(), and amdgpu_vm_map_range() takes
>   eviction_lock itself. Merging them is what would violate that rule.
> 
>> No, they Xe vm->lock and eviction_lock are actually identical in the handling.
> 
>   They are not. Xe's vm->lock is a rw_semaphore, the "outer most lock" of
>   the VM , held down_write across the whole fault. amdgpu's
>   eviction_lock is a mutex taken only inside amdgpu_vm_begin_critical()
>   during a PT update, under memalloc_noreclaim. Xe's eviction/reclaim
>   handling is separate from vm->lock. The amdgpu analogue of Xe's vm->lock
>   is svm_lock, not eviction_lock.
> 
> C) Reusing an existing amdgpu_vm lock as the primary lock needs refactor amdgpu VM
> 
>   Xe can register vm->lock because Xe's VM was designed with an outer
>   rw_semaphore held across faults. amdgpu_vm has no such lock: only
>   eviction_lock , the root PD dma_resv , and a few spinlocks.
> 
>   So do it like Xe means introducing a dedicated, outer, sleepable VM
>   lock held across the fault. That lock is exactly svm_lock. Folding it
>   into struct amdgpu_vm as a general vm->lock is a core amdgpu VM refactor.
> 
> Regards,
> Honglei
> 
> 
>>
>> Regards,
>> Christian.
>>
>>> +
>>> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
>>> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
>>> +
>>> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>>> +void amdgpu_svm_close(struct amdgpu_vm *vm);
>>> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
>>> +
>>> +void amdgpu_svm_put(struct amdgpu_svm *svm);
>>> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
>>> +                           uint32_t pasid);
>>> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
>>> +                uint64_t fault_page, uint64_t ts,
>>> +                bool write_fault);
>>> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
>>> +
>>> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>>> +             struct drm_file *filp);
>>> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
>>> +                struct list_head *work_list);
>>> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
>>> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
>>> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
>>> +                 const struct amdgpu_svm_attrs *old_attrs,
>>> +                 const struct amdgpu_svm_attrs *new_attrs,
>>> +                 unsigned long start_page,
>>> +                 unsigned long last_page);
>>> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
>>> +#else
>>> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
>>> +                  struct amdgpu_vm *vm)
>>> +{
>>> +    return 0;
>>> +}
>>> +
>>> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
>>> +{
>>> +}
>>> +
>>> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
>>> +{
>>> +}
>>> +
>>> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
>>> +                      uint32_t pasid,
>>> +                      uint64_t fault_page,
>>> +                      uint64_t ts,
>>> +                      bool write_fault)
>>> +{
>>> +    return -EOPNOTSUPP;
>>> +}
>>> +
>>> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
>>> +{
>>> +    return false;
>>> +}
>>> +
>>> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>>> +                       struct drm_file *filp)
>>> +{
>>> +    return -EOPNOTSUPP;
>>> +}
>>> +#endif /* CONFIG_DRM_AMDGPU_SVM */
>>> +
>>> +#endif /* __AMDGPU_SVM_H__ */
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> index ec1196d390bb7..30463a83e2e60 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>>>   struct amdgpu_job;
>>>   struct amdgpu_bo_list_entry;
>>>   struct amdgpu_bo_vm;
>>> +struct amdgpu_svm;
>>>     /*
>>>    * GPUVM handling
>>> @@ -373,6 +374,9 @@ struct amdgpu_vm {
>>>         /* cached fault info */
>>>       struct amdgpu_vm_fault_info fault_info;
>>> +
>>> +    /* SVM experimental implementation */
>>> +    struct amdgpu_svm *svm;
>>>   };
>>>     struct amdgpu_vm_manager {
>>
> 

Re: [PATCH v9 02/18] drm/amdgpu: add SVM core header and VM integration

[Huang, Honglei]

On 8/12/2026 4:36 PM, Christian König wrote:
> On 8/11/26 16:06, Huang, Honglei wrote:
> ...
>>>> +/*
>>>> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
>>>> + * Hold it in write mode around structural GPU SVM updates, including
>>>> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
>>>> + */
>>>> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
>>>> +{
>>>> +    down_write(&svm->svm_lock);
>>>> +}
>>>> +
>>>> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
>>>> +{
>>>> +    up_write(&svm->svm_lock);
>>>> +}
>>>> +
>>>> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
>>>> +{
>>>> +    lockdep_assert_held_write(&svm->svm_lock);
>>>> +}
>>>
>>> I'm starting to repeat myself, so once more: This stuff doesn't work like that!
>>>
>>> The lock the SVM subsystem uses to serialize updates *must* be the amdgpu_vm->eviction_lock and *not* a separate one.
>>>
>>> So clear NAK to having this functions here.
>>
>>
>> I have explained why eviction lock can not be used as svm lock in V8, previous version. And it seems like we have a big gap about it.
>>
>> The eviction lock can not be used for svm lcok.
>>
>> And the design of svm lock is the core locking design of drmsvm frame work, without this desgin, this framework lost its soul. So I have to explain how to use this lock.
>>
>> I believe we are talking about two different locks.
> 
> Yeah, that stuff is more than a bit complicated. The key point is that I still don't see any of the mandatory changes to amdgpu_vm.c in this patch set.
> 
>> You may treat svm lock as notifier lock. drm_gpusvm has two of them, and the "no allocation while held in the MMU notifier" rule applies
>> to the other one, not to driver_svm_lock.
>>
>> 1) drm_gpusvm has two distinct locks: drivers/gpu/drm/drm_gpusvm.c
>>
>>    - notifier_lock: safeguards the notifier's range RB tree and list, as
>>      well as the range's DMA mappings and sequence number. ... This lock
>>      corresponds to the driver->update lock mentioned in
>>      Documentation/mm/hmm.rst."
> And that one here *MUST* be identical to the eviction lock in amdgpu_vm.c

Actually I have a question about this,

The notifier lock protects the CPU pages tables, it prevents the 
migration/swap ... from MM logic.

And the eviction lock protects the GPU VM page table, it prevents gpu 
page table changes from TTM logic.

They have different jobs,

when doing a GPU mapping in SVM, cpu page table can not change, casue it
may change the cpu dma addr -> gpu mapping. So must hold it, it is done 
in current code, and it is must required by drm gpu svm frame work.

And at the same time the eviciton lock must hold also, casue the GPU 
page tables may change by TTM logic.

Those two locks are all need be hold, no conflict, this is just my thought.

Regards,
Honglei

> 
> The background is that XE uses a different page table allocation approach than amdgpu and we need to drop this lock in amdgpu to be able to allocate page tables. See function amdgpu_vm_pt_alloc().
> 
> With that design here that currently doesn't work at all.
> 
> We have two options, either use the drm_gpusvm notifier_lock as eviction_lock in amdgpu_vm.c or re-design amdgpu_vm.c to use the same approach for allocating page tables as XE.
> 
> Some engineer from Valve is working on re-designing amdgpu_vm.c, but that will potentially take month if not years.
> 
> So my take is that the new SVM code needs to modify amdgpu_vm.c so that the drm_gpusvm notifier_lock is used as eviction lock by the VM code.
> 

> Regards,
> Christian.
> 
> 
>>
>>    - driver_svm_lock: In addition to the locking mentioned above, the
>>      driver should implement a lock to safeguard core GPU SVM function
>>      calls that modify state, such as drm_gpusvm_range_find_or_insert and
>>      drm_gpusvm_range_remove.
>>
>>    Two locks, two jobs.
>>
>> 2) The lock held in the MMU notifier is notifier_lock, never driver_svm_lock
>>
>>    drm_gpusvm_notifier_invalidate():
>>          down_write(&gpusvm->notifier_lock);
>>          ...
>>          gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
>>
>>    The driver invalidate callback runs under notifier_lock only. Per the
>>    framework's own notifier example it just unmaps pages
>>    and queues the range to the garbage collector no allocation, and it
>>    does not take driver_svm_lock:
>>
>>          drm_gpusvm_range_unmap_pages(...);
>>          drm_gpusvm_range_set_unmapped(...);
>>          driver_garbage_collector_add(...);
>>
>> 3) driver_svm_lock is by design an allocating, process context lock
>>
>>    drm_gpusvm_range_find_or_insert() asserts it and then allocates under it:
>>
>>          drm_gpusvm_range_find_or_insert():
>>                  drm_gpusvm_driver_lock_held(gpusvm);
>>                  ...
>>                  range = drm_gpusvm_range_alloc(...);
>>                  ... mmu_interval_notifier_insert(), kzalloc
>>
>>    drm_gpusvm_range_remove() asserts it and frees. This is only safe
>>    because driver_svm_lock is a sleepable, reclaim friendly lock that is
>>    never taken from the MMU notifier. Reference counting
>>     handles range *lifetime*, but it does not
>>    serialize tree insert/remove, which is exactly why the framework still
>>    asserts driver_svm_lock on those two entry points regardless of refcount.
>>
>> Now the three concrete points:
>>
>> A) Why the primary driver_svm_lock is required
>>
>>    It is a framework requirement, not an amdgpu invention:
>>     - DOC: Locking says the driver "should implement" it.
>>     - drm_gpusvm lockdep-asserts it on every structural entry:
>>       drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove() both
>>       call drm_gpusvm_driver_lock_held().
>>     - The reference fault handler holds it across the whole fault:
>>       GC -> find_or_insert -> migrate -> get_pages -> bind.
>>
>>    Xe does exactly this:
>>     - xe_svm.c:      drm_gpusvm_driver_set_lock(&vm->svm.gpusvm, &vm->lock);
>>     - xe_pagefault.c: down_write(&vm->lock); before dispatching the fault
>>     - __xe_svm_handle_pagefault(): lockdep_assert_held_write(&vm->lock);
>>       held across GC / find_or_insert / alloc_vram / get_pages / rebind
>>     - xe_svm_garbage_collector(): lockdep_assert_held_write(&vm->lock);
>>
>>    amdgpu's svm_lock is the same driver_svm_lock, used the same way.
>>
>> B) Why eviction_lock cannot be that lock
>>
>>> This lock eviction_lock can only be grabbed while updating the mapping range.
>>
>>    and that is precisely why it cannot be driver_svm_lock.
>>    driver_svm_lock must wrap find_or_insert, migration, and
>>    drm_gpusvm_range_get_pages
>>    eviction_lock is the opposite by contract:
>>
>>     - It is taken with memalloc_noreclaim_save() in
>>       amdgpu_vm_begin_critical(), specifically so no reclaim happens while
>>       held (to avoid the reclaim -> MMU-notifier deadlock). Holding it
>>       across get_pages/migration breaks that.
>>     - TTM eviction try-locks it: amdgpu_vm_evictable() does
>>       scoped_cond_guard(mutex_try, return false, &vm->eviction_lock) and
>>       sets vm->evicting. Long holds starve eviction.
>>     - It is a plain mutex that the SVM map path re-enters:
>>       amdgpu_svm_range_update_mapping() -> amdgpu_vm_map_range() ->
>>       amdgpu_vm_begin_critical() -> mutex_lock(&vm->eviction_lock). If
>>       eviction_lock were also the outer SVM lock, this is a self-deadlock.
>>
>>    In short, eviction_lock has the contract of notifier_lock , not of
>>    driver_svm_lock. This is also why the current split is correct:
>>    svm_lock (outer) != eviction_lock (inner). Your own rule - "you can't
>>    call the VM code with the lock held, the VM code must take it itself" -
>>    is satisfied today only because they are separate: svm_lock is held
>>    while calling amdgpu_vm_map_range(), and amdgpu_vm_map_range() takes
>>    eviction_lock itself. Merging them is what would violate that rule.
>>
>>> No, they Xe vm->lock and eviction_lock are actually identical in the handling.
>>
>>    They are not. Xe's vm->lock is a rw_semaphore, the "outer most lock" of
>>    the VM , held down_write across the whole fault. amdgpu's
>>    eviction_lock is a mutex taken only inside amdgpu_vm_begin_critical()
>>    during a PT update, under memalloc_noreclaim. Xe's eviction/reclaim
>>    handling is separate from vm->lock. The amdgpu analogue of Xe's vm->lock
>>    is svm_lock, not eviction_lock.
>>
>> C) Reusing an existing amdgpu_vm lock as the primary lock needs refactor amdgpu VM
>>
>>    Xe can register vm->lock because Xe's VM was designed with an outer
>>    rw_semaphore held across faults. amdgpu_vm has no such lock: only
>>    eviction_lock , the root PD dma_resv , and a few spinlocks.
>>
>>    So do it like Xe means introducing a dedicated, outer, sleepable VM
>>    lock held across the fault. That lock is exactly svm_lock. Folding it
>>    into struct amdgpu_vm as a general vm->lock is a core amdgpu VM refactor.
>>
>> Regards,
>> Honglei
>>
>>
>>>
>>> Regards,
>>> Christian.
>>>
>>>> +
>>>> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
>>>> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
>>>> +
>>>> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>>>> +void amdgpu_svm_close(struct amdgpu_vm *vm);
>>>> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
>>>> +
>>>> +void amdgpu_svm_put(struct amdgpu_svm *svm);
>>>> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
>>>> +                           uint32_t pasid);
>>>> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
>>>> +                uint64_t fault_page, uint64_t ts,
>>>> +                bool write_fault);
>>>> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
>>>> +
>>>> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>>>> +             struct drm_file *filp);
>>>> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
>>>> +                struct list_head *work_list);
>>>> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
>>>> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
>>>> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
>>>> +                 const struct amdgpu_svm_attrs *old_attrs,
>>>> +                 const struct amdgpu_svm_attrs *new_attrs,
>>>> +                 unsigned long start_page,
>>>> +                 unsigned long last_page);
>>>> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
>>>> +#else
>>>> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
>>>> +                  struct amdgpu_vm *vm)
>>>> +{
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
>>>> +{
>>>> +}
>>>> +
>>>> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
>>>> +{
>>>> +}
>>>> +
>>>> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
>>>> +                      uint32_t pasid,
>>>> +                      uint64_t fault_page,
>>>> +                      uint64_t ts,
>>>> +                      bool write_fault)
>>>> +{
>>>> +    return -EOPNOTSUPP;
>>>> +}
>>>> +
>>>> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
>>>> +{
>>>> +    return false;
>>>> +}
>>>> +
>>>> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>>>> +                       struct drm_file *filp)
>>>> +{
>>>> +    return -EOPNOTSUPP;
>>>> +}
>>>> +#endif /* CONFIG_DRM_AMDGPU_SVM */
>>>> +
>>>> +#endif /* __AMDGPU_SVM_H__ */
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>> index ec1196d390bb7..30463a83e2e60 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>>>>    struct amdgpu_job;
>>>>    struct amdgpu_bo_list_entry;
>>>>    struct amdgpu_bo_vm;
>>>> +struct amdgpu_svm;
>>>>      /*
>>>>     * GPUVM handling
>>>> @@ -373,6 +374,9 @@ struct amdgpu_vm {
>>>>          /* cached fault info */
>>>>        struct amdgpu_vm_fault_info fault_info;
>>>> +
>>>> +    /* SVM experimental implementation */
>>>> +    struct amdgpu_svm *svm;
>>>>    };
>>>>      struct amdgpu_vm_manager {
>>>
>>

Re: [PATCH v9 02/18] drm/amdgpu: add SVM core header and VM integration

[Christian König]

On 8/12/26 11:55, Huang, Honglei wrote:
> On 8/12/2026 4:36 PM, Christian König wrote:
>> On 8/11/26 16:06, Huang, Honglei wrote:
>> ...
>>>>> +/*
>>>>> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
>>>>> + * Hold it in write mode around structural GPU SVM updates, including
>>>>> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
>>>>> + */
>>>>> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
>>>>> +{
>>>>> +    down_write(&svm->svm_lock);
>>>>> +}
>>>>> +
>>>>> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
>>>>> +{
>>>>> +    up_write(&svm->svm_lock);
>>>>> +}
>>>>> +
>>>>> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
>>>>> +{
>>>>> +    lockdep_assert_held_write(&svm->svm_lock);
>>>>> +}
>>>>
>>>> I'm starting to repeat myself, so once more: This stuff doesn't work like that!
>>>>
>>>> The lock the SVM subsystem uses to serialize updates *must* be the amdgpu_vm->eviction_lock and *not* a separate one.
>>>>
>>>> So clear NAK to having this functions here.
>>>
>>>
>>> I have explained why eviction lock can not be used as svm lock in V8, previous version. And it seems like we have a big gap about it.
>>>
>>> The eviction lock can not be used for svm lcok.
>>>
>>> And the design of svm lock is the core locking design of drmsvm frame work, without this desgin, this framework lost its soul. So I have to explain how to use this lock.
>>>
>>> I believe we are talking about two different locks.
>>
>> Yeah, that stuff is more than a bit complicated. The key point is that I still don't see any of the mandatory changes to amdgpu_vm.c in this patch set.
>>
>>> You may treat svm lock as notifier lock. drm_gpusvm has two of them, and the "no allocation while held in the MMU notifier" rule applies
>>> to the other one, not to driver_svm_lock.
>>>
>>> 1) drm_gpusvm has two distinct locks: drivers/gpu/drm/drm_gpusvm.c
>>>
>>>    - notifier_lock: safeguards the notifier's range RB tree and list, as
>>>      well as the range's DMA mappings and sequence number. ... This lock
>>>      corresponds to the driver->update lock mentioned in
>>>      Documentation/mm/hmm.rst."
>> And that one here *MUST* be identical to the eviction lock in amdgpu_vm.c
> 
> Actually I have a question about this,
> 
> The notifier lock protects the CPU pages tables, it prevents the migration/swap ... from MM logic.
> 
> And the eviction lock protects the GPU VM page table, it prevents gpu page table changes from TTM logic.
> 
> They have different jobs,

No, exactly that is what we have gotten wrong in the existing KFD SVM implementation.

> 
> when doing a GPU mapping in SVM, cpu page table can not change, casue it
> may change the cpu dma addr -> gpu mapping. So must hold it, it is done in current code, and it is must required by drm gpu svm frame work.
> 
> And at the same time the eviciton lock must hold also, casue the GPU page tables may change by TTM logic.
> 
> Those two locks are all need be hold, no conflict, this is just my thought.

Originally the notifier_lock only made sure that the CPU page table updates where done in order and originally the eviction lock made sure that the GPU page table updates where done in order, but essentially we need the order for both.

The point is that the updates need to be serialized. In other words when one CPU is doing a mapping operation and another CPU is doing an unmap through an MMU notifier we somehow need to make sure that the coresponding GPU page table updates execute in the correct order.

So essentially those two locks need to be the same one, and we need to drop the lock to allocate page tables and when we re-acquire it we need to double check the sequence number to make sure that no unmap operation happened concurrently.

And yeah I know that this makes things much much more complicated, but it is definately necessary.

Regards,
Christian.

> 
> Regards,
> Honglei
> 
>>
>> The background is that XE uses a different page table allocation approach than amdgpu and we need to drop this lock in amdgpu to be able to allocate page tables. See function amdgpu_vm_pt_alloc().
>>
>> With that design here that currently doesn't work at all.
>>
>> We have two options, either use the drm_gpusvm notifier_lock as eviction_lock in amdgpu_vm.c or re-design amdgpu_vm.c to use the same approach for allocating page tables as XE.
>>
>> Some engineer from Valve is working on re-designing amdgpu_vm.c, but that will potentially take month if not years.
>>
>> So my take is that the new SVM code needs to modify amdgpu_vm.c so that the drm_gpusvm notifier_lock is used as eviction lock by the VM code.
>>
> 
>> Regards,
>> Christian.
>>
>>
>>>
>>>    - driver_svm_lock: In addition to the locking mentioned above, the
>>>      driver should implement a lock to safeguard core GPU SVM function
>>>      calls that modify state, such as drm_gpusvm_range_find_or_insert and
>>>      drm_gpusvm_range_remove.
>>>
>>>    Two locks, two jobs.
>>>
>>> 2) The lock held in the MMU notifier is notifier_lock, never driver_svm_lock
>>>
>>>    drm_gpusvm_notifier_invalidate():
>>>          down_write(&gpusvm->notifier_lock);
>>>          ...
>>>          gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
>>>
>>>    The driver invalidate callback runs under notifier_lock only. Per the
>>>    framework's own notifier example it just unmaps pages
>>>    and queues the range to the garbage collector no allocation, and it
>>>    does not take driver_svm_lock:
>>>
>>>          drm_gpusvm_range_unmap_pages(...);
>>>          drm_gpusvm_range_set_unmapped(...);
>>>          driver_garbage_collector_add(...);
>>>
>>> 3) driver_svm_lock is by design an allocating, process context lock
>>>
>>>    drm_gpusvm_range_find_or_insert() asserts it and then allocates under it:
>>>
>>>          drm_gpusvm_range_find_or_insert():
>>>                  drm_gpusvm_driver_lock_held(gpusvm);
>>>                  ...
>>>                  range = drm_gpusvm_range_alloc(...);
>>>                  ... mmu_interval_notifier_insert(), kzalloc
>>>
>>>    drm_gpusvm_range_remove() asserts it and frees. This is only safe
>>>    because driver_svm_lock is a sleepable, reclaim friendly lock that is
>>>    never taken from the MMU notifier. Reference counting
>>>     handles range *lifetime*, but it does not
>>>    serialize tree insert/remove, which is exactly why the framework still
>>>    asserts driver_svm_lock on those two entry points regardless of refcount.
>>>
>>> Now the three concrete points:
>>>
>>> A) Why the primary driver_svm_lock is required
>>>
>>>    It is a framework requirement, not an amdgpu invention:
>>>     - DOC: Locking says the driver "should implement" it.
>>>     - drm_gpusvm lockdep-asserts it on every structural entry:
>>>       drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove() both
>>>       call drm_gpusvm_driver_lock_held().
>>>     - The reference fault handler holds it across the whole fault:
>>>       GC -> find_or_insert -> migrate -> get_pages -> bind.
>>>
>>>    Xe does exactly this:
>>>     - xe_svm.c:      drm_gpusvm_driver_set_lock(&vm->svm.gpusvm, &vm->lock);
>>>     - xe_pagefault.c: down_write(&vm->lock); before dispatching the fault
>>>     - __xe_svm_handle_pagefault(): lockdep_assert_held_write(&vm->lock);
>>>       held across GC / find_or_insert / alloc_vram / get_pages / rebind
>>>     - xe_svm_garbage_collector(): lockdep_assert_held_write(&vm->lock);
>>>
>>>    amdgpu's svm_lock is the same driver_svm_lock, used the same way.
>>>
>>> B) Why eviction_lock cannot be that lock
>>>
>>>> This lock eviction_lock can only be grabbed while updating the mapping range.
>>>
>>>    and that is precisely why it cannot be driver_svm_lock.
>>>    driver_svm_lock must wrap find_or_insert, migration, and
>>>    drm_gpusvm_range_get_pages
>>>    eviction_lock is the opposite by contract:
>>>
>>>     - It is taken with memalloc_noreclaim_save() in
>>>       amdgpu_vm_begin_critical(), specifically so no reclaim happens while
>>>       held (to avoid the reclaim -> MMU-notifier deadlock). Holding it
>>>       across get_pages/migration breaks that.
>>>     - TTM eviction try-locks it: amdgpu_vm_evictable() does
>>>       scoped_cond_guard(mutex_try, return false, &vm->eviction_lock) and
>>>       sets vm->evicting. Long holds starve eviction.
>>>     - It is a plain mutex that the SVM map path re-enters:
>>>       amdgpu_svm_range_update_mapping() -> amdgpu_vm_map_range() ->
>>>       amdgpu_vm_begin_critical() -> mutex_lock(&vm->eviction_lock). If
>>>       eviction_lock were also the outer SVM lock, this is a self-deadlock.
>>>
>>>    In short, eviction_lock has the contract of notifier_lock , not of
>>>    driver_svm_lock. This is also why the current split is correct:
>>>    svm_lock (outer) != eviction_lock (inner). Your own rule - "you can't
>>>    call the VM code with the lock held, the VM code must take it itself" -
>>>    is satisfied today only because they are separate: svm_lock is held
>>>    while calling amdgpu_vm_map_range(), and amdgpu_vm_map_range() takes
>>>    eviction_lock itself. Merging them is what would violate that rule.
>>>
>>>> No, they Xe vm->lock and eviction_lock are actually identical in the handling.
>>>
>>>    They are not. Xe's vm->lock is a rw_semaphore, the "outer most lock" of
>>>    the VM , held down_write across the whole fault. amdgpu's
>>>    eviction_lock is a mutex taken only inside amdgpu_vm_begin_critical()
>>>    during a PT update, under memalloc_noreclaim. Xe's eviction/reclaim
>>>    handling is separate from vm->lock. The amdgpu analogue of Xe's vm->lock
>>>    is svm_lock, not eviction_lock.
>>>
>>> C) Reusing an existing amdgpu_vm lock as the primary lock needs refactor amdgpu VM
>>>
>>>    Xe can register vm->lock because Xe's VM was designed with an outer
>>>    rw_semaphore held across faults. amdgpu_vm has no such lock: only
>>>    eviction_lock , the root PD dma_resv , and a few spinlocks.
>>>
>>>    So do it like Xe means introducing a dedicated, outer, sleepable VM
>>>    lock held across the fault. That lock is exactly svm_lock. Folding it
>>>    into struct amdgpu_vm as a general vm->lock is a core amdgpu VM refactor.
>>>
>>> Regards,
>>> Honglei
>>>
>>>
>>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>>> +
>>>>> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
>>>>> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
>>>>> +
>>>>> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>>>>> +void amdgpu_svm_close(struct amdgpu_vm *vm);
>>>>> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
>>>>> +
>>>>> +void amdgpu_svm_put(struct amdgpu_svm *svm);
>>>>> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
>>>>> +                           uint32_t pasid);
>>>>> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
>>>>> +                uint64_t fault_page, uint64_t ts,
>>>>> +                bool write_fault);
>>>>> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
>>>>> +
>>>>> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>>>>> +             struct drm_file *filp);
>>>>> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
>>>>> +                struct list_head *work_list);
>>>>> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
>>>>> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
>>>>> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
>>>>> +                 const struct amdgpu_svm_attrs *old_attrs,
>>>>> +                 const struct amdgpu_svm_attrs *new_attrs,
>>>>> +                 unsigned long start_page,
>>>>> +                 unsigned long last_page);
>>>>> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
>>>>> +#else
>>>>> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
>>>>> +                  struct amdgpu_vm *vm)
>>>>> +{
>>>>> +    return 0;
>>>>> +}
>>>>> +
>>>>> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
>>>>> +{
>>>>> +}
>>>>> +
>>>>> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
>>>>> +{
>>>>> +}
>>>>> +
>>>>> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
>>>>> +                      uint32_t pasid,
>>>>> +                      uint64_t fault_page,
>>>>> +                      uint64_t ts,
>>>>> +                      bool write_fault)
>>>>> +{
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +
>>>>> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
>>>>> +{
>>>>> +    return false;
>>>>> +}
>>>>> +
>>>>> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>>>>> +                       struct drm_file *filp)
>>>>> +{
>>>>> +    return -EOPNOTSUPP;
>>>>> +}
>>>>> +#endif /* CONFIG_DRM_AMDGPU_SVM */
>>>>> +
>>>>> +#endif /* __AMDGPU_SVM_H__ */
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>> index ec1196d390bb7..30463a83e2e60 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>>>>> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>>>>>    struct amdgpu_job;
>>>>>    struct amdgpu_bo_list_entry;
>>>>>    struct amdgpu_bo_vm;
>>>>> +struct amdgpu_svm;
>>>>>      /*
>>>>>     * GPUVM handling
>>>>> @@ -373,6 +374,9 @@ struct amdgpu_vm {
>>>>>          /* cached fault info */
>>>>>        struct amdgpu_vm_fault_info fault_info;
>>>>> +
>>>>> +    /* SVM experimental implementation */
>>>>> +    struct amdgpu_svm *svm;
>>>>>    };
>>>>>      struct amdgpu_vm_manager {
>>>>
>>>
> 

Re: [PATCH v9 02/18] drm/amdgpu: add SVM core header and VM integration

[Huang Rui]

On Wed, Aug 12, 2026 at 02:16:03PM +0200, Christian KKKnig wrote:
> On 8/12/26 11:55, Huang, Honglei wrote:
> > On 8/12/2026 4:36 PM, Christian König wrote:
> >> On 8/11/26 16:06, Huang, Honglei wrote:
> >> ...
> >>>>> +/*
> >>>>> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
> >>>>> + * Hold it in write mode around structural GPU SVM updates, including
> >>>>> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
> >>>>> + */
> >>>>> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
> >>>>> +{
> >>>>> +    down_write(&svm->svm_lock);
> >>>>> +}
> >>>>> +
> >>>>> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
> >>>>> +{
> >>>>> +    up_write(&svm->svm_lock);
> >>>>> +}
> >>>>> +
> >>>>> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
> >>>>> +{
> >>>>> +    lockdep_assert_held_write(&svm->svm_lock);
> >>>>> +}
> >>>>
> >>>> I'm starting to repeat myself, so once more: This stuff doesn't work like that!
> >>>>
> >>>> The lock the SVM subsystem uses to serialize updates *must* be the amdgpu_vm->eviction_lock and *not* a separate one.
> >>>>
> >>>> So clear NAK to having this functions here.
> >>>
> >>>
> >>> I have explained why eviction lock can not be used as svm lock in V8, previous version. And it seems like we have a big gap about it.
> >>>
> >>> The eviction lock can not be used for svm lcok.
> >>>
> >>> And the design of svm lock is the core locking design of drmsvm frame work, without this desgin, this framework lost its soul. So I have to explain how to use this lock.
> >>>
> >>> I believe we are talking about two different locks.
> >>
> >> Yeah, that stuff is more than a bit complicated. The key point is that I still don't see any of the mandatory changes to amdgpu_vm.c in this patch set.
> >>
> >>> You may treat svm lock as notifier lock. drm_gpusvm has two of them, and the "no allocation while held in the MMU notifier" rule applies
> >>> to the other one, not to driver_svm_lock.
> >>>
> >>> 1) drm_gpusvm has two distinct locks: drivers/gpu/drm/drm_gpusvm.c
> >>>
> >>>    - notifier_lock: safeguards the notifier's range RB tree and list, as
> >>>      well as the range's DMA mappings and sequence number. ... This lock
> >>>      corresponds to the driver->update lock mentioned in
> >>>      Documentation/mm/hmm.rst."
> >> And that one here *MUST* be identical to the eviction lock in amdgpu_vm.c
> > 
> > Actually I have a question about this,
> > 
> > The notifier lock protects the CPU pages tables, it prevents the migration/swap ... from MM logic.
> > 
> > And the eviction lock protects the GPU VM page table, it prevents gpu page table changes from TTM logic.
> > 
> > They have different jobs,
> 
> No, exactly that is what we have gotten wrong in the existing KFD SVM implementation.
> 
> > 
> > when doing a GPU mapping in SVM, cpu page table can not change, casue it
> > may change the cpu dma addr -> gpu mapping. So must hold it, it is done in current code, and it is must required by drm gpu svm frame work.
> > 
> > And at the same time the eviciton lock must hold also, casue the GPU page tables may change by TTM logic.
> > 
> > Those two locks are all need be hold, no conflict, this is just my thought.
> 
> Originally the notifier_lock only made sure that the CPU page table updates where done in order and originally the eviction lock made sure that the GPU page table updates where done in order, but essentially we need the order for both.
> 
> The point is that the updates need to be serialized. In other words when one CPU is doing a mapping operation and another CPU is doing an unmap through an MMU notifier we somehow need to make sure that the coresponding GPU page table updates execute in the correct order.
> 
> So essentially those two locks need to be the same one, and we need to drop the lock to allocate page tables and when we re-acquire it we need to double check the sequence number to make sure that no unmap operation happened concurrently.
> 
> And yeah I know that this makes things much much more complicated, but it is definately necessary.
> 

So, if I understand your point correctly, the best way to solve the
serialization issue between these two locks is to turn them into a single
lock. Given that a full re-design of amdgpu_vm could take a significant
amount of time, it seems that using drm_gpusvm's notifier_lock as a
replacement for eviction_lock in amdgpu_vm would be the more practical
short-term solution.

Please correct me if I've misunderstood your position.

Thanks,
Ray

> Regards,
> Christian.
> 
> > 
> > Regards,
> > Honglei
> > 
> >>
> >> The background is that XE uses a different page table allocation approach than amdgpu and we need to drop this lock in amdgpu to be able to allocate page tables. See function amdgpu_vm_pt_alloc().
> >>
> >> With that design here that currently doesn't work at all.
> >>
> >> We have two options, either use the drm_gpusvm notifier_lock as eviction_lock in amdgpu_vm.c or re-design amdgpu_vm.c to use the same approach for allocating page tables as XE.
> >>
> >> Some engineer from Valve is working on re-designing amdgpu_vm.c, but that will potentially take month if not years.
> >>
> >> So my take is that the new SVM code needs to modify amdgpu_vm.c so that the drm_gpusvm notifier_lock is used as eviction lock by the VM code.
> >>
> > 
> >> Regards,
> >> Christian.
> >>
> >>
> >>>
> >>>    - driver_svm_lock: In addition to the locking mentioned above, the
> >>>      driver should implement a lock to safeguard core GPU SVM function
> >>>      calls that modify state, such as drm_gpusvm_range_find_or_insert and
> >>>      drm_gpusvm_range_remove.
> >>>
> >>>    Two locks, two jobs.
> >>>
> >>> 2) The lock held in the MMU notifier is notifier_lock, never driver_svm_lock
> >>>
> >>>    drm_gpusvm_notifier_invalidate():
> >>>          down_write(&gpusvm->notifier_lock);
> >>>          ...
> >>>          gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
> >>>
> >>>    The driver invalidate callback runs under notifier_lock only. Per the
> >>>    framework's own notifier example it just unmaps pages
> >>>    and queues the range to the garbage collector no allocation, and it
> >>>    does not take driver_svm_lock:
> >>>
> >>>          drm_gpusvm_range_unmap_pages(...);
> >>>          drm_gpusvm_range_set_unmapped(...);
> >>>          driver_garbage_collector_add(...);
> >>>
> >>> 3) driver_svm_lock is by design an allocating, process context lock
> >>>
> >>>    drm_gpusvm_range_find_or_insert() asserts it and then allocates under it:
> >>>
> >>>          drm_gpusvm_range_find_or_insert():
> >>>                  drm_gpusvm_driver_lock_held(gpusvm);
> >>>                  ...
> >>>                  range = drm_gpusvm_range_alloc(...);
> >>>                  ... mmu_interval_notifier_insert(), kzalloc
> >>>
> >>>    drm_gpusvm_range_remove() asserts it and frees. This is only safe
> >>>    because driver_svm_lock is a sleepable, reclaim friendly lock that is
> >>>    never taken from the MMU notifier. Reference counting
> >>>     handles range *lifetime*, but it does not
> >>>    serialize tree insert/remove, which is exactly why the framework still
> >>>    asserts driver_svm_lock on those two entry points regardless of refcount.
> >>>
> >>> Now the three concrete points:
> >>>
> >>> A) Why the primary driver_svm_lock is required
> >>>
> >>>    It is a framework requirement, not an amdgpu invention:
> >>>     - DOC: Locking says the driver "should implement" it.
> >>>     - drm_gpusvm lockdep-asserts it on every structural entry:
> >>>       drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove() both
> >>>       call drm_gpusvm_driver_lock_held().
> >>>     - The reference fault handler holds it across the whole fault:
> >>>       GC -> find_or_insert -> migrate -> get_pages -> bind.
> >>>
> >>>    Xe does exactly this:
> >>>     - xe_svm.c:      drm_gpusvm_driver_set_lock(&vm->svm.gpusvm, &vm->lock);
> >>>     - xe_pagefault.c: down_write(&vm->lock); before dispatching the fault
> >>>     - __xe_svm_handle_pagefault(): lockdep_assert_held_write(&vm->lock);
> >>>       held across GC / find_or_insert / alloc_vram / get_pages / rebind
> >>>     - xe_svm_garbage_collector(): lockdep_assert_held_write(&vm->lock);
> >>>
> >>>    amdgpu's svm_lock is the same driver_svm_lock, used the same way.
> >>>
> >>> B) Why eviction_lock cannot be that lock
> >>>
> >>>> This lock eviction_lock can only be grabbed while updating the mapping range.
> >>>
> >>>    and that is precisely why it cannot be driver_svm_lock.
> >>>    driver_svm_lock must wrap find_or_insert, migration, and
> >>>    drm_gpusvm_range_get_pages
> >>>    eviction_lock is the opposite by contract:
> >>>
> >>>     - It is taken with memalloc_noreclaim_save() in
> >>>       amdgpu_vm_begin_critical(), specifically so no reclaim happens while
> >>>       held (to avoid the reclaim -> MMU-notifier deadlock). Holding it
> >>>       across get_pages/migration breaks that.
> >>>     - TTM eviction try-locks it: amdgpu_vm_evictable() does
> >>>       scoped_cond_guard(mutex_try, return false, &vm->eviction_lock) and
> >>>       sets vm->evicting. Long holds starve eviction.
> >>>     - It is a plain mutex that the SVM map path re-enters:
> >>>       amdgpu_svm_range_update_mapping() -> amdgpu_vm_map_range() ->
> >>>       amdgpu_vm_begin_critical() -> mutex_lock(&vm->eviction_lock). If
> >>>       eviction_lock were also the outer SVM lock, this is a self-deadlock.
> >>>
> >>>    In short, eviction_lock has the contract of notifier_lock , not of
> >>>    driver_svm_lock. This is also why the current split is correct:
> >>>    svm_lock (outer) != eviction_lock (inner). Your own rule - "you can't
> >>>    call the VM code with the lock held, the VM code must take it itself" -
> >>>    is satisfied today only because they are separate: svm_lock is held
> >>>    while calling amdgpu_vm_map_range(), and amdgpu_vm_map_range() takes
> >>>    eviction_lock itself. Merging them is what would violate that rule.
> >>>
> >>>> No, they Xe vm->lock and eviction_lock are actually identical in the handling.
> >>>
> >>>    They are not. Xe's vm->lock is a rw_semaphore, the "outer most lock" of
> >>>    the VM , held down_write across the whole fault. amdgpu's
> >>>    eviction_lock is a mutex taken only inside amdgpu_vm_begin_critical()
> >>>    during a PT update, under memalloc_noreclaim. Xe's eviction/reclaim
> >>>    handling is separate from vm->lock. The amdgpu analogue of Xe's vm->lock
> >>>    is svm_lock, not eviction_lock.
> >>>
> >>> C) Reusing an existing amdgpu_vm lock as the primary lock needs refactor amdgpu VM
> >>>
> >>>    Xe can register vm->lock because Xe's VM was designed with an outer
> >>>    rw_semaphore held across faults. amdgpu_vm has no such lock: only
> >>>    eviction_lock , the root PD dma_resv , and a few spinlocks.
> >>>
> >>>    So do it like Xe means introducing a dedicated, outer, sleepable VM
> >>>    lock held across the fault. That lock is exactly svm_lock. Folding it
> >>>    into struct amdgpu_vm as a general vm->lock is a core amdgpu VM refactor.
> >>>
> >>> Regards,
> >>> Honglei
> >>>
> >>>
> >>>>
> >>>> Regards,
> >>>> Christian.
> >>>>
> >>>>> +
> >>>>> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
> >>>>> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
> >>>>> +
> >>>>> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
> >>>>> +void amdgpu_svm_close(struct amdgpu_vm *vm);
> >>>>> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
> >>>>> +
> >>>>> +void amdgpu_svm_put(struct amdgpu_svm *svm);
> >>>>> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
> >>>>> +                           uint32_t pasid);
> >>>>> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
> >>>>> +                uint64_t fault_page, uint64_t ts,
> >>>>> +                bool write_fault);
> >>>>> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
> >>>>> +
> >>>>> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
> >>>>> +             struct drm_file *filp);
> >>>>> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
> >>>>> +                struct list_head *work_list);
> >>>>> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
> >>>>> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
> >>>>> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
> >>>>> +                 const struct amdgpu_svm_attrs *old_attrs,
> >>>>> +                 const struct amdgpu_svm_attrs *new_attrs,
> >>>>> +                 unsigned long start_page,
> >>>>> +                 unsigned long last_page);
> >>>>> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
> >>>>> +#else
> >>>>> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
> >>>>> +                  struct amdgpu_vm *vm)
> >>>>> +{
> >>>>> +    return 0;
> >>>>> +}
> >>>>> +
> >>>>> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
> >>>>> +{
> >>>>> +}
> >>>>> +
> >>>>> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
> >>>>> +{
> >>>>> +}
> >>>>> +
> >>>>> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
> >>>>> +                      uint32_t pasid,
> >>>>> +                      uint64_t fault_page,
> >>>>> +                      uint64_t ts,
> >>>>> +                      bool write_fault)
> >>>>> +{
> >>>>> +    return -EOPNOTSUPP;
> >>>>> +}
> >>>>> +
> >>>>> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
> >>>>> +{
> >>>>> +    return false;
> >>>>> +}
> >>>>> +
> >>>>> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
> >>>>> +                       struct drm_file *filp)
> >>>>> +{
> >>>>> +    return -EOPNOTSUPP;
> >>>>> +}
> >>>>> +#endif /* CONFIG_DRM_AMDGPU_SVM */
> >>>>> +
> >>>>> +#endif /* __AMDGPU_SVM_H__ */
> >>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >>>>> index ec1196d390bb7..30463a83e2e60 100644
> >>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> >>>>> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
> >>>>>    struct amdgpu_job;
> >>>>>    struct amdgpu_bo_list_entry;
> >>>>>    struct amdgpu_bo_vm;
> >>>>> +struct amdgpu_svm;
> >>>>>      /*
> >>>>>     * GPUVM handling
> >>>>> @@ -373,6 +374,9 @@ struct amdgpu_vm {
> >>>>>          /* cached fault info */
> >>>>>        struct amdgpu_vm_fault_info fault_info;
> >>>>> +
> >>>>> +    /* SVM experimental implementation */
> >>>>> +    struct amdgpu_svm *svm;
> >>>>>    };
> >>>>>      struct amdgpu_vm_manager {
> >>>>
> >>>
> > 
>