[PATCH v10 00/22] drm/amdgpu: AMDGPU SVM support based on DRM (Phase 1: single GPU, XNACK on)

[PATCH v10 00/22] drm/amdgpu: AMDGPU SVM support based on DRM (Phase 1: single GPU, XNACK on)

[Huang Rui]

Hi Christian, Philip, Alex, Felix, and all,

These series introduce a new Shared Virtual Memory (SVM) implementation for
amdgpu that is built directly on top of the common DRM GPUSVM / drm_pagemap
core (drivers/gpu/drm/drm_gpusvm.c, drm_pagemap.c) rather than on the
existing KFD-private SVM code in amdkfd/kfd_svm.c.

The goal is to provide HMM-based unified memory through the same shared
infrastructure that the Xe driver already uses, so that amdgpu and xe
converge on one well-reviewed SVM/page-migration core instead of
maintaining a separate amdgpu-specific stack.

Dependencies
============

This series is rebased on top of Christian's amdgpu VM rework series and
depends on it. The SVM locking work in patches 13-15 builds directly on the
eviction-lock critical section and the amdgpu_vm_update_range() split
introduced there, so this series does not apply or build without it:

  https://lore.kernel.org/amd-gfx/20260928151041.1857-1-christian.koenig@amd.com/

Phased plan
===========

This work is being upstreamed in phases to keep each submission reviewable:

  * Phase 1 (this series): single GPU, XNACK on (retry faults).
     - v9 https://lore.kernel.org/amd-gfx/20260804075417.1234567-1-ray.huang@amd.com/

  * Phase 2 (in progress): single GPU, XNACK off (no-retry / eager mapping).
     - https://lore.kernel.org/amd-gfx/20260529054928.596825-1-honglei1.huang@amd.com/
     - https://lore.kernel.org/amd-gfx/20260605075340.20199-1-Junhua.Shen@amd.com/

  * Phase 3 (in progress): multiple GPUs (XGMI mapping, P2P device-to-device migration).
     - https://lore.kernel.org/amd-gfx/20260603065620.2555316-1-honglei1.huang@amd.com

Accordingly, this first submission targets the simplest useful configuration:

 * Single GPU with local VRAM migration: transparent RAM <-> VRAM page
   migration on one GPU via SDMA, with TTM-based eviction for overcommit.
 * XNACK on (retry faults): GPU page faults are retried after the driver
   lazily populates PTEs on demand. The XNACK-off/eager-mapping path is
   deferred to Phase 2.
 * Compute VMs only (amdgpu_vm_make_compute()), matching the KFD SVM
   use case.

Everything is gated behind a new, default-n Kconfig option
(CONFIG_DRM_AMDGPU_SVM) and is therefore opt-in and isolated from existing
users until the feature matures.

UAPI
====

A new render-node ioctl, DRM_IOCTL_AMDGPU_GEM_SVM, lets userspace describe SVM
attributes over a CPU virtual-address interval (patch 1):

  - operations: SET_ATTR / GET_ATTR / RESET_ATTR
  - access modes: INACCESSIBLE / IN_PLACE / ALLOW_MIGRATE
  - location hints: SYSMEM / UNDEFINED / LOCAL / HIVE (preferred_loc,
    prefetch_loc). The location is a fixed enum rather than a device file
    descriptor on purpose: a descriptor has no stable meaning across a
    checkpoint/restore cycle.
  - per-range flags: HOST_ACCESS, COHERENT, EXT_COHERENT, GPU_RO, GPU_EXEC,
    GPU_READ_MOSTLY, plus a granularity hint

GET_ATTR additionally returns, in @size, the length of the contiguous range
starting at @start_addr over which the queried attributes are constant, so
user space (e.g. CRIU) can walk the whole address space without knowing the
range boundaries in advance.

Attributes are stored in a per-VM interval tree; the GPU page tables are
populated lazily on demand by retry faults, or eagerly on a prefetch request.

User Space Work
===============

  - libdrm exposes the new SVM ioctl:
    https://gitlab.freedesktop.org/mesa/libdrm/-/merge_requests/485

  - ROCm runtime adaptation for the new amdgpu DRM SVM API:
    https://github.com/ROCm/rocm-systems/pull/8318

Design Overview
===============

The implementation is split into clearly layered modules:

  amdgpu_svm.c            Core context (struct amdgpu_svm embeds
                          struct drm_gpusvm), kref lifecycle, PASID lookup,
                          drm_gpusvm_ops, the GEM_SVM ioctl entry point, and
                          the GC workqueue.
  amdgpu_svm_attr.c       The attribute interval tree: validation, gap/overlap
                          split-merge, SET/GET/RESET, and change-trigger
                          classification that decides whether an attribute
                          change needs PTE invalidation or a remap.
  amdgpu_svm_range.c      Per-range GPU mapping: PTE-flag computation per GC IP
                          version, DMA-segment coalescing, MMU-notifier
                          begin/end handling, PTE zapping, and the garbage
                          collector.
  amdgpu_svm_fault.c      The retry-fault entry point amdgpu_svm_handle_fault()
                          and the fault_map_range() pipeline.
  amdgpu_migrate.c        drm_pagemap / ZONE_DEVICE VRAM migration backend.
  amdgpu_svm_range_migrate.c  Per-range RAM<->VRAM migration helpers.

Fault path (XNACK on):
amdgpu_vm_handle_fault() routes a compute-VM retry fault to
amdgpu_svm_handle_fault() when the VM has an SVM context. After PASID lookup
and a checkpoint-timestamp filter that drops stale retry faults left over from
a recent unmap, it looks up (or synthesizes a default) attribute range and runs
fault_map_range(): garbage-collect -> VMA permission check -> find/insert range
-> short-circuit if recently validated or already valid -> drm_gpusvm get_pages
(HMM) -> program GPU PTEs under the notifier lock with a notifier-sequence
re-check.

Locking (patches 13-15):
Following the v9 review, amdgpu_vm's eviction lock is now a selectable
read/write semaphore (struct rw_semaphore *eviction_lock) instead of a
private mutex. On an SVM VM it is pointed at the drm_gpusvm notifier lock,
so page-table updates, MMU-notifier invalidation and TTM eviction all
serialize on one lock rather than on separate, unordered ones. Updates made
with the root PD reserved take the read side; updates without it, and
eviction, take the write side. Because page-table allocation can enter
reclaim - and reclaim re-enters the MMU notifier that wants the same lock -
the lock is dropped around amdgpu_vm_pt_create() and the range sequence
number is rechecked on reacquire, returning -EAGAIN if it moved. Plain
graphics VMs keep their own default lock and are unaffected.

Invalidation / GC:
MMU-notifier callbacks zap the affected PTEs (batched, single heavyweight TLB
flush) and, for unmap events, queue the range to a high-priority GC workqueue
that removes it via drm_gpusvm_range_remove() outside notifier context.

VRAM migration (patches 16-22):
A drm_pagemap is registered over the GPU's VRAM as a ZONE_DEVICE region at
device-init / reset-restore time. struct amdgpu_bo_svm (a new BO subtype)
backs migrated ranges, with SDMA-based copy_to_devmem / copy_to_ram callbacks
using a GART window. TTM eviction of SVM BOs synchronously migrates pages back
to system memory, which keeps VRAM overcommit working. The fault and prefetch
paths call amdgpu_svm_range_migrate_to_vram() before mapping when migration is
requested, with a single -EBUSY retry that evicts conflicting pages first.

Patch 16 ("drm/pagemap: fix page undercount for compound src entries") is a
prerequisite fix in the shared drm_pagemap core, found while enabling THP
migration on amdgpu. It is independent of this series and can be applied on
its own.

Test Results
============

Tested on gfx943 (MI300X) and gfx906 (MI60) with XNACK on:
 - KFD test: 95%+ passed.
 - ROCR test: all passed.
 - HIP catch test: gfx943 (MI300X): 99% passed. gfx906 (MI60): 99% passed.

These numbers come from running the existing KFD and ROCR test suites over
the new DRM SVM path, which is enabled by:

  https://github.com/ROCm/rocm-systems/pull/13719

IGT test suites covering amdgpu DRM SVM directly will follow.

Changes from Old Version
========================

In previous V8 version, we had consolidated all implementations for the
single-GPU XNACK-on mode. This approach followed Christian's suggestion to
make the code review process more straightforward.

Previously, the XNACK-on implementation was split into two parts:

 - Basic V7
   https://lore.kernel.org/amd-gfx/20260529054804.596214-1-honglei1.huang@amd.com/
 - Migration V5
   https://lore.kernel.org/amd-gfx/20260605075340.20199-1-Junhua.Shen@amd.com/

Above two parts are no longer needed to be reviewed separately, please
focus on this series.

Changes from V9 to V10:
 - Rebased on top of Christian's amdgpu VM rework series (see Dependencies
   above), which this series now requires. The GEM_SVM ioctl number moved
   from 0x1B to 0x1C.
 - eviction_lock: the V9 limitation is resolved by three new patches
   (13-15). amdgpu_vm's eviction lock becomes a selectable rwsem aliased
   to the drm_gpusvm notifier lock on an SVM VM, so page table updates,
   MMU notifier invalidation and eviction serialize on one lock. It is
   dropped around amdgpu_vm_pt_create() to avoid the reclaim/notifier
   recursion, with an -EAGAIN recheck on reacquire.
 - Added "drm/pagemap: fix page undercount for compound src entries" as a
   prerequisite, and made the SDMA copy path folio aware, so THP-backed
   ranges migrate correctly.
 - UAPI: the location enum gained LOCAL/HIVE and UNDEFINED changed from
   0xffffffffU to 1, replacing the AMDGPU_SVM_ATTR_HIVE_LOCAL flag.
   GET_ATTR now returns in @size the length of the range over which the
   queried attributes are constant, so CRIU can walk the address space.
 - Review fixes: the GEM_SVM ioctl no longer creates the SVM context
   lazily, the missing root BO reservation around context create/destroy
   was added, the mapping publication moved inside the notifier lock, two
   range-queueing bugs were fixed, and several error paths now report
   failure (-EPERM on a write to a read-only range, -EFAULT on a missing
   VMA, unknown attribute values, TLB flush failures) instead of being
   silently ignored.
 - Cleanups: dropped the private log wrappers in favour of pr_debug() and
   pr_err_ratelimited(), partition-scoped TLB flushing, kzalloc_obj(), and
   removed the leftover clean-svm Makefile target.

Changes from V8 to V9:
 - This series has been rebased on top of the amdgpu vm fix.
   - https://lore.kernel.org/amd-gfx/20260529114031.3714-1-christian.koenig@amd.com/
 - amdgpu_svm.h: dropped the AMDGPU_SVM_KMEM_CACHE_* macros; SVM ranges use
   plain kzalloc()/kfree().
 - amdgpu_svm.h: dropped the flush_tlb callback; amdgpu_svm_flush_tlb() is
   called directly.
 - amdgpu_svm_attr.h: added kerneldoc for the interval tree and list.
 - amdgpu_svm_attr.c: dropped the attr slab cache; uses kzalloc()/kfree().
 - amdgpu_svm_attr.c: squashed the types header into its implementation patch
 - amdgpu_svm_attr.c: renamed the local "flags" to "unsupported_vm_flags".
 - amdgpu_svm_attr.c: dropped amdgpu_svm_attr_validate_range_vma() from the
   fault path; hmm_range_fault() covers it. A lighter sync check is kept
   only at the set/get attr ioctl entry for fix regression.
 - amdgpu_svm_range.h: AMDGPU_SVM_RANGE_DEBUG removed, replaced by a tracepoint.
 - amdgpu_svm_range.h: dropped the UNMAP_WORK macro.
 - amdgpu_svm_range.c: the fast range valid check is now lock free.
 - amdgpu_svm_range.c: zap_ptes always drains the fence regardless of the
   update return value.
 - per IP PTE flags: moved the switch out of the SVM core
   into gmc_v9_0.c/gmc_v12_0.c behind a new gmc_funcs->get_svm_pte_flags()
   callback.
 - eviction_lock: drm_gpusvm framwork's limitation blocks using the
   eviction_lock kept as is.
 - SVM/BO overlap: dropped the scan-based amdgpu_svm_attr_check_vm_bo() helper.
 - Added drm_pagemap refcount-based deferred destroy for amdgpu_pagemap:
   the ZONE_DEVICE mapping is torn down from a destroy worker
   (amdgpu_pagemap_destroy_work) once the drm_pagemap refcount reaches zero,
   using drm_dev_enter()/exit() to guard against device teardown
 - Added pre_migrate_fence handling in the SDMA copy callbacks: wait for
   outstanding async VRAM operations (DMA_RESV_USAGE_KERNEL) to complete

Thanks,
Ray/Honglei/Junhua

Honglei Huang (15):
  drm/amdgpu: add SVM ioctl UAPI definitions
  drm/amdgpu: add SVM core header and VM integration
  drm/amdgpu: implement SVM attribute tree and helper functions
  drm/amdgpu: implement SVM attribute set/get/clear operations
  drm/amdgpu: add SVM range types and work queue interface
  drm/amdgpu/gmc: add get_svm_pte_flags callback
  drm/amdgpu: implement SVM range GPU mapping core
  drm/amdgpu: implement SVM range notifier and GC helpers
  drm/amdgpu: add SVM notifier invalidate callback and checkpoint
  drm/amdgpu: implement SVM initialization and lifecycle
  drm/amdgpu: add SVM ioctl entry and fault handler module
  drm/amdgpu: integrate SVM into build system and VM fault path
  drm/amdgpu: make VM eviction lock read/write to match notifier lock
  drm/amdgpu: recheck range validity after page table allocation
  drm/amdgpu: use drm_gpusvm notifier lock for VM eviction

Junhua Shen (7):
  drm/pagemap: fix page undercount for compound src entries
  drm/amdgpu: add VRAM migration infrastructure for drm_pagemap
  drm/amdgpu: implement drm_pagemap SDMA migration callbacks
  drm/amdgpu: implement synchronous TTM eviction for SVM BOs
  drm/amdgpu: hook up ZONE_DEVICE registration in device init and reset
  drm/amdgpu: add SVM range migration helpers for drm_pagemap
  drm/amdgpu: integrate VRAM migration into SVM fault and prefetch paths

 drivers/gpu/drm/amd/amdgpu/Kconfig            |  10 +
 drivers/gpu/drm/amd/amdgpu/Makefile           |   4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |   8 +
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c        |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |   4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h       |  24 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c   | 899 ++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h   | 107 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |   4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c     |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c       | 858 +++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       | 196 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c  | 977 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h  | 202 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 476 +++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h |  39 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 812 +++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 191 ++++
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 120 +++
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.h |  35 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  30 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |  20 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c    |  24 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c     |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 111 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |  19 +-
 .../gpu/drm/amd/amdgpu/amdgpu_vm_internal.h   |  30 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c     |  32 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c        |  69 ++
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c         | 119 +++
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          |   7 +-
 drivers/gpu/drm/drm_pagemap.c                 |  60 +-
 include/uapi/drm/amdgpu_drm.h                 | 127 +++
 36 files changed, 5541 insertions(+), 85 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h

-- 
2.53.0

[PATCH v10 01/22] drm/amdgpu: add SVM ioctl UAPI definitions

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Define the following ioctl structures and enums:
- DRM_AMDGPU_GEM_SVM ioctl command and DRM_IOCTL_AMDGPU_GEM_SVM macro
- enum amdgpu_ioctl_svm_op: SET_ATTR, GET_ATTR, RESET_ATTR operations
- enum amdgpu_ioctl_svm_access: INACCESSIBLE, IN_PLACE, ALLOW_MIGRATE
- enum amdgpu_ioctl_svm_location: SYSMEM, UNDEFINED, LOCAL, HIVE
- enum amdgpu_ioctl_svm_attr_type: PREFERRED_LOC, PREFETCH_LOC, ACCESS,
  GRANULARITY, HOST_ACCESS, COHERENT, EXT_COHERENT, GPU_RO,
  GPU_EXEC, GPU_READ_MOSTLY
- struct drm_amdgpu_svm_attribute: type and value pair
- struct drm_amdgpu_gem_svm: ioctl payload with start_addr, size,
  operation, nattr, and attrs_ptr

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 include/uapi/drm/amdgpu_drm.h | 127 ++++++++++++++++++++++++++++++++++
 1 file changed, 127 insertions(+)

diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
index cd1b8034a2fb4..cfe463f2114a1 100644
--- a/include/uapi/drm/amdgpu_drm.h
+++ b/include/uapi/drm/amdgpu_drm.h
@@ -60,6 +60,7 @@ extern "C" {
 #define DRM_AMDGPU_GEM_LIST_HANDLES	0x19
 #define DRM_AMDGPU_PROC_OPTIONS		0x1A
 #define DRM_AMDGPU_UALINK_HANDLE	0x1B
+#define DRM_AMDGPU_GEM_SVM		0x1C
 
 #define DRM_IOCTL_AMDGPU_GEM_CREATE	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
 #define DRM_IOCTL_AMDGPU_GEM_MMAP	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)
@@ -83,6 +84,9 @@ extern "C" {
 #define DRM_IOCTL_AMDGPU_GEM_LIST_HANDLES DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_LIST_HANDLES, struct drm_amdgpu_gem_list_handles)
 #define DRM_IOCTL_AMDGPU_PROC_OPTIONS	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_PROC_OPTIONS, struct drm_amdgpu_proc_options)
 #define DRM_IOCTL_AMDGPU_UALINK_HANDLE DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_UALINK_HANDLE, union drm_amdgpu_ualink_handle)
+#define DRM_IOCTL_AMDGPU_GEM_SVM	\
+	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_SVM, \
+		  struct drm_amdgpu_gem_svm)
 
 /**
  * DOC: memory domains
@@ -1735,6 +1739,129 @@ union drm_amdgpu_ualink_handle {
 	union drm_amdgpu_ualink_handle_out out;
 };
 
+/**
+ * enum amdgpu_ioctl_svm_op - operation selector for DRM_IOCTL_AMDGPU_GEM_SVM.
+ * @AMDGPU_SVM_OP_SET_ATTR: apply the attributes in @attrs_ptr to the VA range
+ *                          [@start_addr, @start_addr + @size).
+ * @AMDGPU_SVM_OP_GET_ATTR: query the attributes at @start_addr. The caller
+ *                          lists the attribute types it is interested in via
+ *                          @attrs_ptr; the kernel fills in their current
+ *                          values and updates @size to the length of the
+ *                          contiguous range starting at @start_addr over
+ *                          which those values are constant. Input @size is an
+ *                          upper bound. This lets user space (e.g. CRIU) walk
+ *                          the address space and read back all attributes
+ *                          without knowing the range boundaries in advance.
+ * @AMDGPU_SVM_OP_RESET_ATTR: reset all attributes for the VA range to their
+ *                            default values. @attrs_ptr and @nattr are ignored.
+ */
+enum amdgpu_ioctl_svm_op {
+	AMDGPU_SVM_OP_SET_ATTR = 0,
+	AMDGPU_SVM_OP_GET_ATTR = 1,
+	AMDGPU_SVM_OP_RESET_ATTR = 2,
+};
+
+/**
+ * enum amdgpu_ioctl_svm_access - values for AMDGPU_SVM_ATTR_ACCESS.
+ * @AMDGPU_SVM_ACCESS_INACCESSIBLE: GPU must not access the range; any access
+ *                                  is a fault.
+ * @AMDGPU_SVM_ACCESS_IN_PLACE:     GPU may access the range only at its
+ *                                  current backing store; the driver will
+ *                                  never migrate pages to local VRAM.
+ * @AMDGPU_SVM_ACCESS_ALLOW_MIGRATE: GPU may access the range and the driver
+ *                                  is allowed (but not required) to migrate
+ *                                  pages between system memory and local
+ *                                  VRAM to satisfy the preferred/prefetch
+ *                                  location.
+ */
+enum amdgpu_ioctl_svm_access {
+	AMDGPU_SVM_ACCESS_INACCESSIBLE		= 0,
+	AMDGPU_SVM_ACCESS_IN_PLACE			= 1,
+	AMDGPU_SVM_ACCESS_ALLOW_MIGRATE		= 2,
+};
+
+/**
+ * enum amdgpu_ioctl_svm_location - values for AMDGPU_SVM_ATTR_PREFERRED_LOC /
+ *                            AMDGPU_SVM_ATTR_PREFETCH_LOC.
+ * @AMDGPU_SVM_LOCATION_SYSMEM:    back the range with system memory.
+ * @AMDGPU_SVM_LOCATION_UNDEFINED: no preference; the driver chooses.
+ * @AMDGPU_SVM_LOCATION_LOCAL:     back the range with the local VRAM of the
+ *                                 device the SVM context belongs to.
+ * @AMDGPU_SVM_LOCATION_HIVE:      back the range with the VRAM of any device
+ *                                 in the same XGMI hive as the local device.
+ *
+ * The location is expressed as a fixed enum rather than a device file
+ * descriptor on purpose: a descriptor has no stable meaning across a
+ * checkpoint/restore (CRIU) cycle.
+ */
+enum amdgpu_ioctl_svm_location {
+	AMDGPU_SVM_LOCATION_SYSMEM	= 0,
+	AMDGPU_SVM_LOCATION_UNDEFINED	= 1,
+	AMDGPU_SVM_LOCATION_LOCAL	= 2,
+	AMDGPU_SVM_LOCATION_HIVE	= 3,
+};
+
+/**
+ * enum amdgpu_ioctl_svm_attr_type - attribute selector for
+ *                                   &drm_amdgpu_svm_attribute.type.
+ *
+ * @AMDGPU_SVM_ATTR_PREFERRED_LOC: Preferred backing location for the range.
+ *	Value is one of &enum amdgpu_ioctl_svm_location.
+ * @AMDGPU_SVM_ATTR_PREFETCH_LOC: Prefetch target for the range. Value is
+ *	one of &enum amdgpu_ioctl_svm_location.
+ * @AMDGPU_SVM_ATTR_ACCESS: GPU access policy for the range. Value is one
+ *	of &enum amdgpu_ioctl_svm_access.
+ * @AMDGPU_SVM_ATTR_GRANULARITY: log2 of the migration granularity in pages.
+ * @AMDGPU_SVM_ATTR_HOST_ACCESS: Guarantee host access to memory.
+ * @AMDGPU_SVM_ATTR_COHERENT: Fine-grained coherency between all devices
+ *	with access.
+ * @AMDGPU_SVM_ATTR_EXT_COHERENT: Fine-grained coherency between all devices
+ *	using device-scope atomics.
+ * @AMDGPU_SVM_ATTR_GPU_RO: GPUs only read the range, allowing replication.
+ * @AMDGPU_SVM_ATTR_GPU_EXEC: Allow execution on GPU.
+ * @AMDGPU_SVM_ATTR_GPU_READ_MOSTLY: GPUs mostly read the range; may allow
+ *	optimizations similar to GPU_RO, but writes still fault.
+ */
+enum amdgpu_ioctl_svm_attr_type {
+	AMDGPU_SVM_ATTR_PREFERRED_LOC		= 0,
+	AMDGPU_SVM_ATTR_PREFETCH_LOC		= 1,
+	AMDGPU_SVM_ATTR_ACCESS				= 2,
+	AMDGPU_SVM_ATTR_GRANULARITY			= 3,
+	/* Boolean attributes below: value must be 0 or 1. */
+	AMDGPU_SVM_ATTR_HOST_ACCESS			= 4,
+	AMDGPU_SVM_ATTR_COHERENT			= 5,
+	AMDGPU_SVM_ATTR_EXT_COHERENT		= 6,
+	AMDGPU_SVM_ATTR_GPU_RO				= 7,
+	AMDGPU_SVM_ATTR_GPU_EXEC			= 8,
+	AMDGPU_SVM_ATTR_GPU_READ_MOSTLY		= 9,
+};
+
+/* One (type, value) pair carried by DRM_IOCTL_AMDGPU_GEM_SVM. */
+struct drm_amdgpu_svm_attribute {
+	/** AMDGPU_SVM_ATTR_* */
+	__u32 type;
+	/** Attribute value; interpretation depends on @type */
+	__u32 value;
+};
+
+/* Argument for DRM_IOCTL_AMDGPU_GEM_SVM. */
+struct drm_amdgpu_gem_svm {
+	/** Start of the virtual address range */
+	__u64 start_addr;
+	/**
+	 * Size of the range in bytes. For AMDGPU_SVM_OP_GET_ATTR this is an
+	 * input upper bound and is updated on return to the length of the
+	 * contiguous range starting at @start_addr with constant attributes.
+	 */
+	__u64 size;
+	/** AMDGPU_SVM_OP_* */
+	__u32 operation;
+	/** Number of struct drm_amdgpu_svm_attribute entries in @attrs_ptr */
+	__u32 nattr;
+	/** User pointer to an array of @nattr struct drm_amdgpu_svm_attribute */
+	__u64 attrs_ptr;
+};
+
 #if defined(__cplusplus)
 }
 #endif
-- 
2.53.0

[PATCH v10 02/22] drm/amdgpu: add SVM core header and VM integration

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add amdgpu_svm.h with SVM data structures and public API:
- enum amdgpu_svm_xnack_mode: OFF, ON and DEFAULT mode
- struct amdgpu_svm: core context with drm_gpusvm, kref lifecycle,
  attribute tree, rw_semaphore, GC workqueue, xnack state.
- struct amdgpu_svm_gc: garbage collector with workqueue and work_struct
- Kmem cache helpers: AMDGPU_SVM_KMEM_CACHE_CREATE/DESTROY
- Locking helpers: amdgpu_svm_lock/unlock/assert_locked
- Public API declarations (including clean_queue, sync_work)

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 195 ++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |   4 +
 2 files changed, 199 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
new file mode 100644
index 0000000000000..72f954d2df39e
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
@@ -0,0 +1,195 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_H__
+#define __AMDGPU_SVM_H__
+
+#include <drm/amdgpu_drm.h>
+#include <drm/drm_gpusvm.h>
+#include <linux/atomic.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/printk.h>
+#include <linux/rwsem.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+struct amdgpu_device;
+struct amdgpu_vm;
+struct amdgpu_svm_attr_tree;
+struct amdgpu_svm_attrs;
+struct drm_device;
+struct drm_file;
+
+enum amdgpu_svm_xnack_mode {
+	AMDGPU_SVM_XNACK_OFF,
+	AMDGPU_SVM_XNACK_ON,
+	AMDGPU_SVM_XNACK_DEFAULT,
+};
+
+#define amdgpu_svm_assert_in_notifier(svm__) \
+	lockdep_assert_held_write(&(svm__)->gpusvm.notifier_lock)
+
+/**
+ * struct amdgpu_svm_gc - SVM range garbage collector used for unmapping
+ * SVM ranges in response to mmu_notifier events.
+ * @wq: Workqueue where @work runs on.
+ * @list: Ranges list in garbage collector. Protected by amdgpu_svm.work_lock.
+ * @work: Drains @list under amdgpu_svm.svm_lock.
+ */
+struct amdgpu_svm_gc {
+	struct workqueue_struct *wq;
+	struct list_head list;
+	struct work_struct work;
+};
+
+/**
+ * struct amdgpu_svm - shared virtual memory context
+ * @gpusvm: Embedded drm_gpusvm base, manages the range/notifier trees.
+ * @refcount: Reference count guarding the lifetime of this instance.
+ * @adev: The amdgpu device this SVM context belongs to.
+ * @vm: Back pointer to the owning amdgpu_vm.
+ * @attr_tree: Interval tree of SVM attributes.
+ * @svm_lock: The driver_svm_lock described in the GPU SVM locking section.
+ *            registered by drm_gpusvm_driver_set_lock().
+ * @work_lock: Protects the deferred work queues such as @gc.list.
+ * @gc: garbage collector for ranges queued for destruction.
+ * @exiting: Set once when the context starts tearing down.
+ * @checkpoint_ts: IH ring timestamp captured on unmap, used to drop stale
+ *                 faults.
+ * @default_granularity: Default range granularity.
+ * @xnack_enabled: Whether retry (XNACK) faults are enabled.
+ * @invalidate_ranges: Callback of the MMU notifier invalidate event.
+ */
+struct amdgpu_svm {
+	struct drm_gpusvm gpusvm;
+	struct kref refcount;
+	struct amdgpu_device *adev;
+	struct amdgpu_vm *vm;
+	struct amdgpu_svm_attr_tree *attr_tree;
+	struct rw_semaphore svm_lock;
+	spinlock_t work_lock;
+	struct amdgpu_svm_gc gc;
+	atomic_t exiting;
+	uint64_t checkpoint_ts;
+	u8 default_granularity;
+	bool xnack_enabled;
+	void (*invalidate_ranges)(struct amdgpu_svm *svm,
+				  struct drm_gpusvm_notifier *notifier,
+				  const struct mmu_notifier_range *mmu_range,
+				  struct drm_gpusvm_range *first,
+				  uint64_t adj_start, uint64_t adj_end);
+};
+
+static inline struct amdgpu_svm *to_amdgpu_svm(struct drm_gpusvm *gpusvm)
+{
+	return container_of(gpusvm, struct amdgpu_svm, gpusvm);
+}
+
+/*
+ * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
+ * Hold it in write mode around structural GPU SVM updates, including
+ * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
+ */
+static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
+{
+	down_write(&svm->svm_lock);
+}
+
+static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
+{
+	up_write(&svm->svm_lock);
+}
+
+static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
+{
+	lockdep_assert_held_write(&svm->svm_lock);
+}
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
+
+int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
+void amdgpu_svm_close(struct amdgpu_vm *vm);
+void amdgpu_svm_fini(struct amdgpu_vm *vm);
+
+void amdgpu_svm_put(struct amdgpu_svm *svm);
+struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
+					       uint32_t pasid);
+int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
+			    uint64_t fault_page, uint64_t ts,
+			    bool write_fault);
+bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
+
+int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
+			 struct drm_file *filp);
+void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
+			    struct list_head *work_list);
+void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
+int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
+int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
+				 const struct amdgpu_svm_attrs *old_attrs,
+				 const struct amdgpu_svm_attrs *new_attrs,
+				 unsigned long start_page,
+				 unsigned long last_page);
+bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
+#else
+static inline int amdgpu_svm_init(struct amdgpu_device *adev,
+				  struct amdgpu_vm *vm)
+{
+	return 0;
+}
+
+static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
+{
+}
+
+static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
+{
+}
+
+static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
+					  uint32_t pasid,
+					  uint64_t fault_page,
+					  uint64_t ts,
+					  bool write_fault)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
+{
+	return false;
+}
+
+static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
+				       struct drm_file *filp)
+{
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_DRM_AMDGPU_SVM */
+
+#endif /* __AMDGPU_SVM_H__ */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 9b07df5a2b816..b435e393be154 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -43,6 +43,7 @@ struct amdgpu_bo_va;
 struct amdgpu_job;
 struct amdgpu_bo_list_entry;
 struct amdgpu_bo_vm;
+struct amdgpu_svm;
 
 /*
  * GPUVM handling
@@ -381,6 +382,9 @@ struct amdgpu_vm {
 
 	/* cached fault info */
 	struct amdgpu_vm_fault_info fault_info;
+
+	/* SVM experimental implementation */
+	struct amdgpu_svm *svm;
 };
 
 struct amdgpu_vm_manager {
-- 
2.53.0

[PATCH v10 03/22] drm/amdgpu: implement SVM attribute tree and helper functions

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add amdgpu_svm_attr.h with the SVM attribute types and tree
infrastructure together with the helper implementation in
amdgpu_svm_attr.c that uses them. The header is squashed into the
implementation patch so it does not stand alone as declarations
without an implementation.

amdgpu_svm_attr.h:
- Internal flag bitmask definitions mapped from UAPI attr types
- PTE_FLAG_MASK and MAPPING_FLAG_MASK for change detection
- struct amdgpu_svm_attrs: user set attribute range
- struct amdgpu_svm_attr_range: interval tree node with attrs
- struct amdgpu_svm_attr_tree: mutex protected RB tree for store and
  search
- enum amdgpu_svm_attr_change_trigger: change flags of user attribute
  changes

amdgpu_svm_attr.c:
- Default attribute initialization: amdgpu_svm_attr_set_default
- Device memory and VRAM preference helpers
- VMA validity checker: amdgpu_svm_check_vma
- Attribute equality comparison: attr_equal
- Interval tree CRUD operations: find, get_bounds,
  alloc, insert, and remove
- attr_set_interval helper for range boundary updates

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 235 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h | 182 ++++++++++++++
 2 files changed, 417 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
new file mode 100644
index 0000000000000..a5afa75421fbd
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
@@ -0,0 +1,235 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/gfp.h>
+#include <linux/lockdep.h>
+#include <linux/minmax.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+struct attr_set_ctx {
+	struct amdgpu_svm_attrs old_attrs;
+	struct amdgpu_svm_attrs new_attrs;
+	unsigned long start_page;
+	unsigned long last_page;
+};
+
+struct attr_get_ctx {
+	int32_t preferred_loc;
+	int32_t prefetch_loc;
+	enum amdgpu_ioctl_svm_access access;
+	uint32_t granularity;
+	uint32_t flags_and;
+	bool has_range;
+};
+
+bool amdgpu_svm_attr_prefer_vram(const struct amdgpu_svm_attrs *attrs)
+{
+	if (attrs->preferred_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
+	    attrs->preferred_loc != AMDGPU_SVM_LOCATION_SYSMEM)
+		return true;
+
+	if (attrs->prefetch_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
+	    attrs->prefetch_loc != AMDGPU_SVM_LOCATION_SYSMEM)
+		return true;
+
+	return false;
+}
+
+struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
+					unsigned long addr)
+{
+	const unsigned long unsupported_vm_flags = VM_IO | VM_PFNMAP |
+						   VM_MIXEDMAP;
+	struct vm_area_struct *vma = vma_lookup(mm, addr);
+
+	if (!vma)
+		return ERR_PTR(-EFAULT);
+
+	if (vma->vm_flags & unsupported_vm_flags)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	return vma;
+}
+
+static void attr_set_interval(struct amdgpu_svm_attr_range *range,
+				unsigned long start_page,
+				unsigned long last_page)
+{
+	range->it_node.start = start_page;
+	range->it_node.last = last_page;
+}
+
+void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
+				 struct amdgpu_svm_attrs *attrs)
+{
+	attrs->preferred_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	attrs->prefetch_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	attrs->granularity = svm->default_granularity;
+	attrs->flags = AMDGPU_SVM_ATTR_BIT_HOST_ACCESS | AMDGPU_SVM_ATTR_BIT_COHERENT;
+	attrs->access = svm->xnack_enabled ?
+		AMDGPU_SVM_ACCESS_ALLOW_MIGRATE : AMDGPU_SVM_ACCESS_INACCESSIBLE;
+}
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
+			   unsigned long page)
+{
+	struct interval_tree_node *node;
+
+	node = interval_tree_iter_first(&attr_tree->tree, page, page);
+	if (node)
+		return container_of(node, struct amdgpu_svm_attr_range, it_node);
+
+	return NULL;
+}
+
+/**
+ * amdgpu_svm_attr_get_bounds_locked() - Find attributes  or surrounding bounds
+ * @attr_tree: attribute tree to search.
+ * @page: page index to look up.
+ * @start_page: out, first page of the returned range or gap.
+ * @last_page: out, last page of the returned range or gap.
+ *
+ * If @page is covered by an attribute range, return that range and report its
+ * bounds. Otherwise return NULL and fill @start_page/@last_page with the bounds
+ * of the default attribute gap around @page, clamped by neighboring explicit
+ * ranges or [0, ULONG_MAX] when none.
+ *
+ * Return: the covering range, or NULL if @page falls in a gap.
+ */
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
+				  unsigned long page,
+				  unsigned long *start_page,
+				  unsigned long *last_page)
+{
+	struct amdgpu_svm_attr_range *attr_range;
+	struct interval_tree_node *node;
+	struct rb_node *rb;
+
+	attr_range = amdgpu_svm_attr_find_locked(attr_tree, page);
+	if (attr_range) {
+		*start_page = amdgpu_svm_attr_start_page(attr_range);
+		*last_page = amdgpu_svm_attr_last_page(attr_range);
+		return attr_range;
+	}
+
+	*start_page = 0;
+	*last_page = ULONG_MAX;
+
+	if (page == ULONG_MAX)
+		return NULL;
+
+	node = interval_tree_iter_first(&attr_tree->tree, page + 1, ULONG_MAX);
+	if (node) {
+		if (node->start > page)
+			*last_page = node->start - 1;
+
+		rb = rb_prev(&node->rb);
+		if (rb) {
+			node = container_of(rb, struct interval_tree_node, rb);
+			if (node->last < page)
+				*start_page = node->last + 1;
+		}
+	} else {
+		rb = rb_last(&attr_tree->tree.rb_root);
+
+		if (rb) {
+			node = container_of(rb, struct interval_tree_node, rb);
+			if (node->last < page)
+				*start_page = node->last + 1;
+		}
+	}
+
+	return NULL;
+}
+
+static bool attr_equal(const struct amdgpu_svm_attrs *a,
+				 const struct amdgpu_svm_attrs *b)
+{
+	return a->flags == b->flags &&
+	       a->preferred_loc == b->preferred_loc &&
+	       a->prefetch_loc == b->prefetch_loc &&
+		       a->granularity == b->granularity &&
+		       a->access == b->access;
+}
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_range_alloc(unsigned long start_page,
+			   unsigned long last_page,
+			   const struct amdgpu_svm_attrs *attrs)
+{
+	struct amdgpu_svm_attr_range *range;
+
+	range = kzalloc_obj(*range, GFP_KERNEL);
+	if (!range)
+		return NULL;
+
+	INIT_LIST_HEAD(&range->list);
+	attr_set_interval(range, start_page, last_page);
+	range->attrs = *attrs;
+	return range;
+}
+
+void amdgpu_svm_attr_range_insert_locked(struct amdgpu_svm_attr_tree *attr_tree,
+					 struct amdgpu_svm_attr_range *range)
+{
+	struct interval_tree_node *node;
+	struct amdgpu_svm_attr_range *next;
+
+	lockdep_assert_held(&attr_tree->lock);
+
+	/*
+	 * Keep @range_list ordered by start page: insert before the first range
+	 * starting at or after @range, then add to the interval tree.
+	 */
+	node = interval_tree_iter_first(&attr_tree->tree, amdgpu_svm_attr_start_page(range),
+					ULONG_MAX);
+	if (node) {
+		next = container_of(node, struct amdgpu_svm_attr_range, it_node);
+		list_add_tail(&range->list, &next->list);
+	} else {
+		list_add_tail(&range->list, &attr_tree->range_list);
+	}
+
+	interval_tree_insert(&range->it_node, &attr_tree->tree);
+}
+
+static void attr_remove_range_locked(struct amdgpu_svm_attr_tree *attr_tree,
+					  struct amdgpu_svm_attr_range *range,
+					  bool free_range)
+{
+	lockdep_assert_held(&attr_tree->lock);
+
+	interval_tree_remove(&range->it_node, &attr_tree->tree);
+	list_del_init(&range->list);
+	if (free_range)
+		kfree(range);
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
new file mode 100644
index 0000000000000..36fee11aa9ea0
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
@@ -0,0 +1,182 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_ATTR_H__
+#define __AMDGPU_SVM_ATTR_H__
+
+#include <drm/amdgpu_drm.h>
+#include <linux/interval_tree.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/rbtree.h>
+#include <linux/types.h>
+
+/* Internal SVM attribute bitmask flags mapped from UAPI ioctl definitions */
+#define AMDGPU_SVM_ATTR_BIT_HOST_ACCESS		(1u << 0)
+#define AMDGPU_SVM_ATTR_BIT_COHERENT		(1u << 1)
+#define AMDGPU_SVM_ATTR_BIT_EXT_COHERENT	(1u << 2)
+#define AMDGPU_SVM_ATTR_BIT_GPU_RO		(1u << 3)
+#define AMDGPU_SVM_ATTR_BIT_GPU_EXEC		(1u << 4)
+#define AMDGPU_SVM_ATTR_BIT_GPU_READ_MOSTLY	(1u << 5)
+
+#define AMDGPU_SVM_PTE_FLAG_MASK \
+	(AMDGPU_SVM_ATTR_BIT_COHERENT | AMDGPU_SVM_ATTR_BIT_EXT_COHERENT | \
+	 AMDGPU_SVM_ATTR_BIT_GPU_RO | AMDGPU_SVM_ATTR_BIT_GPU_EXEC)
+
+#define AMDGPU_SVM_MAPPING_FLAG_MASK \
+	(AMDGPU_SVM_ATTR_BIT_HOST_ACCESS | \
+	 AMDGPU_SVM_ATTR_BIT_GPU_READ_MOSTLY)
+
+/**
+ * struct amdgpu_svm_attrs - SVM attributes for an address range
+ * @preferred_loc: Preferred backing location, using AMDGPU_SVM_LOCATION_*.
+ * @prefetch_loc: Target location for prefetch requests, using
+ *                AMDGPU_SVM_LOCATION_*.
+ * @flags: Internal AMDGPU_SVM_ATTR_BIT_* flags mapped from the UAPI.
+ * @granularity: Mapping granularity encoded as a page order.
+ * @access: CPU/GPU access policy from the SVM UAPI.
+ */
+struct amdgpu_svm_attrs {
+	int32_t preferred_loc;
+	int32_t prefetch_loc;
+	uint32_t flags;
+	uint32_t granularity;
+	enum amdgpu_ioctl_svm_access access;
+};
+
+/**
+ * struct amdgpu_svm_attr_range - a range of user attributes
+ * @it_node: interval tree node keyed by [start, last] page index.
+ * @list: links the range into amdgpu_svm_attr_tree.range_list in address order.
+ * @attrs: the attributes applied to this range.
+ */
+struct amdgpu_svm_attr_range {
+	struct interval_tree_node it_node;
+	struct list_head list;
+	struct amdgpu_svm_attrs attrs;
+};
+
+static inline unsigned long
+amdgpu_svm_attr_start_page(const struct amdgpu_svm_attr_range *range)
+{
+	return range->it_node.start;
+}
+
+static inline unsigned long
+amdgpu_svm_attr_last_page(const struct amdgpu_svm_attr_range *range)
+{
+	return range->it_node.last;
+}
+
+static inline unsigned long
+amdgpu_svm_attr_start(const struct amdgpu_svm_attr_range *range)
+{
+	return range->it_node.start << PAGE_SHIFT;
+}
+
+static inline unsigned long
+amdgpu_svm_attr_end(const struct amdgpu_svm_attr_range *range)
+{
+	return (range->it_node.last + 1) << PAGE_SHIFT;
+}
+
+struct amdgpu_svm;
+struct mm_struct;
+struct vm_area_struct;
+
+static inline bool
+amdgpu_svm_attr_has_access(enum amdgpu_ioctl_svm_access access)
+{
+	return access == AMDGPU_SVM_ACCESS_ALLOW_MIGRATE ||
+	       access == AMDGPU_SVM_ACCESS_IN_PLACE;
+}
+
+/**
+ * struct amdgpu_svm_attr_tree - per-SVM store of user attribute ranges
+ * @lock: protects @tree and @range_list.
+ * @tree: interval tree of struct amdgpu_svm_attr_range for fast lookup.
+ * @range_list: address-ordered list of the same ranges, kept in sync with
+ *              @tree to allow ordered traversal during modification.
+ * @svm: back pointer to the owning SVM instance.
+ */
+struct amdgpu_svm_attr_tree {
+	struct mutex lock;
+	struct rb_root_cached tree;
+	struct list_head range_list;
+	struct amdgpu_svm *svm;
+};
+
+/**
+ * enum amdgpu_svm_attr_change_trigger - effects caused by an attribute change
+ * @AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE: Access policy changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE: GPU PTE permission/cache bits changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE: Mapping policy bits changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE: Preferred or prefetch location changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE: Range granularity changed.
+ * @AMDGPU_SVM_ATTR_TRIGGER_PREFETCH: New attributes request a VRAM prefetch.
+ *
+ * Bitmask describing what an attribute update touched. It drives whether the
+ * existing GPU mappings must be invalidated and/or remapped.
+ */
+enum amdgpu_svm_attr_change_trigger {
+	AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE = (1U << 0),
+	AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE = (1U << 1),
+	AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE = (1U << 2),
+	AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE = (1U << 3),
+	AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE = (1U << 4),
+	AMDGPU_SVM_ATTR_TRIGGER_PREFETCH = (1U << 5),
+};
+
+/*
+ * Attribute changes that require invalidating existing GPU mappings.
+ * A granularity-only change does not, so it is intentionally excluded.
+ */
+#define AMDGPU_SVM_ATTR_TRIGGER_NEED_INVALIDATE \
+	(AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE | \
+	 AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE | \
+	 AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE | \
+	 AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE)
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
+			   unsigned long page);
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
+				  unsigned long page,
+				  unsigned long *start_page,
+				  unsigned long *last_page);
+void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
+				 struct amdgpu_svm_attrs *attrs);
+
+struct amdgpu_svm_attr_range *
+amdgpu_svm_attr_range_alloc(unsigned long start_page,
+			   unsigned long last_page,
+			   const struct amdgpu_svm_attrs *attrs);
+void amdgpu_svm_attr_range_insert_locked(struct amdgpu_svm_attr_tree *attr_tree,
+					 struct amdgpu_svm_attr_range *range);
+bool amdgpu_svm_attr_prefer_vram(const struct amdgpu_svm_attrs *attrs);
+struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
+					unsigned long addr);
+
+#endif /* __AMDGPU_SVM_ATTR_H__ */
-- 
2.53.0

[PATCH v10 04/22] drm/amdgpu: implement SVM attribute set/get/clear operations

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Complete attribute subsystem with validation, tree modification, and
public API:
- Attribute validation: per-type checks for preferred_loc, prefetch_loc,
  access, flags, granularity
- Flag application: attr_apply_flag, amdgpu_svm_attr_apply
- VMA range validation: amdgpu_svm_attr_validate_range_vma
- VM BO overlap detection: amdgpu_svm_attr_check_vm_bo
- Tree modification with gap/overlap handling:
  amdgpu_svm_attr_set_hole (new range in gap),
  amdgpu_svm_attr_set_existing (split/update existing range)
- amdgpu_svm_attr_set_range: cursor based iteration with per segment
- Lifecycle: amdgpu_svm_attr_tree_create/destroy
- Public API: amdgpu_svm_attr_set (validate, set_range with retry),
  amdgpu_svm_attr_get (aggregate attrs across interval),
  amdgpu_svm_attr_clear (interval removal with split),
  amdgpu_svm_attr_reset

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 736 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h |  20 +
 2 files changed, 756 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
index a5afa75421fbd..c76e10473968e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
@@ -78,6 +78,43 @@ struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
 	return vma;
 }
 
+/*
+ * amdgpu_svm_attr_range_is_valid - validate an attr ioctl address range
+ *
+ * Synchronous input validation for the SVM set/get-attr ioctls: walk the
+ * CPU VMAs covering [start_page, last_page] and reject the request if any
+ * part is unmapped or backed by an unsupported VMA (-EOPNOTSUPP).
+ *
+ */
+static int amdgpu_svm_attr_range_is_valid(struct amdgpu_svm *svm,
+					  unsigned long start_page,
+					  unsigned long last_page)
+{
+	struct mm_struct *mm = svm->gpusvm.mm;
+	struct vm_area_struct *vma;
+	unsigned long start, end;
+	int r = 0;
+
+	if (!mm)
+		return -EFAULT;
+
+	start = start_page << PAGE_SHIFT;
+	end = (last_page + 1) << PAGE_SHIFT;
+
+	mmap_read_lock(mm);
+	do {
+		vma = amdgpu_svm_check_vma(mm, start);
+		if (IS_ERR(vma)) {
+			r = PTR_ERR(vma);
+			break;
+		}
+		start = min(end, vma->vm_end);
+	} while (start < end);
+	mmap_read_unlock(mm);
+
+	return r;
+}
+
 static void attr_set_interval(struct amdgpu_svm_attr_range *range,
 				unsigned long start_page,
 				unsigned long last_page)
@@ -233,3 +270,702 @@ static void attr_remove_range_locked(struct amdgpu_svm_attr_tree *attr_tree,
 	if (free_range)
 		kfree(range);
 }
+
+static void attr_change_ctx_set(
+		struct attr_set_ctx *change,
+		const struct amdgpu_svm_attrs *old_attrs,
+		const struct amdgpu_svm_attrs *new_attrs,
+		unsigned long start_page,
+		unsigned long last_page)
+{
+	change->old_attrs = *old_attrs;
+	change->new_attrs = *new_attrs;
+	change->start_page = start_page;
+	change->last_page = last_page;
+}
+
+static inline int attr_check_preferred_loc(uint32_t value)
+{
+	/*
+	 * SYSMEM and UNDEFINED are always valid; LOCAL/HIVE select device
+	 * VRAM (see enum amdgpu_ioctl_svm_location).
+	 */
+	switch (value) {
+	case AMDGPU_SVM_LOCATION_SYSMEM:
+	case AMDGPU_SVM_LOCATION_UNDEFINED:
+	case AMDGPU_SVM_LOCATION_LOCAL:
+	case AMDGPU_SVM_LOCATION_HIVE:
+		return 0;
+	default:
+		return -EINVAL;
+	}
+}
+
+static inline int attr_check_prefetch_loc(uint32_t value)
+{
+	/*
+	 * Prefetch accepts the same values as the preferred location (see
+	 * enum amdgpu_ioctl_svm_location); UNDEFINED means no prefetch and is
+	 * what GET_ATTR reports for default or mixed ranges, so it must round
+	 * trip through SET for CRIU restore.
+	 */
+	switch (value) {
+	case AMDGPU_SVM_LOCATION_SYSMEM:
+	case AMDGPU_SVM_LOCATION_UNDEFINED:
+	case AMDGPU_SVM_LOCATION_LOCAL:
+	case AMDGPU_SVM_LOCATION_HIVE:
+		return 0;
+	default:
+		return -EINVAL;
+	}
+}
+
+static inline int attr_check_access(uint32_t value)
+{
+	switch (value) {
+	case AMDGPU_SVM_ACCESS_INACCESSIBLE:
+	case AMDGPU_SVM_ACCESS_IN_PLACE:
+	case AMDGPU_SVM_ACCESS_ALLOW_MIGRATE:
+		return 0;
+	default:
+		return -EINVAL;
+	}
+}
+
+static bool attr_flag_type_to_bit(uint32_t type, uint32_t *flag_bit)
+{
+	if (type < AMDGPU_SVM_ATTR_HOST_ACCESS ||
+	    type > AMDGPU_SVM_ATTR_GPU_READ_MOSTLY)
+		return false;
+
+	*flag_bit = 1u << (type - AMDGPU_SVM_ATTR_HOST_ACCESS);
+	return true;
+}
+
+static inline int attr_check_flag_value(uint32_t value)
+{
+	if (value > 1)
+		return -EINVAL;
+
+	return 0;
+}
+
+static inline int attr_check_flag_attr(uint32_t type, uint32_t value)
+{
+	uint32_t flag_bit;
+	int ret;
+
+	if (!attr_flag_type_to_bit(type, &flag_bit))
+		return -EINVAL;
+
+	ret = attr_check_flag_value(value);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static inline int attr_check_granularity(uint32_t value)
+{
+	return 0;
+}
+
+static int attr_set_validate(const struct drm_amdgpu_svm_attribute *attr)
+{
+	switch (attr->type) {
+	case AMDGPU_SVM_ATTR_PREFERRED_LOC:
+		return attr_check_preferred_loc(attr->value);
+	case AMDGPU_SVM_ATTR_PREFETCH_LOC:
+		return attr_check_prefetch_loc(attr->value);
+	case AMDGPU_SVM_ATTR_ACCESS:
+		return attr_check_access(attr->value);
+	case AMDGPU_SVM_ATTR_GRANULARITY:
+		return attr_check_granularity(attr->value);
+	case AMDGPU_SVM_ATTR_HOST_ACCESS:
+	case AMDGPU_SVM_ATTR_COHERENT:
+	case AMDGPU_SVM_ATTR_GPU_RO:
+	case AMDGPU_SVM_ATTR_GPU_EXEC:
+	case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
+	case AMDGPU_SVM_ATTR_EXT_COHERENT:
+		return attr_check_flag_attr(attr->type, attr->value);
+	default:
+		return -EINVAL;
+	}
+}
+
+static void attr_apply_flag(struct amdgpu_svm_attrs *attrs,
+			    uint32_t type, uint32_t value)
+{
+	uint32_t flag_bit;
+
+	if (!attr_flag_type_to_bit(type, &flag_bit))
+		return;
+
+	if (value)
+		attrs->flags |= flag_bit;
+	else
+		attrs->flags &= ~flag_bit;
+}
+
+static void attr_apply(struct amdgpu_svm_attrs *attrs,
+					uint32_t nattr,
+					const struct drm_amdgpu_svm_attribute *pattrs)
+{
+	const struct drm_amdgpu_svm_attribute *attr;
+
+	for (attr = pattrs; nattr--; attr++) {
+		switch (attr->type) {
+		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
+			attrs->preferred_loc = (int32_t)attr->value;
+			break;
+		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
+			attrs->prefetch_loc = (int32_t)attr->value;
+			break;
+		case AMDGPU_SVM_ATTR_ACCESS:
+			attrs->access = (enum amdgpu_ioctl_svm_access)attr->value;
+			break;
+		case AMDGPU_SVM_ATTR_HOST_ACCESS:
+		case AMDGPU_SVM_ATTR_COHERENT:
+		case AMDGPU_SVM_ATTR_GPU_RO:
+		case AMDGPU_SVM_ATTR_GPU_EXEC:
+		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
+		case AMDGPU_SVM_ATTR_EXT_COHERENT:
+			attr_apply_flag(attrs, attr->type, attr->value);
+			break;
+		case AMDGPU_SVM_ATTR_GRANULARITY:
+			attrs->granularity = min_t(uint32_t, attr->value, 0x3f);
+			break;
+		default:
+			break;
+		}
+	}
+}
+
+static bool attr_same_attrs(const struct amdgpu_svm_attr_range *range,
+			    uint32_t nattr,
+			    const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm_attrs target;
+
+	target = range->attrs;
+	attr_apply(&target, nattr, attrs);
+	return attr_equal(&range->attrs, &target);
+}
+
+static int
+amdgpu_svm_attr_set_hole(struct amdgpu_svm_attr_tree *attr_tree,
+			  const struct amdgpu_svm_attrs *default_attrs,
+			  unsigned long start_page, unsigned long last_page,
+			  uint32_t nattr,
+			  const struct drm_amdgpu_svm_attribute *attrs,
+			  struct attr_set_ctx *change)
+{
+	struct amdgpu_svm_attrs new_attrs;
+	struct amdgpu_svm_attr_range *range;
+
+	lockdep_assert_held(&attr_tree->lock);
+
+	if (start_page > last_page)
+		return 0;
+
+	new_attrs = *default_attrs;
+	attr_apply(&new_attrs, nattr, attrs);
+
+	/* Always create a range entry even when attrs equal defaults */
+	range = amdgpu_svm_attr_range_alloc(start_page, last_page, &new_attrs);
+	if (!range)
+		return -ENOMEM;
+
+	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+
+	attr_change_ctx_set(change, default_attrs,
+				       &new_attrs, start_page, last_page);
+	return 0;
+}
+
+/*
+ * amdgpu_svm_attr_set_existing() - Apply attrs to part of an existing range
+ *
+ * Update @range over [@start_page, @last_page]. If the request fully covers
+ * @range the attrs are updated in place, otherwise the range is split into the
+ * head/tail, carrying the old attrs, and the covered middle, carrying
+ * the new attrs. The resulting change is reported by @change.
+ */
+static int
+amdgpu_svm_attr_set_existing(struct amdgpu_svm_attr_tree *attr_tree,
+			     struct amdgpu_svm_attr_range *range,
+			     unsigned long start_page, unsigned long last_page,
+			     uint32_t nattr,
+			     const struct drm_amdgpu_svm_attribute *attrs,
+			     struct attr_set_ctx *change)
+{
+	unsigned long range_start = amdgpu_svm_attr_start_page(range);
+	unsigned long range_last = amdgpu_svm_attr_last_page(range);
+	struct amdgpu_svm_attr_range *left = NULL;
+	struct amdgpu_svm_attr_range *right = NULL;
+	struct amdgpu_svm_attrs old_attrs;
+	struct amdgpu_svm_attrs new_attrs;
+
+	lockdep_assert_held(&attr_tree->lock);
+
+	old_attrs = range->attrs;
+
+	if (attr_same_attrs(range, nattr, attrs)) {
+		/* Report old==new so apply_attr_change can decide */
+		attr_change_ctx_set(change, &old_attrs,
+					       &old_attrs,
+					       start_page, last_page);
+		return 0;
+	}
+
+	new_attrs = old_attrs;
+	attr_apply(&new_attrs, nattr, attrs);
+
+	/* only need to update attr */
+	if (start_page == range_start && last_page == range_last) {
+		range->attrs = new_attrs;
+		attr_change_ctx_set(change, &old_attrs,
+					       &new_attrs, start_page, last_page);
+		return 0;
+	}
+
+	/* split head */
+	if (start_page > range_start) {
+		left = amdgpu_svm_attr_range_alloc(range_start, start_page - 1, &old_attrs);
+		if (!left)
+			return -ENOMEM;
+	}
+
+	/* split tail */
+	if (last_page < range_last) {
+		right = amdgpu_svm_attr_range_alloc(last_page + 1, range_last, &old_attrs);
+		if (!right) {
+			kfree(left);
+			return -ENOMEM;
+		}
+	}
+
+	attr_remove_range_locked(attr_tree, range, false);
+	if (left)
+		amdgpu_svm_attr_range_insert_locked(attr_tree, left);
+	attr_set_interval(range, start_page, last_page);
+	range->attrs = new_attrs;
+	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+	if (right)
+		amdgpu_svm_attr_range_insert_locked(attr_tree, right);
+
+	attr_change_ctx_set(change, &old_attrs,
+				       &new_attrs, start_page, last_page);
+	return 0;
+}
+
+/*
+ * amdgpu_svm_attr_set_range() - Apply attr changes across [@start_page, @last_page]
+ *
+ * Walk the interval in segments, each segment being either an existing range
+ * or a hole.
+ */
+static int
+amdgpu_svm_attr_set_range(struct amdgpu_svm_attr_tree *attr_tree,
+			  const struct amdgpu_svm_attrs *default_attrs,
+			  unsigned long start_page, unsigned long last_page,
+			  uint32_t nattr,
+			  const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm *svm = attr_tree->svm;
+	unsigned long cursor = start_page;
+	bool need_retry = false;
+
+	while (cursor <= last_page) {
+		struct interval_tree_node *node;
+		unsigned long seg_last;
+		struct attr_set_ctx change = { 0 };
+		int ret;
+
+		amdgpu_svm_lock(svm);
+		mutex_lock(&attr_tree->lock);
+		node = interval_tree_iter_first(&attr_tree->tree, cursor, cursor);
+		if (node) {
+			struct amdgpu_svm_attr_range *range;
+
+			range = container_of(node, struct amdgpu_svm_attr_range, it_node);
+			seg_last = min(last_page, amdgpu_svm_attr_last_page(range));
+			ret = amdgpu_svm_attr_set_existing(attr_tree, range,
+								   cursor, seg_last,
+								   nattr, attrs, &change);
+		} else {
+			struct interval_tree_node *next;
+
+			seg_last = last_page;
+			if (cursor != ULONG_MAX) {
+				next = interval_tree_iter_first(&attr_tree->tree,
+								cursor + 1,
+								ULONG_MAX);
+				if (next) {
+					struct amdgpu_svm_attr_range *next_range;
+
+					next_range = container_of(next,
+						struct amdgpu_svm_attr_range,
+						it_node);
+					seg_last = min(last_page,
+						       amdgpu_svm_attr_start_page(next_range) - 1);
+				}
+			}
+			ret = amdgpu_svm_attr_set_hole(attr_tree,
+							       default_attrs,
+							       cursor, seg_last,
+							       nattr, attrs,
+							       &change);
+		}
+		mutex_unlock(&attr_tree->lock);
+
+		if (ret) {
+			amdgpu_svm_unlock(svm);
+			return ret;
+		}
+
+		ret = amdgpu_svm_apply_attr_change(svm,
+						   &change.old_attrs,
+						   &change.new_attrs,
+						   change.start_page,
+						   change.last_page);
+		amdgpu_svm_unlock(svm);
+
+		if (ret == -EAGAIN) {
+			need_retry = true;
+			ret = 0;
+		}
+
+		if (ret)
+			return ret;
+
+		if (seg_last == ULONG_MAX || seg_last == last_page)
+			break;
+
+		cursor = seg_last + 1;
+	}
+
+	return need_retry ? -EAGAIN : 0;
+}
+
+struct amdgpu_svm_attr_tree *
+amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm)
+{
+	struct amdgpu_svm_attr_tree *attr_tree;
+
+	attr_tree = kzalloc_obj(*attr_tree, GFP_KERNEL);
+	if (!attr_tree)
+		return NULL;
+
+	mutex_init(&attr_tree->lock);
+	attr_tree->tree = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&attr_tree->range_list);
+	attr_tree->svm = svm;
+	return attr_tree;
+}
+
+void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree)
+{
+	struct amdgpu_svm_attr_range *range, *tmp;
+
+	if (!attr_tree)
+		return;
+
+	mutex_lock(&attr_tree->lock);
+	list_for_each_entry_safe(range, tmp, &attr_tree->range_list, list) {
+		interval_tree_remove(&range->it_node, &attr_tree->tree);
+		list_del_init(&range->list);
+		kfree(range);
+	}
+	mutex_unlock(&attr_tree->lock);
+
+	mutex_destroy(&attr_tree->lock);
+	kfree(attr_tree);
+}
+
+int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
+			uint64_t start,
+			uint64_t size,
+			uint32_t nattr,
+			const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm *svm = attr_tree->svm;
+	struct amdgpu_svm_attrs default_attrs;
+	unsigned long start_page, last_page;
+	uint32_t i;
+	int r;
+
+	start_page = start >> PAGE_SHIFT;
+	last_page = (start + size - 1) >> PAGE_SHIFT;
+
+	for (i = 0; i < nattr; i++) {
+		pr_debug("set attr type %u value 0x%08x range [0x%lx-0x%lx] xnack:%d",
+			 attrs[i].type, attrs[i].value, start_page, last_page,
+			 svm->xnack_enabled ? 1 : 0);
+		r = attr_set_validate(&attrs[i]);
+		if (r) {
+			pr_debug("invalid attribute %u value 0x%08x",
+					 attrs[i].type, attrs[i].value);
+			return r;
+		}
+	}
+
+	r = amdgpu_svm_attr_range_is_valid(svm, start_page, last_page);
+	if (r)
+		return r;
+
+	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
+
+retry:
+	r = amdgpu_svm_attr_set_range(attr_tree, &default_attrs,
+					       start_page, last_page,
+					       nattr, attrs);
+	if (r == -EAGAIN) {
+		pr_debug("attr_set retry [0x%lx-0x%lx]\n",
+				 start_page, last_page);
+		amdgpu_svm_sync_work(svm);
+		cond_resched();
+		goto retry;
+	}
+
+	return r;
+}
+
+int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page)
+{
+	struct interval_tree_node *node;
+	int r = 0;
+
+	if (start_page > last_page)
+		return -EINVAL;
+
+	mutex_lock(&attr_tree->lock);
+
+	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
+	while (node) {
+		struct interval_tree_node *next;
+		struct amdgpu_svm_attr_range *range;
+		unsigned long range_start;
+		unsigned long range_last;
+
+		range = container_of(node, struct amdgpu_svm_attr_range, it_node);
+		next = interval_tree_iter_next(node, start_page, last_page);
+		range_start = amdgpu_svm_attr_start_page(range);
+		range_last = amdgpu_svm_attr_last_page(range);
+
+		if (range_start < start_page && range_last > last_page) {
+			struct amdgpu_svm_attr_range *tail;
+
+			tail = amdgpu_svm_attr_range_alloc(last_page + 1, range_last,
+							   &range->attrs);
+			if (!tail) {
+				r = -ENOMEM;
+				break;
+			}
+
+			attr_remove_range_locked(attr_tree, range, false);
+			attr_set_interval(range, range_start, start_page - 1);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, tail);
+		} else if (range_start < start_page) {
+			attr_remove_range_locked(attr_tree, range, false);
+			attr_set_interval(range, range_start, start_page - 1);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+		} else if (range_last > last_page) {
+			attr_remove_range_locked(attr_tree, range, false);
+			attr_set_interval(range, last_page + 1, range_last);
+			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+		} else {
+			attr_remove_range_locked(attr_tree, range, true);
+		}
+
+		node = next;
+	}
+
+	mutex_unlock(&attr_tree->lock);
+	return r;
+}
+
+int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page)
+{
+	/*
+	 * Range with no attr node in this implementation is treated the
+	 * same as one that explicitly stores default attrs. So clear
+	 * the attribute ranges when user wants to reset the attrs.
+	 *
+	 * - GET_ATTR: return default_attrs to userspace when there is no attr
+	 *   range.
+	 * - SET_ATTR: holes are treated as having default_attrs when
+	 *   computing the change trigger.
+	 * - Fault: attr lookup falls back to defaults when no attr range exists,
+	 *   so migration and PTE flag decisions are unchanged.
+	 *
+	 * This approach simplifies the implementation and avoids redundant
+	 * attribute maintenance. This ioctl operation is for attribute only,
+	 * so do not invalidate the GPU mapping here.
+	 */
+	return amdgpu_svm_attr_clear(attr_tree, start_page, last_page);
+}
+
+static void attr_get_ctx_add(struct attr_get_ctx *ctx,
+			       const struct amdgpu_svm_attrs *attrs)
+{
+	if (!ctx->has_range) {
+		ctx->preferred_loc = attrs->preferred_loc;
+		ctx->prefetch_loc = attrs->prefetch_loc;
+		ctx->granularity = attrs->granularity;
+		ctx->access = attrs->access;
+		ctx->flags_and = attrs->flags;
+		ctx->has_range = true;
+		return;
+	}
+
+	if (ctx->preferred_loc != attrs->preferred_loc)
+		ctx->preferred_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	if (ctx->prefetch_loc != attrs->prefetch_loc)
+		ctx->prefetch_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
+	if (attrs->granularity < ctx->granularity)
+		ctx->granularity = attrs->granularity;
+	if (ctx->access != attrs->access)
+		ctx->access = AMDGPU_SVM_ACCESS_INACCESSIBLE;
+	ctx->flags_and &= attrs->flags;
+}
+
+static int attr_get_ctx_to_result(const struct attr_get_ctx *ctx,
+				uint32_t nattr,
+				struct drm_amdgpu_svm_attribute *attrs)
+{
+	uint32_t i;
+
+	for (i = 0; i < nattr; i++) {
+		switch (attrs[i].type) {
+		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
+			attrs[i].value = ctx->preferred_loc;
+			break;
+		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
+			attrs[i].value = ctx->prefetch_loc;
+			break;
+		case AMDGPU_SVM_ATTR_ACCESS:
+			attrs[i].value = (uint32_t)ctx->access;
+			break;
+		case AMDGPU_SVM_ATTR_HOST_ACCESS:
+		case AMDGPU_SVM_ATTR_COHERENT:
+		case AMDGPU_SVM_ATTR_GPU_RO:
+		case AMDGPU_SVM_ATTR_GPU_EXEC:
+		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
+		case AMDGPU_SVM_ATTR_EXT_COHERENT: {
+			uint32_t flag_bit;
+
+			if (!attr_flag_type_to_bit(attrs[i].type, &flag_bit))
+				return -EINVAL;
+
+			attrs[i].value = (ctx->flags_and & flag_bit) ? 1 : 0;
+			break;
+		}
+		case AMDGPU_SVM_ATTR_GRANULARITY:
+			attrs[i].value = ctx->granularity;
+			break;
+		default:
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+/*
+ * On return @out_size is shrunk to the length of the contiguous range starting
+ * at @start over which all attributes are identical, so user space (e.g. CRIU)
+ * can walk the address space one uniform range at a time.
+ */
+int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
+			uint64_t start, uint64_t size,
+			uint32_t nattr,
+			struct drm_amdgpu_svm_attribute *attrs,
+			uint64_t *out_size)
+{
+	struct amdgpu_svm_attrs default_attrs;
+	struct amdgpu_svm_attrs ref_attrs;
+	struct attr_get_ctx ctx = { 0 };
+	struct interval_tree_node *node;
+	unsigned long start_page, last_page, cursor, homog_last;
+	bool have_ref = false;
+	int r;
+
+	start_page = start >> PAGE_SHIFT;
+	last_page = (start + size - 1) >> PAGE_SHIFT;
+
+	r = amdgpu_svm_attr_range_is_valid(attr_tree->svm, start_page, last_page);
+	if (r)
+		return r;
+
+	mutex_lock(&attr_tree->lock);
+	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
+
+	homog_last = start_page;
+	cursor = start_page;
+	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
+
+	while (cursor <= last_page) {
+		const struct amdgpu_svm_attrs *range_attrs;
+		unsigned long range_last = last_page;
+		struct amdgpu_svm_attr_range *range = NULL;
+		unsigned long next;
+
+		if (node) {
+			range = container_of(node, struct amdgpu_svm_attr_range,
+					     it_node);
+
+			if (amdgpu_svm_attr_last_page(range) < cursor) {
+				node = interval_tree_iter_next(node, start_page,
+							       last_page);
+				continue;
+			}
+
+			if (amdgpu_svm_attr_start_page(range) <= cursor) {
+				range_last = min(last_page, amdgpu_svm_attr_last_page(range));
+				node = interval_tree_iter_next(node, start_page,
+							       last_page);
+			} else {
+				range_last = min(last_page,
+						 amdgpu_svm_attr_start_page(range) - 1);
+				range = NULL;
+			}
+		}
+
+		range_attrs = range ? &range->attrs : &default_attrs;
+
+		if (!have_ref) {
+			ref_attrs = *range_attrs;
+			have_ref = true;
+		} else if (!attr_equal(range_attrs, &ref_attrs)) {
+			break;
+		}
+
+		homog_last = range_last;
+
+		if (range_last == ULONG_MAX)
+			break;
+
+		next = range_last + 1;
+		if (next <= cursor)
+			break;
+		cursor = next;
+	}
+
+	if (!have_ref)
+		ref_attrs = default_attrs;
+
+	attr_get_ctx_add(&ctx, &ref_attrs);
+	r = attr_get_ctx_to_result(&ctx, nattr, attrs);
+	mutex_unlock(&attr_tree->lock);
+
+	if (!r && out_size)
+		*out_size = ((uint64_t)(homog_last - start_page) + 1) << PAGE_SHIFT;
+
+	return r;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
index 36fee11aa9ea0..56bcd81c608cb 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
@@ -158,6 +158,9 @@ enum amdgpu_svm_attr_change_trigger {
 	 AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE | \
 	 AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE)
 
+struct amdgpu_svm_attr_tree *
+amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm);
+void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree);
 struct amdgpu_svm_attr_range *
 amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
 			   unsigned long page);
@@ -169,6 +172,23 @@ amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
 void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
 				 struct amdgpu_svm_attrs *attrs);
 
+int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
+			   uint64_t start,
+			   uint64_t size,
+			   uint32_t nattr,
+			   const struct drm_amdgpu_svm_attribute *attrs);
+int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
+				       uint64_t start,
+				       uint64_t size,
+				       uint32_t nattr,
+				       struct drm_amdgpu_svm_attribute *attrs,
+				       uint64_t *out_size);
+int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page);
+int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
+			  unsigned long start_page,
+			  unsigned long last_page);
 struct amdgpu_svm_attr_range *
 amdgpu_svm_attr_range_alloc(unsigned long start_page,
 			   unsigned long last_page,
-- 
2.53.0

[PATCH v10 05/22] drm/amdgpu: add SVM range types and work queue interface

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add amdgpu_svm_range.h with GPU mapped range types and interfaces:
- enum amdgpu_svm_range_queue_state: NOT_QUEUED, IN_GC, PROCESSING
  states for queue work
- struct amdgpu_svm_range: extends drm_gpusvm_range with gpu_mapped
  state, queue_state, attribute flags, work queue node, pending
  ops/pages, and validation timestamp
- enum amdgpu_svm_range_op: NONE, UNMAP operation types
- struct amdgpu_svm_range_op_ctx: dequeue context for GC processing
- Inline helper: amdgpu_svm_range_invalidate_gpu_mapping
- Range operations API declarations: find_or_insert, get_pages,
  update_mapping, update_gpu_range, invalidate

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 129 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  30 ++++
 2 files changed, 159 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
new file mode 100644
index 0000000000000..39db3a18b5f2f
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -0,0 +1,129 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_RANGE_H__
+#define __AMDGPU_SVM_RANGE_H__
+
+#include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
+
+#include "amdgpu_svm.h"
+#include "amdgpu_vm.h"
+
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/types.h>
+
+struct amdgpu_svm;
+struct amdgpu_svm_attr_range;
+struct amdgpu_svm_attrs;
+struct dma_fence;
+struct drm_exec;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_range;
+struct mmu_notifier_range;
+
+/**
+ * enum amdgpu_svm_range_queue_state - Work queue state of an SVM range
+ * @AMDGPU_SVM_RANGE_NOT_QUEUED: Not on any work list.
+ * @AMDGPU_SVM_RANGE_IN_GC: Queued on the GC list.
+ * @AMDGPU_SVM_RANGE_PROCESSING: Being processed in workqueue.
+ */
+enum amdgpu_svm_range_queue_state {
+	AMDGPU_SVM_RANGE_NOT_QUEUED = 0,
+	AMDGPU_SVM_RANGE_IN_GC,
+	AMDGPU_SVM_RANGE_PROCESSING,
+};
+
+/**
+ * struct amdgpu_svm_range - amdgpu GPU mapped SVM range
+ * @base: Embedded drm_gpusvm_range base.
+ * @work_node: Links the range into the work list. Protected by amdgpu_svm.work_lock.
+ * @gpu_mapped: True while the range has a live GPU mapping. Accessed with
+ *              READ_ONCE/WRITE_ONCE.
+ * @queue_state: Current work queue state @amdgpu_svm_range_queue_state.
+ *               Protected by amdgpu_svm.work_lock.
+ * @pending_ops: Pending amdgpu_svm_range_op bitmask. Protected by work_lock.
+ * @pending_start_page: First page of the accumulated pending interval.
+ * @pending_last_page: Last page of the accumulated pending interval.
+ * @attr_flags: Attribute flags last applied to the GPU mapping.
+ * @validate_timestamp: Time of the last successful mapping validation.
+ */
+struct amdgpu_svm_range {
+	struct drm_gpusvm_range base;
+	struct list_head work_node;
+	bool gpu_mapped;
+	u8 queue_state;
+	u8 pending_ops;
+	unsigned long pending_start_page;
+	unsigned long pending_last_page;
+	uint32_t attr_flags;
+	ktime_t validate_timestamp;
+};
+
+static inline struct amdgpu_svm_range *
+to_amdgpu_svm_range(struct drm_gpusvm_range *range)
+{
+	return container_of(range, struct amdgpu_svm_range, base);
+}
+
+static inline void
+amdgpu_svm_range_invalidate_gpu_mapping(struct amdgpu_svm_range *range)
+{
+	WRITE_ONCE(range->gpu_mapped, false);
+}
+
+#define AMDGPU_SVM_RANGE_TRACE(r__, op__)				    \
+	trace_amdgpu_svm_range((op__),					    \
+		to_amdgpu_svm((r__)->base.gpusvm)->vm->pasid,		    \
+		(r__)->base.gpusvm, READ_ONCE((r__)->gpu_mapped),	    \
+		(r__)->base.pages.notifier_seq,				    \
+		drm_gpusvm_range_start(&(r__)->base) >> PAGE_SHIFT,	    \
+		drm_gpusvm_range_end(&(r__)->base) >> PAGE_SHIFT)
+
+/**
+ * enum amdgpu_svm_range_op - Pending operation bits for an SVM range
+ * @AMDGPU_SVM_RANGE_OP_NONE: No pending operation.
+ * @AMDGPU_SVM_RANGE_OP_UNMAP: Range needs to be unmapped/removed by the GC.
+ */
+enum amdgpu_svm_range_op {
+	AMDGPU_SVM_RANGE_OP_NONE    = 0,
+	AMDGPU_SVM_RANGE_OP_UNMAP   = BIT(0),
+};
+
+/**
+ * struct amdgpu_svm_range_op_ctx - context of a range workqueue operation
+ * @range: The dequeued range to process.
+ * @start_page: First page for workqueue operation.
+ * @last_page: Last page for workqueue operation.
+ * @pending_ops: amdgpu_svm_range_op bitmask.
+ */
+struct amdgpu_svm_range_op_ctx {
+	struct amdgpu_svm_range *range;
+	unsigned long start_page;
+	unsigned long last_page;
+	uint8_t pending_ops;
+};
+
+#endif /* __AMDGPU_SVM_RANGE_H__ */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
index 599c1d0e512b5..86ba09cba34c9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
@@ -116,6 +116,36 @@ TRACE_EVENT(amdgpu_iv,
 		      __entry->src_data[2], __entry->src_data[3])
 );
 
+TRACE_EVENT(amdgpu_svm_range,
+	    TP_PROTO(const char *op, u32 pasid, void *gpusvm, bool mapped,
+		     unsigned long seqno, unsigned long start_page,
+		     unsigned long end_page),
+	    TP_ARGS(op, pasid, gpusvm, mapped, seqno, start_page, end_page),
+	    TP_STRUCT__entry(
+			     __string(op, op)
+			     __field(u32, pasid)
+			     __field(void *, gpusvm)
+			     __field(bool, mapped)
+			     __field(unsigned long, seqno)
+			     __field(unsigned long, start_page)
+			     __field(unsigned long, end_page)
+			    ),
+	    TP_fast_assign(
+			   __assign_str(op);
+			   __entry->pasid = pasid;
+			   __entry->gpusvm = gpusvm;
+			   __entry->mapped = mapped;
+			   __entry->seqno = seqno;
+			   __entry->start_page = start_page;
+			   __entry->end_page = end_page;
+			   ),
+	    TP_printk("%s: pasid=%u gpusvm=%p mapped=%d seqno=%lu range=[0x%lx-0x%lx]-0x%lx",
+		      __get_str(op), __entry->pasid, __entry->gpusvm,
+		      __entry->mapped, __entry->seqno,
+		      __entry->start_page, __entry->end_page,
+		      __entry->end_page - __entry->start_page)
+);
+
 
 TRACE_EVENT(amdgpu_bo_create,
 	    TP_PROTO(struct amdgpu_bo *bo),
-- 
2.53.0

[PATCH v10 06/22] drm/amdgpu/gmc: add get_svm_pte_flags callback

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add a gmc_funcs callback for computing the GPU PTE flags of an SVM
mapping, plus a HW-agnostic input struct (amdgpu_svm_pte_flags_params)
and the amdgpu_gmc_get_svm_pte_flags() wrapper. This lets the per-IP
MTYPE/PTE selection live in the HW IP specific gmc_vX_0.c files instead
of the SVM core, mirroring how get_vm_pte handles BO mappings.

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  24 +++++
 drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c  |  69 ++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 119 ++++++++++++++++++++++++
 3 files changed, 212 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
index cffd5ca3fc66b..3f668fb71a632 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
@@ -150,6 +150,24 @@ struct amdgpu_vmhub {
 	const struct amdgpu_vmhub_funcs *vmhub_funcs;
 };
 
+/* Decoded inputs for the SVM PTE flag selection, gmc_funcs::get_svm_pte_flags */
+struct amdgpu_svm_pte_flags_params {
+	/* SVM attribute: coherent */
+	bool coherent;
+	/* SVM attribute: extended coherent */
+	bool ext_coherent;
+	/* SVM attribute: GPU read only */
+	bool gpu_ro;
+	/* SVM attribute: GPU executable */
+	bool gpu_exec;
+	/* mapping is in VRAM local device or remote P2P */
+	bool is_vram;
+	/* mapping is in this GPU's local VRAM */
+	bool is_local;
+	/* remote VRAM BO is on a GPU in the same XGMI hive as this GPU. */
+	bool same_hive;
+};
+
 /*
  * GPU MC structures, functions & helpers
  */
@@ -178,6 +196,10 @@ struct amdgpu_gmc_funcs {
 			   struct amdgpu_bo *bo,
 			   uint32_t vm_flags,
 			   uint64_t *pte_flags);
+	/* get the full pte flags for an SVM mapping */
+	uint64_t (*get_svm_pte_flags)(struct amdgpu_device *adev,
+				      struct amdgpu_vm *vm,
+				      const struct amdgpu_svm_pte_flags_params *params);
 	/* override per-page pte flags */
 	void (*override_vm_pte_flags)(struct amdgpu_device *dev,
 				      struct amdgpu_vm *vm,
@@ -378,6 +400,8 @@ struct amdgpu_gmc {
 #define amdgpu_gmc_get_vm_pte(adev, vm, bo, vm_flags, pte_flags) \
 	((adev)->gmc.gmc_funcs->get_vm_pte((adev), (vm), (bo), (vm_flags), \
 					   (pte_flags)))
+#define amdgpu_gmc_get_svm_pte_flags(adev, vm, params) \
+	((adev)->gmc.gmc_funcs->get_svm_pte_flags((adev), (vm), (params)))
 #define amdgpu_gmc_override_vm_pte_flags(adev, vm, addr, pte_flags)	\
 	(adev)->gmc.gmc_funcs->override_vm_pte_flags			\
 		((adev), (vm), (addr), (pte_flags))
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
index 0cc73982ed6f9..a541173e34a4e 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
@@ -551,6 +551,74 @@ static void gmc_v12_0_get_vm_pte(struct amdgpu_device *adev,
 		*flags = AMDGPU_PTE_MTYPE_GFX12(*flags, MTYPE_UC);
 }
 
+/*
+ * Mirror of the GFX12 cases of svm_range_get_pte_flags(), with the
+ * per IP logic kept here in the HW specific file.
+ */
+static uint64_t
+gmc_v12_0_get_svm_pte_flags(struct amdgpu_device *adev,
+			    struct amdgpu_vm *vm,
+			    const struct amdgpu_svm_pte_flags_params *params)
+{
+	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
+	bool coherent = params->coherent;
+	bool ext_coherent = params->ext_coherent;
+	bool is_local = params->is_local;
+	bool is_vram = params->is_vram;
+	bool snoop = !is_vram;
+	uint32_t mapping_flags = 0;
+	unsigned int mtype_local, mtype_remote;
+	bool is_aid_a1;
+	uint64_t pte_flags;
+
+	switch (gc_ip_version) {
+	case IP_VERSION(12, 0, 0):
+	case IP_VERSION(12, 0, 1):
+		mapping_flags |= AMDGPU_VM_MTYPE_NC;
+		break;
+	case IP_VERSION(12, 1, 0):
+		is_aid_a1 = (adev->rev_id & 0x10);
+		mtype_local = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_RW :
+				amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
+				is_aid_a1 ? AMDGPU_VM_MTYPE_RW : AMDGPU_VM_MTYPE_NC;
+		mtype_remote = is_aid_a1 ? AMDGPU_VM_MTYPE_NC : AMDGPU_VM_MTYPE_UC;
+		snoop = true;
+
+		if (is_local) {
+			mapping_flags |= mtype_local;
+		} else if (ext_coherent) {
+			mapping_flags |= AMDGPU_VM_MTYPE_UC;
+		} else {
+			/* system memory or remote VRAM */
+			mapping_flags |= mtype_remote;
+		}
+		break;
+	default:
+		mapping_flags |= coherent ?
+			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		break;
+	}
+
+	if (params->gpu_exec)
+		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
+
+	pte_flags = AMDGPU_PTE_VALID;
+	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
+	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
+	pte_flags |= AMDGPU_PTE_IS_PTE;
+
+	gmc_v12_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
+	pte_flags |= AMDGPU_PTE_READABLE;
+	if (!params->gpu_ro)
+		pte_flags |= AMDGPU_PTE_WRITEABLE;
+
+	if (gc_ip_version == IP_VERSION(12, 1, 0) &&
+	    adev->have_atomics_support)
+		pte_flags |= AMDGPU_PTE_BUS_ATOMICS;
+
+	return pte_flags;
+}
+
 static unsigned gmc_v12_0_get_vbios_fb_size(struct amdgpu_device *adev)
 {
 	return 0;
@@ -580,6 +648,7 @@ static const struct amdgpu_gmc_funcs gmc_v12_0_gmc_funcs = {
 	.emit_pasid_mapping = gmc_v12_0_emit_pasid_mapping,
 	.get_vm_pde = gmc_v12_0_get_vm_pde,
 	.get_vm_pte = gmc_v12_0_get_vm_pte,
+	.get_svm_pte_flags = gmc_v12_0_get_svm_pte_flags,
 	.get_vbios_fb_size = gmc_v12_0_get_vbios_fb_size,
 	.get_dcc_alignment = gmc_v12_0_get_dcc_alignment,
 };
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
index 633772fa71fda..a3a5fa048a393 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
@@ -1199,6 +1199,124 @@ static void gmc_v9_0_get_vm_pte(struct amdgpu_device *adev,
 		gmc_v9_0_get_coherence_flags(adev, vm, bo, vm_flags, flags);
 }
 
+/*
+ * Mirror of the GFX9 cases of svm_range_get_pte_flags(), with the
+ * per IP logic kept here in the HW specific file.
+ */
+static uint64_t
+gmc_v9_0_get_svm_pte_flags(struct amdgpu_device *adev,
+			   struct amdgpu_vm *vm,
+			   const struct amdgpu_svm_pte_flags_params *params)
+{
+	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
+	bool coherent = params->coherent;
+	bool ext_coherent = params->ext_coherent;
+	bool is_local = params->is_local;
+	bool is_vram = params->is_vram;
+	bool same_hive = params->same_hive;
+	bool snoop = !is_vram;
+	uint32_t mapping_flags = 0;
+	unsigned int mtype_local;
+	uint64_t pte_flags;
+
+	switch (gc_ip_version) {
+	case IP_VERSION(9, 4, 1):
+		if (is_vram) {
+			if (is_local) {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
+			} else {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+				if (same_hive)
+					snoop = true;
+			}
+		} else {
+			mapping_flags |= coherent ?
+				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		}
+		break;
+	case IP_VERSION(9, 4, 2):
+		if (is_vram) {
+			if (is_local) {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
+				if (adev->gmc.xgmi.connected_to_cpu)
+					snoop = true;
+			} else {
+				mapping_flags |= coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+				if (same_hive)
+					snoop = true;
+			}
+		} else {
+			mapping_flags |= coherent ?
+				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		}
+		break;
+	case IP_VERSION(9, 4, 3):
+	case IP_VERSION(9, 4, 4):
+	case IP_VERSION(9, 5, 0):
+		if (ext_coherent)
+			mtype_local = AMDGPU_VM_MTYPE_CC;
+		else
+			mtype_local = amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
+				amdgpu_mtype_local == 2 ? AMDGPU_VM_MTYPE_CC :
+				AMDGPU_VM_MTYPE_RW;
+		snoop = true;
+		if (is_vram) {
+			/* local HBM region close to partition */
+			if (is_local)
+				mapping_flags |= mtype_local;
+			/* local HBM region far from partition or remote XGMI GPU
+			 * with regular system scope coherence
+			 */
+			else if (same_hive && !ext_coherent)
+				mapping_flags |= AMDGPU_VM_MTYPE_NC;
+			/* PCIe P2P on GPUs pre-9.5.0 */
+			else if (gc_ip_version < IP_VERSION(9, 5, 0) && !same_hive)
+				mapping_flags |= AMDGPU_VM_MTYPE_UC;
+			/* Other remote memory */
+			else
+				mapping_flags |= ext_coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		} else if (adev->flags & AMD_IS_APU) {
+			/* On NUMA systems, locality is determined per-page
+			 * in gmc_v9_0_override_vm_pte_flags.
+			 */
+			if (num_possible_nodes() <= 1)
+				mapping_flags |= mtype_local;
+			else
+				mapping_flags |= ext_coherent ?
+					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		} else {
+			if (gc_ip_version < IP_VERSION(9, 5, 0) || ext_coherent)
+				mapping_flags |= AMDGPU_VM_MTYPE_UC;
+			else
+				mapping_flags |= AMDGPU_VM_MTYPE_NC;
+		}
+		break;
+	default:
+		mapping_flags |= coherent ?
+			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+		break;
+	}
+
+	if (params->gpu_exec)
+		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
+
+	pte_flags = AMDGPU_PTE_VALID;
+	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
+	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
+
+	gmc_v9_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
+	pte_flags |= AMDGPU_PTE_READABLE;
+	if (!params->gpu_ro)
+		pte_flags |= AMDGPU_PTE_WRITEABLE;
+
+	return pte_flags;
+}
+
 static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev,
 					   struct amdgpu_vm *vm,
 					   uint64_t addr, uint64_t *flags)
@@ -1316,6 +1434,7 @@ static const struct amdgpu_gmc_funcs gmc_v9_0_gmc_funcs = {
 	.emit_pasid_mapping = gmc_v9_0_emit_pasid_mapping,
 	.get_vm_pde = gmc_v9_0_get_vm_pde,
 	.get_vm_pte = gmc_v9_0_get_vm_pte,
+	.get_svm_pte_flags = gmc_v9_0_get_svm_pte_flags,
 	.override_vm_pte_flags = gmc_v9_0_override_vm_pte_flags,
 	.get_vbios_fb_size = gmc_v9_0_get_vbios_fb_size,
 	.query_mem_partition_mode = &amdgpu_gmc_query_memory_partition,
-- 
2.53.0

[PATCH v10 07/22] drm/amdgpu: implement SVM range GPU mapping core

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Implement GPU page table mapping in amdgpu_svm_range.c:
- amdgpu_svm_range_pages_valid: check notifier sequence validity
- amdgpu_svm_range_is_valid: validate gpu_mapped and attr_flags
  against the requested attribute set, under MMU notifier lock.
- amdgpu_svm_range_zap_ptes: clear GPU PTEs via amdgpu_vm_update_range
- amdgpu_svm_range_attr_pte_flags: compute PTE flags per GC IP version
- amdgpu_svm_range_lock_vm_pd: acquire VM page directory by drm_exec
- amdgpu_svm_range_update_gpu_range: DMA segment coalescing; recomputes
  PTE flags per segment from entry->proto via attr_pte_flags, then
  programs PTEs under notifier lock
- amdgpu_svm_range_find_or_insert: wrapper of drm gpu SVM find or insert
  API.
- amdgpu_svm_range_get_pages: wrapper of drm gpu SVM get pages.
- amdgpu_svm_range_evict: thin wrapper of drm_gpusvm_range_evict
- amdgpu_svm_range_update_mapping: full pipeline (lock PD, validate
  pages, program PTEs, update PDEs, flush TLB, record attr_flags
  and mapped state); takes attrs + read_only and forwards them to
  update_gpu_range
- amdgpu_svm_range_map_attrs: cursor based iteration across attr tree,
  creating/mapping ranges

- Add to amdgpu_svm.h driver private interconnect tags used in
drm_pagemap_addr.proto:
  AMDGPU_INTERCONNECT_VRAM = DRM_INTERCONNECT_DRIVER
  AMDGPU_INTERCONNECT_P2P  = AMDGPU_INTERCONNECT_VRAM + 1

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       |   4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 417 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  40 ++
 3 files changed, 461 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
index 72f954d2df39e..cfdc4e6d045b7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
@@ -27,6 +27,7 @@
 
 #include <drm/amdgpu_drm.h>
 #include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
 #include <linux/atomic.h>
 #include <linux/kref.h>
 #include <linux/list.h>
@@ -44,6 +45,9 @@ struct amdgpu_svm_attrs;
 struct drm_device;
 struct drm_file;
 
+#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
+#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
+
 enum amdgpu_svm_xnack_mode {
 	AMDGPU_SVM_XNACK_OFF,
 	AMDGPU_SVM_XNACK_ON,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
new file mode 100644
index 0000000000000..d4830e53f3d8e
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -0,0 +1,417 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu_svm_fault.h"
+#include "amdgpu.h"
+#include "amdgpu_vm.h"
+#include "amdgpu_trace.h"
+
+#include <drm/drm_exec.h>
+#include <drm/drm_pagemap.h>
+
+#include <linux/mmu_notifier.h>
+#include <uapi/linux/kfd_ioctl.h>
+
+bool
+amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
+		  struct amdgpu_svm_range *range)
+{
+	struct drm_gpusvm_range *base = &range->base;
+
+	lockdep_assert_held(&svm->gpusvm.notifier_lock);
+
+	if (base->pages.flags.unmapped || base->pages.flags.partial_unmap)
+		return false;
+
+	return drm_gpusvm_range_pages_valid(&svm->gpusvm, base);
+}
+
+bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
+			       const struct amdgpu_svm_attrs *attrs)
+{
+	return READ_ONCE(range->gpu_mapped) &&
+	       READ_ONCE(range->attr_flags) == attrs->flags;
+}
+
+
+int
+amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
+			  struct amdgpu_svm_range *range,
+			  unsigned long start_page,
+			  unsigned long last_page)
+{
+	struct dma_fence *fence = NULL;
+	unsigned int flags;
+	int ret;
+
+	if (last_page < start_page)
+		return 0;
+
+	flags = memalloc_noreclaim_save();
+	ret = amdgpu_vm_unmap_range(svm->adev, svm->vm, NULL,
+				    start_page, last_page, 0, &fence);
+	memalloc_noreclaim_restore(flags);
+
+	if (fence) {
+		dma_fence_wait(fence, false);
+		dma_fence_put(fence);
+	}
+
+	return ret;
+}
+
+uint64_t
+amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
+			    const struct amdgpu_svm_attrs *attrs,
+			    bool read_only,
+			    enum drm_interconnect_protocol proto)
+{
+	struct amdgpu_device *adev = svm->adev;
+	uint32_t flags = attrs->flags;
+	bool is_local = (proto == AMDGPU_INTERCONNECT_VRAM);
+	struct amdgpu_svm_pte_flags_params params = {
+		.coherent = flags & (AMDGPU_SVM_ATTR_BIT_COHERENT |
+				     AMDGPU_SVM_ATTR_BIT_EXT_COHERENT),
+		.ext_coherent = flags & AMDGPU_SVM_ATTR_BIT_EXT_COHERENT,
+		.gpu_ro = flags & AMDGPU_SVM_ATTR_BIT_GPU_RO,
+		.gpu_exec = flags & AMDGPU_SVM_ATTR_BIT_GPU_EXEC,
+		.is_local = is_local,
+		.is_vram = is_local || (proto == AMDGPU_INTERCONNECT_P2P),
+		/* TODO: add hive check. */
+		.same_hive = false,
+	};
+	uint32_t gc_ip_version;
+	uint32_t mapping_flags = 0;
+	uint64_t pte_flags;
+	bool snoop;
+
+	/*
+	 * The per-IP MTYPE / PTE flag selection lives in the HW IP specific
+	 * gmc_vX_0.c files mirroring amdgpu_gmc_get_vm_pte for BO mappings.
+	 */
+	if (adev->gmc.gmc_funcs->get_svm_pte_flags) {
+		pte_flags = amdgpu_gmc_get_svm_pte_flags(adev, svm->vm, &params);
+	} else {
+		gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
+		snoop = !is_local;
+		mapping_flags |= params.coherent ?
+			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
+
+		if (params.gpu_exec)
+			mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
+
+		pte_flags = AMDGPU_PTE_VALID;
+		pte_flags |= is_local ? 0 : AMDGPU_PTE_SYSTEM;
+		pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
+		if (gc_ip_version >= IP_VERSION(12, 0, 0))
+			pte_flags |= AMDGPU_PTE_IS_PTE;
+
+		amdgpu_gmc_get_vm_pte(adev, svm->vm, NULL, mapping_flags,
+				      &pte_flags);
+		pte_flags |= AMDGPU_PTE_READABLE;
+		if (!params.gpu_ro)
+			pte_flags |= AMDGPU_PTE_WRITEABLE;
+	}
+
+	if (read_only)
+		pte_flags &= ~AMDGPU_PTE_WRITEABLE;
+
+	return pte_flags;
+}
+
+
+
+int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm, struct drm_exec *exec,
+				bool intr)
+{
+	unsigned int exec_flags = DRM_EXEC_IGNORE_DUPLICATES;
+	int ret;
+
+	if (intr)
+		exec_flags |= DRM_EXEC_INTERRUPTIBLE_WAIT;
+
+	drm_exec_init(exec, exec_flags, 0);
+	drm_exec_until_all_locked(exec) {
+		ret = amdgpu_vm_lock_pd(svm->vm, exec, 1);
+		drm_exec_retry_on_contention(exec);
+		if (ret) {
+			drm_exec_fini(exec);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
+amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
+				  struct amdgpu_svm_range *range,
+				  const struct amdgpu_svm_attrs *attrs,
+				  bool read_only,
+				  bool flush_tlb,
+				  bool wait_fence,
+				  struct dma_fence **fence)
+{
+	struct drm_gpusvm_range *base = &range->base;
+
+	lockdep_assert_held(&svm->gpusvm.notifier_lock);
+
+	const unsigned long range_start_page = drm_gpusvm_range_start(base) >> PAGE_SHIFT;
+	const unsigned long range_end_page = drm_gpusvm_range_end(base) >> PAGE_SHIFT;
+	const unsigned long npages = range_end_page - range_start_page;
+	unsigned long mapped_pages = 0;
+	unsigned long dma_idx = 0;
+	int ret;
+
+	if (!base->pages.dma_addr || !npages)
+		return -EINVAL;
+
+	while (mapped_pages < npages) {
+		const struct drm_pagemap_addr *entry = &base->pages.dma_addr[dma_idx++];
+		unsigned long seg_pages = min_t(unsigned long, 1UL << entry->order,
+						npages - mapped_pages);
+		uint64_t pte_flags;
+		unsigned long start_page, last_page;
+		bool is_last_seg;
+
+		if (entry->proto != DRM_INTERCONNECT_SYSTEM)
+			return -EOPNOTSUPP;
+
+		pte_flags = amdgpu_svm_range_attr_pte_flags(svm, attrs,
+							    read_only,
+							    entry->proto);
+
+		start_page = range_start_page + mapped_pages;
+		last_page = start_page + seg_pages - 1;
+		mapped_pages += seg_pages;
+		is_last_seg = mapped_pages == npages;
+
+		/*
+		 * The struct hmm_range hmm_range field inside amdgpu_hmm_range
+		 * is drm_gpusvm internal state. It is not exposed to the
+		 * implementing driver and is managed entirely by the framework
+		 * as part of the framework logic. This amdgpu_vm_map_range call
+		 * always runs inside the drm_gpusvm notifier, so omitting
+		 * hmm_range here is safe.
+		 */
+		ret = amdgpu_vm_map_range(svm->adev, svm->vm,
+					  flush_tlb && is_last_seg, true, NULL,
+					  start_page, last_page, pte_flags,
+					  0, entry->addr, NULL, NULL, NULL,
+					  wait_fence && is_last_seg ? fence : NULL);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+struct amdgpu_svm_range *
+amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
+				unsigned long gpuva_start, unsigned long gpuva_end,
+				struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_range *r;
+
+retry:
+	r = drm_gpusvm_range_find_or_insert(&svm->gpusvm, addr,
+					    gpuva_start, gpuva_end, ctx);
+	/*
+	 * UMD doesn't set RO for some RO VMAs, but the drm gpu svm framework
+	 * denies no RO flag range insert for RO VMAs, so treat
+	 * -EPERM as an indication of RO and retry if not set.
+	 */
+	if (PTR_ERR_OR_ZERO(r) == -EPERM && !ctx->read_only) {
+		ctx->read_only = true;
+		goto retry;
+	}
+
+	if (IS_ERR(r))
+		return ERR_CAST(r);
+
+	return to_amdgpu_svm_range(r);
+}
+
+int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
+			       struct drm_gpusvm_range *range,
+			       struct drm_gpusvm_ctx *ctx)
+{
+	int ret;
+
+retry:
+	ret = drm_gpusvm_range_get_pages(&svm->gpusvm, range, ctx);
+	/*
+	 * HMM returns -EPERM when write access is requested for a read-only
+	 * VMA. Retry as read-only so the eventual GPU mapping follows the CPU
+	 * VMA permissions.
+	 */
+	if (ret == -EPERM && !ctx->read_only) {
+		ctx->read_only = true;
+		goto retry;
+	}
+
+	if (ret == -EOPNOTSUPP) {
+		pr_err_ratelimited("range get pages -EOPNOTSUPP, evict and retry: gpuva=[0x%lx-0x%lx) ret=%d\n",
+				   drm_gpusvm_range_start(range),
+				   drm_gpusvm_range_end(range), ret);
+		amdgpu_svm_range_evict(svm, range);
+	}
+
+	return ret;
+}
+
+void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
+			    struct drm_gpusvm_range *range)
+{
+	if (!range->pages.flags.has_devmem_pages)
+		return;
+
+	drm_gpusvm_range_evict(&svm->gpusvm, range);
+}
+
+int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
+				    struct amdgpu_svm_range *range,
+				    const struct amdgpu_svm_attrs *attrs,
+				    bool read_only,
+				    bool intr, bool wait,
+				    bool flush_tlb)
+{
+	struct drm_exec exec;
+	struct dma_fence *fence = NULL;
+	unsigned int flags;
+	int ret;
+
+	ret = amdgpu_svm_range_lock_vm_pd(svm, &exec, intr);
+	if (ret)
+		return ret;
+
+	flags = memalloc_noreclaim_save();
+	drm_gpusvm_notifier_lock(&svm->gpusvm);
+
+	if (!amdgpu_svm_range_pages_valid(svm, range)) {
+		amdgpu_svm_range_invalidate_gpu_mapping(range);
+		ret = -EAGAIN;
+	} else {
+		ret = amdgpu_svm_range_update_gpu_range(svm, range, attrs,
+							read_only, flush_tlb,
+							wait, wait ? &fence : NULL);
+	}
+
+	drm_gpusvm_notifier_unlock(&svm->gpusvm);
+	memalloc_noreclaim_restore(flags);
+
+	if (!ret && fence)
+		dma_fence_wait(fence, intr);
+	dma_fence_put(fence);
+
+	if (!ret)
+		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, false);
+
+	if (!ret) {
+		if (flush_tlb)
+			amdgpu_svm_flush_tlb(svm);
+		WRITE_ONCE(range->attr_flags, attrs->flags);
+		WRITE_ONCE(range->gpu_mapped, true);
+		range->validate_timestamp = ktime_get_boottime();
+	}
+
+	drm_exec_fini(&exec);
+	return ret;
+}
+
+int
+amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
+		       const struct amdgpu_svm_attrs *attrs,
+		       unsigned long start, unsigned long end)
+{
+	unsigned long addr = start;
+	int ret;
+	bool devmem_possible = amdgpu_svm_devmem_possible(svm);
+	bool need_vram_migration = devmem_possible &&
+				   amdgpu_svm_attr_prefer_vram(attrs);
+	struct drm_gpusvm_ctx map_ctx = {
+		.devmem_possible = devmem_possible,
+		.devmem_only = need_vram_migration,
+		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
+	};
+
+	while (addr < end) {
+		struct amdgpu_svm_range *range;
+		unsigned long next_addr;
+		/* reset read_only every iteration, amdgpu_svm_range_find_or_insert may change it */
+		map_ctx.read_only = !!(attrs->flags & AMDGPU_SVM_ATTR_BIT_GPU_RO);
+
+		range = amdgpu_svm_range_find_or_insert(svm, addr,
+							addr, end,
+							&map_ctx);
+		if (IS_ERR(range)) {
+			pr_err_ratelimited("failed to find/insert range for gpuva 0x%lx [0x%lx-0x%lx), ret=%ld\n",
+					   addr, start, end, PTR_ERR(range));
+			return PTR_ERR(range);
+		}
+
+		next_addr = drm_gpusvm_range_end(&range->base);
+		if (next_addr <= addr)
+			return -EINVAL;
+
+		if (amdgpu_svm_range_is_valid(range, attrs)) {
+			addr = next_addr;
+			continue;
+		}
+
+		/* TODO: add migration */
+
+		AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
+
+		ret = amdgpu_svm_range_get_pages(svm, &range->base,
+						 &map_ctx);
+		if (ret) {
+			pr_err_ratelimited("failed to get pages for range [0x%lx-0x%lx), ret=%d\n",
+					   drm_gpusvm_range_start(&range->base),
+					   drm_gpusvm_range_end(&range->base), ret);
+			return ret;
+		}
+
+		AMDGPU_SVM_RANGE_TRACE(range, "UPDATE MAPPING");
+
+		ret = amdgpu_svm_range_update_mapping(svm, range, attrs,
+						      map_ctx.read_only,
+						      true, true,
+						      true);
+		if (ret) {
+			pr_err_ratelimited("failed to update gpu map for range [0x%lx-0x%lx), ret=%d\n",
+					   drm_gpusvm_range_start(&range->base),
+					   drm_gpusvm_range_end(&range->base), ret);
+			return ret;
+		}
+
+		addr = next_addr;
+	}
+
+	return 0;
+}
+
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
index 39db3a18b5f2f..e5a5f52166576 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -126,4 +126,44 @@ struct amdgpu_svm_range_op_ctx {
 	uint8_t pending_ops;
 };
 
+uint64_t amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
+					 const struct amdgpu_svm_attrs *attrs,
+					 bool read_only,
+					 enum drm_interconnect_protocol proto);
+int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm,
+				struct drm_exec *exec,
+				bool intr);
+bool amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
+				  struct amdgpu_svm_range *range);
+bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
+			       const struct amdgpu_svm_attrs *attrs);
+int amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
+				      struct amdgpu_svm_range *range,
+				      const struct amdgpu_svm_attrs *attrs,
+				      bool read_only,
+				      bool flush_tlb, bool wait,
+				      struct dma_fence **fence);
+int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
+				    struct amdgpu_svm_range *range,
+				    const struct amdgpu_svm_attrs *attrs,
+				    bool read_only,
+				    bool intr, bool wait,
+				    bool flush_tlb);
+int amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
+			       const struct amdgpu_svm_attrs *attrs,
+			       unsigned long start, unsigned long end);
+int amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
+			      struct amdgpu_svm_range *range,
+			      unsigned long start_page,
+			      unsigned long last_page);
+void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
+			    struct drm_gpusvm_range *range);
+struct amdgpu_svm_range *
+amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
+				unsigned long gpuva_start, unsigned long gpuva_end,
+				struct drm_gpusvm_ctx *ctx);
+int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
+			       struct drm_gpusvm_range *range,
+			       struct drm_gpusvm_ctx *ctx);
+
 #endif /* __AMDGPU_SVM_RANGE_H__ */
-- 
2.53.0

[PATCH v10 08/22] drm/amdgpu: implement SVM range notifier and GC helpers

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add MMU notifier event handling and garbage collection infrastructure:
- remove invalidated ranges from gpusvm
- zap affected PTEs and invalidate GPU mappings
- queue unmapped ranges for deferred garbage collection
- serialize GC queue state and range references
- drain queued GC before changing each attribute segment
- reject ranges already marked unmapped during lookup

The last two steps prevent attribute updates and prefetch mapping from
reusing a stale range that has been marked unmapped but has not yet been
removed from the gpusvm tree.

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c  |   6 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 315 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  16 +
 3 files changed, 337 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
index c76e10473968e..014785a6db742 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
@@ -583,6 +583,12 @@ amdgpu_svm_attr_set_range(struct amdgpu_svm_attr_tree *attr_tree,
 		int ret;
 
 		amdgpu_svm_lock(svm);
+		ret = amdgpu_svm_garbage_collector(svm);
+		if (ret) {
+			amdgpu_svm_unlock(svm);
+			return ret;
+		}
+
 		mutex_lock(&attr_tree->lock);
 		node = interval_tree_iter_first(&attr_tree->tree, cursor, cursor);
 		if (node) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index d4830e53f3d8e..b907f516cdbe6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -235,6 +235,7 @@ amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
 				unsigned long gpuva_start, unsigned long gpuva_end,
 				struct drm_gpusvm_ctx *ctx)
 {
+	struct drm_gpusvm_pages_flags flags;
 	struct drm_gpusvm_range *r;
 
 retry:
@@ -253,6 +254,10 @@ amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
 	if (IS_ERR(r))
 		return ERR_CAST(r);
 
+	flags.__flags = READ_ONCE(r->pages.flags.__flags);
+	if (flags.unmapped)
+		return ERR_PTR(-EAGAIN);
+
 	return to_amdgpu_svm_range(r);
 }
 
@@ -415,3 +420,313 @@ amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 	return 0;
 }
 
+void amdgpu_svm_range_remove(struct amdgpu_svm *svm,
+			     struct amdgpu_svm_range *range,
+			     struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_range *base = &range->base;
+
+	amdgpu_svm_assert_locked(svm);
+
+	if (!base->pages.flags.unmapped && !base->pages.flags.partial_unmap)
+		drm_gpusvm_range_unmap_pages(&svm->gpusvm, base, ctx);
+
+	amdgpu_svm_range_invalidate_gpu_mapping(range);
+	drm_gpusvm_range_remove(&svm->gpusvm, base);
+}
+
+/**
+ * amdgpu_svm_range_notifier_event_begin() - First MMU notifier phase
+ * @svm: The SVM context.
+ * @range: The GPU SVM range covered by the notifier event.
+ * @mmu_range: The MMU notifier range describing the invalidation window.
+ *
+ * Invalidate the GPU view of @range over the notifier window: zap the
+ * affected PTEs and clear the gpu_mapped flag. The range is not removed
+ * here; structural removal if needed happens in the event-end phase.
+ *
+ * Return: true if PTEs were zapped, false otherwise.
+ */
+bool
+amdgpu_svm_range_notifier_event_begin(struct amdgpu_svm *svm,
+				      struct drm_gpusvm_range *range,
+				      const struct mmu_notifier_range *mmu_range)
+{
+	struct amdgpu_svm_range *svm_range = to_amdgpu_svm_range(range);
+	unsigned long start_page, last_page;
+
+	amdgpu_svm_assert_in_notifier(svm);
+
+	AMDGPU_SVM_RANGE_TRACE(svm_range, "NOTIFIER");
+
+	if (range->pages.flags.unmapped || !svm_range->gpu_mapped)
+		return false;
+
+	AMDGPU_SVM_RANGE_TRACE(svm_range, "NOTIFIER - EXECUTE");
+
+	start_page = max(drm_gpusvm_range_start(range),
+			 mmu_range->start) >> PAGE_SHIFT;
+	last_page = (min(drm_gpusvm_range_end(range),
+			 mmu_range->end) >> PAGE_SHIFT) - 1;
+
+	amdgpu_svm_range_zap_ptes(svm, svm_range, start_page, last_page);
+	amdgpu_svm_range_invalidate_gpu_mapping(svm_range);
+
+	return true;
+}
+
+/**
+ * amdgpu_svm_gc_enqueue() - Queue a range for work queue processing
+ * @svm: The SVM context.
+ * @range: The SVM range to queue.
+ * @start_page: First page of the pending invalidation interval.
+ * @last_page: Last page of the pending invalidation interval.
+ *
+ * Queue @range on the GC list and kick the GC worker. A range reference
+ * is taken when it first enters IN_GC and dropped only when it returns
+ * to NOT_QUEUED, so the range stays alive while on the GC list or being
+ * processed. All transitions are serialized by svm->work_lock.
+ */
+static void
+amdgpu_svm_gc_enqueue(struct amdgpu_svm *svm,
+		      struct amdgpu_svm_range *range,
+		      unsigned long start_page, unsigned long last_page)
+{
+	if (atomic_read(&svm->exiting))
+		return;
+
+	spin_lock(&svm->work_lock);
+	if (range->queue_state == AMDGPU_SVM_RANGE_NOT_QUEUED) {
+		drm_gpusvm_range_get(&range->base);
+		range->queue_state = AMDGPU_SVM_RANGE_IN_GC;
+		list_add_tail(&range->work_node, &svm->gc.list);
+	}
+
+	range->pending_start_page = min(range->pending_start_page, start_page);
+	range->pending_last_page = max(range->pending_last_page, last_page);
+	range->pending_ops |= AMDGPU_SVM_RANGE_OP_UNMAP;
+	spin_unlock(&svm->work_lock);
+
+	queue_work(svm->gc.wq, &svm->gc.work);
+}
+
+static void
+amdgpu_svm_gc_add_range(struct amdgpu_svm *svm,
+			struct amdgpu_svm_range *svm_range,
+			const struct mmu_notifier_range *mmu_range)
+{
+	unsigned long start_page = max(drm_gpusvm_range_start(&svm_range->base),
+				       mmu_range->start) >> PAGE_SHIFT;
+	unsigned long last_page = (min(drm_gpusvm_range_end(&svm_range->base),
+				       mmu_range->end) >> PAGE_SHIFT) - 1;
+
+	AMDGPU_SVM_RANGE_TRACE(svm_range, "GARBAGE COLLECTOR ADD");
+
+	drm_gpusvm_range_set_unmapped(&svm_range->base, mmu_range);
+	amdgpu_svm_gc_enqueue(svm, svm_range, start_page, last_page);
+}
+
+/**
+ * amdgpu_svm_range_notifier_event_end() - Second MMU notifier phase
+ * @svm: The SVM context.
+ * @range: The GPU SVM range covered by the notifier event.
+ * @mmu_range: The MMU notifier range describing the event.
+ *
+ * DMA-unmap the pages for @range. On an actual unmap event, also queue the
+ * range to the GC so it gets structurally removed from the gpusvm trees.
+ */
+static void
+amdgpu_svm_range_notifier_event_end(struct amdgpu_svm *svm,
+				    struct drm_gpusvm_range *range,
+				    const struct mmu_notifier_range *mmu_range)
+{
+	struct drm_gpusvm_ctx ctx = { .in_notifier = true, };
+
+	amdgpu_svm_assert_in_notifier(svm);
+
+	drm_gpusvm_range_unmap_pages(&svm->gpusvm, range, &ctx);
+	if (mmu_range->event == MMU_NOTIFY_UNMAP)
+		amdgpu_svm_gc_add_range(svm, to_amdgpu_svm_range(range),
+					mmu_range);
+}
+
+int
+amdgpu_svm_range_invalidate_interval(struct amdgpu_svm *svm,
+				     unsigned long start_page,
+				     unsigned long last_page)
+{
+	unsigned long start = start_page << PAGE_SHIFT;
+	unsigned long end = (last_page + 1) << PAGE_SHIFT;
+	struct drm_gpusvm_notifier *notifier, *next_notifier;
+	struct drm_gpusvm_ctx ctx = { .in_notifier = false };
+	struct drm_exec exec;
+	bool needs_flush = false;
+	int ret;
+
+	amdgpu_svm_assert_locked(svm);
+
+	ret = amdgpu_svm_range_lock_vm_pd(svm, &exec, true);
+	if (ret)
+		return ret;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next_notifier, &svm->gpusvm,
+					  start, end) {
+		struct drm_gpusvm_range *range, *next_range;
+
+		drm_gpusvm_for_each_range_safe(range, next_range, notifier,
+					       start, end) {
+			struct amdgpu_svm_range *svm_range = to_amdgpu_svm_range(range);
+			bool crosses_boundary = start > drm_gpusvm_range_start(range) ||
+						end < drm_gpusvm_range_end(range);
+
+			if (svm_range->gpu_mapped) {
+				AMDGPU_SVM_RANGE_TRACE(svm_range,
+					crosses_boundary ? "ATTR DESTROY" :
+							   "ATTR ZAP PTE");
+
+				ret = amdgpu_svm_range_zap_ptes(svm, svm_range,
+						drm_gpusvm_range_start(range) >> PAGE_SHIFT,
+						(drm_gpusvm_range_end(range) >> PAGE_SHIFT) - 1);
+				if (ret < 0) {
+					drm_exec_fini(&exec);
+					return ret;
+				}
+
+				amdgpu_svm_range_invalidate_gpu_mapping(svm_range);
+				needs_flush = true;
+			}
+
+			if (crosses_boundary) {
+				/* remove ranges crossing the boundary so GPU fault
+				 * creates new ranges bounded by the updated
+				 * attr_range boundaries.
+				 * Evict devmem-backed pages back to sysmem first
+				 * so VRAM-resident data is not lost when the range
+				 * is destroyed. No-op for sysmem-only ranges.
+				 */
+				drm_gpusvm_range_get(range);
+
+				amdgpu_svm_range_evict(svm, range);
+				amdgpu_svm_range_remove(svm, svm_range, &ctx);
+
+				amdgpu_svm_range_unqueue(svm, svm_range);
+
+				drm_gpusvm_range_put(range);
+			}
+		}
+	}
+
+	drm_exec_fini(&exec);
+
+	if (needs_flush)
+		amdgpu_svm_flush_tlb(svm);
+
+	pr_debug("attr invalidate done [0x%lx-0x%lx]-0x%lx needs_flush=%d\n",
+			 start_page, last_page, last_page - start_page + 1,
+			 needs_flush ? 1 : 0);
+
+	return 0;
+}
+
+/**
+ * amdgpu_svm_range_dequeue_locked() - Pull the next range off a work list
+ * @svm: The SVM context.
+ * @work_list: The work list to dequeue from.
+ * @op_ctx: Output context filled with the dequeued range's pending state.
+ *
+ * Move the first range on @work_list to PROCESSING and atomically snapshot
+ * its pending op/interval into @op_ctx while resetting them on the range, so
+ * invalidations that arrive during processing accumulate afresh. Caller must
+ * hold svm->work_lock.
+ *
+ * Return: true if a range was dequeued, false if @work_list was empty.
+ */
+bool
+amdgpu_svm_range_dequeue_locked(struct amdgpu_svm *svm,
+					struct list_head *work_list,
+					struct amdgpu_svm_range_op_ctx *op_ctx)
+{
+	struct amdgpu_svm_range *range;
+
+	lockdep_assert_held(&svm->work_lock);
+
+	range = list_first_entry_or_null(work_list, struct amdgpu_svm_range,
+					work_node);
+	if (!range)
+		return false;
+
+	list_del_init(&range->work_node);
+	range->queue_state = AMDGPU_SVM_RANGE_PROCESSING;
+
+	op_ctx->range = range;
+	op_ctx->start_page = range->pending_start_page;
+	op_ctx->last_page = range->pending_last_page;
+	op_ctx->pending_ops = range->pending_ops;
+
+	range->pending_start_page = ULONG_MAX;
+	range->pending_last_page = 0;
+	range->pending_ops = AMDGPU_SVM_RANGE_OP_NONE;
+
+	return true;
+}
+
+/**
+ * amdgpu_svm_range_put_if_dequeued() - Finish processing a dequeued range
+ * @svm: The SVM context.
+ * @range: The range previously moved to PROCESSING.
+ *
+ * Called after a PROCESSING range has been handled. If new UNMAP work was
+ * enqueued while it was being processed, re-queue it on the GC list and kick
+ * the worker again; otherwise return it to NOT_QUEUED and drop the reference
+ * taken at enqueue time.
+ */
+void
+amdgpu_svm_range_put_if_dequeued(struct amdgpu_svm *svm,
+				     struct amdgpu_svm_range *range)
+{
+	bool release_kref = false;
+	bool queue_gc = false;
+
+	spin_lock(&svm->work_lock);
+
+	if (range->queue_state != AMDGPU_SVM_RANGE_PROCESSING) {
+		spin_unlock(&svm->work_lock);
+		return;
+	}
+
+	if (range->pending_ops & AMDGPU_SVM_RANGE_OP_UNMAP) {
+		list_add_tail(&range->work_node, &svm->gc.list);
+		range->queue_state = AMDGPU_SVM_RANGE_IN_GC;
+		queue_gc = true;
+	} else {
+		range->queue_state = AMDGPU_SVM_RANGE_NOT_QUEUED;
+		release_kref = true;
+	}
+
+	spin_unlock(&svm->work_lock);
+
+	if (queue_gc)
+		queue_work(svm->gc.wq, &svm->gc.work);
+	if (release_kref)
+		drm_gpusvm_range_put(&range->base);
+}
+
+void amdgpu_svm_range_unqueue(struct amdgpu_svm *svm,
+					struct amdgpu_svm_range *range)
+{
+	bool put = false;
+
+	amdgpu_svm_assert_locked(svm);
+
+	spin_lock(&svm->work_lock);
+	if (range->queue_state == AMDGPU_SVM_RANGE_IN_GC) {
+		list_del_init(&range->work_node);
+		range->queue_state = AMDGPU_SVM_RANGE_NOT_QUEUED;
+		range->pending_ops = AMDGPU_SVM_RANGE_OP_NONE;
+		put = true;
+	}
+	spin_unlock(&svm->work_lock);
+
+	if (put)
+		drm_gpusvm_range_put(&range->base);
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
index e5a5f52166576..f423be277a85a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -149,6 +149,16 @@ int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
 				    bool read_only,
 				    bool intr, bool wait,
 				    bool flush_tlb);
+bool amdgpu_svm_range_dequeue_locked(struct amdgpu_svm *svm,
+				     struct list_head *work_list,
+				     struct amdgpu_svm_range_op_ctx *op_ctx);
+void amdgpu_svm_range_put_if_dequeued(struct amdgpu_svm *svm,
+				      struct amdgpu_svm_range *range);
+void amdgpu_svm_range_unqueue(struct amdgpu_svm *svm,
+					struct amdgpu_svm_range *range);
+void amdgpu_svm_range_remove(struct amdgpu_svm *svm,
+			     struct amdgpu_svm_range *range,
+			     struct drm_gpusvm_ctx *ctx);
 int amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 			       const struct amdgpu_svm_attrs *attrs,
 			       unsigned long start, unsigned long end);
@@ -165,5 +175,11 @@ amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
 int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
 			       struct drm_gpusvm_range *range,
 			       struct drm_gpusvm_ctx *ctx);
+bool amdgpu_svm_range_notifier_event_begin(struct amdgpu_svm *svm,
+					   struct drm_gpusvm_range *range,
+					   const struct mmu_notifier_range *mmu_range);
+int amdgpu_svm_range_invalidate_interval(struct amdgpu_svm *svm,
+					 unsigned long start_page,
+					 unsigned long last_page);
 
 #endif /* __AMDGPU_SVM_RANGE_H__ */
-- 
2.53.0

[PATCH v10 09/22] drm/amdgpu: add SVM notifier invalidate callback and checkpoint

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add invalidate_ranges callback and checkpoint timestamp:
- amdgpu_svm_capture_checkpoint_ts: capture interrupt handler write
  pointer timestamp for stale retry fault filtering
- amdgpu_svm_range_invalidate: invalidate_ranges callback dispatching
  per range notifier begin/end with TLB flush batching, checkpoint
  timestamp capture on MMU_NOTIFY_UNMAP events

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 71 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  6 ++
 2 files changed, 77 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index b907f516cdbe6..62c78c3ce6762 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -730,3 +730,74 @@ void amdgpu_svm_range_unqueue(struct amdgpu_svm *svm,
 	if (put)
 		drm_gpusvm_range_put(&range->base);
 }
+
+/**
+ * amdgpu_svm_capture_checkpoint_ts() - Record the IH ring write pointer time
+ * @svm: The SVM context.
+ *
+ * Capture the timestamp of the most recent interrupt handler write pointer
+ * into @svm->checkpoint_ts. Retry faults whose timestamp predates this
+ * checkpoint are stale and are dropped by the fault handler.
+ */
+void amdgpu_svm_capture_checkpoint_ts(struct amdgpu_svm *svm)
+{
+	struct amdgpu_device *adev = svm->adev;
+	struct amdgpu_ih_ring *ih;
+	uint32_t checkpoint_wptr;
+
+	if (!adev->irq.retry_cam_enabled && adev->irq.ih1.ring_size) {
+		ih = &adev->irq.ih1;
+		checkpoint_wptr = amdgpu_ih_get_wptr(adev, ih);
+		if (ih->rptr != checkpoint_wptr) {
+			WRITE_ONCE(svm->checkpoint_ts,
+				   amdgpu_ih_decode_iv_ts(adev, ih,
+							  checkpoint_wptr, -1));
+			return;
+		}
+	}
+
+	ih = &adev->irq.ih_soft;
+	checkpoint_wptr = amdgpu_ih_get_wptr(adev, ih);
+	if (ih->rptr != checkpoint_wptr)
+		WRITE_ONCE(svm->checkpoint_ts,
+			   amdgpu_ih_decode_iv_ts(adev, ih,
+						  checkpoint_wptr, -1));
+}
+
+/**
+ * amdgpu_svm_range_invalidate() - MMU notifier invalidate_ranges callback
+ * @svm: The SVM context.
+ * @notifier: The GPU SVM notifier covering the invalidated ranges.
+ * @mmu_range: The MMU notifier range describing the event.
+ * @first: First GPU SVM range in the affected interval.
+ * @adj_start: Adjusted start address of the affected interval.
+ * @adj_end: Adjusted end address of the affected interval.
+ *
+ * Dispatch the two phase notifier handling across every range in
+ * [@adj_start, @adj_end): run the begin phase for all ranges, then
+ * the end phase. On an unmap event, also capture a checkpoint timestamp
+ * so stale retry faults can be filtered.
+ */
+void amdgpu_svm_range_invalidate(struct amdgpu_svm *svm,
+				 struct drm_gpusvm_notifier *notifier,
+				 const struct mmu_notifier_range *mmu_range,
+				 struct drm_gpusvm_range *first,
+				 uint64_t adj_start, uint64_t adj_end)
+{
+	struct drm_gpusvm_range *r;
+	bool needs_flush = false;
+
+	if (mmu_range->event == MMU_NOTIFY_UNMAP)
+		amdgpu_svm_capture_checkpoint_ts(svm);
+
+	r = first;
+	drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end)
+		needs_flush |= amdgpu_svm_range_notifier_event_begin(svm, r,
+								     mmu_range);
+	if (needs_flush)
+		amdgpu_svm_flush_tlb(svm);
+
+	r = first;
+	drm_gpusvm_for_each_range(r, notifier, adj_start, adj_end)
+		amdgpu_svm_range_notifier_event_end(svm, r, mmu_range);
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
index f423be277a85a..5e49d774711ce 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -175,6 +175,12 @@ amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
 int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
 			       struct drm_gpusvm_range *range,
 			       struct drm_gpusvm_ctx *ctx);
+void amdgpu_svm_capture_checkpoint_ts(struct amdgpu_svm *svm);
+void amdgpu_svm_range_invalidate(struct amdgpu_svm *svm,
+				 struct drm_gpusvm_notifier *notifier,
+				 const struct mmu_notifier_range *mmu_range,
+				 struct drm_gpusvm_range *first,
+				 uint64_t adj_start, uint64_t adj_end);
 bool amdgpu_svm_range_notifier_event_begin(struct amdgpu_svm *svm,
 					   struct drm_gpusvm_range *range,
 					   const struct mmu_notifier_range *mmu_range);
-- 
2.53.0

[PATCH v10 10/22] drm/amdgpu: implement SVM initialization and lifecycle

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Implement amdgpu_svm.c core module:
- XNACK_OFF/ON helper macros for xnack state checks
- Static amdgpu_svm_cache_lock mutex for slab cache lifecycle
- drm_gpusvm_ops callbacks: range_alloc (kmem_cache), range_free,
  invalidate (dispatches via svm->invalidate_ranges callback)
- kref-based lifecycle: amdgpu_svm_release, amdgpu_svm_put
- PASID lookup: amdgpu_svm_lookup_by_pasid
- Slab cache management: amdgpu_svm_cache_init/fini
- Ioctl operation wrappers: op_set_attr, op_get_attr, op_reset_attr
- Attribute change detection and application:
  attr_change_trigger classifies changes into trigger types,
  amdgpu_svm_apply_attr_change dispatches invalidate or remap
  based on trigger flags and xnack state
- Hardware detection: amdgpu_svm_default_xnack_enabled per GC IP
- TLB flushing with logical, partition-scoped XCC masks
- xnack mode: amdgpu_svm_init_xnack_mode validates requested mode
- Initialization: amdgpu_svm_init_with_ops (drm_gpusvm_init with
  2M/64K/4K chunk sizes, attr tree, invalidate_ranges/flush_tlb
  callbacks), amdgpu_svm_init_compute with xnack_mode parameter
- Teardown: amdgpu_svm_close (mark exiting, sync work),
  amdgpu_svm_fini (gpusvm_fini, destroy attr tree, release ref)

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 641 ++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  |   2 +-
 2 files changed, 642 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
new file mode 100644
index 0000000000000..f5ed917329944
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -0,0 +1,641 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include <linux/sched/mm.h>
+#include <linux/uaccess.h>
+#include <linux/xarray.h>
+
+#include <drm/drm_file.h>
+
+#include "amdgpu.h"
+#include "amdgpu_ih.h"
+#include "amdgpu_reset.h"
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+#include "amdgpu_svm_fault.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu_vm.h"
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+
+#define AMDGPU_SVM_MAX_ATTRS 64
+#define AMDGPU_SVM_DEFAULT_SVM_NOTIFIER_SIZE 512
+
+static const unsigned long amdgpu_svm_chunk_sizes[] = {
+	SZ_2M,
+	SZ_64K,
+	SZ_4K,
+};
+
+#define AMDGPU_SVM_GC_WQ_NAME "amdgpu_svm_gc"
+#define XNACK_OFF(svm)	((svm)->xnack_enabled == false)
+#define XNACK_ON(svm)	((svm)->xnack_enabled == true)
+
+/**
+ * amdgpu_svm_invalidate() - drm_gpusvm invalidate callback
+ * @gpusvm: The drm_gpusvm instance.
+ * @notifier: The GPU SVM notifier reporting the event.
+ * @mmu_range: The MMU notifier range describing the invalidation.
+ *
+ * Clamp the event to the notifier window, find the first affected range and
+ * dispatch to the driver's @invalidate_ranges handler.
+ */
+static void amdgpu_svm_invalidate(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_notifier *notifier,
+				  const struct mmu_notifier_range *mmu_range)
+{
+	struct amdgpu_svm *svm = to_amdgpu_svm(gpusvm);
+	struct drm_gpusvm_range *first;
+	uint64_t adj_start = mmu_range->start, adj_end = mmu_range->end;
+
+	amdgpu_svm_assert_in_notifier(svm);
+
+	pr_debug(
+		"INVALIDATE: pasid=%u, gpusvm=%p, seqno=%lu, [0x%016lx-0x%016lx]-0x%lx, ev=%d\n",
+		svm->vm->pasid, &svm->gpusvm,
+		notifier->notifier.invalidate_seq,
+			 mmu_range->start, mmu_range->end,
+			 mmu_range->end - mmu_range->start, mmu_range->event);
+
+	if (mmu_range->event == MMU_NOTIFY_RELEASE)
+		return;
+	if (atomic_read(&svm->exiting))
+		return;
+
+	adj_start = max(drm_gpusvm_notifier_start(notifier), adj_start);
+	adj_end = min(drm_gpusvm_notifier_end(notifier), adj_end);
+
+	first = drm_gpusvm_range_find(notifier, adj_start, adj_end);
+	if (!first)
+		return;
+
+	svm->invalidate_ranges(svm, notifier, mmu_range, first,
+			       adj_start, adj_end);
+}
+
+static struct drm_gpusvm_range *amdgpu_svm_range_alloc(struct drm_gpusvm *gpusvm)
+{
+	struct amdgpu_svm_range *range;
+
+	range = kzalloc_obj(*range, GFP_KERNEL);
+	if (!range)
+		return NULL;
+
+	INIT_LIST_HEAD(&range->work_node);
+	range->pending_start_page = ULONG_MAX;
+	return &range->base;
+}
+
+static void amdgpu_svm_range_free(struct drm_gpusvm_range *range)
+{
+	kfree(to_amdgpu_svm_range(range));
+}
+
+static const struct drm_gpusvm_ops amdgpu_gpusvm_ops = {
+	.range_alloc = amdgpu_svm_range_alloc,
+	.range_free = amdgpu_svm_range_free,
+	.invalidate = amdgpu_svm_invalidate,
+};
+
+static void amdgpu_svm_release(struct kref *ref)
+{
+	kfree(container_of(ref, struct amdgpu_svm, refcount));
+}
+
+/**
+ * amdgpu_svm_put() - Drop a reference on an SVM context
+ * @svm: The SVM context.
+ *
+ * Release a reference taken on @svm and free it once the last reference is
+ * dropped.
+ */
+void amdgpu_svm_put(struct amdgpu_svm *svm)
+{
+	if (svm)
+		kref_put(&svm->refcount, amdgpu_svm_release);
+}
+
+/**
+ * amdgpu_svm_lookup_by_pasid() - Find the SVM context for a PASID
+ * @adev: The amdgpu device.
+ * @pasid: The PASID to look up.
+ *
+ * Look up the VM bound to @pasid and return its SVM context with a reference
+ * taken. The caller must drop it with amdgpu_svm_put().
+ *
+ * Return: The referenced SVM context, or %NULL if none is bound.
+ */
+struct amdgpu_svm *
+amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev, uint32_t pasid)
+{
+	struct amdgpu_svm *svm = NULL;
+	struct amdgpu_fpriv *fpriv;
+	struct amdgpu_vm *vm;
+	unsigned long irqflags;
+
+	amdgpu_pasid_lock(&irqflags);
+	fpriv = amdgpu_pasid_get_fpriv_locked(pasid);
+	vm = fpriv ? &fpriv->vm : NULL;
+	if (vm && vm->svm) {
+		svm = vm->svm;
+		kref_get(&svm->refcount);
+	}
+	amdgpu_pasid_unlock(irqflags);
+
+	return svm;
+}
+
+static int amdgpu_svm_op_set_attr(struct amdgpu_vm *vm,
+				  uint64_t start,
+				  uint64_t size,
+				  uint32_t nattr,
+				  const struct drm_amdgpu_svm_attribute *attrs)
+{
+	struct amdgpu_svm *svm = vm->svm;
+
+	amdgpu_svm_sync_work(svm);
+
+	return amdgpu_svm_attr_set(svm->attr_tree, start, size, nattr,
+				   attrs);
+}
+
+static int amdgpu_svm_op_get_attr(struct amdgpu_vm *vm,
+				  uint64_t start,
+				  uint64_t size,
+				  uint32_t nattr,
+				  struct drm_amdgpu_svm_attribute *attrs,
+				  uint64_t *out_size)
+{
+	amdgpu_svm_sync_work(vm->svm);
+
+	return amdgpu_svm_attr_get(vm->svm->attr_tree, start, size, nattr,
+				   attrs, out_size);
+}
+
+static int amdgpu_svm_op_reset_attr(struct amdgpu_vm *vm,
+				    uint64_t start, uint64_t size)
+{
+	struct amdgpu_svm *svm = vm->svm;
+	unsigned long start_page = start >> PAGE_SHIFT;
+	unsigned long last_page = (start + size - 1) >> PAGE_SHIFT;
+
+	amdgpu_svm_sync_work(svm);
+
+	return amdgpu_svm_attr_reset(svm->attr_tree,
+				     start_page, last_page);
+}
+
+/**
+ * attr_change_trigger() - Classify what an attribute update changed
+ * @old_attrs: Attributes before the update.
+ * @new_attrs: Attributes after the update.
+ *
+ * Compare the two attribute sets and return an
+ * amdgpu_svm_attr_change_trigger bitmask describing which aspects changed:
+ * access, PTE flags, mapping flags, location, granularity, prefetch.
+ *
+ * Return: The trigger bitmask.
+ */
+static uint32_t
+attr_change_trigger(const struct amdgpu_svm_attrs *old_attrs,
+		    const struct amdgpu_svm_attrs *new_attrs)
+{
+	uint32_t trigger = 0;
+	uint32_t changed_flags = old_attrs->flags ^ new_attrs->flags;
+
+	if (old_attrs->access != new_attrs->access)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE;
+	if (changed_flags & AMDGPU_SVM_PTE_FLAG_MASK)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE;
+	if (changed_flags & AMDGPU_SVM_MAPPING_FLAG_MASK)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE;
+	if (old_attrs->preferred_loc != new_attrs->preferred_loc ||
+	    old_attrs->prefetch_loc != new_attrs->prefetch_loc)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE;
+	if (old_attrs->granularity != new_attrs->granularity)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE;
+	if (new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
+	    new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_SYSMEM)
+		trigger |= AMDGPU_SVM_ATTR_TRIGGER_PREFETCH;
+
+	return trigger;
+}
+
+/**
+ * amdgpu_svm_apply_attr_change() - React to an attribute change on a range
+ * @svm: The SVM context.
+ * @old_attrs: Attributes before the change.
+ * @new_attrs: Attributes after the change.
+ * @start_page: First page of the affected interval.
+ * @last_page: Last page of the affected interval.
+ *
+ * Classify the change and act on it: when XNACK is on and the change affects
+ * existing GPU mappings, invalidate the interval; when the new attributes
+ * request a prefetch, map / remap the interval with the new attributes.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
+				 const struct amdgpu_svm_attrs *old_attrs,
+				 const struct amdgpu_svm_attrs *new_attrs,
+				 unsigned long start_page,
+				 unsigned long last_page)
+{
+	bool old_access, new_access;
+	bool needs_invalidate = false;
+	bool needs_mapping = false;
+	uint32_t trigger;
+	int ret;
+
+	amdgpu_svm_assert_locked(svm);
+
+	if (!start_page && !last_page)
+		return 0;
+
+	trigger = attr_change_trigger(old_attrs, new_attrs);
+	old_access = amdgpu_svm_attr_has_access(old_attrs->access);
+	new_access = amdgpu_svm_attr_has_access(new_attrs->access);
+	if (XNACK_ON(svm) &&
+	    (trigger & AMDGPU_SVM_ATTR_TRIGGER_NEED_INVALIDATE))
+		needs_invalidate = true;
+
+	if (trigger & AMDGPU_SVM_ATTR_TRIGGER_PREFETCH)
+		needs_mapping = true;
+
+	if (!trigger && !needs_mapping)
+		return 0;
+
+	pr_debug("attr change trigger=0x%x old=%d new=%d [0x%lx-0x%lx]-0x%lx, xnack=%d\n",
+			 trigger, old_access, new_access, start_page, last_page,
+			 last_page - start_page + 1,
+			 svm->xnack_enabled ? 1 : 0);
+
+	if (needs_invalidate) {
+		pr_debug("attr change invalidate [0x%lx-0x%lx]-0x%lx trigger=0x%x\n",
+				 start_page, last_page,
+				 last_page - start_page + 1, trigger);
+		ret = amdgpu_svm_range_invalidate_interval(svm, start_page,
+							   last_page);
+		if (ret) {
+			pr_err_ratelimited("failed to invalidate range for attr change: [0x%lx-0x%lx], ret=%d\n",
+					   start_page, last_page, ret);
+			return ret;
+		}
+	}
+
+	if (!needs_mapping)
+		return 0;
+
+	return amdgpu_svm_range_map_attrs(svm, new_attrs,
+					  start_page << PAGE_SHIFT,
+					  (last_page + 1) << PAGE_SHIFT);
+}
+
+bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm)
+{
+	if (svm->adev->apu_prefer_gtt)
+		return false;
+
+	/* TODO: add amdgpu_pagemap_capable() */
+
+	return false;
+}
+
+/**
+ * amdgpu_svm_default_xnack_enabled() - Whether XNACK defaults to on for the HW
+ * @adev: The amdgpu device.
+ *
+ * Decide the default retry fault (XNACK) policy from the GC IP version and
+ * platform constraints.
+ *
+ * Return: true if XNACK should default to enabled.
+ */
+static bool amdgpu_svm_default_xnack_enabled(struct amdgpu_device *adev)
+{
+	uint32_t gc_ver = amdgpu_ip_version(adev, GC_HWIP, 0);
+
+	if (gc_ver < IP_VERSION(9, 0, 1))
+		return false;
+	if (!amdgpu_sriov_xnack_support(adev))
+		return false;
+
+	if (adev->gmc.noretry)
+		return false;
+
+	switch (gc_ver) {
+	case IP_VERSION(9, 4, 2):
+	case IP_VERSION(9, 4, 3):
+	case IP_VERSION(9, 4, 4):
+	case IP_VERSION(9, 5, 0):
+		return true;
+	default:
+		break;
+	}
+	if (gc_ver >= IP_VERSION(10, 1, 1))
+		return false;
+
+	return true;
+}
+
+void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm)
+{
+	struct amdgpu_fpriv *fpriv = container_of(svm->vm, struct amdgpu_fpriv, vm);
+	struct amdgpu_device *adev = svm->adev;
+	struct amdgpu_xcp_mgr *xcp_mgr = adev->xcp_mgr;
+	u32 xcc_mask;
+	int r;
+
+	if (!xcp_mgr || fpriv->xcp_id >= xcp_mgr->num_xcps ||
+	    amdgpu_xcp_get_inst_details(&xcp_mgr->xcp[fpriv->xcp_id],
+					AMDGPU_XCP_GFX, &xcc_mask))
+		xcc_mask = (1U << NUM_XCC(adev->gfx.xcc_mask)) - 1;
+
+	r = amdgpu_vm_flush_compute_tlb(adev, svm->vm, TLB_FLUSH_HEAVYWEIGHT,
+					xcc_mask);
+	if (r)
+		dev_err(adev->dev, "SVM TLB flush failed (%d), GPU may use stale translations\n",
+			r);
+}
+
+static int amdgpu_svm_work_init(struct amdgpu_svm *svm,
+				void (*gc_work_func)(struct work_struct *));
+static void amdgpu_svm_work_fini(struct amdgpu_svm *svm);
+
+/**
+ * amdgpu_svm_init_xnack_mode() - Resolve the requested XNACK mode
+ * @adev: The amdgpu device.
+ * @mode: The requested XNACK mode.
+ * @xnack_enabled: Output, set to the resolved enable state.
+ *
+ * Validate @mode against the hardware default: DEFAULT follows the HW policy,
+ * ON is rejected if the HW does not support it, OFF always disables.
+ *
+ * Return: 0 on success, -EOPNOTSUPP if ON is unavailable, -EINVAL on a bad
+ * mode.
+ */
+static int amdgpu_svm_init_xnack_mode(struct amdgpu_device *adev,
+				    enum amdgpu_svm_xnack_mode mode,
+				    bool *xnack_enabled)
+{
+	bool xnack_default = amdgpu_svm_default_xnack_enabled(adev);
+
+	switch (mode) {
+	case AMDGPU_SVM_XNACK_DEFAULT:
+		*xnack_enabled = xnack_default;
+		break;
+	case AMDGPU_SVM_XNACK_ON:
+		if (!xnack_default) {
+			pr_err("xnack on not available (mode=%d)\n",
+					mode);
+			*xnack_enabled = xnack_default;
+			return -EOPNOTSUPP;
+		}
+		*xnack_enabled = true;
+		break;
+	case AMDGPU_SVM_XNACK_OFF:
+		*xnack_enabled = false;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+/**
+ * amdgpu_svm_init_with_ops() - Initialize the SVM core with driver callbacks
+ * @svm: The SVM context to initialize.
+ * @invalidate_ranges: Callback invoked from the MMU notifier path.
+ * @gc_work_func: Work function draining the garbage collector.
+ *
+ * Set up the work queues, attribute tree and the embedded drm_gpusvm (with
+ * the 2M/64K/4K chunk sizes and the driver lock), wiring the supplied
+ * callbacks.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int amdgpu_svm_init_with_ops(struct amdgpu_svm *svm,
+				    void (*invalidate_ranges)(struct amdgpu_svm *,
+						struct drm_gpusvm_notifier *,
+						const struct mmu_notifier_range *,
+						struct drm_gpusvm_range *,
+						uint64_t, uint64_t),
+				    void (*gc_work_func)(struct work_struct *))
+{
+	struct amdgpu_device *adev = svm->adev;
+	int ret;
+
+	svm->invalidate_ranges = invalidate_ranges;
+
+	ret = amdgpu_svm_work_init(svm, gc_work_func);
+	if (ret)
+		return ret;
+
+	svm->attr_tree = amdgpu_svm_attr_tree_create(svm);
+	if (!svm->attr_tree) {
+		ret = -ENOMEM;
+		goto err_work_fini;
+	}
+
+	ret = drm_gpusvm_init(&svm->gpusvm, "AMDGPU SVM",
+						adev_to_drm(adev), current->mm, 0,
+						adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT,
+						AMDGPU_SVM_DEFAULT_SVM_NOTIFIER_SIZE * SZ_1M,
+						&amdgpu_gpusvm_ops,
+						amdgpu_svm_chunk_sizes,
+						ARRAY_SIZE(amdgpu_svm_chunk_sizes));
+
+	if (ret)
+		goto err_attr_tree_destroy;
+
+	drm_gpusvm_driver_set_lock(&svm->gpusvm, &svm->svm_lock);
+
+	return 0;
+
+err_attr_tree_destroy:
+	amdgpu_svm_attr_tree_destroy(svm->attr_tree);
+err_work_fini:
+	amdgpu_svm_work_fini(svm);
+	return ret;
+}
+
+static void amdgpu_svm_gc_work_func(struct work_struct *w);
+
+/**
+ * amdgpu_svm_init_compute() - Create the SVM context for a compute VM
+ * @adev: The amdgpu device.
+ * @vm: The VM to attach the SVM context to.
+ * @xnack_mode: The requested XNACK mode.
+ *
+ * Allocate and initialize an SVM context for @vm (idempotent if one already
+ * exists), resolving the XNACK mode and wiring the compute callbacks. XNACK
+ * off is not supported yet.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int amdgpu_svm_init_compute(struct amdgpu_device *adev,
+				   struct amdgpu_vm *vm,
+				   enum amdgpu_svm_xnack_mode xnack_mode)
+{
+	struct amdgpu_svm *svm;
+	int ret;
+
+	if (vm->svm)
+		return 0;
+
+	svm = kzalloc_obj(*svm, GFP_KERNEL);
+	if (!svm)
+		return -ENOMEM;
+
+	kref_init(&svm->refcount);
+	svm->adev = adev;
+	svm->vm = vm;
+	svm->default_granularity = min_t(u8, amdgpu_svm_default_granularity, 0x1B);
+	atomic_set(&svm->exiting, 0);
+
+	ret = amdgpu_svm_init_xnack_mode(adev, xnack_mode,
+					  &svm->xnack_enabled);
+	if (ret)
+		goto err_free;
+
+	if (svm->xnack_enabled) {
+		ret = amdgpu_svm_init_with_ops(svm,
+					       amdgpu_svm_range_invalidate,
+					       amdgpu_svm_gc_work_func);
+	} else {
+		pr_err("xnack off is not supported yet\n");
+		ret = -EOPNOTSUPP;
+	}
+
+	if (ret)
+		goto err_free;
+
+	pr_debug("AMDGPU SVM initialized: default granularity 0x%lx bytes, xnack: %s\n",
+	       1UL << (svm->default_granularity + PAGE_SHIFT),
+	       svm->xnack_enabled ? "enabled" : "disabled");
+
+	vm->svm = svm;
+	return 0;
+
+err_free:
+	kfree(svm);
+	return ret;
+}
+
+/**
+ * amdgpu_svm_init() - Initialize SVM for a VM
+ * @adev: The amdgpu device.
+ * @vm: The VM to enable SVM on.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
+{
+	/* graphics svm init maybe different */
+
+	return amdgpu_svm_init_compute(adev, vm, AMDGPU_SVM_XNACK_DEFAULT);
+}
+
+/**
+ * amdgpu_svm_drain_retry_fault() - Wait for retry faults to drain
+ * @adev: The amdgpu device.
+ *
+ * Wait until the interrupt handler has processed up to the current checkpoint
+ * on the relevant IH rings, so no stale retry faults remain in flight. Skips
+ * draining during a GPU reset or if the reset domain cannot be entered.
+ */
+static void amdgpu_svm_drain_retry_fault(struct amdgpu_device *adev)
+{
+	if (!adev)
+		return;
+
+	if (amdgpu_in_reset(adev))
+		return;
+
+	if (!down_read_trylock(&adev->reset_domain->sem))
+		return;
+
+	amdgpu_ih_wait_on_checkpoint_process_ts(adev,
+		adev->irq.retry_cam_enabled ?
+			&adev->irq.ih : &adev->irq.ih1);
+	if (adev->irq.retry_cam_enabled)
+		amdgpu_ih_wait_on_checkpoint_process_ts(adev,
+			&adev->irq.ih_soft);
+
+	up_read(&adev->reset_domain->sem);
+}
+
+/**
+ * amdgpu_svm_close() - Begin SVM teardown for a VM
+ * @vm: The VM whose SVM context is closing.
+ *
+ * Mark the context as exiting (once), flush pending GC work and drain
+ * in-flight retry faults. Safe to call on a VM without an SVM context.
+ */
+void amdgpu_svm_close(struct amdgpu_vm *vm)
+{
+	struct amdgpu_svm *svm = vm->svm;
+
+	if (!svm)
+		return;
+
+	if (atomic_xchg(&svm->exiting, 1))
+		return;
+
+	amdgpu_svm_sync_work(svm);
+	amdgpu_svm_drain_retry_fault(svm->adev);
+}
+
+/**
+ * amdgpu_svm_fini() - Finalize and release a VM's SVM context
+ * @vm: The VM whose SVM context is being torn down.
+ *
+ * Close the context, tear down the embedded drm_gpusvm under the SVM lock,
+ * destroy the attribute tree and work queues, and drop the context
+ * reference. Safe to call on a VM without an SVM context.
+ */
+void amdgpu_svm_fini(struct amdgpu_vm *vm)
+{
+	struct amdgpu_svm *svm = vm->svm;
+
+	if (!svm)
+		return;
+
+	amdgpu_svm_close(vm);
+	amdgpu_svm_lock(svm);
+	drm_gpusvm_fini(&svm->gpusvm);
+	amdgpu_svm_unlock(svm);
+
+	amdgpu_svm_attr_tree_destroy(svm->attr_tree);
+	amdgpu_svm_work_fini(svm);
+	vm->svm = NULL;
+	amdgpu_svm_put(svm);
+}
+
+bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
+{
+	return vm->svm != NULL;
+}
+
+#endif /* CONFIG_DRM_AMDGPU_SVM */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index c22c804f7d716..4fb22195607a5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -1780,7 +1780,7 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev,
  * @adev: amdgpu_device pointer
  * @vm: requested vm
  * @flush_type: flush type
- * @xcc_mask: mask of XCCs that belong to the compute partition in need of a TLB flush.
+ * @xcc_mask: mask of logical XCC indices to flush
  *
  * Flush TLB if needed for a compute VM.
  *
-- 
2.53.0

[PATCH v10 11/22] drm/amdgpu: add SVM ioctl entry and fault handler module

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Add ioctl entry point and fault handling:

amdgpu_svm.c additions:
- amdgpu_svm_copy_attrs: memdup_user for attribute array
- amdgpu_svm_garbage_collector: dequeue and remove unmapped ranges
- amdgpu_svm_clean_queue: drain work list releasing refs
- amdgpu_svm_gc_work_func: static GC work handler
- amdgpu_svm_gc_init/fini/flush: static workqueue lifecycle
- amdgpu_svm_work_init/work_fini: wrappers for GC lifecycle
- amdgpu_svm_sync_work: wrapper for GC flush
- amdgpu_gem_svm_ioctl: validate args, dispatch
  SET_ATTR/GET_ATTR/RESET_ATTR with copy_to_user for GET

amdgpu_svm_fault.c (new):
- fault_check_allowed: VMA permission check for fault address
- fault_get_unregistered_attrs: create default attr range for
  faulting address not covered by explicit attributes, bounded
  by VMA and granularity alignment
- fault_map_range: fault mapping pipeline with retry, GC,
  permission check, find/insert range, timestamp dedup,
  validity check, get_pages, update_mapping.
- amdgpu_svm_handle_fault: top level entry point from VM fault path

The VM fault path treats zero as a successfully serviced or benignly
dropped fault. Return an error for faults which cannot be serviced so the
caller installs AMDGPU_VM_NORETRY_FLAGS and terminates the faulting wave.

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c       | 196 ++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 467 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h |  39 ++
 3 files changed, 702 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
index f5ed917329944..c7810e6d3035f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -638,4 +638,200 @@ bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
 	return vm->svm != NULL;
 }
 
+static int amdgpu_svm_copy_attrs(const struct drm_amdgpu_gem_svm *args,
+					   struct drm_amdgpu_svm_attribute **attrs,
+					   size_t *size)
+{
+	if (!args->nattr || args->nattr > AMDGPU_SVM_MAX_ATTRS)
+		return -EINVAL;
+	if (!args->attrs_ptr)
+		return -EINVAL;
+
+	*size = args->nattr * sizeof(**attrs);
+	*attrs = memdup_user(u64_to_user_ptr(args->attrs_ptr), *size);
+
+	return PTR_ERR_OR_ZERO(*attrs);
+}
+
+/**
+ * amdgpu_svm_garbage_collector() - Remove ranges queued for unmapping
+ * @svm: The SVM context.
+ *
+ * Drain the GC list: for each queued range, remove it from the gpusvm trees
+ * and drop the reference held while queued. Caller must hold the SVM lock.
+ *
+ * Return: 0 on success.
+ */
+int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm)
+{
+	struct amdgpu_svm_range_op_ctx op_ctx;
+
+	amdgpu_svm_assert_locked(svm);
+
+	spin_lock(&svm->work_lock);
+	while (amdgpu_svm_range_dequeue_locked(svm, &svm->gc.list, &op_ctx)) {
+		spin_unlock(&svm->work_lock);
+
+		WARN_ON(!(op_ctx.pending_ops & AMDGPU_SVM_RANGE_OP_UNMAP));
+
+		drm_gpusvm_range_remove(&svm->gpusvm,
+					&op_ctx.range->base);
+
+		amdgpu_svm_range_put_if_dequeued(svm, op_ctx.range);
+		spin_lock(&svm->work_lock);
+	}
+	spin_unlock(&svm->work_lock);
+	return 0;
+}
+
+/**
+ * amdgpu_svm_clean_queue() - Drain a work list, releasing range references
+ * @svm: The SVM context.
+ * @work_list: The work list to drain.
+ *
+ * Dequeue every range on @work_list and drop the reference held while it was
+ * queued, without performing any unmapping. Used during teardown.
+ */
+void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
+			   struct list_head *work_list)
+{
+	struct amdgpu_svm_range_op_ctx op_ctx;
+
+	spin_lock(&svm->work_lock);
+	while (amdgpu_svm_range_dequeue_locked(svm, work_list,
+						 &op_ctx)) {
+		spin_unlock(&svm->work_lock);
+		amdgpu_svm_range_put_if_dequeued(svm, op_ctx.range);
+		spin_lock(&svm->work_lock);
+	}
+	spin_unlock(&svm->work_lock);
+}
+
+static void amdgpu_svm_gc_work_func(struct work_struct *w)
+{
+	struct amdgpu_svm_gc *gc = container_of(w, struct amdgpu_svm_gc, work);
+	struct amdgpu_svm *svm = container_of(gc, struct amdgpu_svm, gc);
+
+	amdgpu_svm_lock(svm);
+	amdgpu_svm_garbage_collector(svm);
+	amdgpu_svm_unlock(svm);
+}
+
+static int amdgpu_svm_gc_init(struct amdgpu_svm *svm,
+			      void (*gc_work_func)(struct work_struct *))
+{
+	svm->gc.wq = alloc_workqueue(AMDGPU_SVM_GC_WQ_NAME,
+					WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM, 0);
+	if (!svm->gc.wq)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&svm->gc.list);
+	INIT_WORK(&svm->gc.work, gc_work_func);
+
+	return 0;
+}
+
+static void amdgpu_svm_gc_fini(struct amdgpu_svm *svm)
+{
+	flush_work(&svm->gc.work);
+	amdgpu_svm_clean_queue(svm, &svm->gc.list);
+	destroy_workqueue(svm->gc.wq);
+	svm->gc.wq = NULL;
+}
+
+static void amdgpu_svm_gc_flush(struct amdgpu_svm *svm)
+{
+	flush_work(&svm->gc.work);
+}
+
+static int amdgpu_svm_work_init(struct amdgpu_svm *svm,
+				void (*gc_work_func)(struct work_struct *))
+{
+	int ret;
+
+	init_rwsem(&svm->svm_lock);
+	spin_lock_init(&svm->work_lock);
+
+	ret = amdgpu_svm_gc_init(svm, gc_work_func);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static void amdgpu_svm_work_fini(struct amdgpu_svm *svm)
+{
+	amdgpu_svm_gc_fini(svm);
+}
+
+void amdgpu_svm_sync_work(struct amdgpu_svm *svm)
+{
+	amdgpu_svm_gc_flush(svm);
+}
+
+/**
+ * amdgpu_gem_svm_ioctl() - SVM GEM ioctl entry point
+ * @dev: The DRM device.
+ * @data: The ioctl argument (struct drm_amdgpu_gem_svm).
+ * @filp: The DRM file.
+ *
+ * Validate the arguments and dispatch to the SET_ATTR / GET_ATTR /
+ * RESET_ATTR operation, copying results back to user space for GET.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
+			 struct drm_file *filp)
+{
+	struct amdgpu_fpriv *fpriv = filp->driver_priv;
+	struct drm_amdgpu_gem_svm *args = data;
+	struct drm_amdgpu_svm_attribute *attrs = NULL;
+	struct amdgpu_vm *vm;
+	size_t attrs_size = 0;
+	int ret = 0;
+
+	pr_debug("ioctl op=%u va:[0x%llx-0x%llx)-0x%llx nattr=%u\n",
+			 args->operation, args->start_addr, args->start_addr + args->size,
+			 args->size, args->nattr);
+
+	vm = &fpriv->vm;
+	if (!amdgpu_svm_is_enabled(vm))
+		return -EOPNOTSUPP;
+
+	if ((args->start_addr & ~PAGE_MASK) || (args->size & ~PAGE_MASK))
+		return -EINVAL;
+
+	if (!args->start_addr || !args->size)
+		return -EINVAL;
+
+	if (args->operation != AMDGPU_SVM_OP_RESET_ATTR) {
+		ret = amdgpu_svm_copy_attrs(args, &attrs, &attrs_size);
+		if (ret)
+			return ret;
+	}
+
+	switch (args->operation) {
+	case AMDGPU_SVM_OP_SET_ATTR:
+		ret = amdgpu_svm_op_set_attr(vm, args->start_addr, args->size,
+					 args->nattr, attrs);
+		break;
+	case AMDGPU_SVM_OP_GET_ATTR:
+		ret = amdgpu_svm_op_get_attr(vm, args->start_addr, args->size,
+					 args->nattr, attrs, &args->size);
+		if (!ret && copy_to_user(u64_to_user_ptr(args->attrs_ptr),
+					 attrs, attrs_size))
+			ret = -EFAULT;
+		break;
+	case AMDGPU_SVM_OP_RESET_ATTR:
+		ret = amdgpu_svm_op_reset_attr(vm, args->start_addr, args->size);
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
+
+	kvfree(attrs);
+	return ret;
+}
+
 #endif /* CONFIG_DRM_AMDGPU_SVM */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
new file mode 100644
index 0000000000000..2dd77fe78303c
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
@@ -0,0 +1,467 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_attr.h"
+#include "amdgpu_svm_fault.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu.h"
+#include "amdgpu_vm.h"
+#include "amdgpu_gmc.h"
+#include "amdgpu_ih.h"
+#include "amdgpu_trace.h"
+
+#include <drm/drm_exec.h>
+#include <drm/drm_gpusvm.h>
+
+#include <linux/delay.h>
+#include <linux/mm.h>
+#include <linux/sched/mm.h>
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+
+#define AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING	(2UL * NSEC_PER_MSEC)
+
+/**
+ * fault_get_unregistered_attrs() - Create a default attr range for a fault
+ * @svm: The SVM context.
+ * @fault_addr: The faulting address.
+ * @attr_start_page: Lower bound (first page) of the surrounding attribute gap.
+ * @attr_last_page: Upper bound (last page) of the surrounding attribute gap.
+ * @out_attrs: Output, a snapshot of the attributes for the new range.
+ * @out_start_page: Output, first page of the new range.
+ * @out_last_page: Output, last page of the new range.
+ *
+ * For a fault that lands in an address gap not covered by any explicit
+ * attribute range, create a default attribute range. Its bounds are the
+ * faulting page's granularity aligned window, clamped to the CPU VMA and to
+ * the surrounding attribute gap [@attr_start_page, @attr_last_page].
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int fault_get_unregistered_attrs(struct amdgpu_svm *svm,
+					    unsigned long fault_addr,
+					    unsigned long attr_start_page,
+					    unsigned long attr_last_page,
+					    struct amdgpu_svm_attrs *out_attrs,
+					    unsigned long *out_start_page,
+					    unsigned long *out_last_page)
+{
+	struct amdgpu_svm_attr_tree *attr_tree = svm->attr_tree;
+	struct amdgpu_svm_attr_range *range;
+	struct amdgpu_svm_attrs attrs;
+	struct mm_struct *mm = svm->gpusvm.mm;
+	struct vm_area_struct *vma;
+	unsigned long fault_page = fault_addr >> PAGE_SHIFT;
+	unsigned long start_page, last_page;
+	unsigned long vma_start_page, vma_last_page;
+
+	amdgpu_svm_attr_set_default(svm, &attrs);
+
+	mmap_read_lock(mm);
+
+	vma = amdgpu_svm_check_vma(mm, fault_addr);
+	if (IS_ERR(vma)) {
+		mmap_read_unlock(mm);
+		pr_debug("get_unregistered_attrs: invalid VMA for fault_addr=0x%lx\n",
+			 fault_addr);
+		return PTR_ERR(vma);
+	}
+	vma_start_page = vma->vm_start >> PAGE_SHIFT;
+	vma_last_page = (vma->vm_end >> PAGE_SHIFT) - 1;
+
+	if (vma_is_initial_heap(vma) || vma_is_initial_stack(vma))
+		attrs.preferred_loc = AMDGPU_SVM_LOCATION_SYSMEM;
+
+	mmap_read_unlock(mm);
+
+	start_page = max_t(unsigned long, vma_start_page,
+			   ALIGN_DOWN(fault_page, 1UL << attrs.granularity));
+	last_page = min_t(unsigned long, vma_last_page,
+			  ALIGN(fault_page + 1, 1UL << attrs.granularity) - 1);
+
+	start_page = max(start_page, attr_start_page);
+	last_page = min(last_page, attr_last_page);
+
+	mutex_lock(&attr_tree->lock);
+	range = amdgpu_svm_attr_range_alloc(start_page, last_page, &attrs);
+	if (!range) {
+		mutex_unlock(&attr_tree->lock);
+		return -ENOMEM;
+	}
+	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
+	*out_attrs = range->attrs;
+	*out_start_page = amdgpu_svm_attr_start_page(range);
+	*out_last_page = amdgpu_svm_attr_last_page(range);
+	mutex_unlock(&attr_tree->lock);
+
+	pr_debug(
+		"Created unregistered range for fault_addr=0x%lx: attr range=[0x%lx-0x%lx] size: 0x%lx attrs={preferred_loc=%d, prefetch_loc=%d, flags=0x%x, granularity=%u, access=%u}\n",
+		fault_addr, *out_start_page, *out_last_page + 1,
+		*out_last_page - *out_start_page + 1,
+		out_attrs->preferred_loc, out_attrs->prefetch_loc,
+		out_attrs->flags, out_attrs->granularity,
+		out_attrs->access);
+
+	return 0;
+}
+
+/**
+ * fault_check_allowed() - Verify the CPU VMA
+ * @svm: The SVM context.
+ * @fault_addr: The faulting address.
+ * @write_fault: True if the fault was caused by a write.
+ *
+ * Check that the VMA covering @fault_addr grants the requested access
+ *
+ * Return: 0 if allowed, -EPERM if the required permission is missing.
+ */
+static int fault_check_allowed(struct amdgpu_svm *svm,
+				   unsigned long fault_addr, bool write_fault)
+{
+	struct mm_struct *mm = svm->gpusvm.mm;
+	struct vm_area_struct *vma;
+	unsigned long requested = VM_READ;
+	int ret = 0;
+
+	if (write_fault)
+		requested |= VM_WRITE;
+
+	mmap_read_lock(mm);
+	vma = vma_lookup(mm, fault_addr);
+	if (vma && (vma->vm_flags & requested) != requested) {
+		pr_err_ratelimited("fault addr 0x%lx no %s permission\n",
+				   fault_addr, write_fault ? "write" : "read");
+		ret = -EPERM;
+	}
+	mmap_read_unlock(mm);
+
+	return ret;
+}
+
+/**
+ * fault_map_range() - Map the faulting range into the GPU page tables
+ * @svm: The SVM context.
+ * @fault_addr: The faulting address.
+ * @attrs: Snapshot of the attributes covering @fault_addr.
+ * @attr_start_page: First page covered by @attrs.
+ * @attr_last_page: Last page covered by @attrs.
+ * @write_fault: True if the fault was caused by a write.
+ *
+ * Run the fault mapping pipeline under the SVM lock: drain the garbage
+ * collector, check VMA permissions, find or insert the gpusvm range, skip
+ * work that a concurrent fault already handled, collect the pages and do
+ * the GPU mapping. Transient failures (HMM retry, changed CPU mappings)
+ * are retried a bounded number of times.
+ *
+ * Return: 0 on success or a benign drop, negative error code on failure.
+ */
+static int fault_map_range(struct amdgpu_svm *svm,
+			       unsigned long fault_addr,
+			       const struct amdgpu_svm_attrs *attrs,
+			       unsigned long attr_start_page,
+			       unsigned long attr_last_page,
+			       bool write_fault)
+{
+	bool devmem_possible = amdgpu_svm_devmem_possible(svm);
+	bool need_vram_migration = devmem_possible &&
+				   amdgpu_svm_attr_prefer_vram(attrs);
+	struct drm_gpusvm_ctx map_ctx = {
+		.read_only = !!(attrs->flags & AMDGPU_SVM_ATTR_BIT_GPU_RO),
+		.devmem_possible = devmem_possible,
+		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
+		.devmem_only = need_vram_migration,
+		.timeslice_ms = need_vram_migration ? 5 : 0,
+	};
+	struct amdgpu_svm_range *range;
+	ktime_t timestamp = ktime_get_boottime();
+	int retry_count = 3;
+	int ret;
+
+	amdgpu_svm_assert_locked(svm);
+	WARN_ON(!svm->xnack_enabled);
+
+retry:
+	ret = amdgpu_svm_garbage_collector(svm);
+	if (ret) {
+		pr_err_ratelimited("fault garbage collector failed: ret=%d, fault_addr=0x%lx\n",
+				   ret, fault_addr);
+		return ret;
+	}
+
+	ret = fault_check_allowed(svm, fault_addr, write_fault);
+	if (ret)
+		return ret;
+
+	range = amdgpu_svm_range_find_or_insert(
+		svm, fault_addr,
+		attr_start_page << PAGE_SHIFT,
+		(attr_last_page + 1) << PAGE_SHIFT,
+		&map_ctx);
+	if (IS_ERR(range)) {
+		ret = PTR_ERR(range);
+		/*
+		 * -EINVAL: fault_addr out of gpusvm range, or no chunk size
+		 *          fits within VMA/notifier/attr_range bounds.
+		 * -EFAULT: mmget_not_zero failed (mm is dying).
+		 * -ENOENT: No VMA at fault_addr.
+		 */
+		if (ret == -EFAULT || ret == -ENOENT) {
+			pr_debug("no vma or mm dying: 0x%lx ret=%d\n",
+					 fault_addr, ret);
+			ret = 0;
+			goto out;
+		}
+
+		pr_err_ratelimited("map_fault: range_find_or_insert failed: fault=0x%lx ret=%d\n",
+				   fault_addr, ret);
+		return ret;
+	}
+
+	if (write_fault && map_ctx.read_only) {
+		pr_debug("write fault on read-only range: fault=0x%lx range=[0x%lx-0x%lx)\n",
+			 fault_addr, drm_gpusvm_range_start(&range->base),
+			 drm_gpusvm_range_end(&range->base));
+		ret = -EPERM;
+		goto err_out;
+	}
+
+	if (READ_ONCE(range->gpu_mapped) &&
+	    ktime_before(timestamp,
+			 ktime_add_ns(range->validate_timestamp,
+				      AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING))) {
+		pr_debug("already restored, skip: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				 fault_addr, drm_gpusvm_range_start(&range->base),
+				 drm_gpusvm_range_end(&range->base));
+		goto out;
+	}
+
+	if (amdgpu_svm_range_is_valid(range, attrs)) {
+		pr_debug("valid range, skip: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				 fault_addr, drm_gpusvm_range_start(&range->base),
+				 drm_gpusvm_range_end(&range->base));
+		goto out;
+	}
+
+	AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT");
+	/* TODO: add migration*/
+
+	AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
+	ret = amdgpu_svm_range_get_pages(svm, &range->base, &map_ctx);
+	if (ret == -EOPNOTSUPP || ret == -EFAULT || ret == -EBUSY) {
+		/*
+		 * -EOPNOTSUPP  Mixed page types within range.
+		 * -EFAULT      mm is dying，range was unmapped，DMA mapping failed，
+		 *              devmem_only requested but system page encountered，
+		 *              no VMA, page fault error, bad pte/pmd.
+		 * -EBUSY       HMM retry loop timed out.
+		 * -ENOMEM      PFN or DMA address array allocation failed.
+		 * -EINVAL      hmm_range_fault: invalid VMA type.
+		 */
+		map_ctx.timeslice_ms <<= 1;
+		if (!map_ctx.devmem_only && --retry_count > 0) {
+			pr_debug(
+				"retry: get_pages failed %d, left=%d: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				ret, retry_count, fault_addr,
+				drm_gpusvm_range_start(&range->base),
+				drm_gpusvm_range_end(&range->base));
+			goto retry;
+		}
+		pr_err_ratelimited("map_fault: get_pages retry exhausted, ret=%d: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				   ret, fault_addr,
+				   drm_gpusvm_range_start(&range->base),
+				   drm_gpusvm_range_end(&range->base));
+	}
+
+	if (ret == -EPERM) {
+		/* According to AMD UMD contract, treat -EPERM as a error fault */
+		pr_err_ratelimited("get_pages -EPERM: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				   fault_addr, drm_gpusvm_range_start(&range->base),
+				   drm_gpusvm_range_end(&range->base));
+		return ret;
+	}
+
+	if (ret == -ENOMEM) {
+		pr_err_ratelimited("get_pages -ENOMEM: fault=0x%lx range=[0x%lx-0x%lx)\n",
+				   fault_addr, drm_gpusvm_range_start(&range->base),
+				   drm_gpusvm_range_end(&range->base));
+		return ret;
+	}
+
+	if (ret) {
+		AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT - FAIL PAGE COLLECT");
+		goto err_out;
+	}
+
+	AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT - GPU MAP");
+
+	ret = amdgpu_svm_range_update_mapping(svm, range, attrs,
+					      map_ctx.read_only,
+					      false, false, false);
+
+	if (ret)
+		goto err_out;
+
+out:
+	return 0;
+
+err_out:
+	if (ret == -EAGAIN && --retry_count > 0) {
+		map_ctx.timeslice_ms <<= 1;
+		AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT - RETRY GPU MAP");
+		goto retry;
+	}
+
+	return ret;
+}
+
+/**
+ * amdgpu_svm_handle_fault() - Top-level SVM retry fault handler
+ * @adev: The amdgpu device.
+ * @pasid: PASID of the faulting context.
+ * @fault_page: Faulting page index.
+ * @ts: IH ring timestamp of the fault.
+ * @write_fault: True if the fault was caused by a write.
+ *
+ * Entry point from the VM fault path. Look up the SVM context, drop the
+ * fault if the context is gone/exiting or the mm is dying, filter stale
+ * retry faults using the checkpoint timestamp, then resolve the covering
+ * attribute range (creating a default one if needed) and map it.
+ *
+ * Return: 0 on success or a dropped fault, negative error code on failure.
+ */
+int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
+			    uint64_t fault_page, uint64_t ts,
+			    bool write_fault)
+{
+	struct amdgpu_svm *svm;
+	struct amdgpu_svm_attr_range *attr_range;
+	struct amdgpu_svm_attrs attrs;
+	unsigned long attr_start_page, attr_last_page;
+	uint64_t fault_addr = fault_page << PAGE_SHIFT;
+	struct mm_struct *mm = NULL;
+	uint64_t ckpt;
+	int ret;
+
+	if (amdgpu_in_reset(adev))
+		return 0;
+
+	svm = amdgpu_svm_lookup_by_pasid(adev, pasid);
+	if (!svm) {
+		pr_debug("no SVM context for pasid %u, drop fault 0x%llx\n",
+				 pasid, fault_page << PAGE_SHIFT);
+		return 0;
+	}
+
+	if (atomic_read(&svm->exiting)) {
+		pr_debug("SVM context is exiting for pasid %u, drop fault 0x%llx\n",
+				 pasid, fault_page << PAGE_SHIFT);
+		ret = 0;
+		goto out_put;
+	}
+
+	mm = svm->gpusvm.mm;
+	if (!mm || !mmget_not_zero(mm)) {
+		pr_debug("mm dying for pasid %u, drop fault 0x%llx\n",
+				 pasid, fault_page << PAGE_SHIFT);
+		mm = NULL;
+		ret = 0;
+		goto out_put;
+	}
+
+	if (!svm->xnack_enabled) {
+		pr_err_ratelimited("xnack not enabled for pasid %u\n",
+				   pasid);
+		ret = -EOPNOTSUPP;
+		goto out_put;
+	}
+
+	ckpt = READ_ONCE(svm->checkpoint_ts);
+	if (ckpt != 0) {
+		if (amdgpu_ih_ts_after_or_equal(ts, ckpt)) {
+			pr_debug(
+				"draining stale retry fault, drop fault 0x%llx ts=%llu checkpoint=%llu\n",
+				fault_addr, ts, ckpt);
+			amdgpu_gmc_filter_faults_remove(
+				adev, fault_page, pasid);
+			ret = 0;
+			goto out_put;
+		} else {
+			WRITE_ONCE(svm->checkpoint_ts, 0);
+		}
+	}
+
+	amdgpu_svm_lock(svm);
+
+	mutex_lock(&svm->attr_tree->lock);
+	attr_range = amdgpu_svm_attr_get_bounds_locked(svm->attr_tree,
+						       fault_page,
+						       &attr_start_page, &attr_last_page);
+	if (attr_range) {
+		attrs = attr_range->attrs;
+		attr_start_page = amdgpu_svm_attr_start_page(attr_range);
+		attr_last_page = amdgpu_svm_attr_last_page(attr_range);
+	}
+	mutex_unlock(&svm->attr_tree->lock);
+	if (!attr_range) {
+		ret = fault_get_unregistered_attrs(svm, fault_addr,
+							      attr_start_page,
+							      attr_last_page,
+							      &attrs,
+							      &attr_start_page,
+							      &attr_last_page);
+		if (ret) {
+			if (ret == -EFAULT)
+				goto out_no_vma;
+			goto out_unlock;
+		}
+	}
+	ret = fault_map_range(svm, fault_addr, &attrs, attr_start_page,
+			      attr_last_page, write_fault);
+
+	if (ret == -EAGAIN) {
+		pr_debug("retry fault=0x%llx\n",
+				 fault_addr);
+		amdgpu_gmc_filter_faults_remove(adev, fault_page, pasid);
+		ret = 0;
+	}
+
+	goto out_unlock;
+
+out_no_vma:
+	pr_debug("no VMA for fault=0x%llx (stale retry or GPU NULL deref)\n",
+		 fault_addr);
+	ret = -EFAULT;
+
+out_unlock:
+	amdgpu_svm_unlock(svm);
+
+out_put:
+	if (mm)
+		mmput_async(mm);
+	amdgpu_svm_put(svm);
+	return ret;
+}
+
+#endif /* CONFIG_DRM_AMDGPU_SVM */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h
new file mode 100644
index 0000000000000..c169a96600de4
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_FAULT_H__
+#define __AMDGPU_SVM_FAULT_H__
+
+#include <linux/types.h>
+
+struct amdgpu_device;
+struct amdgpu_svm;
+struct amdgpu_svm_attr_range;
+struct amdgpu_svm_attrs;
+
+int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
+			    uint64_t fault_page, uint64_t ts,
+			    bool write_fault);
+
+#endif /* __AMDGPU_SVM_FAULT_H__ */
-- 
2.53.0

[PATCH v10 12/22] drm/amdgpu: integrate SVM into build system and VM fault path

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Wire up the SVM subsystem:

Kconfig:
- CONFIG_DRM_AMDGPU_SVM option, depends on DEVICE_PRIVATE, selects
  DRM_GPUSVM, HMM_MIRROR, MMU_NOTIFIER, default n

Makefile:
- Build amdgpu_svm.o, amdgpu_svm_attr.o, amdgpu_svm_fault.o,
  amdgpu_svm_range.o when CONFIG_DRM_AMDGPU_SVM=y
- KBUILD_EXTRA_SYMBOLS for drm Module.symvers

amdgpu_drv.c:
- Register DRM_IOCTL_DEF_DRV in ioctl table

amdgpu_vm.c:
- Initialize vm->svm = NULL in amdgpu_vm_init
- Call amdgpu_svm_init in amdgpu_vm_make_compute
- Call amdgpu_svm_close + amdgpu_svm_fini in amdgpu_vm_fini
- Route GPU page faults through amdgpu_svm_handle_fault when SVM
  context exists, without falling back to svm_range_restore_pages

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Kconfig      | 10 ++++++++++
 drivers/gpu/drm/amd/amdgpu/Makefile     |  4 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 23 +++++++++++++++++++++--
 4 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/Kconfig b/drivers/gpu/drm/amd/amdgpu/Kconfig
index 12e4a41bf1f04..68f519b3479db 100644
--- a/drivers/gpu/drm/amd/amdgpu/Kconfig
+++ b/drivers/gpu/drm/amd/amdgpu/Kconfig
@@ -81,6 +81,16 @@ config DRM_AMDGPU_USERPTR
 	  This option selects CONFIG_HMM and CONFIG_HMM_MIRROR if it
 	  isn't already selected to enabled full userptr support.
 
+config DRM_AMDGPU_SVM
+	bool "Enable AMDGPU SVM support (experimental)"
+	depends on DRM_AMDGPU
+	depends on DEVICE_PRIVATE
+	select DRM_GPUSVM
+	select HMM_MIRROR
+	select MMU_NOTIFIER
+	help
+	  Experimental SVM support based on DRM GPUSVM.
+
 config DRM_AMD_ISP
 	bool "Enable AMD Image Signal Processor IP support"
 	depends on DRM_AMDGPU && ACPI
diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index 7b08814a24da4..96fd2b195c295 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -326,6 +326,10 @@ amdgpu-$(CONFIG_VGA_SWITCHEROO) += amdgpu_atpx_handler.o
 amdgpu-$(CONFIG_ACPI) += amdgpu_acpi.o
 amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
 
+# svm support
+amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
+	amdgpu_svm_fault.o amdgpu_svm_range.o
+
 include $(FULL_AMD_PATH)/pm/Makefile
 
 amdgpu-y += $(AMD_POWERPLAY_FILES)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 9c5e93cd3ee64..2a03d5cd78e85 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -50,6 +50,7 @@
 #include "amdgpu_ras.h"
 #include "amdgpu_reset.h"
 #include "amdgpu_sched.h"
+#include "amdgpu_svm.h"
 #include "amdgpu_xgmi.h"
 #include "amdgpu_userq.h"
 #include "amdgpu_userq_fence.h"
@@ -3121,7 +3122,8 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = {
 	DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(AMDGPU_GEM_LIST_HANDLES, amdgpu_gem_list_handles_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 	DRM_IOCTL_DEF_DRV(AMDGPU_PROC_OPTIONS, amdgpu_proc_options_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
-	DRM_IOCTL_DEF_DRV(AMDGPU_UALINK_HANDLE, amdgpu_gem_ualink_handle_ioctl, DRM_AUTH|DRM_RENDER_ALLOW)
+	DRM_IOCTL_DEF_DRV(AMDGPU_UALINK_HANDLE, amdgpu_gem_ualink_handle_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
+	DRM_IOCTL_DEF_DRV(AMDGPU_GEM_SVM, amdgpu_gem_svm_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
 };
 
 static const struct drm_driver amdgpu_kms_driver = {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 4fb22195607a5..77f1794eb5011 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -44,6 +44,7 @@
 #include "amdgpu_xgmi.h"
 #include "amdgpu_dma_buf.h"
 #include "amdgpu_res_cursor.h"
+#include "amdgpu_svm.h"
 #include "kfd_svm.h"
 
 /**
@@ -2672,6 +2673,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	int r, i;
 
 	vm->va = RB_ROOT_CACHED;
+	vm->svm = NULL;
 	for (i = 0; i < AMDGPU_MAX_VMHUBS; i++)
 		vm->reserved_vmid[i] = NULL;
 
@@ -2816,6 +2818,10 @@ int amdgpu_vm_make_compute(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	vm->is_compute_context = true;
 	vm->need_tlb_fence = true;
 
+	r = amdgpu_svm_init(adev, vm);
+	if (r)
+		goto unreserve_bo;
+
 unreserve_bo:
 	amdgpu_bo_unreserve(vm->root.bo);
 	return r;
@@ -2888,6 +2894,9 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	unsigned long flags;
 	int i;
 
+	amdgpu_svm_close(vm);
+	amdgpu_svm_fini(vm);
+
 	amdgpu_amdkfd_gpuvm_destroy_cb(adev, vm);
 
 	root = amdgpu_bo_ref(vm->root.bo);
@@ -3127,6 +3136,7 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 {
 	struct amdgpu_vm_update_params params;
 	bool is_compute_context = false;
+	bool use_amdgpu_svm = false;
 	uint64_t *dst, flags[AMDGPU_VM_MAX_LEVEL];
 	struct amdgpu_vm *vm;
 	struct drm_exec exec;
@@ -3146,6 +3156,7 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 	}
 
 	is_compute_context = vm->is_compute_context;
+	use_amdgpu_svm = !!vm->svm;
 
 	if (is_compute_context) {
 		/* Release the root PD lock since svm_range_restore_pages
@@ -3155,8 +3166,16 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 		 */
 		drm_exec_fini(&exec);
 
-		if (!svm_range_restore_pages(adev, pasid, vmid,
-					     node_id, addr >> PAGE_SHIFT, ts, write_fault))
+		if (use_amdgpu_svm)
+			r = amdgpu_svm_handle_fault(adev, pasid,
+						    addr >> PAGE_SHIFT, ts,
+						    write_fault);
+		else
+			r = svm_range_restore_pages(adev, pasid, vmid, node_id,
+						    addr >> PAGE_SHIFT, ts,
+						    write_fault);
+
+		if (!r)
 			return true;
 
 		/* Re-acquire the VM lock, could be that the VM was freed in between. */
-- 
2.53.0

[PATCH v10 13/22] drm/amdgpu: make VM eviction lock read/write to match notifier lock

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Make the VM eviction lock read/write to match the drm_gpusvm notifier
lock. This allows VM updates, MMU notifier invalidation and eviction to
share the same rwsem.

Replace the VM eviction mutex with a selectable rwsem. Updates made with
the root BO reserved take the read side. Updates without the root BO
reservation take the write side, as does eviction. A VM using
drm_gpusvm can use the notifier lock as its eviction lock.

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Yu Huang <Yu.Huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c          |  9 +++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h          |  8 ++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h | 12 ++++++++++--
 3 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 77f1794eb5011..e851912999a51 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -617,7 +617,7 @@ int amdgpu_vm_validate(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	 * As soon as all page tables are in place we can start updating them
 	 * again.
 	 */
-	scoped_guard(mutex, &vm->eviction_lock)
+	scoped_guard(rwsem_write, vm->eviction_lock)
 		vm->evicting = false;
 
 	list_for_each_entry_safe(bo_base, tmp, &vm->always_valid.evicted,
@@ -678,7 +678,7 @@ bool amdgpu_vm_ready(struct amdgpu_vm *vm)
 
 	amdgpu_vm_assert_locked(vm);
 
-	scoped_guard(mutex, &vm->eviction_lock)
+	scoped_guard(rwsem_read, vm->eviction_lock)
 		ret = !vm->evicting;
 
 	ret &= list_empty(&vm->kernel.evicted);
@@ -2357,7 +2357,7 @@ bool amdgpu_vm_evictable(struct amdgpu_bo *bo)
 
 	/* Try to block ongoing updates */
 	vm = bo_base->vm;
-	scoped_cond_guard(mutex_try, return false, &vm->eviction_lock) {
+	scoped_cond_guard(rwsem_write_try, return false, vm->eviction_lock) {
 
 		/* Don't evict VM page tables while they are updated */
 		if (!dma_fence_is_signaled(vm->last_unlocked))
@@ -2713,7 +2713,8 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	vm->last_tlb_flush = dma_fence_get_stub();
 	vm->generation = amdgpu_vm_generation(adev, NULL);
 
-	mutex_init(&vm->eviction_lock);
+	init_rwsem(&vm->default_eviction_lock);
+	vm->eviction_lock = &vm->default_eviction_lock;
 	vm->evicting = false;
 	vm->tlb_fence_context = dma_fence_context_alloc(1);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index b435e393be154..ffdfaadc90544 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -27,6 +27,7 @@
 #include <linux/idr.h>
 #include <linux/kfifo.h>
 #include <linux/rbtree.h>
+#include <linux/rwsem.h>
 #include <drm/gpu_scheduler.h>
 #include <drm/drm_file.h>
 #include <drm/ttm/ttm_bo.h>
@@ -288,10 +289,9 @@ struct amdgpu_vm {
 	/* tree of virtual addresses mapped */
 	struct rb_root_cached	va;
 
-	/* Lock to prevent eviction while we are updating page tables
-	 * use vm_eviction_lock/unlock(vm)
-	 */
-	struct mutex		eviction_lock;
+	/* SVM can replace this pointer with its notifier lock. */
+	struct rw_semaphore	default_eviction_lock;
+	struct rw_semaphore	*eviction_lock;
 	bool			evicting;
 
 	/* Memory statistics for this vm, protected by stats_lock */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h
index 8a0e5fcbd55ba..920d0b3935807 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h
@@ -160,7 +160,12 @@ void amdgpu_vm_pt_free_dummies(struct amdgpu_device *adev);
  */
 static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)
 {
-	mutex_lock(&p->vm->eviction_lock);
+	/* Use the write lock to serialize updates when the root BO is unlocked. */
+	if (p->unlocked)
+		down_write(p->vm->eviction_lock);
+	else
+		down_read(p->vm->eviction_lock);
+
 	p->saved_flags = memalloc_noreclaim_save();
 	if (p->vm->evicting)
 		return -EBUSY;
@@ -178,7 +183,10 @@ static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)
 static inline void amdgpu_vm_end_critical(struct amdgpu_vm_update_params *p)
 {
 	memalloc_noreclaim_restore(p->saved_flags);
-	mutex_unlock(&p->vm->eviction_lock);
+	if (p->unlocked)
+		up_write(p->vm->eviction_lock);
+	else
+		up_read(p->vm->eviction_lock);
 }
 
 #endif
-- 
2.53.0

[PATCH v10 14/22] drm/amdgpu: recheck range validity after page table allocation

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

The notifier and eviction locks need to be the same lock. Page table
allocation can enter memory reclaim, and reclaim can invoke an MMU
notifier that takes the same lock. Release the lock before allocating
page tables to avoid a deadlock.

Dropping the lock allows an unmap operation to invalidate the range
concurrently. Recheck the HMM or GPU SVM range after reacquiring the
lock and return -EAGAIN if its sequence number has changed.

Allow callers that already hold the eviction lock to pass it to the VM
update helpers. Drop the lock around amdgpu_vm_pt_create() and reacquire
it before continuing. Callers that pass NULL retain the existing
internal locking path.

Update the remaining amdgpu_vm_update_pdes() call sites (amdkfd_gpuvm.c,
cs.c, gem.c, userq.c) and amdgpu_ualink.c's update_pdes/map_range call
sites to pass NULL, since these callers do not hold the eviction lock
externally. amdgpu_ualink.c only exists on top of amd-staging-drm-next
and was never covered by the original patch against the old base.

Drop the now-unused error_free_pt label in amdgpu_vm_pt_alloc(): the
r2-failure path was reworked to use amdgpu_vm_update_unlock()/relock()
instead of a goto, leaving error_free_pt reachable only by fallthrough
from error_unpin.

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Yu Huang <Yu.Huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c        |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c |  7 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c    | 24 +++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c     |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 79 ++++++++++++-------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |  9 ++-
 .../gpu/drm/amd/amdgpu/amdgpu_vm_internal.h   | 28 +++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c     | 32 ++++++--
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          |  7 +-
 11 files changed, 131 insertions(+), 63 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index cb46111b2dc8c..321f23cba15c3 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -503,7 +503,7 @@ static int vm_update_pds(struct amdgpu_vm *vm, struct amdgpu_sync *sync)
 	struct amdgpu_device *adev = amdgpu_ttm_adev(pd->tbo.bdev);
 	int ret;
 
-	ret = amdgpu_vm_update_pdes(adev, vm);
+	ret = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (ret)
 		return ret;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
index 9e0b9a50d2e72..90829b6adb1f4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -1184,7 +1184,7 @@ static int amdgpu_cs_vm_handling(struct amdgpu_cs_parser *p)
 	if (r)
 		return r;
 
-	r = amdgpu_vm_update_pdes(adev, vm);
+	r = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (r)
 		return r;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index 907b25397a621..669ca9d1de82a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -804,7 +804,7 @@ amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
 	}
 
 	/* Always update PDEs after we touched the mappings. */
-	r = amdgpu_vm_update_pdes(adev, vm);
+	r = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (r)
 		goto error;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index 62c78c3ce6762..a1aa84974a43a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -72,8 +72,8 @@ amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
 		return 0;
 
 	flags = memalloc_noreclaim_save();
-	ret = amdgpu_vm_unmap_range(svm->adev, svm->vm, NULL,
-				    start_page, last_page, 0, &fence);
+	ret = amdgpu_vm_unmap_range(svm->adev, svm->vm, true, NULL,
+				    start_page, last_page, NULL, &fence);
 	memalloc_noreclaim_restore(flags);
 
 	if (fence) {
@@ -222,6 +222,7 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
 					  flush_tlb && is_last_seg, true, NULL,
 					  start_page, last_page, pte_flags,
 					  0, entry->addr, NULL, NULL, NULL,
+					  NULL, NULL,
 					  wait_fence && is_last_seg ? fence : NULL);
 		if (ret)
 			return ret;
@@ -334,7 +335,7 @@ int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
 	dma_fence_put(fence);
 
 	if (!ret)
-		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, false);
+		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, NULL);
 
 	if (!ret) {
 		if (flush_tlb)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
index de913131a246f..eecc85caadcd2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
@@ -2051,14 +2051,14 @@ static int amdgpu_ualink_unmap_npa_addr(struct amdgpu_device *adev,
 
 	r = amdgpu_vm_map_range(adev, vm, true, false, NULL, npa_addr,
 				npa_addr + size - 1, pte_value, 0, 0, NULL,
-				NULL, NULL, &vm->last_update);
+				NULL, NULL, NULL, NULL, &vm->last_update);
 	if (r) {
 		dev_err(adev->dev,
 			"Failed to unmap NPA addr (%llx) from NPA VM\n", npa_addr);
 		goto out;
 	}
 
-	r = amdgpu_vm_update_pdes(adev, vm);
+	r = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (r) {
 		dev_err(adev->dev,
 			"Failed %d to update page directories during unmapping NPA: 0x%llx\n",
@@ -2104,7 +2104,7 @@ static int amdgpu_ualink_map_npa_addr(struct amdgpu_device *adev, u64 npa_addr,
 	r = amdgpu_vm_map_range(adev, vm, true, false, NULL, npa_addr,
 				npa_addr + size - 1, pte_flags, offset,
 				adev->vm_manager.vram_base_offset,
-				bo->tbo.resource, NULL, NULL, &vm->last_update);
+				bo->tbo.resource, NULL, NULL, NULL, NULL, &vm->last_update);
 	if (r) {
 		dev_warn(adev->dev,
 			"Failed to map NPA addr (%llx) into NPA VM\n", npa_addr);
@@ -2112,7 +2112,7 @@ static int amdgpu_ualink_map_npa_addr(struct amdgpu_device *adev, u64 npa_addr,
 		goto out;
 	}
 
-	r = amdgpu_vm_update_pdes(adev, vm);
+	r = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (r) {
 		dev_err(adev->dev,
 			"failed %d to update page directories for NPA: 0x%llx\n",
@@ -2610,7 +2610,7 @@ static void amdgpu_ualink_force_retry_rpcs(struct amdgpu_device *adev,
 					pte_flags, 0,
 					adev->vm_manager.vram_base_offset,
 					bo->tbo.resource, NULL, NULL,
-					&vm->last_update);
+					NULL, NULL, &vm->last_update);
 
 		if (r)
 			dev_warn(adev->dev,
@@ -2621,7 +2621,7 @@ static void amdgpu_ualink_force_retry_rpcs(struct amdgpu_device *adev,
 			break;
 	}
 
-	r = amdgpu_vm_update_pdes(adev, vm);
+	r = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (r) {
 		dev_err(adev->dev,
 			"Failed %d to update page directories during force retry rpcs\n",
@@ -2746,7 +2746,7 @@ static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
 		r = amdgpu_vm_map_range(adev, vm, true, false, NULL,
 					npa_addr, npa_addr + size - 1,
 					pte_value, 0, 0, NULL, NULL, NULL,
-					&vm->last_update);
+					NULL, NULL, &vm->last_update);
 
 		if (r)
 			dev_err(adev->dev,
@@ -2758,7 +2758,7 @@ static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
 			break;
 	}
 
-	r = amdgpu_vm_update_pdes(adev, vm);
+	r = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (r) {
 		dev_err(adev->dev,
 			"Failed %d to update page directories during all NPA addresses unmapping\n",
@@ -4527,7 +4527,7 @@ static int amdgpu_ualink_npa_vm_map_range(struct amdgpu_device *adev, struct amd
 				npa_in_pages, npa_in_pages + size_in_pages - 1,
 				pte_flags, offset, adev->vm_manager.vram_base_offset,
 				bo->tbo.resource, NULL, NULL,
-				&npa_vm->last_update);
+				NULL, NULL, &npa_vm->last_update);
 	if (r)
 		dev_dbg(adev->dev, "failed %d to map npa 0x%llx to NPA VM\n", r,
 			npa_in_pages << AMDGPU_GPU_PAGE_SHIFT);
@@ -4561,7 +4561,7 @@ static int amdgpu_ualink_npa_vm_unmap_range(struct amdgpu_device *adev, struct a
 	r = amdgpu_vm_map_range(adev, npa_vm, true, false, NULL,
 				npa_in_pages, npa_in_pages + size_in_pages - 1,
 				pte_flags, offset, 0, bo->tbo.resource, NULL,
-				NULL, &npa_vm->last_update);
+				NULL, NULL, NULL, &npa_vm->last_update);
 	if (r)
 		dev_dbg(adev->dev, "failed %d to unmap npa 0x%llx from NPA VM\n", r,
 			npa_in_pages << AMDGPU_GPU_PAGE_SHIFT);
@@ -4828,7 +4828,7 @@ static void amdgpu_ualink_metadata_npa_unmapping(struct amdgpu_device *adev)
 		}
 	}
 
-	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm);
+	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, NULL);
 	if (r) {
 		dev_dbg(adev->dev, "failed %d to update directories\n", r);
 		goto out_unreserve;
@@ -4971,7 +4971,7 @@ static int amdgpu_ualink_metadata_npa_mapping(struct amdgpu_device *adev)
 		}
 	}
 
-	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm);
+	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, NULL);
 	if (r) {
 		dev_dbg(adev->dev, "failed %d to update directories\n", r);
 		goto error_npa_mapping;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
index 5df3c9af2667c..53c3259c6a189 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
@@ -1169,7 +1169,7 @@ amdgpu_userq_vm_validate_and_restore_queue(struct amdgpu_userq_mgr *uq_mgr)
 		goto retry_lock;
 	}
 
-	ret = amdgpu_vm_update_pdes(adev, vm);
+	ret = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (ret)
 		goto unlock_all;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index e851912999a51..81a83e6b76e00 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -985,13 +985,15 @@ uint64_t amdgpu_vm_map_gart(const dma_addr_t *pages_addr, uint64_t addr)
  *
  * @adev: amdgpu_device pointer
  * @vm: requested vm
+ * @eviction_lock: optional caller-held eviction lock
  *
  * Makes sure all directories are up to date.
  *
  * Returns:
  * 0 for success, error for failure.
  */
-int amdgpu_vm_update_pdes(struct amdgpu_device *adev, struct amdgpu_vm *vm)
+int amdgpu_vm_update_pdes(struct amdgpu_device *adev, struct amdgpu_vm *vm,
+			  struct rw_semaphore *eviction_lock)
 {
 	struct amdgpu_vm_update_params params;
 	struct amdgpu_vm_bo_base *entry, *tmp;
@@ -1010,6 +1012,16 @@ int amdgpu_vm_update_pdes(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	params.adev = adev;
 	params.vm = vm;
 
+	/*
+	 * Keep PDE updates under the SVM eviction lock so MMU notifier
+	 * invalidation won't happen between the PTE and PDE updates.
+	 */
+	params.eviction_lock = eviction_lock;
+
+	r = amdgpu_vm_begin_critical(&params);
+	if (r)
+		goto error;
+
 	r = vm->update_funcs->prepare(&params, NULL,
 				      AMDGPU_KERNEL_JOB_ID_VM_UPDATE_PDES);
 	if (r)
@@ -1036,6 +1048,7 @@ int amdgpu_vm_update_pdes(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 		amdgpu_vm_bo_idle(entry);
 
 error:
+	amdgpu_vm_end_critical(&params);
 	drm_dev_exit(idx);
 	return r;
 }
@@ -1116,6 +1129,8 @@ amdgpu_vm_tlb_flush(struct amdgpu_vm_update_params *params,
  * @res: ttm_resource to map
  * @pages_addr: DMA addresses to use for mapping
  * @hmm_range: to check validity of DMA addresses
+ * @eviction_lock: optional caller-held eviction lock
+ * @gpusvm_range: optional SVM range to revalidate
  * @fence: optional resulting fence
  *
  * Fill in the page table entries between @start and @last. Allocate and free
@@ -1131,6 +1146,8 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			uint64_t vram_base, struct ttm_resource *res,
 			dma_addr_t *pages_addr,
 			struct amdgpu_hmm_range *hmm_range,
+			struct rw_semaphore *eviction_lock,
+			struct drm_gpusvm_range *gpusvm_range,
 			struct dma_fence **fence)
 {
 	struct amdgpu_vm_tlb_seq_struct *tlb_cb;
@@ -1143,10 +1160,25 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	if (!drm_dev_enter(adev_to_drm(adev), &idx))
 		return -ENODEV;
 
+	memset(&params, 0, sizeof(params));
+	params.adev = adev;
+	params.vm = vm;
+	params.pages_addr = pages_addr;
+	params.hmm_range = hmm_range;
+	params.eviction_lock = eviction_lock;
+	params.gpusvm_range = gpusvm_range;
+	params.override_pte = allow_override && adev->gmc.override_pte;
+	INIT_LIST_HEAD(&params.tlb_flush_waitlist);
+
+	/* Enter noreclaim before allocating the small tlb_cb. */
+	r = amdgpu_vm_begin_critical(&params);
+	if (r)
+		goto error_critical;
+
 	tlb_cb = kmalloc_obj(*tlb_cb);
 	if (!tlb_cb) {
-		drm_dev_exit(idx);
-		return -ENOMEM;
+		r = -ENOMEM;
+		goto error_critical;
 	}
 
 	/* Vega20+XGMI where PTEs get inadvertently cached in L2 texture cache,
@@ -1160,18 +1192,7 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	 */
 	flush_tlb |= amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(9, 0, 0);
 
-	memset(&params, 0, sizeof(params));
-	params.adev = adev;
-	params.vm = vm;
-	params.pages_addr = pages_addr;
-	params.hmm_range = hmm_range;
 	params.needs_flush = flush_tlb;
-	params.override_pte = allow_override && adev->gmc.override_pte;
-	INIT_LIST_HEAD(&params.tlb_flush_waitlist);
-
-	r = amdgpu_vm_begin_critical(&params);
-	if (r)
-		goto error_free;
 
 	if (!dma_fence_is_signaled(vm->last_unlocked)) {
 		struct dma_fence *tmp = dma_fence_get_stub();
@@ -1255,6 +1276,7 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 error_free:
 	kfree(tlb_cb);
+error_critical:
 	amdgpu_vm_end_critical(&params);
 	drm_dev_exit(idx);
 	return r;
@@ -1269,6 +1291,7 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
  * @sync: fences we need to sync to
  * @start: start of unmapped range
  * @last: last unmapped entry
+ * @eviction_lock: optional caller-held eviction lock
  * @fence: optional resulting fence
  *
  * Fill in the page table entries between @start and @last with a fixed flags
@@ -1281,6 +1304,7 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 int amdgpu_vm_unmap_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			  bool unlocked, struct amdgpu_sync *sync,
 			  uint64_t start, uint64_t last,
+			  struct rw_semaphore *eviction_lock,
 			  struct dma_fence **fence)
 {
 	struct amdgpu_vm_manager *vm_mgr = &adev->vm_manager;
@@ -1292,22 +1316,24 @@ int amdgpu_vm_unmap_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	if (!drm_dev_enter(adev_to_drm(adev), &idx))
 		return -ENODEV;
 
-	tlb_cb = kmalloc(sizeof(*tlb_cb), GFP_KERNEL);
-	if (!tlb_cb) {
-		drm_dev_exit(idx);
-		return -ENOMEM;
-	}
-
 	memset(&params, 0, sizeof(params));
 	params.adev = adev;
 	params.vm = vm;
 	params.needs_flush = true;
 	params.unlocked = unlocked;
+	params.eviction_lock = eviction_lock;
 	INIT_LIST_HEAD(&params.tlb_flush_waitlist);
 
+	/* Enter noreclaim before allocating the small tlb_cb. */
 	r = amdgpu_vm_begin_critical(&params);
 	if (r)
-		goto error_end_critical;
+		goto error_critical;
+
+	tlb_cb = kmalloc_obj(*tlb_cb);
+	if (!tlb_cb) {
+		r = -ENOMEM;
+		goto error_critical;
+	}
 
 	r = vm->update_funcs->prepare(&params, sync,
 				      AMDGPU_KERNEL_JOB_ID_VM_UNMAP_RANGE);
@@ -1329,8 +1355,7 @@ int amdgpu_vm_unmap_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 error_free:
 	kfree(tlb_cb);
-
-error_end_critical:
+error_critical:
 	amdgpu_vm_end_critical(&params);
 	drm_dev_exit(idx);
 	return r;
@@ -1468,7 +1493,7 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va,
 					mapping->start, mapping->last,
 					update_flags, mapping->offset,
 					vram_base, mem, pages_addr, NULL,
-					last_update);
+					NULL, NULL, last_update);
 		if (r)
 			goto error_free;
 	}
@@ -1675,8 +1700,9 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
 			struct amdgpu_bo_va_mapping, list);
 		list_del(&mapping->list);
 
-		r = amdgpu_vm_unmap_range(adev, vm, false, &sync,
-					  mapping->start, mapping->last, &f);
+                r = amdgpu_vm_unmap_range(adev, vm, false, &sync,
+                                          mapping->start, mapping->last,
+                                          NULL, &f);
 		amdgpu_vm_free_mapping(adev, vm, mapping, f);
 		if (r) {
 			dma_fence_put(f);
@@ -2897,7 +2923,6 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 
 	amdgpu_svm_close(vm);
 	amdgpu_svm_fini(vm);
-
 	amdgpu_amdkfd_gpuvm_destroy_cb(adev, vm);
 
 	root = amdgpu_bo_ref(vm->root.bo);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index ffdfaadc90544..574c1af58c2a2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -39,6 +39,7 @@
 #include "amdgpu_ttm.h"
 
 struct drm_exec;
+struct drm_gpusvm_range;
 
 struct amdgpu_bo_va;
 struct amdgpu_job;
@@ -289,7 +290,7 @@ struct amdgpu_vm {
 	/* tree of virtual addresses mapped */
 	struct rb_root_cached	va;
 
-	/* SVM can replace this pointer with its notifier lock. */
+	/* SVM uses gpusvm.notifier_lock, other VMs use default_eviction_lock. */
 	struct rw_semaphore	default_eviction_lock;
 	struct rw_semaphore	*eviction_lock;
 	bool			evicting;
@@ -455,7 +456,8 @@ int amdgpu_vm_validate(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 void amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
 		     bool *need_pipe_sync, bool *emit_spm_needed,
 		     bool *emit_gds_needed);
-int amdgpu_vm_update_pdes(struct amdgpu_device *adev, struct amdgpu_vm *vm);
+int amdgpu_vm_update_pdes(struct amdgpu_device *adev, struct amdgpu_vm *vm,
+			  struct rw_semaphore *eviction_lock);
 int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
 			  struct amdgpu_vm *vm,
 			  struct dma_fence **fence);
@@ -475,10 +477,13 @@ int amdgpu_vm_map_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			uint64_t vram_base, struct ttm_resource *res,
 			dma_addr_t *pages_addr,
 			struct amdgpu_hmm_range *hmm_range,
+			struct rw_semaphore *eviction_lock,
+			struct drm_gpusvm_range *gpusvm_range,
 			struct dma_fence **fence);
 int amdgpu_vm_unmap_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			  bool unlocked, struct amdgpu_sync *sync,
 			  uint64_t start, uint64_t last,
+			  struct rw_semaphore *eviction_lock,
 			  struct dma_fence **fence);
 int amdgpu_vm_bo_update(struct amdgpu_device *adev,
 			struct amdgpu_bo_va *bo_va,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h
index 920d0b3935807..ffb8f824af9b5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_internal.h
@@ -24,6 +24,7 @@
 #ifndef __AMDGPU_VM_INTERNAL_H__
 #define __AMDGPU_VM_INTERNAL_H__
 
+#include <drm/drm_gpusvm.h>
 #include <linux/types.h>
 #include <linux/list.h>
 #include "amdgpu_hmm.h"
@@ -80,6 +81,12 @@ struct amdgpu_vm_update_params {
 	 */
 	struct amdgpu_hmm_range *hmm_range;
 
+	/** @eviction_lock: Optional caller-held eviction lock. */
+	struct rw_semaphore *eviction_lock;
+
+	/** @gpusvm_range: Optional SVM range to revalidate after taking the lock. */
+	struct drm_gpusvm_range *gpusvm_range;
+
 	/**
 	 * @job: job to used for hw submission
 	 */
@@ -152,6 +159,7 @@ void amdgpu_vm_pt_free_dummies(struct amdgpu_device *adev);
  *
  * Serialize all updates, check parameters and make sure that memory allocations
  * don't enter the reclaim path so that we don't deadlock with MMU notifiers.
+ * A non-NULL eviction_lock means the caller already holds the selected lock.
  *
  * Returns:
  *
@@ -161,7 +169,9 @@ void amdgpu_vm_pt_free_dummies(struct amdgpu_device *adev);
 static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)
 {
 	/* Use the write lock to serialize updates when the root BO is unlocked. */
-	if (p->unlocked)
+	if (p->eviction_lock)
+		lockdep_assert_held(p->eviction_lock);
+	else if (p->unlocked)
 		down_write(p->vm->eviction_lock);
 	else
 		down_read(p->vm->eviction_lock);
@@ -171,6 +181,10 @@ static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)
 		return -EBUSY;
 	if (p->hmm_range && !amdgpu_hmm_range_valid(p->hmm_range))
 		return -EAGAIN;
+	if (p->gpusvm_range &&
+	    !drm_gpusvm_range_pages_valid(p->gpusvm_range->gpusvm,
+					   p->gpusvm_range))
+		return -EAGAIN;
 	return 0;
 }
 
@@ -178,15 +192,17 @@ static inline int amdgpu_vm_begin_critical(struct amdgpu_vm_update_params *p)
  * amdgpu_vm_end_critical - end the critical section of the update
  * @p: The update parameters
  *
- * Restore the GFP flags and drop the lock.
+ * Restore the GFP flags.
  */
 static inline void amdgpu_vm_end_critical(struct amdgpu_vm_update_params *p)
 {
 	memalloc_noreclaim_restore(p->saved_flags);
-	if (p->unlocked)
-		up_write(p->vm->eviction_lock);
-	else
-		up_read(p->vm->eviction_lock);
+	if (!p->eviction_lock) {
+		if (p->unlocked)
+			up_write(p->vm->eviction_lock);
+		else
+			up_read(p->vm->eviction_lock);
+	}
 }
 
 #endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
index dea80796f1393..68a94c4a2dd68 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
@@ -505,6 +505,23 @@ int amdgpu_vm_pt_create(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	return 0;
 }
 
+static void amdgpu_vm_update_unlock(struct amdgpu_vm_update_params *p)
+{
+	amdgpu_vm_end_critical(p);
+	if (p->eviction_lock) {
+		lockdep_assert_held_read(p->eviction_lock);
+		up_read(p->eviction_lock);
+	}
+}
+
+static int amdgpu_vm_update_relock(struct amdgpu_vm_update_params *p)
+{
+	if (p->eviction_lock)
+		down_read(p->eviction_lock);
+
+	return amdgpu_vm_begin_critical(p);
+}
+
 /**
  * amdgpu_vm_pt_alloc - Allocate a specific page table
  *
@@ -528,19 +545,23 @@ static int amdgpu_vm_pt_alloc(struct amdgpu_vm_update_params *p,
 	if (entry->bo)
 		return 0;
 
-	amdgpu_vm_end_critical(p);
+	amdgpu_vm_update_unlock(p);
 	r = amdgpu_vm_pt_create(p->adev, p->vm, cursor->level,
 				&pt, p->vm->root.bo->xcp_id);
-	r2 = amdgpu_vm_begin_critical(p);
+	r2 = amdgpu_vm_update_relock(p);
 	if (r)
 		return r;
-	if (r2)
-		goto error_free_pt;
+	pt_bo = &pt->bo;
+	if (r2) {
+		amdgpu_vm_update_unlock(p);
+		amdgpu_bo_unref(&pt_bo);
+		(void)amdgpu_vm_update_relock(p);
+		return r2;
+	}
 
 	/* Keep a reference to the root directory to avoid
 	 * freeing them up in the wrong order.
 	 */
-	pt_bo = &pt->bo;
 	pt_bo->parent = amdgpu_bo_ref(cursor->parent->bo);
 	amdgpu_vm_bo_base_init(entry, p->vm, pt_bo);
 	r = amdgpu_vm_pt_clear(p->adev, p->vm, pt);
@@ -552,7 +573,6 @@ static int amdgpu_vm_pt_alloc(struct amdgpu_vm_update_params *p,
 error_unpin:
 	if (p->vm->is_npa)
 		amdgpu_bo_unpin(pt_bo);
-error_free_pt:
 	amdgpu_bo_unref(&pt_bo);
 	return r;
 }
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 3d90257e68ab6..d22a40af379f1 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -1411,7 +1411,7 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	}
 
 	return amdgpu_vm_unmap_range(adev, vm, true, NULL, gpu_start, gpu_end,
-				     fence);
+				     NULL, fence);
 }
 
 static int
@@ -1526,7 +1526,8 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 					gpu_start, gpu_end, pte_flags,
 					(last_start - prange->start) << PAGE_SHIFT,
 					bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
-					NULL, dma_addr, hmm_range, &vm->last_update);
+					NULL, dma_addr, hmm_range,
+					NULL, NULL, &vm->last_update);
 
 		for (j = last_start - prange->start; j <= i; j++)
 			dma_addr[j] |= last_domain;
@@ -1538,7 +1539,7 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 		last_start = prange->start + i + 1;
 	}
 
-	r = amdgpu_vm_update_pdes(adev, vm);
+	r = amdgpu_vm_update_pdes(adev, vm, NULL);
 	if (r) {
 		pr_debug("failed %d to update directories 0x%lx\n", r,
 			 prange->start);
-- 
2.53.0

[PATCH v10 15/22] drm/amdgpu: use drm_gpusvm notifier lock for VM eviction

[Huang Rui]

From: Honglei Huang <honghuan@amd.com>

Make amdgpu_vm's eviction lock a read/write semaphore so it can be
shared with drm_gpusvm's notifier lock: VM updates and eviction take
it for read, MMU notifier invalidation and eviction take it for write.

Pass the held eviction lock through to amdgpu_vm_update_pdes() in the
SVM context so callers that already hold it do not try to reacquire it.

Fix the eviction_lock argument at this call site: it must be the
3-argument form (adev, vm, eviction_lock) introduced by "recheck range
validity after page table allocation" - a stray leftover 4th argument
(an old "immediate" bool) was being passed here, which would otherwise
fail to build.

Suggested-by: Christian König <christian.koenig@amd.com>
Suggested-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Yu Huang <Yu.Huang@amd.com>
Signed-off-by: Honglei Huang <honghuan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c       | 37 ++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 51 ++++++++-----------
 2 files changed, 49 insertions(+), 39 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
index c7810e6d3035f..5ae8a649644ec 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -157,10 +157,13 @@ amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev, uint32_t pasid)
 	amdgpu_pasid_lock(&irqflags);
 	fpriv = amdgpu_pasid_get_fpriv_locked(pasid);
 	vm = fpriv ? &fpriv->vm : NULL;
-	if (vm && vm->svm) {
-		svm = vm->svm;
+	/*
+	 * eviction_lock changes when SVM is enabled, so acquire vm->svm
+	 * after the new lock alias is published.
+	 */
+	svm = vm ? smp_load_acquire(&vm->svm) : NULL;
+	if (svm)
 		kref_get(&svm->refcount);
-	}
 	amdgpu_pasid_unlock(irqflags);
 
 	return svm;
@@ -501,6 +504,8 @@ static int amdgpu_svm_init_compute(struct amdgpu_device *adev,
 	struct amdgpu_svm *svm;
 	int ret;
 
+	dma_resv_assert_held(vm->root.bo->tbo.base.resv);
+
 	if (vm->svm)
 		return 0;
 
@@ -535,7 +540,9 @@ static int amdgpu_svm_init_compute(struct amdgpu_device *adev,
 	       1UL << (svm->default_granularity + PAGE_SHIFT),
 	       svm->xnack_enabled ? "enabled" : "disabled");
 
-	vm->svm = svm;
+	vm->eviction_lock = &svm->gpusvm.notifier_lock;
+	/* Publish the context after its notifier lock alias. */
+	smp_store_release(&vm->svm, svm);
 	return 0;
 
 err_free:
@@ -611,31 +618,43 @@ void amdgpu_svm_close(struct amdgpu_vm *vm)
  * amdgpu_svm_fini() - Finalize and release a VM's SVM context
  * @vm: The VM whose SVM context is being torn down.
  *
- * Close the context, tear down the embedded drm_gpusvm under the SVM lock,
- * destroy the attribute tree and work queues, and drop the context
- * reference. Safe to call on a VM without an SVM context.
+ * Tear down the embedded drm_gpusvm, attribute tree and work queues, and drop
+ * the context reference.
  */
 void amdgpu_svm_fini(struct amdgpu_vm *vm)
 {
 	struct amdgpu_svm *svm = vm->svm;
+	struct amdgpu_bo *root;
 
 	if (!svm)
 		return;
 
 	amdgpu_svm_close(vm);
 	amdgpu_svm_lock(svm);
+
+	root = amdgpu_bo_ref(vm->root.bo);
+	amdgpu_bo_reserve(root, true);
+
 	drm_gpusvm_fini(&svm->gpusvm);
+
+	vm->eviction_lock = &vm->default_eviction_lock;
+	/* Publish the lock reset before making SVM unavailable. */
+	smp_store_release(&vm->svm, NULL);
+
 	amdgpu_svm_unlock(svm);
 
 	amdgpu_svm_attr_tree_destroy(svm->attr_tree);
 	amdgpu_svm_work_fini(svm);
-	vm->svm = NULL;
 	amdgpu_svm_put(svm);
+
+	amdgpu_bo_unreserve(root);
+	amdgpu_bo_unref(&root);
 }
 
 bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
 {
-	return vm->svm != NULL;
+	/* Pairs with publishing vm->svm after its lock alias is initialized. */
+	return smp_load_acquire(&vm->svm) != NULL;
 }
 
 static int amdgpu_svm_copy_attrs(const struct drm_amdgpu_gem_svm *args,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index a1aa84974a43a..787afe0767e5a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -65,16 +65,14 @@ amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
 			  unsigned long last_page)
 {
 	struct dma_fence *fence = NULL;
-	unsigned int flags;
 	int ret;
 
 	if (last_page < start_page)
 		return 0;
 
-	flags = memalloc_noreclaim_save();
 	ret = amdgpu_vm_unmap_range(svm->adev, svm->vm, true, NULL,
-				    start_page, last_page, NULL, &fence);
-	memalloc_noreclaim_restore(flags);
+				    start_page, last_page,
+				    svm->vm->eviction_lock, &fence);
 
 	if (fence) {
 		dma_fence_wait(fence, false);
@@ -210,20 +208,11 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
 		mapped_pages += seg_pages;
 		is_last_seg = mapped_pages == npages;
 
-		/*
-		 * The struct hmm_range hmm_range field inside amdgpu_hmm_range
-		 * is drm_gpusvm internal state. It is not exposed to the
-		 * implementing driver and is managed entirely by the framework
-		 * as part of the framework logic. This amdgpu_vm_map_range call
-		 * always runs inside the drm_gpusvm notifier, so omitting
-		 * hmm_range here is safe.
-		 */
-		ret = amdgpu_vm_map_range(svm->adev, svm->vm,
-					  flush_tlb && is_last_seg, true, NULL,
-					  start_page, last_page, pte_flags,
-					  0, entry->addr, NULL, NULL, NULL,
-					  NULL, NULL,
-					  wait_fence && is_last_seg ? fence : NULL);
+		ret = amdgpu_vm_map_range(
+			svm->adev, svm->vm, flush_tlb && is_last_seg, true,
+			NULL, start_page, last_page, pte_flags, 0, entry->addr,
+			NULL, NULL, NULL, svm->vm->eviction_lock, base,
+			wait_fence && is_last_seg ? fence : NULL);
 		if (ret)
 			return ret;
 	}
@@ -308,14 +297,12 @@ int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
 {
 	struct drm_exec exec;
 	struct dma_fence *fence = NULL;
-	unsigned int flags;
 	int ret;
 
 	ret = amdgpu_svm_range_lock_vm_pd(svm, &exec, intr);
 	if (ret)
 		return ret;
 
-	flags = memalloc_noreclaim_save();
 	drm_gpusvm_notifier_lock(&svm->gpusvm);
 
 	if (!amdgpu_svm_range_pages_valid(svm, range)) {
@@ -327,24 +314,25 @@ int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
 							wait, wait ? &fence : NULL);
 	}
 
-	drm_gpusvm_notifier_unlock(&svm->gpusvm);
-	memalloc_noreclaim_restore(flags);
-
-	if (!ret && fence)
-		dma_fence_wait(fence, intr);
-	dma_fence_put(fence);
-
 	if (!ret)
-		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, NULL);
+		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm,
+					    svm->vm->eviction_lock);
 
 	if (!ret) {
-		if (flush_tlb)
-			amdgpu_svm_flush_tlb(svm);
 		WRITE_ONCE(range->attr_flags, attrs->flags);
 		WRITE_ONCE(range->gpu_mapped, true);
 		range->validate_timestamp = ktime_get_boottime();
 	}
 
+	drm_gpusvm_notifier_unlock(&svm->gpusvm);
+
+	if (!ret && fence)
+		dma_fence_wait(fence, intr);
+	dma_fence_put(fence);
+
+	if (!ret && flush_tlb)
+		amdgpu_svm_flush_tlb(svm);
+
 	drm_exec_fini(&exec);
 	return ret;
 }
@@ -585,9 +573,12 @@ amdgpu_svm_range_invalidate_interval(struct amdgpu_svm *svm,
 					crosses_boundary ? "ATTR DESTROY" :
 							   "ATTR ZAP PTE");
 
+				/* Use the unified eviction/MMU notifier lock for PTE zaps. */
+				down_write(svm->vm->eviction_lock);
 				ret = amdgpu_svm_range_zap_ptes(svm, svm_range,
 						drm_gpusvm_range_start(range) >> PAGE_SHIFT,
 						(drm_gpusvm_range_end(range) >> PAGE_SHIFT) - 1);
+				up_write(svm->vm->eviction_lock);
 				if (ret < 0) {
 					drm_exec_fini(&exec);
 					return ret;
-- 
2.53.0

[PATCH v10 16/22] drm/pagemap: fix page undercount for compound src entries

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

A migrate.src[] entry with MIGRATE_PFN_VALID=0 && MIGRATE_PFN_COMPOUND=1
represents a compound (THP) source page, but migrate_pfn_to_page() returns
NULL for it. Both accounting loops in drm_pagemap_migrate_to_devmem()
derive the count from that NULL page and fall back to nr_pages = 1 instead
of HPAGE_PMD_NR, undercounting migrated_pages and risking a spurious
-EBUSY in the final migration race check. drm_pagemap_cpages() already
handles this case, so the open-coded sites disagree.

Factor the decoding out into drm_pagemap_src_pfn_nr_pages() and use it in
drm_pagemap_cpages() and both accounting loops of
drm_pagemap_migrate_to_devmem(), so a COMPOUND-only entry is consistently
counted as HPAGE_PMD_NR base pages.

Fixes: 192cb1f5cb16 ("drm/pagemap: Enable THP support for GPU memory migration")
Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
Reviewed-by: Francois Dugast <francois.dugast@intel.com>
---
 drivers/gpu/drm/drm_pagemap.c | 60 +++++++++++++++++++++++++----------
 1 file changed, 43 insertions(+), 17 deletions(-)

diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
index 15c78eca180b7..df23721434d88 100644
--- a/drivers/gpu/drm/drm_pagemap.c
+++ b/drivers/gpu/drm/drm_pagemap.c
@@ -554,6 +554,37 @@ static int drm_pagemap_migrate_range(struct drm_pagemap_devmem *devmem,
 	return ret;
 }
 
+/**
+ * drm_pagemap_src_pfn_nr_pages() - Decode src entry and return base-page count
+ * @src_pfn: Source migrate entry
+ * @src_page: Optional decoded source page when MIGRATE_PFN_VALID is set
+ *
+ * Decode @src_pfn to compute how many base pages it represents: use folio
+ * page count for valid entries, HPAGE_PMD_NR for COMPOUND-only entries,
+ * otherwise 1.
+ *
+ * Return: Number of base pages represented by @src_pfn.
+ */
+static unsigned long drm_pagemap_src_pfn_nr_pages(unsigned long src_pfn,
+						   struct page **src_page)
+{
+	struct page *page = NULL;
+	unsigned long nr_pages = 1;
+
+	if (src_pfn & MIGRATE_PFN_VALID) {
+		page = migrate_pfn_to_page(src_pfn);
+		if (page)
+			nr_pages = NR_PAGES(folio_order(page_folio(page)));
+	} else if (src_pfn & MIGRATE_PFN_COMPOUND) {
+		nr_pages = HPAGE_PMD_NR;
+	}
+
+	if (src_page)
+		*src_page = page;
+
+	return nr_pages;
+}
+
 /**
  * drm_pagemap_cpages() - Count collected pages
  * @migrate_pfn: Array of migrate_pfn entries to account
@@ -570,20 +601,14 @@ static int drm_pagemap_cpages(unsigned long *migrate_pfn, unsigned long npages)
 	unsigned long i, cpages = 0;
 
 	for (i = 0; i < npages;) {
-		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
-		struct folio *folio;
-		unsigned int order = 0;
+		unsigned long src_pfn = migrate_pfn[i];
+		struct page *page;
+		unsigned long nr_pages = drm_pagemap_src_pfn_nr_pages(src_pfn, &page);
 
-		if (page) {
-			folio = page_folio(page);
-			order = folio_order(folio);
-			cpages += NR_PAGES(order);
-		} else if (migrate_pfn[i] & MIGRATE_PFN_COMPOUND) {
-			order = HPAGE_PMD_ORDER;
-			cpages += NR_PAGES(order);
-		}
+		if (page || (src_pfn & MIGRATE_PFN_COMPOUND))
+			cpages += nr_pages;
 
-		i += NR_PAGES(order);
+		i += nr_pages;
 	}
 
 	return cpages;
@@ -703,8 +728,9 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
 
 	/* Count device-private pages to migrate */
 	for (i = 0; i < npages;) {
-		struct page *src_page = migrate_pfn_to_page(migrate.src[i]);
-		unsigned long nr_pages = src_page ? NR_PAGES(folio_order(page_folio(src_page))) : 1;
+		unsigned long src_pfn = migrate.src[i];
+		struct page *src_page;
+		unsigned long nr_pages = drm_pagemap_src_pfn_nr_pages(src_pfn, &src_page);
 
 		if (src_page && is_zone_device_page(src_page)) {
 			if (page_pgmap(src_page) == pagemap)
@@ -813,10 +839,10 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
 	migrate_vma_pages(&migrate);
 
 	for (i = 0; !err && i < npages;) {
-		struct page *page = migrate_pfn_to_page(migrate.src[i]);
-		unsigned long nr_pages = page ? NR_PAGES(folio_order(page_folio(page))) : 1;
+		unsigned long src_pfn = migrate.src[i];
+		unsigned long nr_pages = drm_pagemap_src_pfn_nr_pages(src_pfn, NULL);
 
-		if (migrate.src[i] & MIGRATE_PFN_MIGRATE)
+		if (src_pfn & MIGRATE_PFN_MIGRATE)
 			migrated_pages += nr_pages;
 
 		i += nr_pages;
-- 
2.53.0

[PATCH v10 17/22] drm/amdgpu: add VRAM migration infrastructure for drm_pagemap

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Add the drm_pagemap-based VRAM migration infrastructure:

- Define struct amdgpu_pagemap wrapping dev_pagemap + drm_pagemap
- Define AMDGPU_PGMAP_OWNER() and AMDGPU_INTERCONNECT_VRAM macros
- Implement amdgpu_svm_migration_init() to register ZONE_DEVICE via
  devm_memremap_pages() and initialize the drm_pagemap
- Add amdgpu_pagemap pointer (apagemap) to struct amdgpu_device

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Makefile         |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |   8 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 252 ++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h |  92 +++++++
 4 files changed, 353 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index 96fd2b195c295..c78e9a0474425 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -328,7 +328,7 @@ amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
 
 # svm support
 amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
-	amdgpu_svm_fault.o amdgpu_svm_range.o
+	amdgpu_svm_fault.o amdgpu_svm_range.o amdgpu_migrate.o
 
 include $(FULL_AMD_PATH)/pm/Makefile
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 79b69d74eb2e5..bd4ffa85d5871 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -326,6 +326,7 @@ struct amdgpu_fpriv;
 struct amdgpu_bo_va_mapping;
 struct kfd_vm_fault_info;
 struct amdgpu_hive_info;
+struct amdgpu_pagemap;
 struct amdgpu_reset_context;
 struct amdgpu_reset_control;
 struct amdgpu_coredump_info;
@@ -965,6 +966,13 @@ struct amdgpu_device {
 
 	struct amdgpu_uma_carveout_info uma_info;
 
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+	/* SVM VRAM migration via drm_pagemap (drm_gpusvm path).
+	 * Allocated in amdgpu_svm_migration_init(), NULL if SVM disabled.
+	 */
+	struct amdgpu_pagemap		*apagemap;
+#endif
+
 	/* KFD
 	 * Must be last --ends in a flexible-array member.
 	 */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
new file mode 100644
index 0000000000000..054520d3ff1f7
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -0,0 +1,252 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+/**
+ * DOC: AMDGPU SVM Migration
+ *
+ * This file implements the drm_pagemap-based migration infrastructure for
+ * AMDGPU SVM. It provides the callbacks that the DRM GPUSVM / drm_pagemap
+ * framework needs to:
+ *
+ * 1. Map ZONE_DEVICE pages to GPU-visible VRAM MC addresses (device_map)
+ * 2. Allocate VRAM and migrate pages from system memory (populate_mm)
+ * 3. Copy data between RAM and VRAM using SDMA (copy_to_devmem / copy_to_ram)
+ * 4. Release VRAM backing when pages migrate back to system memory (devmem_release)
+ *
+ * Architecture overview::
+ *
+ *   adev->apagemap->dpagemap   (struct drm_pagemap)
+ *       .ops = &amdgpu_svm_drm_pagemap_ops
+ *           |
+ *       +---+-------------------+
+ *       |                       |
+ *  .populate_mm            .device_map
+ *  (alloc BO + migrate)    (page -> VRAM MC addr)
+ *       |
+ *       v
+ *   drm_pagemap_devmem_ops  (per-BO migration mechanics)
+ *       .populate_devmem_pfn  -> BO buddy blocks -> PFN array
+ *       .copy_to_devmem       -> SDMA copy RAM -> VRAM
+ *       .copy_to_ram          -> SDMA copy VRAM -> RAM
+ *       .devmem_release       -> release BO reference
+ *
+ * The three address spaces involved:
+ *
+ *   VRAM offset  [0, real_vram_size)     - buddy allocator managed
+ *       + hpa_base
+ *   HPA / PFN    [hpa_base, hpa_base+..) - ZONE_DEVICE struct page management
+ *       + vm_manager.vram_base_offset
+ *   PTE address  [vram_base_offset, ..]  - GPU page table entries (from MMHUB FB_OFFSET)
+ */
+
+#include <drm/drm_drv.h>
+#include <drm/drm_pagemap.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+
+#include "amdgpu_migrate.h"
+#include "amdgpu.h"
+
+static inline struct amdgpu_pagemap *
+dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
+{
+	return container_of(dpagemap, struct amdgpu_pagemap, dpagemap);
+}
+
+static inline struct amdgpu_device *
+dpagemap_to_adev(struct drm_pagemap *dpagemap)
+{
+	return drm_to_adev(dpagemap->drm);
+}
+
+/**
+ * amdgpu_svm_page_to_apagemap - Get amdgpu_pagemap from a ZONE_DEVICE page
+ * @page: A ZONE_DEVICE page backed by VRAM
+ *
+ * Follows: page -> pgmap -> container_of(apagemap)
+ */
+static inline struct amdgpu_pagemap *
+amdgpu_svm_page_to_apagemap(struct page *page)
+{
+	struct dev_pagemap *pgmap = page_pgmap(page);
+
+	return container_of(pgmap, struct amdgpu_pagemap, pgmap);
+}
+
+/**
+ * amdgpu_pagemap_destroy_work - Deferred destruction of amdgpu_pagemap
+ * @work: work_struct embedded in amdgpu_pagemap
+ *
+ * Called when drm_pagemap refcount drops to zero.  Releases the
+ * ZONE_DEVICE mapping and frees the amdgpu_pagemap structure.
+ * Uses drm_dev_enter/exit to ensure device is still valid.
+ */
+static void amdgpu_pagemap_destroy_work(struct work_struct *work)
+{
+	struct amdgpu_pagemap *apagemap =
+		container_of(work, struct amdgpu_pagemap, destroy_work);
+	struct dev_pagemap *pgmap = &apagemap->pgmap;
+	struct drm_device *drm = apagemap->dpagemap.drm;
+	int idx;
+
+	/*
+	 * Only unmap/release if devm release hasn't run yet.
+	 * Otherwise the devm callbacks have already released, or
+	 * will do shortly.
+	 */
+	if (drm_dev_enter(drm, &idx)) {
+		devm_memunmap_pages(drm->dev, pgmap);
+		if (pgmap->type == MEMORY_DEVICE_PRIVATE)
+			devm_release_mem_region(drm->dev, pgmap->range.start,
+						pgmap->range.end - pgmap->range.start + 1);
+		drm_dev_exit(idx);
+	}
+
+	kfree(apagemap);
+}
+
+/**
+ * amdgpu_pagemap_destroy - drm_pagemap_ops.destroy callback
+ * @dpagemap: the drm_pagemap being destroyed
+ * @from_atomic_or_reclaim: true if called from atomic/reclaim context
+ *
+ * Called when the drm_pagemap refcount drops to zero.  If called from
+ * atomic or reclaim context, queues the destroy work to run later.
+ * Otherwise performs cleanup synchronously.
+ */
+static void amdgpu_pagemap_destroy(struct drm_pagemap *dpagemap,
+				   bool from_atomic_or_reclaim)
+{
+	struct amdgpu_pagemap *apagemap = dpagemap_to_apagemap(dpagemap);
+
+	if (from_atomic_or_reclaim)
+		schedule_work(&apagemap->destroy_work);
+	else
+		amdgpu_pagemap_destroy_work(&apagemap->destroy_work);
+}
+
+const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = {
+	.destroy = amdgpu_pagemap_destroy,
+};
+
+/**
+ * amdgpu_svm_migration_init - Register ZONE_DEVICE and initialize drm_pagemap
+ * @adev: AMDGPU device to set up VRAM migration for
+ *
+ * Allocates a ZONE_DEVICE region covering the GPU's VRAM, registers it
+ * via devm_memremap_pages() with drm_pagemap's generic dev_pagemap_ops,
+ * and then initializes the drm_pagemap (dpagemap) that provides the
+ * device_map / populate_mm callbacks for the DRM GPUSVM migration path.
+ *
+ * For XGMI-connected CPUs, uses the device's aperture directly
+ * (MEMORY_DEVICE_COHERENT).  For discrete GPUs, requests a free
+ * iomem region for MEMORY_DEVICE_PRIVATE pages.
+ *
+ * Return: 0 on success, -EINVAL if GPU IP too old, negative error on failure
+ */
+int amdgpu_svm_migration_init(struct amdgpu_device *adev)
+{
+	struct amdgpu_pagemap *apagemap;
+	struct drm_pagemap *dpagemap;
+	struct dev_pagemap *pgmap;
+	struct resource *res = NULL;
+	unsigned long size;
+	void *r;
+	int err;
+
+	if (amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(9, 0, 1))
+		return -EINVAL;
+
+	if (adev->apu_prefer_gtt)
+		return 0;
+
+	if (adev->apagemap && adev->apagemap->initialized)
+		return 0;
+
+	apagemap = kzalloc(sizeof(*apagemap), GFP_KERNEL);
+	if (!apagemap)
+		return -ENOMEM;
+
+	pgmap = &apagemap->pgmap;
+	dpagemap = &apagemap->dpagemap;
+	INIT_WORK(&apagemap->destroy_work, amdgpu_pagemap_destroy_work);
+	apagemap->adev = adev;
+
+	/* Initialize drm_pagemap first — takes refs on drm_device and module */
+	err = drm_pagemap_init(dpagemap, pgmap, adev_to_drm(adev),
+			       &amdgpu_svm_drm_pagemap_ops);
+	if (err) {
+		dev_err(adev->dev, "SVM: failed to init drm_pagemap\n");
+		goto err_free;
+	}
+
+	/* Now allocate ZONE_DEVICE resources */
+	size = ALIGN(adev->gmc.real_vram_size, 2ULL << 20);
+	if (adev->gmc.xgmi.connected_to_cpu) {
+		pgmap->range.start = adev->gmc.aper_base;
+		pgmap->range.end = adev->gmc.aper_base + adev->gmc.aper_size - 1;
+		pgmap->type = MEMORY_DEVICE_COHERENT;
+	} else {
+		res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
+		if (IS_ERR(res)) {
+			err = PTR_ERR(res);
+			goto err_pagemap;
+		}
+		pgmap->range.start = res->start;
+		pgmap->range.end = res->end;
+		pgmap->type = MEMORY_DEVICE_PRIVATE;
+	}
+
+	pgmap->nr_range = 1;
+	pgmap->flags = 0;
+	pgmap->ops = drm_pagemap_pagemap_ops_get();
+	pgmap->owner = AMDGPU_PGMAP_OWNER(adev);
+
+	r = devm_memremap_pages(adev->dev, pgmap);
+	if (IS_ERR(r)) {
+		dev_err(adev->dev, "SVM: failed to register HMM device memory\n");
+		err = PTR_ERR(r);
+		goto err_mem_region;
+	}
+
+	apagemap->hpa_base = pgmap->range.start;
+	apagemap->initialized = true;
+	adev->apagemap = apagemap;
+
+	dev_info(adev->dev, "SVM: registered %ldMB device memory, hpa_base=0x%llx\n",
+		 size >> 20, apagemap->hpa_base);
+	return 0;
+
+err_mem_region:
+	if (pgmap->type == MEMORY_DEVICE_PRIVATE && res)
+		devm_release_mem_region(adev->dev, res->start, resource_size(res));
+	pgmap->type = 0;
+err_pagemap:
+	drm_pagemap_put(dpagemap);
+	return err;
+
+err_free:
+	kfree(apagemap);
+	return err;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
new file mode 100644
index 0000000000000..f5f726a740469
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_MIGRATE_H__
+#define __AMDGPU_MIGRATE_H__
+
+#include <drm/drm_pagemap.h>
+#include <linux/memremap.h>
+#include <linux/workqueue.h>
+
+struct amdgpu_device;
+
+/*
+ * AMDGPU_PGMAP_OWNER - Unique owner token for dev_pagemap registration.
+ *
+ * migrate_vma_setup() uses pgmap->owner to distinguish "own" device pages
+ * from "foreign" device pages (e.g., another GPU in an XGMI hive).
+ * Pages whose page->pgmap->owner matches the migration source are skipped
+ * (they're already in the right place).
+ *
+ * For XGMI hive: all GPUs in the hive share the same owner (the hive pointer)
+ * so intra-hive pages are treated as local.
+ * For standalone GPU: use the adev pointer itself as a unique per-device token.
+ */
+#define AMDGPU_PGMAP_OWNER(adev) \
+	((adev)->hive ? (void *)(adev)->hive : (void *)(adev))
+
+/**
+ * struct amdgpu_pagemap - VRAM migration infrastructure for drm_pagemap
+ * @dpagemap: DRM pagemap wrapper providing device_map/populate_mm callbacks
+ * @adev: back-pointer to the owning amdgpu_device
+ * @hpa_base: host physical address base of the ZONE_DEVICE region
+ * @initialized: set to true after successful registration
+ * @destroy_work: work struct for deferred destruction when called from
+ *                atomic/reclaim context
+ * @pgmap: the dev_pagemap registered with devm_memremap_pages();
+ *         must be last — contains a flexible-array member (ranges[])
+ *
+ * Allocated with kzalloc() in amdgpu_svm_migration_init() and stored
+ * as adev->apagemap.  Freed via drm_pagemap_ops.destroy callback when
+ * dpagemap refcount drops to zero.
+ */
+struct amdgpu_pagemap {
+	struct drm_pagemap	dpagemap;
+	struct amdgpu_device	*adev;
+	resource_size_t		hpa_base;
+	bool			initialized;
+	struct work_struct	destroy_work;
+	struct dev_pagemap	pgmap;	/* must be last — flex-array */
+};
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+int amdgpu_svm_migration_init(struct amdgpu_device *adev);
+#else
+static inline
+int amdgpu_svm_migration_init(struct amdgpu_device *adev)
+{
+	return 0;
+}
+#endif
+
+/**
+ * amdgpu_svm_drm_pagemap_ops - drm_pagemap_ops for AMDGPU VRAM migration
+ *
+ * Provides:
+ *   .device_map  - Convert ZONE_DEVICE page to VRAM address
+ *   .populate_mm - Allocate VRAM BO and migrate pages from system memory
+ */
+extern const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops;
+
+#endif /* __AMDGPU_MIGRATE_H__ */
-- 
2.53.0

[PATCH v10 18/22] drm/amdgpu: implement drm_pagemap SDMA migration callbacks

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Implement the drm_pagemap_devmem_ops and drm_pagemap_ops callbacks
that the DRM GPUSVM migration framework requires.

Introduce struct amdgpu_bo_svm as a BO subtype (following the
amdgpu_bo_user/amdgpu_bo_vm pattern) that embeds struct amdgpu_bo
and carries a drm_pagemap_devmem allocation.

drm_pagemap_ops (top-level entry points):
  - device_map:   convert ZONE_DEVICE page to GPU PTE address
  - populate_mm:  allocate amdgpu_bo_svm and trigger migration

drm_pagemap_devmem_ops (per-BO migration mechanics):
  - populate_devmem_pfn: walk BO buddy blocks to build PFN array
  - copy_to_devmem:      SDMA copy system RAM -> VRAM via GART window
  - copy_to_ram:         SDMA copy VRAM -> system RAM via GART window
  - devmem_release:      drop BO ref (triggers destroy to free amdgpu_bo_svm)

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 601 ++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h |   3 +
 2 files changed, 604 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
index 054520d3ff1f7..20daf2922e0f6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -67,6 +67,14 @@
 
 #include "amdgpu_migrate.h"
 #include "amdgpu.h"
+#include "amdgpu_ttm.h"
+#include "amdgpu_res_cursor.h"
+
+/* SDMA copy direction */
+#define FROM_RAM_TO_VRAM	0
+#define FROM_VRAM_TO_RAM	1
+
+static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops;
 
 static inline struct amdgpu_pagemap *
 dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
@@ -94,6 +102,597 @@ amdgpu_svm_page_to_apagemap(struct page *page)
 	return container_of(pgmap, struct amdgpu_pagemap, pgmap);
 }
 
+/**
+ * struct amdgpu_bo_svm - SVM BO subtype with drm_pagemap devmem allocation
+ *
+ * @bo: Embedded base amdgpu_bo
+ * @devmem: drm_pagemap device memory allocation (passed to framework)
+ *
+ * Lifecycle is managed by the drm_pagemap framework's internal zdd refcount:
+ *   - zdd->devmem_allocation points to &svm_bo->devmem
+ *   - When zdd refcount drops to zero, framework calls devmem_release()
+ *   - devmem_release() drops the BO reference (triggering destroy callback)
+ */
+struct amdgpu_bo_svm {
+	struct amdgpu_bo bo;
+	struct drm_pagemap_devmem devmem;
+};
+
+#define to_amdgpu_bo_svm(abo) container_of((abo), struct amdgpu_bo_svm, bo)
+
+static inline struct amdgpu_bo_svm *
+devmem_to_amdgpu_bo_svm(struct drm_pagemap_devmem *devmem_allocation)
+{
+	return container_of(devmem_allocation, struct amdgpu_bo_svm, devmem);
+}
+
+/**
+ * amdgpu_bo_svm_destroy - TTM destroy callback for SVM BO
+ *
+ * Called when the last reference to the BO is dropped.
+ * Follows amdgpu_bo_user_destroy pattern: delegates to amdgpu_bo_base_destroy.
+ */
+static void amdgpu_bo_svm_destroy(struct ttm_buffer_object *tbo)
+{
+	struct amdgpu_bo *bo = ttm_to_amdgpu_bo(tbo);
+	struct amdgpu_bo_svm *svm_bo = to_amdgpu_bo_svm(bo);
+
+	amdgpu_bo_kunmap(bo);
+	drm_gem_object_release(&bo->tbo.base);
+	amdgpu_bo_unref(&bo->parent);
+	kvfree(svm_bo);
+}
+
+/**
+ * amdgpu_bo_svm_alloc - Allocate an amdgpu_bo_svm with VRAM backing
+ * @adev: AMDGPU device
+ * @dpagemap: The drm_pagemap for this device
+ * @mm: mm_struct of the owning process
+ * @size: Allocation size in bytes
+ *
+ * Uses bp->bo_ptr_size = sizeof(struct amdgpu_bo_svm) so that
+ * amdgpu_bo_create() allocates the full amdgpu_bo_svm structure,
+ * with the embedded amdgpu_bo as the base.
+ *
+ * Return: Pointer to allocated amdgpu_bo_svm on success, ERR_PTR on failure
+ */
+static struct amdgpu_bo_svm *
+amdgpu_bo_svm_alloc(struct amdgpu_device *adev,
+		     struct drm_pagemap *dpagemap,
+		     struct mm_struct *mm, unsigned long size)
+{
+	struct dma_fence *pre_migrate_fence = NULL;
+	struct amdgpu_bo_param bp = {};
+	struct amdgpu_bo_svm *svm_bo;
+	struct amdgpu_bo *bo;
+	int ret;
+
+	bp.size = size;
+	bp.bo_ptr_size = sizeof(struct amdgpu_bo_svm);
+	bp.destroy = &amdgpu_bo_svm_destroy;
+	bp.domain = AMDGPU_GEM_DOMAIN_VRAM;
+	bp.type = ttm_bo_type_device;
+	bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS |
+		   AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
+		   AMDGPU_GEM_CREATE_VRAM_CLEARED;
+
+	ret = amdgpu_bo_create(adev, &bp, &bo);
+	if (ret) {
+		pr_debug("Failed to create SVM BO\n");
+		return ERR_PTR(ret);
+	}
+
+	svm_bo = to_amdgpu_bo_svm(bo);
+
+	/* Ensure any async VRAM operations complete before migration copy */
+	if (!dma_resv_test_signaled(bo->tbo.base.resv,
+					 DMA_RESV_USAGE_KERNEL)) {
+		ret = dma_resv_get_singleton(bo->tbo.base.resv,
+					      DMA_RESV_USAGE_KERNEL,
+					      &pre_migrate_fence);
+		if (ret) {
+			amdgpu_bo_unref(&bo);
+			return ERR_PTR(ret);
+		}
+	}
+
+	drm_pagemap_devmem_init(&svm_bo->devmem, adev->dev, mm,
+				&amdgpu_pagemap_ops, dpagemap, size,
+				pre_migrate_fence);
+
+	return svm_bo;
+}
+
+/**
+ * amdgpu_svm_devmem_release - Release BO when all device pages migrate back
+ *
+ * Called by the drm_pagemap framework (via drm_pagemap_zdd_destroy) when the
+ * last device-private page backed by this allocation has been migrated back
+ * to system memory (or the owning process exits).
+ *
+ * Drops the BO reference, which triggers the destroy callback to free
+ * the amdgpu_bo_svm structure.
+ */
+static void
+amdgpu_svm_devmem_release(struct drm_pagemap_devmem *devmem_allocation)
+{
+	struct amdgpu_bo_svm *svm_bo = devmem_to_amdgpu_bo_svm(devmem_allocation);
+	struct amdgpu_bo *bo = &svm_bo->bo;
+
+	pr_debug("Release svm_bo=%px bo=%px\n", svm_bo, bo);
+	dma_fence_put(devmem_allocation->pre_migrate_fence);
+	devmem_allocation->pre_migrate_fence = NULL;
+	amdgpu_bo_unref(&bo);
+}
+
+/**
+ * amdgpu_svm_populate_devmem_pfn - Convert BO VRAM allocation to PFN array
+ * @devmem_allocation: The devmem allocation in the amdgpu_bo_svm wrapper
+ * @npages: Number of PFN entries to fill
+ * @pfn: Output PFN array
+ *
+ * Iterates over the BO's TTM vram_mgr buddy blocks and converts each
+ * block's VRAM offset to ZONE_DEVICE PFNs:
+ *
+ *   PFN = PHYS_PFN(block_offset + apagemap.hpa_base) + page_index
+ *
+ * This is called by drm_pagemap_migrate_to_devmem() to build the
+ * destination PFN array for migrate_vma_pages().
+ *
+ * Return: 0 on success
+ */
+static int
+amdgpu_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocation,
+				unsigned long npages, unsigned long *pfn)
+{
+	struct amdgpu_pagemap *svm_dm = dpagemap_to_apagemap(devmem_allocation->dpagemap);
+	struct amdgpu_bo_svm *svm_bo = devmem_to_amdgpu_bo_svm(devmem_allocation);
+	struct amdgpu_bo *bo = &svm_bo->bo;
+	struct amdgpu_res_cursor cursor;
+	unsigned long i = 0;
+
+	dma_resv_assert_held(bo->tbo.base.resv);
+
+	amdgpu_res_first(bo->tbo.resource, 0, npages << PAGE_SHIFT, &cursor);
+	while (cursor.remaining && i < npages) {
+		u64 pfn_base = PHYS_PFN(cursor.start + svm_dm->hpa_base);
+		u64 pages = cursor.size >> PAGE_SHIFT;
+		unsigned long j;
+
+		for (j = 0; j < pages && i < npages; j++, i++)
+			pfn[i] = pfn_base + j;
+
+		amdgpu_res_next(&cursor, cursor.size);
+	}
+
+	pr_debug("populate_devmem_pfn: npages=%lu first_pfn=0x%lx\n",
+		 npages, npages > 0 ? pfn[0] : 0);
+
+	return 0;
+}
+
+/* SDMA copy helpers — GART window based data transfer */
+
+/**
+ * amdgpu_svm_direct_mapping_addr - Convert VRAM offset to MC address
+ * @adev: AMDGPU device
+ * @vram_offset: Byte offset within VRAM
+ *
+ * Return: MC address suitable for SDMA src/dst
+ */
+static u64
+amdgpu_svm_direct_mapping_addr(struct amdgpu_device *adev, u64 vram_offset)
+{
+	return vram_offset + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM);
+}
+
+/**
+ * amdgpu_svm_gart_map - Map system DMA addresses into GART window
+ * @ring: SDMA ring for the GART update job
+ * @npages: Number of pages to map
+ * @addr: Array of system memory DMA addresses
+ * @gart_addr: Output — GART base address to use in SDMA copy
+ * @flags: PTE flags (e.g. writeable for RAM-to-VRAM src)
+ *
+ * Builds GART PTEs pointing at the given DMA addresses, submits an
+ * SDMA job to update the GART entries, and returns the GART address
+ * that can be used as src or dst in a subsequent amdgpu_copy_buffer().
+ *
+ * Uses GART window 0, protected by gtt_window_lock.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_gart_map(struct amdgpu_ring *ring,
+		     struct amdgpu_ttm_buffer_entity *entity,
+		     u64 npages,
+		     dma_addr_t *addr, u64 *gart_addr, u64 flags)
+{
+	struct amdgpu_device *adev = ring->adev;
+	struct amdgpu_job *job;
+	unsigned int num_dw, num_bytes;
+	struct dma_fence *fence;
+	u64 src_addr, dst_addr;
+	u64 pte_flags;
+	void *cpu_addr;
+	int r;
+
+	/* Use entity's GART window 0 */
+	*gart_addr = amdgpu_compute_gart_address(&adev->gmc, entity, 0);
+
+	num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
+	num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+
+	r = amdgpu_job_alloc_with_ib(adev, &entity->base,
+				     AMDGPU_FENCE_OWNER_UNDEFINED,
+				     num_dw * 4 + num_bytes,
+				     AMDGPU_IB_POOL_DELAYED,
+				     AMDGPU_KERNEL_JOB_ID_KFD_GART_MAP,
+				     &job);
+	if (r)
+		return r;
+
+	src_addr = num_dw * 4;
+	src_addr += job->ibs[0].gpu_addr;
+
+	dst_addr = amdgpu_bo_gpu_offset(adev->gart.bo);
+	dst_addr += (entity->gart_window_offs[0] >> AMDGPU_GPU_PAGE_SHIFT) * 8;
+	amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr,
+				dst_addr, num_bytes, 0);
+
+	amdgpu_ring_pad_ib(ring, &job->ibs[0]);
+	WARN_ON(job->ibs[0].length_dw > num_dw);
+
+	pte_flags = AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE;
+	pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED;
+	if (flags & AMDGPU_PTE_WRITEABLE)
+		pte_flags |= AMDGPU_PTE_WRITEABLE;
+	pte_flags |= adev->gart.gart_pte_flags;
+
+	cpu_addr = &job->ibs[0].ptr[num_dw];
+
+	amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr);
+	fence = amdgpu_job_submit(job);
+	dma_fence_put(fence);
+
+	return 0;
+}
+
+/**
+ * amdgpu_svm_copy_memory_gart - SDMA copy between system RAM and VRAM
+ * @adev: AMDGPU device
+ * @sys: Array of DMA addresses for system memory pages
+ * @vram: Array of VRAM byte offsets (relative to start of VRAM)
+ * @npages: Number of pages to copy
+ * @direction: FROM_RAM_TO_VRAM or FROM_VRAM_TO_RAM
+ * @mfence: In/out — carries the last SDMA fence for serialization
+ *
+ * Maps system memory pages into the GART window and uses SDMA to copy
+ * data to/from VRAM. Handles splitting into AMDGPU_GTT_MAX_TRANSFER_SIZE
+ * chunks. Acquires entity->lock internally to protect the GART window,
+ * matching the KFD svm_migrate_copy_memory_gart() pattern.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys,
+			    u64 *vram, u64 npages, int direction,
+			    struct dma_fence **mfence)
+{
+	const u64 max_pages = AMDGPU_GTT_MAX_TRANSFER_SIZE >> PAGE_SHIFT;
+	struct amdgpu_ring *ring = to_amdgpu_ring(adev->mman.buffer_funcs_scheds[0]);
+	struct amdgpu_ttm_buffer_entity *entity = &adev->mman.move_entities[0];
+	u64 gart_s, gart_d;
+	struct dma_fence *next;
+	u64 size;
+	int r;
+
+	mutex_lock(&entity->lock);
+
+	while (npages) {
+		size = min(max_pages, npages);
+
+		if (direction == FROM_VRAM_TO_RAM) {
+			gart_s = amdgpu_svm_direct_mapping_addr(adev, *vram);
+			r = amdgpu_svm_gart_map(ring, entity, size, sys,
+						&gart_d, AMDGPU_PTE_WRITEABLE);
+		} else {
+			r = amdgpu_svm_gart_map(ring, entity, size, sys,
+						&gart_s, 0);
+			gart_d = amdgpu_svm_direct_mapping_addr(adev, *vram);
+		}
+		if (r) {
+			dev_err(adev->dev, "failed %d to map GART for SDMA\n", r);
+			goto out_unlock;
+		}
+
+		pr_debug("SDMA_COPY: %s npages=%llu vram_off=0x%llx\n",
+			 direction == FROM_RAM_TO_VRAM ? "RAM->VRAM" : "VRAM->RAM",
+			 size, (u64)*vram);
+
+		r = amdgpu_copy_buffer(adev, entity, gart_s, gart_d,
+				       size * PAGE_SIZE,
+				       NULL, &next, true, 0);
+		if (r) {
+			dev_err(adev->dev, "failed %d to copy buffer\n", r);
+			goto out_unlock;
+		}
+
+		dma_fence_put(*mfence);
+		*mfence = next;
+		npages -= size;
+		if (npages) {
+			sys += size;
+			vram += size;
+		}
+	}
+
+out_unlock:
+	mutex_unlock(&entity->lock);
+
+	return r;
+}
+
+/**
+ * amdgpu_svm_copy_pages - SDMA copy between system memory and VRAM
+ * @pages: ZONE_DEVICE pages — destination when direction=FROM_RAM_TO_VRAM,
+ *         source when direction=FROM_VRAM_TO_RAM
+ * @pagemap_addr: DMA addresses — source when direction=FROM_RAM_TO_VRAM,
+ *                destination when direction=FROM_VRAM_TO_RAM
+ * @npages: Number of framework slots in the arrays
+ * @pre_migrate_fence: Optional fence to wait before the first SDMA submission
+ * @direction: FROM_RAM_TO_VRAM or FROM_VRAM_TO_RAM
+ *
+ * Builds parallel sys[] (DMA addresses) and vram[] (VRAM byte offsets) arrays
+ * from the framework-provided inputs, then submits batched SDMA copies via
+ * the GART window.  Contiguous VRAM runs are coalesced into a single
+ * amdgpu_svm_copy_memory_gart() call; discontiguity flushes the current batch
+ * and starts a new one.  The pre_migrate_fence wait is deferred until the
+ * first actual SDMA submission.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_pages(struct page **pages,
+		       struct drm_pagemap_addr *pagemap_addr,
+		       unsigned long npages,
+		       struct dma_fence *pre_migrate_fence,
+		       int direction)
+{
+	struct amdgpu_device *adev;
+	struct amdgpu_pagemap *svm_dm;
+	struct dma_fence *mfence = NULL;
+	dma_addr_t *sys = NULL;
+	u64 *vram = NULL;
+	unsigned long i, j;
+	bool waited = false;
+	int ret = 0;
+
+	if (!npages)
+		return 0;
+
+	for (i = 0; i < npages; i++) {
+		if (pages[i] && pagemap_addr[i].addr)
+			break;
+	}
+	if (i == npages) {
+		/*
+		 * No data to copy (e.g. anonymous zero-hole migration to
+		 * device).  We still must wait for the pre_migrate_fence (the
+		 * async VRAM clear) before the freshly populated device pages
+		 * become user-visible, otherwise a late clear could overwrite
+		 * subsequent user writes.
+		 */
+		if (pre_migrate_fence)
+			dma_fence_wait(pre_migrate_fence, false);
+		return 0;
+	}
+
+	svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
+	adev = svm_dm->adev;
+
+	sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
+	vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
+	if (!sys || !vram) {
+		ret = -ENOMEM;
+		goto out_free;
+	}
+
+	for (i = 0, j = 0; i < npages; i++) {
+		unsigned long nr_pages;
+		dma_addr_t sys_base;
+		u64 vram_base;
+		unsigned long k;
+
+		if (!pagemap_addr[i].addr || !pages[i])
+			goto flush;
+
+		nr_pages = folio_nr_pages(page_folio(pages[i]));
+		sys_base = pagemap_addr[i].addr;
+		vram_base = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+			    svm_dm->hpa_base;
+
+		if (j > 0 && vram_base != vram[j - 1] + PAGE_SIZE)
+			goto flush;
+
+		for (k = 0; k < nr_pages; k++) {
+			sys[j] = sys_base + k * PAGE_SIZE;
+			vram[j] = vram_base + k * PAGE_SIZE;
+			j++;
+		}
+		continue;
+flush:
+		if (j) {
+			if (pre_migrate_fence && !waited) {
+				dma_fence_wait(pre_migrate_fence, false);
+				waited = true;
+			}
+			ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+							  direction, &mfence);
+			if (ret)
+				goto out_fence;
+			j = 0;
+		}
+		/* Re-process current page if it was valid but broke contiguity */
+		if (pagemap_addr[i].addr && pages[i]) {
+			nr_pages = folio_nr_pages(page_folio(pages[i]));
+			sys_base = pagemap_addr[i].addr;
+			vram_base = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+				    svm_dm->hpa_base;
+			for (k = 0; k < nr_pages; k++) {
+				sys[k] = sys_base + k * PAGE_SIZE;
+				vram[k] = vram_base + k * PAGE_SIZE;
+			}
+			j = nr_pages;
+		}
+	}
+
+	/* Flush remaining batch */
+	if (j) {
+		if (pre_migrate_fence && !waited)
+			dma_fence_wait(pre_migrate_fence, false);
+		ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+						  direction, &mfence);
+	}
+
+out_fence:
+	if (mfence) {
+		dma_fence_wait(mfence, false);
+		dma_fence_put(mfence);
+	}
+
+out_free:
+	kvfree(vram);
+	kvfree(sys);
+
+	pr_debug("copy_pages done: npages=%ld direction=%d ret=%d\n",
+		 npages, direction, ret);
+	return ret;
+}
+
+static int
+amdgpu_svm_copy_to_devmem(struct page **pages,
+			   struct drm_pagemap_addr *pagemap_addr,
+			   unsigned long npages,
+			   struct dma_fence *pre_migrate_fence)
+{
+	return amdgpu_svm_copy_pages(pages, pagemap_addr, npages,
+				     pre_migrate_fence, FROM_RAM_TO_VRAM);
+}
+
+static int
+amdgpu_svm_copy_to_ram(struct page **pages,
+		        struct drm_pagemap_addr *pagemap_addr,
+		        unsigned long npages,
+		        struct dma_fence *pre_migrate_fence)
+{
+	return amdgpu_svm_copy_pages(pages, pagemap_addr, npages,
+				     pre_migrate_fence, FROM_VRAM_TO_RAM);
+}
+
+static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = {
+	.devmem_release      = amdgpu_svm_devmem_release,
+	.populate_devmem_pfn = amdgpu_svm_populate_devmem_pfn,
+	.copy_to_devmem      = amdgpu_svm_copy_to_devmem,
+	.copy_to_ram         = amdgpu_svm_copy_to_ram,
+};
+
+/* drm_pagemap_ops — top-level migration entry points */
+
+/**
+ * amdgpu_svm_device_map - Convert ZONE_DEVICE page to GPU PTE address
+ * @dpagemap: The drm_pagemap for this device
+ * @dev: Requesting device (for P2P check)
+ * @page: ZONE_DEVICE page backed by VRAM
+ * @order: Page order (0 = 4K, 9 = 2M, etc.)
+ * @dir: DMA direction (unused for local VRAM)
+ *
+ * Address conversion chain:
+ *   page -> PFN -> HPA -> VRAM offset -> PTE address
+ *
+ *   HPA = page_to_pfn(page) << PAGE_SHIFT
+ *   VRAM offset = HPA - apagemap.hpa_base
+ *   PTE address = VRAM offset + adev->vm_manager.vram_base_offset
+ *
+ * Return: drm_pagemap_addr with PTE address and AMDGPU_INTERCONNECT_VRAM protocol
+ */
+static struct drm_pagemap_addr
+amdgpu_svm_device_map(struct drm_pagemap *dpagemap,
+		       struct device *dev,
+		       struct page *page,
+		       unsigned int order,
+		       enum dma_data_direction dir)
+{
+	struct amdgpu_pagemap *svm_dm = dpagemap_to_apagemap(dpagemap);
+	struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
+	dma_addr_t addr;
+
+	if (dpagemap->drm->dev == dev) {
+		/* Same device: return VRAM PTE address */
+		u64 hpa = (u64)page_to_pfn(page) << PAGE_SHIFT;
+		u64 vram_offset = hpa - svm_dm->hpa_base;
+
+		addr = vram_offset + adev->vm_manager.vram_base_offset;
+	} else {
+		/* Cross-device P2P: not yet supported */
+		addr = DMA_MAPPING_ERROR;
+	}
+
+	return drm_pagemap_addr_encode(addr,
+				AMDGPU_INTERCONNECT_VRAM, order, dir);
+}
+
+/**
+ * amdgpu_svm_populate_mm - Allocate VRAM BO and migrate pages
+ * @dpagemap: The drm_pagemap for this device
+ * @start: Start virtual address of the range to migrate
+ * @end: End virtual address (exclusive)
+ * @mm: mm_struct of the owning process
+ * @timeslice_ms: Maximum time to spend migrating (for fairness)
+ *
+ * Core migration entry point called by drm_pagemap_populate_mm().
+ * Allocates an amdgpu_bo_svm via amdgpu_bo_svm_alloc(), then calls
+ * drm_pagemap_migrate_to_devmem() to execute the actual migration.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_populate_mm(struct drm_pagemap *dpagemap,
+			unsigned long start, unsigned long end,
+			struct mm_struct *mm,
+			unsigned long timeslice_ms)
+{
+	struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
+	struct drm_pagemap_migrate_details mdetails = {
+		.timeslice_ms = timeslice_ms,
+	};
+	struct amdgpu_bo_svm *svm_bo;
+	int ret;
+
+	svm_bo = amdgpu_bo_svm_alloc(adev, dpagemap, mm, end - start);
+	if (IS_ERR(svm_bo))
+		return PTR_ERR(svm_bo);
+
+	pr_debug("populate_mm: [0x%lx-0x%lx] size=%lu\n",
+		 start, end, end - start);
+
+	ret = amdgpu_bo_reserve(&svm_bo->bo, true);
+	if (ret) {
+		struct amdgpu_bo *bo = &svm_bo->bo;
+
+		amdgpu_bo_unref(&bo);
+		return ret;
+	}
+
+	ret = drm_pagemap_migrate_to_devmem(&svm_bo->devmem,
+					     mm, start, end,
+					     &mdetails);
+
+	amdgpu_bo_unreserve(&svm_bo->bo);
+
+	return ret;
+}
+
 /**
  * amdgpu_pagemap_destroy_work - Deferred destruction of amdgpu_pagemap
  * @work: work_struct embedded in amdgpu_pagemap
@@ -148,6 +747,8 @@ static void amdgpu_pagemap_destroy(struct drm_pagemap *dpagemap,
 
 const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = {
 	.destroy = amdgpu_pagemap_destroy,
+	.device_map = amdgpu_svm_device_map,
+	.populate_mm = amdgpu_svm_populate_mm,
 };
 
 /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
index f5f726a740469..56c66851f0a47 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
@@ -31,6 +31,9 @@
 
 struct amdgpu_device;
 
+#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
+#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
+
 /*
  * AMDGPU_PGMAP_OWNER - Unique owner token for dev_pagemap registration.
  *
-- 
2.53.0

[PATCH v10 19/22] drm/amdgpu: implement synchronous TTM eviction for SVM BOs

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Implement the TTM eviction path for SVM (Shared Virtual Memory) BOs,
enabling VRAM overcommit scenarios where device-private pages must be
migrated back to system memory when VRAM is full.

- amdgpu_ttm.c: Add SVM BO detection in amdgpu_evict_flags() to evict
  directly to SYSTEM domain. Add SVM BO handling in amdgpu_bo_move()
  to call amdgpu_svm_bo_evict() for VRAM->SYSTEM transitions, which
  synchronously migrates device-private pages back to RAM via
  SDMA.

- amdgpu_migrate.c: Add amdgpu_svm_bo_evict() and
  amdgpu_svm_bo_is_svm() helpers.

- amdgpu_object.c: Register SVM BO destroy callback in
  amdgpu_bo_is_amdgpu_bo() so TTM eviction can identify and handle
  SVM BOs correctly.

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 46 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h | 12 ++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c     | 20 +++++++++
 4 files changed, 81 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
index 20daf2922e0f6..cd7483ad3cc3d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -64,6 +64,7 @@
 #include <drm/drm_pagemap.h>
 #include <linux/memremap.h>
 #include <linux/migrate.h>
+#include <linux/sched/mm.h>
 
 #include "amdgpu_migrate.h"
 #include "amdgpu.h"
@@ -597,6 +598,51 @@ static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = {
 	.copy_to_ram         = amdgpu_svm_copy_to_ram,
 };
 
+/**
+ * amdgpu_svm_bo_evict - Evict SVM BO by migrating device pages back to RAM
+ * @bo: The amdgpu_bo to evict (must be an amdgpu_bo_svm)
+ *
+ * Called from amdgpu_bo_move() when TTM needs to evict a SVM BO from VRAM.
+ * Uses drm_pagemap_evict_to_ram() to synchronously migrate all device-private
+ * pages back to system memory via SDMA.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+int amdgpu_svm_bo_evict(struct amdgpu_bo *bo)
+{
+	struct amdgpu_bo_svm *sbo = to_amdgpu_bo_svm(bo);
+	struct mm_struct *mm = sbo->devmem.mm;
+	int r;
+
+	/*
+	 * If the owning process is exiting, there is nothing to migrate; return
+	 * -EBUSY so the TTM eviction LRU walk skips this BO instead of aborting
+	 * the whole walk (which would fail the allocation that triggered the
+	 * eviction). Hold an mm reference across the call so the framework's own
+	 * dead-mm path stays unreachable and any -EFAULT it returns is
+	 * unambiguously a real error.
+	 */
+	if (!mmget_not_zero(mm))
+		return -EBUSY;
+
+	r = drm_pagemap_evict_to_ram(&sbo->devmem);
+
+	mmput_async(mm);
+
+	return r;
+}
+
+/**
+ * amdgpu_svm_bo_is_svm - Check if a TTM BO is an SVM BO
+ * @tbo: TTM buffer object
+ *
+ * Return: true if this is an amdgpu_bo_svm (identified by destroy callback)
+ */
+bool amdgpu_svm_bo_is_svm(struct ttm_buffer_object *tbo)
+{
+	return tbo->destroy == &amdgpu_bo_svm_destroy;
+}
+
 /* drm_pagemap_ops — top-level migration entry points */
 
 /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
index 56c66851f0a47..e9159e8167dd7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
@@ -30,6 +30,8 @@
 #include <linux/workqueue.h>
 
 struct amdgpu_device;
+struct amdgpu_bo;
+struct ttm_buffer_object;
 
 #define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
 #define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
@@ -75,12 +77,22 @@ struct amdgpu_pagemap {
 
 #if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
 int amdgpu_svm_migration_init(struct amdgpu_device *adev);
+int amdgpu_svm_bo_evict(struct amdgpu_bo *bo);
+bool amdgpu_svm_bo_is_svm(struct ttm_buffer_object *tbo);
 #else
 static inline
 int amdgpu_svm_migration_init(struct amdgpu_device *adev)
 {
 	return 0;
 }
+static inline int amdgpu_svm_bo_evict(struct amdgpu_bo *bo)
+{
+	return -ENODEV;
+}
+static inline bool amdgpu_svm_bo_is_svm(struct ttm_buffer_object *tbo)
+{
+	return false;
+}
 #endif
 
 /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index fc6d3fa62d794..2615946165bb1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -44,6 +44,7 @@
 #include "amdgpu_vm.h"
 #include "amdgpu_dma_buf.h"
 #include "kfd_svm.h"
+#include "amdgpu_migrate.h"
 
 /**
  * DOC: amdgpu_object
@@ -95,7 +96,8 @@ bool amdgpu_bo_is_amdgpu_bo(struct ttm_buffer_object *bo)
 {
 	if (bo->destroy == &amdgpu_bo_destroy ||
 	    bo->destroy == &amdgpu_bo_user_destroy ||
-	    bo->destroy == &svm_range_bo_destroy)
+	    bo->destroy == &svm_range_bo_destroy ||
+	    amdgpu_svm_bo_is_svm(bo))
 		return true;
 
 	return false;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 3d620ec2937f4..6f1b109b3fd83 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -60,6 +60,7 @@
 #include "amdgpu_hmm.h"
 #include "amdgpu_atomfirmware.h"
 #include "amdgpu_res_cursor.h"
+#include "amdgpu_migrate.h"
 #include "bif/bif_4_1_d.h"
 #include "kfd_svm.h"
 
@@ -123,6 +124,15 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,
 		return;
 	}
 
+	/* SVM BOs must evict directly to system memory for drm_pagemap
+	 * migration back to RAM in amdgpu_bo_move().
+	 */
+	if (amdgpu_svm_bo_is_svm(bo)) {
+		amdgpu_bo_placement_from_domain(abo, AMDGPU_GEM_DOMAIN_CPU);
+		*placement = abo->placement;
+		return;
+	}
+
 	switch (bo->resource->mem_type) {
 	case AMDGPU_PL_GDS:
 	case AMDGPU_PL_GWS:
@@ -610,6 +620,16 @@ static int amdgpu_bo_move(struct ttm_buffer_object *bo, bool evict,
 		abo->flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
 	}
 
+	/* SVM BO eviction: migrate device-private pages back to RAM */
+	if (amdgpu_svm_bo_is_svm(bo) &&
+	    old_mem->mem_type == TTM_PL_VRAM &&
+	    new_mem->mem_type == TTM_PL_SYSTEM) {
+		r = amdgpu_svm_bo_evict(abo);
+		if (!r)
+			ttm_bo_move_null(bo, new_mem);
+		return r;
+	}
+
 	if (adev->mman.buffer_funcs_enabled &&
 	    ((old_mem->mem_type == TTM_PL_SYSTEM &&
 	      new_mem->mem_type == TTM_PL_VRAM) ||
-- 
2.53.0

[PATCH v10 20/22] drm/amdgpu: hook up ZONE_DEVICE registration in device init and reset

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Call amdgpu_svm_migration_init() in the device initialization and
XGMI reset-restore paths to register the GPU's VRAM as a ZONE_DEVICE
region before KFD initialization.

This activates the drm_pagemap migration infrastructure.

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c  | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 44091ac12fa9c..ef3a5f844444a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -80,6 +80,7 @@
 #include "amdgpu_reset.h"
 #include "amdgpu_virt.h"
 #include "amdgpu_dev_coredump.h"
+#include "amdgpu_migrate.h"
 
 #include <linux/suspend.h>
 #include <drm/task_barrier.h>
@@ -4181,6 +4182,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 
 	/* Don't init kfd if whole hive need to be reset during init */
 	if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) {
+		amdgpu_svm_migration_init(adev);
 		kgd2kfd_init_zone_device(adev);
 		kfd_update_svm_support_properties(adev);
 	}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
index c8016fa55a92f..571dd062547ce 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
@@ -26,6 +26,7 @@
 #include "sienna_cichlid.h"
 #include "smu_v13_0_10.h"
 #include "soc_v1_0.h"
+#include "amdgpu_migrate.h"
 
 static int amdgpu_reset_xgmi_reset_on_init_suspend(struct amdgpu_device *adev)
 {
@@ -88,6 +89,7 @@ static int amdgpu_reset_xgmi_reset_on_init_restore_hwctxt(
 		return r;
 	list_for_each_entry(tmp_adev, reset_device_list, reset_list) {
 		if (!tmp_adev->kfd.init_complete) {
+			amdgpu_svm_migration_init(tmp_adev);
 			kgd2kfd_init_zone_device(tmp_adev);
 			amdgpu_amdkfd_device_init(tmp_adev);
 			amdgpu_amdkfd_drm_client_create(tmp_adev);
-- 
2.53.0

[PATCH v10 21/22] drm/amdgpu: add SVM range migration helpers for drm_pagemap

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

Add amdgpu_svm_range_migrate.c/.h implementing the per-range migration
helpers for SVM VRAM migration via drm_pagemap:

- Implement range_needs_migrate_to_vram() checking migrate_devmem
  capability and current backing location.
- Add amdgpu_svm_range_migrate_to_vram() wrapping
  drm_pagemap_populate_mm() for RAM-to-VRAM migration. On -EBUSY
  (partial migration due to races), evict conflicting pages back to
  system memory via drm_gpusvm_range_evict() and retry once.
- Add amdgpu_pagemap_capable() to check device memory support.

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/Makefile           |   2 +-
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 120 ++++++++++++++++++
 .../drm/amd/amdgpu/amdgpu_svm_range_migrate.h |  35 +++++
 3 files changed, 156 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h

diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index c78e9a0474425..9a233012beb44 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -328,7 +328,7 @@ amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
 
 # svm support
 amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
-	amdgpu_svm_fault.o amdgpu_svm_range.o amdgpu_migrate.o
+	amdgpu_svm_fault.o amdgpu_svm_range.o amdgpu_svm_range_migrate.o amdgpu_migrate.o
 
 include $(FULL_AMD_PATH)/pm/Makefile
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
new file mode 100644
index 0000000000000..0c44ad2920328
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
@@ -0,0 +1,120 @@
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu.h"
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu_migrate.h"
+#include "amdgpu_svm_range_migrate.h"
+
+static bool
+range_in_vram(struct drm_gpusvm_range *range)
+{
+	struct drm_gpusvm_pages_flags flags = {
+		/* Pairs with WRITE_ONCE in drm_gpusvm_get_pages() */
+		.__flags = READ_ONCE(range->pages.flags.__flags),
+	};
+
+	return flags.has_devmem_pages;
+}
+
+static bool
+range_needs_migrate_to_vram(struct drm_gpusvm_range *range)
+{
+	if (!range->pages.flags.migrate_devmem)
+		return false;
+	if (range_in_vram(range))
+		return false;
+
+	return true;
+}
+
+static struct drm_pagemap *
+amdgpu_svm_get_dpagemap(struct amdgpu_svm *svm)
+{
+	struct amdgpu_pagemap *apagemap = svm->adev->apagemap;
+
+	if (!apagemap || !apagemap->initialized)
+		return NULL;
+
+	return &apagemap->dpagemap;
+}
+
+bool
+amdgpu_pagemap_capable(struct amdgpu_svm *svm)
+{
+	if (svm->adev->gmc.is_app_apu)
+		return false;
+
+	if (!amdgpu_svm_get_dpagemap(svm))
+		return false;
+
+	return true;
+}
+
+/**
+ * amdgpu_svm_range_migrate_to_vram - Migrate range pages to VRAM
+ * @svm: Pointer to the AMDGPU SVM structure
+ * @range: The GPU SVM range to migrate
+ *
+ * Wraps drm_pagemap_populate_mm() to migrate system memory pages
+ * backing the given range into device VRAM. Skips migration if the
+ * range doesn't support devmem or is already in VRAM.
+ *
+ * Return: 0 on success or no-op, negative error code on failure
+ */
+int
+amdgpu_svm_range_migrate_to_vram(struct amdgpu_svm *svm,
+				 struct drm_gpusvm_range *range)
+{
+	struct drm_pagemap *dpagemap;
+	unsigned long start, end;
+	int ret, retries = 1;
+
+	if (!range_needs_migrate_to_vram(range))
+		return 0;
+
+	dpagemap = amdgpu_svm_get_dpagemap(svm);
+	if (!dpagemap)
+		return -ENODEV;
+
+	start = drm_gpusvm_range_start(range);
+	end = drm_gpusvm_range_end(range);
+
+	do {
+		ret = drm_pagemap_populate_mm(dpagemap, start, end,
+					      svm->gpusvm.mm, 0);
+
+		if (ret == -EBUSY && retries)
+			drm_gpusvm_range_evict(&svm->gpusvm, range);
+	} while (ret == -EBUSY && retries--);
+
+	if (ret) {
+		pr_debug("migrate_to_vram failed: ret=%d [0x%lx-0x%lx]\n",
+				 ret, start, end);
+		return ret;
+	}
+
+	return 0;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
new file mode 100644
index 0000000000000..6e4b6e45512e7
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_RANGE_MIGRATE_H__
+#define __AMDGPU_SVM_RANGE_MIGRATE_H__
+
+struct amdgpu_svm;
+struct drm_gpusvm_range;
+
+bool amdgpu_pagemap_capable(struct amdgpu_svm *svm);
+int amdgpu_svm_range_migrate_to_vram(struct amdgpu_svm *svm,
+				     struct drm_gpusvm_range *range);
+
+#endif /* __AMDGPU_SVM_RANGE_MIGRATE_H__ */
-- 
2.53.0

[PATCH v10 22/22] drm/amdgpu: integrate VRAM migration into SVM fault and prefetch paths

[Huang Rui]

From: Junhua Shen <Junhua.Shen@amd.com>

- Call amdgpu_svm_range_migrate_to_vram() before GPU mapping in
  fault and prefetch paths when VRAM migration is needed
- Set device_private_page_owner in drm_gpusvm_ctx for devmem awareness
- Allow AMDGPU_INTERCONNECT_VRAM entries in update_gpu_range proto check
- Skip valid-mapping fast path in prefetch when VRAM migration is
  requested, so prefetch can actively move pages to VRAM
- Enable amdgpu_pagemap_capable() check in amdgpu_svm_devmem_possible()
- Move AMDGPU_INTERCONNECT_VRAM/P2P defines to migration header

Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c       |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       |  3 ---
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 11 ++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 23 ++++++++++++++++---
 4 files changed, 33 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
index 5ae8a649644ec..31d0786012934 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -35,6 +35,7 @@
 #include "amdgpu_svm_attr.h"
 #include "amdgpu_svm_fault.h"
 #include "amdgpu_svm_range.h"
+#include "amdgpu_svm_range_migrate.h"
 #include "amdgpu_vm.h"
 
 #if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
@@ -320,7 +321,8 @@ bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm)
 	if (svm->adev->apu_prefer_gtt)
 		return false;
 
-	/* TODO: add amdgpu_pagemap_capable() */
+	if (amdgpu_pagemap_capable(svm))
+		return true;
 
 	return false;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
index cfdc4e6d045b7..e3ebd865d0732 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
@@ -45,9 +45,6 @@ struct amdgpu_svm_attrs;
 struct drm_device;
 struct drm_file;
 
-#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
-#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
-
 enum amdgpu_svm_xnack_mode {
 	AMDGPU_SVM_XNACK_OFF,
 	AMDGPU_SVM_XNACK_ON,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
index 2dd77fe78303c..c135a3f74d313 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
@@ -26,6 +26,8 @@
 #include "amdgpu_svm_attr.h"
 #include "amdgpu_svm_fault.h"
 #include "amdgpu_svm_range.h"
+#include "amdgpu_svm_range_migrate.h"
+#include "amdgpu_migrate.h"
 #include "amdgpu.h"
 #include "amdgpu_vm.h"
 #include "amdgpu_gmc.h"
@@ -193,6 +195,8 @@ static int fault_map_range(struct amdgpu_svm *svm,
 		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
 		.devmem_only = need_vram_migration,
 		.timeslice_ms = need_vram_migration ? 5 : 0,
+		.device_private_page_owner = devmem_possible ?
+			AMDGPU_PGMAP_OWNER(svm->adev) : NULL,
 	};
 	struct amdgpu_svm_range *range;
 	ktime_t timestamp = ktime_get_boottime();
@@ -265,7 +269,12 @@ static int fault_map_range(struct amdgpu_svm *svm,
 	}
 
 	AMDGPU_SVM_RANGE_TRACE(range, "PAGE FAULT");
-	/* TODO: add migration*/
+	if (need_vram_migration) {
+		AMDGPU_SVM_RANGE_TRACE(range, "MIGRATE PAGES");
+		ret = amdgpu_svm_range_migrate_to_vram(svm, &range->base);
+		if (ret)
+			return ret;
+	}
 
 	AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
 	ret = amdgpu_svm_range_get_pages(svm, &range->base, &map_ctx);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index 787afe0767e5a..2bcd9cbeb3037 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -26,6 +26,8 @@
 #include "amdgpu_svm_attr.h"
 #include "amdgpu_svm_range.h"
 #include "amdgpu_svm_fault.h"
+#include "amdgpu_svm_range_migrate.h"
+#include "amdgpu_migrate.h"
 #include "amdgpu.h"
 #include "amdgpu_vm.h"
 #include "amdgpu_trace.h"
@@ -196,7 +198,8 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
 		unsigned long start_page, last_page;
 		bool is_last_seg;
 
-		if (entry->proto != DRM_INTERCONNECT_SYSTEM)
+		if (entry->proto != DRM_INTERCONNECT_SYSTEM &&
+		    entry->proto != AMDGPU_INTERCONNECT_VRAM)
 			return -EOPNOTSUPP;
 
 		pte_flags = amdgpu_svm_range_attr_pte_flags(svm, attrs,
@@ -351,6 +354,8 @@ amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 		.devmem_possible = devmem_possible,
 		.devmem_only = need_vram_migration,
 		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
+		.device_private_page_owner = devmem_possible ?
+			AMDGPU_PGMAP_OWNER(svm->adev) : NULL,
 	};
 
 	while (addr < end) {
@@ -372,12 +377,24 @@ amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
 		if (next_addr <= addr)
 			return -EINVAL;
 
-		if (amdgpu_svm_range_is_valid(range, attrs)) {
+		/*
+		 * Prefetch to device memory should still run migration logic even if
+		 * the current GPU mapping is already valid. This allows prefetch
+		 * requests to actively move backing pages to VRAM instead of being
+		 * skipped by the valid-mapping fast path.
+		 */
+		if (amdgpu_svm_range_is_valid(range, attrs) &&
+		    !need_vram_migration) {
 			addr = next_addr;
 			continue;
 		}
 
-		/* TODO: add migration */
+		if (need_vram_migration) {
+			AMDGPU_SVM_RANGE_TRACE(range, "MIGRATE PAGES");
+			ret = amdgpu_svm_range_migrate_to_vram(svm, &range->base);
+			if (ret)
+				return ret;
+		}
 
 		AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
 
-- 
2.53.0

Re: [PATCH v10 01/22] drm/amdgpu: add SVM ioctl UAPI definitions

[Christian König]

On 10/21/26 04:58, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Define the following ioctl structures and enums:
> - DRM_AMDGPU_GEM_SVM ioctl command and DRM_IOCTL_AMDGPU_GEM_SVM macro
> - enum amdgpu_ioctl_svm_op: SET_ATTR, GET_ATTR, RESET_ATTR operations
> - enum amdgpu_ioctl_svm_access: INACCESSIBLE, IN_PLACE, ALLOW_MIGRATE
> - enum amdgpu_ioctl_svm_location: SYSMEM, UNDEFINED, LOCAL, HIVE
> - enum amdgpu_ioctl_svm_attr_type: PREFERRED_LOC, PREFETCH_LOC, ACCESS,
>   GRANULARITY, HOST_ACCESS, COHERENT, EXT_COHERENT, GPU_RO,
>   GPU_EXEC, GPU_READ_MOSTLY
> - struct drm_amdgpu_svm_attribute: type and value pair
> - struct drm_amdgpu_gem_svm: ioctl payload with start_addr, size,
>   operation, nattr, and attrs_ptr
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Huang Rui <ray.huang@amd.com>
> Signed-off-by: Honglei Huang <honghuan@amd.com>

Reviewed-by: Christian König <christian.koenig@amd.com>

> ---
>  include/uapi/drm/amdgpu_drm.h | 127 ++++++++++++++++++++++++++++++++++
>  1 file changed, 127 insertions(+)
> 
> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
> index cd1b8034a2fb4..cfe463f2114a1 100644
> --- a/include/uapi/drm/amdgpu_drm.h
> +++ b/include/uapi/drm/amdgpu_drm.h
> @@ -60,6 +60,7 @@ extern "C" {
>  #define DRM_AMDGPU_GEM_LIST_HANDLES	0x19
>  #define DRM_AMDGPU_PROC_OPTIONS		0x1A
>  #define DRM_AMDGPU_UALINK_HANDLE	0x1B
> +#define DRM_AMDGPU_GEM_SVM		0x1C
>  
>  #define DRM_IOCTL_AMDGPU_GEM_CREATE	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_CREATE, union drm_amdgpu_gem_create)
>  #define DRM_IOCTL_AMDGPU_GEM_MMAP	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_MMAP, union drm_amdgpu_gem_mmap)
> @@ -83,6 +84,9 @@ extern "C" {
>  #define DRM_IOCTL_AMDGPU_GEM_LIST_HANDLES DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_LIST_HANDLES, struct drm_amdgpu_gem_list_handles)
>  #define DRM_IOCTL_AMDGPU_PROC_OPTIONS	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_PROC_OPTIONS, struct drm_amdgpu_proc_options)
>  #define DRM_IOCTL_AMDGPU_UALINK_HANDLE DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_UALINK_HANDLE, union drm_amdgpu_ualink_handle)
> +#define DRM_IOCTL_AMDGPU_GEM_SVM	\
> +	DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_GEM_SVM, \
> +		  struct drm_amdgpu_gem_svm)
>  
>  /**
>   * DOC: memory domains
> @@ -1735,6 +1739,129 @@ union drm_amdgpu_ualink_handle {
>  	union drm_amdgpu_ualink_handle_out out;
>  };
>  
> +/**
> + * enum amdgpu_ioctl_svm_op - operation selector for DRM_IOCTL_AMDGPU_GEM_SVM.
> + * @AMDGPU_SVM_OP_SET_ATTR: apply the attributes in @attrs_ptr to the VA range
> + *                          [@start_addr, @start_addr + @size).
> + * @AMDGPU_SVM_OP_GET_ATTR: query the attributes at @start_addr. The caller
> + *                          lists the attribute types it is interested in via
> + *                          @attrs_ptr; the kernel fills in their current
> + *                          values and updates @size to the length of the
> + *                          contiguous range starting at @start_addr over
> + *                          which those values are constant. Input @size is an
> + *                          upper bound. This lets user space (e.g. CRIU) walk
> + *                          the address space and read back all attributes
> + *                          without knowing the range boundaries in advance.
> + * @AMDGPU_SVM_OP_RESET_ATTR: reset all attributes for the VA range to their
> + *                            default values. @attrs_ptr and @nattr are ignored.
> + */
> +enum amdgpu_ioctl_svm_op {
> +	AMDGPU_SVM_OP_SET_ATTR = 0,
> +	AMDGPU_SVM_OP_GET_ATTR = 1,
> +	AMDGPU_SVM_OP_RESET_ATTR = 2,
> +};
> +
> +/**
> + * enum amdgpu_ioctl_svm_access - values for AMDGPU_SVM_ATTR_ACCESS.
> + * @AMDGPU_SVM_ACCESS_INACCESSIBLE: GPU must not access the range; any access
> + *                                  is a fault.
> + * @AMDGPU_SVM_ACCESS_IN_PLACE:     GPU may access the range only at its
> + *                                  current backing store; the driver will
> + *                                  never migrate pages to local VRAM.
> + * @AMDGPU_SVM_ACCESS_ALLOW_MIGRATE: GPU may access the range and the driver
> + *                                  is allowed (but not required) to migrate
> + *                                  pages between system memory and local
> + *                                  VRAM to satisfy the preferred/prefetch
> + *                                  location.
> + */
> +enum amdgpu_ioctl_svm_access {
> +	AMDGPU_SVM_ACCESS_INACCESSIBLE		= 0,
> +	AMDGPU_SVM_ACCESS_IN_PLACE			= 1,
> +	AMDGPU_SVM_ACCESS_ALLOW_MIGRATE		= 2,
> +};
> +
> +/**
> + * enum amdgpu_ioctl_svm_location - values for AMDGPU_SVM_ATTR_PREFERRED_LOC /
> + *                            AMDGPU_SVM_ATTR_PREFETCH_LOC.
> + * @AMDGPU_SVM_LOCATION_SYSMEM:    back the range with system memory.
> + * @AMDGPU_SVM_LOCATION_UNDEFINED: no preference; the driver chooses.
> + * @AMDGPU_SVM_LOCATION_LOCAL:     back the range with the local VRAM of the
> + *                                 device the SVM context belongs to.
> + * @AMDGPU_SVM_LOCATION_HIVE:      back the range with the VRAM of any device
> + *                                 in the same XGMI hive as the local device.
> + *
> + * The location is expressed as a fixed enum rather than a device file
> + * descriptor on purpose: a descriptor has no stable meaning across a
> + * checkpoint/restore (CRIU) cycle.
> + */
> +enum amdgpu_ioctl_svm_location {
> +	AMDGPU_SVM_LOCATION_SYSMEM	= 0,
> +	AMDGPU_SVM_LOCATION_UNDEFINED	= 1,
> +	AMDGPU_SVM_LOCATION_LOCAL	= 2,
> +	AMDGPU_SVM_LOCATION_HIVE	= 3,
> +};
> +
> +/**
> + * enum amdgpu_ioctl_svm_attr_type - attribute selector for
> + *                                   &drm_amdgpu_svm_attribute.type.
> + *
> + * @AMDGPU_SVM_ATTR_PREFERRED_LOC: Preferred backing location for the range.
> + *	Value is one of &enum amdgpu_ioctl_svm_location.
> + * @AMDGPU_SVM_ATTR_PREFETCH_LOC: Prefetch target for the range. Value is
> + *	one of &enum amdgpu_ioctl_svm_location.
> + * @AMDGPU_SVM_ATTR_ACCESS: GPU access policy for the range. Value is one
> + *	of &enum amdgpu_ioctl_svm_access.
> + * @AMDGPU_SVM_ATTR_GRANULARITY: log2 of the migration granularity in pages.
> + * @AMDGPU_SVM_ATTR_HOST_ACCESS: Guarantee host access to memory.
> + * @AMDGPU_SVM_ATTR_COHERENT: Fine-grained coherency between all devices
> + *	with access.
> + * @AMDGPU_SVM_ATTR_EXT_COHERENT: Fine-grained coherency between all devices
> + *	using device-scope atomics.
> + * @AMDGPU_SVM_ATTR_GPU_RO: GPUs only read the range, allowing replication.
> + * @AMDGPU_SVM_ATTR_GPU_EXEC: Allow execution on GPU.
> + * @AMDGPU_SVM_ATTR_GPU_READ_MOSTLY: GPUs mostly read the range; may allow
> + *	optimizations similar to GPU_RO, but writes still fault.
> + */
> +enum amdgpu_ioctl_svm_attr_type {
> +	AMDGPU_SVM_ATTR_PREFERRED_LOC		= 0,
> +	AMDGPU_SVM_ATTR_PREFETCH_LOC		= 1,
> +	AMDGPU_SVM_ATTR_ACCESS				= 2,
> +	AMDGPU_SVM_ATTR_GRANULARITY			= 3,
> +	/* Boolean attributes below: value must be 0 or 1. */
> +	AMDGPU_SVM_ATTR_HOST_ACCESS			= 4,
> +	AMDGPU_SVM_ATTR_COHERENT			= 5,
> +	AMDGPU_SVM_ATTR_EXT_COHERENT		= 6,
> +	AMDGPU_SVM_ATTR_GPU_RO				= 7,
> +	AMDGPU_SVM_ATTR_GPU_EXEC			= 8,
> +	AMDGPU_SVM_ATTR_GPU_READ_MOSTLY		= 9,
> +};
> +
> +/* One (type, value) pair carried by DRM_IOCTL_AMDGPU_GEM_SVM. */
> +struct drm_amdgpu_svm_attribute {
> +	/** AMDGPU_SVM_ATTR_* */
> +	__u32 type;
> +	/** Attribute value; interpretation depends on @type */
> +	__u32 value;
> +};
> +
> +/* Argument for DRM_IOCTL_AMDGPU_GEM_SVM. */
> +struct drm_amdgpu_gem_svm {
> +	/** Start of the virtual address range */
> +	__u64 start_addr;
> +	/**
> +	 * Size of the range in bytes. For AMDGPU_SVM_OP_GET_ATTR this is an
> +	 * input upper bound and is updated on return to the length of the
> +	 * contiguous range starting at @start_addr with constant attributes.
> +	 */
> +	__u64 size;
> +	/** AMDGPU_SVM_OP_* */
> +	__u32 operation;
> +	/** Number of struct drm_amdgpu_svm_attribute entries in @attrs_ptr */
> +	__u32 nattr;
> +	/** User pointer to an array of @nattr struct drm_amdgpu_svm_attribute */
> +	__u64 attrs_ptr;
> +};
> +
>  #if defined(__cplusplus)
>  }
>  #endif

Re: [PATCH v10 02/22] drm/amdgpu: add SVM core header and VM integration

[Christian König]

On 10/21/26 04:58, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Add amdgpu_svm.h with SVM data structures and public API:
> - enum amdgpu_svm_xnack_mode: OFF, ON and DEFAULT mode
> - struct amdgpu_svm: core context with drm_gpusvm, kref lifecycle,
>   attribute tree, rw_semaphore, GC workqueue, xnack state.
> - struct amdgpu_svm_gc: garbage collector with workqueue and work_struct
> - Kmem cache helpers: AMDGPU_SVM_KMEM_CACHE_CREATE/DESTROY
> - Locking helpers: amdgpu_svm_lock/unlock/assert_locked
> - Public API declarations (including clean_queue, sync_work)
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Huang Rui <ray.huang@amd.com>
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 195 ++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |   4 +
>  2 files changed, 199 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> new file mode 100644
> index 0000000000000..72f954d2df39e
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> @@ -0,0 +1,195 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright 2026 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + */
> +
> +#ifndef __AMDGPU_SVM_H__
> +#define __AMDGPU_SVM_H__
> +
> +#include <drm/amdgpu_drm.h>
> +#include <drm/drm_gpusvm.h>
> +#include <linux/atomic.h>
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/printk.h>
> +#include <linux/rwsem.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/types.h>
> +#include <linux/workqueue.h>
> +
> +struct amdgpu_device;
> +struct amdgpu_vm;
> +struct amdgpu_svm_attr_tree;
> +struct amdgpu_svm_attrs;
> +struct drm_device;
> +struct drm_file;
> +
> +enum amdgpu_svm_xnack_mode {
> +	AMDGPU_SVM_XNACK_OFF,
> +	AMDGPU_SVM_XNACK_ON,
> +	AMDGPU_SVM_XNACK_DEFAULT,
> +};
> +
> +#define amdgpu_svm_assert_in_notifier(svm__) \
> +	lockdep_assert_held_write(&(svm__)->gpusvm.notifier_lock)
> +
> +/**
> + * struct amdgpu_svm_gc - SVM range garbage collector used for unmapping
> + * SVM ranges in response to mmu_notifier events.
> + * @wq: Workqueue where @work runs on.
> + * @list: Ranges list in garbage collector. Protected by amdgpu_svm.work_lock.
> + * @work: Drains @list under amdgpu_svm.svm_lock.
> + */
> +struct amdgpu_svm_gc {
> +	struct workqueue_struct *wq;
> +	struct list_head list;
> +	struct work_struct work;
> +};
> +
> +/**
> + * struct amdgpu_svm - shared virtual memory context
> + * @gpusvm: Embedded drm_gpusvm base, manages the range/notifier trees.
> + * @refcount: Reference count guarding the lifetime of this instance.
> + * @adev: The amdgpu device this SVM context belongs to.
> + * @vm: Back pointer to the owning amdgpu_vm.
> + * @attr_tree: Interval tree of SVM attributes.
> + * @svm_lock: The driver_svm_lock described in the GPU SVM locking section.
> + *            registered by drm_gpusvm_driver_set_lock().
> + * @work_lock: Protects the deferred work queues such as @gc.list.
> + * @gc: garbage collector for ranges queued for destruction.
> + * @exiting: Set once when the context starts tearing down.
> + * @checkpoint_ts: IH ring timestamp captured on unmap, used to drop stale
> + *                 faults.
> + * @default_granularity: Default range granularity.
> + * @xnack_enabled: Whether retry (XNACK) faults are enabled.
> + * @invalidate_ranges: Callback of the MMU notifier invalidate event.
> + */
> +struct amdgpu_svm {
> +	struct drm_gpusvm gpusvm;
> +	struct kref refcount;
> +	struct amdgpu_device *adev;
> +	struct amdgpu_vm *vm;
> +	struct amdgpu_svm_attr_tree *attr_tree;
> +	struct rw_semaphore svm_lock;
> +	spinlock_t work_lock;
> +	struct amdgpu_svm_gc gc;
> +	atomic_t exiting;

> +	uint64_t checkpoint_ts;

By now we have figured out that TS in faults are not useable for fault handling.

Background is that they reset from time to time because of GFXOFF and may vary depending on the engine who sends the fault.

Adding Timur from Valve on CC, he send out a patch to completely remove the existing timestamp handling from the fault filter.

We need to come up with a different approach here.

> +	u8 default_granularity;

> +	bool xnack_enabled;

It's probably ok to have that here for now, but that will become a VM property in the future I think.

> +	void (*invalidate_ranges)(struct amdgpu_svm *svm,
> +				  struct drm_gpusvm_notifier *notifier,
> +				  const struct mmu_notifier_range *mmu_range,
> +				  struct drm_gpusvm_range *first,
> +				  uint64_t adj_start, uint64_t adj_end);

That looks a bit questionable. Why is that a pointer inside the amdgpu_svm structure?

Apart from those questions that looks rather reasonable.

Regards,
Christian.

> +};
> +
> +static inline struct amdgpu_svm *to_amdgpu_svm(struct drm_gpusvm *gpusvm)
> +{
> +	return container_of(gpusvm, struct amdgpu_svm, gpusvm);
> +}
> +
> +/*
> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
> + * Hold it in write mode around structural GPU SVM updates, including
> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
> + */
> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
> +{
> +	down_write(&svm->svm_lock);
> +}
> +
> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
> +{
> +	up_write(&svm->svm_lock);
> +}
> +
> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
> +{
> +	lockdep_assert_held_write(&svm->svm_lock);
> +}
> +
> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
> +
> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
> +void amdgpu_svm_close(struct amdgpu_vm *vm);
> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
> +
> +void amdgpu_svm_put(struct amdgpu_svm *svm);
> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
> +					       uint32_t pasid);
> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
> +			    uint64_t fault_page, uint64_t ts,
> +			    bool write_fault);
> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
> +
> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
> +			 struct drm_file *filp);
> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
> +			    struct list_head *work_list);
> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
> +				 const struct amdgpu_svm_attrs *old_attrs,
> +				 const struct amdgpu_svm_attrs *new_attrs,
> +				 unsigned long start_page,
> +				 unsigned long last_page);
> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
> +#else
> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
> +				  struct amdgpu_vm *vm)
> +{
> +	return 0;
> +}
> +
> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
> +{
> +}
> +
> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
> +{
> +}
> +
> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
> +					  uint32_t pasid,
> +					  uint64_t fault_page,
> +					  uint64_t ts,
> +					  bool write_fault)
> +{
> +	return -EOPNOTSUPP;
> +}
> +
> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
> +{
> +	return false;
> +}
> +
> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
> +				       struct drm_file *filp)
> +{
> +	return -EOPNOTSUPP;
> +}
> +#endif /* CONFIG_DRM_AMDGPU_SVM */
> +
> +#endif /* __AMDGPU_SVM_H__ */
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> index 9b07df5a2b816..b435e393be154 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>  struct amdgpu_job;
>  struct amdgpu_bo_list_entry;
>  struct amdgpu_bo_vm;
> +struct amdgpu_svm;
>  
>  /*
>   * GPUVM handling
> @@ -381,6 +382,9 @@ struct amdgpu_vm {
>  
>  	/* cached fault info */
>  	struct amdgpu_vm_fault_info fault_info;
> +
> +	/* SVM experimental implementation */
> +	struct amdgpu_svm *svm;
>  };
>  
>  struct amdgpu_vm_manager {

Re: [PATCH v10 03/22] drm/amdgpu: implement SVM attribute tree and helper functions

[Christian König]

On 10/21/26 04:58, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Add amdgpu_svm_attr.h with the SVM attribute types and tree
> infrastructure together with the helper implementation in
> amdgpu_svm_attr.c that uses them. The header is squashed into the
> implementation patch so it does not stand alone as declarations
> without an implementation.
> 
> amdgpu_svm_attr.h:
> - Internal flag bitmask definitions mapped from UAPI attr types
> - PTE_FLAG_MASK and MAPPING_FLAG_MASK for change detection
> - struct amdgpu_svm_attrs: user set attribute range
> - struct amdgpu_svm_attr_range: interval tree node with attrs
> - struct amdgpu_svm_attr_tree: mutex protected RB tree for store and
>   search
> - enum amdgpu_svm_attr_change_trigger: change flags of user attribute
>   changes
> 
> amdgpu_svm_attr.c:
> - Default attribute initialization: amdgpu_svm_attr_set_default
> - Device memory and VRAM preference helpers
> - VMA validity checker: amdgpu_svm_check_vma
> - Attribute equality comparison: attr_equal
> - Interval tree CRUD operations: find, get_bounds,
>   alloc, insert, and remove
> - attr_set_interval helper for range boundary updates
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Huang Rui <ray.huang@amd.com>
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 235 +++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h | 182 ++++++++++++++
>  2 files changed, 417 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
>  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
> new file mode 100644
> index 0000000000000..a5afa75421fbd
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
> @@ -0,0 +1,235 @@
> +// SPDX-License-Identifier: GPL-2.0 OR MIT
> +/*
> + * Copyright 2026 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + */
> +
> +#include "amdgpu_svm.h"
> +#include "amdgpu_svm_attr.h"
> +
> +#include <linux/err.h>
> +#include <linux/errno.h>
> +#include <linux/gfp.h>
> +#include <linux/lockdep.h>
> +#include <linux/minmax.h>
> +#include <linux/mm.h>
> +#include <linux/slab.h>
> +
> +struct attr_set_ctx {
> +	struct amdgpu_svm_attrs old_attrs;
> +	struct amdgpu_svm_attrs new_attrs;
> +	unsigned long start_page;
> +	unsigned long last_page;
> +};
> +
> +struct attr_get_ctx {
> +	int32_t preferred_loc;
> +	int32_t prefetch_loc;
> +	enum amdgpu_ioctl_svm_access access;
> +	uint32_t granularity;
> +	uint32_t flags_and;
> +	bool has_range;
> +};
> +


> +bool amdgpu_svm_attr_prefer_vram(const struct amdgpu_svm_attrs *attrs)
> +{
> +	if (attrs->preferred_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
> +	    attrs->preferred_loc != AMDGPU_SVM_LOCATION_SYSMEM)
> +		return true;
> +
> +	if (attrs->prefetch_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
> +	    attrs->prefetch_loc != AMDGPU_SVM_LOCATION_SYSMEM)
> +		return true;
> +
> +	return false;
> +}
> +
> +struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
> +					unsigned long addr)
> +{
> +	const unsigned long unsupported_vm_flags = VM_IO | VM_PFNMAP |
> +						   VM_MIXEDMAP;
> +	struct vm_area_struct *vma = vma_lookup(mm, addr);
> +
> +	if (!vma)
> +		return ERR_PTR(-EFAULT);
> +
> +	if (vma->vm_flags & unsupported_vm_flags)
> +		return ERR_PTR(-EOPNOTSUPP);
> +
> +	return vma;
> +}
> +
> +static void attr_set_interval(struct amdgpu_svm_attr_range *range,
> +				unsigned long start_page,
> +				unsigned long last_page)
> +{
> +	range->it_node.start = start_page;
> +	range->it_node.last = last_page;
> +}
> +
> +void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
> +				 struct amdgpu_svm_attrs *attrs)
> +{
> +	attrs->preferred_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
> +	attrs->prefetch_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
> +	attrs->granularity = svm->default_granularity;
> +	attrs->flags = AMDGPU_SVM_ATTR_BIT_HOST_ACCESS | AMDGPU_SVM_ATTR_BIT_COHERENT;
> +	attrs->access = svm->xnack_enabled ?
> +		AMDGPU_SVM_ACCESS_ALLOW_MIGRATE : AMDGPU_SVM_ACCESS_INACCESSIBLE;
> +}
> +
> +struct amdgpu_svm_attr_range *
> +amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
> +			   unsigned long page)
> +{
> +	struct interval_tree_node *node;
> +
> +	node = interval_tree_iter_first(&attr_tree->tree, page, page);
> +	if (node)
> +		return container_of(node, struct amdgpu_svm_attr_range, it_node);
> +
> +	return NULL;
> +}

I think this whole block of functions can be static, but not 100% sure.

If they are used outside of this C file please add kerneldoc for them.

Apart from those generic comments I don't feel like I knew enough about drm_gpusvm and the ROCm requirements to judge if that is really correct or not.

Writing that I added Aveline on CC, it would be really cool if you could take a look as well.

Feel free to add Acked-by: Christian König <christian.koenig@amd.com>.

Regards,
Christian.

> +
> +/**
> + * amdgpu_svm_attr_get_bounds_locked() - Find attributes  or surrounding bounds
> + * @attr_tree: attribute tree to search.
> + * @page: page index to look up.
> + * @start_page: out, first page of the returned range or gap.
> + * @last_page: out, last page of the returned range or gap.
> + *
> + * If @page is covered by an attribute range, return that range and report its
> + * bounds. Otherwise return NULL and fill @start_page/@last_page with the bounds
> + * of the default attribute gap around @page, clamped by neighboring explicit
> + * ranges or [0, ULONG_MAX] when none.
> + *
> + * Return: the covering range, or NULL if @page falls in a gap.
> + */
> +struct amdgpu_svm_attr_range *
> +amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
> +				  unsigned long page,
> +				  unsigned long *start_page,
> +				  unsigned long *last_page)
> +{
> +	struct amdgpu_svm_attr_range *attr_range;
> +	struct interval_tree_node *node;
> +	struct rb_node *rb;
> +
> +	attr_range = amdgpu_svm_attr_find_locked(attr_tree, page);
> +	if (attr_range) {
> +		*start_page = amdgpu_svm_attr_start_page(attr_range);
> +		*last_page = amdgpu_svm_attr_last_page(attr_range);
> +		return attr_range;
> +	}
> +
> +	*start_page = 0;
> +	*last_page = ULONG_MAX;
> +
> +	if (page == ULONG_MAX)
> +		return NULL;
> +
> +	node = interval_tree_iter_first(&attr_tree->tree, page + 1, ULONG_MAX);
> +	if (node) {
> +		if (node->start > page)
> +			*last_page = node->start - 1;
> +
> +		rb = rb_prev(&node->rb);
> +		if (rb) {
> +			node = container_of(rb, struct interval_tree_node, rb);
> +			if (node->last < page)
> +				*start_page = node->last + 1;
> +		}
> +	} else {
> +		rb = rb_last(&attr_tree->tree.rb_root);
> +
> +		if (rb) {
> +			node = container_of(rb, struct interval_tree_node, rb);
> +			if (node->last < page)
> +				*start_page = node->last + 1;
> +		}
> +	}
> +
> +	return NULL;
> +}
> +
> +static bool attr_equal(const struct amdgpu_svm_attrs *a,
> +				 const struct amdgpu_svm_attrs *b)
> +{
> +	return a->flags == b->flags &&
> +	       a->preferred_loc == b->preferred_loc &&
> +	       a->prefetch_loc == b->prefetch_loc &&
> +		       a->granularity == b->granularity &&
> +		       a->access == b->access;
> +}
> +
> +struct amdgpu_svm_attr_range *
> +amdgpu_svm_attr_range_alloc(unsigned long start_page,
> +			   unsigned long last_page,
> +			   const struct amdgpu_svm_attrs *attrs)
> +{
> +	struct amdgpu_svm_attr_range *range;
> +
> +	range = kzalloc_obj(*range, GFP_KERNEL);
> +	if (!range)
> +		return NULL;
> +
> +	INIT_LIST_HEAD(&range->list);
> +	attr_set_interval(range, start_page, last_page);
> +	range->attrs = *attrs;
> +	return range;
> +}
> +
> +void amdgpu_svm_attr_range_insert_locked(struct amdgpu_svm_attr_tree *attr_tree,
> +					 struct amdgpu_svm_attr_range *range)
> +{
> +	struct interval_tree_node *node;
> +	struct amdgpu_svm_attr_range *next;
> +
> +	lockdep_assert_held(&attr_tree->lock);
> +
> +	/*
> +	 * Keep @range_list ordered by start page: insert before the first range
> +	 * starting at or after @range, then add to the interval tree.
> +	 */
> +	node = interval_tree_iter_first(&attr_tree->tree, amdgpu_svm_attr_start_page(range),
> +					ULONG_MAX);
> +	if (node) {
> +		next = container_of(node, struct amdgpu_svm_attr_range, it_node);
> +		list_add_tail(&range->list, &next->list);
> +	} else {
> +		list_add_tail(&range->list, &attr_tree->range_list);
> +	}
> +
> +	interval_tree_insert(&range->it_node, &attr_tree->tree);
> +}
> +
> +static void attr_remove_range_locked(struct amdgpu_svm_attr_tree *attr_tree,
> +					  struct amdgpu_svm_attr_range *range,
> +					  bool free_range)
> +{
> +	lockdep_assert_held(&attr_tree->lock);
> +
> +	interval_tree_remove(&range->it_node, &attr_tree->tree);
> +	list_del_init(&range->list);
> +	if (free_range)
> +		kfree(range);
> +}
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
> new file mode 100644
> index 0000000000000..36fee11aa9ea0
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
> @@ -0,0 +1,182 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright 2026 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + */
> +
> +#ifndef __AMDGPU_SVM_ATTR_H__
> +#define __AMDGPU_SVM_ATTR_H__
> +
> +#include <drm/amdgpu_drm.h>
> +#include <linux/interval_tree.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/rbtree.h>
> +#include <linux/types.h>
> +
> +/* Internal SVM attribute bitmask flags mapped from UAPI ioctl definitions */
> +#define AMDGPU_SVM_ATTR_BIT_HOST_ACCESS		(1u << 0)
> +#define AMDGPU_SVM_ATTR_BIT_COHERENT		(1u << 1)
> +#define AMDGPU_SVM_ATTR_BIT_EXT_COHERENT	(1u << 2)
> +#define AMDGPU_SVM_ATTR_BIT_GPU_RO		(1u << 3)
> +#define AMDGPU_SVM_ATTR_BIT_GPU_EXEC		(1u << 4)
> +#define AMDGPU_SVM_ATTR_BIT_GPU_READ_MOSTLY	(1u << 5)
> +
> +#define AMDGPU_SVM_PTE_FLAG_MASK \
> +	(AMDGPU_SVM_ATTR_BIT_COHERENT | AMDGPU_SVM_ATTR_BIT_EXT_COHERENT | \
> +	 AMDGPU_SVM_ATTR_BIT_GPU_RO | AMDGPU_SVM_ATTR_BIT_GPU_EXEC)
> +
> +#define AMDGPU_SVM_MAPPING_FLAG_MASK \
> +	(AMDGPU_SVM_ATTR_BIT_HOST_ACCESS | \
> +	 AMDGPU_SVM_ATTR_BIT_GPU_READ_MOSTLY)
> +
> +/**
> + * struct amdgpu_svm_attrs - SVM attributes for an address range
> + * @preferred_loc: Preferred backing location, using AMDGPU_SVM_LOCATION_*.
> + * @prefetch_loc: Target location for prefetch requests, using
> + *                AMDGPU_SVM_LOCATION_*.
> + * @flags: Internal AMDGPU_SVM_ATTR_BIT_* flags mapped from the UAPI.
> + * @granularity: Mapping granularity encoded as a page order.
> + * @access: CPU/GPU access policy from the SVM UAPI.
> + */
> +struct amdgpu_svm_attrs {
> +	int32_t preferred_loc;
> +	int32_t prefetch_loc;
> +	uint32_t flags;
> +	uint32_t granularity;
> +	enum amdgpu_ioctl_svm_access access;
> +};
> +
> +/**
> + * struct amdgpu_svm_attr_range - a range of user attributes
> + * @it_node: interval tree node keyed by [start, last] page index.
> + * @list: links the range into amdgpu_svm_attr_tree.range_list in address order.
> + * @attrs: the attributes applied to this range.
> + */
> +struct amdgpu_svm_attr_range {
> +	struct interval_tree_node it_node;
> +	struct list_head list;
> +	struct amdgpu_svm_attrs attrs;
> +};
> +
> +static inline unsigned long
> +amdgpu_svm_attr_start_page(const struct amdgpu_svm_attr_range *range)
> +{
> +	return range->it_node.start;
> +}
> +
> +static inline unsigned long
> +amdgpu_svm_attr_last_page(const struct amdgpu_svm_attr_range *range)
> +{
> +	return range->it_node.last;
> +}
> +
> +static inline unsigned long
> +amdgpu_svm_attr_start(const struct amdgpu_svm_attr_range *range)
> +{
> +	return range->it_node.start << PAGE_SHIFT;
> +}
> +
> +static inline unsigned long
> +amdgpu_svm_attr_end(const struct amdgpu_svm_attr_range *range)
> +{
> +	return (range->it_node.last + 1) << PAGE_SHIFT;
> +}
> +
> +struct amdgpu_svm;
> +struct mm_struct;
> +struct vm_area_struct;
> +
> +static inline bool
> +amdgpu_svm_attr_has_access(enum amdgpu_ioctl_svm_access access)
> +{
> +	return access == AMDGPU_SVM_ACCESS_ALLOW_MIGRATE ||
> +	       access == AMDGPU_SVM_ACCESS_IN_PLACE;
> +}
> +
> +/**
> + * struct amdgpu_svm_attr_tree - per-SVM store of user attribute ranges
> + * @lock: protects @tree and @range_list.
> + * @tree: interval tree of struct amdgpu_svm_attr_range for fast lookup.
> + * @range_list: address-ordered list of the same ranges, kept in sync with
> + *              @tree to allow ordered traversal during modification.
> + * @svm: back pointer to the owning SVM instance.
> + */
> +struct amdgpu_svm_attr_tree {
> +	struct mutex lock;
> +	struct rb_root_cached tree;
> +	struct list_head range_list;
> +	struct amdgpu_svm *svm;
> +};
> +
> +/**
> + * enum amdgpu_svm_attr_change_trigger - effects caused by an attribute change
> + * @AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE: Access policy changed.
> + * @AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE: GPU PTE permission/cache bits changed.
> + * @AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE: Mapping policy bits changed.
> + * @AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE: Preferred or prefetch location changed.
> + * @AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE: Range granularity changed.
> + * @AMDGPU_SVM_ATTR_TRIGGER_PREFETCH: New attributes request a VRAM prefetch.
> + *
> + * Bitmask describing what an attribute update touched. It drives whether the
> + * existing GPU mappings must be invalidated and/or remapped.
> + */
> +enum amdgpu_svm_attr_change_trigger {
> +	AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE = (1U << 0),
> +	AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE = (1U << 1),
> +	AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE = (1U << 2),
> +	AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE = (1U << 3),
> +	AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE = (1U << 4),
> +	AMDGPU_SVM_ATTR_TRIGGER_PREFETCH = (1U << 5),
> +};
> +
> +/*
> + * Attribute changes that require invalidating existing GPU mappings.
> + * A granularity-only change does not, so it is intentionally excluded.
> + */
> +#define AMDGPU_SVM_ATTR_TRIGGER_NEED_INVALIDATE \
> +	(AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE | \
> +	 AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE | \
> +	 AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE | \
> +	 AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE)
> +
> +struct amdgpu_svm_attr_range *
> +amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
> +			   unsigned long page);
> +struct amdgpu_svm_attr_range *
> +amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
> +				  unsigned long page,
> +				  unsigned long *start_page,
> +				  unsigned long *last_page);
> +void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
> +				 struct amdgpu_svm_attrs *attrs);
> +
> +struct amdgpu_svm_attr_range *
> +amdgpu_svm_attr_range_alloc(unsigned long start_page,
> +			   unsigned long last_page,
> +			   const struct amdgpu_svm_attrs *attrs);
> +void amdgpu_svm_attr_range_insert_locked(struct amdgpu_svm_attr_tree *attr_tree,
> +					 struct amdgpu_svm_attr_range *range);
> +bool amdgpu_svm_attr_prefer_vram(const struct amdgpu_svm_attrs *attrs);
> +struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
> +					unsigned long addr);
> +
> +#endif /* __AMDGPU_SVM_ATTR_H__ */

Re: [PATCH v10 04/22] drm/amdgpu: implement SVM attribute set/get/clear operations

[Christian König]

On 10/21/26 04:58, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Complete attribute subsystem with validation, tree modification, and
> public API:
> - Attribute validation: per-type checks for preferred_loc, prefetch_loc,
>   access, flags, granularity
> - Flag application: attr_apply_flag, amdgpu_svm_attr_apply
> - VMA range validation: amdgpu_svm_attr_validate_range_vma
> - VM BO overlap detection: amdgpu_svm_attr_check_vm_bo
> - Tree modification with gap/overlap handling:
>   amdgpu_svm_attr_set_hole (new range in gap),
>   amdgpu_svm_attr_set_existing (split/update existing range)
> - amdgpu_svm_attr_set_range: cursor based iteration with per segment
> - Lifecycle: amdgpu_svm_attr_tree_create/destroy
> - Public API: amdgpu_svm_attr_set (validate, set_range with retry),
>   amdgpu_svm_attr_get (aggregate attrs across interval),
>   amdgpu_svm_attr_clear (interval removal with split),
>   amdgpu_svm_attr_reset
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Huang Rui <ray.huang@amd.com>
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 736 +++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h |  20 +
>  2 files changed, 756 insertions(+)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
> index a5afa75421fbd..c76e10473968e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
> @@ -78,6 +78,43 @@ struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
>  	return vma;
>  }
>  
> +/*
> + * amdgpu_svm_attr_range_is_valid - validate an attr ioctl address range
> + *
> + * Synchronous input validation for the SVM set/get-attr ioctls: walk the
> + * CPU VMAs covering [start_page, last_page] and reject the request if any
> + * part is unmapped or backed by an unsupported VMA (-EOPNOTSUPP).
> + *
> + */
> +static int amdgpu_svm_attr_range_is_valid(struct amdgpu_svm *svm,
> +					  unsigned long start_page,
> +					  unsigned long last_page)
> +{
> +	struct mm_struct *mm = svm->gpusvm.mm;
> +	struct vm_area_struct *vma;
> +	unsigned long start, end;
> +	int r = 0;
> +
> +	if (!mm)
> +		return -EFAULT;
> +
> +	start = start_page << PAGE_SHIFT;
> +	end = (last_page + 1) << PAGE_SHIFT;
> +
> +	mmap_read_lock(mm);
> +	do {
> +		vma = amdgpu_svm_check_vma(mm, start);
> +		if (IS_ERR(vma)) {
> +			r = PTR_ERR(vma);
> +			break;
> +		}
> +		start = min(end, vma->vm_end);
> +	} while (start < end);
> +	mmap_read_unlock(mm);

Again a clear NAK to any of those static checks!

The result of the check becomes invalid ass soon as the mman_read_lock is dropped, so that here only gives a completely false picture if some operation is valid or not.

We have two possibilities here, either keep the mman_read_lock while installing the SVM range or completely drop the check and make sure that faulting on a range which isn't backed by any VMA or an invalid VMA is properly handled in the fault handler.

I think we need to have a resilent fault handler who can deal with such errors anyway, so it is probably the best approach to go down that route.

Regards,
Christian.

> +
> +	return r;
> +}
> +
>  static void attr_set_interval(struct amdgpu_svm_attr_range *range,
>  				unsigned long start_page,
>  				unsigned long last_page)
> @@ -233,3 +270,702 @@ static void attr_remove_range_locked(struct amdgpu_svm_attr_tree *attr_tree,
>  	if (free_range)
>  		kfree(range);
>  }
> +
> +static void attr_change_ctx_set(
> +		struct attr_set_ctx *change,
> +		const struct amdgpu_svm_attrs *old_attrs,
> +		const struct amdgpu_svm_attrs *new_attrs,
> +		unsigned long start_page,
> +		unsigned long last_page)
> +{
> +	change->old_attrs = *old_attrs;
> +	change->new_attrs = *new_attrs;
> +	change->start_page = start_page;
> +	change->last_page = last_page;
> +}
> +
> +static inline int attr_check_preferred_loc(uint32_t value)
> +{
> +	/*
> +	 * SYSMEM and UNDEFINED are always valid; LOCAL/HIVE select device
> +	 * VRAM (see enum amdgpu_ioctl_svm_location).
> +	 */
> +	switch (value) {
> +	case AMDGPU_SVM_LOCATION_SYSMEM:
> +	case AMDGPU_SVM_LOCATION_UNDEFINED:
> +	case AMDGPU_SVM_LOCATION_LOCAL:
> +	case AMDGPU_SVM_LOCATION_HIVE:
> +		return 0;
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
> +static inline int attr_check_prefetch_loc(uint32_t value)
> +{
> +	/*
> +	 * Prefetch accepts the same values as the preferred location (see
> +	 * enum amdgpu_ioctl_svm_location); UNDEFINED means no prefetch and is
> +	 * what GET_ATTR reports for default or mixed ranges, so it must round
> +	 * trip through SET for CRIU restore.
> +	 */
> +	switch (value) {
> +	case AMDGPU_SVM_LOCATION_SYSMEM:
> +	case AMDGPU_SVM_LOCATION_UNDEFINED:
> +	case AMDGPU_SVM_LOCATION_LOCAL:
> +	case AMDGPU_SVM_LOCATION_HIVE:
> +		return 0;
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
> +static inline int attr_check_access(uint32_t value)
> +{
> +	switch (value) {
> +	case AMDGPU_SVM_ACCESS_INACCESSIBLE:
> +	case AMDGPU_SVM_ACCESS_IN_PLACE:
> +	case AMDGPU_SVM_ACCESS_ALLOW_MIGRATE:
> +		return 0;
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
> +static bool attr_flag_type_to_bit(uint32_t type, uint32_t *flag_bit)
> +{
> +	if (type < AMDGPU_SVM_ATTR_HOST_ACCESS ||
> +	    type > AMDGPU_SVM_ATTR_GPU_READ_MOSTLY)
> +		return false;
> +
> +	*flag_bit = 1u << (type - AMDGPU_SVM_ATTR_HOST_ACCESS);
> +	return true;
> +}
> +
> +static inline int attr_check_flag_value(uint32_t value)
> +{
> +	if (value > 1)
> +		return -EINVAL;
> +
> +	return 0;
> +}
> +
> +static inline int attr_check_flag_attr(uint32_t type, uint32_t value)
> +{
> +	uint32_t flag_bit;
> +	int ret;
> +
> +	if (!attr_flag_type_to_bit(type, &flag_bit))
> +		return -EINVAL;
> +
> +	ret = attr_check_flag_value(value);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static inline int attr_check_granularity(uint32_t value)
> +{
> +	return 0;
> +}
> +
> +static int attr_set_validate(const struct drm_amdgpu_svm_attribute *attr)
> +{
> +	switch (attr->type) {
> +	case AMDGPU_SVM_ATTR_PREFERRED_LOC:
> +		return attr_check_preferred_loc(attr->value);
> +	case AMDGPU_SVM_ATTR_PREFETCH_LOC:
> +		return attr_check_prefetch_loc(attr->value);
> +	case AMDGPU_SVM_ATTR_ACCESS:
> +		return attr_check_access(attr->value);
> +	case AMDGPU_SVM_ATTR_GRANULARITY:
> +		return attr_check_granularity(attr->value);
> +	case AMDGPU_SVM_ATTR_HOST_ACCESS:
> +	case AMDGPU_SVM_ATTR_COHERENT:
> +	case AMDGPU_SVM_ATTR_GPU_RO:
> +	case AMDGPU_SVM_ATTR_GPU_EXEC:
> +	case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
> +	case AMDGPU_SVM_ATTR_EXT_COHERENT:
> +		return attr_check_flag_attr(attr->type, attr->value);
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
> +static void attr_apply_flag(struct amdgpu_svm_attrs *attrs,
> +			    uint32_t type, uint32_t value)
> +{
> +	uint32_t flag_bit;
> +
> +	if (!attr_flag_type_to_bit(type, &flag_bit))
> +		return;
> +
> +	if (value)
> +		attrs->flags |= flag_bit;
> +	else
> +		attrs->flags &= ~flag_bit;
> +}
> +
> +static void attr_apply(struct amdgpu_svm_attrs *attrs,
> +					uint32_t nattr,
> +					const struct drm_amdgpu_svm_attribute *pattrs)
> +{
> +	const struct drm_amdgpu_svm_attribute *attr;
> +
> +	for (attr = pattrs; nattr--; attr++) {
> +		switch (attr->type) {
> +		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
> +			attrs->preferred_loc = (int32_t)attr->value;
> +			break;
> +		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
> +			attrs->prefetch_loc = (int32_t)attr->value;
> +			break;
> +		case AMDGPU_SVM_ATTR_ACCESS:
> +			attrs->access = (enum amdgpu_ioctl_svm_access)attr->value;
> +			break;
> +		case AMDGPU_SVM_ATTR_HOST_ACCESS:
> +		case AMDGPU_SVM_ATTR_COHERENT:
> +		case AMDGPU_SVM_ATTR_GPU_RO:
> +		case AMDGPU_SVM_ATTR_GPU_EXEC:
> +		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
> +		case AMDGPU_SVM_ATTR_EXT_COHERENT:
> +			attr_apply_flag(attrs, attr->type, attr->value);
> +			break;
> +		case AMDGPU_SVM_ATTR_GRANULARITY:
> +			attrs->granularity = min_t(uint32_t, attr->value, 0x3f);
> +			break;
> +		default:
> +			break;
> +		}
> +	}
> +}
> +
> +static bool attr_same_attrs(const struct amdgpu_svm_attr_range *range,
> +			    uint32_t nattr,
> +			    const struct drm_amdgpu_svm_attribute *attrs)
> +{
> +	struct amdgpu_svm_attrs target;
> +
> +	target = range->attrs;
> +	attr_apply(&target, nattr, attrs);
> +	return attr_equal(&range->attrs, &target);
> +}
> +
> +static int
> +amdgpu_svm_attr_set_hole(struct amdgpu_svm_attr_tree *attr_tree,
> +			  const struct amdgpu_svm_attrs *default_attrs,
> +			  unsigned long start_page, unsigned long last_page,
> +			  uint32_t nattr,
> +			  const struct drm_amdgpu_svm_attribute *attrs,
> +			  struct attr_set_ctx *change)
> +{
> +	struct amdgpu_svm_attrs new_attrs;
> +	struct amdgpu_svm_attr_range *range;
> +
> +	lockdep_assert_held(&attr_tree->lock);
> +
> +	if (start_page > last_page)
> +		return 0;
> +
> +	new_attrs = *default_attrs;
> +	attr_apply(&new_attrs, nattr, attrs);
> +
> +	/* Always create a range entry even when attrs equal defaults */
> +	range = amdgpu_svm_attr_range_alloc(start_page, last_page, &new_attrs);
> +	if (!range)
> +		return -ENOMEM;
> +
> +	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
> +
> +	attr_change_ctx_set(change, default_attrs,
> +				       &new_attrs, start_page, last_page);
> +	return 0;
> +}
> +
> +/*
> + * amdgpu_svm_attr_set_existing() - Apply attrs to part of an existing range
> + *
> + * Update @range over [@start_page, @last_page]. If the request fully covers
> + * @range the attrs are updated in place, otherwise the range is split into the
> + * head/tail, carrying the old attrs, and the covered middle, carrying
> + * the new attrs. The resulting change is reported by @change.
> + */
> +static int
> +amdgpu_svm_attr_set_existing(struct amdgpu_svm_attr_tree *attr_tree,
> +			     struct amdgpu_svm_attr_range *range,
> +			     unsigned long start_page, unsigned long last_page,
> +			     uint32_t nattr,
> +			     const struct drm_amdgpu_svm_attribute *attrs,
> +			     struct attr_set_ctx *change)
> +{
> +	unsigned long range_start = amdgpu_svm_attr_start_page(range);
> +	unsigned long range_last = amdgpu_svm_attr_last_page(range);
> +	struct amdgpu_svm_attr_range *left = NULL;
> +	struct amdgpu_svm_attr_range *right = NULL;
> +	struct amdgpu_svm_attrs old_attrs;
> +	struct amdgpu_svm_attrs new_attrs;
> +
> +	lockdep_assert_held(&attr_tree->lock);
> +
> +	old_attrs = range->attrs;
> +
> +	if (attr_same_attrs(range, nattr, attrs)) {
> +		/* Report old==new so apply_attr_change can decide */
> +		attr_change_ctx_set(change, &old_attrs,
> +					       &old_attrs,
> +					       start_page, last_page);
> +		return 0;
> +	}
> +
> +	new_attrs = old_attrs;
> +	attr_apply(&new_attrs, nattr, attrs);
> +
> +	/* only need to update attr */
> +	if (start_page == range_start && last_page == range_last) {
> +		range->attrs = new_attrs;
> +		attr_change_ctx_set(change, &old_attrs,
> +					       &new_attrs, start_page, last_page);
> +		return 0;
> +	}
> +
> +	/* split head */
> +	if (start_page > range_start) {
> +		left = amdgpu_svm_attr_range_alloc(range_start, start_page - 1, &old_attrs);
> +		if (!left)
> +			return -ENOMEM;
> +	}
> +
> +	/* split tail */
> +	if (last_page < range_last) {
> +		right = amdgpu_svm_attr_range_alloc(last_page + 1, range_last, &old_attrs);
> +		if (!right) {
> +			kfree(left);
> +			return -ENOMEM;
> +		}
> +	}
> +
> +	attr_remove_range_locked(attr_tree, range, false);
> +	if (left)
> +		amdgpu_svm_attr_range_insert_locked(attr_tree, left);
> +	attr_set_interval(range, start_page, last_page);
> +	range->attrs = new_attrs;
> +	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
> +	if (right)
> +		amdgpu_svm_attr_range_insert_locked(attr_tree, right);
> +
> +	attr_change_ctx_set(change, &old_attrs,
> +				       &new_attrs, start_page, last_page);
> +	return 0;
> +}
> +
> +/*
> + * amdgpu_svm_attr_set_range() - Apply attr changes across [@start_page, @last_page]
> + *
> + * Walk the interval in segments, each segment being either an existing range
> + * or a hole.
> + */
> +static int
> +amdgpu_svm_attr_set_range(struct amdgpu_svm_attr_tree *attr_tree,
> +			  const struct amdgpu_svm_attrs *default_attrs,
> +			  unsigned long start_page, unsigned long last_page,
> +			  uint32_t nattr,
> +			  const struct drm_amdgpu_svm_attribute *attrs)
> +{
> +	struct amdgpu_svm *svm = attr_tree->svm;
> +	unsigned long cursor = start_page;
> +	bool need_retry = false;
> +
> +	while (cursor <= last_page) {
> +		struct interval_tree_node *node;
> +		unsigned long seg_last;
> +		struct attr_set_ctx change = { 0 };
> +		int ret;
> +
> +		amdgpu_svm_lock(svm);
> +		mutex_lock(&attr_tree->lock);
> +		node = interval_tree_iter_first(&attr_tree->tree, cursor, cursor);
> +		if (node) {
> +			struct amdgpu_svm_attr_range *range;
> +
> +			range = container_of(node, struct amdgpu_svm_attr_range, it_node);
> +			seg_last = min(last_page, amdgpu_svm_attr_last_page(range));
> +			ret = amdgpu_svm_attr_set_existing(attr_tree, range,
> +								   cursor, seg_last,
> +								   nattr, attrs, &change);
> +		} else {
> +			struct interval_tree_node *next;
> +
> +			seg_last = last_page;
> +			if (cursor != ULONG_MAX) {
> +				next = interval_tree_iter_first(&attr_tree->tree,
> +								cursor + 1,
> +								ULONG_MAX);
> +				if (next) {
> +					struct amdgpu_svm_attr_range *next_range;
> +
> +					next_range = container_of(next,
> +						struct amdgpu_svm_attr_range,
> +						it_node);
> +					seg_last = min(last_page,
> +						       amdgpu_svm_attr_start_page(next_range) - 1);
> +				}
> +			}
> +			ret = amdgpu_svm_attr_set_hole(attr_tree,
> +							       default_attrs,
> +							       cursor, seg_last,
> +							       nattr, attrs,
> +							       &change);
> +		}
> +		mutex_unlock(&attr_tree->lock);
> +
> +		if (ret) {
> +			amdgpu_svm_unlock(svm);
> +			return ret;
> +		}
> +
> +		ret = amdgpu_svm_apply_attr_change(svm,
> +						   &change.old_attrs,
> +						   &change.new_attrs,
> +						   change.start_page,
> +						   change.last_page);
> +		amdgpu_svm_unlock(svm);
> +
> +		if (ret == -EAGAIN) {
> +			need_retry = true;
> +			ret = 0;
> +		}
> +
> +		if (ret)
> +			return ret;
> +
> +		if (seg_last == ULONG_MAX || seg_last == last_page)
> +			break;
> +
> +		cursor = seg_last + 1;
> +	}
> +
> +	return need_retry ? -EAGAIN : 0;
> +}
> +
> +struct amdgpu_svm_attr_tree *
> +amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm)
> +{
> +	struct amdgpu_svm_attr_tree *attr_tree;
> +
> +	attr_tree = kzalloc_obj(*attr_tree, GFP_KERNEL);
> +	if (!attr_tree)
> +		return NULL;
> +
> +	mutex_init(&attr_tree->lock);
> +	attr_tree->tree = RB_ROOT_CACHED;
> +	INIT_LIST_HEAD(&attr_tree->range_list);
> +	attr_tree->svm = svm;
> +	return attr_tree;
> +}
> +
> +void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree)
> +{
> +	struct amdgpu_svm_attr_range *range, *tmp;
> +
> +	if (!attr_tree)
> +		return;
> +
> +	mutex_lock(&attr_tree->lock);
> +	list_for_each_entry_safe(range, tmp, &attr_tree->range_list, list) {
> +		interval_tree_remove(&range->it_node, &attr_tree->tree);
> +		list_del_init(&range->list);
> +		kfree(range);
> +	}
> +	mutex_unlock(&attr_tree->lock);
> +
> +	mutex_destroy(&attr_tree->lock);
> +	kfree(attr_tree);
> +}
> +
> +int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
> +			uint64_t start,
> +			uint64_t size,
> +			uint32_t nattr,
> +			const struct drm_amdgpu_svm_attribute *attrs)
> +{
> +	struct amdgpu_svm *svm = attr_tree->svm;
> +	struct amdgpu_svm_attrs default_attrs;
> +	unsigned long start_page, last_page;
> +	uint32_t i;
> +	int r;
> +
> +	start_page = start >> PAGE_SHIFT;
> +	last_page = (start + size - 1) >> PAGE_SHIFT;
> +
> +	for (i = 0; i < nattr; i++) {
> +		pr_debug("set attr type %u value 0x%08x range [0x%lx-0x%lx] xnack:%d",
> +			 attrs[i].type, attrs[i].value, start_page, last_page,
> +			 svm->xnack_enabled ? 1 : 0);
> +		r = attr_set_validate(&attrs[i]);
> +		if (r) {
> +			pr_debug("invalid attribute %u value 0x%08x",
> +					 attrs[i].type, attrs[i].value);
> +			return r;
> +		}
> +	}
> +
> +	r = amdgpu_svm_attr_range_is_valid(svm, start_page, last_page);
> +	if (r)
> +		return r;
> +
> +	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
> +
> +retry:
> +	r = amdgpu_svm_attr_set_range(attr_tree, &default_attrs,
> +					       start_page, last_page,
> +					       nattr, attrs);
> +	if (r == -EAGAIN) {
> +		pr_debug("attr_set retry [0x%lx-0x%lx]\n",
> +				 start_page, last_page);
> +		amdgpu_svm_sync_work(svm);
> +		cond_resched();
> +		goto retry;
> +	}
> +
> +	return r;
> +}
> +
> +int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
> +			  unsigned long start_page,
> +			  unsigned long last_page)
> +{
> +	struct interval_tree_node *node;
> +	int r = 0;
> +
> +	if (start_page > last_page)
> +		return -EINVAL;
> +
> +	mutex_lock(&attr_tree->lock);
> +
> +	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
> +	while (node) {
> +		struct interval_tree_node *next;
> +		struct amdgpu_svm_attr_range *range;
> +		unsigned long range_start;
> +		unsigned long range_last;
> +
> +		range = container_of(node, struct amdgpu_svm_attr_range, it_node);
> +		next = interval_tree_iter_next(node, start_page, last_page);
> +		range_start = amdgpu_svm_attr_start_page(range);
> +		range_last = amdgpu_svm_attr_last_page(range);
> +
> +		if (range_start < start_page && range_last > last_page) {
> +			struct amdgpu_svm_attr_range *tail;
> +
> +			tail = amdgpu_svm_attr_range_alloc(last_page + 1, range_last,
> +							   &range->attrs);
> +			if (!tail) {
> +				r = -ENOMEM;
> +				break;
> +			}
> +
> +			attr_remove_range_locked(attr_tree, range, false);
> +			attr_set_interval(range, range_start, start_page - 1);
> +			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
> +			amdgpu_svm_attr_range_insert_locked(attr_tree, tail);
> +		} else if (range_start < start_page) {
> +			attr_remove_range_locked(attr_tree, range, false);
> +			attr_set_interval(range, range_start, start_page - 1);
> +			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
> +		} else if (range_last > last_page) {
> +			attr_remove_range_locked(attr_tree, range, false);
> +			attr_set_interval(range, last_page + 1, range_last);
> +			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
> +		} else {
> +			attr_remove_range_locked(attr_tree, range, true);
> +		}
> +
> +		node = next;
> +	}
> +
> +	mutex_unlock(&attr_tree->lock);
> +	return r;
> +}
> +
> +int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
> +			  unsigned long start_page,
> +			  unsigned long last_page)
> +{
> +	/*
> +	 * Range with no attr node in this implementation is treated the
> +	 * same as one that explicitly stores default attrs. So clear
> +	 * the attribute ranges when user wants to reset the attrs.
> +	 *
> +	 * - GET_ATTR: return default_attrs to userspace when there is no attr
> +	 *   range.
> +	 * - SET_ATTR: holes are treated as having default_attrs when
> +	 *   computing the change trigger.
> +	 * - Fault: attr lookup falls back to defaults when no attr range exists,
> +	 *   so migration and PTE flag decisions are unchanged.
> +	 *
> +	 * This approach simplifies the implementation and avoids redundant
> +	 * attribute maintenance. This ioctl operation is for attribute only,
> +	 * so do not invalidate the GPU mapping here.
> +	 */
> +	return amdgpu_svm_attr_clear(attr_tree, start_page, last_page);
> +}
> +
> +static void attr_get_ctx_add(struct attr_get_ctx *ctx,
> +			       const struct amdgpu_svm_attrs *attrs)
> +{
> +	if (!ctx->has_range) {
> +		ctx->preferred_loc = attrs->preferred_loc;
> +		ctx->prefetch_loc = attrs->prefetch_loc;
> +		ctx->granularity = attrs->granularity;
> +		ctx->access = attrs->access;
> +		ctx->flags_and = attrs->flags;
> +		ctx->has_range = true;
> +		return;
> +	}
> +
> +	if (ctx->preferred_loc != attrs->preferred_loc)
> +		ctx->preferred_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
> +	if (ctx->prefetch_loc != attrs->prefetch_loc)
> +		ctx->prefetch_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
> +	if (attrs->granularity < ctx->granularity)
> +		ctx->granularity = attrs->granularity;
> +	if (ctx->access != attrs->access)
> +		ctx->access = AMDGPU_SVM_ACCESS_INACCESSIBLE;
> +	ctx->flags_and &= attrs->flags;
> +}
> +
> +static int attr_get_ctx_to_result(const struct attr_get_ctx *ctx,
> +				uint32_t nattr,
> +				struct drm_amdgpu_svm_attribute *attrs)
> +{
> +	uint32_t i;
> +
> +	for (i = 0; i < nattr; i++) {
> +		switch (attrs[i].type) {
> +		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
> +			attrs[i].value = ctx->preferred_loc;
> +			break;
> +		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
> +			attrs[i].value = ctx->prefetch_loc;
> +			break;
> +		case AMDGPU_SVM_ATTR_ACCESS:
> +			attrs[i].value = (uint32_t)ctx->access;
> +			break;
> +		case AMDGPU_SVM_ATTR_HOST_ACCESS:
> +		case AMDGPU_SVM_ATTR_COHERENT:
> +		case AMDGPU_SVM_ATTR_GPU_RO:
> +		case AMDGPU_SVM_ATTR_GPU_EXEC:
> +		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
> +		case AMDGPU_SVM_ATTR_EXT_COHERENT: {
> +			uint32_t flag_bit;
> +
> +			if (!attr_flag_type_to_bit(attrs[i].type, &flag_bit))
> +				return -EINVAL;
> +
> +			attrs[i].value = (ctx->flags_and & flag_bit) ? 1 : 0;
> +			break;
> +		}
> +		case AMDGPU_SVM_ATTR_GRANULARITY:
> +			attrs[i].value = ctx->granularity;
> +			break;
> +		default:
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +/*
> + * On return @out_size is shrunk to the length of the contiguous range starting
> + * at @start over which all attributes are identical, so user space (e.g. CRIU)
> + * can walk the address space one uniform range at a time.
> + */
> +int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
> +			uint64_t start, uint64_t size,
> +			uint32_t nattr,
> +			struct drm_amdgpu_svm_attribute *attrs,
> +			uint64_t *out_size)
> +{
> +	struct amdgpu_svm_attrs default_attrs;
> +	struct amdgpu_svm_attrs ref_attrs;
> +	struct attr_get_ctx ctx = { 0 };
> +	struct interval_tree_node *node;
> +	unsigned long start_page, last_page, cursor, homog_last;
> +	bool have_ref = false;
> +	int r;
> +
> +	start_page = start >> PAGE_SHIFT;
> +	last_page = (start + size - 1) >> PAGE_SHIFT;
> +
> +	r = amdgpu_svm_attr_range_is_valid(attr_tree->svm, start_page, last_page);
> +	if (r)
> +		return r;
> +
> +	mutex_lock(&attr_tree->lock);
> +	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
> +
> +	homog_last = start_page;
> +	cursor = start_page;
> +	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
> +
> +	while (cursor <= last_page) {
> +		const struct amdgpu_svm_attrs *range_attrs;
> +		unsigned long range_last = last_page;
> +		struct amdgpu_svm_attr_range *range = NULL;
> +		unsigned long next;
> +
> +		if (node) {
> +			range = container_of(node, struct amdgpu_svm_attr_range,
> +					     it_node);
> +
> +			if (amdgpu_svm_attr_last_page(range) < cursor) {
> +				node = interval_tree_iter_next(node, start_page,
> +							       last_page);
> +				continue;
> +			}
> +
> +			if (amdgpu_svm_attr_start_page(range) <= cursor) {
> +				range_last = min(last_page, amdgpu_svm_attr_last_page(range));
> +				node = interval_tree_iter_next(node, start_page,
> +							       last_page);
> +			} else {
> +				range_last = min(last_page,
> +						 amdgpu_svm_attr_start_page(range) - 1);
> +				range = NULL;
> +			}
> +		}
> +
> +		range_attrs = range ? &range->attrs : &default_attrs;
> +
> +		if (!have_ref) {
> +			ref_attrs = *range_attrs;
> +			have_ref = true;
> +		} else if (!attr_equal(range_attrs, &ref_attrs)) {
> +			break;
> +		}
> +
> +		homog_last = range_last;
> +
> +		if (range_last == ULONG_MAX)
> +			break;
> +
> +		next = range_last + 1;
> +		if (next <= cursor)
> +			break;
> +		cursor = next;
> +	}
> +
> +	if (!have_ref)
> +		ref_attrs = default_attrs;
> +
> +	attr_get_ctx_add(&ctx, &ref_attrs);
> +	r = attr_get_ctx_to_result(&ctx, nattr, attrs);
> +	mutex_unlock(&attr_tree->lock);
> +
> +	if (!r && out_size)
> +		*out_size = ((uint64_t)(homog_last - start_page) + 1) << PAGE_SHIFT;
> +
> +	return r;
> +}
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
> index 36fee11aa9ea0..56bcd81c608cb 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
> @@ -158,6 +158,9 @@ enum amdgpu_svm_attr_change_trigger {
>  	 AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE | \
>  	 AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE)
>  
> +struct amdgpu_svm_attr_tree *
> +amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm);
> +void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree);
>  struct amdgpu_svm_attr_range *
>  amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
>  			   unsigned long page);
> @@ -169,6 +172,23 @@ amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
>  void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
>  				 struct amdgpu_svm_attrs *attrs);
>  
> +int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
> +			   uint64_t start,
> +			   uint64_t size,
> +			   uint32_t nattr,
> +			   const struct drm_amdgpu_svm_attribute *attrs);
> +int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
> +				       uint64_t start,
> +				       uint64_t size,
> +				       uint32_t nattr,
> +				       struct drm_amdgpu_svm_attribute *attrs,
> +				       uint64_t *out_size);
> +int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
> +			  unsigned long start_page,
> +			  unsigned long last_page);
> +int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
> +			  unsigned long start_page,
> +			  unsigned long last_page);
>  struct amdgpu_svm_attr_range *
>  amdgpu_svm_attr_range_alloc(unsigned long start_page,
>  			   unsigned long last_page,

Re: [PATCH v10 05/22] drm/amdgpu: add SVM range types and work queue interface

[Christian König]

On 10/21/26 04:58, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Add amdgpu_svm_range.h with GPU mapped range types and interfaces:
> - enum amdgpu_svm_range_queue_state: NOT_QUEUED, IN_GC, PROCESSING
>   states for queue work
> - struct amdgpu_svm_range: extends drm_gpusvm_range with gpu_mapped
>   state, queue_state, attribute flags, work queue node, pending
>   ops/pages, and validation timestamp
> - enum amdgpu_svm_range_op: NONE, UNMAP operation types
> - struct amdgpu_svm_range_op_ctx: dequeue context for GC processing
> - Inline helper: amdgpu_svm_range_invalidate_gpu_mapping
> - Range operations API declarations: find_or_insert, get_pages,
>   update_mapping, update_gpu_range, invalidate
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Huang Rui <ray.huang@amd.com>
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 129 ++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  30 ++++
>  2 files changed, 159 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
> new file mode 100644
> index 0000000000000..39db3a18b5f2f
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
> @@ -0,0 +1,129 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright 2026 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + */
> +
> +#ifndef __AMDGPU_SVM_RANGE_H__
> +#define __AMDGPU_SVM_RANGE_H__
> +
> +#include <drm/drm_gpusvm.h>
> +#include <drm/drm_pagemap.h>
> +
> +#include "amdgpu_svm.h"
> +#include "amdgpu_vm.h"
> +
> +#include <linux/ktime.h>
> +#include <linux/list.h>
> +#include <linux/types.h>
> +
> +struct amdgpu_svm;
> +struct amdgpu_svm_attr_range;
> +struct amdgpu_svm_attrs;
> +struct dma_fence;
> +struct drm_exec;
> +struct drm_gpusvm_notifier;
> +struct drm_gpusvm_range;
> +struct mmu_notifier_range;
> +
> +/**
> + * enum amdgpu_svm_range_queue_state - Work queue state of an SVM range
> + * @AMDGPU_SVM_RANGE_NOT_QUEUED: Not on any work list.
> + * @AMDGPU_SVM_RANGE_IN_GC: Queued on the GC list.
> + * @AMDGPU_SVM_RANGE_PROCESSING: Being processed in workqueue.
> + */
> +enum amdgpu_svm_range_queue_state {
> +	AMDGPU_SVM_RANGE_NOT_QUEUED = 0,
> +	AMDGPU_SVM_RANGE_IN_GC,
> +	AMDGPU_SVM_RANGE_PROCESSING,
> +};
> +
> +/**
> + * struct amdgpu_svm_range - amdgpu GPU mapped SVM range
> + * @base: Embedded drm_gpusvm_range base.
> + * @work_node: Links the range into the work list. Protected by amdgpu_svm.work_lock.
> + * @gpu_mapped: True while the range has a live GPU mapping. Accessed with
> + *              READ_ONCE/WRITE_ONCE.
> + * @queue_state: Current work queue state @amdgpu_svm_range_queue_state.
> + *               Protected by amdgpu_svm.work_lock.
> + * @pending_ops: Pending amdgpu_svm_range_op bitmask. Protected by work_lock.
> + * @pending_start_page: First page of the accumulated pending interval.
> + * @pending_last_page: Last page of the accumulated pending interval.
> + * @attr_flags: Attribute flags last applied to the GPU mapping.
> + * @validate_timestamp: Time of the last successful mapping validation.
> + */
> +struct amdgpu_svm_range {
> +	struct drm_gpusvm_range base;
> +	struct list_head work_node;
> +	bool gpu_mapped;

> +	u8 queue_state;

What is that good for and why is it needed?

Regards,
Christian.

> +	u8 pending_ops;
> +	unsigned long pending_start_page;
> +	unsigned long pending_last_page;
> +	uint32_t attr_flags;
> +	ktime_t validate_timestamp;
> +};
> +
> +static inline struct amdgpu_svm_range *
> +to_amdgpu_svm_range(struct drm_gpusvm_range *range)
> +{
> +	return container_of(range, struct amdgpu_svm_range, base);
> +}
> +
> +static inline void
> +amdgpu_svm_range_invalidate_gpu_mapping(struct amdgpu_svm_range *range)
> +{
> +	WRITE_ONCE(range->gpu_mapped, false);
> +}
> +
> +#define AMDGPU_SVM_RANGE_TRACE(r__, op__)				    \
> +	trace_amdgpu_svm_range((op__),					    \
> +		to_amdgpu_svm((r__)->base.gpusvm)->vm->pasid,		    \
> +		(r__)->base.gpusvm, READ_ONCE((r__)->gpu_mapped),	    \
> +		(r__)->base.pages.notifier_seq,				    \
> +		drm_gpusvm_range_start(&(r__)->base) >> PAGE_SHIFT,	    \
> +		drm_gpusvm_range_end(&(r__)->base) >> PAGE_SHIFT)
> +
> +/**
> + * enum amdgpu_svm_range_op - Pending operation bits for an SVM range
> + * @AMDGPU_SVM_RANGE_OP_NONE: No pending operation.
> + * @AMDGPU_SVM_RANGE_OP_UNMAP: Range needs to be unmapped/removed by the GC.
> + */
> +enum amdgpu_svm_range_op {
> +	AMDGPU_SVM_RANGE_OP_NONE    = 0,
> +	AMDGPU_SVM_RANGE_OP_UNMAP   = BIT(0),
> +};
> +
> +/**
> + * struct amdgpu_svm_range_op_ctx - context of a range workqueue operation
> + * @range: The dequeued range to process.
> + * @start_page: First page for workqueue operation.
> + * @last_page: Last page for workqueue operation.
> + * @pending_ops: amdgpu_svm_range_op bitmask.
> + */
> +struct amdgpu_svm_range_op_ctx {
> +	struct amdgpu_svm_range *range;
> +	unsigned long start_page;
> +	unsigned long last_page;
> +	uint8_t pending_ops;
> +};
> +
> +#endif /* __AMDGPU_SVM_RANGE_H__ */
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> index 599c1d0e512b5..86ba09cba34c9 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> @@ -116,6 +116,36 @@ TRACE_EVENT(amdgpu_iv,
>  		      __entry->src_data[2], __entry->src_data[3])
>  );
>  
> +TRACE_EVENT(amdgpu_svm_range,
> +	    TP_PROTO(const char *op, u32 pasid, void *gpusvm, bool mapped,
> +		     unsigned long seqno, unsigned long start_page,
> +		     unsigned long end_page),
> +	    TP_ARGS(op, pasid, gpusvm, mapped, seqno, start_page, end_page),
> +	    TP_STRUCT__entry(
> +			     __string(op, op)
> +			     __field(u32, pasid)
> +			     __field(void *, gpusvm)
> +			     __field(bool, mapped)
> +			     __field(unsigned long, seqno)
> +			     __field(unsigned long, start_page)
> +			     __field(unsigned long, end_page)
> +			    ),
> +	    TP_fast_assign(
> +			   __assign_str(op);
> +			   __entry->pasid = pasid;
> +			   __entry->gpusvm = gpusvm;
> +			   __entry->mapped = mapped;
> +			   __entry->seqno = seqno;
> +			   __entry->start_page = start_page;
> +			   __entry->end_page = end_page;
> +			   ),
> +	    TP_printk("%s: pasid=%u gpusvm=%p mapped=%d seqno=%lu range=[0x%lx-0x%lx]-0x%lx",
> +		      __get_str(op), __entry->pasid, __entry->gpusvm,
> +		      __entry->mapped, __entry->seqno,
> +		      __entry->start_page, __entry->end_page,
> +		      __entry->end_page - __entry->start_page)
> +);
> +
>  
>  TRACE_EVENT(amdgpu_bo_create,
>  	    TP_PROTO(struct amdgpu_bo *bo),

Re: [PATCH v10 06/22] drm/amdgpu/gmc: add get_svm_pte_flags callback

[Christian König]

On 10/21/26 04:58, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Add a gmc_funcs callback for computing the GPU PTE flags of an SVM
> mapping, plus a HW-agnostic input struct (amdgpu_svm_pte_flags_params)
> and the amdgpu_gmc_get_svm_pte_flags() wrapper. This lets the per-IP
> MTYPE/PTE selection live in the HW IP specific gmc_vX_0.c files instead
> of the SVM core, mirroring how get_vm_pte handles BO mappings.
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Huang Rui <ray.huang@amd.com>
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  24 +++++
>  drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c  |  69 ++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 119 ++++++++++++++++++++++++
>  3 files changed, 212 insertions(+)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> index cffd5ca3fc66b..3f668fb71a632 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> @@ -150,6 +150,24 @@ struct amdgpu_vmhub {
>  	const struct amdgpu_vmhub_funcs *vmhub_funcs;
>  };
>  
> +/* Decoded inputs for the SVM PTE flag selection, gmc_funcs::get_svm_pte_flags */
> +struct amdgpu_svm_pte_flags_params {
> +	/* SVM attribute: coherent */
> +	bool coherent;
> +	/* SVM attribute: extended coherent */
> +	bool ext_coherent;
> +	/* SVM attribute: GPU read only */
> +	bool gpu_ro;
> +	/* SVM attribute: GPU executable */
> +	bool gpu_exec;
> +	/* mapping is in VRAM local device or remote P2P */
> +	bool is_vram;
> +	/* mapping is in this GPU's local VRAM */
> +	bool is_local;
> +	/* remote VRAM BO is on a GPU in the same XGMI hive as this GPU. */
> +	bool same_hive;
> +};
> +
>  /*
>   * GPU MC structures, functions & helpers
>   */
> @@ -178,6 +196,10 @@ struct amdgpu_gmc_funcs {
>  			   struct amdgpu_bo *bo,
>  			   uint32_t vm_flags,
>  			   uint64_t *pte_flags);
> +	/* get the full pte flags for an SVM mapping */
> +	uint64_t (*get_svm_pte_flags)(struct amdgpu_device *adev,
> +				      struct amdgpu_vm *vm,
> +				      const struct amdgpu_svm_pte_flags_params *params);

That looks like a really really bad idea to me.

Please add the SVM flags to the existing amdgpu_gmc_get_vm_pte() function.

Additional to that you also need to modify the destination addr and not just the flags.

Regards,
Christian.

>  	/* override per-page pte flags */
>  	void (*override_vm_pte_flags)(struct amdgpu_device *dev,
>  				      struct amdgpu_vm *vm,
> @@ -378,6 +400,8 @@ struct amdgpu_gmc {
>  #define amdgpu_gmc_get_vm_pte(adev, vm, bo, vm_flags, pte_flags) \
>  	((adev)->gmc.gmc_funcs->get_vm_pte((adev), (vm), (bo), (vm_flags), \
>  					   (pte_flags)))
> +#define amdgpu_gmc_get_svm_pte_flags(adev, vm, params) \
> +	((adev)->gmc.gmc_funcs->get_svm_pte_flags((adev), (vm), (params)))
>  #define amdgpu_gmc_override_vm_pte_flags(adev, vm, addr, pte_flags)	\
>  	(adev)->gmc.gmc_funcs->override_vm_pte_flags			\
>  		((adev), (vm), (addr), (pte_flags))
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
> index 0cc73982ed6f9..a541173e34a4e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
> @@ -551,6 +551,74 @@ static void gmc_v12_0_get_vm_pte(struct amdgpu_device *adev,
>  		*flags = AMDGPU_PTE_MTYPE_GFX12(*flags, MTYPE_UC);
>  }
>  
> +/*
> + * Mirror of the GFX12 cases of svm_range_get_pte_flags(), with the
> + * per IP logic kept here in the HW specific file.
> + */
> +static uint64_t
> +gmc_v12_0_get_svm_pte_flags(struct amdgpu_device *adev,
> +			    struct amdgpu_vm *vm,
> +			    const struct amdgpu_svm_pte_flags_params *params)
> +{
> +	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
> +	bool coherent = params->coherent;
> +	bool ext_coherent = params->ext_coherent;
> +	bool is_local = params->is_local;
> +	bool is_vram = params->is_vram;
> +	bool snoop = !is_vram;
> +	uint32_t mapping_flags = 0;
> +	unsigned int mtype_local, mtype_remote;
> +	bool is_aid_a1;
> +	uint64_t pte_flags;
> +
> +	switch (gc_ip_version) {
> +	case IP_VERSION(12, 0, 0):
> +	case IP_VERSION(12, 0, 1):
> +		mapping_flags |= AMDGPU_VM_MTYPE_NC;
> +		break;
> +	case IP_VERSION(12, 1, 0):
> +		is_aid_a1 = (adev->rev_id & 0x10);
> +		mtype_local = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_RW :
> +				amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
> +				is_aid_a1 ? AMDGPU_VM_MTYPE_RW : AMDGPU_VM_MTYPE_NC;
> +		mtype_remote = is_aid_a1 ? AMDGPU_VM_MTYPE_NC : AMDGPU_VM_MTYPE_UC;
> +		snoop = true;
> +
> +		if (is_local) {
> +			mapping_flags |= mtype_local;
> +		} else if (ext_coherent) {
> +			mapping_flags |= AMDGPU_VM_MTYPE_UC;
> +		} else {
> +			/* system memory or remote VRAM */
> +			mapping_flags |= mtype_remote;
> +		}
> +		break;
> +	default:
> +		mapping_flags |= coherent ?
> +			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +		break;
> +	}
> +
> +	if (params->gpu_exec)
> +		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
> +
> +	pte_flags = AMDGPU_PTE_VALID;
> +	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
> +	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
> +	pte_flags |= AMDGPU_PTE_IS_PTE;
> +
> +	gmc_v12_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
> +	pte_flags |= AMDGPU_PTE_READABLE;
> +	if (!params->gpu_ro)
> +		pte_flags |= AMDGPU_PTE_WRITEABLE;
> +
> +	if (gc_ip_version == IP_VERSION(12, 1, 0) &&
> +	    adev->have_atomics_support)
> +		pte_flags |= AMDGPU_PTE_BUS_ATOMICS;
> +
> +	return pte_flags;
> +}
> +
>  static unsigned gmc_v12_0_get_vbios_fb_size(struct amdgpu_device *adev)
>  {
>  	return 0;
> @@ -580,6 +648,7 @@ static const struct amdgpu_gmc_funcs gmc_v12_0_gmc_funcs = {
>  	.emit_pasid_mapping = gmc_v12_0_emit_pasid_mapping,
>  	.get_vm_pde = gmc_v12_0_get_vm_pde,
>  	.get_vm_pte = gmc_v12_0_get_vm_pte,
> +	.get_svm_pte_flags = gmc_v12_0_get_svm_pte_flags,
>  	.get_vbios_fb_size = gmc_v12_0_get_vbios_fb_size,
>  	.get_dcc_alignment = gmc_v12_0_get_dcc_alignment,
>  };
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> index 633772fa71fda..a3a5fa048a393 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> @@ -1199,6 +1199,124 @@ static void gmc_v9_0_get_vm_pte(struct amdgpu_device *adev,
>  		gmc_v9_0_get_coherence_flags(adev, vm, bo, vm_flags, flags);
>  }
>  
> +/*
> + * Mirror of the GFX9 cases of svm_range_get_pte_flags(), with the
> + * per IP logic kept here in the HW specific file.
> + */
> +static uint64_t
> +gmc_v9_0_get_svm_pte_flags(struct amdgpu_device *adev,
> +			   struct amdgpu_vm *vm,
> +			   const struct amdgpu_svm_pte_flags_params *params)
> +{
> +	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
> +	bool coherent = params->coherent;
> +	bool ext_coherent = params->ext_coherent;
> +	bool is_local = params->is_local;
> +	bool is_vram = params->is_vram;
> +	bool same_hive = params->same_hive;
> +	bool snoop = !is_vram;
> +	uint32_t mapping_flags = 0;
> +	unsigned int mtype_local;
> +	uint64_t pte_flags;
> +
> +	switch (gc_ip_version) {
> +	case IP_VERSION(9, 4, 1):
> +		if (is_vram) {
> +			if (is_local) {
> +				mapping_flags |= coherent ?
> +					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
> +			} else {
> +				mapping_flags |= coherent ?
> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +				if (same_hive)
> +					snoop = true;
> +			}
> +		} else {
> +			mapping_flags |= coherent ?
> +				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +		}
> +		break;
> +	case IP_VERSION(9, 4, 2):
> +		if (is_vram) {
> +			if (is_local) {
> +				mapping_flags |= coherent ?
> +					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
> +				if (adev->gmc.xgmi.connected_to_cpu)
> +					snoop = true;
> +			} else {
> +				mapping_flags |= coherent ?
> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +				if (same_hive)
> +					snoop = true;
> +			}
> +		} else {
> +			mapping_flags |= coherent ?
> +				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +		}
> +		break;
> +	case IP_VERSION(9, 4, 3):
> +	case IP_VERSION(9, 4, 4):
> +	case IP_VERSION(9, 5, 0):
> +		if (ext_coherent)
> +			mtype_local = AMDGPU_VM_MTYPE_CC;
> +		else
> +			mtype_local = amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
> +				amdgpu_mtype_local == 2 ? AMDGPU_VM_MTYPE_CC :
> +				AMDGPU_VM_MTYPE_RW;
> +		snoop = true;
> +		if (is_vram) {
> +			/* local HBM region close to partition */
> +			if (is_local)
> +				mapping_flags |= mtype_local;
> +			/* local HBM region far from partition or remote XGMI GPU
> +			 * with regular system scope coherence
> +			 */
> +			else if (same_hive && !ext_coherent)
> +				mapping_flags |= AMDGPU_VM_MTYPE_NC;
> +			/* PCIe P2P on GPUs pre-9.5.0 */
> +			else if (gc_ip_version < IP_VERSION(9, 5, 0) && !same_hive)
> +				mapping_flags |= AMDGPU_VM_MTYPE_UC;
> +			/* Other remote memory */
> +			else
> +				mapping_flags |= ext_coherent ?
> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +		} else if (adev->flags & AMD_IS_APU) {
> +			/* On NUMA systems, locality is determined per-page
> +			 * in gmc_v9_0_override_vm_pte_flags.
> +			 */
> +			if (num_possible_nodes() <= 1)
> +				mapping_flags |= mtype_local;
> +			else
> +				mapping_flags |= ext_coherent ?
> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +		} else {
> +			if (gc_ip_version < IP_VERSION(9, 5, 0) || ext_coherent)
> +				mapping_flags |= AMDGPU_VM_MTYPE_UC;
> +			else
> +				mapping_flags |= AMDGPU_VM_MTYPE_NC;
> +		}
> +		break;
> +	default:
> +		mapping_flags |= coherent ?
> +			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +		break;
> +	}
> +
> +	if (params->gpu_exec)
> +		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
> +
> +	pte_flags = AMDGPU_PTE_VALID;
> +	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
> +	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
> +
> +	gmc_v9_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
> +	pte_flags |= AMDGPU_PTE_READABLE;
> +	if (!params->gpu_ro)
> +		pte_flags |= AMDGPU_PTE_WRITEABLE;
> +
> +	return pte_flags;
> +}
> +
>  static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev,
>  					   struct amdgpu_vm *vm,
>  					   uint64_t addr, uint64_t *flags)
> @@ -1316,6 +1434,7 @@ static const struct amdgpu_gmc_funcs gmc_v9_0_gmc_funcs = {
>  	.emit_pasid_mapping = gmc_v9_0_emit_pasid_mapping,
>  	.get_vm_pde = gmc_v9_0_get_vm_pde,
>  	.get_vm_pte = gmc_v9_0_get_vm_pte,
> +	.get_svm_pte_flags = gmc_v9_0_get_svm_pte_flags,
>  	.override_vm_pte_flags = gmc_v9_0_override_vm_pte_flags,
>  	.get_vbios_fb_size = gmc_v9_0_get_vbios_fb_size,
>  	.query_mem_partition_mode = &amdgpu_gmc_query_memory_partition,

Re: [PATCH v10 07/22] drm/amdgpu: implement SVM range GPU mapping core

[Christian König]

On 10/21/26 04:58, Huang Rui wrote:
> From: Honglei Huang <honghuan@amd.com>
> 
> Implement GPU page table mapping in amdgpu_svm_range.c:
> - amdgpu_svm_range_pages_valid: check notifier sequence validity
> - amdgpu_svm_range_is_valid: validate gpu_mapped and attr_flags
>   against the requested attribute set, under MMU notifier lock.
> - amdgpu_svm_range_zap_ptes: clear GPU PTEs via amdgpu_vm_update_range
> - amdgpu_svm_range_attr_pte_flags: compute PTE flags per GC IP version
> - amdgpu_svm_range_lock_vm_pd: acquire VM page directory by drm_exec
> - amdgpu_svm_range_update_gpu_range: DMA segment coalescing; recomputes
>   PTE flags per segment from entry->proto via attr_pte_flags, then
>   programs PTEs under notifier lock
> - amdgpu_svm_range_find_or_insert: wrapper of drm gpu SVM find or insert
>   API.
> - amdgpu_svm_range_get_pages: wrapper of drm gpu SVM get pages.
> - amdgpu_svm_range_evict: thin wrapper of drm_gpusvm_range_evict
> - amdgpu_svm_range_update_mapping: full pipeline (lock PD, validate
>   pages, program PTEs, update PDEs, flush TLB, record attr_flags
>   and mapped state); takes attrs + read_only and forwards them to
>   update_gpu_range
> - amdgpu_svm_range_map_attrs: cursor based iteration across attr tree,
>   creating/mapping ranges
> 
> - Add to amdgpu_svm.h driver private interconnect tags used in
> drm_pagemap_addr.proto:
>   AMDGPU_INTERCONNECT_VRAM = DRM_INTERCONNECT_DRIVER
>   AMDGPU_INTERCONNECT_P2P  = AMDGPU_INTERCONNECT_VRAM + 1
> 
> Suggested-by: Christian König <christian.koenig@amd.com>
> Suggested-by: Huang Rui <ray.huang@amd.com>
> Signed-off-by: Honglei Huang <honghuan@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       |   4 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 417 ++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  40 ++
>  3 files changed, 461 insertions(+)
>  create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> index 72f954d2df39e..cfdc4e6d045b7 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
> @@ -27,6 +27,7 @@
>  
>  #include <drm/amdgpu_drm.h>
>  #include <drm/drm_gpusvm.h>
> +#include <drm/drm_pagemap.h>
>  #include <linux/atomic.h>
>  #include <linux/kref.h>
>  #include <linux/list.h>
> @@ -44,6 +45,9 @@ struct amdgpu_svm_attrs;
>  struct drm_device;
>  struct drm_file;
>  
> +#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
> +#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
> +
>  enum amdgpu_svm_xnack_mode {
>  	AMDGPU_SVM_XNACK_OFF,
>  	AMDGPU_SVM_XNACK_ON,
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
> new file mode 100644
> index 0000000000000..d4830e53f3d8e
> --- /dev/null
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
> @@ -0,0 +1,417 @@
> +// SPDX-License-Identifier: GPL-2.0 OR MIT
> +/*
> + * Copyright 2026 Advanced Micro Devices, Inc.
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a
> + * copy of this software and associated documentation files (the "Software"),
> + * to deal in the Software without restriction, including without limitation
> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
> + * and/or sell copies of the Software, and to permit persons to whom the
> + * Software is furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
> + * OTHER DEALINGS IN THE SOFTWARE.
> + *
> + */
> +
> +#include "amdgpu_svm.h"
> +#include "amdgpu_svm_attr.h"
> +#include "amdgpu_svm_range.h"
> +#include "amdgpu_svm_fault.h"
> +#include "amdgpu.h"
> +#include "amdgpu_vm.h"
> +#include "amdgpu_trace.h"
> +
> +#include <drm/drm_exec.h>
> +#include <drm/drm_pagemap.h>
> +
> +#include <linux/mmu_notifier.h>
> +#include <uapi/linux/kfd_ioctl.h>
> +
> +bool
> +amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
> +		  struct amdgpu_svm_range *range)
> +{
> +	struct drm_gpusvm_range *base = &range->base;
> +
> +	lockdep_assert_held(&svm->gpusvm.notifier_lock);
> +
> +	if (base->pages.flags.unmapped || base->pages.flags.partial_unmap)
> +		return false;
> +
> +	return drm_gpusvm_range_pages_valid(&svm->gpusvm, base);
> +}
> +
> +bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
> +			       const struct amdgpu_svm_attrs *attrs)
> +{
> +	return READ_ONCE(range->gpu_mapped) &&
> +	       READ_ONCE(range->attr_flags) == attrs->flags;
> +}
> +
> +
> +int
> +amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
> +			  struct amdgpu_svm_range *range,
> +			  unsigned long start_page,
> +			  unsigned long last_page)
> +{
> +	struct dma_fence *fence = NULL;
> +	unsigned int flags;
> +	int ret;
> +
> +	if (last_page < start_page)
> +		return 0;
> +
> +	flags = memalloc_noreclaim_save();

Completely drop that, this needs to be handled by the VM code and not here.

> +	ret = amdgpu_vm_unmap_range(svm->adev, svm->vm, NULL,
> +				    start_page, last_page, 0, &fence);
> +	memalloc_noreclaim_restore(flags);
> +
> +	if (fence) {
> +		dma_fence_wait(fence, false);
> +		dma_fence_put(fence);
> +	}
> +
> +	return ret;
> +}
> +
> +uint64_t
> +amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
> +			    const struct amdgpu_svm_attrs *attrs,
> +			    bool read_only,
> +			    enum drm_interconnect_protocol proto)
> +{
> +	struct amdgpu_device *adev = svm->adev;
> +	uint32_t flags = attrs->flags;
> +	bool is_local = (proto == AMDGPU_INTERCONNECT_VRAM);
> +	struct amdgpu_svm_pte_flags_params params = {
> +		.coherent = flags & (AMDGPU_SVM_ATTR_BIT_COHERENT |
> +				     AMDGPU_SVM_ATTR_BIT_EXT_COHERENT),
> +		.ext_coherent = flags & AMDGPU_SVM_ATTR_BIT_EXT_COHERENT,
> +		.gpu_ro = flags & AMDGPU_SVM_ATTR_BIT_GPU_RO,
> +		.gpu_exec = flags & AMDGPU_SVM_ATTR_BIT_GPU_EXEC,
> +		.is_local = is_local,
> +		.is_vram = is_local || (proto == AMDGPU_INTERCONNECT_P2P),
> +		/* TODO: add hive check. */
> +		.same_hive = false,
> +	};
> +	uint32_t gc_ip_version;
> +	uint32_t mapping_flags = 0;
> +	uint64_t pte_flags;
> +	bool snoop;
> +
> +	/*
> +	 * The per-IP MTYPE / PTE flag selection lives in the HW IP specific
> +	 * gmc_vX_0.c files mirroring amdgpu_gmc_get_vm_pte for BO mappings.
> +	 */
> +	if (adev->gmc.gmc_funcs->get_svm_pte_flags) {
> +		pte_flags = amdgpu_gmc_get_svm_pte_flags(adevamdgpu_gmc_get_vm_pte, svm->vm, &params);
> +	} else {
> +		gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
> +		snoop = !is_local;
> +		mapping_flags |= params.coherent ?
> +			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
> +
> +		if (params.gpu_exec)
> +			mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
> +
> +		pte_flags = AMDGPU_PTE_VALID;
> +		pte_flags |= is_local ? 0 : AMDGPU_PTE_SYSTEM;
> +		pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
> +		if (gc_ip_version >= IP_VERSION(12, 0, 0))
> +			pte_flags |= AMDGPU_PTE_IS_PTE;
> +
> +		amdgpu_gmc_get_vm_pte(adev, svm->vm, NULL, mapping_flags,
> +				      &pte_flags);
> +		pte_flags |= AMDGPU_PTE_READABLE;
> +		if (!params.gpu_ro)
> +			pte_flags |= AMDGPU_PTE_WRITEABLE;

Don't touch the flags after calling amdgpu_gmc_get_vm_pte().

> +	}
> +
> +	if (read_only)
> +		pte_flags &= ~AMDGPU_PTE_WRITEABLE;

Same here.

> +
> +	return pte_flags;
> +}
> +
> +
> +
> +int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm, struct drm_exec *exec,
> +				bool intr)
> +{
> +	unsigned int exec_flags = DRM_EXEC_IGNORE_DUPLICATES;
> +	int ret;
> +
> +	if (intr)
> +		exec_flags |= DRM_EXEC_INTERRUPTIBLE_WAIT;
> +
> +	drm_exec_init(exec, exec_flags, 0);
> +	drm_exec_until_all_locked(exec) {
> +		ret = amdgpu_vm_lock_pd(svm->vm, exec, 1);
> +		drm_exec_retry_on_contention(exec);
> +		if (ret) {
> +			drm_exec_fini(exec);
> +			return ret;
> +		}
> +	}
> +
> +	return 0;
> +}

Please drop that helper and always use the drm_exec loop directly.

Regards,
Christian.

> +
> +int
> +amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
> +				  struct amdgpu_svm_range *range,
> +				  const struct amdgpu_svm_attrs *attrs,
> +				  bool read_only,
> +				  bool flush_tlb,
> +				  bool wait_fence,
> +				  struct dma_fence **fence)
> +{
> +	struct drm_gpusvm_range *base = &range->base;
> +
> +	lockdep_assert_held(&svm->gpusvm.notifier_lock);
> +
> +	const unsigned long range_start_page = drm_gpusvm_range_start(base) >> PAGE_SHIFT;
> +	const unsigned long range_end_page = drm_gpusvm_range_end(base) >> PAGE_SHIFT;
> +	const unsigned long npages = range_end_page - range_start_page;
> +	unsigned long mapped_pages = 0;
> +	unsigned long dma_idx = 0;
> +	int ret;
> +
> +	if (!base->pages.dma_addr || !npages)
> +		return -EINVAL;
> +
> +	while (mapped_pages < npages) {
> +		const struct drm_pagemap_addr *entry = &base->pages.dma_addr[dma_idx++];
> +		unsigned long seg_pages = min_t(unsigned long, 1UL << entry->order,
> +						npages - mapped_pages);
> +		uint64_t pte_flags;
> +		unsigned long start_page, last_page;
> +		bool is_last_seg;
> +
> +		if (entry->proto != DRM_INTERCONNECT_SYSTEM)
> +			return -EOPNOTSUPP;
> +
> +		pte_flags = amdgpu_svm_range_attr_pte_flags(svm, attrs,
> +							    read_only,
> +							    entry->proto);
> +
> +		start_page = range_start_page + mapped_pages;
> +		last_page = start_page + seg_pages - 1;
> +		mapped_pages += seg_pages;
> +		is_last_seg = mapped_pages == npages;
> +
> +		/*
> +		 * The struct hmm_range hmm_range field inside amdgpu_hmm_range
> +		 * is drm_gpusvm internal state. It is not exposed to the
> +		 * implementing driver and is managed entirely by the framework
> +		 * as part of the framework logic. This amdgpu_vm_map_range call
> +		 * always runs inside the drm_gpusvm notifier, so omitting
> +		 * hmm_range here is safe.
> +		 */
> +		ret = amdgpu_vm_map_range(svm->adev, svm->vm,
> +					  flush_tlb && is_last_seg, true, NULL,
> +					  start_page, last_page, pte_flags,
> +					  0, entry->addr, NULL, NULL, NULL,
> +					  wait_fence && is_last_seg ? fence : NULL);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +struct amdgpu_svm_range *
> +amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
> +				unsigned long gpuva_start, unsigned long gpuva_end,
> +				struct drm_gpusvm_ctx *ctx)
> +{
> +	struct drm_gpusvm_range *r;
> +
> +retry:
> +	r = drm_gpusvm_range_find_or_insert(&svm->gpusvm, addr,
> +					    gpuva_start, gpuva_end, ctx);
> +	/*
> +	 * UMD doesn't set RO for some RO VMAs, but the drm gpu svm framework
> +	 * denies no RO flag range insert for RO VMAs, so treat
> +	 * -EPERM as an indication of RO and retry if not set.
> +	 */
> +	if (PTR_ERR_OR_ZERO(r) == -EPERM && !ctx->read_only) {
> +		ctx->read_only = true;
> +		goto retry;
> +	}
> +
> +	if (IS_ERR(r))
> +		return ERR_CAST(r);
> +
> +	return to_amdgpu_svm_range(r);
> +}
> +
> +int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
> +			       struct drm_gpusvm_range *range,
> +			       struct drm_gpusvm_ctx *ctx)
> +{
> +	int ret;
> +
> +retry:
> +	ret = drm_gpusvm_range_get_pages(&svm->gpusvm, range, ctx);
> +	/*
> +	 * HMM returns -EPERM when write access is requested for a read-only
> +	 * VMA. Retry as read-only so the eventual GPU mapping follows the CPU
> +	 * VMA permissions.
> +	 */
> +	if (ret == -EPERM && !ctx->read_only) {
> +		ctx->read_only = true;
> +		goto retry;
> +	}
> +
> +	if (ret == -EOPNOTSUPP) {
> +		pr_err_ratelimited("range get pages -EOPNOTSUPP, evict and retry: gpuva=[0x%lx-0x%lx) ret=%d\n",
> +				   drm_gpusvm_range_start(range),
> +				   drm_gpusvm_range_end(range), ret);
> +		amdgpu_svm_range_evict(svm, range);
> +	}
> +
> +	return ret;
> +}
> +
> +void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
> +			    struct drm_gpusvm_range *range)
> +{
> +	if (!range->pages.flags.has_devmem_pages)
> +		return;
> +
> +	drm_gpusvm_range_evict(&svm->gpusvm, range);
> +}
> +
> +int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
> +				    struct amdgpu_svm_range *range,
> +				    const struct amdgpu_svm_attrs *attrs,
> +				    bool read_only,
> +				    bool intr, bool wait,
> +				    bool flush_tlb)
> +{
> +	struct drm_exec exec;
> +	struct dma_fence *fence = NULL;
> +	unsigned int flags;
> +	int ret;
> +
> +	ret = amdgpu_svm_range_lock_vm_pd(svm, &exec, intr);
> +	if (ret)
> +		return ret;
> +
> +	flags = memalloc_noreclaim_save();
> +	drm_gpusvm_notifier_lock(&svm->gpusvm);
> +
> +	if (!amdgpu_svm_range_pages_valid(svm, range)) {
> +		amdgpu_svm_range_invalidate_gpu_mapping(range);
> +		ret = -EAGAIN;
> +	} else {
> +		ret = amdgpu_svm_range_update_gpu_range(svm, range, attrs,
> +							read_only, flush_tlb,
> +							wait, wait ? &fence : NULL);
> +	}
> +
> +	drm_gpusvm_notifier_unlock(&svm->gpusvm);
> +	memalloc_noreclaim_restore(flags);
> +
> +	if (!ret && fence)
> +		dma_fence_wait(fence, intr);
> +	dma_fence_put(fence);
> +
> +	if (!ret)
> +		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, false);
> +
> +	if (!ret) {
> +		if (flush_tlb)
> +			amdgpu_svm_flush_tlb(svm);
> +		WRITE_ONCE(range->attr_flags, attrs->flags);
> +		WRITE_ONCE(range->gpu_mapped, true);
> +		range->validate_timestamp = ktime_get_boottime();
> +	}
> +
> +	drm_exec_fini(&exec);
> +	return ret;
> +}
> +
> +int
> +amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
> +		       const struct amdgpu_svm_attrs *attrs,
> +		       unsigned long start, unsigned long end)
> +{
> +	unsigned long addr = start;
> +	int ret;
> +	bool devmem_possible = amdgpu_svm_devmem_possible(svm);
> +	bool need_vram_migration = devmem_possible &&
> +				   amdgpu_svm_attr_prefer_vram(attrs);
> +	struct drm_gpusvm_ctx map_ctx = {
> +		.devmem_possible = devmem_possible,
> +		.devmem_only = need_vram_migration,
> +		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
> +	};
> +
> +	while (addr < end) {
> +		struct amdgpu_svm_range *range;
> +		unsigned long next_addr;
> +		/* reset read_only every iteration, amdgpu_svm_range_find_or_insert may change it */
> +		map_ctx.read_only = !!(attrs->flags & AMDGPU_SVM_ATTR_BIT_GPU_RO);
> +
> +		range = amdgpu_svm_range_find_or_insert(svm, addr,
> +							addr, end,
> +							&map_ctx);
> +		if (IS_ERR(range)) {
> +			pr_err_ratelimited("failed to find/insert range for gpuva 0x%lx [0x%lx-0x%lx), ret=%ld\n",
> +					   addr, start, end, PTR_ERR(range));
> +			return PTR_ERR(range);
> +		}
> +
> +		next_addr = drm_gpusvm_range_end(&range->base);
> +		if (next_addr <= addr)
> +			return -EINVAL;
> +
> +		if (amdgpu_svm_range_is_valid(range, attrs)) {
> +			addr = next_addr;
> +			continue;
> +		}
> +
> +		/* TODO: add migration */
> +
> +		AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
> +
> +		ret = amdgpu_svm_range_get_pages(svm, &range->base,
> +						 &map_ctx);
> +		if (ret) {
> +			pr_err_ratelimited("failed to get pages for range [0x%lx-0x%lx), ret=%d\n",
> +					   drm_gpusvm_range_start(&range->base),
> +					   drm_gpusvm_range_end(&range->base), ret);
> +			return ret;
> +		}
> +
> +		AMDGPU_SVM_RANGE_TRACE(range, "UPDATE MAPPING");
> +
> +		ret = amdgpu_svm_range_update_mapping(svm, range, attrs,
> +						      map_ctx.read_only,
> +						      true, true,
> +						      true);
> +		if (ret) {
> +			pr_err_ratelimited("failed to update gpu map for range [0x%lx-0x%lx), ret=%d\n",
> +					   drm_gpusvm_range_start(&range->base),
> +					   drm_gpusvm_range_end(&range->base), ret);
> +			return ret;
> +		}
> +
> +		addr = next_addr;
> +	}
> +
> +	return 0;
> +}
> +
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
> index 39db3a18b5f2f..e5a5f52166576 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
> @@ -126,4 +126,44 @@ struct amdgpu_svm_range_op_ctx {
>  	uint8_t pending_ops;
>  };
>  
> +uint64_t amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
> +					 const struct amdgpu_svm_attrs *attrs,
> +					 bool read_only,
> +					 enum drm_interconnect_protocol proto);
> +int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm,
> +				struct drm_exec *exec,
> +				bool intr);
> +bool amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
> +				  struct amdgpu_svm_range *range);
> +bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
> +			       const struct amdgpu_svm_attrs *attrs);
> +int amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
> +				      struct amdgpu_svm_range *range,
> +				      const struct amdgpu_svm_attrs *attrs,
> +				      bool read_only,
> +				      bool flush_tlb, bool wait,
> +				      struct dma_fence **fence);
> +int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
> +				    struct amdgpu_svm_range *range,
> +				    const struct amdgpu_svm_attrs *attrs,
> +				    bool read_only,
> +				    bool intr, bool wait,
> +				    bool flush_tlb);
> +int amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
> +			       const struct amdgpu_svm_attrs *attrs,
> +			       unsigned long start, unsigned long end);
> +int amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
> +			      struct amdgpu_svm_range *range,
> +			      unsigned long start_page,
> +			      unsigned long last_page);
> +void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
> +			    struct drm_gpusvm_range *range);
> +struct amdgpu_svm_range *
> +amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
> +				unsigned long gpuva_start, unsigned long gpuva_end,
> +				struct drm_gpusvm_ctx *ctx);
> +int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
> +			       struct drm_gpusvm_range *range,
> +			       struct drm_gpusvm_ctx *ctx);
> +
>  #endif /* __AMDGPU_SVM_RANGE_H__ */

Re: [PATCH v10 11/22] drm/amdgpu: add SVM ioctl entry and fault handler module

[Aveline Noir]

Hi Ray,

On Wed, 21 Oct 2026 10:58:50 +0800 Huang Rui wrote:

> +	ret = fault_check_allowed(svm, fault_addr, write_fault);
> +	if (ret)
> +		return ret;

This checks the VMA permissions but not attrs->access, so I think
INACCESSIBLE isn't really enforced. SET_ATTR invalidates the existing
GPU mapping, but the next retry fault just maps the range again.

IN_PLACE has a similar problem. With preferred_loc=LOCAL, the
migration added by 22/22 can still move migratable system-memory pages
into VRAM on a pagemap-capable GPU. Both seem to go against the access
definitions in 01/22, and the prefetch path doesn't check access
either.

Could the fault and prefetch paths check access before mapping or
migrating?

Thanks,
Aveline Noir

Re: [PATCH v10 03/22] drm/amdgpu: implement SVM attribute tree and helper functions

[Aveline Noir]

Hi Christian,

On Mon, 09 Nov 2026 13:41:48 +0100 Christian König wrote:

> I think this whole block of functions can be static, but not 100% sure.
>
> If they are used outside of this C file please add kerneldoc for them.

Several of them are used by other SVM files, so maybe those can't be static.
amdgpu_svm_attr_find_locked() seems to be the exception, since I only
see it used in this file. And yes, they should have kerneldoc, ideally
with the locking requirements.

> Writing that I added Aveline on CC, it would be really cool if you
> could take a look as well.

Thanks for adding :) I have some comments on 08/22, 10/22 and 11/22
and will reply on those patches. Sadly I don't have a setup to test
this on a GPU, so any testing would be very welcome :)

Thanks,
Aveline Noir

Re: [PATCH v10 10/22] drm/amdgpu: implement SVM initialization and lifecycle

[Aveline Noir]

Hi Ray,

On Wed, 21 Oct 2026 10:58:49 +0800 Huang Rui wrote:

> +	if (new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
> +	    new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_SYSMEM)
> +		trigger |= AMDGPU_SVM_ATTR_TRIGGER_PREFETCH;

I'm not sure about excluding SYSMEM here... The proposed libhsakmt
adaptation maps a CPU prefetch target to SYSMEM. If the pages are in
VRAM and the update fully covers an existing mapped SVM range, nothing
gets mapped or migrated eagerly. The GPU PTEs can be zapped, but the
pages stay in VRAM even though SET_ATTR can return success.

Is SYSMEM prefetch meant to move pages back to RAM, or only to set the
attribute for a later CPU fault?

Thanks,
Aveline Noir

Re: [PATCH v10 08/22] drm/amdgpu: implement SVM range notifier and GC helpers

[Aveline Noir]

Hi Ray,

On Wed, 21 Oct 2026 10:58:47 +0800 Huang Rui wrote:

> + * Invalidate the GPU view of @range over the notifier window: zap the
> + * affected PTEs and clear the gpu_mapped flag. The range is not removed
> + * here; structural removal if needed happens in the event-end phase.

I think this can leave stale PTEs behind when only part of a range is
invalidated. Take a one-page notifier event on a larger system-memory
range mapped with 4K PTEs. This zaps that one PTE but clears gpu_mapped
for the whole range, and then the event-end callback unmaps DMA for the
whole range. The other PTEs are still valid at that point, but their DMA
mappings are gone. A later notifier won't zap them either, because
gpu_mapped is already false.

Maybe the zap should cover the full range when its DMA mapping is
going away?

> +			if (crosses_boundary) {

Another case is GPU_RO going from 1 to 0 over a whole mapped
GPUSVM range. This branch isn't taken, so the PTEs are zapped but the
old DMA_TO_DEVICE mapping stays around. The next get_pages() sees valid
pages and reuses it, so the writable GPU PTE goes in without an
HMM_PFN_REQ_WRITE fault or a DMA_BIDIRECTIONAL remap. If it's a VM_WRITE
MAP_PRIVATE mapping whose CPU PTE still needs COW, I think it can miss
the write fault. So could the old DMA mapping be dropped before the
writable PTE goes in?

I couldn't tell from the cover which drm_gpusvm base this is on, so I
used 671b7825dbfe for both of these. I haven't reproduced either one on
a GPU, though.

Thanks,
Aveline Noir

Re: [PATCH v10 02/22] drm/amdgpu: add SVM core header and VM integration

[Huang, Honglei]

On 11/9/2026 8:33 PM, Christian König wrote:
> On 10/21/26 04:58, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Add amdgpu_svm.h with SVM data structures and public API:
>> - enum amdgpu_svm_xnack_mode: OFF, ON and DEFAULT mode
>> - struct amdgpu_svm: core context with drm_gpusvm, kref lifecycle,
>>    attribute tree, rw_semaphore, GC workqueue, xnack state.
>> - struct amdgpu_svm_gc: garbage collector with workqueue and work_struct
>> - Kmem cache helpers: AMDGPU_SVM_KMEM_CACHE_CREATE/DESTROY
>> - Locking helpers: amdgpu_svm_lock/unlock/assert_locked
>> - Public API declarations (including clean_queue, sync_work)
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Huang Rui <ray.huang@amd.com>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 195 ++++++++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |   4 +
>>   2 files changed, 199 insertions(+)
>>   create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> new file mode 100644
>> index 0000000000000..72f954d2df39e
>> --- /dev/null
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> @@ -0,0 +1,195 @@
>> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
>> +/*
>> + * Copyright 2026 Advanced Micro Devices, Inc.
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice shall be included in
>> + * all copies or substantial portions of the Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
>> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
>> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>> + * OTHER DEALINGS IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#ifndef __AMDGPU_SVM_H__
>> +#define __AMDGPU_SVM_H__
>> +
>> +#include <drm/amdgpu_drm.h>
>> +#include <drm/drm_gpusvm.h>
>> +#include <linux/atomic.h>
>> +#include <linux/kref.h>
>> +#include <linux/list.h>
>> +#include <linux/printk.h>
>> +#include <linux/rwsem.h>
>> +#include <linux/slab.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/types.h>
>> +#include <linux/workqueue.h>
>> +
>> +struct amdgpu_device;
>> +struct amdgpu_vm;
>> +struct amdgpu_svm_attr_tree;
>> +struct amdgpu_svm_attrs;
>> +struct drm_device;
>> +struct drm_file;
>> +
>> +enum amdgpu_svm_xnack_mode {
>> +	AMDGPU_SVM_XNACK_OFF,
>> +	AMDGPU_SVM_XNACK_ON,
>> +	AMDGPU_SVM_XNACK_DEFAULT,
>> +};
>> +
>> +#define amdgpu_svm_assert_in_notifier(svm__) \
>> +	lockdep_assert_held_write(&(svm__)->gpusvm.notifier_lock)
>> +
>> +/**
>> + * struct amdgpu_svm_gc - SVM range garbage collector used for unmapping
>> + * SVM ranges in response to mmu_notifier events.
>> + * @wq: Workqueue where @work runs on.
>> + * @list: Ranges list in garbage collector. Protected by amdgpu_svm.work_lock.
>> + * @work: Drains @list under amdgpu_svm.svm_lock.
>> + */
>> +struct amdgpu_svm_gc {
>> +	struct workqueue_struct *wq;
>> +	struct list_head list;
>> +	struct work_struct work;
>> +};
>> +
>> +/**
>> + * struct amdgpu_svm - shared virtual memory context
>> + * @gpusvm: Embedded drm_gpusvm base, manages the range/notifier trees.
>> + * @refcount: Reference count guarding the lifetime of this instance.
>> + * @adev: The amdgpu device this SVM context belongs to.
>> + * @vm: Back pointer to the owning amdgpu_vm.
>> + * @attr_tree: Interval tree of SVM attributes.
>> + * @svm_lock: The driver_svm_lock described in the GPU SVM locking section.
>> + *            registered by drm_gpusvm_driver_set_lock().
>> + * @work_lock: Protects the deferred work queues such as @gc.list.
>> + * @gc: garbage collector for ranges queued for destruction.
>> + * @exiting: Set once when the context starts tearing down.
>> + * @checkpoint_ts: IH ring timestamp captured on unmap, used to drop stale
>> + *                 faults.
>> + * @default_granularity: Default range granularity.
>> + * @xnack_enabled: Whether retry (XNACK) faults are enabled.
>> + * @invalidate_ranges: Callback of the MMU notifier invalidate event.
>> + */
>> +struct amdgpu_svm {
>> +	struct drm_gpusvm gpusvm;
>> +	struct kref refcount;
>> +	struct amdgpu_device *adev;
>> +	struct amdgpu_vm *vm;
>> +	struct amdgpu_svm_attr_tree *attr_tree;
>> +	struct rw_semaphore svm_lock;
>> +	spinlock_t work_lock;
>> +	struct amdgpu_svm_gc gc;
>> +	atomic_t exiting;
> 
>> +	uint64_t checkpoint_ts;
> 
> By now we have figured out that TS in faults are not useable for fault handling.
> 
> Background is that they reset from time to time because of GFXOFF and may vary depending on the engine who sends the fault.
> 
> Adding Timur from Valve on CC, he send out a patch to completely remove the existing timestamp handling from the fault filter.
> 
> We need to come up with a different approach here.

Got it, is that valid for you to remove the checkpoint_ts check in this 
series?
I check the series of Timur, didn't find speific new approach, can you 
please help to point out the new approach.

> 
>> +	u8 default_granularity;
> 
>> +	bool xnack_enabled;
> 
> It's probably ok to have that here for now, but that will become a VM property in the future I think.

Will try to move this into VM.

> 
>> +	void (*invalidate_ranges)(struct amdgpu_svm *svm,
>> +				  struct drm_gpusvm_notifier *notifier,
>> +				  const struct mmu_notifier_range *mmu_range,
>> +				  struct drm_gpusvm_range *first,
>> +				  uint64_t adj_start, uint64_t adj_end);
> 
> That looks a bit questionable. Why is that a pointer inside the amdgpu_svm structure?
> 
> Apart from those questions that looks rather reasonable.

The invalidate callback is for both xnack-on and xnack-off mode, the 
logic of those two modes are different. Differnet callback is set in
different mode.
How about remove this call back then put two modes logic in a fixed 
function?

Regards,
Honglei

> 
> Regards,
> Christian.
> 
>> +};
>> +
>> +static inline struct amdgpu_svm *to_amdgpu_svm(struct drm_gpusvm *gpusvm)
>> +{
>> +	return container_of(gpusvm, struct amdgpu_svm, gpusvm);
>> +}
>> +
>> +/*
>> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
>> + * Hold it in write mode around structural GPU SVM updates, including
>> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
>> + */
>> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
>> +{
>> +	down_write(&svm->svm_lock);
>> +}
>> +
>> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
>> +{
>> +	up_write(&svm->svm_lock);
>> +}
>> +
>> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
>> +{
>> +	lockdep_assert_held_write(&svm->svm_lock);
>> +}
>> +
>> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
>> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
>> +
>> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>> +void amdgpu_svm_close(struct amdgpu_vm *vm);
>> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
>> +
>> +void amdgpu_svm_put(struct amdgpu_svm *svm);
>> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
>> +					       uint32_t pasid);
>> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
>> +			    uint64_t fault_page, uint64_t ts,
>> +			    bool write_fault);
>> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
>> +
>> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>> +			 struct drm_file *filp);
>> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
>> +			    struct list_head *work_list);
>> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
>> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
>> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
>> +				 const struct amdgpu_svm_attrs *old_attrs,
>> +				 const struct amdgpu_svm_attrs *new_attrs,
>> +				 unsigned long start_page,
>> +				 unsigned long last_page);
>> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
>> +#else
>> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
>> +				  struct amdgpu_vm *vm)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
>> +{
>> +}
>> +
>> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
>> +{
>> +}
>> +
>> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
>> +					  uint32_t pasid,
>> +					  uint64_t fault_page,
>> +					  uint64_t ts,
>> +					  bool write_fault)
>> +{
>> +	return -EOPNOTSUPP;
>> +}
>> +
>> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
>> +{
>> +	return false;
>> +}
>> +
>> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>> +				       struct drm_file *filp)
>> +{
>> +	return -EOPNOTSUPP;
>> +}
>> +#endif /* CONFIG_DRM_AMDGPU_SVM */
>> +
>> +#endif /* __AMDGPU_SVM_H__ */
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> index 9b07df5a2b816..b435e393be154 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>>   struct amdgpu_job;
>>   struct amdgpu_bo_list_entry;
>>   struct amdgpu_bo_vm;
>> +struct amdgpu_svm;
>>   
>>   /*
>>    * GPUVM handling
>> @@ -381,6 +382,9 @@ struct amdgpu_vm {
>>   
>>   	/* cached fault info */
>>   	struct amdgpu_vm_fault_info fault_info;
>> +
>> +	/* SVM experimental implementation */
>> +	struct amdgpu_svm *svm;
>>   };
>>   
>>   struct amdgpu_vm_manager {
> 

Re: [PATCH v10 03/22] drm/amdgpu: implement SVM attribute tree and helper functions

[Huang, Honglei]

On 11/10/2026 12:54 AM, Aveline Noir wrote:
> Hi Christian,
> 
> On Mon, 09 Nov 2026 13:41:48 +0100 Christian König wrote:
> 
>> I think this whole block of functions can be static, but not 100% sure.
>>
>> If they are used outside of this C file please add kerneldoc for them.
> 
> Several of them are used by other SVM files, so maybe those can't be static.
> amdgpu_svm_attr_find_locked() seems to be the exception, since I only
> see it used in this file. And yes, they should have kerneldoc, ideally
> with the locking requirements.

Will change amdgpu_svm_attr_find_locked to static, and will add kernel doc.

> 
>> Writing that I added Aveline on CC, it would be really cool if you
>> could take a look as well.
> 
> Thanks for adding :) I have some comments on 08/22, 10/22 and 11/22
> and will reply on those patches. Sadly I don't have a setup to test
> this on a GPU, so any testing would be very welcome :)

Thanks for the review, will address them in next version.

Regards,
Honglei

> 
> Thanks,
> Aveline Noir

Re: [PATCH v10 04/22] drm/amdgpu: implement SVM attribute set/get/clear operations

[Huang, Honglei]

On 11/9/2026 8:52 PM, Christian König wrote:
> On 10/21/26 04:58, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Complete attribute subsystem with validation, tree modification, and
>> public API:
>> - Attribute validation: per-type checks for preferred_loc, prefetch_loc,
>>    access, flags, granularity
>> - Flag application: attr_apply_flag, amdgpu_svm_attr_apply
>> - VMA range validation: amdgpu_svm_attr_validate_range_vma
>> - VM BO overlap detection: amdgpu_svm_attr_check_vm_bo
>> - Tree modification with gap/overlap handling:
>>    amdgpu_svm_attr_set_hole (new range in gap),
>>    amdgpu_svm_attr_set_existing (split/update existing range)
>> - amdgpu_svm_attr_set_range: cursor based iteration with per segment
>> - Lifecycle: amdgpu_svm_attr_tree_create/destroy
>> - Public API: amdgpu_svm_attr_set (validate, set_range with retry),
>>    amdgpu_svm_attr_get (aggregate attrs across interval),
>>    amdgpu_svm_attr_clear (interval removal with split),
>>    amdgpu_svm_attr_reset
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Huang Rui <ray.huang@amd.com>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 736 +++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h |  20 +
>>   2 files changed, 756 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
>> index a5afa75421fbd..c76e10473968e 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
>> @@ -78,6 +78,43 @@ struct vm_area_struct *amdgpu_svm_check_vma(struct mm_struct *mm,
>>   	return vma;
>>   }
>>   
>> +/*
>> + * amdgpu_svm_attr_range_is_valid - validate an attr ioctl address range
>> + *
>> + * Synchronous input validation for the SVM set/get-attr ioctls: walk the
>> + * CPU VMAs covering [start_page, last_page] and reject the request if any
>> + * part is unmapped or backed by an unsupported VMA (-EOPNOTSUPP).
>> + *
>> + */
>> +static int amdgpu_svm_attr_range_is_valid(struct amdgpu_svm *svm,
>> +					  unsigned long start_page,
>> +					  unsigned long last_page)
>> +{
>> +	struct mm_struct *mm = svm->gpusvm.mm;
>> +	struct vm_area_struct *vma;
>> +	unsigned long start, end;
>> +	int r = 0;
>> +
>> +	if (!mm)
>> +		return -EFAULT;
>> +
>> +	start = start_page << PAGE_SHIFT;
>> +	end = (last_page + 1) << PAGE_SHIFT;
>> +
>> +	mmap_read_lock(mm);
>> +	do {
>> +		vma = amdgpu_svm_check_vma(mm, start);
>> +		if (IS_ERR(vma)) {
>> +			r = PTR_ERR(vma);
>> +			break;
>> +		}
>> +		start = min(end, vma->vm_end);
>> +	} while (start < end);
>> +	mmap_read_unlock(mm);
> 
> Again a clear NAK to any of those static checks!
> 
> The result of the check becomes invalid ass soon as the mman_read_lock is dropped, so that here only gives a completely false picture if some operation is valid or not.
> 
> We have two possibilities here, either keep the mman_read_lock while installing the SVM range or completely drop the check and make sure that faulting on a range which isn't backed by any VMA or an invalid VMA is properly handled in the fault handler.
> 
> I think we need to have a resilent fault handler who can deal with such errors anyway, so it is probably the best approach to go down that route.
> 

Got it, wil drop that check, and some negative test will become fail for 
some no vma setting SVM test.
And the behaviour for xnack-on and xnack-off will be different, xnack-on 
will get a silent fail, no error return when SVM set ioctl, xnack-on 
won't do fault when setting ioctl, but our fault patch will handle no 
vma situlation corretcly.
But for xnack-off, it always do fault when ioctl setting, so it can make 
the negative test pass, return error in iotcl.

Regards,
Honglei


> Regards,
> Christian.
> 
>> +
>> +	return r;
>> +}
>> +
>>   static void attr_set_interval(struct amdgpu_svm_attr_range *range,
>>   				unsigned long start_page,
>>   				unsigned long last_page)
>> @@ -233,3 +270,702 @@ static void attr_remove_range_locked(struct amdgpu_svm_attr_tree *attr_tree,
>>   	if (free_range)
>>   		kfree(range);
>>   }
>> +
>> +static void attr_change_ctx_set(
>> +		struct attr_set_ctx *change,
>> +		const struct amdgpu_svm_attrs *old_attrs,
>> +		const struct amdgpu_svm_attrs *new_attrs,
>> +		unsigned long start_page,
>> +		unsigned long last_page)
>> +{
>> +	change->old_attrs = *old_attrs;
>> +	change->new_attrs = *new_attrs;
>> +	change->start_page = start_page;
>> +	change->last_page = last_page;
>> +}
>> +
>> +static inline int attr_check_preferred_loc(uint32_t value)
>> +{
>> +	/*
>> +	 * SYSMEM and UNDEFINED are always valid; LOCAL/HIVE select device
>> +	 * VRAM (see enum amdgpu_ioctl_svm_location).
>> +	 */
>> +	switch (value) {
>> +	case AMDGPU_SVM_LOCATION_SYSMEM:
>> +	case AMDGPU_SVM_LOCATION_UNDEFINED:
>> +	case AMDGPU_SVM_LOCATION_LOCAL:
>> +	case AMDGPU_SVM_LOCATION_HIVE:
>> +		return 0;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +}
>> +
>> +static inline int attr_check_prefetch_loc(uint32_t value)
>> +{
>> +	/*
>> +	 * Prefetch accepts the same values as the preferred location (see
>> +	 * enum amdgpu_ioctl_svm_location); UNDEFINED means no prefetch and is
>> +	 * what GET_ATTR reports for default or mixed ranges, so it must round
>> +	 * trip through SET for CRIU restore.
>> +	 */
>> +	switch (value) {
>> +	case AMDGPU_SVM_LOCATION_SYSMEM:
>> +	case AMDGPU_SVM_LOCATION_UNDEFINED:
>> +	case AMDGPU_SVM_LOCATION_LOCAL:
>> +	case AMDGPU_SVM_LOCATION_HIVE:
>> +		return 0;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +}
>> +
>> +static inline int attr_check_access(uint32_t value)
>> +{
>> +	switch (value) {
>> +	case AMDGPU_SVM_ACCESS_INACCESSIBLE:
>> +	case AMDGPU_SVM_ACCESS_IN_PLACE:
>> +	case AMDGPU_SVM_ACCESS_ALLOW_MIGRATE:
>> +		return 0;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +}
>> +
>> +static bool attr_flag_type_to_bit(uint32_t type, uint32_t *flag_bit)
>> +{
>> +	if (type < AMDGPU_SVM_ATTR_HOST_ACCESS ||
>> +	    type > AMDGPU_SVM_ATTR_GPU_READ_MOSTLY)
>> +		return false;
>> +
>> +	*flag_bit = 1u << (type - AMDGPU_SVM_ATTR_HOST_ACCESS);
>> +	return true;
>> +}
>> +
>> +static inline int attr_check_flag_value(uint32_t value)
>> +{
>> +	if (value > 1)
>> +		return -EINVAL;
>> +
>> +	return 0;
>> +}
>> +
>> +static inline int attr_check_flag_attr(uint32_t type, uint32_t value)
>> +{
>> +	uint32_t flag_bit;
>> +	int ret;
>> +
>> +	if (!attr_flag_type_to_bit(type, &flag_bit))
>> +		return -EINVAL;
>> +
>> +	ret = attr_check_flag_value(value);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return 0;
>> +}
>> +
>> +static inline int attr_check_granularity(uint32_t value)
>> +{
>> +	return 0;
>> +}
>> +
>> +static int attr_set_validate(const struct drm_amdgpu_svm_attribute *attr)
>> +{
>> +	switch (attr->type) {
>> +	case AMDGPU_SVM_ATTR_PREFERRED_LOC:
>> +		return attr_check_preferred_loc(attr->value);
>> +	case AMDGPU_SVM_ATTR_PREFETCH_LOC:
>> +		return attr_check_prefetch_loc(attr->value);
>> +	case AMDGPU_SVM_ATTR_ACCESS:
>> +		return attr_check_access(attr->value);
>> +	case AMDGPU_SVM_ATTR_GRANULARITY:
>> +		return attr_check_granularity(attr->value);
>> +	case AMDGPU_SVM_ATTR_HOST_ACCESS:
>> +	case AMDGPU_SVM_ATTR_COHERENT:
>> +	case AMDGPU_SVM_ATTR_GPU_RO:
>> +	case AMDGPU_SVM_ATTR_GPU_EXEC:
>> +	case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
>> +	case AMDGPU_SVM_ATTR_EXT_COHERENT:
>> +		return attr_check_flag_attr(attr->type, attr->value);
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +}
>> +
>> +static void attr_apply_flag(struct amdgpu_svm_attrs *attrs,
>> +			    uint32_t type, uint32_t value)
>> +{
>> +	uint32_t flag_bit;
>> +
>> +	if (!attr_flag_type_to_bit(type, &flag_bit))
>> +		return;
>> +
>> +	if (value)
>> +		attrs->flags |= flag_bit;
>> +	else
>> +		attrs->flags &= ~flag_bit;
>> +}
>> +
>> +static void attr_apply(struct amdgpu_svm_attrs *attrs,
>> +					uint32_t nattr,
>> +					const struct drm_amdgpu_svm_attribute *pattrs)
>> +{
>> +	const struct drm_amdgpu_svm_attribute *attr;
>> +
>> +	for (attr = pattrs; nattr--; attr++) {
>> +		switch (attr->type) {
>> +		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
>> +			attrs->preferred_loc = (int32_t)attr->value;
>> +			break;
>> +		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
>> +			attrs->prefetch_loc = (int32_t)attr->value;
>> +			break;
>> +		case AMDGPU_SVM_ATTR_ACCESS:
>> +			attrs->access = (enum amdgpu_ioctl_svm_access)attr->value;
>> +			break;
>> +		case AMDGPU_SVM_ATTR_HOST_ACCESS:
>> +		case AMDGPU_SVM_ATTR_COHERENT:
>> +		case AMDGPU_SVM_ATTR_GPU_RO:
>> +		case AMDGPU_SVM_ATTR_GPU_EXEC:
>> +		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
>> +		case AMDGPU_SVM_ATTR_EXT_COHERENT:
>> +			attr_apply_flag(attrs, attr->type, attr->value);
>> +			break;
>> +		case AMDGPU_SVM_ATTR_GRANULARITY:
>> +			attrs->granularity = min_t(uint32_t, attr->value, 0x3f);
>> +			break;
>> +		default:
>> +			break;
>> +		}
>> +	}
>> +}
>> +
>> +static bool attr_same_attrs(const struct amdgpu_svm_attr_range *range,
>> +			    uint32_t nattr,
>> +			    const struct drm_amdgpu_svm_attribute *attrs)
>> +{
>> +	struct amdgpu_svm_attrs target;
>> +
>> +	target = range->attrs;
>> +	attr_apply(&target, nattr, attrs);
>> +	return attr_equal(&range->attrs, &target);
>> +}
>> +
>> +static int
>> +amdgpu_svm_attr_set_hole(struct amdgpu_svm_attr_tree *attr_tree,
>> +			  const struct amdgpu_svm_attrs *default_attrs,
>> +			  unsigned long start_page, unsigned long last_page,
>> +			  uint32_t nattr,
>> +			  const struct drm_amdgpu_svm_attribute *attrs,
>> +			  struct attr_set_ctx *change)
>> +{
>> +	struct amdgpu_svm_attrs new_attrs;
>> +	struct amdgpu_svm_attr_range *range;
>> +
>> +	lockdep_assert_held(&attr_tree->lock);
>> +
>> +	if (start_page > last_page)
>> +		return 0;
>> +
>> +	new_attrs = *default_attrs;
>> +	attr_apply(&new_attrs, nattr, attrs);
>> +
>> +	/* Always create a range entry even when attrs equal defaults */
>> +	range = amdgpu_svm_attr_range_alloc(start_page, last_page, &new_attrs);
>> +	if (!range)
>> +		return -ENOMEM;
>> +
>> +	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
>> +
>> +	attr_change_ctx_set(change, default_attrs,
>> +				       &new_attrs, start_page, last_page);
>> +	return 0;
>> +}
>> +
>> +/*
>> + * amdgpu_svm_attr_set_existing() - Apply attrs to part of an existing range
>> + *
>> + * Update @range over [@start_page, @last_page]. If the request fully covers
>> + * @range the attrs are updated in place, otherwise the range is split into the
>> + * head/tail, carrying the old attrs, and the covered middle, carrying
>> + * the new attrs. The resulting change is reported by @change.
>> + */
>> +static int
>> +amdgpu_svm_attr_set_existing(struct amdgpu_svm_attr_tree *attr_tree,
>> +			     struct amdgpu_svm_attr_range *range,
>> +			     unsigned long start_page, unsigned long last_page,
>> +			     uint32_t nattr,
>> +			     const struct drm_amdgpu_svm_attribute *attrs,
>> +			     struct attr_set_ctx *change)
>> +{
>> +	unsigned long range_start = amdgpu_svm_attr_start_page(range);
>> +	unsigned long range_last = amdgpu_svm_attr_last_page(range);
>> +	struct amdgpu_svm_attr_range *left = NULL;
>> +	struct amdgpu_svm_attr_range *right = NULL;
>> +	struct amdgpu_svm_attrs old_attrs;
>> +	struct amdgpu_svm_attrs new_attrs;
>> +
>> +	lockdep_assert_held(&attr_tree->lock);
>> +
>> +	old_attrs = range->attrs;
>> +
>> +	if (attr_same_attrs(range, nattr, attrs)) {
>> +		/* Report old==new so apply_attr_change can decide */
>> +		attr_change_ctx_set(change, &old_attrs,
>> +					       &old_attrs,
>> +					       start_page, last_page);
>> +		return 0;
>> +	}
>> +
>> +	new_attrs = old_attrs;
>> +	attr_apply(&new_attrs, nattr, attrs);
>> +
>> +	/* only need to update attr */
>> +	if (start_page == range_start && last_page == range_last) {
>> +		range->attrs = new_attrs;
>> +		attr_change_ctx_set(change, &old_attrs,
>> +					       &new_attrs, start_page, last_page);
>> +		return 0;
>> +	}
>> +
>> +	/* split head */
>> +	if (start_page > range_start) {
>> +		left = amdgpu_svm_attr_range_alloc(range_start, start_page - 1, &old_attrs);
>> +		if (!left)
>> +			return -ENOMEM;
>> +	}
>> +
>> +	/* split tail */
>> +	if (last_page < range_last) {
>> +		right = amdgpu_svm_attr_range_alloc(last_page + 1, range_last, &old_attrs);
>> +		if (!right) {
>> +			kfree(left);
>> +			return -ENOMEM;
>> +		}
>> +	}
>> +
>> +	attr_remove_range_locked(attr_tree, range, false);
>> +	if (left)
>> +		amdgpu_svm_attr_range_insert_locked(attr_tree, left);
>> +	attr_set_interval(range, start_page, last_page);
>> +	range->attrs = new_attrs;
>> +	amdgpu_svm_attr_range_insert_locked(attr_tree, range);
>> +	if (right)
>> +		amdgpu_svm_attr_range_insert_locked(attr_tree, right);
>> +
>> +	attr_change_ctx_set(change, &old_attrs,
>> +				       &new_attrs, start_page, last_page);
>> +	return 0;
>> +}
>> +
>> +/*
>> + * amdgpu_svm_attr_set_range() - Apply attr changes across [@start_page, @last_page]
>> + *
>> + * Walk the interval in segments, each segment being either an existing range
>> + * or a hole.
>> + */
>> +static int
>> +amdgpu_svm_attr_set_range(struct amdgpu_svm_attr_tree *attr_tree,
>> +			  const struct amdgpu_svm_attrs *default_attrs,
>> +			  unsigned long start_page, unsigned long last_page,
>> +			  uint32_t nattr,
>> +			  const struct drm_amdgpu_svm_attribute *attrs)
>> +{
>> +	struct amdgpu_svm *svm = attr_tree->svm;
>> +	unsigned long cursor = start_page;
>> +	bool need_retry = false;
>> +
>> +	while (cursor <= last_page) {
>> +		struct interval_tree_node *node;
>> +		unsigned long seg_last;
>> +		struct attr_set_ctx change = { 0 };
>> +		int ret;
>> +
>> +		amdgpu_svm_lock(svm);
>> +		mutex_lock(&attr_tree->lock);
>> +		node = interval_tree_iter_first(&attr_tree->tree, cursor, cursor);
>> +		if (node) {
>> +			struct amdgpu_svm_attr_range *range;
>> +
>> +			range = container_of(node, struct amdgpu_svm_attr_range, it_node);
>> +			seg_last = min(last_page, amdgpu_svm_attr_last_page(range));
>> +			ret = amdgpu_svm_attr_set_existing(attr_tree, range,
>> +								   cursor, seg_last,
>> +								   nattr, attrs, &change);
>> +		} else {
>> +			struct interval_tree_node *next;
>> +
>> +			seg_last = last_page;
>> +			if (cursor != ULONG_MAX) {
>> +				next = interval_tree_iter_first(&attr_tree->tree,
>> +								cursor + 1,
>> +								ULONG_MAX);
>> +				if (next) {
>> +					struct amdgpu_svm_attr_range *next_range;
>> +
>> +					next_range = container_of(next,
>> +						struct amdgpu_svm_attr_range,
>> +						it_node);
>> +					seg_last = min(last_page,
>> +						       amdgpu_svm_attr_start_page(next_range) - 1);
>> +				}
>> +			}
>> +			ret = amdgpu_svm_attr_set_hole(attr_tree,
>> +							       default_attrs,
>> +							       cursor, seg_last,
>> +							       nattr, attrs,
>> +							       &change);
>> +		}
>> +		mutex_unlock(&attr_tree->lock);
>> +
>> +		if (ret) {
>> +			amdgpu_svm_unlock(svm);
>> +			return ret;
>> +		}
>> +
>> +		ret = amdgpu_svm_apply_attr_change(svm,
>> +						   &change.old_attrs,
>> +						   &change.new_attrs,
>> +						   change.start_page,
>> +						   change.last_page);
>> +		amdgpu_svm_unlock(svm);
>> +
>> +		if (ret == -EAGAIN) {
>> +			need_retry = true;
>> +			ret = 0;
>> +		}
>> +
>> +		if (ret)
>> +			return ret;
>> +
>> +		if (seg_last == ULONG_MAX || seg_last == last_page)
>> +			break;
>> +
>> +		cursor = seg_last + 1;
>> +	}
>> +
>> +	return need_retry ? -EAGAIN : 0;
>> +}
>> +
>> +struct amdgpu_svm_attr_tree *
>> +amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm)
>> +{
>> +	struct amdgpu_svm_attr_tree *attr_tree;
>> +
>> +	attr_tree = kzalloc_obj(*attr_tree, GFP_KERNEL);
>> +	if (!attr_tree)
>> +		return NULL;
>> +
>> +	mutex_init(&attr_tree->lock);
>> +	attr_tree->tree = RB_ROOT_CACHED;
>> +	INIT_LIST_HEAD(&attr_tree->range_list);
>> +	attr_tree->svm = svm;
>> +	return attr_tree;
>> +}
>> +
>> +void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree)
>> +{
>> +	struct amdgpu_svm_attr_range *range, *tmp;
>> +
>> +	if (!attr_tree)
>> +		return;
>> +
>> +	mutex_lock(&attr_tree->lock);
>> +	list_for_each_entry_safe(range, tmp, &attr_tree->range_list, list) {
>> +		interval_tree_remove(&range->it_node, &attr_tree->tree);
>> +		list_del_init(&range->list);
>> +		kfree(range);
>> +	}
>> +	mutex_unlock(&attr_tree->lock);
>> +
>> +	mutex_destroy(&attr_tree->lock);
>> +	kfree(attr_tree);
>> +}
>> +
>> +int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
>> +			uint64_t start,
>> +			uint64_t size,
>> +			uint32_t nattr,
>> +			const struct drm_amdgpu_svm_attribute *attrs)
>> +{
>> +	struct amdgpu_svm *svm = attr_tree->svm;
>> +	struct amdgpu_svm_attrs default_attrs;
>> +	unsigned long start_page, last_page;
>> +	uint32_t i;
>> +	int r;
>> +
>> +	start_page = start >> PAGE_SHIFT;
>> +	last_page = (start + size - 1) >> PAGE_SHIFT;
>> +
>> +	for (i = 0; i < nattr; i++) {
>> +		pr_debug("set attr type %u value 0x%08x range [0x%lx-0x%lx] xnack:%d",
>> +			 attrs[i].type, attrs[i].value, start_page, last_page,
>> +			 svm->xnack_enabled ? 1 : 0);
>> +		r = attr_set_validate(&attrs[i]);
>> +		if (r) {
>> +			pr_debug("invalid attribute %u value 0x%08x",
>> +					 attrs[i].type, attrs[i].value);
>> +			return r;
>> +		}
>> +	}
>> +
>> +	r = amdgpu_svm_attr_range_is_valid(svm, start_page, last_page);
>> +	if (r)
>> +		return r;
>> +
>> +	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
>> +
>> +retry:
>> +	r = amdgpu_svm_attr_set_range(attr_tree, &default_attrs,
>> +					       start_page, last_page,
>> +					       nattr, attrs);
>> +	if (r == -EAGAIN) {
>> +		pr_debug("attr_set retry [0x%lx-0x%lx]\n",
>> +				 start_page, last_page);
>> +		amdgpu_svm_sync_work(svm);
>> +		cond_resched();
>> +		goto retry;
>> +	}
>> +
>> +	return r;
>> +}
>> +
>> +int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
>> +			  unsigned long start_page,
>> +			  unsigned long last_page)
>> +{
>> +	struct interval_tree_node *node;
>> +	int r = 0;
>> +
>> +	if (start_page > last_page)
>> +		return -EINVAL;
>> +
>> +	mutex_lock(&attr_tree->lock);
>> +
>> +	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
>> +	while (node) {
>> +		struct interval_tree_node *next;
>> +		struct amdgpu_svm_attr_range *range;
>> +		unsigned long range_start;
>> +		unsigned long range_last;
>> +
>> +		range = container_of(node, struct amdgpu_svm_attr_range, it_node);
>> +		next = interval_tree_iter_next(node, start_page, last_page);
>> +		range_start = amdgpu_svm_attr_start_page(range);
>> +		range_last = amdgpu_svm_attr_last_page(range);
>> +
>> +		if (range_start < start_page && range_last > last_page) {
>> +			struct amdgpu_svm_attr_range *tail;
>> +
>> +			tail = amdgpu_svm_attr_range_alloc(last_page + 1, range_last,
>> +							   &range->attrs);
>> +			if (!tail) {
>> +				r = -ENOMEM;
>> +				break;
>> +			}
>> +
>> +			attr_remove_range_locked(attr_tree, range, false);
>> +			attr_set_interval(range, range_start, start_page - 1);
>> +			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
>> +			amdgpu_svm_attr_range_insert_locked(attr_tree, tail);
>> +		} else if (range_start < start_page) {
>> +			attr_remove_range_locked(attr_tree, range, false);
>> +			attr_set_interval(range, range_start, start_page - 1);
>> +			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
>> +		} else if (range_last > last_page) {
>> +			attr_remove_range_locked(attr_tree, range, false);
>> +			attr_set_interval(range, last_page + 1, range_last);
>> +			amdgpu_svm_attr_range_insert_locked(attr_tree, range);
>> +		} else {
>> +			attr_remove_range_locked(attr_tree, range, true);
>> +		}
>> +
>> +		node = next;
>> +	}
>> +
>> +	mutex_unlock(&attr_tree->lock);
>> +	return r;
>> +}
>> +
>> +int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
>> +			  unsigned long start_page,
>> +			  unsigned long last_page)
>> +{
>> +	/*
>> +	 * Range with no attr node in this implementation is treated the
>> +	 * same as one that explicitly stores default attrs. So clear
>> +	 * the attribute ranges when user wants to reset the attrs.
>> +	 *
>> +	 * - GET_ATTR: return default_attrs to userspace when there is no attr
>> +	 *   range.
>> +	 * - SET_ATTR: holes are treated as having default_attrs when
>> +	 *   computing the change trigger.
>> +	 * - Fault: attr lookup falls back to defaults when no attr range exists,
>> +	 *   so migration and PTE flag decisions are unchanged.
>> +	 *
>> +	 * This approach simplifies the implementation and avoids redundant
>> +	 * attribute maintenance. This ioctl operation is for attribute only,
>> +	 * so do not invalidate the GPU mapping here.
>> +	 */
>> +	return amdgpu_svm_attr_clear(attr_tree, start_page, last_page);
>> +}
>> +
>> +static void attr_get_ctx_add(struct attr_get_ctx *ctx,
>> +			       const struct amdgpu_svm_attrs *attrs)
>> +{
>> +	if (!ctx->has_range) {
>> +		ctx->preferred_loc = attrs->preferred_loc;
>> +		ctx->prefetch_loc = attrs->prefetch_loc;
>> +		ctx->granularity = attrs->granularity;
>> +		ctx->access = attrs->access;
>> +		ctx->flags_and = attrs->flags;
>> +		ctx->has_range = true;
>> +		return;
>> +	}
>> +
>> +	if (ctx->preferred_loc != attrs->preferred_loc)
>> +		ctx->preferred_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
>> +	if (ctx->prefetch_loc != attrs->prefetch_loc)
>> +		ctx->prefetch_loc = AMDGPU_SVM_LOCATION_UNDEFINED;
>> +	if (attrs->granularity < ctx->granularity)
>> +		ctx->granularity = attrs->granularity;
>> +	if (ctx->access != attrs->access)
>> +		ctx->access = AMDGPU_SVM_ACCESS_INACCESSIBLE;
>> +	ctx->flags_and &= attrs->flags;
>> +}
>> +
>> +static int attr_get_ctx_to_result(const struct attr_get_ctx *ctx,
>> +				uint32_t nattr,
>> +				struct drm_amdgpu_svm_attribute *attrs)
>> +{
>> +	uint32_t i;
>> +
>> +	for (i = 0; i < nattr; i++) {
>> +		switch (attrs[i].type) {
>> +		case AMDGPU_SVM_ATTR_PREFERRED_LOC:
>> +			attrs[i].value = ctx->preferred_loc;
>> +			break;
>> +		case AMDGPU_SVM_ATTR_PREFETCH_LOC:
>> +			attrs[i].value = ctx->prefetch_loc;
>> +			break;
>> +		case AMDGPU_SVM_ATTR_ACCESS:
>> +			attrs[i].value = (uint32_t)ctx->access;
>> +			break;
>> +		case AMDGPU_SVM_ATTR_HOST_ACCESS:
>> +		case AMDGPU_SVM_ATTR_COHERENT:
>> +		case AMDGPU_SVM_ATTR_GPU_RO:
>> +		case AMDGPU_SVM_ATTR_GPU_EXEC:
>> +		case AMDGPU_SVM_ATTR_GPU_READ_MOSTLY:
>> +		case AMDGPU_SVM_ATTR_EXT_COHERENT: {
>> +			uint32_t flag_bit;
>> +
>> +			if (!attr_flag_type_to_bit(attrs[i].type, &flag_bit))
>> +				return -EINVAL;
>> +
>> +			attrs[i].value = (ctx->flags_and & flag_bit) ? 1 : 0;
>> +			break;
>> +		}
>> +		case AMDGPU_SVM_ATTR_GRANULARITY:
>> +			attrs[i].value = ctx->granularity;
>> +			break;
>> +		default:
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +/*
>> + * On return @out_size is shrunk to the length of the contiguous range starting
>> + * at @start over which all attributes are identical, so user space (e.g. CRIU)
>> + * can walk the address space one uniform range at a time.
>> + */
>> +int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
>> +			uint64_t start, uint64_t size,
>> +			uint32_t nattr,
>> +			struct drm_amdgpu_svm_attribute *attrs,
>> +			uint64_t *out_size)
>> +{
>> +	struct amdgpu_svm_attrs default_attrs;
>> +	struct amdgpu_svm_attrs ref_attrs;
>> +	struct attr_get_ctx ctx = { 0 };
>> +	struct interval_tree_node *node;
>> +	unsigned long start_page, last_page, cursor, homog_last;
>> +	bool have_ref = false;
>> +	int r;
>> +
>> +	start_page = start >> PAGE_SHIFT;
>> +	last_page = (start + size - 1) >> PAGE_SHIFT;
>> +
>> +	r = amdgpu_svm_attr_range_is_valid(attr_tree->svm, start_page, last_page);
>> +	if (r)
>> +		return r;
>> +
>> +	mutex_lock(&attr_tree->lock);
>> +	amdgpu_svm_attr_set_default(attr_tree->svm, &default_attrs);
>> +
>> +	homog_last = start_page;
>> +	cursor = start_page;
>> +	node = interval_tree_iter_first(&attr_tree->tree, start_page, last_page);
>> +
>> +	while (cursor <= last_page) {
>> +		const struct amdgpu_svm_attrs *range_attrs;
>> +		unsigned long range_last = last_page;
>> +		struct amdgpu_svm_attr_range *range = NULL;
>> +		unsigned long next;
>> +
>> +		if (node) {
>> +			range = container_of(node, struct amdgpu_svm_attr_range,
>> +					     it_node);
>> +
>> +			if (amdgpu_svm_attr_last_page(range) < cursor) {
>> +				node = interval_tree_iter_next(node, start_page,
>> +							       last_page);
>> +				continue;
>> +			}
>> +
>> +			if (amdgpu_svm_attr_start_page(range) <= cursor) {
>> +				range_last = min(last_page, amdgpu_svm_attr_last_page(range));
>> +				node = interval_tree_iter_next(node, start_page,
>> +							       last_page);
>> +			} else {
>> +				range_last = min(last_page,
>> +						 amdgpu_svm_attr_start_page(range) - 1);
>> +				range = NULL;
>> +			}
>> +		}
>> +
>> +		range_attrs = range ? &range->attrs : &default_attrs;
>> +
>> +		if (!have_ref) {
>> +			ref_attrs = *range_attrs;
>> +			have_ref = true;
>> +		} else if (!attr_equal(range_attrs, &ref_attrs)) {
>> +			break;
>> +		}
>> +
>> +		homog_last = range_last;
>> +
>> +		if (range_last == ULONG_MAX)
>> +			break;
>> +
>> +		next = range_last + 1;
>> +		if (next <= cursor)
>> +			break;
>> +		cursor = next;
>> +	}
>> +
>> +	if (!have_ref)
>> +		ref_attrs = default_attrs;
>> +
>> +	attr_get_ctx_add(&ctx, &ref_attrs);
>> +	r = attr_get_ctx_to_result(&ctx, nattr, attrs);
>> +	mutex_unlock(&attr_tree->lock);
>> +
>> +	if (!r && out_size)
>> +		*out_size = ((uint64_t)(homog_last - start_page) + 1) << PAGE_SHIFT;
>> +
>> +	return r;
>> +}
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
>> index 36fee11aa9ea0..56bcd81c608cb 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
>> @@ -158,6 +158,9 @@ enum amdgpu_svm_attr_change_trigger {
>>   	 AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE | \
>>   	 AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE)
>>   
>> +struct amdgpu_svm_attr_tree *
>> +amdgpu_svm_attr_tree_create(struct amdgpu_svm *svm);
>> +void amdgpu_svm_attr_tree_destroy(struct amdgpu_svm_attr_tree *attr_tree);
>>   struct amdgpu_svm_attr_range *
>>   amdgpu_svm_attr_find_locked(struct amdgpu_svm_attr_tree *attr_tree,
>>   			   unsigned long page);
>> @@ -169,6 +172,23 @@ amdgpu_svm_attr_get_bounds_locked(struct amdgpu_svm_attr_tree *attr_tree,
>>   void amdgpu_svm_attr_set_default(struct amdgpu_svm *svm,
>>   				 struct amdgpu_svm_attrs *attrs);
>>   
>> +int amdgpu_svm_attr_set(struct amdgpu_svm_attr_tree *attr_tree,
>> +			   uint64_t start,
>> +			   uint64_t size,
>> +			   uint32_t nattr,
>> +			   const struct drm_amdgpu_svm_attribute *attrs);
>> +int amdgpu_svm_attr_get(struct amdgpu_svm_attr_tree *attr_tree,
>> +				       uint64_t start,
>> +				       uint64_t size,
>> +				       uint32_t nattr,
>> +				       struct drm_amdgpu_svm_attribute *attrs,
>> +				       uint64_t *out_size);
>> +int amdgpu_svm_attr_clear(struct amdgpu_svm_attr_tree *attr_tree,
>> +			  unsigned long start_page,
>> +			  unsigned long last_page);
>> +int amdgpu_svm_attr_reset(struct amdgpu_svm_attr_tree *attr_tree,
>> +			  unsigned long start_page,
>> +			  unsigned long last_page);
>>   struct amdgpu_svm_attr_range *
>>   amdgpu_svm_attr_range_alloc(unsigned long start_page,
>>   			   unsigned long last_page,
> 

Re: [PATCH v10 05/22] drm/amdgpu: add SVM range types and work queue interface

[Huang, Honglei]

On 11/9/2026 9:48 PM, Christian König wrote:
> On 10/21/26 04:58, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Add amdgpu_svm_range.h with GPU mapped range types and interfaces:
>> - enum amdgpu_svm_range_queue_state: NOT_QUEUED, IN_GC, PROCESSING
>>    states for queue work
>> - struct amdgpu_svm_range: extends drm_gpusvm_range with gpu_mapped
>>    state, queue_state, attribute flags, work queue node, pending
>>    ops/pages, and validation timestamp
>> - enum amdgpu_svm_range_op: NONE, UNMAP operation types
>> - struct amdgpu_svm_range_op_ctx: dequeue context for GC processing
>> - Inline helper: amdgpu_svm_range_invalidate_gpu_mapping
>> - Range operations API declarations: find_or_insert, get_pages,
>>    update_mapping, update_gpu_range, invalidate
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Huang Rui <ray.huang@amd.com>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 129 ++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h     |  30 ++++
>>   2 files changed, 159 insertions(+)
>>   create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
>> new file mode 100644
>> index 0000000000000..39db3a18b5f2f
>> --- /dev/null
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
>> @@ -0,0 +1,129 @@
>> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
>> +/*
>> + * Copyright 2026 Advanced Micro Devices, Inc.
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice shall be included in
>> + * all copies or substantial portions of the Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
>> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
>> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>> + * OTHER DEALINGS IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#ifndef __AMDGPU_SVM_RANGE_H__
>> +#define __AMDGPU_SVM_RANGE_H__
>> +
>> +#include <drm/drm_gpusvm.h>
>> +#include <drm/drm_pagemap.h>
>> +
>> +#include "amdgpu_svm.h"
>> +#include "amdgpu_vm.h"
>> +
>> +#include <linux/ktime.h>
>> +#include <linux/list.h>
>> +#include <linux/types.h>
>> +
>> +struct amdgpu_svm;
>> +struct amdgpu_svm_attr_range;
>> +struct amdgpu_svm_attrs;
>> +struct dma_fence;
>> +struct drm_exec;
>> +struct drm_gpusvm_notifier;
>> +struct drm_gpusvm_range;
>> +struct mmu_notifier_range;
>> +
>> +/**
>> + * enum amdgpu_svm_range_queue_state - Work queue state of an SVM range
>> + * @AMDGPU_SVM_RANGE_NOT_QUEUED: Not on any work list.
>> + * @AMDGPU_SVM_RANGE_IN_GC: Queued on the GC list.
>> + * @AMDGPU_SVM_RANGE_PROCESSING: Being processed in workqueue.
>> + */
>> +enum amdgpu_svm_range_queue_state {
>> +	AMDGPU_SVM_RANGE_NOT_QUEUED = 0,
>> +	AMDGPU_SVM_RANGE_IN_GC,
>> +	AMDGPU_SVM_RANGE_PROCESSING,
>> +};
>> +
>> +/**
>> + * struct amdgpu_svm_range - amdgpu GPU mapped SVM range
>> + * @base: Embedded drm_gpusvm_range base.
>> + * @work_node: Links the range into the work list. Protected by amdgpu_svm.work_lock.
>> + * @gpu_mapped: True while the range has a live GPU mapping. Accessed with
>> + *              READ_ONCE/WRITE_ONCE.
>> + * @queue_state: Current work queue state @amdgpu_svm_range_queue_state.
>> + *               Protected by amdgpu_svm.work_lock.
>> + * @pending_ops: Pending amdgpu_svm_range_op bitmask. Protected by work_lock.
>> + * @pending_start_page: First page of the accumulated pending interval.
>> + * @pending_last_page: Last page of the accumulated pending interval.
>> + * @attr_flags: Attribute flags last applied to the GPU mapping.
>> + * @validate_timestamp: Time of the last successful mapping validation.
>> + */
>> +struct amdgpu_svm_range {
>> +	struct drm_gpusvm_range base;
>> +	struct list_head work_node;
>> +	bool gpu_mapped;
> 
>> +	u8 queue_state;
> 
> What is that good for and why is it needed?

Previous SVM implemetation need pop the range task under mmap write lock 
to prevent the UAF. This is not allowed under drmsvm helper.
The current series use queue state to trace the range work op state and 
seralize the different op by using work_lock instead of mmap lock.

Regards,
Honglei

> 
> Regards,
> Christian.
> 
>> +	u8 pending_ops;
>> +	unsigned long pending_start_page;
>> +	unsigned long pending_last_page;
>> +	uint32_t attr_flags;
>> +	ktime_t validate_timestamp;
>> +};
>> +
>> +static inline struct amdgpu_svm_range *
>> +to_amdgpu_svm_range(struct drm_gpusvm_range *range)
>> +{
>> +	return container_of(range, struct amdgpu_svm_range, base);
>> +}
>> +
>> +static inline void
>> +amdgpu_svm_range_invalidate_gpu_mapping(struct amdgpu_svm_range *range)
>> +{
>> +	WRITE_ONCE(range->gpu_mapped, false);
>> +}
>> +
>> +#define AMDGPU_SVM_RANGE_TRACE(r__, op__)				    \
>> +	trace_amdgpu_svm_range((op__),					    \
>> +		to_amdgpu_svm((r__)->base.gpusvm)->vm->pasid,		    \
>> +		(r__)->base.gpusvm, READ_ONCE((r__)->gpu_mapped),	    \
>> +		(r__)->base.pages.notifier_seq,				    \
>> +		drm_gpusvm_range_start(&(r__)->base) >> PAGE_SHIFT,	    \
>> +		drm_gpusvm_range_end(&(r__)->base) >> PAGE_SHIFT)
>> +
>> +/**
>> + * enum amdgpu_svm_range_op - Pending operation bits for an SVM range
>> + * @AMDGPU_SVM_RANGE_OP_NONE: No pending operation.
>> + * @AMDGPU_SVM_RANGE_OP_UNMAP: Range needs to be unmapped/removed by the GC.
>> + */
>> +enum amdgpu_svm_range_op {
>> +	AMDGPU_SVM_RANGE_OP_NONE    = 0,
>> +	AMDGPU_SVM_RANGE_OP_UNMAP   = BIT(0),
>> +};
>> +
>> +/**
>> + * struct amdgpu_svm_range_op_ctx - context of a range workqueue operation
>> + * @range: The dequeued range to process.
>> + * @start_page: First page for workqueue operation.
>> + * @last_page: Last page for workqueue operation.
>> + * @pending_ops: amdgpu_svm_range_op bitmask.
>> + */
>> +struct amdgpu_svm_range_op_ctx {
>> +	struct amdgpu_svm_range *range;
>> +	unsigned long start_page;
>> +	unsigned long last_page;
>> +	uint8_t pending_ops;
>> +};
>> +
>> +#endif /* __AMDGPU_SVM_RANGE_H__ */
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> index 599c1d0e512b5..86ba09cba34c9 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> @@ -116,6 +116,36 @@ TRACE_EVENT(amdgpu_iv,
>>   		      __entry->src_data[2], __entry->src_data[3])
>>   );
>>   
>> +TRACE_EVENT(amdgpu_svm_range,
>> +	    TP_PROTO(const char *op, u32 pasid, void *gpusvm, bool mapped,
>> +		     unsigned long seqno, unsigned long start_page,
>> +		     unsigned long end_page),
>> +	    TP_ARGS(op, pasid, gpusvm, mapped, seqno, start_page, end_page),
>> +	    TP_STRUCT__entry(
>> +			     __string(op, op)
>> +			     __field(u32, pasid)
>> +			     __field(void *, gpusvm)
>> +			     __field(bool, mapped)
>> +			     __field(unsigned long, seqno)
>> +			     __field(unsigned long, start_page)
>> +			     __field(unsigned long, end_page)
>> +			    ),
>> +	    TP_fast_assign(
>> +			   __assign_str(op);
>> +			   __entry->pasid = pasid;
>> +			   __entry->gpusvm = gpusvm;
>> +			   __entry->mapped = mapped;
>> +			   __entry->seqno = seqno;
>> +			   __entry->start_page = start_page;
>> +			   __entry->end_page = end_page;
>> +			   ),
>> +	    TP_printk("%s: pasid=%u gpusvm=%p mapped=%d seqno=%lu range=[0x%lx-0x%lx]-0x%lx",
>> +		      __get_str(op), __entry->pasid, __entry->gpusvm,
>> +		      __entry->mapped, __entry->seqno,
>> +		      __entry->start_page, __entry->end_page,
>> +		      __entry->end_page - __entry->start_page)
>> +);
>> +
>>   
>>   TRACE_EVENT(amdgpu_bo_create,
>>   	    TP_PROTO(struct amdgpu_bo *bo),
> 

Re: [PATCH v10 06/22] drm/amdgpu/gmc: add get_svm_pte_flags callback

[Huang, Honglei]

On 11/9/2026 9:52 PM, Christian König wrote:
> On 10/21/26 04:58, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Add a gmc_funcs callback for computing the GPU PTE flags of an SVM
>> mapping, plus a HW-agnostic input struct (amdgpu_svm_pte_flags_params)
>> and the amdgpu_gmc_get_svm_pte_flags() wrapper. This lets the per-IP
>> MTYPE/PTE selection live in the HW IP specific gmc_vX_0.c files instead
>> of the SVM core, mirroring how get_vm_pte handles BO mappings.
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Huang Rui <ray.huang@amd.com>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  24 +++++
>>   drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c  |  69 ++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 119 ++++++++++++++++++++++++
>>   3 files changed, 212 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> index cffd5ca3fc66b..3f668fb71a632 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>> @@ -150,6 +150,24 @@ struct amdgpu_vmhub {
>>   	const struct amdgpu_vmhub_funcs *vmhub_funcs;
>>   };
>>   
>> +/* Decoded inputs for the SVM PTE flag selection, gmc_funcs::get_svm_pte_flags */
>> +struct amdgpu_svm_pte_flags_params {
>> +	/* SVM attribute: coherent */
>> +	bool coherent;
>> +	/* SVM attribute: extended coherent */
>> +	bool ext_coherent;
>> +	/* SVM attribute: GPU read only */
>> +	bool gpu_ro;
>> +	/* SVM attribute: GPU executable */
>> +	bool gpu_exec;
>> +	/* mapping is in VRAM local device or remote P2P */
>> +	bool is_vram;
>> +	/* mapping is in this GPU's local VRAM */
>> +	bool is_local;
>> +	/* remote VRAM BO is on a GPU in the same XGMI hive as this GPU. */
>> +	bool same_hive;
>> +};
>> +
>>   /*
>>    * GPU MC structures, functions & helpers
>>    */
>> @@ -178,6 +196,10 @@ struct amdgpu_gmc_funcs {
>>   			   struct amdgpu_bo *bo,
>>   			   uint32_t vm_flags,
>>   			   uint64_t *pte_flags);
>> +	/* get the full pte flags for an SVM mapping */
>> +	uint64_t (*get_svm_pte_flags)(struct amdgpu_device *adev,
>> +				      struct amdgpu_vm *vm,
>> +				      const struct amdgpu_svm_pte_flags_params *params);
> 
> That looks like a really really bad idea to me.
> 
> Please add the SVM flags to the existing amdgpu_gmc_get_vm_pte() function.
> 
> Additional to that you also need to modify the destination addr and not just the flags.

Will change to existing amdgpu_gmc_get_vm_pte function

Regards,
Honglei

> 
> Regards,
> Christian.
> 
>>   	/* override per-page pte flags */
>>   	void (*override_vm_pte_flags)(struct amdgpu_device *dev,
>>   				      struct amdgpu_vm *vm,
>> @@ -378,6 +400,8 @@ struct amdgpu_gmc {
>>   #define amdgpu_gmc_get_vm_pte(adev, vm, bo, vm_flags, pte_flags) \
>>   	((adev)->gmc.gmc_funcs->get_vm_pte((adev), (vm), (bo), (vm_flags), \
>>   					   (pte_flags)))
>> +#define amdgpu_gmc_get_svm_pte_flags(adev, vm, params) \
>> +	((adev)->gmc.gmc_funcs->get_svm_pte_flags((adev), (vm), (params)))
>>   #define amdgpu_gmc_override_vm_pte_flags(adev, vm, addr, pte_flags)	\
>>   	(adev)->gmc.gmc_funcs->override_vm_pte_flags			\
>>   		((adev), (vm), (addr), (pte_flags))
>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>> index 0cc73982ed6f9..a541173e34a4e 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>> @@ -551,6 +551,74 @@ static void gmc_v12_0_get_vm_pte(struct amdgpu_device *adev,
>>   		*flags = AMDGPU_PTE_MTYPE_GFX12(*flags, MTYPE_UC);
>>   }
>>   
>> +/*
>> + * Mirror of the GFX12 cases of svm_range_get_pte_flags(), with the
>> + * per IP logic kept here in the HW specific file.
>> + */
>> +static uint64_t
>> +gmc_v12_0_get_svm_pte_flags(struct amdgpu_device *adev,
>> +			    struct amdgpu_vm *vm,
>> +			    const struct amdgpu_svm_pte_flags_params *params)
>> +{
>> +	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
>> +	bool coherent = params->coherent;
>> +	bool ext_coherent = params->ext_coherent;
>> +	bool is_local = params->is_local;
>> +	bool is_vram = params->is_vram;
>> +	bool snoop = !is_vram;
>> +	uint32_t mapping_flags = 0;
>> +	unsigned int mtype_local, mtype_remote;
>> +	bool is_aid_a1;
>> +	uint64_t pte_flags;
>> +
>> +	switch (gc_ip_version) {
>> +	case IP_VERSION(12, 0, 0):
>> +	case IP_VERSION(12, 0, 1):
>> +		mapping_flags |= AMDGPU_VM_MTYPE_NC;
>> +		break;
>> +	case IP_VERSION(12, 1, 0):
>> +		is_aid_a1 = (adev->rev_id & 0x10);
>> +		mtype_local = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_RW :
>> +				amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
>> +				is_aid_a1 ? AMDGPU_VM_MTYPE_RW : AMDGPU_VM_MTYPE_NC;
>> +		mtype_remote = is_aid_a1 ? AMDGPU_VM_MTYPE_NC : AMDGPU_VM_MTYPE_UC;
>> +		snoop = true;
>> +
>> +		if (is_local) {
>> +			mapping_flags |= mtype_local;
>> +		} else if (ext_coherent) {
>> +			mapping_flags |= AMDGPU_VM_MTYPE_UC;
>> +		} else {
>> +			/* system memory or remote VRAM */
>> +			mapping_flags |= mtype_remote;
>> +		}
>> +		break;
>> +	default:
>> +		mapping_flags |= coherent ?
>> +			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +		break;
>> +	}
>> +
>> +	if (params->gpu_exec)
>> +		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
>> +
>> +	pte_flags = AMDGPU_PTE_VALID;
>> +	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
>> +	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
>> +	pte_flags |= AMDGPU_PTE_IS_PTE;
>> +
>> +	gmc_v12_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
>> +	pte_flags |= AMDGPU_PTE_READABLE;
>> +	if (!params->gpu_ro)
>> +		pte_flags |= AMDGPU_PTE_WRITEABLE;
>> +
>> +	if (gc_ip_version == IP_VERSION(12, 1, 0) &&
>> +	    adev->have_atomics_support)
>> +		pte_flags |= AMDGPU_PTE_BUS_ATOMICS;
>> +
>> +	return pte_flags;
>> +}
>> +
>>   static unsigned gmc_v12_0_get_vbios_fb_size(struct amdgpu_device *adev)
>>   {
>>   	return 0;
>> @@ -580,6 +648,7 @@ static const struct amdgpu_gmc_funcs gmc_v12_0_gmc_funcs = {
>>   	.emit_pasid_mapping = gmc_v12_0_emit_pasid_mapping,
>>   	.get_vm_pde = gmc_v12_0_get_vm_pde,
>>   	.get_vm_pte = gmc_v12_0_get_vm_pte,
>> +	.get_svm_pte_flags = gmc_v12_0_get_svm_pte_flags,
>>   	.get_vbios_fb_size = gmc_v12_0_get_vbios_fb_size,
>>   	.get_dcc_alignment = gmc_v12_0_get_dcc_alignment,
>>   };
>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>> index 633772fa71fda..a3a5fa048a393 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>> @@ -1199,6 +1199,124 @@ static void gmc_v9_0_get_vm_pte(struct amdgpu_device *adev,
>>   		gmc_v9_0_get_coherence_flags(adev, vm, bo, vm_flags, flags);
>>   }
>>   
>> +/*
>> + * Mirror of the GFX9 cases of svm_range_get_pte_flags(), with the
>> + * per IP logic kept here in the HW specific file.
>> + */
>> +static uint64_t
>> +gmc_v9_0_get_svm_pte_flags(struct amdgpu_device *adev,
>> +			   struct amdgpu_vm *vm,
>> +			   const struct amdgpu_svm_pte_flags_params *params)
>> +{
>> +	uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
>> +	bool coherent = params->coherent;
>> +	bool ext_coherent = params->ext_coherent;
>> +	bool is_local = params->is_local;
>> +	bool is_vram = params->is_vram;
>> +	bool same_hive = params->same_hive;
>> +	bool snoop = !is_vram;
>> +	uint32_t mapping_flags = 0;
>> +	unsigned int mtype_local;
>> +	uint64_t pte_flags;
>> +
>> +	switch (gc_ip_version) {
>> +	case IP_VERSION(9, 4, 1):
>> +		if (is_vram) {
>> +			if (is_local) {
>> +				mapping_flags |= coherent ?
>> +					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
>> +			} else {
>> +				mapping_flags |= coherent ?
>> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +				if (same_hive)
>> +					snoop = true;
>> +			}
>> +		} else {
>> +			mapping_flags |= coherent ?
>> +				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +		}
>> +		break;
>> +	case IP_VERSION(9, 4, 2):
>> +		if (is_vram) {
>> +			if (is_local) {
>> +				mapping_flags |= coherent ?
>> +					AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
>> +				if (adev->gmc.xgmi.connected_to_cpu)
>> +					snoop = true;
>> +			} else {
>> +				mapping_flags |= coherent ?
>> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +				if (same_hive)
>> +					snoop = true;
>> +			}
>> +		} else {
>> +			mapping_flags |= coherent ?
>> +				AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +		}
>> +		break;
>> +	case IP_VERSION(9, 4, 3):
>> +	case IP_VERSION(9, 4, 4):
>> +	case IP_VERSION(9, 5, 0):
>> +		if (ext_coherent)
>> +			mtype_local = AMDGPU_VM_MTYPE_CC;
>> +		else
>> +			mtype_local = amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
>> +				amdgpu_mtype_local == 2 ? AMDGPU_VM_MTYPE_CC :
>> +				AMDGPU_VM_MTYPE_RW;
>> +		snoop = true;
>> +		if (is_vram) {
>> +			/* local HBM region close to partition */
>> +			if (is_local)
>> +				mapping_flags |= mtype_local;
>> +			/* local HBM region far from partition or remote XGMI GPU
>> +			 * with regular system scope coherence
>> +			 */
>> +			else if (same_hive && !ext_coherent)
>> +				mapping_flags |= AMDGPU_VM_MTYPE_NC;
>> +			/* PCIe P2P on GPUs pre-9.5.0 */
>> +			else if (gc_ip_version < IP_VERSION(9, 5, 0) && !same_hive)
>> +				mapping_flags |= AMDGPU_VM_MTYPE_UC;
>> +			/* Other remote memory */
>> +			else
>> +				mapping_flags |= ext_coherent ?
>> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +		} else if (adev->flags & AMD_IS_APU) {
>> +			/* On NUMA systems, locality is determined per-page
>> +			 * in gmc_v9_0_override_vm_pte_flags.
>> +			 */
>> +			if (num_possible_nodes() <= 1)
>> +				mapping_flags |= mtype_local;
>> +			else
>> +				mapping_flags |= ext_coherent ?
>> +					AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +		} else {
>> +			if (gc_ip_version < IP_VERSION(9, 5, 0) || ext_coherent)
>> +				mapping_flags |= AMDGPU_VM_MTYPE_UC;
>> +			else
>> +				mapping_flags |= AMDGPU_VM_MTYPE_NC;
>> +		}
>> +		break;
>> +	default:
>> +		mapping_flags |= coherent ?
>> +			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +		break;
>> +	}
>> +
>> +	if (params->gpu_exec)
>> +		mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
>> +
>> +	pte_flags = AMDGPU_PTE_VALID;
>> +	pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
>> +	pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
>> +
>> +	gmc_v9_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
>> +	pte_flags |= AMDGPU_PTE_READABLE;
>> +	if (!params->gpu_ro)
>> +		pte_flags |= AMDGPU_PTE_WRITEABLE;
>> +
>> +	return pte_flags;
>> +}
>> +
>>   static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev,
>>   					   struct amdgpu_vm *vm,
>>   					   uint64_t addr, uint64_t *flags)
>> @@ -1316,6 +1434,7 @@ static const struct amdgpu_gmc_funcs gmc_v9_0_gmc_funcs = {
>>   	.emit_pasid_mapping = gmc_v9_0_emit_pasid_mapping,
>>   	.get_vm_pde = gmc_v9_0_get_vm_pde,
>>   	.get_vm_pte = gmc_v9_0_get_vm_pte,
>> +	.get_svm_pte_flags = gmc_v9_0_get_svm_pte_flags,
>>   	.override_vm_pte_flags = gmc_v9_0_override_vm_pte_flags,
>>   	.get_vbios_fb_size = gmc_v9_0_get_vbios_fb_size,
>>   	.query_mem_partition_mode = &amdgpu_gmc_query_memory_partition,
> 

Re: [PATCH v10 07/22] drm/amdgpu: implement SVM range GPU mapping core

[Huang, Honglei]

On 11/9/2026 9:55 PM, Christian König wrote:
> On 10/21/26 04:58, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Implement GPU page table mapping in amdgpu_svm_range.c:
>> - amdgpu_svm_range_pages_valid: check notifier sequence validity
>> - amdgpu_svm_range_is_valid: validate gpu_mapped and attr_flags
>>    against the requested attribute set, under MMU notifier lock.
>> - amdgpu_svm_range_zap_ptes: clear GPU PTEs via amdgpu_vm_update_range
>> - amdgpu_svm_range_attr_pte_flags: compute PTE flags per GC IP version
>> - amdgpu_svm_range_lock_vm_pd: acquire VM page directory by drm_exec
>> - amdgpu_svm_range_update_gpu_range: DMA segment coalescing; recomputes
>>    PTE flags per segment from entry->proto via attr_pte_flags, then
>>    programs PTEs under notifier lock
>> - amdgpu_svm_range_find_or_insert: wrapper of drm gpu SVM find or insert
>>    API.
>> - amdgpu_svm_range_get_pages: wrapper of drm gpu SVM get pages.
>> - amdgpu_svm_range_evict: thin wrapper of drm_gpusvm_range_evict
>> - amdgpu_svm_range_update_mapping: full pipeline (lock PD, validate
>>    pages, program PTEs, update PDEs, flush TLB, record attr_flags
>>    and mapped state); takes attrs + read_only and forwards them to
>>    update_gpu_range
>> - amdgpu_svm_range_map_attrs: cursor based iteration across attr tree,
>>    creating/mapping ranges
>>
>> - Add to amdgpu_svm.h driver private interconnect tags used in
>> drm_pagemap_addr.proto:
>>    AMDGPU_INTERCONNECT_VRAM = DRM_INTERCONNECT_DRIVER
>>    AMDGPU_INTERCONNECT_P2P  = AMDGPU_INTERCONNECT_VRAM + 1
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Huang Rui <ray.huang@amd.com>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h       |   4 +
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 417 ++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h |  40 ++
>>   3 files changed, 461 insertions(+)
>>   create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> index 72f954d2df39e..cfdc4e6d045b7 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> @@ -27,6 +27,7 @@
>>   
>>   #include <drm/amdgpu_drm.h>
>>   #include <drm/drm_gpusvm.h>
>> +#include <drm/drm_pagemap.h>
>>   #include <linux/atomic.h>
>>   #include <linux/kref.h>
>>   #include <linux/list.h>
>> @@ -44,6 +45,9 @@ struct amdgpu_svm_attrs;
>>   struct drm_device;
>>   struct drm_file;
>>   
>> +#define AMDGPU_INTERCONNECT_VRAM	DRM_INTERCONNECT_DRIVER
>> +#define AMDGPU_INTERCONNECT_P2P		(AMDGPU_INTERCONNECT_VRAM + 1)
>> +
>>   enum amdgpu_svm_xnack_mode {
>>   	AMDGPU_SVM_XNACK_OFF,
>>   	AMDGPU_SVM_XNACK_ON,
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
>> new file mode 100644
>> index 0000000000000..d4830e53f3d8e
>> --- /dev/null
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
>> @@ -0,0 +1,417 @@
>> +// SPDX-License-Identifier: GPL-2.0 OR MIT
>> +/*
>> + * Copyright 2026 Advanced Micro Devices, Inc.
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice shall be included in
>> + * all copies or substantial portions of the Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
>> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
>> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>> + * OTHER DEALINGS IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#include "amdgpu_svm.h"
>> +#include "amdgpu_svm_attr.h"
>> +#include "amdgpu_svm_range.h"
>> +#include "amdgpu_svm_fault.h"
>> +#include "amdgpu.h"
>> +#include "amdgpu_vm.h"
>> +#include "amdgpu_trace.h"
>> +
>> +#include <drm/drm_exec.h>
>> +#include <drm/drm_pagemap.h>
>> +
>> +#include <linux/mmu_notifier.h>
>> +#include <uapi/linux/kfd_ioctl.h>
>> +
>> +bool
>> +amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
>> +		  struct amdgpu_svm_range *range)
>> +{
>> +	struct drm_gpusvm_range *base = &range->base;
>> +
>> +	lockdep_assert_held(&svm->gpusvm.notifier_lock);
>> +
>> +	if (base->pages.flags.unmapped || base->pages.flags.partial_unmap)
>> +		return false;
>> +
>> +	return drm_gpusvm_range_pages_valid(&svm->gpusvm, base);
>> +}
>> +
>> +bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
>> +			       const struct amdgpu_svm_attrs *attrs)
>> +{
>> +	return READ_ONCE(range->gpu_mapped) &&
>> +	       READ_ONCE(range->attr_flags) == attrs->flags;
>> +}
>> +
>> +
>> +int
>> +amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
>> +			  struct amdgpu_svm_range *range,
>> +			  unsigned long start_page,
>> +			  unsigned long last_page)
>> +{
>> +	struct dma_fence *fence = NULL;
>> +	unsigned int flags;
>> +	int ret;
>> +
>> +	if (last_page < start_page)
>> +		return 0;
>> +
>> +	flags = memalloc_noreclaim_save();
> 
> Completely drop that, this needs to be handled by the VM code and not here.

Will drop it.

> 
>> +	ret = amdgpu_vm_unmap_range(svm->adev, svm->vm, NULL,
>> +				    start_page, last_page, 0, &fence);
>> +	memalloc_noreclaim_restore(flags);
>> +
>> +	if (fence) {
>> +		dma_fence_wait(fence, false);
>> +		dma_fence_put(fence);
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +uint64_t
>> +amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
>> +			    const struct amdgpu_svm_attrs *attrs,
>> +			    bool read_only,
>> +			    enum drm_interconnect_protocol proto)
>> +{
>> +	struct amdgpu_device *adev = svm->adev;
>> +	uint32_t flags = attrs->flags;
>> +	bool is_local = (proto == AMDGPU_INTERCONNECT_VRAM);
>> +	struct amdgpu_svm_pte_flags_params params = {
>> +		.coherent = flags & (AMDGPU_SVM_ATTR_BIT_COHERENT |
>> +				     AMDGPU_SVM_ATTR_BIT_EXT_COHERENT),
>> +		.ext_coherent = flags & AMDGPU_SVM_ATTR_BIT_EXT_COHERENT,
>> +		.gpu_ro = flags & AMDGPU_SVM_ATTR_BIT_GPU_RO,
>> +		.gpu_exec = flags & AMDGPU_SVM_ATTR_BIT_GPU_EXEC,
>> +		.is_local = is_local,
>> +		.is_vram = is_local || (proto == AMDGPU_INTERCONNECT_P2P),
>> +		/* TODO: add hive check. */
>> +		.same_hive = false,
>> +	};
>> +	uint32_t gc_ip_version;
>> +	uint32_t mapping_flags = 0;
>> +	uint64_t pte_flags;
>> +	bool snoop;
>> +
>> +	/*
>> +	 * The per-IP MTYPE / PTE flag selection lives in the HW IP specific
>> +	 * gmc_vX_0.c files mirroring amdgpu_gmc_get_vm_pte for BO mappings.
>> +	 */
>> +	if (adev->gmc.gmc_funcs->get_svm_pte_flags) {
>> +		pte_flags = amdgpu_gmc_get_svm_pte_flags(adevamdgpu_gmc_get_vm_pte, svm->vm, &params);
>> +	} else {
>> +		gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
>> +		snoop = !is_local;
>> +		mapping_flags |= params.coherent ?
>> +			AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>> +
>> +		if (params.gpu_exec)
>> +			mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
>> +
>> +		pte_flags = AMDGPU_PTE_VALID;
>> +		pte_flags |= is_local ? 0 : AMDGPU_PTE_SYSTEM;
>> +		pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
>> +		if (gc_ip_version >= IP_VERSION(12, 0, 0))
>> +			pte_flags |= AMDGPU_PTE_IS_PTE;
>> +
>> +		amdgpu_gmc_get_vm_pte(adev, svm->vm, NULL, mapping_flags,
>> +				      &pte_flags);
>> +		pte_flags |= AMDGPU_PTE_READABLE;
>> +		if (!params.gpu_ro)
>> +			pte_flags |= AMDGPU_PTE_WRITEABLE;
> 
> Don't touch the flags after calling amdgpu_gmc_get_vm_pte().

Will remove this.

> 
>> +	}
>> +
>> +	if (read_only)
>> +		pte_flags &= ~AMDGPU_PTE_WRITEABLE;
> 
> Same here.

Will remove.


> 
>> +
>> +	return pte_flags;
>> +}
>> +
>> +
>> +
>> +int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm, struct drm_exec *exec,
>> +				bool intr)
>> +{
>> +	unsigned int exec_flags = DRM_EXEC_IGNORE_DUPLICATES;
>> +	int ret;
>> +
>> +	if (intr)
>> +		exec_flags |= DRM_EXEC_INTERRUPTIBLE_WAIT;
>> +
>> +	drm_exec_init(exec, exec_flags, 0);
>> +	drm_exec_until_all_locked(exec) {
>> +		ret = amdgpu_vm_lock_pd(svm->vm, exec, 1);
>> +		drm_exec_retry_on_contention(exec);
>> +		if (ret) {
>> +			drm_exec_fini(exec);
>> +			return ret;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
> 
> Please drop that helper and always use the drm_exec loop directly.

Will drop this helper.

Regards,
Honglei

> 
> Regards,
> Christian.
> 
>> +
>> +int
>> +amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
>> +				  struct amdgpu_svm_range *range,
>> +				  const struct amdgpu_svm_attrs *attrs,
>> +				  bool read_only,
>> +				  bool flush_tlb,
>> +				  bool wait_fence,
>> +				  struct dma_fence **fence)
>> +{
>> +	struct drm_gpusvm_range *base = &range->base;
>> +
>> +	lockdep_assert_held(&svm->gpusvm.notifier_lock);
>> +
>> +	const unsigned long range_start_page = drm_gpusvm_range_start(base) >> PAGE_SHIFT;
>> +	const unsigned long range_end_page = drm_gpusvm_range_end(base) >> PAGE_SHIFT;
>> +	const unsigned long npages = range_end_page - range_start_page;
>> +	unsigned long mapped_pages = 0;
>> +	unsigned long dma_idx = 0;
>> +	int ret;
>> +
>> +	if (!base->pages.dma_addr || !npages)
>> +		return -EINVAL;
>> +
>> +	while (mapped_pages < npages) {
>> +		const struct drm_pagemap_addr *entry = &base->pages.dma_addr[dma_idx++];
>> +		unsigned long seg_pages = min_t(unsigned long, 1UL << entry->order,
>> +						npages - mapped_pages);
>> +		uint64_t pte_flags;
>> +		unsigned long start_page, last_page;
>> +		bool is_last_seg;
>> +
>> +		if (entry->proto != DRM_INTERCONNECT_SYSTEM)
>> +			return -EOPNOTSUPP;
>> +
>> +		pte_flags = amdgpu_svm_range_attr_pte_flags(svm, attrs,
>> +							    read_only,
>> +							    entry->proto);
>> +
>> +		start_page = range_start_page + mapped_pages;
>> +		last_page = start_page + seg_pages - 1;
>> +		mapped_pages += seg_pages;
>> +		is_last_seg = mapped_pages == npages;
>> +
>> +		/*
>> +		 * The struct hmm_range hmm_range field inside amdgpu_hmm_range
>> +		 * is drm_gpusvm internal state. It is not exposed to the
>> +		 * implementing driver and is managed entirely by the framework
>> +		 * as part of the framework logic. This amdgpu_vm_map_range call
>> +		 * always runs inside the drm_gpusvm notifier, so omitting
>> +		 * hmm_range here is safe.
>> +		 */
>> +		ret = amdgpu_vm_map_range(svm->adev, svm->vm,
>> +					  flush_tlb && is_last_seg, true, NULL,
>> +					  start_page, last_page, pte_flags,
>> +					  0, entry->addr, NULL, NULL, NULL,
>> +					  wait_fence && is_last_seg ? fence : NULL);
>> +		if (ret)
>> +			return ret;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +struct amdgpu_svm_range *
>> +amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
>> +				unsigned long gpuva_start, unsigned long gpuva_end,
>> +				struct drm_gpusvm_ctx *ctx)
>> +{
>> +	struct drm_gpusvm_range *r;
>> +
>> +retry:
>> +	r = drm_gpusvm_range_find_or_insert(&svm->gpusvm, addr,
>> +					    gpuva_start, gpuva_end, ctx);
>> +	/*
>> +	 * UMD doesn't set RO for some RO VMAs, but the drm gpu svm framework
>> +	 * denies no RO flag range insert for RO VMAs, so treat
>> +	 * -EPERM as an indication of RO and retry if not set.
>> +	 */
>> +	if (PTR_ERR_OR_ZERO(r) == -EPERM && !ctx->read_only) {
>> +		ctx->read_only = true;
>> +		goto retry;
>> +	}
>> +
>> +	if (IS_ERR(r))
>> +		return ERR_CAST(r);
>> +
>> +	return to_amdgpu_svm_range(r);
>> +}
>> +
>> +int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
>> +			       struct drm_gpusvm_range *range,
>> +			       struct drm_gpusvm_ctx *ctx)
>> +{
>> +	int ret;
>> +
>> +retry:
>> +	ret = drm_gpusvm_range_get_pages(&svm->gpusvm, range, ctx);
>> +	/*
>> +	 * HMM returns -EPERM when write access is requested for a read-only
>> +	 * VMA. Retry as read-only so the eventual GPU mapping follows the CPU
>> +	 * VMA permissions.
>> +	 */
>> +	if (ret == -EPERM && !ctx->read_only) {
>> +		ctx->read_only = true;
>> +		goto retry;
>> +	}
>> +
>> +	if (ret == -EOPNOTSUPP) {
>> +		pr_err_ratelimited("range get pages -EOPNOTSUPP, evict and retry: gpuva=[0x%lx-0x%lx) ret=%d\n",
>> +				   drm_gpusvm_range_start(range),
>> +				   drm_gpusvm_range_end(range), ret);
>> +		amdgpu_svm_range_evict(svm, range);
>> +	}
>> +
>> +	return ret;
>> +}
>> +
>> +void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
>> +			    struct drm_gpusvm_range *range)
>> +{
>> +	if (!range->pages.flags.has_devmem_pages)
>> +		return;
>> +
>> +	drm_gpusvm_range_evict(&svm->gpusvm, range);
>> +}
>> +
>> +int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
>> +				    struct amdgpu_svm_range *range,
>> +				    const struct amdgpu_svm_attrs *attrs,
>> +				    bool read_only,
>> +				    bool intr, bool wait,
>> +				    bool flush_tlb)
>> +{
>> +	struct drm_exec exec;
>> +	struct dma_fence *fence = NULL;
>> +	unsigned int flags;
>> +	int ret;
>> +
>> +	ret = amdgpu_svm_range_lock_vm_pd(svm, &exec, intr);
>> +	if (ret)
>> +		return ret;
>> +
>> +	flags = memalloc_noreclaim_save();
>> +	drm_gpusvm_notifier_lock(&svm->gpusvm);
>> +
>> +	if (!amdgpu_svm_range_pages_valid(svm, range)) {
>> +		amdgpu_svm_range_invalidate_gpu_mapping(range);
>> +		ret = -EAGAIN;
>> +	} else {
>> +		ret = amdgpu_svm_range_update_gpu_range(svm, range, attrs,
>> +							read_only, flush_tlb,
>> +							wait, wait ? &fence : NULL);
>> +	}
>> +
>> +	drm_gpusvm_notifier_unlock(&svm->gpusvm);
>> +	memalloc_noreclaim_restore(flags);
>> +
>> +	if (!ret && fence)
>> +		dma_fence_wait(fence, intr);
>> +	dma_fence_put(fence);
>> +
>> +	if (!ret)
>> +		ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, false);
>> +
>> +	if (!ret) {
>> +		if (flush_tlb)
>> +			amdgpu_svm_flush_tlb(svm);
>> +		WRITE_ONCE(range->attr_flags, attrs->flags);
>> +		WRITE_ONCE(range->gpu_mapped, true);
>> +		range->validate_timestamp = ktime_get_boottime();
>> +	}
>> +
>> +	drm_exec_fini(&exec);
>> +	return ret;
>> +}
>> +
>> +int
>> +amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
>> +		       const struct amdgpu_svm_attrs *attrs,
>> +		       unsigned long start, unsigned long end)
>> +{
>> +	unsigned long addr = start;
>> +	int ret;
>> +	bool devmem_possible = amdgpu_svm_devmem_possible(svm);
>> +	bool need_vram_migration = devmem_possible &&
>> +				   amdgpu_svm_attr_prefer_vram(attrs);
>> +	struct drm_gpusvm_ctx map_ctx = {
>> +		.devmem_possible = devmem_possible,
>> +		.devmem_only = need_vram_migration,
>> +		.check_pages_threshold = devmem_possible ? SZ_64K : 0,
>> +	};
>> +
>> +	while (addr < end) {
>> +		struct amdgpu_svm_range *range;
>> +		unsigned long next_addr;
>> +		/* reset read_only every iteration, amdgpu_svm_range_find_or_insert may change it */
>> +		map_ctx.read_only = !!(attrs->flags & AMDGPU_SVM_ATTR_BIT_GPU_RO);
>> +
>> +		range = amdgpu_svm_range_find_or_insert(svm, addr,
>> +							addr, end,
>> +							&map_ctx);
>> +		if (IS_ERR(range)) {
>> +			pr_err_ratelimited("failed to find/insert range for gpuva 0x%lx [0x%lx-0x%lx), ret=%ld\n",
>> +					   addr, start, end, PTR_ERR(range));
>> +			return PTR_ERR(range);
>> +		}
>> +
>> +		next_addr = drm_gpusvm_range_end(&range->base);
>> +		if (next_addr <= addr)
>> +			return -EINVAL;
>> +
>> +		if (amdgpu_svm_range_is_valid(range, attrs)) {
>> +			addr = next_addr;
>> +			continue;
>> +		}
>> +
>> +		/* TODO: add migration */
>> +
>> +		AMDGPU_SVM_RANGE_TRACE(range, "GET PAGES");
>> +
>> +		ret = amdgpu_svm_range_get_pages(svm, &range->base,
>> +						 &map_ctx);
>> +		if (ret) {
>> +			pr_err_ratelimited("failed to get pages for range [0x%lx-0x%lx), ret=%d\n",
>> +					   drm_gpusvm_range_start(&range->base),
>> +					   drm_gpusvm_range_end(&range->base), ret);
>> +			return ret;
>> +		}
>> +
>> +		AMDGPU_SVM_RANGE_TRACE(range, "UPDATE MAPPING");
>> +
>> +		ret = amdgpu_svm_range_update_mapping(svm, range, attrs,
>> +						      map_ctx.read_only,
>> +						      true, true,
>> +						      true);
>> +		if (ret) {
>> +			pr_err_ratelimited("failed to update gpu map for range [0x%lx-0x%lx), ret=%d\n",
>> +					   drm_gpusvm_range_start(&range->base),
>> +					   drm_gpusvm_range_end(&range->base), ret);
>> +			return ret;
>> +		}
>> +
>> +		addr = next_addr;
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
>> index 39db3a18b5f2f..e5a5f52166576 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
>> @@ -126,4 +126,44 @@ struct amdgpu_svm_range_op_ctx {
>>   	uint8_t pending_ops;
>>   };
>>   
>> +uint64_t amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
>> +					 const struct amdgpu_svm_attrs *attrs,
>> +					 bool read_only,
>> +					 enum drm_interconnect_protocol proto);
>> +int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm,
>> +				struct drm_exec *exec,
>> +				bool intr);
>> +bool amdgpu_svm_range_pages_valid(struct amdgpu_svm *svm,
>> +				  struct amdgpu_svm_range *range);
>> +bool amdgpu_svm_range_is_valid(struct amdgpu_svm_range *range,
>> +			       const struct amdgpu_svm_attrs *attrs);
>> +int amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
>> +				      struct amdgpu_svm_range *range,
>> +				      const struct amdgpu_svm_attrs *attrs,
>> +				      bool read_only,
>> +				      bool flush_tlb, bool wait,
>> +				      struct dma_fence **fence);
>> +int amdgpu_svm_range_update_mapping(struct amdgpu_svm *svm,
>> +				    struct amdgpu_svm_range *range,
>> +				    const struct amdgpu_svm_attrs *attrs,
>> +				    bool read_only,
>> +				    bool intr, bool wait,
>> +				    bool flush_tlb);
>> +int amdgpu_svm_range_map_attrs(struct amdgpu_svm *svm,
>> +			       const struct amdgpu_svm_attrs *attrs,
>> +			       unsigned long start, unsigned long end);
>> +int amdgpu_svm_range_zap_ptes(struct amdgpu_svm *svm,
>> +			      struct amdgpu_svm_range *range,
>> +			      unsigned long start_page,
>> +			      unsigned long last_page);
>> +void amdgpu_svm_range_evict(struct amdgpu_svm *svm,
>> +			    struct drm_gpusvm_range *range);
>> +struct amdgpu_svm_range *
>> +amdgpu_svm_range_find_or_insert(struct amdgpu_svm *svm, unsigned long addr,
>> +				unsigned long gpuva_start, unsigned long gpuva_end,
>> +				struct drm_gpusvm_ctx *ctx);
>> +int amdgpu_svm_range_get_pages(struct amdgpu_svm *svm,
>> +			       struct drm_gpusvm_range *range,
>> +			       struct drm_gpusvm_ctx *ctx);
>> +
>>   #endif /* __AMDGPU_SVM_RANGE_H__ */
> 

Re: [PATCH v10 08/22] drm/amdgpu: implement SVM range notifier and GC helpers

[Huang, Honglei]

On 11/10/2026 1:12 AM, Aveline Noir wrote:
> Hi Ray,
> 
> On Wed, 21 Oct 2026 10:58:47 +0800 Huang Rui wrote:
> 
>> + * Invalidate the GPU view of @range over the notifier window: zap the
>> + * affected PTEs and clear the gpu_mapped flag. The range is not removed
>> + * here; structural removal if needed happens in the event-end phase.
> 
> I think this can leave stale PTEs behind when only part of a range is
> invalidated. Take a one-page notifier event on a larger system-memory
> range mapped with 4K PTEs. This zaps that one PTE but clears gpu_mapped
> for the whole range, and then the event-end callback unmaps DMA for the
> whole range. The other PTEs are still valid at that point, but their DMA
> mappings are gone. A later notifier won't zap them either, because
> gpu_mapped is already false.
> 
> Maybe the zap should cover the full range when its DMA mapping is
> going away?

Yes, the zep should cover the entire range, will fix it in next version.

> 
>> +			if (crosses_boundary) {
> 
> Another case is GPU_RO going from 1 to 0 over a whole mapped
> GPUSVM range. This branch isn't taken, so the PTEs are zapped but the
> old DMA_TO_DEVICE mapping stays around. The next get_pages() sees valid
> pages and reuses it, so the writable GPU PTE goes in without an
> HMM_PFN_REQ_WRITE fault or a DMA_BIDIRECTIONAL remap. If it's a VM_WRITE
> MAP_PRIVATE mapping whose CPU PTE still needs COW, I think it can miss
> the write fault. So could the old DMA mapping be dropped before the
> writable PTE goes in?

Will apply the drm_gpusvm_unmap_pages when attr is change, regardless of 
whether crosses_boundary.

> 
> I couldn't tell from the cover which drm_gpusvm base this is on, so I
> used 671b7825dbfe for both of these. I haven't reproduced either one on
> a GPU, though.

Regards,
Honglei

> 
> Thanks,
> Aveline Noir

Re: [PATCH v10 10/22] drm/amdgpu: implement SVM initialization and lifecycle

[Huang, Honglei]

On 11/10/2026 12:59 AM, Aveline Noir wrote:
> Hi Ray,
> 
> On Wed, 21 Oct 2026 10:58:49 +0800 Huang Rui wrote:
> 
>> +	if (new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_UNDEFINED &&
>> +	    new_attrs->prefetch_loc != AMDGPU_SVM_LOCATION_SYSMEM)
>> +		trigger |= AMDGPU_SVM_ATTR_TRIGGER_PREFETCH;
> 
> I'm not sure about excluding SYSMEM here... The proposed libhsakmt
> adaptation maps a CPU prefetch target to SYSMEM. If the pages are in
> VRAM and the update fully covers an existing mapped SVM range, nothing
> gets mapped or migrated eagerly. The GPU PTEs can be zapped, but the
> pages stay in VRAM even though SET_ATTR can return success.
> 
> Is SYSMEM prefetch meant to move pages back to RAM, or only to set the
> attribute for a later CPU fault?

This is indeed a issue, will add local system into prefetch in next version.

Regards,
Honglei


> 
> Thanks,
> Aveline Noir

Re: [PATCH v10 11/22] drm/amdgpu: add SVM ioctl entry and fault handler module

[Huang, Honglei]

On 11/10/2026 12:47 AM, Aveline Noir wrote:
> Hi Ray,
> 
> On Wed, 21 Oct 2026 10:58:50 +0800 Huang Rui wrote:
> 
>> +	ret = fault_check_allowed(svm, fault_addr, write_fault);
>> +	if (ret)
>> +		return ret;
> 
> This checks the VMA permissions but not attrs->access, so I think
> INACCESSIBLE isn't really enforced. SET_ATTR invalidates the existing
> GPU mapping, but the next retry fault just maps the range again.
> 
> IN_PLACE has a similar problem. With preferred_loc=LOCAL, the
> migration added by 22/22 can still move migratable system-memory pages
> into VRAM on a pagemap-capable GPU. Both seem to go against the access
> definitions in 01/22, and the prefetch path doesn't check access
> either.
> 
> Could the fault and prefetch paths check access before mapping or
> migrating?

Will add access check in fault_check_allowed, IN_PLACE access, prefetch 
access path.

Really thanks for the review!

Regards,
Honglei

> 
> Thanks,
> Aveline Noir

Re: [PATCH v10 06/22] drm/amdgpu/gmc: add get_svm_pte_flags callback

[Christian König]

On 11/10/26 07:57, Huang, Honglei wrote:
> 
> 
> On 11/9/2026 9:52 PM, Christian König wrote:
>> On 10/21/26 04:58, Huang Rui wrote:
>>> From: Honglei Huang <honghuan@amd.com>
>>>
>>> Add a gmc_funcs callback for computing the GPU PTE flags of an SVM
>>> mapping, plus a HW-agnostic input struct (amdgpu_svm_pte_flags_params)
>>> and the amdgpu_gmc_get_svm_pte_flags() wrapper. This lets the per-IP
>>> MTYPE/PTE selection live in the HW IP specific gmc_vX_0.c files instead
>>> of the SVM core, mirroring how get_vm_pte handles BO mappings.
>>>
>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>> Suggested-by: Huang Rui <ray.huang@amd.com>
>>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>>> ---
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  24 +++++
>>>   drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c  |  69 ++++++++++++++
>>>   drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 119 ++++++++++++++++++++++++
>>>   3 files changed, 212 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> index cffd5ca3fc66b..3f668fb71a632 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> @@ -150,6 +150,24 @@ struct amdgpu_vmhub {
>>>       const struct amdgpu_vmhub_funcs *vmhub_funcs;
>>>   };
>>>   +/* Decoded inputs for the SVM PTE flag selection, gmc_funcs::get_svm_pte_flags */
>>> +struct amdgpu_svm_pte_flags_params {
>>> +    /* SVM attribute: coherent */
>>> +    bool coherent;
>>> +    /* SVM attribute: extended coherent */
>>> +    bool ext_coherent;
>>> +    /* SVM attribute: GPU read only */
>>> +    bool gpu_ro;
>>> +    /* SVM attribute: GPU executable */
>>> +    bool gpu_exec;
>>> +    /* mapping is in VRAM local device or remote P2P */
>>> +    bool is_vram;
>>> +    /* mapping is in this GPU's local VRAM */
>>> +    bool is_local;
>>> +    /* remote VRAM BO is on a GPU in the same XGMI hive as this GPU. */
>>> +    bool same_hive;
>>> +};
>>> +
>>>   /*
>>>    * GPU MC structures, functions & helpers
>>>    */
>>> @@ -178,6 +196,10 @@ struct amdgpu_gmc_funcs {
>>>                  struct amdgpu_bo *bo,
>>>                  uint32_t vm_flags,
>>>                  uint64_t *pte_flags);
>>> +    /* get the full pte flags for an SVM mapping */
>>> +    uint64_t (*get_svm_pte_flags)(struct amdgpu_device *adev,
>>> +                      struct amdgpu_vm *vm,
>>> +                      const struct amdgpu_svm_pte_flags_params *params);
>>
>> That looks like a really really bad idea to me.
>>
>> Please add the SVM flags to the existing amdgpu_gmc_get_vm_pte() function.
>>
>> Additional to that you also need to modify the destination addr and not just the flags.
> 
> Will change to existing amdgpu_gmc_get_vm_pte function

It might be necessary to rework the existing amdgpu_gmc_get_vm_pte function to get the BO flags as parameter instead of the BO itself so that it can be used for both use cases.

Maybe split that into a separate patch.

Regards,
Christian.

> 
> Regards,
> Honglei
> 
>>
>> Regards,
>> Christian.
>>
>>>       /* override per-page pte flags */
>>>       void (*override_vm_pte_flags)(struct amdgpu_device *dev,
>>>                         struct amdgpu_vm *vm,
>>> @@ -378,6 +400,8 @@ struct amdgpu_gmc {
>>>   #define amdgpu_gmc_get_vm_pte(adev, vm, bo, vm_flags, pte_flags) \
>>>       ((adev)->gmc.gmc_funcs->get_vm_pte((adev), (vm), (bo), (vm_flags), \
>>>                          (pte_flags)))
>>> +#define amdgpu_gmc_get_svm_pte_flags(adev, vm, params) \
>>> +    ((adev)->gmc.gmc_funcs->get_svm_pte_flags((adev), (vm), (params)))
>>>   #define amdgpu_gmc_override_vm_pte_flags(adev, vm, addr, pte_flags)    \
>>>       (adev)->gmc.gmc_funcs->override_vm_pte_flags            \
>>>           ((adev), (vm), (addr), (pte_flags))
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>>> index 0cc73982ed6f9..a541173e34a4e 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>>> @@ -551,6 +551,74 @@ static void gmc_v12_0_get_vm_pte(struct amdgpu_device *adev,
>>>           *flags = AMDGPU_PTE_MTYPE_GFX12(*flags, MTYPE_UC);
>>>   }
>>>   +/*
>>> + * Mirror of the GFX12 cases of svm_range_get_pte_flags(), with the
>>> + * per IP logic kept here in the HW specific file.
>>> + */
>>> +static uint64_t
>>> +gmc_v12_0_get_svm_pte_flags(struct amdgpu_device *adev,
>>> +                struct amdgpu_vm *vm,
>>> +                const struct amdgpu_svm_pte_flags_params *params)
>>> +{
>>> +    uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
>>> +    bool coherent = params->coherent;
>>> +    bool ext_coherent = params->ext_coherent;
>>> +    bool is_local = params->is_local;
>>> +    bool is_vram = params->is_vram;
>>> +    bool snoop = !is_vram;
>>> +    uint32_t mapping_flags = 0;
>>> +    unsigned int mtype_local, mtype_remote;
>>> +    bool is_aid_a1;
>>> +    uint64_t pte_flags;
>>> +
>>> +    switch (gc_ip_version) {
>>> +    case IP_VERSION(12, 0, 0):
>>> +    case IP_VERSION(12, 0, 1):
>>> +        mapping_flags |= AMDGPU_VM_MTYPE_NC;
>>> +        break;
>>> +    case IP_VERSION(12, 1, 0):
>>> +        is_aid_a1 = (adev->rev_id & 0x10);
>>> +        mtype_local = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_RW :
>>> +                amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
>>> +                is_aid_a1 ? AMDGPU_VM_MTYPE_RW : AMDGPU_VM_MTYPE_NC;
>>> +        mtype_remote = is_aid_a1 ? AMDGPU_VM_MTYPE_NC : AMDGPU_VM_MTYPE_UC;
>>> +        snoop = true;
>>> +
>>> +        if (is_local) {
>>> +            mapping_flags |= mtype_local;
>>> +        } else if (ext_coherent) {
>>> +            mapping_flags |= AMDGPU_VM_MTYPE_UC;
>>> +        } else {
>>> +            /* system memory or remote VRAM */
>>> +            mapping_flags |= mtype_remote;
>>> +        }
>>> +        break;
>>> +    default:
>>> +        mapping_flags |= coherent ?
>>> +            AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +        break;
>>> +    }
>>> +
>>> +    if (params->gpu_exec)
>>> +        mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
>>> +
>>> +    pte_flags = AMDGPU_PTE_VALID;
>>> +    pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
>>> +    pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
>>> +    pte_flags |= AMDGPU_PTE_IS_PTE;
>>> +
>>> +    gmc_v12_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
>>> +    pte_flags |= AMDGPU_PTE_READABLE;
>>> +    if (!params->gpu_ro)
>>> +        pte_flags |= AMDGPU_PTE_WRITEABLE;
>>> +
>>> +    if (gc_ip_version == IP_VERSION(12, 1, 0) &&
>>> +        adev->have_atomics_support)
>>> +        pte_flags |= AMDGPU_PTE_BUS_ATOMICS;
>>> +
>>> +    return pte_flags;
>>> +}
>>> +
>>>   static unsigned gmc_v12_0_get_vbios_fb_size(struct amdgpu_device *adev)
>>>   {
>>>       return 0;
>>> @@ -580,6 +648,7 @@ static const struct amdgpu_gmc_funcs gmc_v12_0_gmc_funcs = {
>>>       .emit_pasid_mapping = gmc_v12_0_emit_pasid_mapping,
>>>       .get_vm_pde = gmc_v12_0_get_vm_pde,
>>>       .get_vm_pte = gmc_v12_0_get_vm_pte,
>>> +    .get_svm_pte_flags = gmc_v12_0_get_svm_pte_flags,
>>>       .get_vbios_fb_size = gmc_v12_0_get_vbios_fb_size,
>>>       .get_dcc_alignment = gmc_v12_0_get_dcc_alignment,
>>>   };
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>> index 633772fa71fda..a3a5fa048a393 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>> @@ -1199,6 +1199,124 @@ static void gmc_v9_0_get_vm_pte(struct amdgpu_device *adev,
>>>           gmc_v9_0_get_coherence_flags(adev, vm, bo, vm_flags, flags);
>>>   }
>>>   +/*
>>> + * Mirror of the GFX9 cases of svm_range_get_pte_flags(), with the
>>> + * per IP logic kept here in the HW specific file.
>>> + */
>>> +static uint64_t
>>> +gmc_v9_0_get_svm_pte_flags(struct amdgpu_device *adev,
>>> +               struct amdgpu_vm *vm,
>>> +               const struct amdgpu_svm_pte_flags_params *params)
>>> +{
>>> +    uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
>>> +    bool coherent = params->coherent;
>>> +    bool ext_coherent = params->ext_coherent;
>>> +    bool is_local = params->is_local;
>>> +    bool is_vram = params->is_vram;
>>> +    bool same_hive = params->same_hive;
>>> +    bool snoop = !is_vram;
>>> +    uint32_t mapping_flags = 0;
>>> +    unsigned int mtype_local;
>>> +    uint64_t pte_flags;
>>> +
>>> +    switch (gc_ip_version) {
>>> +    case IP_VERSION(9, 4, 1):
>>> +        if (is_vram) {
>>> +            if (is_local) {
>>> +                mapping_flags |= coherent ?
>>> +                    AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
>>> +            } else {
>>> +                mapping_flags |= coherent ?
>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +                if (same_hive)
>>> +                    snoop = true;
>>> +            }
>>> +        } else {
>>> +            mapping_flags |= coherent ?
>>> +                AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +        }
>>> +        break;
>>> +    case IP_VERSION(9, 4, 2):
>>> +        if (is_vram) {
>>> +            if (is_local) {
>>> +                mapping_flags |= coherent ?
>>> +                    AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
>>> +                if (adev->gmc.xgmi.connected_to_cpu)
>>> +                    snoop = true;
>>> +            } else {
>>> +                mapping_flags |= coherent ?
>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +                if (same_hive)
>>> +                    snoop = true;
>>> +            }
>>> +        } else {
>>> +            mapping_flags |= coherent ?
>>> +                AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +        }
>>> +        break;
>>> +    case IP_VERSION(9, 4, 3):
>>> +    case IP_VERSION(9, 4, 4):
>>> +    case IP_VERSION(9, 5, 0):
>>> +        if (ext_coherent)
>>> +            mtype_local = AMDGPU_VM_MTYPE_CC;
>>> +        else
>>> +            mtype_local = amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
>>> +                amdgpu_mtype_local == 2 ? AMDGPU_VM_MTYPE_CC :
>>> +                AMDGPU_VM_MTYPE_RW;
>>> +        snoop = true;
>>> +        if (is_vram) {
>>> +            /* local HBM region close to partition */
>>> +            if (is_local)
>>> +                mapping_flags |= mtype_local;
>>> +            /* local HBM region far from partition or remote XGMI GPU
>>> +             * with regular system scope coherence
>>> +             */
>>> +            else if (same_hive && !ext_coherent)
>>> +                mapping_flags |= AMDGPU_VM_MTYPE_NC;
>>> +            /* PCIe P2P on GPUs pre-9.5.0 */
>>> +            else if (gc_ip_version < IP_VERSION(9, 5, 0) && !same_hive)
>>> +                mapping_flags |= AMDGPU_VM_MTYPE_UC;
>>> +            /* Other remote memory */
>>> +            else
>>> +                mapping_flags |= ext_coherent ?
>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +        } else if (adev->flags & AMD_IS_APU) {
>>> +            /* On NUMA systems, locality is determined per-page
>>> +             * in gmc_v9_0_override_vm_pte_flags.
>>> +             */
>>> +            if (num_possible_nodes() <= 1)
>>> +                mapping_flags |= mtype_local;
>>> +            else
>>> +                mapping_flags |= ext_coherent ?
>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +        } else {
>>> +            if (gc_ip_version < IP_VERSION(9, 5, 0) || ext_coherent)
>>> +                mapping_flags |= AMDGPU_VM_MTYPE_UC;
>>> +            else
>>> +                mapping_flags |= AMDGPU_VM_MTYPE_NC;
>>> +        }
>>> +        break;
>>> +    default:
>>> +        mapping_flags |= coherent ?
>>> +            AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>> +        break;
>>> +    }
>>> +
>>> +    if (params->gpu_exec)
>>> +        mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
>>> +
>>> +    pte_flags = AMDGPU_PTE_VALID;
>>> +    pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
>>> +    pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
>>> +
>>> +    gmc_v9_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
>>> +    pte_flags |= AMDGPU_PTE_READABLE;
>>> +    if (!params->gpu_ro)
>>> +        pte_flags |= AMDGPU_PTE_WRITEABLE;
>>> +
>>> +    return pte_flags;
>>> +}
>>> +
>>>   static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev,
>>>                          struct amdgpu_vm *vm,
>>>                          uint64_t addr, uint64_t *flags)
>>> @@ -1316,6 +1434,7 @@ static const struct amdgpu_gmc_funcs gmc_v9_0_gmc_funcs = {
>>>       .emit_pasid_mapping = gmc_v9_0_emit_pasid_mapping,
>>>       .get_vm_pde = gmc_v9_0_get_vm_pde,
>>>       .get_vm_pte = gmc_v9_0_get_vm_pte,
>>> +    .get_svm_pte_flags = gmc_v9_0_get_svm_pte_flags,
>>>       .override_vm_pte_flags = gmc_v9_0_override_vm_pte_flags,
>>>       .get_vbios_fb_size = gmc_v9_0_get_vbios_fb_size,
>>>       .query_mem_partition_mode = &amdgpu_gmc_query_memory_partition,
>>
> 

Re: [PATCH v10 06/22] drm/amdgpu/gmc: add get_svm_pte_flags callback

[Huang, Honglei]

On 11/10/2026 6:01 PM, Christian König wrote:
> On 11/10/26 07:57, Huang, Honglei wrote:
>>
>>
>> On 11/9/2026 9:52 PM, Christian König wrote:
>>> On 10/21/26 04:58, Huang Rui wrote:
>>>> From: Honglei Huang <honghuan@amd.com>
>>>>
>>>> Add a gmc_funcs callback for computing the GPU PTE flags of an SVM
>>>> mapping, plus a HW-agnostic input struct (amdgpu_svm_pte_flags_params)
>>>> and the amdgpu_gmc_get_svm_pte_flags() wrapper. This lets the per-IP
>>>> MTYPE/PTE selection live in the HW IP specific gmc_vX_0.c files instead
>>>> of the SVM core, mirroring how get_vm_pte handles BO mappings.
>>>>
>>>> Suggested-by: Christian König <christian.koenig@amd.com>
>>>> Suggested-by: Huang Rui <ray.huang@amd.com>
>>>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>>>> ---
>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  24 +++++
>>>>    drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c  |  69 ++++++++++++++
>>>>    drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 119 ++++++++++++++++++++++++
>>>>    3 files changed, 212 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>> index cffd5ca3fc66b..3f668fb71a632 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>>> @@ -150,6 +150,24 @@ struct amdgpu_vmhub {
>>>>        const struct amdgpu_vmhub_funcs *vmhub_funcs;
>>>>    };
>>>>    +/* Decoded inputs for the SVM PTE flag selection, gmc_funcs::get_svm_pte_flags */
>>>> +struct amdgpu_svm_pte_flags_params {
>>>> +    /* SVM attribute: coherent */
>>>> +    bool coherent;
>>>> +    /* SVM attribute: extended coherent */
>>>> +    bool ext_coherent;
>>>> +    /* SVM attribute: GPU read only */
>>>> +    bool gpu_ro;
>>>> +    /* SVM attribute: GPU executable */
>>>> +    bool gpu_exec;
>>>> +    /* mapping is in VRAM local device or remote P2P */
>>>> +    bool is_vram;
>>>> +    /* mapping is in this GPU's local VRAM */
>>>> +    bool is_local;
>>>> +    /* remote VRAM BO is on a GPU in the same XGMI hive as this GPU. */
>>>> +    bool same_hive;
>>>> +};
>>>> +
>>>>    /*
>>>>     * GPU MC structures, functions & helpers
>>>>     */
>>>> @@ -178,6 +196,10 @@ struct amdgpu_gmc_funcs {
>>>>                   struct amdgpu_bo *bo,
>>>>                   uint32_t vm_flags,
>>>>                   uint64_t *pte_flags);
>>>> +    /* get the full pte flags for an SVM mapping */
>>>> +    uint64_t (*get_svm_pte_flags)(struct amdgpu_device *adev,
>>>> +                      struct amdgpu_vm *vm,
>>>> +                      const struct amdgpu_svm_pte_flags_params *params);
>>>
>>> That looks like a really really bad idea to me.
>>>
>>> Please add the SVM flags to the existing amdgpu_gmc_get_vm_pte() function.
>>>
>>> Additional to that you also need to modify the destination addr and not just the flags.
>>
>> Will change to existing amdgpu_gmc_get_vm_pte function
> 
> It might be necessary to rework the existing amdgpu_gmc_get_vm_pte function to get the BO flags as parameter instead of the BO itself so that it can be used for both use cases.
> 
> Maybe split that into a separate patch.

Got it, will split that into a new patch.

Regards,
Honglei

> 
> Regards,
> Christian.
> 
>>
>> Regards,
>> Honglei
>>
>>>
>>> Regards,
>>> Christian.
>>>
>>>>        /* override per-page pte flags */
>>>>        void (*override_vm_pte_flags)(struct amdgpu_device *dev,
>>>>                          struct amdgpu_vm *vm,
>>>> @@ -378,6 +400,8 @@ struct amdgpu_gmc {
>>>>    #define amdgpu_gmc_get_vm_pte(adev, vm, bo, vm_flags, pte_flags) \
>>>>        ((adev)->gmc.gmc_funcs->get_vm_pte((adev), (vm), (bo), (vm_flags), \
>>>>                           (pte_flags)))
>>>> +#define amdgpu_gmc_get_svm_pte_flags(adev, vm, params) \
>>>> +    ((adev)->gmc.gmc_funcs->get_svm_pte_flags((adev), (vm), (params)))
>>>>    #define amdgpu_gmc_override_vm_pte_flags(adev, vm, addr, pte_flags)    \
>>>>        (adev)->gmc.gmc_funcs->override_vm_pte_flags            \
>>>>            ((adev), (vm), (addr), (pte_flags))
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>>>> index 0cc73982ed6f9..a541173e34a4e 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c
>>>> @@ -551,6 +551,74 @@ static void gmc_v12_0_get_vm_pte(struct amdgpu_device *adev,
>>>>            *flags = AMDGPU_PTE_MTYPE_GFX12(*flags, MTYPE_UC);
>>>>    }
>>>>    +/*
>>>> + * Mirror of the GFX12 cases of svm_range_get_pte_flags(), with the
>>>> + * per IP logic kept here in the HW specific file.
>>>> + */
>>>> +static uint64_t
>>>> +gmc_v12_0_get_svm_pte_flags(struct amdgpu_device *adev,
>>>> +                struct amdgpu_vm *vm,
>>>> +                const struct amdgpu_svm_pte_flags_params *params)
>>>> +{
>>>> +    uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
>>>> +    bool coherent = params->coherent;
>>>> +    bool ext_coherent = params->ext_coherent;
>>>> +    bool is_local = params->is_local;
>>>> +    bool is_vram = params->is_vram;
>>>> +    bool snoop = !is_vram;
>>>> +    uint32_t mapping_flags = 0;
>>>> +    unsigned int mtype_local, mtype_remote;
>>>> +    bool is_aid_a1;
>>>> +    uint64_t pte_flags;
>>>> +
>>>> +    switch (gc_ip_version) {
>>>> +    case IP_VERSION(12, 0, 0):
>>>> +    case IP_VERSION(12, 0, 1):
>>>> +        mapping_flags |= AMDGPU_VM_MTYPE_NC;
>>>> +        break;
>>>> +    case IP_VERSION(12, 1, 0):
>>>> +        is_aid_a1 = (adev->rev_id & 0x10);
>>>> +        mtype_local = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_RW :
>>>> +                amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
>>>> +                is_aid_a1 ? AMDGPU_VM_MTYPE_RW : AMDGPU_VM_MTYPE_NC;
>>>> +        mtype_remote = is_aid_a1 ? AMDGPU_VM_MTYPE_NC : AMDGPU_VM_MTYPE_UC;
>>>> +        snoop = true;
>>>> +
>>>> +        if (is_local) {
>>>> +            mapping_flags |= mtype_local;
>>>> +        } else if (ext_coherent) {
>>>> +            mapping_flags |= AMDGPU_VM_MTYPE_UC;
>>>> +        } else {
>>>> +            /* system memory or remote VRAM */
>>>> +            mapping_flags |= mtype_remote;
>>>> +        }
>>>> +        break;
>>>> +    default:
>>>> +        mapping_flags |= coherent ?
>>>> +            AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +        break;
>>>> +    }
>>>> +
>>>> +    if (params->gpu_exec)
>>>> +        mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
>>>> +
>>>> +    pte_flags = AMDGPU_PTE_VALID;
>>>> +    pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
>>>> +    pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
>>>> +    pte_flags |= AMDGPU_PTE_IS_PTE;
>>>> +
>>>> +    gmc_v12_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
>>>> +    pte_flags |= AMDGPU_PTE_READABLE;
>>>> +    if (!params->gpu_ro)
>>>> +        pte_flags |= AMDGPU_PTE_WRITEABLE;
>>>> +
>>>> +    if (gc_ip_version == IP_VERSION(12, 1, 0) &&
>>>> +        adev->have_atomics_support)
>>>> +        pte_flags |= AMDGPU_PTE_BUS_ATOMICS;
>>>> +
>>>> +    return pte_flags;
>>>> +}
>>>> +
>>>>    static unsigned gmc_v12_0_get_vbios_fb_size(struct amdgpu_device *adev)
>>>>    {
>>>>        return 0;
>>>> @@ -580,6 +648,7 @@ static const struct amdgpu_gmc_funcs gmc_v12_0_gmc_funcs = {
>>>>        .emit_pasid_mapping = gmc_v12_0_emit_pasid_mapping,
>>>>        .get_vm_pde = gmc_v12_0_get_vm_pde,
>>>>        .get_vm_pte = gmc_v12_0_get_vm_pte,
>>>> +    .get_svm_pte_flags = gmc_v12_0_get_svm_pte_flags,
>>>>        .get_vbios_fb_size = gmc_v12_0_get_vbios_fb_size,
>>>>        .get_dcc_alignment = gmc_v12_0_get_dcc_alignment,
>>>>    };
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>> index 633772fa71fda..a3a5fa048a393 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>> @@ -1199,6 +1199,124 @@ static void gmc_v9_0_get_vm_pte(struct amdgpu_device *adev,
>>>>            gmc_v9_0_get_coherence_flags(adev, vm, bo, vm_flags, flags);
>>>>    }
>>>>    +/*
>>>> + * Mirror of the GFX9 cases of svm_range_get_pte_flags(), with the
>>>> + * per IP logic kept here in the HW specific file.
>>>> + */
>>>> +static uint64_t
>>>> +gmc_v9_0_get_svm_pte_flags(struct amdgpu_device *adev,
>>>> +               struct amdgpu_vm *vm,
>>>> +               const struct amdgpu_svm_pte_flags_params *params)
>>>> +{
>>>> +    uint32_t gc_ip_version = amdgpu_ip_version(adev, GC_HWIP, 0);
>>>> +    bool coherent = params->coherent;
>>>> +    bool ext_coherent = params->ext_coherent;
>>>> +    bool is_local = params->is_local;
>>>> +    bool is_vram = params->is_vram;
>>>> +    bool same_hive = params->same_hive;
>>>> +    bool snoop = !is_vram;
>>>> +    uint32_t mapping_flags = 0;
>>>> +    unsigned int mtype_local;
>>>> +    uint64_t pte_flags;
>>>> +
>>>> +    switch (gc_ip_version) {
>>>> +    case IP_VERSION(9, 4, 1):
>>>> +        if (is_vram) {
>>>> +            if (is_local) {
>>>> +                mapping_flags |= coherent ?
>>>> +                    AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
>>>> +            } else {
>>>> +                mapping_flags |= coherent ?
>>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +                if (same_hive)
>>>> +                    snoop = true;
>>>> +            }
>>>> +        } else {
>>>> +            mapping_flags |= coherent ?
>>>> +                AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +        }
>>>> +        break;
>>>> +    case IP_VERSION(9, 4, 2):
>>>> +        if (is_vram) {
>>>> +            if (is_local) {
>>>> +                mapping_flags |= coherent ?
>>>> +                    AMDGPU_VM_MTYPE_CC : AMDGPU_VM_MTYPE_RW;
>>>> +                if (adev->gmc.xgmi.connected_to_cpu)
>>>> +                    snoop = true;
>>>> +            } else {
>>>> +                mapping_flags |= coherent ?
>>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +                if (same_hive)
>>>> +                    snoop = true;
>>>> +            }
>>>> +        } else {
>>>> +            mapping_flags |= coherent ?
>>>> +                AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +        }
>>>> +        break;
>>>> +    case IP_VERSION(9, 4, 3):
>>>> +    case IP_VERSION(9, 4, 4):
>>>> +    case IP_VERSION(9, 5, 0):
>>>> +        if (ext_coherent)
>>>> +            mtype_local = AMDGPU_VM_MTYPE_CC;
>>>> +        else
>>>> +            mtype_local = amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
>>>> +                amdgpu_mtype_local == 2 ? AMDGPU_VM_MTYPE_CC :
>>>> +                AMDGPU_VM_MTYPE_RW;
>>>> +        snoop = true;
>>>> +        if (is_vram) {
>>>> +            /* local HBM region close to partition */
>>>> +            if (is_local)
>>>> +                mapping_flags |= mtype_local;
>>>> +            /* local HBM region far from partition or remote XGMI GPU
>>>> +             * with regular system scope coherence
>>>> +             */
>>>> +            else if (same_hive && !ext_coherent)
>>>> +                mapping_flags |= AMDGPU_VM_MTYPE_NC;
>>>> +            /* PCIe P2P on GPUs pre-9.5.0 */
>>>> +            else if (gc_ip_version < IP_VERSION(9, 5, 0) && !same_hive)
>>>> +                mapping_flags |= AMDGPU_VM_MTYPE_UC;
>>>> +            /* Other remote memory */
>>>> +            else
>>>> +                mapping_flags |= ext_coherent ?
>>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +        } else if (adev->flags & AMD_IS_APU) {
>>>> +            /* On NUMA systems, locality is determined per-page
>>>> +             * in gmc_v9_0_override_vm_pte_flags.
>>>> +             */
>>>> +            if (num_possible_nodes() <= 1)
>>>> +                mapping_flags |= mtype_local;
>>>> +            else
>>>> +                mapping_flags |= ext_coherent ?
>>>> +                    AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +        } else {
>>>> +            if (gc_ip_version < IP_VERSION(9, 5, 0) || ext_coherent)
>>>> +                mapping_flags |= AMDGPU_VM_MTYPE_UC;
>>>> +            else
>>>> +                mapping_flags |= AMDGPU_VM_MTYPE_NC;
>>>> +        }
>>>> +        break;
>>>> +    default:
>>>> +        mapping_flags |= coherent ?
>>>> +            AMDGPU_VM_MTYPE_UC : AMDGPU_VM_MTYPE_NC;
>>>> +        break;
>>>> +    }
>>>> +
>>>> +    if (params->gpu_exec)
>>>> +        mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE;
>>>> +
>>>> +    pte_flags = AMDGPU_PTE_VALID;
>>>> +    pte_flags |= is_vram ? 0 : AMDGPU_PTE_SYSTEM;
>>>> +    pte_flags |= snoop ? AMDGPU_PTE_SNOOPED : 0;
>>>> +
>>>> +    gmc_v9_0_get_vm_pte(adev, vm, NULL, mapping_flags, &pte_flags);
>>>> +    pte_flags |= AMDGPU_PTE_READABLE;
>>>> +    if (!params->gpu_ro)
>>>> +        pte_flags |= AMDGPU_PTE_WRITEABLE;
>>>> +
>>>> +    return pte_flags;
>>>> +}
>>>> +
>>>>    static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev,
>>>>                           struct amdgpu_vm *vm,
>>>>                           uint64_t addr, uint64_t *flags)
>>>> @@ -1316,6 +1434,7 @@ static const struct amdgpu_gmc_funcs gmc_v9_0_gmc_funcs = {
>>>>        .emit_pasid_mapping = gmc_v9_0_emit_pasid_mapping,
>>>>        .get_vm_pde = gmc_v9_0_get_vm_pde,
>>>>        .get_vm_pte = gmc_v9_0_get_vm_pte,
>>>> +    .get_svm_pte_flags = gmc_v9_0_get_svm_pte_flags,
>>>>        .override_vm_pte_flags = gmc_v9_0_override_vm_pte_flags,
>>>>        .get_vbios_fb_size = gmc_v9_0_get_vbios_fb_size,
>>>>        .query_mem_partition_mode = &amdgpu_gmc_query_memory_partition,
>>>
>>
> 

Re: [PATCH v10 02/22] drm/amdgpu: add SVM core header and VM integration

[Kuehling, Felix]

On 2026-11-09 07:33, Christian König wrote:
> On 10/21/26 04:58, Huang Rui wrote:
>> From: Honglei Huang <honghuan@amd.com>
>>
>> Add amdgpu_svm.h with SVM data structures and public API:
>> - enum amdgpu_svm_xnack_mode: OFF, ON and DEFAULT mode
>> - struct amdgpu_svm: core context with drm_gpusvm, kref lifecycle,
>>    attribute tree, rw_semaphore, GC workqueue, xnack state.
>> - struct amdgpu_svm_gc: garbage collector with workqueue and work_struct
>> - Kmem cache helpers: AMDGPU_SVM_KMEM_CACHE_CREATE/DESTROY
>> - Locking helpers: amdgpu_svm_lock/unlock/assert_locked
>> - Public API declarations (including clean_queue, sync_work)
>>
>> Suggested-by: Christian König <christian.koenig@amd.com>
>> Suggested-by: Huang Rui <ray.huang@amd.com>
>> Signed-off-by: Honglei Huang <honghuan@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 195 ++++++++++++++++++++++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h  |   4 +
>>   2 files changed, 199 insertions(+)
>>   create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> new file mode 100644
>> index 0000000000000..72f954d2df39e
>> --- /dev/null
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
>> @@ -0,0 +1,195 @@
>> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
>> +/*
>> + * Copyright 2026 Advanced Micro Devices, Inc.
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice shall be included in
>> + * all copies or substantial portions of the Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
>> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
>> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>> + * OTHER DEALINGS IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#ifndef __AMDGPU_SVM_H__
>> +#define __AMDGPU_SVM_H__
>> +
>> +#include <drm/amdgpu_drm.h>
>> +#include <drm/drm_gpusvm.h>
>> +#include <linux/atomic.h>
>> +#include <linux/kref.h>
>> +#include <linux/list.h>
>> +#include <linux/printk.h>
>> +#include <linux/rwsem.h>
>> +#include <linux/slab.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/types.h>
>> +#include <linux/workqueue.h>
>> +
>> +struct amdgpu_device;
>> +struct amdgpu_vm;
>> +struct amdgpu_svm_attr_tree;
>> +struct amdgpu_svm_attrs;
>> +struct drm_device;
>> +struct drm_file;
>> +
>> +enum amdgpu_svm_xnack_mode {
>> +	AMDGPU_SVM_XNACK_OFF,
>> +	AMDGPU_SVM_XNACK_ON,
>> +	AMDGPU_SVM_XNACK_DEFAULT,
>> +};
>> +
>> +#define amdgpu_svm_assert_in_notifier(svm__) \
>> +	lockdep_assert_held_write(&(svm__)->gpusvm.notifier_lock)
>> +
>> +/**
>> + * struct amdgpu_svm_gc - SVM range garbage collector used for unmapping
>> + * SVM ranges in response to mmu_notifier events.
>> + * @wq: Workqueue where @work runs on.
>> + * @list: Ranges list in garbage collector. Protected by amdgpu_svm.work_lock.
>> + * @work: Drains @list under amdgpu_svm.svm_lock.
>> + */
>> +struct amdgpu_svm_gc {
>> +	struct workqueue_struct *wq;
>> +	struct list_head list;
>> +	struct work_struct work;
>> +};
>> +
>> +/**
>> + * struct amdgpu_svm - shared virtual memory context
>> + * @gpusvm: Embedded drm_gpusvm base, manages the range/notifier trees.
>> + * @refcount: Reference count guarding the lifetime of this instance.
>> + * @adev: The amdgpu device this SVM context belongs to.
>> + * @vm: Back pointer to the owning amdgpu_vm.
>> + * @attr_tree: Interval tree of SVM attributes.
>> + * @svm_lock: The driver_svm_lock described in the GPU SVM locking section.
>> + *            registered by drm_gpusvm_driver_set_lock().
>> + * @work_lock: Protects the deferred work queues such as @gc.list.
>> + * @gc: garbage collector for ranges queued for destruction.
>> + * @exiting: Set once when the context starts tearing down.
>> + * @checkpoint_ts: IH ring timestamp captured on unmap, used to drop stale
>> + *                 faults.
>> + * @default_granularity: Default range granularity.
>> + * @xnack_enabled: Whether retry (XNACK) faults are enabled.
>> + * @invalidate_ranges: Callback of the MMU notifier invalidate event.
>> + */
>> +struct amdgpu_svm {
>> +	struct drm_gpusvm gpusvm;
>> +	struct kref refcount;
>> +	struct amdgpu_device *adev;
>> +	struct amdgpu_vm *vm;
>> +	struct amdgpu_svm_attr_tree *attr_tree;
>> +	struct rw_semaphore svm_lock;
>> +	spinlock_t work_lock;
>> +	struct amdgpu_svm_gc gc;
>> +	atomic_t exiting;
>> +	uint64_t checkpoint_ts;
> By now we have figured out that TS in faults are not useable for fault handling.
>
> Background is that they reset from time to time because of GFXOFF and may vary depending on the engine who sends the fault.
>
> Adding Timur from Valve on CC, he send out a patch to completely remove the existing timestamp handling from the fault filter.
>
> We need to come up with a different approach here.

At least for the fault filter that shouldn't be a big problem. We don't 
need the SW fault filter on any GPU that has a CAM. That should be any 
GPU that supports retry faults starting with Vega20. AFAIK the only GPUs 
that still need the SW fault filter are Vega10 and Raven. If the 
timestamps work on those, we're fine. If there are any newer GPUs left 
that don't enable the CAM for whatever reason, let's fix that instead.

Regards,
   Felix


>
>> +	u8 default_granularity;
>> +	bool xnack_enabled;
> It's probably ok to have that here for now, but that will become a VM property in the future I think.
>
>> +	void (*invalidate_ranges)(struct amdgpu_svm *svm,
>> +				  struct drm_gpusvm_notifier *notifier,
>> +				  const struct mmu_notifier_range *mmu_range,
>> +				  struct drm_gpusvm_range *first,
>> +				  uint64_t adj_start, uint64_t adj_end);
> That looks a bit questionable. Why is that a pointer inside the amdgpu_svm structure?
>
> Apart from those questions that looks rather reasonable.
>
> Regards,
> Christian.
>
>> +};
>> +
>> +static inline struct amdgpu_svm *to_amdgpu_svm(struct drm_gpusvm *gpusvm)
>> +{
>> +	return container_of(gpusvm, struct amdgpu_svm, gpusvm);
>> +}
>> +
>> +/*
>> + * Helpers for amdgpu_svm.svm_lock, the driver_svm_lock registered with GPU SVM.
>> + * Hold it in write mode around structural GPU SVM updates, including
>> + * drm_gpusvm_range_find_or_insert() and drm_gpusvm_range_remove().
>> + */
>> +static inline void amdgpu_svm_lock(struct amdgpu_svm *svm)
>> +{
>> +	down_write(&svm->svm_lock);
>> +}
>> +
>> +static inline void amdgpu_svm_unlock(struct amdgpu_svm *svm)
>> +{
>> +	up_write(&svm->svm_lock);
>> +}
>> +
>> +static inline void amdgpu_svm_assert_locked(struct amdgpu_svm *svm)
>> +{
>> +	lockdep_assert_held_write(&svm->svm_lock);
>> +}
>> +
>> +#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
>> +void amdgpu_svm_flush_tlb(struct amdgpu_svm *svm);
>> +
>> +int amdgpu_svm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
>> +void amdgpu_svm_close(struct amdgpu_vm *vm);
>> +void amdgpu_svm_fini(struct amdgpu_vm *vm);
>> +
>> +void amdgpu_svm_put(struct amdgpu_svm *svm);
>> +struct amdgpu_svm *amdgpu_svm_lookup_by_pasid(struct amdgpu_device *adev,
>> +					       uint32_t pasid);
>> +int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
>> +			    uint64_t fault_page, uint64_t ts,
>> +			    bool write_fault);
>> +bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm);
>> +
>> +int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>> +			 struct drm_file *filp);
>> +void amdgpu_svm_clean_queue(struct amdgpu_svm *svm,
>> +			    struct list_head *work_list);
>> +void amdgpu_svm_sync_work(struct amdgpu_svm *svm);
>> +int amdgpu_svm_garbage_collector(struct amdgpu_svm *svm);
>> +int amdgpu_svm_apply_attr_change(struct amdgpu_svm *svm,
>> +				 const struct amdgpu_svm_attrs *old_attrs,
>> +				 const struct amdgpu_svm_attrs *new_attrs,
>> +				 unsigned long start_page,
>> +				 unsigned long last_page);
>> +bool amdgpu_svm_devmem_possible(struct amdgpu_svm *svm);
>> +#else
>> +static inline int amdgpu_svm_init(struct amdgpu_device *adev,
>> +				  struct amdgpu_vm *vm)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void amdgpu_svm_close(struct amdgpu_vm *vm)
>> +{
>> +}
>> +
>> +static inline void amdgpu_svm_fini(struct amdgpu_vm *vm)
>> +{
>> +}
>> +
>> +static inline int amdgpu_svm_handle_fault(struct amdgpu_device *adev,
>> +					  uint32_t pasid,
>> +					  uint64_t fault_page,
>> +					  uint64_t ts,
>> +					  bool write_fault)
>> +{
>> +	return -EOPNOTSUPP;
>> +}
>> +
>> +static inline bool amdgpu_svm_is_enabled(struct amdgpu_vm *vm)
>> +{
>> +	return false;
>> +}
>> +
>> +static inline int amdgpu_gem_svm_ioctl(struct drm_device *dev, void *data,
>> +				       struct drm_file *filp)
>> +{
>> +	return -EOPNOTSUPP;
>> +}
>> +#endif /* CONFIG_DRM_AMDGPU_SVM */
>> +
>> +#endif /* __AMDGPU_SVM_H__ */
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> index 9b07df5a2b816..b435e393be154 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
>> @@ -43,6 +43,7 @@ struct amdgpu_bo_va;
>>   struct amdgpu_job;
>>   struct amdgpu_bo_list_entry;
>>   struct amdgpu_bo_vm;
>> +struct amdgpu_svm;
>>   
>>   /*
>>    * GPUVM handling
>> @@ -381,6 +382,9 @@ struct amdgpu_vm {
>>   
>>   	/* cached fault info */
>>   	struct amdgpu_vm_fault_info fault_info;
>> +
>> +	/* SVM experimental implementation */
>> +	struct amdgpu_svm *svm;
>>   };
>>   
>>   struct amdgpu_vm_manager {

Re: [PATCH v10 00/22] drm/amdgpu: AMDGPU SVM support based on DRM (Phase 1: single GPU, XNACK on)

[Huang, Honglei]

Hi Matt and Thomas,

We are working on amdgpu SVM based on drm svm.
Christian has a very full schedule at the moment. He mentioned your 
expertise in GPU mm and SVM, and suggested that us reach out to you for 
a review of our AMDGPU SVM work:

the v10 is in:
https://lore.kernel.org/amd-gfx/20261021025915.2433837-1-ray.huang@amd.com/

If you have some time, we would appreciate your review of the series.
Your feedback would be very valuable in helping us move this work forward.
Thank you for your time.

Regards,
Honglei