* [PATCH v3 0/5] drm/amdgpu: SVM VRAM migration via drm_pagemap
@ 2026-04-27 10:05 Junhua Shen
2026-04-27 10:05 ` [PATCH v3 1/5] drm/amdgpu: add VRAM migration infrastructure for drm_pagemap Junhua Shen
` (4 more replies)
0 siblings, 5 replies; 8+ messages in thread
From: Junhua Shen @ 2026-04-27 10:05 UTC (permalink / raw)
To: Alexander.Deucher, Felix.Kuehling, Christian.Koenig, Oak.Zeng,
Jenny-Jing.Liu, Philip.Yang, Xiaogang.Chen, Ray.Huang,
honglei1.huang, Lingshan.Zhu
Cc: amd-gfx, dri-devel, Junhua Shen
This series adds VRAM migration support to the amdgpu SVM (Shared
Virtual Memory) subsystem, built on top of the drm_pagemap
infrastructure [1].
It enables transparent page migration between system RAM and device
VRAM using SDMA, driven by userspace SVM attribute hints (prefetch
location, access attributes) through the existing AMDGPU SVM ioctl
interface.
Limitations:
- Single GPU only; multi-GPU migration is not addressed
- No XNACK-on GPU fault-driven migration (XNACK-off ioctl
path only)
- No VRAM-to-VRAM (peer GPU) migration
- No eviction fence / VRAM overcommit handling yet
Design highlights:
- ZONE_DEVICE pages managed via devm_memremap_pages / drm_pagemap
- SDMA-based migration with proper DMA fence synchronization
- Migration decision layer that evaluates SVM attributes to
determine when and where to migrate
- Zero modifications to the KFD subsystem
Patch breakdown:
1. Core VRAM migration infrastructure (ZONE_DEVICE, drm_pagemap_ops)
2. SDMA migration callbacks (copy_to_ram / copy_to_dev)
3. Migration decision layer (policy evaluation)
4. SVM attribute extensions (prefetch force-trigger)
5. Integration into SVM range map path + ZONE_DEVICE registration
Built on top of the drm_pagemap SVM series [1].
Changes since v2:
- Moved amdgpu_pagemap entirely to amdgpu side, eliminating all KFD
modifications
- Split commits for better reviewability: separated infrastructure
from SDMA callbacks, decision layer from integration
- Merged ZONE_DEVICE registration hook into the integration patch
Changes since v1:
- Dropped the eviction fence patch per Christian König's review
(violates dma_fence contract)
[1] https://lore.kernel.org/all/20260317-drm-svm-v2-0-4bceef04e41e@amd.com/
v1: https://lore.kernel.org/all/20260410113146.146212-1-Junhua.Shen@amd.com/
v2: https://lore.kernel.org/all/20260413103031.181953-1-Junhua.Shen@amd.com/
Junhua Shen (5):
drm/amdgpu: add VRAM migration infrastructure for drm_pagemap
drm/amdgpu: implement drm_pagemap SDMA migration callbacks
drm/amdgpu: introduce SVM range migration decision layer
drm/amdgpu: add SVM attr prefetch/force-trigger functionality
drm/amdgpu: integrate VRAM migration into SVM range map path
drivers/gpu/drm/amd/amdgpu/Makefile | 6 +-
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 8 +
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +
drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 789 ++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h | 98 +++
drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c | 4 +
drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 4 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 34 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 136 +--
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 5 +-
.../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 140 ++++
.../drm/amd/amdgpu/amdgpu_svm_range_migrate.h | 60 ++
12 files changed, 1212 insertions(+), 76 deletions(-)
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
--
2.34.1
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v3 1/5] drm/amdgpu: add VRAM migration infrastructure for drm_pagemap
2026-04-27 10:05 [PATCH v3 0/5] drm/amdgpu: SVM VRAM migration via drm_pagemap Junhua Shen
@ 2026-04-27 10:05 ` Junhua Shen
2026-04-27 10:05 ` [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks Junhua Shen
` (3 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: Junhua Shen @ 2026-04-27 10:05 UTC (permalink / raw)
To: Alexander.Deucher, Felix.Kuehling, Christian.Koenig, Oak.Zeng,
Jenny-Jing.Liu, Philip.Yang, Xiaogang.Chen, Ray.Huang,
honglei1.huang, Lingshan.Zhu
Cc: amd-gfx, dri-devel, Junhua Shen
Add the drm_pagemap-based VRAM migration infrastructure:
- Define struct amdgpu_pagemap wrapping dev_pagemap + drm_pagemap
- Define AMDGPU_SVM_PGMAP_OWNER() and AMDGPU_INTERCONNECT_VRAM macros
- Implement amdgpu_svm_migration_init() to register ZONE_DEVICE via
devm_memremap_pages() and initialize the drm_pagemap
- Add amdgpu_pagemap pointer (apagemap) to struct amdgpu_device
Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
drivers/gpu/drm/amd/amdgpu/Makefile | 6 +-
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 8 +
drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 179 ++++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h | 98 +++++++++++
4 files changed, 289 insertions(+), 2 deletions(-)
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index 7700f81a246e..e64abb5c8ab8 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -323,12 +323,14 @@ amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
# svm support
amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
- amdgpu_svm_range.o
+ amdgpu_svm_range.o amdgpu_migrate.o
.PHONY: clean-svm
clean-svm:
rm -f $(obj)/amdgpu_svm.o $(obj)/amdgpu_svm_attr.o $(obj)/amdgpu_svm_range.o \
- $(obj)/.amdgpu_svm.o.cmd $(obj)/.amdgpu_svm_attr.o.cmd $(obj)/.amdgpu_svm_range.o.cmd
+ $(obj)/amdgpu_migrate.o \
+ $(obj)/.amdgpu_svm.o.cmd $(obj)/.amdgpu_svm_attr.o.cmd $(obj)/.amdgpu_svm_range.o.cmd \
+ $(obj)/.amdgpu_migrate.o.cmd
include $(FULL_AMD_PATH)/pm/Makefile
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 49e7881750fa..fe6ba9911d9f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -325,6 +325,7 @@ struct amdgpu_fpriv;
struct amdgpu_bo_va_mapping;
struct kfd_vm_fault_info;
struct amdgpu_hive_info;
+struct amdgpu_pagemap;
struct amdgpu_reset_context;
struct amdgpu_reset_control;
struct amdgpu_coredump_info;
@@ -1200,6 +1201,13 @@ struct amdgpu_device {
struct amdgpu_uma_carveout_info uma_info;
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+ /* SVM VRAM migration via drm_pagemap (drm_gpusvm path).
+ * Allocated in amdgpu_svm_migration_init(), NULL if SVM disabled.
+ */
+ struct amdgpu_pagemap *apagemap;
+#endif
+
/* KFD
* Must be last --ends in a flexible-array member.
*/
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
new file mode 100644
index 000000000000..170e2eadc106
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -0,0 +1,179 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+/**
+ * DOC: AMDGPU SVM Migration
+ *
+ * This file implements the drm_pagemap-based migration infrastructure for
+ * AMDGPU SVM. It provides the callbacks that the DRM GPUSVM / drm_pagemap
+ * framework needs to:
+ *
+ * 1. Map ZONE_DEVICE pages to GPU-visible VRAM MC addresses (device_map)
+ * 2. Allocate VRAM and migrate pages from system memory (populate_mm)
+ * 3. Copy data between RAM and VRAM using SDMA (copy_to_devmem / copy_to_ram)
+ * 4. Release VRAM backing when pages migrate back to system memory (devmem_release)
+ *
+ * Architecture overview::
+ *
+ * adev->apagemap->dpagemap (struct drm_pagemap)
+ * .ops = &amdgpu_svm_drm_pagemap_ops
+ * |
+ * +---+-------------------+
+ * | |
+ * .populate_mm .device_map
+ * (alloc BO + migrate) (page -> VRAM MC addr)
+ * |
+ * v
+ * drm_pagemap_devmem_ops (per-BO migration mechanics)
+ * .populate_devmem_pfn -> BO buddy blocks -> PFN array
+ * .copy_to_devmem -> SDMA copy RAM -> VRAM
+ * .copy_to_ram -> SDMA copy VRAM -> RAM
+ * .devmem_release -> release BO reference
+ *
+ * The three address spaces involved:
+ *
+ * VRAM offset [0, real_vram_size) - buddy allocator managed
+ * + hpa_base
+ * HPA / PFN [hpa_base, hpa_base+..) - ZONE_DEVICE struct page management
+ * + vm_manager.vram_base_offset
+ * PTE address [vram_base_offset, ..] - GPU page table entries (from MMHUB FB_OFFSET)
+ */
+
+#include <drm/drm_pagemap.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+
+#include "amdgpu_amdkfd.h"
+#include "amdgpu_migrate.h"
+#include "amdgpu.h"
+
+static inline struct amdgpu_pagemap *
+dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
+{
+ return container_of(dpagemap, struct amdgpu_pagemap, dpagemap);
+}
+
+static inline struct amdgpu_device *
+dpagemap_to_adev(struct drm_pagemap *dpagemap)
+{
+ return drm_to_adev(dpagemap->drm);
+}
+
+/**
+ * amdgpu_svm_page_to_apagemap - Get amdgpu_pagemap from a ZONE_DEVICE page
+ * @page: A ZONE_DEVICE page backed by VRAM
+ *
+ * Follows: page -> pgmap -> container_of(apagemap)
+ */
+static inline struct amdgpu_pagemap *
+amdgpu_svm_page_to_apagemap(struct page *page)
+{
+ struct dev_pagemap *pgmap = page_pgmap(page);
+
+ return container_of(pgmap, struct amdgpu_pagemap, pgmap);
+}
+
+
+const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = { };
+
+/**
+ * amdgpu_svm_migration_init - Register ZONE_DEVICE and initialize drm_pagemap
+ * @adev: AMDGPU device to set up VRAM migration for
+ *
+ * Allocates a ZONE_DEVICE region covering the GPU's VRAM, registers it
+ * via devm_memremap_pages() with drm_pagemap's generic dev_pagemap_ops,
+ * and then initializes the drm_pagemap (dpagemap) that provides the
+ * device_map / populate_mm callbacks for the DRM GPUSVM migration path.
+ *
+ * For XGMI-connected CPUs, uses the device's aperture directly
+ * (MEMORY_DEVICE_COHERENT). For discrete GPUs, requests a free
+ * iomem region for MEMORY_DEVICE_PRIVATE pages.
+ *
+ * Return: 0 on success, -EINVAL if GPU IP too old, negative error on failure
+ */
+int amdgpu_svm_migration_init(struct amdgpu_device *adev)
+{
+ struct amdgpu_pagemap *svm_dm;
+ struct dev_pagemap *pgmap;
+ struct resource *res = NULL;
+ unsigned long size;
+ void *r;
+
+ if (amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(9, 0, 1))
+ return -EINVAL;
+
+ if (adev->apu_prefer_gtt)
+ return 0;
+
+ if (adev->apagemap && adev->apagemap->initialized)
+ return 0;
+
+ svm_dm = devm_kzalloc(adev->dev, sizeof(*svm_dm), GFP_KERNEL);
+ if (!svm_dm)
+ return -ENOMEM;
+
+ pgmap = &svm_dm->pgmap;
+
+ size = ALIGN(adev->gmc.real_vram_size, 2ULL << 20);
+ if (adev->gmc.xgmi.connected_to_cpu) {
+ pgmap->range.start = adev->gmc.aper_base;
+ pgmap->range.end = adev->gmc.aper_base + adev->gmc.aper_size - 1;
+ pgmap->type = MEMORY_DEVICE_COHERENT;
+ } else {
+ res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
+ if (IS_ERR(res))
+ return PTR_ERR(res);
+ pgmap->range.start = res->start;
+ pgmap->range.end = res->end;
+ pgmap->type = MEMORY_DEVICE_PRIVATE;
+ }
+
+ pgmap->nr_range = 1;
+ pgmap->flags = 0;
+ pgmap->ops = drm_pagemap_pagemap_ops_get();
+ pgmap->owner = AMDGPU_SVM_PGMAP_OWNER(adev);
+
+ r = devm_memremap_pages(adev->dev, pgmap);
+ if (IS_ERR(r)) {
+ dev_err(adev->dev, "SVM: failed to register HMM device memory\n");
+ if (pgmap->type == MEMORY_DEVICE_PRIVATE && res)
+ devm_release_mem_region(adev->dev, res->start, resource_size(res));
+ pgmap->type = 0;
+ return PTR_ERR(r);
+ }
+
+ if (drm_pagemap_init(&svm_dm->dpagemap, pgmap, adev_to_drm(adev),
+ &amdgpu_svm_drm_pagemap_ops)) {
+ dev_err(adev->dev, "SVM: failed to init drm_pagemap\n");
+ return -EINVAL;
+ }
+ svm_dm->adev = adev;
+ svm_dm->hpa_base = pgmap->range.start;
+ svm_dm->initialized = true;
+ adev->apagemap = svm_dm;
+
+ dev_info(adev->dev, "SVM: registered %ldMB device memory, hpa_base=0x%llx\n",
+ size >> 20, svm_dm->hpa_base);
+ return 0;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
new file mode 100644
index 000000000000..e20698fb1597
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_MIGRATE_H__
+#define __AMDGPU_MIGRATE_H__
+
+#include <drm/drm_pagemap.h>
+#include <linux/memremap.h>
+
+struct amdgpu_device;
+
+/*
+ * AMDGPU_INTERCONNECT_VRAM - Protocol identifier for local VRAM access.
+ *
+ * Used in drm_pagemap_addr to distinguish device-local VRAM addresses from
+ * DMA-mapped system memory addresses. drm_gpusvm_get_pages() uses this to
+ * identify pages that are already in device memory and need no DMA mapping.
+ *
+ * Value must be non-zero (0 == DRM_INTERCONNECT_SYSTEM).
+ */
+#define AMDGPU_INTERCONNECT_VRAM 1
+
+/*
+ * AMDGPU_SVM_PGMAP_OWNER - Unique owner token for dev_pagemap registration.
+ *
+ * migrate_vma_setup() uses pgmap->owner to distinguish "own" device pages
+ * from "foreign" device pages (e.g., another GPU in an XGMI hive).
+ * Pages whose page->pgmap->owner matches the migration source are skipped
+ * (they're already in the right place).
+ *
+ * For XGMI hive: all GPUs in the hive share the same owner (the hive pointer)
+ * so intra-hive pages are treated as local.
+ * For standalone GPU: use the adev pointer itself as a unique per-device token.
+ */
+#define AMDGPU_SVM_PGMAP_OWNER(adev) \
+ ((adev)->hive ? (void *)(adev)->hive : (void *)(adev))
+
+/**
+ * struct amdgpu_pagemap - VRAM migration infrastructure for drm_pagemap
+ * @dpagemap: DRM pagemap wrapper providing device_map/populate_mm callbacks
+ * @adev: back-pointer to the owning amdgpu_device
+ * @hpa_base: host physical address base of the ZONE_DEVICE region
+ * @initialized: set to true after successful registration
+ * @pgmap: the dev_pagemap registered with devm_memremap_pages();
+ * must be last — contains a flexible-array member (ranges[])
+ *
+ * Allocated with devm_kzalloc() in amdgpu_svm_migration_init() and stored
+ * as adev->apagemap. Lifetime is tied to the device via devres.
+ */
+struct amdgpu_pagemap {
+ struct drm_pagemap dpagemap;
+ struct amdgpu_device *adev;
+ resource_size_t hpa_base;
+ bool initialized;
+ struct dev_pagemap pgmap; /* must be last — flex-array */
+};
+
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+int amdgpu_svm_migration_init(struct amdgpu_device *adev);
+#else
+static inline
+int amdgpu_svm_migration_init(struct amdgpu_device *adev)
+{
+ return 0;
+}
+#endif
+
+/**
+ * amdgpu_svm_drm_pagemap_ops - drm_pagemap_ops for AMDGPU VRAM migration
+ *
+ * Provides:
+ * .device_map - Convert ZONE_DEVICE page to VRAM address
+ * .populate_mm - Allocate VRAM BO and migrate pages from system memory
+ */
+extern const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops;
+
+#endif /* __AMDGPU_MIGRATE_H__ */
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks
2026-04-27 10:05 [PATCH v3 0/5] drm/amdgpu: SVM VRAM migration via drm_pagemap Junhua Shen
2026-04-27 10:05 ` [PATCH v3 1/5] drm/amdgpu: add VRAM migration infrastructure for drm_pagemap Junhua Shen
@ 2026-04-27 10:05 ` Junhua Shen
2026-04-27 22:20 ` Felix Kuehling
2026-04-27 10:05 ` [PATCH v3 3/5] drm/amdgpu: introduce SVM range migration decision layer Junhua Shen
` (2 subsequent siblings)
4 siblings, 1 reply; 8+ messages in thread
From: Junhua Shen @ 2026-04-27 10:05 UTC (permalink / raw)
To: Alexander.Deucher, Felix.Kuehling, Christian.Koenig, Oak.Zeng,
Jenny-Jing.Liu, Philip.Yang, Xiaogang.Chen, Ray.Huang,
honglei1.huang, Lingshan.Zhu
Cc: amd-gfx, dri-devel, Junhua Shen
Implement the drm_pagemap_devmem_ops and drm_pagemap_ops callbacks
that the DRM GPUSVM migration framework requires:
drm_pagemap_ops (top-level entry points):
- device_map: convert ZONE_DEVICE page to GPU PTE address
- populate_mm: allocate VRAM BO and trigger migration
drm_pagemap_devmem_ops (per-BO migration mechanics):
- populate_devmem_pfn: walk BO buddy blocks to build PFN array
- copy_to_devmem: SDMA copy system RAM -> VRAM via GART window
- copy_to_ram: SDMA copy VRAM -> system RAM via GART window
- devmem_release: free BO when all pages migrate back
Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 616 +++++++++++++++++++-
1 file changed, 613 insertions(+), 3 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
index 170e2eadc106..42092651b4d5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
@@ -64,12 +64,20 @@
#include <linux/memremap.h>
#include <linux/migrate.h>
-#include "amdgpu_amdkfd.h"
#include "amdgpu_migrate.h"
#include "amdgpu.h"
+#include "amdgpu_ttm.h"
+#include "amdgpu_res_cursor.h"
+
+#define AMDGPU_MIGRATE_TRACE(fmt, ...) \
+ pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
+
+/* SDMA copy direction */
+#define FROM_RAM_TO_VRAM 0
+#define FROM_VRAM_TO_RAM 1
static inline struct amdgpu_pagemap *
-dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
+to_amdgpu_pagemap(struct drm_pagemap *dpagemap)
{
return container_of(dpagemap, struct amdgpu_pagemap, dpagemap);
}
@@ -94,8 +102,610 @@ amdgpu_svm_page_to_apagemap(struct page *page)
return container_of(pgmap, struct amdgpu_pagemap, pgmap);
}
+/* drm_pagemap_devmem_ops — per-BO migration mechanics */
+
+/**
+ * struct amdgpu_svm_bo - Wrapper linking drm_pagemap_devmem to amdgpu_bo
+ *
+ * @devmem: drm_pagemap device memory allocation (passed to framework)
+ * @bo: The backing VRAM amdgpu_bo
+ *
+ * It is allocated per-migration in populate_mm() and freed by
+ * devmem_release() when all device-private pages have migrated
+ * back to system memory.
+ *
+ * Lifecycle is managed by the drm_pagemap framework's internal zdd refcount:
+ * - zdd->devmem_allocation points to &svm_bo->devmem
+ * - When zdd refcount drops to zero, framework calls devmem_release()
+ * - devmem_release() frees both the BO reference and the svm_bo itself
+ */
+struct amdgpu_svm_bo {
+ struct amdgpu_bo *bo;
+ struct drm_pagemap_devmem devmem;
+};
+
+static inline struct amdgpu_svm_bo *
+to_amdgpu_svm_bo(struct drm_pagemap_devmem *devmem_allocation)
+{
+ return container_of(devmem_allocation, struct amdgpu_svm_bo, devmem);
+}
+
+/**
+ * amdgpu_svm_devmem_release - Release BO when all device pages migrate back
+ *
+ * Called by the drm_pagemap framework (via drm_pagemap_zdd_destroy) when the
+ * last device-private page backed by this allocation has been migrated back
+ * to system memory (or the owning process exits).
+ *
+ * Frees both the amdgpu_bo reference and the wrapper amdgpu_svm_bo itself.
+ */
+static void
+amdgpu_svm_devmem_release(struct drm_pagemap_devmem *devmem_allocation)
+{
+ struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation);
+
+ AMDGPU_MIGRATE_TRACE("Release svm_bo=%px bo=%px\n", svm_bo, svm_bo->bo);
+ amdgpu_bo_unref(&svm_bo->bo);
+ kfree(svm_bo);
+}
+
+/**
+ * amdgpu_svm_populate_devmem_pfn - Convert BO VRAM allocation to PFN array
+ * @devmem_allocation: The devmem allocation in the amdgpu_svm_bo wrapper
+ * @npages: Number of PFN entries to fill
+ * @pfn: Output PFN array
+ *
+ * Iterates over the BO's TTM vram_mgr buddy blocks and converts each
+ * block's VRAM offset to ZONE_DEVICE PFNs:
+ *
+ * PFN = PHYS_PFN(block_offset + apagemap.hpa_base) + page_index
+ *
+ * This is called by drm_pagemap_migrate_to_devmem() to build the
+ * destination PFN array for migrate_vma_pages().
+ *
+ * Return: 0 on success
+ */
+static int
+amdgpu_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocation,
+ unsigned long npages, unsigned long *pfn)
+{
+ struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(devmem_allocation->dpagemap);
+ struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation);
+ struct amdgpu_bo *bo = svm_bo->bo;
+ struct amdgpu_res_cursor cursor;
+ unsigned long i = 0;
+ int ret;
+
+ ret = amdgpu_bo_reserve(bo, false);
+ if (ret)
+ return ret;
+
+ amdgpu_res_first(bo->tbo.resource, 0, npages << PAGE_SHIFT, &cursor);
+ while (cursor.remaining && i < npages) {
+ u64 pfn_base = PHYS_PFN(cursor.start + svm_dm->hpa_base);
+ u64 pages = cursor.size >> PAGE_SHIFT;
+ unsigned long j;
+
+ for (j = 0; j < pages && i < npages; j++, i++)
+ pfn[i] = pfn_base + j;
+
+ amdgpu_res_next(&cursor, cursor.size);
+ }
+
+ amdgpu_bo_unreserve(bo);
+
+ AMDGPU_MIGRATE_TRACE("populate_devmem_pfn: npages=%lu first_pfn=0x%lx\n",
+ npages, npages > 0 ? pfn[0] : 0);
+
+ return 0;
+}
+
+/* SDMA copy helpers — GART window based data transfer */
+
+/**
+ * amdgpu_svm_direct_mapping_addr - Convert VRAM offset to MC address
+ * @adev: AMDGPU device
+ * @vram_offset: Byte offset within VRAM
+ *
+ * Return: MC address suitable for SDMA src/dst
+ */
+static u64
+amdgpu_svm_direct_mapping_addr(struct amdgpu_device *adev, u64 vram_offset)
+{
+ return vram_offset + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM);
+}
+
+/**
+ * amdgpu_svm_gart_map - Map system DMA addresses into GART window
+ * @ring: SDMA ring for the GART update job
+ * @npages: Number of pages to map
+ * @addr: Array of system memory DMA addresses
+ * @gart_addr: Output — GART base address to use in SDMA copy
+ * @flags: PTE flags (e.g. writeable for RAM-to-VRAM src)
+ *
+ * Builds GART PTEs pointing at the given DMA addresses, submits an
+ * SDMA job to update the GART entries, and returns the GART address
+ * that can be used as src or dst in a subsequent amdgpu_copy_buffer().
+ *
+ * Uses GART window 0, protected by gtt_window_lock.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_gart_map(struct amdgpu_ring *ring,
+ struct amdgpu_ttm_buffer_entity *entity,
+ u64 npages,
+ dma_addr_t *addr, u64 *gart_addr, u64 flags)
+{
+ struct amdgpu_device *adev = ring->adev;
+ struct amdgpu_job *job;
+ unsigned int num_dw, num_bytes;
+ struct dma_fence *fence;
+ u64 src_addr, dst_addr;
+ u64 pte_flags;
+ void *cpu_addr;
+ int r;
+
+ /* Use entity's GART window 0 */
+ *gart_addr = amdgpu_compute_gart_address(&adev->gmc, entity, 0);
+
+ num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
+ num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+
+ r = amdgpu_job_alloc_with_ib(adev, &entity->base,
+ AMDGPU_FENCE_OWNER_UNDEFINED,
+ num_dw * 4 + num_bytes,
+ AMDGPU_IB_POOL_DELAYED,
+ &job,
+ AMDGPU_KERNEL_JOB_ID_KFD_GART_MAP);
+ if (r)
+ return r;
+
+ src_addr = num_dw * 4;
+ src_addr += job->ibs[0].gpu_addr;
+
+ dst_addr = amdgpu_bo_gpu_offset(adev->gart.bo);
+ dst_addr += (entity->gart_window_offs[0] >> AMDGPU_GPU_PAGE_SHIFT) * 8;
+ amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr,
+ dst_addr, num_bytes, 0);
+
+ amdgpu_ring_pad_ib(ring, &job->ibs[0]);
+ WARN_ON(job->ibs[0].length_dw > num_dw);
+
+ pte_flags = AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE;
+ pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED;
+ if (flags & AMDGPU_PTE_WRITEABLE)
+ pte_flags |= AMDGPU_PTE_WRITEABLE;
+ pte_flags |= adev->gart.gart_pte_flags;
+
+ cpu_addr = &job->ibs[0].ptr[num_dw];
+
+ amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr);
+ fence = amdgpu_job_submit(job);
+ dma_fence_put(fence);
+
+ return 0;
+}
+
+/**
+ * amdgpu_svm_copy_memory_gart - SDMA copy between system RAM and VRAM
+ * @adev: AMDGPU device
+ * @sys: Array of DMA addresses for system memory pages
+ * @vram: Array of VRAM byte offsets (relative to start of VRAM)
+ * @npages: Number of pages to copy
+ * @direction: FROM_RAM_TO_VRAM or FROM_VRAM_TO_RAM
+ * @mfence: In/out — carries the last SDMA fence for serialization
+ *
+ * Maps system memory pages into the GART window and uses SDMA to copy
+ * data to/from VRAM. Handles splitting into AMDGPU_GTT_MAX_TRANSFER_SIZE
+ * chunks. Acquires entity->lock internally to protect the GART window,
+ * matching the KFD svm_migrate_copy_memory_gart() pattern.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys,
+ u64 *vram, u64 npages, int direction,
+ struct dma_fence **mfence)
+{
+ const u64 max_pages = AMDGPU_GTT_MAX_TRANSFER_SIZE;
+ struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+ struct amdgpu_ttm_buffer_entity *entity = &adev->mman.move_entity;
+ u64 gart_s, gart_d;
+ struct dma_fence *next;
+ u64 size;
+ int r;
+
+ mutex_lock(&entity->lock);
+
+ while (npages) {
+ size = min(max_pages, npages);
+
+ if (direction == FROM_VRAM_TO_RAM) {
+ gart_s = amdgpu_svm_direct_mapping_addr(adev, *vram);
+ r = amdgpu_svm_gart_map(ring, entity, size, sys,
+ &gart_d, AMDGPU_PTE_WRITEABLE);
+ } else {
+ r = amdgpu_svm_gart_map(ring, entity, size, sys,
+ &gart_s, 0);
+ gart_d = amdgpu_svm_direct_mapping_addr(adev, *vram);
+ }
+ if (r) {
+ dev_err(adev->dev, "failed %d to map GART for SDMA\n", r);
+ goto out_unlock;
+ }
+
+ AMDGPU_MIGRATE_TRACE("SDMA_COPY: %s npages=%llu vram_off=0x%llx\n",
+ direction == FROM_RAM_TO_VRAM ? "RAM->VRAM" : "VRAM->RAM",
+ size, (u64)*vram);
+
+ r = amdgpu_copy_buffer(adev, entity, gart_s, gart_d,
+ size * PAGE_SIZE,
+ NULL, &next, true, 0);
+ if (r) {
+ dev_err(adev->dev, "failed %d to copy buffer\n", r);
+ goto out_unlock;
+ }
+
+ dma_fence_put(*mfence);
+ *mfence = next;
+ npages -= size;
+ if (npages) {
+ sys += size;
+ vram += size;
+ }
+ }
+
+out_unlock:
+ mutex_unlock(&entity->lock);
+
+ return r;
+}
+
+/**
+ * amdgpu_svm_copy_to_devmem - SDMA copy system memory -> VRAM
+ * @pages: Array of destination ZONE_DEVICE pages (VRAM-backed)
+ * @pagemap_addr: Array of source DMA addresses (system memory, already mapped)
+ * @npages: Number of pages to copy
+ *
+ * Builds parallel sys[] and vram[] arrays from the framework-provided
+ * pagemap_addr and device pages, then submits batched SDMA copies via
+ * the GART window.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_to_devmem(struct page **pages,
+ struct drm_pagemap_addr *pagemap_addr,
+ unsigned long npages,
+ struct dma_fence *pre_migrate_fence)
+{
+ struct amdgpu_device *adev;
+ struct amdgpu_pagemap *svm_dm;
+ struct dma_fence *mfence = NULL;
+ dma_addr_t *sys;
+ u64 *vram;
+ unsigned long i, j;
+ int ret = 0;
+
+ if (!npages)
+ return 0;
+
+ /*
+ * Find the first non-NULL page to derive the device.
+ * The pages array may contain NULL entries for positions where
+ * no valid device page exists.
+ */
+ for (i = 0; i < npages; i++) {
+ if (pages[i])
+ break;
+ }
+ if (i == npages)
+ return 0;
+
+ svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
+ adev = svm_dm->adev;
+
+ sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
+ vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
+ if (!sys || !vram) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+
+ for (i = 0, j = 0; i < npages; i++) {
+ if (!pagemap_addr[i].addr)
+ goto flush;
+
+ sys[j] = pagemap_addr[i].addr;
+ vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+ svm_dm->hpa_base;
+
+ /* Check if next vram page is contiguous with current */
+ if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
+ goto flush;
+
+ j++;
+ continue;
+flush:
+ if (j) {
+ ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+ FROM_RAM_TO_VRAM,
+ &mfence);
+ if (ret)
+ goto out_fence;
+ j = 0;
+ }
+ /* Re-process current page if it was valid but broke contiguity */
+ if (pagemap_addr[i].addr) {
+ sys[0] = pagemap_addr[i].addr;
+ vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+ svm_dm->hpa_base;
+ j = 1;
+ }
+ }
+
+ /* Flush remaining batch */
+ if (j)
+ ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+ FROM_RAM_TO_VRAM, &mfence);
+
+out_fence:
+ if (mfence) {
+ dma_fence_wait(mfence, false);
+ dma_fence_put(mfence);
+ }
+
+ AMDGPU_MIGRATE_TRACE("copy_to_devmem done: npages=%ld ret=%d\n",
+ npages, ret);
+
+out_free:
+ kvfree(vram);
+ kvfree(sys);
+ return ret;
+}
+
+/**
+ * amdgpu_svm_copy_to_ram - SDMA copy VRAM -> system memory
+ * @pages: Array of source ZONE_DEVICE pages (VRAM-backed)
+ * @pagemap_addr: Array of destination DMA addresses (system memory, already mapped)
+ * @npages: Number of pages to copy
+ *
+ * Mirror of copy_to_devmem with src/dst swapped.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_copy_to_ram(struct page **pages,
+ struct drm_pagemap_addr *pagemap_addr,
+ unsigned long npages,
+ struct dma_fence *pre_migrate_fence)
+{
+ struct amdgpu_device *adev;
+ struct amdgpu_pagemap *svm_dm;
+ struct dma_fence *mfence = NULL;
+ dma_addr_t *sys;
+ u64 *vram;
+ unsigned long i, j;
+ int ret = 0;
+
+ if (!npages)
+ return 0;
+
+ for (i = 0; i < npages; i++) {
+ if (pages[i])
+ break;
+ }
+ if (i == npages)
+ return 0;
+
+ svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
+ adev = svm_dm->adev;
+
+ sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
+ vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
+ if (!sys || !vram) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+
+ for (i = 0, j = 0; i < npages; i++) {
+ if (!pagemap_addr[i].addr || !pages[i])
+ goto flush;
+
+ vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+ svm_dm->hpa_base;
+ sys[j] = pagemap_addr[i].addr;
+
+ /* Check if next vram page is contiguous with current */
+ if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
+ goto flush;
+
+ j++;
+ continue;
+flush:
+ if (j) {
+ ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+ FROM_VRAM_TO_RAM,
+ &mfence);
+ if (ret)
+ goto out_fence;
+ j = 0;
+ }
+ /* Re-process current page if it was valid but broke contiguity */
+ if (pagemap_addr[i].addr && pages[i]) {
+ vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
+ svm_dm->hpa_base;
+ sys[0] = pagemap_addr[i].addr;
+ j = 1;
+ }
+ }
+
+ /* Flush remaining batch */
+ if (j)
+ ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
+ FROM_VRAM_TO_RAM, &mfence);
+
+out_fence:
+ if (mfence) {
+ dma_fence_wait(mfence, false);
+ dma_fence_put(mfence);
+ }
+
+ AMDGPU_MIGRATE_TRACE("copy_to_ram done: npages=%ld ret=%d\n", npages, ret);
+
+out_free:
+ kvfree(vram);
+ kvfree(sys);
+ return ret;
+}
+
+static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = {
+ .devmem_release = amdgpu_svm_devmem_release,
+ .populate_devmem_pfn = amdgpu_svm_populate_devmem_pfn,
+ .copy_to_devmem = amdgpu_svm_copy_to_devmem,
+ .copy_to_ram = amdgpu_svm_copy_to_ram,
+};
+
+/* drm_pagemap_ops — top-level migration entry points */
+
+/**
+ * amdgpu_svm_device_map - Convert ZONE_DEVICE page to GPU PTE address
+ * @dpagemap: The drm_pagemap for this device
+ * @dev: Requesting device (for P2P check)
+ * @page: ZONE_DEVICE page backed by VRAM
+ * @order: Page order (0 = 4K, 9 = 2M, etc.)
+ * @dir: DMA direction (unused for local VRAM)
+ *
+ * Address conversion chain:
+ * page -> PFN -> HPA -> VRAM offset -> PTE address
+ *
+ * HPA = page_to_pfn(page) << PAGE_SHIFT
+ * VRAM offset = HPA - apagemap.hpa_base
+ * PTE address = VRAM offset + adev->vm_manager.vram_base_offset
+ *
+ * Return: drm_pagemap_addr with PTE address and AMDGPU_INTERCONNECT_VRAM protocol
+ */
+static struct drm_pagemap_addr
+amdgpu_svm_device_map(struct drm_pagemap *dpagemap,
+ struct device *dev,
+ struct page *page,
+ unsigned int order,
+ enum dma_data_direction dir)
+{
+ struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(dpagemap);
+ struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
+ dma_addr_t addr;
+
+ if (dpagemap->drm->dev == dev) {
+ /* Same device: return VRAM PTE address */
+ u64 hpa = (u64)page_to_pfn(page) << PAGE_SHIFT;
+ u64 vram_offset = hpa - svm_dm->hpa_base;
+
+ addr = vram_offset + adev->vm_manager.vram_base_offset;
+ } else {
+ /* Cross-device P2P: not yet supported */
+ addr = DMA_MAPPING_ERROR;
+ }
+
+ return drm_pagemap_addr_encode(addr,
+ AMDGPU_INTERCONNECT_VRAM, order, dir);
+}
+
+/**
+ * amdgpu_svm_bo_alloc - Allocate an amdgpu_svm_bo wrapper with VRAM backing
+ * @adev: AMDGPU device
+ * @dpagemap: The drm_pagemap for this device
+ * @mm: mm_struct of the owning process
+ * @size: Allocation size in bytes
+ *
+ * Return: Pointer to allocated amdgpu_svm_bo on success, ERR_PTR on failure
+ */
+static struct amdgpu_svm_bo *
+amdgpu_svm_bo_alloc(struct amdgpu_device *adev,
+ struct drm_pagemap *dpagemap,
+ struct mm_struct *mm, unsigned long size)
+{
+ struct amdgpu_svm_bo *svm_bo;
+ struct amdgpu_bo_param bp = {};
+ struct amdgpu_bo *bo;
+ int ret;
+
+ svm_bo = kzalloc(sizeof(*svm_bo), GFP_KERNEL);
+ if (!svm_bo)
+ return ERR_PTR(-ENOMEM);
+
+ bp.size = size;
+ bp.bo_ptr_size = sizeof(struct amdgpu_bo);
+ bp.domain = AMDGPU_GEM_DOMAIN_VRAM;
+ bp.type = ttm_bo_type_device;
+ bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS |
+ AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
+ AMDGPU_GEM_CREATE_VRAM_CLEARED;
+
+ ret = amdgpu_bo_create(adev, &bp, &bo);
+ if (ret) {
+ AMDGPU_MIGRATE_TRACE("Failed to create SVM BO\n");
+ kfree(svm_bo);
+ return ERR_PTR(ret);
+ }
+
+ amdgpu_bo_unreserve(bo);
+ svm_bo->bo = bo;
+
+ drm_pagemap_devmem_init(&svm_bo->devmem,
+ adev->dev, mm,
+ &amdgpu_pagemap_ops,
+ dpagemap, size, NULL);
+
+ return svm_bo;
+}
+
+/**
+ * amdgpu_svm_populate_mm - Allocate VRAM BO and migrate pages
+ * @dpagemap: The drm_pagemap for this device
+ * @start: Start virtual address of the range to migrate
+ * @end: End virtual address (exclusive)
+ * @mm: mm_struct of the owning process
+ * @timeslice_ms: Maximum time to spend migrating (for fairness)
+ *
+ * Core migration entry point called by drm_pagemap_populate_mm().
+ * Allocates an amdgpu_svm_bo via amdgpu_svm_bo_alloc(), then calls
+ * drm_pagemap_migrate_to_devmem() to execute the actual migration.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int
+amdgpu_svm_populate_mm(struct drm_pagemap *dpagemap,
+ unsigned long start, unsigned long end,
+ struct mm_struct *mm,
+ unsigned long timeslice_ms)
+{
+ struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
+ struct drm_pagemap_migrate_details mdetails = {
+ .timeslice_ms = timeslice_ms,
+ };
+ struct amdgpu_svm_bo *svm_bo;
+ int ret;
+
+ svm_bo = amdgpu_svm_bo_alloc(adev, dpagemap, mm, end - start);
+ if (IS_ERR(svm_bo))
+ return PTR_ERR(svm_bo);
+
+ AMDGPU_MIGRATE_TRACE("populate_mm: [0x%lx-0x%lx] size=%lu\n",
+ start, end, end - start);
+
+ ret = drm_pagemap_migrate_to_devmem(&svm_bo->devmem,
+ mm, start, end,
+ &mdetails);
+
+ return ret;
+}
-const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = { };
+const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = {
+ .device_map = amdgpu_svm_device_map,
+ .populate_mm = amdgpu_svm_populate_mm,
+};
/**
* amdgpu_svm_migration_init - Register ZONE_DEVICE and initialize drm_pagemap
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v3 3/5] drm/amdgpu: introduce SVM range migration decision layer
2026-04-27 10:05 [PATCH v3 0/5] drm/amdgpu: SVM VRAM migration via drm_pagemap Junhua Shen
2026-04-27 10:05 ` [PATCH v3 1/5] drm/amdgpu: add VRAM migration infrastructure for drm_pagemap Junhua Shen
2026-04-27 10:05 ` [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks Junhua Shen
@ 2026-04-27 10:05 ` Junhua Shen
2026-04-27 10:05 ` [PATCH v3 4/5] drm/amdgpu: add SVM attr prefetch/force-trigger functionality Junhua Shen
2026-04-27 10:05 ` [PATCH v3 5/5] drm/amdgpu: integrate VRAM migration into SVM range map path Junhua Shen
4 siblings, 0 replies; 8+ messages in thread
From: Junhua Shen @ 2026-04-27 10:05 UTC (permalink / raw)
To: Alexander.Deucher, Felix.Kuehling, Christian.Koenig, Oak.Zeng,
Jenny-Jing.Liu, Philip.Yang, Xiaogang.Chen, Ray.Huang,
honglei1.huang, Lingshan.Zhu
Cc: amd-gfx, dri-devel, Junhua Shen
Add the migration decision infrastructure for drm_pagemap-based SVM
VRAM migration:
- Define enum amdgpu_svm_migrate_mode (NONE, TO_VRAM, TO_SYSMEM,
PREFERRED) to carry migration intent
- Implement amdgpu_svm_range_migrate_range() which handles both
directions: TO_VRAM via drm_pagemap_populate_mm(), TO_SYSMEM via
drm_gpusvm_range_evict()
Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
drivers/gpu/drm/amd/amdgpu/Makefile | 6 +-
.../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 140 ++++++++++++++++++
.../drm/amd/amdgpu/amdgpu_svm_range_migrate.h | 60 ++++++++
3 files changed, 203 insertions(+), 3 deletions(-)
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile
index e64abb5c8ab8..cf4f453b7e68 100644
--- a/drivers/gpu/drm/amd/amdgpu/Makefile
+++ b/drivers/gpu/drm/amd/amdgpu/Makefile
@@ -323,14 +323,14 @@ amdgpu-$(CONFIG_HMM_MIRROR) += amdgpu_hmm.o
# svm support
amdgpu-$(CONFIG_DRM_AMDGPU_SVM) += amdgpu_svm.o amdgpu_svm_attr.o \
- amdgpu_svm_range.o amdgpu_migrate.o
+ amdgpu_svm_range.o amdgpu_svm_range_migrate.o amdgpu_migrate.o
.PHONY: clean-svm
clean-svm:
rm -f $(obj)/amdgpu_svm.o $(obj)/amdgpu_svm_attr.o $(obj)/amdgpu_svm_range.o \
- $(obj)/amdgpu_migrate.o \
+ $(obj)/amdgpu_svm_range_migrate.o $(obj)/amdgpu_migrate.o \
$(obj)/.amdgpu_svm.o.cmd $(obj)/.amdgpu_svm_attr.o.cmd $(obj)/.amdgpu_svm_range.o.cmd \
- $(obj)/.amdgpu_migrate.o.cmd
+ $(obj)/.amdgpu_svm_range_migrate.o.cmd $(obj)/.amdgpu_migrate.o.cmd
include $(FULL_AMD_PATH)/pm/Makefile
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
new file mode 100644
index 000000000000..f29d4cdb5c64
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "amdgpu.h"
+#include "amdgpu_svm.h"
+#include "amdgpu_svm_range.h"
+#include "amdgpu_migrate.h"
+#include "amdgpu_svm_range_migrate.h"
+
+static bool
+range_in_vram(struct drm_gpusvm_range *range)
+{
+ struct drm_gpusvm_pages_flags flags = {
+ /* Pairs with WRITE_ONCE in drm_gpusvm_get_pages() */
+ .__flags = READ_ONCE(range->pages.flags.__flags),
+ };
+
+ return flags.has_devmem_pages;
+}
+
+static struct drm_pagemap *
+amdgpu_svm_get_dpagemap(struct amdgpu_svm *svm)
+{
+ struct amdgpu_pagemap *apagemap = svm->adev->apagemap;
+
+ if (!apagemap->initialized)
+ return NULL;
+
+ return &apagemap->dpagemap;
+}
+
+static bool
+range_needs_migrate_to_vram(struct drm_gpusvm_range *range,
+ enum amdgpu_svm_migrate_mode mode)
+{
+ if (!range->pages.flags.migrate_devmem)
+ return false;
+
+ if (mode == AMDGPU_SVM_MIGRATE_TO_SYSMEM)
+ return false;
+
+ if (range_in_vram(range))
+ return false;
+
+ return true;
+}
+
+/**
+ * amdgpu_svm_range_migrate_to_vram - Migrate a single range's pages to VRAM
+ * @svm: Pointer to the AMDGPU SVM structure
+ * @range: Pointer to the GPU SVM range to migrate
+ *
+ * Return: 0 on success, negative errno on failure
+ */
+static int
+amdgpu_svm_range_migrate_to_vram(struct amdgpu_svm *svm,
+ struct drm_gpusvm_range *range)
+{
+ struct drm_pagemap *dpagemap = amdgpu_svm_get_dpagemap(svm);
+ unsigned long start = drm_gpusvm_range_start(range);
+ unsigned long end = drm_gpusvm_range_end(range);
+ int ret;
+
+ if (!dpagemap)
+ return -ENODEV;
+
+ ret = drm_pagemap_populate_mm(dpagemap, start, end,
+ svm->gpusvm.mm, 0);
+ if (ret) {
+ AMDGPU_SVM_TRACE("migrate_to_vram failed: ret=%d [0x%lx-0x%lx]\n",
+ ret, start, end);
+ return ret;
+ }
+
+ return 0;
+}
+
+bool
+amdgpu_pagemap_capable(struct amdgpu_svm *svm)
+{
+ if (svm->adev->gmc.is_app_apu)
+ return false;
+
+ if (!amdgpu_svm_get_dpagemap(svm))
+ return false;
+
+ return true;
+}
+
+/**
+ * amdgpu_svm_range_migrate_range - Per-range migration
+ * @svm: Pointer to the AMDGPU SVM structure
+ * @range: The GPU SVM range to migrate
+ * @migrate_mode: Migration intent
+ *
+ * NOTE: Only AMDGPU_SVM_MIGRATE_TO_VRAM and
+ * AMDGPU_SVM_MIGRATE_TO_SYSMEM are valid inputs. Passing any other
+ * migrate mode will trigger a WARN_ON() and the function will return
+ * -EINVAL. The caller should resolve PREFERRED/OTHER higher-level
+ * intents into one of the two concrete directions before calling this
+ * helper.
+ */
+int
+amdgpu_svm_range_migrate_range(struct amdgpu_svm *svm,
+ struct drm_gpusvm_range *range,
+ enum amdgpu_svm_migrate_mode migrate_mode)
+{
+ if (WARN_ON(migrate_mode != AMDGPU_SVM_MIGRATE_TO_VRAM &&
+ migrate_mode != AMDGPU_SVM_MIGRATE_TO_SYSMEM))
+ return -EINVAL;
+
+ if (range_needs_migrate_to_vram(range, migrate_mode))
+ return amdgpu_svm_range_migrate_to_vram(svm, range);
+ else if (migrate_mode == AMDGPU_SVM_MIGRATE_TO_SYSMEM &&
+ range_in_vram(range))
+ return drm_gpusvm_range_evict(&svm->gpusvm, range);
+
+ return 0;
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
new file mode 100644
index 000000000000..eba0b5f90e6e
--- /dev/null
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __AMDGPU_SVM_RANGE_MIGRATE_H__
+#define __AMDGPU_SVM_RANGE_MIGRATE_H__
+
+struct amdgpu_svm;
+struct drm_gpusvm_range;
+
+/**
+ * enum amdgpu_svm_migrate_mode - Migration mode for SVM range migration
+ *
+ * @AMDGPU_SVM_MIGRATE_NONE: No migration requested. Callers use it to
+ * skip the migration call and allow early-out for already-mapped ranges.
+ *
+ * @AMDGPU_SVM_MIGRATE_TO_VRAM: Migrate pages to VRAM before mapping.
+ *
+ * @AMDGPU_SVM_MIGRATE_TO_SYSMEM: Evict VRAM pages back to system memory.
+ *
+ * @AMDGPU_SVM_MIGRATE_PREFERRED: Follow preferred_loc to decide migration.
+ * Callers resolve this to TO_VRAM or TO_SYSMEM before calling
+ * amdgpu_svm_range_migrate_range().
+ *
+ * Only TO_VRAM and TO_SYSMEM are valid inputs to
+ * amdgpu_svm_range_migrate_range().
+ */
+enum amdgpu_svm_migrate_mode {
+ AMDGPU_SVM_MIGRATE_NONE,
+ AMDGPU_SVM_MIGRATE_TO_VRAM,
+ AMDGPU_SVM_MIGRATE_TO_SYSMEM,
+ AMDGPU_SVM_MIGRATE_PREFERRED
+};
+
+bool amdgpu_pagemap_capable(struct amdgpu_svm *svm);
+int amdgpu_svm_range_migrate_range(struct amdgpu_svm *svm,
+ struct drm_gpusvm_range *range,
+ enum amdgpu_svm_migrate_mode migrate_mode);
+
+#endif /* __AMDGPU_SVM_RANGE_MIGRATE_H__ */
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v3 4/5] drm/amdgpu: add SVM attr prefetch/force-trigger functionality
2026-04-27 10:05 [PATCH v3 0/5] drm/amdgpu: SVM VRAM migration via drm_pagemap Junhua Shen
` (2 preceding siblings ...)
2026-04-27 10:05 ` [PATCH v3 3/5] drm/amdgpu: introduce SVM range migration decision layer Junhua Shen
@ 2026-04-27 10:05 ` Junhua Shen
2026-04-27 10:05 ` [PATCH v3 5/5] drm/amdgpu: integrate VRAM migration into SVM range map path Junhua Shen
4 siblings, 0 replies; 8+ messages in thread
From: Junhua Shen @ 2026-04-27 10:05 UTC (permalink / raw)
To: Alexander.Deucher, Felix.Kuehling, Christian.Koenig, Oak.Zeng,
Jenny-Jing.Liu, Philip.Yang, Xiaogang.Chen, Ray.Huang,
honglei1.huang, Lingshan.Zhu
Cc: amd-gfx, dri-devel, Junhua Shen
Add attr_has_prefetch_loc() helper to detect prefetch-loc attributes
and use it to force-trigger migration even when the stored value has
not changed (prefetch is a one-shot command).
Refine attr_change_ctx_trigger() to only fire LOCATION_CHANGE on
prefetch_loc changes, not preferred_loc changes.
Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 34 +++++++++++++++-----
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
index cd972026f39b..52e402c18b0e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
@@ -159,8 +159,7 @@ attr_change_ctx_trigger(const struct amdgpu_svm_attrs *prev_attrs,
trigger |= AMDGPU_SVM_ATTR_TRIGGER_PTE_FLAG_CHANGE;
if (changed_flags & AMDGPU_SVM_MAPPING_FLAG_MASK)
trigger |= AMDGPU_SVM_ATTR_TRIGGER_MAPPING_FLAG_CHANGE;
- if (prev_attrs->preferred_loc != new_attrs->preferred_loc ||
- prev_attrs->prefetch_loc != new_attrs->prefetch_loc)
+ if (prev_attrs->prefetch_loc != new_attrs->prefetch_loc)
trigger |= AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE;
if (prev_attrs->granularity != new_attrs->granularity)
trigger |= AMDGPU_SVM_ATTR_TRIGGER_GRANULARITY_CHANGE;
@@ -187,9 +186,22 @@ static bool attr_has_access(uint32_t nattr,
return false;
}
+static bool attr_has_prefetch_loc(uint32_t nattr,
+ const struct drm_amdgpu_svm_attribute *attrs)
+{
+ uint32_t i;
+
+ for (i = 0; i < nattr; i++) {
+ if (attrs[i].type == AMDGPU_SVM_ATTR_PREFETCH_LOC)
+ return true;
+ }
+
+ return false;
+}
+
static struct amdgpu_svm_attr_range *
attr_alloc_range(unsigned long start,
- unsigned long last,
+ unsigned long last,
const struct amdgpu_svm_attrs *attrs)
{
struct amdgpu_svm_attr_range *range;
@@ -477,23 +489,29 @@ amdgpu_svm_attr_set_existing(struct amdgpu_svm_attr_tree *attr_tree,
struct amdgpu_svm_attrs old_attrs;
struct amdgpu_svm_attrs new_attrs;
uint32_t trigger;
- bool force_trigger;
+ uint32_t force_trigger = 0;
lockdep_assert_held(&attr_tree->lock);
old_attrs = range->attrs;
- /* The attr layer doesn't store the gpu mapped state, and for align with KFD,
- * need force trigger range layer to check if gpu mapped.
+ /*
+ * Force the range layer to act even when the attr value is
+ * unchanged: ACCESS needs re-check on xnack-off because the
+ * attr layer has no gpu_mapped state; PREFETCH_LOC is one-shot
+ * so repeated prefetch must always trigger migration.
*/
- force_trigger = !attr_tree->svm->xnack_enabled && attr_has_access(nattr, attrs);
+ if (!attr_tree->svm->xnack_enabled && attr_has_access(nattr, attrs))
+ force_trigger |= AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE;
+ if (attr_has_prefetch_loc(nattr, attrs))
+ force_trigger |= AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE;
if (attr_same_attrs(range, nattr, attrs)) {
if (!force_trigger)
return 0;
amdgpu_svm_attr_change_ctx_set(change, start, last,
- AMDGPU_SVM_ATTR_TRIGGER_ACCESS_CHANGE,
+ force_trigger,
&old_attrs, &old_attrs);
return 0;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v3 5/5] drm/amdgpu: integrate VRAM migration into SVM range map path
2026-04-27 10:05 [PATCH v3 0/5] drm/amdgpu: SVM VRAM migration via drm_pagemap Junhua Shen
` (3 preceding siblings ...)
2026-04-27 10:05 ` [PATCH v3 4/5] drm/amdgpu: add SVM attr prefetch/force-trigger functionality Junhua Shen
@ 2026-04-27 10:05 ` Junhua Shen
4 siblings, 0 replies; 8+ messages in thread
From: Junhua Shen @ 2026-04-27 10:05 UTC (permalink / raw)
To: Alexander.Deucher, Felix.Kuehling, Christian.Koenig, Oak.Zeng,
Jenny-Jing.Liu, Philip.Yang, Xiaogang.Chen, Ray.Huang,
honglei1.huang, Lingshan.Zhu
Cc: amd-gfx, dri-devel, Junhua Shen
hook up ZONE_DEVICE registration in device init and reset.
Wire the migration decision layer into the SVM range map call chain:
- Add migrate_mode parameter to amdgpu_svm_range_map_attr_ranges()
and propagate it through map_interval -> per-range map loop
- Call amdgpu_svm_range_migrate_range() on each drm_gpusvm_range
before GPU mapping to perform VRAM migration or eviction
- Pass appropriate migrate_mode from SVM attr prefetch triggers
and the restore worker
Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +
drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c | 4 +
drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 4 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 136 +++++++++---------
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 5 +-
5 files changed, 87 insertions(+), 66 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index fbe553c38583..3be51a2c0106 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -78,6 +78,7 @@
#include "amdgpu_reset.h"
#include "amdgpu_virt.h"
#include "amdgpu_dev_coredump.h"
+#include "amdgpu_migrate.h"
#include <linux/suspend.h>
#include <drm/task_barrier.h>
@@ -4076,6 +4077,9 @@ int amdgpu_device_init(struct amdgpu_device *adev,
/* Don't init kfd if whole hive need to be reset during init */
if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) {
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+ amdgpu_svm_migration_init(adev);
+#endif
kgd2kfd_init_zone_device(adev);
kfd_update_svm_support_properties(adev);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
index 28c4ad62f50e..c94d43f3ab42 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c
@@ -25,6 +25,7 @@
#include "aldebaran.h"
#include "sienna_cichlid.h"
#include "smu_v13_0_10.h"
+#include "amdgpu_migrate.h"
static int amdgpu_reset_xgmi_reset_on_init_suspend(struct amdgpu_device *adev)
{
@@ -87,6 +88,9 @@ static int amdgpu_reset_xgmi_reset_on_init_restore_hwctxt(
return r;
list_for_each_entry(tmp_adev, reset_device_list, reset_list) {
if (!tmp_adev->kfd.init_complete) {
+#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
+ amdgpu_svm_migration_init(tmp_adev);
+#endif
kgd2kfd_init_zone_device(tmp_adev);
amdgpu_amdkfd_device_init(tmp_adev);
amdgpu_amdkfd_drm_client_create(tmp_adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
index 57103a140164..cd8dbe56a9e9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
@@ -32,6 +32,7 @@
#include "amdgpu_svm.h"
#include "amdgpu_svm_attr.h"
#include "amdgpu_svm_range.h"
+#include "amdgpu_svm_range_migrate.h"
#include "amdgpu_vm.h"
#if IS_ENABLED(CONFIG_DRM_AMDGPU_SVM)
@@ -341,7 +342,8 @@ int amdgpu_svm_handle_fault(struct amdgpu_device *adev, uint32_t pasid,
AMDGPU_SVM_TRACE("handle_fault: map_attr page=0x%lx\n", fault_page);
down_write(&svm->svm_lock);
- ret = amdgpu_svm_range_map_attr_ranges(svm, fault_page, fault_page);
+ ret = amdgpu_svm_range_map_attr_ranges(svm, fault_page, fault_page,
+ AMDGPU_SVM_MIGRATE_PREFERRED);
up_write(&svm->svm_lock);
if (ret)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
index 472a641fb836..8880e2ba79cc 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
@@ -25,6 +25,7 @@
#include "amdgpu_svm.h"
#include "amdgpu_svm_attr.h"
#include "amdgpu_svm_range.h"
+#include "amdgpu_migrate.h"
#include "amdgpu.h"
#include "amdgpu_amdkfd.h"
#include "amdgpu_vm.h"
@@ -114,7 +115,6 @@ range_pages_valid(struct amdgpu_svm *svm,
return drm_gpusvm_range_pages_valid(&svm->gpusvm, range);
}
-
static int
amdgpu_svm_range_gpu_unmap_in_notifier(struct amdgpu_svm *svm,
struct drm_gpusvm_range *range,
@@ -247,11 +247,11 @@ amdgpu_svm_range_attr_pte_flags(struct amdgpu_svm *svm,
return pte_flags;
}
- /*
- * POC/WA: reuse kfd apis for queue quiesce/resume
- * But kfd apis are for process level, not for GPU VM level
- * need consider potential issues
- */
+/*
+ * POC/WA: reuse kfd apis for queue quiesce/resume
+ * But kfd apis are for process level, not for GPU VM level
+ * need consider potential issues
+ */
void amdgpu_svm_range_restore_begin_compute(struct amdgpu_svm *svm)
{
int ret;
@@ -317,39 +317,6 @@ static int amdgpu_svm_range_lock_vm_pd(struct amdgpu_svm *svm, struct drm_exec *
return 0;
}
-static int
-amdgpu_svm_range_update_gpu(struct amdgpu_svm *svm, unsigned long start_page,
- unsigned long last_page, uint64_t pte_flags,
- dma_addr_t *pages_addr, bool flush_tlb,
- bool update_pdes, bool wait_fence)
-{
- struct drm_exec exec;
- struct dma_fence *fence = NULL;
- int ret;
-
- ret = amdgpu_svm_range_lock_vm_pd(svm, &exec);
- if (ret)
- return ret;
-
- ret = amdgpu_vm_update_range(svm->adev, svm->vm, false, false,
- flush_tlb, true,
- NULL, start_page, last_page, pte_flags, 0, 0,
- NULL, pages_addr, wait_fence ? &fence : NULL);
- if (!ret && wait_fence && fence) {
- ret = dma_fence_wait(fence, false);
- if (ret < 0)
- AMDGPU_SVM_TRACE("wait unmap fence failed: ret=%d [0x%lx-0x%lx]-0x%lx\n",
- ret, start_page, last_page,
- last_page - start_page + 1);
- }
- if (!ret && update_pdes)
- ret = amdgpu_vm_update_pdes(svm->adev, svm->vm, false);
-
- dma_fence_put(fence);
- drm_exec_fini(&exec);
- return ret;
-}
-
static int
amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
struct drm_gpusvm_range *range,
@@ -376,9 +343,11 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
npages - mapped_pages);
dma_addr_t seg_addr = entry->addr;
unsigned long start_page, last_page;
+ uint64_t seg_pte_flags = pte_flags;
bool is_last_seg;
- if (entry->proto != DRM_INTERCONNECT_SYSTEM)
+ if (entry->proto != DRM_INTERCONNECT_SYSTEM &&
+ entry->proto != AMDGPU_INTERCONNECT_VRAM)
return -EOPNOTSUPP;
while (mapped_pages + seg_pages < npages) {
@@ -399,9 +368,13 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
last_page = start_page + seg_pages - 1;
is_last_seg = mapped_pages + seg_pages == npages;
+ /* For VRAM pages, Clear the SYSTEM and SNOOPED bits */
+ if (entry->proto == AMDGPU_INTERCONNECT_VRAM)
+ seg_pte_flags &= ~(AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED);
+
ret = amdgpu_vm_update_range(svm->adev, svm->vm, false, false,
flush_tlb && is_last_seg, true, NULL,
- start_page, last_page, pte_flags,
+ start_page, last_page, seg_pte_flags,
0, seg_addr, NULL, NULL,
wait_fence && is_last_seg ? fence : NULL);
if (ret)
@@ -413,17 +386,35 @@ amdgpu_svm_range_update_gpu_range(struct amdgpu_svm *svm,
return 0;
}
+static bool
+preferred_loc_is_vram(const struct amdgpu_svm_attrs *attrs)
+{
+ return attrs->preferred_loc != AMDGPU_SVM_LOCATION_SYSMEM &&
+ attrs->preferred_loc != AMDGPU_SVM_LOCATION_UNDEFINED;
+}
+
static int
amdgpu_svm_range_map(struct amdgpu_svm *svm,
unsigned long start,
unsigned long end,
const struct amdgpu_svm_attrs *attrs,
const struct drm_gpusvm_ctx *gpusvm_ctx,
- uint64_t pte_flags)
+ uint64_t pte_flags,
+ enum amdgpu_svm_migrate_mode migrate_mode)
{
unsigned long addr = start;
int ret;
+ /*
+ * For preferred migration mode, determine the
+ * actual migration direction based on the preferred
+ * location attribute.
+ */
+ if (migrate_mode == AMDGPU_SVM_MIGRATE_PREFERRED)
+ migrate_mode = preferred_loc_is_vram(attrs) ?
+ AMDGPU_SVM_MIGRATE_TO_VRAM :
+ AMDGPU_SVM_MIGRATE_TO_SYSMEM;
+
while (addr < end) {
struct drm_exec exec;
struct drm_gpusvm_ctx map_ctx;
@@ -462,6 +453,10 @@ amdgpu_svm_range_map(struct amdgpu_svm *svm,
if (next_addr <= addr)
return -EINVAL;
+ /* Per-range migration */
+ if (migrate_mode != AMDGPU_SVM_MIGRATE_NONE)
+ amdgpu_svm_range_migrate_range(svm, range, migrate_mode);
+
range_pte_flags = map_ctx.read_only ?
(pte_flags & ~AMDGPU_PTE_WRITEABLE) : pte_flags;
@@ -529,10 +524,16 @@ amdgpu_svm_range_map(struct amdgpu_svm *svm,
static int
amdgpu_svm_range_map_interval(struct amdgpu_svm *svm, unsigned long start_page,
unsigned long last_page,
- const struct amdgpu_svm_attrs *attrs)
+ const struct amdgpu_svm_attrs *attrs,
+ enum amdgpu_svm_migrate_mode migrate_mode)
{
+ bool hw_devmem = amdgpu_pagemap_capable(svm);
+
struct drm_gpusvm_ctx gpusvm_ctx = {
.read_only = !!(attrs->flags & AMDGPU_SVM_FLAG_GPU_RO),
+ .devmem_possible = hw_devmem,
+ .device_private_page_owner = hw_devmem ?
+ AMDGPU_SVM_PGMAP_OWNER(svm->adev) : NULL,
};
unsigned long start = start_page << PAGE_SHIFT;
unsigned long end = (last_page + 1) << PAGE_SHIFT;
@@ -542,7 +543,7 @@ amdgpu_svm_range_map_interval(struct amdgpu_svm *svm, unsigned long start_page,
pte_flags = amdgpu_svm_range_attr_pte_flags(svm, attrs);
ret = amdgpu_svm_range_map(svm, start, end, attrs, &gpusvm_ctx,
- pte_flags);
+ pte_flags, migrate_mode);
if (ret)
AMDGPU_SVM_TRACE("map_interval failed: ret=%d [0x%lx-0x%lx)-0x%lx\n",
ret, start, end, end - start);
@@ -553,7 +554,8 @@ amdgpu_svm_range_map_interval(struct amdgpu_svm *svm, unsigned long start_page,
int
amdgpu_svm_range_map_attr_ranges(struct amdgpu_svm *svm,
unsigned long start_page,
- unsigned long last_page)
+ unsigned long last_page,
+ enum amdgpu_svm_migrate_mode migrate_mode)
{
lockdep_assert_held_write(&svm->svm_lock);
@@ -573,9 +575,10 @@ amdgpu_svm_range_map_attr_ranges(struct amdgpu_svm *svm,
seg_last = min(seg_last, last_page);
if (range_has_access(attrs.access)) {
- /* map may fail here cause no vma or access deny */
- ret = amdgpu_svm_range_map_interval(svm, cursor, seg_last,
- &attrs);
+ ret = amdgpu_svm_range_map_interval(svm, cursor,
+ seg_last,
+ &attrs,
+ migrate_mode);
if (ret)
return ret;
}
@@ -657,7 +660,6 @@ static int amdgpu_svm_range_rebuild_locked(struct amdgpu_svm *svm,
unsigned long rebuild_start = start_page;
unsigned long rebuild_last = last_page;
bool removed;
- int ret;
lockdep_assert_held_write(&svm->svm_lock);
@@ -673,14 +675,10 @@ static int amdgpu_svm_range_rebuild_locked(struct amdgpu_svm *svm,
/* scan rebuild start end to build the extra removed ranges */
if (rebuild)
return amdgpu_svm_range_map_attr_ranges(svm, rebuild_start,
- rebuild_last);
+ rebuild_last,
+ AMDGPU_SVM_MIGRATE_PREFERRED);
- ret = amdgpu_svm_range_update_gpu(svm, rebuild_start, rebuild_last,
- 0, NULL, true, true, true);
- if (!ret)
- svm->flush_tlb(svm);
-
- return ret;
+ return 0;
}
static void
@@ -706,10 +704,11 @@ amdgpu_svm_range_process_notifier_ranges(struct amdgpu_svm *svm,
if (clear_pte) {
amdgpu_svm_range_gpu_unmap_in_notifier(svm, range,
mmu_range);
- range_invalidate_gpu_mapping(range);
}
drm_gpusvm_range_unmap_pages(&svm->gpusvm, range, &ctx);
+ range_invalidate_gpu_mapping(range);
+
if (is_unmap)
drm_gpusvm_range_set_unmapped(range, mmu_range);
@@ -758,6 +757,7 @@ int amdgpu_svm_range_apply_attr_change(struct amdgpu_svm *svm,
{
lockdep_assert_held_write(&svm->svm_lock);
+ enum amdgpu_svm_migrate_mode migrate_mode = AMDGPU_SVM_MIGRATE_NONE;
bool old_access, new_access;
bool update_mapping = false;
@@ -788,16 +788,24 @@ int amdgpu_svm_range_apply_attr_change(struct amdgpu_svm *svm,
new_access)
update_mapping = true;
- if (trigger & AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE) {
- /* TODO: add migration */
+ if ((trigger & AMDGPU_SVM_ATTR_TRIGGER_LOCATION_CHANGE) &&
+ new_access) {
+ int32_t loc = new_attrs->prefetch_loc;
+
+ if (loc == AMDGPU_SVM_LOCATION_SYSMEM) {
+ migrate_mode = AMDGPU_SVM_MIGRATE_TO_SYSMEM;
+ update_mapping = true;
+ } else if (loc != AMDGPU_SVM_LOCATION_UNDEFINED) {
+ migrate_mode = AMDGPU_SVM_MIGRATE_TO_VRAM;
+ update_mapping = true;
+ }
}
if (!update_mapping)
return 0;
- AMDGPU_SVM_TRACE("mapping update: remap interval [0x%lx-0x%lx]-0x%lx\n",
- start, last, last - start + 1);
- return amdgpu_svm_range_map_interval(svm, start, last, new_attrs);
+ return amdgpu_svm_range_map_interval(svm, start, last, new_attrs,
+ migrate_mode);
}
static bool
@@ -1004,7 +1012,8 @@ static void amdgpu_svm_range_restore_worker(struct work_struct *w)
down_write(&svm->svm_lock);
ret = amdgpu_svm_range_map_attr_ranges(svm, op_ctx.start,
- op_ctx.last);
+ op_ctx.last,
+ AMDGPU_SVM_MIGRATE_NONE);
up_write(&svm->svm_lock);
if (ret) {
@@ -1042,7 +1051,6 @@ static void amdgpu_svm_range_restore_worker(struct work_struct *w)
drm_gpusvm_notifier_unlock(&svm->gpusvm);
svm->end_restore(svm);
return;
-
}
drm_gpusvm_notifier_unlock(&svm->gpusvm);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
index 18bf3dad13fd..0065ae50c700 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
@@ -30,6 +30,8 @@
#include <linux/list.h>
#include <linux/types.h>
+#include "amdgpu_svm_range_migrate.h"
+
struct amdgpu_svm;
struct amdgpu_svm_attrs;
struct drm_gpusvm_notifier;
@@ -62,7 +64,8 @@ void amdgpu_svm_range_flush(struct amdgpu_svm *svm);
void amdgpu_svm_range_sync_work(struct amdgpu_svm *svm);
int amdgpu_svm_range_map_attr_ranges(struct amdgpu_svm *svm,
unsigned long start_page,
- unsigned long last_page);
+ unsigned long last_page,
+ enum amdgpu_svm_migrate_mode migrate_mode);
int amdgpu_svm_range_apply_attr_change(
struct amdgpu_svm *svm, unsigned long start, unsigned long last,
uint32_t trigger, const struct amdgpu_svm_attrs *prev_attrs,
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks
2026-04-27 10:05 ` [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks Junhua Shen
@ 2026-04-27 22:20 ` Felix Kuehling
2026-04-28 7:39 ` Junhua Shen
0 siblings, 1 reply; 8+ messages in thread
From: Felix Kuehling @ 2026-04-27 22:20 UTC (permalink / raw)
To: Junhua Shen, Alexander.Deucher, Christian.Koenig, Oak.Zeng,
Jenny-Jing.Liu, Philip.Yang, Xiaogang.Chen, Ray.Huang,
honglei1.huang, Lingshan.Zhu
Cc: amd-gfx, dri-devel
On 2026-04-27 06:05, Junhua Shen wrote:
> Implement the drm_pagemap_devmem_ops and drm_pagemap_ops callbacks
> that the DRM GPUSVM migration framework requires:
>
> drm_pagemap_ops (top-level entry points):
> - device_map: convert ZONE_DEVICE page to GPU PTE address
> - populate_mm: allocate VRAM BO and trigger migration
>
> drm_pagemap_devmem_ops (per-BO migration mechanics):
> - populate_devmem_pfn: walk BO buddy blocks to build PFN array
> - copy_to_devmem: SDMA copy system RAM -> VRAM via GART window
> - copy_to_ram: SDMA copy VRAM -> system RAM via GART window
> - devmem_release: free BO when all pages migrate back
>
> Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 616 +++++++++++++++++++-
> 1 file changed, 613 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
> index 170e2eadc106..42092651b4d5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
> @@ -64,12 +64,20 @@
> #include <linux/memremap.h>
> #include <linux/migrate.h>
>
> -#include "amdgpu_amdkfd.h"
> #include "amdgpu_migrate.h"
> #include "amdgpu.h"
> +#include "amdgpu_ttm.h"
> +#include "amdgpu_res_cursor.h"
> +
> +#define AMDGPU_MIGRATE_TRACE(fmt, ...) \
> + pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
> +
> +/* SDMA copy direction */
> +#define FROM_RAM_TO_VRAM 0
> +#define FROM_VRAM_TO_RAM 1
>
> static inline struct amdgpu_pagemap *
> -dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
> +to_amdgpu_pagemap(struct drm_pagemap *dpagemap)
> {
> return container_of(dpagemap, struct amdgpu_pagemap, dpagemap);
> }
> @@ -94,8 +102,610 @@ amdgpu_svm_page_to_apagemap(struct page *page)
> return container_of(pgmap, struct amdgpu_pagemap, pgmap);
> }
>
> +/* drm_pagemap_devmem_ops — per-BO migration mechanics */
> +
> +/**
> + * struct amdgpu_svm_bo - Wrapper linking drm_pagemap_devmem to amdgpu_bo
> + *
> + * @devmem: drm_pagemap device memory allocation (passed to framework)
> + * @bo: The backing VRAM amdgpu_bo
> + *
> + * It is allocated per-migration in populate_mm() and freed by
> + * devmem_release() when all device-private pages have migrated
> + * back to system memory.
> + *
> + * Lifecycle is managed by the drm_pagemap framework's internal zdd refcount:
> + * - zdd->devmem_allocation points to &svm_bo->devmem
> + * - When zdd refcount drops to zero, framework calls devmem_release()
> + * - devmem_release() frees both the BO reference and the svm_bo itself
> + */
> +struct amdgpu_svm_bo {
> + struct amdgpu_bo *bo;
> + struct drm_pagemap_devmem devmem;
> +};
> +
> +static inline struct amdgpu_svm_bo *
> +to_amdgpu_svm_bo(struct drm_pagemap_devmem *devmem_allocation)
> +{
> + return container_of(devmem_allocation, struct amdgpu_svm_bo, devmem);
> +}
> +
> +/**
> + * amdgpu_svm_devmem_release - Release BO when all device pages migrate back
> + *
> + * Called by the drm_pagemap framework (via drm_pagemap_zdd_destroy) when the
> + * last device-private page backed by this allocation has been migrated back
> + * to system memory (or the owning process exits).
> + *
> + * Frees both the amdgpu_bo reference and the wrapper amdgpu_svm_bo itself.
> + */
> +static void
> +amdgpu_svm_devmem_release(struct drm_pagemap_devmem *devmem_allocation)
> +{
> + struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation);
> +
> + AMDGPU_MIGRATE_TRACE("Release svm_bo=%px bo=%px\n", svm_bo, svm_bo->bo);
> + amdgpu_bo_unref(&svm_bo->bo);
> + kfree(svm_bo);
> +}
> +
> +/**
> + * amdgpu_svm_populate_devmem_pfn - Convert BO VRAM allocation to PFN array
> + * @devmem_allocation: The devmem allocation in the amdgpu_svm_bo wrapper
> + * @npages: Number of PFN entries to fill
> + * @pfn: Output PFN array
> + *
> + * Iterates over the BO's TTM vram_mgr buddy blocks and converts each
> + * block's VRAM offset to ZONE_DEVICE PFNs:
> + *
> + * PFN = PHYS_PFN(block_offset + apagemap.hpa_base) + page_index
> + *
> + * This is called by drm_pagemap_migrate_to_devmem() to build the
> + * destination PFN array for migrate_vma_pages().
> + *
> + * Return: 0 on success
> + */
> +static int
> +amdgpu_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocation,
> + unsigned long npages, unsigned long *pfn)
> +{
> + struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(devmem_allocation->dpagemap);
> + struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation);
> + struct amdgpu_bo *bo = svm_bo->bo;
> + struct amdgpu_res_cursor cursor;
> + unsigned long i = 0;
> + int ret;
> +
> + ret = amdgpu_bo_reserve(bo, false);
> + if (ret)
> + return ret;
> +
> + amdgpu_res_first(bo->tbo.resource, 0, npages << PAGE_SHIFT, &cursor);
How do you ensure that the BO is valid and actually in VRAM at the time?
And how do you ensure that it stays there as long as the zone_device
pages are in use? As far as I can tell, the BO is not pinned and there
is no fence that prevents it from being evicted to GTT by TTM without
warning.
Regards,
Felix
> + while (cursor.remaining && i < npages) {
> + u64 pfn_base = PHYS_PFN(cursor.start + svm_dm->hpa_base);
> + u64 pages = cursor.size >> PAGE_SHIFT;
> + unsigned long j;
> +
> + for (j = 0; j < pages && i < npages; j++, i++)
> + pfn[i] = pfn_base + j;
> +
> + amdgpu_res_next(&cursor, cursor.size);
> + }
> +
> + amdgpu_bo_unreserve(bo);
> +
> + AMDGPU_MIGRATE_TRACE("populate_devmem_pfn: npages=%lu first_pfn=0x%lx\n",
> + npages, npages > 0 ? pfn[0] : 0);
> +
> + return 0;
> +}
> +
> +/* SDMA copy helpers — GART window based data transfer */
> +
> +/**
> + * amdgpu_svm_direct_mapping_addr - Convert VRAM offset to MC address
> + * @adev: AMDGPU device
> + * @vram_offset: Byte offset within VRAM
> + *
> + * Return: MC address suitable for SDMA src/dst
> + */
> +static u64
> +amdgpu_svm_direct_mapping_addr(struct amdgpu_device *adev, u64 vram_offset)
> +{
> + return vram_offset + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM);
> +}
> +
> +/**
> + * amdgpu_svm_gart_map - Map system DMA addresses into GART window
> + * @ring: SDMA ring for the GART update job
> + * @npages: Number of pages to map
> + * @addr: Array of system memory DMA addresses
> + * @gart_addr: Output — GART base address to use in SDMA copy
> + * @flags: PTE flags (e.g. writeable for RAM-to-VRAM src)
> + *
> + * Builds GART PTEs pointing at the given DMA addresses, submits an
> + * SDMA job to update the GART entries, and returns the GART address
> + * that can be used as src or dst in a subsequent amdgpu_copy_buffer().
> + *
> + * Uses GART window 0, protected by gtt_window_lock.
> + *
> + * Return: 0 on success, negative error code on failure
> + */
> +static int
> +amdgpu_svm_gart_map(struct amdgpu_ring *ring,
> + struct amdgpu_ttm_buffer_entity *entity,
> + u64 npages,
> + dma_addr_t *addr, u64 *gart_addr, u64 flags)
> +{
> + struct amdgpu_device *adev = ring->adev;
> + struct amdgpu_job *job;
> + unsigned int num_dw, num_bytes;
> + struct dma_fence *fence;
> + u64 src_addr, dst_addr;
> + u64 pte_flags;
> + void *cpu_addr;
> + int r;
> +
> + /* Use entity's GART window 0 */
> + *gart_addr = amdgpu_compute_gart_address(&adev->gmc, entity, 0);
> +
> + num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
> + num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
> +
> + r = amdgpu_job_alloc_with_ib(adev, &entity->base,
> + AMDGPU_FENCE_OWNER_UNDEFINED,
> + num_dw * 4 + num_bytes,
> + AMDGPU_IB_POOL_DELAYED,
> + &job,
> + AMDGPU_KERNEL_JOB_ID_KFD_GART_MAP);
> + if (r)
> + return r;
> +
> + src_addr = num_dw * 4;
> + src_addr += job->ibs[0].gpu_addr;
> +
> + dst_addr = amdgpu_bo_gpu_offset(adev->gart.bo);
> + dst_addr += (entity->gart_window_offs[0] >> AMDGPU_GPU_PAGE_SHIFT) * 8;
> + amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr,
> + dst_addr, num_bytes, 0);
> +
> + amdgpu_ring_pad_ib(ring, &job->ibs[0]);
> + WARN_ON(job->ibs[0].length_dw > num_dw);
> +
> + pte_flags = AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE;
> + pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED;
> + if (flags & AMDGPU_PTE_WRITEABLE)
> + pte_flags |= AMDGPU_PTE_WRITEABLE;
> + pte_flags |= adev->gart.gart_pte_flags;
> +
> + cpu_addr = &job->ibs[0].ptr[num_dw];
> +
> + amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr);
> + fence = amdgpu_job_submit(job);
> + dma_fence_put(fence);
> +
> + return 0;
> +}
> +
> +/**
> + * amdgpu_svm_copy_memory_gart - SDMA copy between system RAM and VRAM
> + * @adev: AMDGPU device
> + * @sys: Array of DMA addresses for system memory pages
> + * @vram: Array of VRAM byte offsets (relative to start of VRAM)
> + * @npages: Number of pages to copy
> + * @direction: FROM_RAM_TO_VRAM or FROM_VRAM_TO_RAM
> + * @mfence: In/out — carries the last SDMA fence for serialization
> + *
> + * Maps system memory pages into the GART window and uses SDMA to copy
> + * data to/from VRAM. Handles splitting into AMDGPU_GTT_MAX_TRANSFER_SIZE
> + * chunks. Acquires entity->lock internally to protect the GART window,
> + * matching the KFD svm_migrate_copy_memory_gart() pattern.
> + *
> + * Return: 0 on success, negative error code on failure
> + */
> +static int
> +amdgpu_svm_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys,
> + u64 *vram, u64 npages, int direction,
> + struct dma_fence **mfence)
> +{
> + const u64 max_pages = AMDGPU_GTT_MAX_TRANSFER_SIZE;
> + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
> + struct amdgpu_ttm_buffer_entity *entity = &adev->mman.move_entity;
> + u64 gart_s, gart_d;
> + struct dma_fence *next;
> + u64 size;
> + int r;
> +
> + mutex_lock(&entity->lock);
> +
> + while (npages) {
> + size = min(max_pages, npages);
> +
> + if (direction == FROM_VRAM_TO_RAM) {
> + gart_s = amdgpu_svm_direct_mapping_addr(adev, *vram);
> + r = amdgpu_svm_gart_map(ring, entity, size, sys,
> + &gart_d, AMDGPU_PTE_WRITEABLE);
> + } else {
> + r = amdgpu_svm_gart_map(ring, entity, size, sys,
> + &gart_s, 0);
> + gart_d = amdgpu_svm_direct_mapping_addr(adev, *vram);
> + }
> + if (r) {
> + dev_err(adev->dev, "failed %d to map GART for SDMA\n", r);
> + goto out_unlock;
> + }
> +
> + AMDGPU_MIGRATE_TRACE("SDMA_COPY: %s npages=%llu vram_off=0x%llx\n",
> + direction == FROM_RAM_TO_VRAM ? "RAM->VRAM" : "VRAM->RAM",
> + size, (u64)*vram);
> +
> + r = amdgpu_copy_buffer(adev, entity, gart_s, gart_d,
> + size * PAGE_SIZE,
> + NULL, &next, true, 0);
> + if (r) {
> + dev_err(adev->dev, "failed %d to copy buffer\n", r);
> + goto out_unlock;
> + }
> +
> + dma_fence_put(*mfence);
> + *mfence = next;
> + npages -= size;
> + if (npages) {
> + sys += size;
> + vram += size;
> + }
> + }
> +
> +out_unlock:
> + mutex_unlock(&entity->lock);
> +
> + return r;
> +}
> +
> +/**
> + * amdgpu_svm_copy_to_devmem - SDMA copy system memory -> VRAM
> + * @pages: Array of destination ZONE_DEVICE pages (VRAM-backed)
> + * @pagemap_addr: Array of source DMA addresses (system memory, already mapped)
> + * @npages: Number of pages to copy
> + *
> + * Builds parallel sys[] and vram[] arrays from the framework-provided
> + * pagemap_addr and device pages, then submits batched SDMA copies via
> + * the GART window.
> + *
> + * Return: 0 on success, negative error code on failure
> + */
> +static int
> +amdgpu_svm_copy_to_devmem(struct page **pages,
> + struct drm_pagemap_addr *pagemap_addr,
> + unsigned long npages,
> + struct dma_fence *pre_migrate_fence)
> +{
> + struct amdgpu_device *adev;
> + struct amdgpu_pagemap *svm_dm;
> + struct dma_fence *mfence = NULL;
> + dma_addr_t *sys;
> + u64 *vram;
> + unsigned long i, j;
> + int ret = 0;
> +
> + if (!npages)
> + return 0;
> +
> + /*
> + * Find the first non-NULL page to derive the device.
> + * The pages array may contain NULL entries for positions where
> + * no valid device page exists.
> + */
> + for (i = 0; i < npages; i++) {
> + if (pages[i])
> + break;
> + }
> + if (i == npages)
> + return 0;
> +
> + svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
> + adev = svm_dm->adev;
> +
> + sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
> + vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
> + if (!sys || !vram) {
> + ret = -ENOMEM;
> + goto out_free;
> + }
> +
> + for (i = 0, j = 0; i < npages; i++) {
> + if (!pagemap_addr[i].addr)
> + goto flush;
> +
> + sys[j] = pagemap_addr[i].addr;
> + vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> + svm_dm->hpa_base;
> +
> + /* Check if next vram page is contiguous with current */
> + if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
> + goto flush;
> +
> + j++;
> + continue;
> +flush:
> + if (j) {
> + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> + FROM_RAM_TO_VRAM,
> + &mfence);
> + if (ret)
> + goto out_fence;
> + j = 0;
> + }
> + /* Re-process current page if it was valid but broke contiguity */
> + if (pagemap_addr[i].addr) {
> + sys[0] = pagemap_addr[i].addr;
> + vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> + svm_dm->hpa_base;
> + j = 1;
> + }
> + }
> +
> + /* Flush remaining batch */
> + if (j)
> + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> + FROM_RAM_TO_VRAM, &mfence);
> +
> +out_fence:
> + if (mfence) {
> + dma_fence_wait(mfence, false);
> + dma_fence_put(mfence);
> + }
> +
> + AMDGPU_MIGRATE_TRACE("copy_to_devmem done: npages=%ld ret=%d\n",
> + npages, ret);
> +
> +out_free:
> + kvfree(vram);
> + kvfree(sys);
> + return ret;
> +}
> +
> +/**
> + * amdgpu_svm_copy_to_ram - SDMA copy VRAM -> system memory
> + * @pages: Array of source ZONE_DEVICE pages (VRAM-backed)
> + * @pagemap_addr: Array of destination DMA addresses (system memory, already mapped)
> + * @npages: Number of pages to copy
> + *
> + * Mirror of copy_to_devmem with src/dst swapped.
> + *
> + * Return: 0 on success, negative error code on failure
> + */
> +static int
> +amdgpu_svm_copy_to_ram(struct page **pages,
> + struct drm_pagemap_addr *pagemap_addr,
> + unsigned long npages,
> + struct dma_fence *pre_migrate_fence)
> +{
> + struct amdgpu_device *adev;
> + struct amdgpu_pagemap *svm_dm;
> + struct dma_fence *mfence = NULL;
> + dma_addr_t *sys;
> + u64 *vram;
> + unsigned long i, j;
> + int ret = 0;
> +
> + if (!npages)
> + return 0;
> +
> + for (i = 0; i < npages; i++) {
> + if (pages[i])
> + break;
> + }
> + if (i == npages)
> + return 0;
> +
> + svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
> + adev = svm_dm->adev;
> +
> + sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
> + vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
> + if (!sys || !vram) {
> + ret = -ENOMEM;
> + goto out_free;
> + }
> +
> + for (i = 0, j = 0; i < npages; i++) {
> + if (!pagemap_addr[i].addr || !pages[i])
> + goto flush;
> +
> + vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> + svm_dm->hpa_base;
> + sys[j] = pagemap_addr[i].addr;
> +
> + /* Check if next vram page is contiguous with current */
> + if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
> + goto flush;
> +
> + j++;
> + continue;
> +flush:
> + if (j) {
> + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> + FROM_VRAM_TO_RAM,
> + &mfence);
> + if (ret)
> + goto out_fence;
> + j = 0;
> + }
> + /* Re-process current page if it was valid but broke contiguity */
> + if (pagemap_addr[i].addr && pages[i]) {
> + vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> + svm_dm->hpa_base;
> + sys[0] = pagemap_addr[i].addr;
> + j = 1;
> + }
> + }
> +
> + /* Flush remaining batch */
> + if (j)
> + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> + FROM_VRAM_TO_RAM, &mfence);
> +
> +out_fence:
> + if (mfence) {
> + dma_fence_wait(mfence, false);
> + dma_fence_put(mfence);
> + }
> +
> + AMDGPU_MIGRATE_TRACE("copy_to_ram done: npages=%ld ret=%d\n", npages, ret);
> +
> +out_free:
> + kvfree(vram);
> + kvfree(sys);
> + return ret;
> +}
> +
> +static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = {
> + .devmem_release = amdgpu_svm_devmem_release,
> + .populate_devmem_pfn = amdgpu_svm_populate_devmem_pfn,
> + .copy_to_devmem = amdgpu_svm_copy_to_devmem,
> + .copy_to_ram = amdgpu_svm_copy_to_ram,
> +};
> +
> +/* drm_pagemap_ops — top-level migration entry points */
> +
> +/**
> + * amdgpu_svm_device_map - Convert ZONE_DEVICE page to GPU PTE address
> + * @dpagemap: The drm_pagemap for this device
> + * @dev: Requesting device (for P2P check)
> + * @page: ZONE_DEVICE page backed by VRAM
> + * @order: Page order (0 = 4K, 9 = 2M, etc.)
> + * @dir: DMA direction (unused for local VRAM)
> + *
> + * Address conversion chain:
> + * page -> PFN -> HPA -> VRAM offset -> PTE address
> + *
> + * HPA = page_to_pfn(page) << PAGE_SHIFT
> + * VRAM offset = HPA - apagemap.hpa_base
> + * PTE address = VRAM offset + adev->vm_manager.vram_base_offset
> + *
> + * Return: drm_pagemap_addr with PTE address and AMDGPU_INTERCONNECT_VRAM protocol
> + */
> +static struct drm_pagemap_addr
> +amdgpu_svm_device_map(struct drm_pagemap *dpagemap,
> + struct device *dev,
> + struct page *page,
> + unsigned int order,
> + enum dma_data_direction dir)
> +{
> + struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(dpagemap);
> + struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
> + dma_addr_t addr;
> +
> + if (dpagemap->drm->dev == dev) {
> + /* Same device: return VRAM PTE address */
> + u64 hpa = (u64)page_to_pfn(page) << PAGE_SHIFT;
> + u64 vram_offset = hpa - svm_dm->hpa_base;
> +
> + addr = vram_offset + adev->vm_manager.vram_base_offset;
> + } else {
> + /* Cross-device P2P: not yet supported */
> + addr = DMA_MAPPING_ERROR;
> + }
> +
> + return drm_pagemap_addr_encode(addr,
> + AMDGPU_INTERCONNECT_VRAM, order, dir);
> +}
> +
> +/**
> + * amdgpu_svm_bo_alloc - Allocate an amdgpu_svm_bo wrapper with VRAM backing
> + * @adev: AMDGPU device
> + * @dpagemap: The drm_pagemap for this device
> + * @mm: mm_struct of the owning process
> + * @size: Allocation size in bytes
> + *
> + * Return: Pointer to allocated amdgpu_svm_bo on success, ERR_PTR on failure
> + */
> +static struct amdgpu_svm_bo *
> +amdgpu_svm_bo_alloc(struct amdgpu_device *adev,
> + struct drm_pagemap *dpagemap,
> + struct mm_struct *mm, unsigned long size)
> +{
> + struct amdgpu_svm_bo *svm_bo;
> + struct amdgpu_bo_param bp = {};
> + struct amdgpu_bo *bo;
> + int ret;
> +
> + svm_bo = kzalloc(sizeof(*svm_bo), GFP_KERNEL);
> + if (!svm_bo)
> + return ERR_PTR(-ENOMEM);
> +
> + bp.size = size;
> + bp.bo_ptr_size = sizeof(struct amdgpu_bo);
> + bp.domain = AMDGPU_GEM_DOMAIN_VRAM;
> + bp.type = ttm_bo_type_device;
> + bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS |
> + AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
> + AMDGPU_GEM_CREATE_VRAM_CLEARED;
> +
> + ret = amdgpu_bo_create(adev, &bp, &bo);
> + if (ret) {
> + AMDGPU_MIGRATE_TRACE("Failed to create SVM BO\n");
> + kfree(svm_bo);
> + return ERR_PTR(ret);
> + }
> +
> + amdgpu_bo_unreserve(bo);
> + svm_bo->bo = bo;
> +
> + drm_pagemap_devmem_init(&svm_bo->devmem,
> + adev->dev, mm,
> + &amdgpu_pagemap_ops,
> + dpagemap, size, NULL);
> +
> + return svm_bo;
> +}
> +
> +/**
> + * amdgpu_svm_populate_mm - Allocate VRAM BO and migrate pages
> + * @dpagemap: The drm_pagemap for this device
> + * @start: Start virtual address of the range to migrate
> + * @end: End virtual address (exclusive)
> + * @mm: mm_struct of the owning process
> + * @timeslice_ms: Maximum time to spend migrating (for fairness)
> + *
> + * Core migration entry point called by drm_pagemap_populate_mm().
> + * Allocates an amdgpu_svm_bo via amdgpu_svm_bo_alloc(), then calls
> + * drm_pagemap_migrate_to_devmem() to execute the actual migration.
> + *
> + * Return: 0 on success, negative error code on failure
> + */
> +static int
> +amdgpu_svm_populate_mm(struct drm_pagemap *dpagemap,
> + unsigned long start, unsigned long end,
> + struct mm_struct *mm,
> + unsigned long timeslice_ms)
> +{
> + struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
> + struct drm_pagemap_migrate_details mdetails = {
> + .timeslice_ms = timeslice_ms,
> + };
> + struct amdgpu_svm_bo *svm_bo;
> + int ret;
> +
> + svm_bo = amdgpu_svm_bo_alloc(adev, dpagemap, mm, end - start);
> + if (IS_ERR(svm_bo))
> + return PTR_ERR(svm_bo);
> +
> + AMDGPU_MIGRATE_TRACE("populate_mm: [0x%lx-0x%lx] size=%lu\n",
> + start, end, end - start);
> +
> + ret = drm_pagemap_migrate_to_devmem(&svm_bo->devmem,
> + mm, start, end,
> + &mdetails);
> +
> + return ret;
> +}
>
> -const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = { };
> +const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = {
> + .device_map = amdgpu_svm_device_map,
> + .populate_mm = amdgpu_svm_populate_mm,
> +};
>
> /**
> * amdgpu_svm_migration_init - Register ZONE_DEVICE and initialize drm_pagemap
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks
2026-04-27 22:20 ` Felix Kuehling
@ 2026-04-28 7:39 ` Junhua Shen
0 siblings, 0 replies; 8+ messages in thread
From: Junhua Shen @ 2026-04-28 7:39 UTC (permalink / raw)
To: Felix Kuehling
Cc: Alexander.Deucher, Christian.Koenig, Oak.Zeng, Jenny-Jing.Liu,
Philip.Yang, Xiaogang.Chen, Ray.Huang, honglei1.huang,
Lingshan.Zhu, simona, amd-gfx, dri-devel
On Mon, Apr 27, 2026 at 06:20:32PM -0400, Felix Kuehling wrote:
>
> On 2026-04-27 06:05, Junhua Shen wrote:
> > Implement the drm_pagemap_devmem_ops and drm_pagemap_ops callbacks
> > that the DRM GPUSVM migration framework requires:
> >
> > drm_pagemap_ops (top-level entry points):
> > - device_map: convert ZONE_DEVICE page to GPU PTE address
> > - populate_mm: allocate VRAM BO and trigger migration
> >
> > drm_pagemap_devmem_ops (per-BO migration mechanics):
> > - populate_devmem_pfn: walk BO buddy blocks to build PFN array
> > - copy_to_devmem: SDMA copy system RAM -> VRAM via GART window
> > - copy_to_ram: SDMA copy VRAM -> system RAM via GART window
> > - devmem_release: free BO when all pages migrate back
> >
> > Signed-off-by: Junhua Shen <Junhua.Shen@amd.com>
> > ---
> > drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 616 +++++++++++++++++++-
> > 1 file changed, 613 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
> > index 170e2eadc106..42092651b4d5 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
> > @@ -64,12 +64,20 @@
> > #include <linux/memremap.h>
> > #include <linux/migrate.h>
> > -#include "amdgpu_amdkfd.h"
> > #include "amdgpu_migrate.h"
> > #include "amdgpu.h"
> > +#include "amdgpu_ttm.h"
> > +#include "amdgpu_res_cursor.h"
> > +
> > +#define AMDGPU_MIGRATE_TRACE(fmt, ...) \
> > + pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
> > +
> > +/* SDMA copy direction */
> > +#define FROM_RAM_TO_VRAM 0
> > +#define FROM_VRAM_TO_RAM 1
> > static inline struct amdgpu_pagemap *
> > -dpagemap_to_apagemap(struct drm_pagemap *dpagemap)
> > +to_amdgpu_pagemap(struct drm_pagemap *dpagemap)
> > {
> > return container_of(dpagemap, struct amdgpu_pagemap, dpagemap);
> > }
> > @@ -94,8 +102,610 @@ amdgpu_svm_page_to_apagemap(struct page *page)
> > return container_of(pgmap, struct amdgpu_pagemap, pgmap);
> > }
> > +/* drm_pagemap_devmem_ops — per-BO migration mechanics */
> > +
> > +/**
> > + * struct amdgpu_svm_bo - Wrapper linking drm_pagemap_devmem to amdgpu_bo
> > + *
> > + * @devmem: drm_pagemap device memory allocation (passed to framework)
> > + * @bo: The backing VRAM amdgpu_bo
> > + *
> > + * It is allocated per-migration in populate_mm() and freed by
> > + * devmem_release() when all device-private pages have migrated
> > + * back to system memory.
> > + *
> > + * Lifecycle is managed by the drm_pagemap framework's internal zdd refcount:
> > + * - zdd->devmem_allocation points to &svm_bo->devmem
> > + * - When zdd refcount drops to zero, framework calls devmem_release()
> > + * - devmem_release() frees both the BO reference and the svm_bo itself
> > + */
> > +struct amdgpu_svm_bo {
> > + struct amdgpu_bo *bo;
> > + struct drm_pagemap_devmem devmem;
> > +};
> > +
> > +static inline struct amdgpu_svm_bo *
> > +to_amdgpu_svm_bo(struct drm_pagemap_devmem *devmem_allocation)
> > +{
> > + return container_of(devmem_allocation, struct amdgpu_svm_bo, devmem);
> > +}
> > +
> > +/**
> > + * amdgpu_svm_devmem_release - Release BO when all device pages migrate back
> > + *
> > + * Called by the drm_pagemap framework (via drm_pagemap_zdd_destroy) when the
> > + * last device-private page backed by this allocation has been migrated back
> > + * to system memory (or the owning process exits).
> > + *
> > + * Frees both the amdgpu_bo reference and the wrapper amdgpu_svm_bo itself.
> > + */
> > +static void
> > +amdgpu_svm_devmem_release(struct drm_pagemap_devmem *devmem_allocation)
> > +{
> > + struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation);
> > +
> > + AMDGPU_MIGRATE_TRACE("Release svm_bo=%px bo=%px\n", svm_bo, svm_bo->bo);
> > + amdgpu_bo_unref(&svm_bo->bo);
> > + kfree(svm_bo);
> > +}
> > +
> > +/**
> > + * amdgpu_svm_populate_devmem_pfn - Convert BO VRAM allocation to PFN array
> > + * @devmem_allocation: The devmem allocation in the amdgpu_svm_bo wrapper
> > + * @npages: Number of PFN entries to fill
> > + * @pfn: Output PFN array
> > + *
> > + * Iterates over the BO's TTM vram_mgr buddy blocks and converts each
> > + * block's VRAM offset to ZONE_DEVICE PFNs:
> > + *
> > + * PFN = PHYS_PFN(block_offset + apagemap.hpa_base) + page_index
> > + *
> > + * This is called by drm_pagemap_migrate_to_devmem() to build the
> > + * destination PFN array for migrate_vma_pages().
> > + *
> > + * Return: 0 on success
> > + */
> > +static int
> > +amdgpu_svm_populate_devmem_pfn(struct drm_pagemap_devmem *devmem_allocation,
> > + unsigned long npages, unsigned long *pfn)
> > +{
> > + struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(devmem_allocation->dpagemap);
> > + struct amdgpu_svm_bo *svm_bo = to_amdgpu_svm_bo(devmem_allocation);
> > + struct amdgpu_bo *bo = svm_bo->bo;
> > + struct amdgpu_res_cursor cursor;
> > + unsigned long i = 0;
> > + int ret;
> > +
> > + ret = amdgpu_bo_reserve(bo, false);
> > + if (ret)
> > + return ret;
> > +
> > + amdgpu_res_first(bo->tbo.resource, 0, npages << PAGE_SHIFT, &cursor);
>
> How do you ensure that the BO is valid and actually in VRAM at the time? And
> how do you ensure that it stays there as long as the zone_device pages are
> in use? As far as I can tell, the BO is not pinned and there is no fence
> that prevents it from being evicted to GTT by TTM without warning.
>
> Regards,
> Felix
>
Hi Felix,
Thanks for the review.
You're right — the BO is not pinned and the current version lacks
protection against TTM eviction, so there is no guarantee that the
BO stays in VRAM while zone_device pages are in use.
The v1 series had an eviction fence patch to address this, but the
implementation had issues, so it was dropped in v2.
I'm working on a proper eviction support implementation that will
migrate zone_device pages back to system RAM before TTM reclaims
the VRAM backing. This will be added in a follow-up series.
Regards,
Junhua
> > + while (cursor.remaining && i < npages) {
> > + u64 pfn_base = PHYS_PFN(cursor.start + svm_dm->hpa_base);
> > + u64 pages = cursor.size >> PAGE_SHIFT;
> > + unsigned long j;
> > +
> > + for (j = 0; j < pages && i < npages; j++, i++)
> > + pfn[i] = pfn_base + j;
> > +
> > + amdgpu_res_next(&cursor, cursor.size);
> > + }
> > +
> > + amdgpu_bo_unreserve(bo);
> > +
> > + AMDGPU_MIGRATE_TRACE("populate_devmem_pfn: npages=%lu first_pfn=0x%lx\n",
> > + npages, npages > 0 ? pfn[0] : 0);
> > +
> > + return 0;
> > +}
> > +
> > +/* SDMA copy helpers — GART window based data transfer */
> > +
> > +/**
> > + * amdgpu_svm_direct_mapping_addr - Convert VRAM offset to MC address
> > + * @adev: AMDGPU device
> > + * @vram_offset: Byte offset within VRAM
> > + *
> > + * Return: MC address suitable for SDMA src/dst
> > + */
> > +static u64
> > +amdgpu_svm_direct_mapping_addr(struct amdgpu_device *adev, u64 vram_offset)
> > +{
> > + return vram_offset + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM);
> > +}
> > +
> > +/**
> > + * amdgpu_svm_gart_map - Map system DMA addresses into GART window
> > + * @ring: SDMA ring for the GART update job
> > + * @npages: Number of pages to map
> > + * @addr: Array of system memory DMA addresses
> > + * @gart_addr: Output — GART base address to use in SDMA copy
> > + * @flags: PTE flags (e.g. writeable for RAM-to-VRAM src)
> > + *
> > + * Builds GART PTEs pointing at the given DMA addresses, submits an
> > + * SDMA job to update the GART entries, and returns the GART address
> > + * that can be used as src or dst in a subsequent amdgpu_copy_buffer().
> > + *
> > + * Uses GART window 0, protected by gtt_window_lock.
> > + *
> > + * Return: 0 on success, negative error code on failure
> > + */
> > +static int
> > +amdgpu_svm_gart_map(struct amdgpu_ring *ring,
> > + struct amdgpu_ttm_buffer_entity *entity,
> > + u64 npages,
> > + dma_addr_t *addr, u64 *gart_addr, u64 flags)
> > +{
> > + struct amdgpu_device *adev = ring->adev;
> > + struct amdgpu_job *job;
> > + unsigned int num_dw, num_bytes;
> > + struct dma_fence *fence;
> > + u64 src_addr, dst_addr;
> > + u64 pte_flags;
> > + void *cpu_addr;
> > + int r;
> > +
> > + /* Use entity's GART window 0 */
> > + *gart_addr = amdgpu_compute_gart_address(&adev->gmc, entity, 0);
> > +
> > + num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
> > + num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
> > +
> > + r = amdgpu_job_alloc_with_ib(adev, &entity->base,
> > + AMDGPU_FENCE_OWNER_UNDEFINED,
> > + num_dw * 4 + num_bytes,
> > + AMDGPU_IB_POOL_DELAYED,
> > + &job,
> > + AMDGPU_KERNEL_JOB_ID_KFD_GART_MAP);
> > + if (r)
> > + return r;
> > +
> > + src_addr = num_dw * 4;
> > + src_addr += job->ibs[0].gpu_addr;
> > +
> > + dst_addr = amdgpu_bo_gpu_offset(adev->gart.bo);
> > + dst_addr += (entity->gart_window_offs[0] >> AMDGPU_GPU_PAGE_SHIFT) * 8;
> > + amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr,
> > + dst_addr, num_bytes, 0);
> > +
> > + amdgpu_ring_pad_ib(ring, &job->ibs[0]);
> > + WARN_ON(job->ibs[0].length_dw > num_dw);
> > +
> > + pte_flags = AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE;
> > + pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED;
> > + if (flags & AMDGPU_PTE_WRITEABLE)
> > + pte_flags |= AMDGPU_PTE_WRITEABLE;
> > + pte_flags |= adev->gart.gart_pte_flags;
> > +
> > + cpu_addr = &job->ibs[0].ptr[num_dw];
> > +
> > + amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr);
> > + fence = amdgpu_job_submit(job);
> > + dma_fence_put(fence);
> > +
> > + return 0;
> > +}
> > +
> > +/**
> > + * amdgpu_svm_copy_memory_gart - SDMA copy between system RAM and VRAM
> > + * @adev: AMDGPU device
> > + * @sys: Array of DMA addresses for system memory pages
> > + * @vram: Array of VRAM byte offsets (relative to start of VRAM)
> > + * @npages: Number of pages to copy
> > + * @direction: FROM_RAM_TO_VRAM or FROM_VRAM_TO_RAM
> > + * @mfence: In/out — carries the last SDMA fence for serialization
> > + *
> > + * Maps system memory pages into the GART window and uses SDMA to copy
> > + * data to/from VRAM. Handles splitting into AMDGPU_GTT_MAX_TRANSFER_SIZE
> > + * chunks. Acquires entity->lock internally to protect the GART window,
> > + * matching the KFD svm_migrate_copy_memory_gart() pattern.
> > + *
> > + * Return: 0 on success, negative error code on failure
> > + */
> > +static int
> > +amdgpu_svm_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys,
> > + u64 *vram, u64 npages, int direction,
> > + struct dma_fence **mfence)
> > +{
> > + const u64 max_pages = AMDGPU_GTT_MAX_TRANSFER_SIZE;
> > + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
> > + struct amdgpu_ttm_buffer_entity *entity = &adev->mman.move_entity;
> > + u64 gart_s, gart_d;
> > + struct dma_fence *next;
> > + u64 size;
> > + int r;
> > +
> > + mutex_lock(&entity->lock);
> > +
> > + while (npages) {
> > + size = min(max_pages, npages);
> > +
> > + if (direction == FROM_VRAM_TO_RAM) {
> > + gart_s = amdgpu_svm_direct_mapping_addr(adev, *vram);
> > + r = amdgpu_svm_gart_map(ring, entity, size, sys,
> > + &gart_d, AMDGPU_PTE_WRITEABLE);
> > + } else {
> > + r = amdgpu_svm_gart_map(ring, entity, size, sys,
> > + &gart_s, 0);
> > + gart_d = amdgpu_svm_direct_mapping_addr(adev, *vram);
> > + }
> > + if (r) {
> > + dev_err(adev->dev, "failed %d to map GART for SDMA\n", r);
> > + goto out_unlock;
> > + }
> > +
> > + AMDGPU_MIGRATE_TRACE("SDMA_COPY: %s npages=%llu vram_off=0x%llx\n",
> > + direction == FROM_RAM_TO_VRAM ? "RAM->VRAM" : "VRAM->RAM",
> > + size, (u64)*vram);
> > +
> > + r = amdgpu_copy_buffer(adev, entity, gart_s, gart_d,
> > + size * PAGE_SIZE,
> > + NULL, &next, true, 0);
> > + if (r) {
> > + dev_err(adev->dev, "failed %d to copy buffer\n", r);
> > + goto out_unlock;
> > + }
> > +
> > + dma_fence_put(*mfence);
> > + *mfence = next;
> > + npages -= size;
> > + if (npages) {
> > + sys += size;
> > + vram += size;
> > + }
> > + }
> > +
> > +out_unlock:
> > + mutex_unlock(&entity->lock);
> > +
> > + return r;
> > +}
> > +
> > +/**
> > + * amdgpu_svm_copy_to_devmem - SDMA copy system memory -> VRAM
> > + * @pages: Array of destination ZONE_DEVICE pages (VRAM-backed)
> > + * @pagemap_addr: Array of source DMA addresses (system memory, already mapped)
> > + * @npages: Number of pages to copy
> > + *
> > + * Builds parallel sys[] and vram[] arrays from the framework-provided
> > + * pagemap_addr and device pages, then submits batched SDMA copies via
> > + * the GART window.
> > + *
> > + * Return: 0 on success, negative error code on failure
> > + */
> > +static int
> > +amdgpu_svm_copy_to_devmem(struct page **pages,
> > + struct drm_pagemap_addr *pagemap_addr,
> > + unsigned long npages,
> > + struct dma_fence *pre_migrate_fence)
> > +{
> > + struct amdgpu_device *adev;
> > + struct amdgpu_pagemap *svm_dm;
> > + struct dma_fence *mfence = NULL;
> > + dma_addr_t *sys;
> > + u64 *vram;
> > + unsigned long i, j;
> > + int ret = 0;
> > +
> > + if (!npages)
> > + return 0;
> > +
> > + /*
> > + * Find the first non-NULL page to derive the device.
> > + * The pages array may contain NULL entries for positions where
> > + * no valid device page exists.
> > + */
> > + for (i = 0; i < npages; i++) {
> > + if (pages[i])
> > + break;
> > + }
> > + if (i == npages)
> > + return 0;
> > +
> > + svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
> > + adev = svm_dm->adev;
> > +
> > + sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
> > + vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
> > + if (!sys || !vram) {
> > + ret = -ENOMEM;
> > + goto out_free;
> > + }
> > +
> > + for (i = 0, j = 0; i < npages; i++) {
> > + if (!pagemap_addr[i].addr)
> > + goto flush;
> > +
> > + sys[j] = pagemap_addr[i].addr;
> > + vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> > + svm_dm->hpa_base;
> > +
> > + /* Check if next vram page is contiguous with current */
> > + if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
> > + goto flush;
> > +
> > + j++;
> > + continue;
> > +flush:
> > + if (j) {
> > + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> > + FROM_RAM_TO_VRAM,
> > + &mfence);
> > + if (ret)
> > + goto out_fence;
> > + j = 0;
> > + }
> > + /* Re-process current page if it was valid but broke contiguity */
> > + if (pagemap_addr[i].addr) {
> > + sys[0] = pagemap_addr[i].addr;
> > + vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> > + svm_dm->hpa_base;
> > + j = 1;
> > + }
> > + }
> > +
> > + /* Flush remaining batch */
> > + if (j)
> > + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> > + FROM_RAM_TO_VRAM, &mfence);
> > +
> > +out_fence:
> > + if (mfence) {
> > + dma_fence_wait(mfence, false);
> > + dma_fence_put(mfence);
> > + }
> > +
> > + AMDGPU_MIGRATE_TRACE("copy_to_devmem done: npages=%ld ret=%d\n",
> > + npages, ret);
> > +
> > +out_free:
> > + kvfree(vram);
> > + kvfree(sys);
> > + return ret;
> > +}
> > +
> > +/**
> > + * amdgpu_svm_copy_to_ram - SDMA copy VRAM -> system memory
> > + * @pages: Array of source ZONE_DEVICE pages (VRAM-backed)
> > + * @pagemap_addr: Array of destination DMA addresses (system memory, already mapped)
> > + * @npages: Number of pages to copy
> > + *
> > + * Mirror of copy_to_devmem with src/dst swapped.
> > + *
> > + * Return: 0 on success, negative error code on failure
> > + */
> > +static int
> > +amdgpu_svm_copy_to_ram(struct page **pages,
> > + struct drm_pagemap_addr *pagemap_addr,
> > + unsigned long npages,
> > + struct dma_fence *pre_migrate_fence)
> > +{
> > + struct amdgpu_device *adev;
> > + struct amdgpu_pagemap *svm_dm;
> > + struct dma_fence *mfence = NULL;
> > + dma_addr_t *sys;
> > + u64 *vram;
> > + unsigned long i, j;
> > + int ret = 0;
> > +
> > + if (!npages)
> > + return 0;
> > +
> > + for (i = 0; i < npages; i++) {
> > + if (pages[i])
> > + break;
> > + }
> > + if (i == npages)
> > + return 0;
> > +
> > + svm_dm = amdgpu_svm_page_to_apagemap(pages[i]);
> > + adev = svm_dm->adev;
> > +
> > + sys = kvcalloc(npages, sizeof(*sys), GFP_KERNEL);
> > + vram = kvcalloc(npages, sizeof(*vram), GFP_KERNEL);
> > + if (!sys || !vram) {
> > + ret = -ENOMEM;
> > + goto out_free;
> > + }
> > +
> > + for (i = 0, j = 0; i < npages; i++) {
> > + if (!pagemap_addr[i].addr || !pages[i])
> > + goto flush;
> > +
> > + vram[j] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> > + svm_dm->hpa_base;
> > + sys[j] = pagemap_addr[i].addr;
> > +
> > + /* Check if next vram page is contiguous with current */
> > + if (j > 0 && vram[j] != vram[j - 1] + PAGE_SIZE)
> > + goto flush;
> > +
> > + j++;
> > + continue;
> > +flush:
> > + if (j) {
> > + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> > + FROM_VRAM_TO_RAM,
> > + &mfence);
> > + if (ret)
> > + goto out_fence;
> > + j = 0;
> > + }
> > + /* Re-process current page if it was valid but broke contiguity */
> > + if (pagemap_addr[i].addr && pages[i]) {
> > + vram[0] = ((u64)page_to_pfn(pages[i]) << PAGE_SHIFT) -
> > + svm_dm->hpa_base;
> > + sys[0] = pagemap_addr[i].addr;
> > + j = 1;
> > + }
> > + }
> > +
> > + /* Flush remaining batch */
> > + if (j)
> > + ret = amdgpu_svm_copy_memory_gart(adev, sys, vram, j,
> > + FROM_VRAM_TO_RAM, &mfence);
> > +
> > +out_fence:
> > + if (mfence) {
> > + dma_fence_wait(mfence, false);
> > + dma_fence_put(mfence);
> > + }
> > +
> > + AMDGPU_MIGRATE_TRACE("copy_to_ram done: npages=%ld ret=%d\n", npages, ret);
> > +
> > +out_free:
> > + kvfree(vram);
> > + kvfree(sys);
> > + return ret;
> > +}
> > +
> > +static const struct drm_pagemap_devmem_ops amdgpu_pagemap_ops = {
> > + .devmem_release = amdgpu_svm_devmem_release,
> > + .populate_devmem_pfn = amdgpu_svm_populate_devmem_pfn,
> > + .copy_to_devmem = amdgpu_svm_copy_to_devmem,
> > + .copy_to_ram = amdgpu_svm_copy_to_ram,
> > +};
> > +
> > +/* drm_pagemap_ops — top-level migration entry points */
> > +
> > +/**
> > + * amdgpu_svm_device_map - Convert ZONE_DEVICE page to GPU PTE address
> > + * @dpagemap: The drm_pagemap for this device
> > + * @dev: Requesting device (for P2P check)
> > + * @page: ZONE_DEVICE page backed by VRAM
> > + * @order: Page order (0 = 4K, 9 = 2M, etc.)
> > + * @dir: DMA direction (unused for local VRAM)
> > + *
> > + * Address conversion chain:
> > + * page -> PFN -> HPA -> VRAM offset -> PTE address
> > + *
> > + * HPA = page_to_pfn(page) << PAGE_SHIFT
> > + * VRAM offset = HPA - apagemap.hpa_base
> > + * PTE address = VRAM offset + adev->vm_manager.vram_base_offset
> > + *
> > + * Return: drm_pagemap_addr with PTE address and AMDGPU_INTERCONNECT_VRAM protocol
> > + */
> > +static struct drm_pagemap_addr
> > +amdgpu_svm_device_map(struct drm_pagemap *dpagemap,
> > + struct device *dev,
> > + struct page *page,
> > + unsigned int order,
> > + enum dma_data_direction dir)
> > +{
> > + struct amdgpu_pagemap *svm_dm = to_amdgpu_pagemap(dpagemap);
> > + struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
> > + dma_addr_t addr;
> > +
> > + if (dpagemap->drm->dev == dev) {
> > + /* Same device: return VRAM PTE address */
> > + u64 hpa = (u64)page_to_pfn(page) << PAGE_SHIFT;
> > + u64 vram_offset = hpa - svm_dm->hpa_base;
> > +
> > + addr = vram_offset + adev->vm_manager.vram_base_offset;
> > + } else {
> > + /* Cross-device P2P: not yet supported */
> > + addr = DMA_MAPPING_ERROR;
> > + }
> > +
> > + return drm_pagemap_addr_encode(addr,
> > + AMDGPU_INTERCONNECT_VRAM, order, dir);
> > +}
> > +
> > +/**
> > + * amdgpu_svm_bo_alloc - Allocate an amdgpu_svm_bo wrapper with VRAM backing
> > + * @adev: AMDGPU device
> > + * @dpagemap: The drm_pagemap for this device
> > + * @mm: mm_struct of the owning process
> > + * @size: Allocation size in bytes
> > + *
> > + * Return: Pointer to allocated amdgpu_svm_bo on success, ERR_PTR on failure
> > + */
> > +static struct amdgpu_svm_bo *
> > +amdgpu_svm_bo_alloc(struct amdgpu_device *adev,
> > + struct drm_pagemap *dpagemap,
> > + struct mm_struct *mm, unsigned long size)
> > +{
> > + struct amdgpu_svm_bo *svm_bo;
> > + struct amdgpu_bo_param bp = {};
> > + struct amdgpu_bo *bo;
> > + int ret;
> > +
> > + svm_bo = kzalloc(sizeof(*svm_bo), GFP_KERNEL);
> > + if (!svm_bo)
> > + return ERR_PTR(-ENOMEM);
> > +
> > + bp.size = size;
> > + bp.bo_ptr_size = sizeof(struct amdgpu_bo);
> > + bp.domain = AMDGPU_GEM_DOMAIN_VRAM;
> > + bp.type = ttm_bo_type_device;
> > + bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS |
> > + AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
> > + AMDGPU_GEM_CREATE_VRAM_CLEARED;
> > +
> > + ret = amdgpu_bo_create(adev, &bp, &bo);
> > + if (ret) {
> > + AMDGPU_MIGRATE_TRACE("Failed to create SVM BO\n");
> > + kfree(svm_bo);
> > + return ERR_PTR(ret);
> > + }
> > +
> > + amdgpu_bo_unreserve(bo);
> > + svm_bo->bo = bo;
> > +
> > + drm_pagemap_devmem_init(&svm_bo->devmem,
> > + adev->dev, mm,
> > + &amdgpu_pagemap_ops,
> > + dpagemap, size, NULL);
> > +
> > + return svm_bo;
> > +}
> > +
> > +/**
> > + * amdgpu_svm_populate_mm - Allocate VRAM BO and migrate pages
> > + * @dpagemap: The drm_pagemap for this device
> > + * @start: Start virtual address of the range to migrate
> > + * @end: End virtual address (exclusive)
> > + * @mm: mm_struct of the owning process
> > + * @timeslice_ms: Maximum time to spend migrating (for fairness)
> > + *
> > + * Core migration entry point called by drm_pagemap_populate_mm().
> > + * Allocates an amdgpu_svm_bo via amdgpu_svm_bo_alloc(), then calls
> > + * drm_pagemap_migrate_to_devmem() to execute the actual migration.
> > + *
> > + * Return: 0 on success, negative error code on failure
> > + */
> > +static int
> > +amdgpu_svm_populate_mm(struct drm_pagemap *dpagemap,
> > + unsigned long start, unsigned long end,
> > + struct mm_struct *mm,
> > + unsigned long timeslice_ms)
> > +{
> > + struct amdgpu_device *adev = dpagemap_to_adev(dpagemap);
> > + struct drm_pagemap_migrate_details mdetails = {
> > + .timeslice_ms = timeslice_ms,
> > + };
> > + struct amdgpu_svm_bo *svm_bo;
> > + int ret;
> > +
> > + svm_bo = amdgpu_svm_bo_alloc(adev, dpagemap, mm, end - start);
> > + if (IS_ERR(svm_bo))
> > + return PTR_ERR(svm_bo);
> > +
> > + AMDGPU_MIGRATE_TRACE("populate_mm: [0x%lx-0x%lx] size=%lu\n",
> > + start, end, end - start);
> > +
> > + ret = drm_pagemap_migrate_to_devmem(&svm_bo->devmem,
> > + mm, start, end,
> > + &mdetails);
> > +
> > + return ret;
> > +}
> > -const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = { };
> > +const struct drm_pagemap_ops amdgpu_svm_drm_pagemap_ops = {
> > + .device_map = amdgpu_svm_device_map,
> > + .populate_mm = amdgpu_svm_populate_mm,
> > +};
> > /**
> > * amdgpu_svm_migration_init - Register ZONE_DEVICE and initialize drm_pagemap
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-04-28 7:39 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-04-27 10:05 [PATCH v3 0/5] drm/amdgpu: SVM VRAM migration via drm_pagemap Junhua Shen
2026-04-27 10:05 ` [PATCH v3 1/5] drm/amdgpu: add VRAM migration infrastructure for drm_pagemap Junhua Shen
2026-04-27 10:05 ` [PATCH v3 2/5] drm/amdgpu: implement drm_pagemap SDMA migration callbacks Junhua Shen
2026-04-27 22:20 ` Felix Kuehling
2026-04-28 7:39 ` Junhua Shen
2026-04-27 10:05 ` [PATCH v3 3/5] drm/amdgpu: introduce SVM range migration decision layer Junhua Shen
2026-04-27 10:05 ` [PATCH v3 4/5] drm/amdgpu: add SVM attr prefetch/force-trigger functionality Junhua Shen
2026-04-27 10:05 ` [PATCH v3 5/5] drm/amdgpu: integrate VRAM migration into SVM range map path Junhua Shen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox