* [PATCH v2] drm/pagemap: Prevent double migration of device pages
@ 2026-08-06 6:11 Arvind Yadav
2026-08-06 7:21 ` Matthew Brost
0 siblings, 1 reply; 2+ messages in thread
From: Arvind Yadav @ 2026-08-06 6:11 UTC (permalink / raw)
To: dri-devel, intel-xe, linux-kernel
Cc: matthew.brost, thomas.hellstrom, maarten.lankhorst, mripard,
tzimmermann, airlied, simona, himal.prasad.ghimiray
A device page migrated to system memory by a CPU fault can remain
referenced after migration completes. During this window, raw-PFN
eviction can collect the same device PFN and migrate it again.
The first migration has already transferred the memcg charge from the
source folio. Migrating that source again can create an uncharged system
folio. Adding such a folio to the LRU can spin indefinitely in
folio_lruvec_lock_irqsave(), causing a soft lockup and an RCU stall.
Track successful device-page migrations in an allocation-relative bitmap
stored in drm_pagemap_zdd. Record successful migrations before finalize
unlocks and drops the migration reference on the source.
Make raw-PFN eviction skip retired folios. Record raw-PFN migrations as
well so an eviction retry cannot select a folio migrated by an earlier
pass.
Mark every base-page bit covered by a migrated folio so the retirement
state remains valid if the source folio is later split.
v2:
- Replace the retired-PFN XArray with an embedded bitmap.(Matthew Brost)
- Mark every base page covered by a migrated folio so retirement remains
valid if the folio is later split.
The lockup was observed as:
==============================
[10109.860465] watchdog: BUG: soft lockup - CPU#9 stuck for 26s! [kworker/u65:5:6557]
[10109.860508] irq event stamp: 308085402
[10109.860508] hardirqs last enabled at (308085401): [<ffffffff829571e1>] _raw_spin_unlock_irqrestore+0x51/0x80
[10109.860514] hardirqs last disabled at (308085402): [<ffffffff8293dd71>] sysvec_apic_timer_interrupt+0x11/0xc0
[10109.860516] softirqs last enabled at (307538534): [<ffffffff813d111b>] __irq_exit_rcu+0xdb/0x1c0
[10109.860519] softirqs last disabled at (307538529): [<ffffffff813d111b>] __irq_exit_rcu+0xdb/0x1c0
[10109.860521] CPU: 9 UID: 0 PID: 6557 Comm: kworker/u65:5 Kdump: loaded Tainted: G S O 7.2.0-rc3-lgci-xepurge-
[10109.860524] Tainted: [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE
[10109.860524] Hardware name: ASUS System Product Name/PRIME Z790-P WIFI, BIOS 0812 02/24/2023
[10109.860525] Workqueue: xe_page_fault_work_queue xe_pagefault_queue_work [xe]
[10109.860644] RIP: 0010:_raw_spin_unlock_irqrestore+0x57/0x80
[10109.860647] Code: 00 75 1c 65 ff 0d 69 ee 84 01 74 20 5b 41 5c 5d 31 c0 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc e8 8f 2e c9
[10109.860648] RSP: 0018:ffffc9000c017030 EFLAGS: 00000246
[10109.860649] RAX: 0000000000000000 RBX: ffff8881012c00d0 RCX: 0000000000000000
[10109.860650] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[10109.860650] RBP: ffffc9000c017040 R08: 0000000000000000 R09: 0000000000000000
[10109.860651] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
[10109.860651] R13: ffffc9000c0170a0 R14: ffff8881012c00d0 R15: ffff88810e9e6f00
[10109.860652] FS: 0000000000000000(0000) GS:ffff8888db2f2000(0000) knlGS:0000000000000000
[10109.860653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10109.860653] CR2: 00007a26c0e00048 CR3: 00000001f366d006 CR4: 0000000000f72ef0
[10109.860654] PKRU: 55555554
[10109.860655] Call Trace:
[10109.860655] <TASK>
[10109.860657] folio_lruvec_lock_irqsave+0x216/0x220
[10109.860661] ? __pfx_lru_add+0x10/0x10
[10109.860665] folio_batch_move_lru+0xc8/0x450
[10109.860670] ? lock_acquire+0xc4/0x2d0
[10109.860674] ? __folio_batch_add_and_move+0x60/0x2e0
[10109.860677] ? folio_migrate_mapping+0xa6/0x110
[10109.860679] ? folio_migrate_flags+0x13b/0x1b0
[10109.860681] ? __pfx_lru_add+0x10/0x10
[10109.860683] __folio_batch_add_and_move+0xe7/0x2e0
[10109.860685] ? dma_iova_try_alloc+0xb0/0x140
[10109.860689] folio_add_lru+0x64/0x80
[10109.860691] __migrate_device_finalize+0x12c/0x270
[10109.860695] migrate_device_finalize+0x10/0x20
[10109.860698] drm_pagemap_evict_to_ram+0x185/0x370 [drm_gpusvm_helper]
[10109.860704] ? drm_pagemap_evict_to_ram+0x96/0x370 [drm_gpusvm_helper]
[10109.860709] xe_svm_bo_evict+0x15/0x20 [xe]
[10109.860819] ? xe_svm_bo_evict+0x15/0x20 [xe]
[10109.860921] xe_bo_move+0x107e/0x1570 [xe]
[10109.860992] ? xe_ttm_tt_create+0x168/0x340 [xe]
[10109.861059] ? __up_read+0x98/0x2b0
[10109.861061] ? lock_is_held_type+0xa3/0x130
[10109.861067] ttm_bo_handle_move_mem+0xe8/0x1e0 [ttm]
[10109.861075] ttm_bo_evict+0x141/0x1c0 [ttm]
[10109.861081] ttm_bo_evict_cb+0x9f/0x100 [ttm]
[10109.861086] ttm_lru_walk_for_evict+0x84/0x190 [ttm]
[10109.861091] ? xe_ttm_vram_mgr_new+0x258/0x3a0 [xe]
[10109.861198] ttm_bo_alloc_resource+0x219/0x750 [ttm]
[10109.861203] ? ttm_bo_alloc_resource+0xa9/0x750 [ttm]
[10109.861208] ? lock_acquire+0xc4/0x2d0
[10109.861214] ttm_bo_validate+0x94/0x1c0 [ttm]
[10109.861218] ? ww_mutex_trylock+0x19d/0x3d0
[10109.861219] ? _raw_write_unlock+0x22/0x50
[10109.861223] ttm_bo_init_reserved+0x17d/0x1f0 [ttm]
[10109.861228] xe_bo_init_locked+0x20a/0x620 [xe]
[10109.861294] ? __pfx_xe_ttm_bo_destroy+0x10/0x10 [xe]
[10109.861359] ? mark_held_locks+0x46/0x90
[10109.861361] ? __create_object+0x68/0xc0
[10109.861366] __xe_bo_create_locked+0x384/0xa20 [xe]
[10109.861432] ? lock_acquire+0xc4/0x2d0
[10109.861434] ? xe_drm_pagemap_populate_mm+0xd3/0x340 [xe]
[10109.861542] xe_bo_create_locked+0x23/0x40 [xe]
[10109.861609] xe_drm_pagemap_populate_mm+0x12e/0x340 [xe]
[10109.861707] ? __lock_acquire+0x43e/0x2930
[10109.861716] drm_pagemap_populate_mm+0x74/0xe0 [drm_gpusvm_helper]
[10109.861720] xe_svm_alloc_vram+0xb5/0x2c0 [xe]
[10109.861817] ? seqcount_lockdep_reader_access.constprop.0+0x9f/0xc0
[10109.861819] ? ktime_get+0x23/0x130
[10109.861821] ? trace_hardirqs_on+0x22/0xe0
[10109.861823] ? seqcount_lockdep_reader_access.constprop.0+0x9f/0xc0
[10109.861826] __xe_svm_handle_pagefault+0x77d/0xbf0 [xe]
[10109.861924] ? rwsem_down_write_slowpath+0x43a/0x9a0
[10109.861926] ? _raw_spin_unlock_irq+0x27/0x70
[10109.861928] ? rwsem_down_write_slowpath+0x43a/0x9a0
[10109.861929] ? trace_hardirqs_on+0x22/0xe0
[10109.861931] ? _raw_spin_unlock_irq+0x27/0x70
[10109.861933] ? rwsem_down_write_slowpath+0x459/0x9a0
[10109.861937] xe_svm_handle_pagefault+0x3d/0xb0 [xe]
[10109.862030] xe_pagefault_queue_work+0x1a9/0x520 [xe]
[10109.862122] process_one_work+0x239/0x730
[10109.862127] worker_thread+0x200/0x3f0
[10109.862130] ? __pfx_worker_thread+0x10/0x10
[10109.862132] kthread+0x10d/0x150
[10109.862133] ? __pfx_kthread+0x10/0x10
[10109.862135] ret_from_fork+0x3bd/0x470
[10109.862138] ? __pfx_kthread+0x10/0x10
[10109.862140] ret_from_fork_asm+0x1a/0x30
[10109.862146] </TASK>
Fixes: 99624bdff867 ("drm/gpusvm: Add support for GPU Shared Virtual Memory")
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Arvind Yadav <arvind.yadav@intel.com>
---
drivers/gpu/drm/drm_pagemap.c | 143 ++++++++++++++++++++++++++++++++--
1 file changed, 136 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
index 7a056592ac66..713095e27006 100644
--- a/drivers/gpu/drm/drm_pagemap.c
+++ b/drivers/gpu/drm/drm_pagemap.c
@@ -7,6 +7,7 @@
#include <linux/dma-mapping.h>
#include <linux/migrate.h>
#include <linux/pagemap.h>
+#include <linux/bitmap.h>
#include <drm/drm_drv.h>
#include <drm/drm_pagemap.h>
#include <drm/drm_pagemap_util.h>
@@ -66,6 +67,12 @@
* @refcount: Reference count for the zdd
* @devmem_allocation: device memory allocation
* @dpagemap: Refcounted pointer to the underlying struct drm_pagemap.
+ * @range_start: Virtual start address of the device memory allocation. Used to
+ * translate a clipped CPU-fault range into an allocation-relative page offset.
+ * @range_npages: Number of pages covered by the allocation, i.e. the number of
+ * valid bits in @retire_map.
+ * @retire_map: Allocation-relative bitmap. Every base page covered by a
+ * migrated folio remains marked until this mapping generation is destroyed.
*
* This structure serves as a generic wrapper installed in
* page->zone_device_data. It provides infrastructure for looking up a device
@@ -78,29 +85,38 @@ struct drm_pagemap_zdd {
struct kref refcount;
struct drm_pagemap_devmem *devmem_allocation;
struct drm_pagemap *dpagemap;
+ unsigned long range_start;
+ unsigned long range_npages;
+ unsigned long retire_map[];
};
/**
* drm_pagemap_zdd_alloc() - Allocate a zdd structure.
* @dpagemap: Pointer to the underlying struct drm_pagemap.
+ * @start: Virtual start address of the device memory allocation.
+ * @npages: Number of pages in the device memory allocation.
*
* This function allocates and initializes a new zdd structure. It sets up the
- * reference count and initializes the destroy work.
+ * reference count and a zeroed retirement bitmap sized for @npages.
*
- * Return: Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ * Return: Pointer to the allocated zdd on success, NULL on failure.
*/
static struct drm_pagemap_zdd *
-drm_pagemap_zdd_alloc(struct drm_pagemap *dpagemap)
+drm_pagemap_zdd_alloc(struct drm_pagemap *dpagemap, unsigned long start,
+ unsigned long npages)
{
struct drm_pagemap_zdd *zdd;
- zdd = kmalloc_obj(*zdd);
+ zdd = kzalloc(struct_size(zdd, retire_map, BITS_TO_LONGS(npages)),
+ GFP_KERNEL);
if (!zdd)
return NULL;
kref_init(&zdd->refcount);
zdd->devmem_allocation = NULL;
zdd->dpagemap = drm_pagemap_get(dpagemap);
+ zdd->range_start = start;
+ zdd->range_npages = npages;
return zdd;
}
@@ -669,7 +685,7 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
pagemap_addr = buf + (2 * sizeof(*migrate.src) * npages);
pages = buf + (2 * sizeof(*migrate.src) + sizeof(*pagemap_addr)) * npages;
- zdd = drm_pagemap_zdd_alloc(dpagemap);
+ zdd = drm_pagemap_zdd_alloc(dpagemap, start, npages);
if (!zdd) {
err = -ENOMEM;
kvfree(buf);
@@ -1102,12 +1118,115 @@ void drm_pagemap_put(struct drm_pagemap *dpagemap)
}
EXPORT_SYMBOL(drm_pagemap_put);
+/**
+ * drm_pagemap_is_devmem_page() - Check whether a page is device memory
+ * @page: page to check
+ *
+ * Return: true if @page is device-private or device-coherent
+ */
+static bool drm_pagemap_is_devmem_page(const struct page *page)
+{
+ return is_device_private_page(page) || is_device_coherent_page(page);
+}
+
+/**
+ * drm_pagemap_retire_migrated_pages() - Record migrated device folios
+ * @src_pfns: source array after migrate_vma_pages() or migrate_device_pages()
+ * @npages: number of entries in @src_pfns
+ * @first: allocation-relative page offset of @src_pfns[0]
+ *
+ * Record successful migrations in the zdd retirement bitmap before finalize
+ * unlocks the sources. The bit is set only after a confirmed migration, and
+ * test_and_set_bit() is used because concurrent CPU faults on clipped ranges
+ * can update different bits in the same word.
+ */
+static void drm_pagemap_retire_migrated_pages(unsigned long *src_pfns,
+ unsigned long npages,
+ unsigned long first)
+{
+ unsigned long i = 0;
+
+ while (i < npages) {
+ struct page *page = migrate_pfn_to_page(src_pfns[i]);
+ struct drm_pagemap_zdd *zdd;
+ unsigned long bit, j, nr = 1;
+
+ if (!page) {
+ i++;
+ continue;
+ }
+
+ nr = folio_nr_pages(page_folio(page));
+
+ if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) ||
+ !drm_pagemap_is_devmem_page(page))
+ goto next;
+
+ zdd = drm_pagemap_page_zone_device_data(page);
+ bit = first + i;
+
+ if (WARN_ON_ONCE(bit >= zdd->range_npages ||
+ nr > zdd->range_npages - bit))
+ goto next;
+
+ /* Keep later folio splits covered. */
+ for (j = 0; j < nr; j++)
+ WARN_ON_ONCE(test_and_set_bit(bit + j, zdd->retire_map));
+next:
+ i += nr;
+ }
+}
+
+/**
+ * drm_pagemap_skip_retired_pages() - Drop retired PFNs from a raw-PFN eviction
+ * @src_pfns: source array after migrate_device_pfns() (MIGRATE_PFN encoded)
+ * @npages: number of entries in @src_pfns
+ *
+ * Skip source PFNs already migrated to RAM by either migration path. The
+ * raw-PFN eviction array starts at allocation offset zero, so the array index
+ * is also the retirement bitmap index.
+ */
+static void drm_pagemap_skip_retired_pages(unsigned long *src_pfns,
+ unsigned long npages)
+{
+ unsigned long i = 0;
+
+ while (i < npages) {
+ struct page *page = migrate_pfn_to_page(src_pfns[i]);
+ struct drm_pagemap_zdd *zdd;
+ unsigned long nr = 1;
+
+ if (!page) {
+ i++;
+ continue;
+ }
+
+ nr = folio_nr_pages(page_folio(page));
+
+ if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) ||
+ !drm_pagemap_is_devmem_page(page))
+ goto next;
+
+ zdd = drm_pagemap_page_zone_device_data(page);
+ if (WARN_ON_ONCE(i >= zdd->range_npages)) {
+ src_pfns[i] &= ~MIGRATE_PFN_MIGRATE;
+ goto next;
+ }
+
+ if (test_bit(i, zdd->retire_map))
+ src_pfns[i] &= ~MIGRATE_PFN_MIGRATE;
+next:
+ i += nr;
+ }
+}
+
/**
* drm_pagemap_evict_to_ram() - Evict GPU SVM range to RAM
* @devmem_allocation: Pointer to the device memory allocation
*
- * Similar to __drm_pagemap_migrate_to_ram but does not require mmap lock and
- * migration done via migrate_device_* functions.
+ * Similar to __drm_pagemap_migrate_to_ram(), but uses the
+ * migrate_device_* helpers and does not require the mmap lock. Device
+ * PFNs already migrated to RAM by either migration path are skipped.
*
* Return: 0 on success, negative error code on failure.
*/
@@ -1149,6 +1268,8 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
if (err)
goto err_free;
+ drm_pagemap_skip_retired_pages(src, npages);
+
err = drm_pagemap_migrate_populate_ram_pfn(NULL, NULL, npages, &mpages,
src, dst, 0);
if (err || !mpages)
@@ -1179,6 +1300,8 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
if (err)
drm_pagemap_migration_unlock_put_pages(npages, dst);
migrate_device_pages(src, dst, npages);
+ /* Raw-PFN eviction: array starts at allocation offset zero. */
+ drm_pagemap_retire_migrated_pages(src, npages, 0);
migrate_device_finalize(src, dst, npages);
drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, dst, npages,
DMA_FROM_DEVICE, &state);
@@ -1251,6 +1374,10 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
if (end > vas->vm_end)
end = vas->vm_end;
+ /* Keep the range within the ZDD allocation so retirement offsets stay valid. */
+ start = max(start, zdd->range_start);
+ end = min(end, zdd->range_start + (zdd->range_npages << PAGE_SHIFT));
+
migrate.start = start;
migrate.end = end;
npages = npages_in_range(start, end);
@@ -1309,6 +1436,8 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
if (err)
drm_pagemap_migration_unlock_put_pages(npages, migrate.dst);
migrate_vma_pages(&migrate);
+ drm_pagemap_retire_migrated_pages(migrate.src, npages,
+ (start - zdd->range_start) >> PAGE_SHIFT);
migrate_vma_finalize(&migrate);
if (dev)
drm_pagemap_migrate_unmap_pages(dev, pagemap_addr, migrate.dst,
--
2.43.0
^ permalink raw reply related [flat|nested] 2+ messages in thread* Re: [PATCH v2] drm/pagemap: Prevent double migration of device pages
2026-08-06 6:11 [PATCH v2] drm/pagemap: Prevent double migration of device pages Arvind Yadav
@ 2026-08-06 7:21 ` Matthew Brost
0 siblings, 0 replies; 2+ messages in thread
From: Matthew Brost @ 2026-08-06 7:21 UTC (permalink / raw)
To: Arvind Yadav
Cc: dri-devel, intel-xe, linux-kernel, thomas.hellstrom,
maarten.lankhorst, mripard, tzimmermann, airlied, simona,
himal.prasad.ghimiray
On Thu, Aug 06, 2026 at 11:41:26AM +0530, Arvind Yadav wrote:
> A device page migrated to system memory by a CPU fault can remain
> referenced after migration completes. During this window, raw-PFN
> eviction can collect the same device PFN and migrate it again.
>
> The first migration has already transferred the memcg charge from the
> source folio. Migrating that source again can create an uncharged system
> folio. Adding such a folio to the LRU can spin indefinitely in
> folio_lruvec_lock_irqsave(), causing a soft lockup and an RCU stall.
>
> Track successful device-page migrations in an allocation-relative bitmap
> stored in drm_pagemap_zdd. Record successful migrations before finalize
> unlocks and drops the migration reference on the source.
>
> Make raw-PFN eviction skip retired folios. Record raw-PFN migrations as
> well so an eviction retry cannot select a folio migrated by an earlier
> pass.
>
> Mark every base-page bit covered by a migrated folio so the retirement
> state remains valid if the source folio is later split.
>
> v2:
> - Replace the retired-PFN XArray with an embedded bitmap.(Matthew Brost)
> - Mark every base page covered by a migrated folio so retirement remains
> valid if the folio is later split.
>
> The lockup was observed as:
> ==============================
> [10109.860465] watchdog: BUG: soft lockup - CPU#9 stuck for 26s! [kworker/u65:5:6557]
> [10109.860508] irq event stamp: 308085402
> [10109.860508] hardirqs last enabled at (308085401): [<ffffffff829571e1>] _raw_spin_unlock_irqrestore+0x51/0x80
> [10109.860514] hardirqs last disabled at (308085402): [<ffffffff8293dd71>] sysvec_apic_timer_interrupt+0x11/0xc0
> [10109.860516] softirqs last enabled at (307538534): [<ffffffff813d111b>] __irq_exit_rcu+0xdb/0x1c0
> [10109.860519] softirqs last disabled at (307538529): [<ffffffff813d111b>] __irq_exit_rcu+0xdb/0x1c0
> [10109.860521] CPU: 9 UID: 0 PID: 6557 Comm: kworker/u65:5 Kdump: loaded Tainted: G S O 7.2.0-rc3-lgci-xepurge-
> [10109.860524] Tainted: [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE
> [10109.860524] Hardware name: ASUS System Product Name/PRIME Z790-P WIFI, BIOS 0812 02/24/2023
> [10109.860525] Workqueue: xe_page_fault_work_queue xe_pagefault_queue_work [xe]
> [10109.860644] RIP: 0010:_raw_spin_unlock_irqrestore+0x57/0x80
> [10109.860647] Code: 00 75 1c 65 ff 0d 69 ee 84 01 74 20 5b 41 5c 5d 31 c0 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc e8 8f 2e c9
> [10109.860648] RSP: 0018:ffffc9000c017030 EFLAGS: 00000246
> [10109.860649] RAX: 0000000000000000 RBX: ffff8881012c00d0 RCX: 0000000000000000
> [10109.860650] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> [10109.860650] RBP: ffffc9000c017040 R08: 0000000000000000 R09: 0000000000000000
> [10109.860651] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
> [10109.860651] R13: ffffc9000c0170a0 R14: ffff8881012c00d0 R15: ffff88810e9e6f00
> [10109.860652] FS: 0000000000000000(0000) GS:ffff8888db2f2000(0000) knlGS:0000000000000000
> [10109.860653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [10109.860653] CR2: 00007a26c0e00048 CR3: 00000001f366d006 CR4: 0000000000f72ef0
> [10109.860654] PKRU: 55555554
> [10109.860655] Call Trace:
> [10109.860655] <TASK>
> [10109.860657] folio_lruvec_lock_irqsave+0x216/0x220
> [10109.860661] ? __pfx_lru_add+0x10/0x10
> [10109.860665] folio_batch_move_lru+0xc8/0x450
> [10109.860670] ? lock_acquire+0xc4/0x2d0
> [10109.860674] ? __folio_batch_add_and_move+0x60/0x2e0
> [10109.860677] ? folio_migrate_mapping+0xa6/0x110
> [10109.860679] ? folio_migrate_flags+0x13b/0x1b0
> [10109.860681] ? __pfx_lru_add+0x10/0x10
> [10109.860683] __folio_batch_add_and_move+0xe7/0x2e0
> [10109.860685] ? dma_iova_try_alloc+0xb0/0x140
> [10109.860689] folio_add_lru+0x64/0x80
> [10109.860691] __migrate_device_finalize+0x12c/0x270
> [10109.860695] migrate_device_finalize+0x10/0x20
> [10109.860698] drm_pagemap_evict_to_ram+0x185/0x370 [drm_gpusvm_helper]
> [10109.860704] ? drm_pagemap_evict_to_ram+0x96/0x370 [drm_gpusvm_helper]
> [10109.860709] xe_svm_bo_evict+0x15/0x20 [xe]
> [10109.860819] ? xe_svm_bo_evict+0x15/0x20 [xe]
> [10109.860921] xe_bo_move+0x107e/0x1570 [xe]
> [10109.860992] ? xe_ttm_tt_create+0x168/0x340 [xe]
> [10109.861059] ? __up_read+0x98/0x2b0
> [10109.861061] ? lock_is_held_type+0xa3/0x130
> [10109.861067] ttm_bo_handle_move_mem+0xe8/0x1e0 [ttm]
> [10109.861075] ttm_bo_evict+0x141/0x1c0 [ttm]
> [10109.861081] ttm_bo_evict_cb+0x9f/0x100 [ttm]
> [10109.861086] ttm_lru_walk_for_evict+0x84/0x190 [ttm]
> [10109.861091] ? xe_ttm_vram_mgr_new+0x258/0x3a0 [xe]
> [10109.861198] ttm_bo_alloc_resource+0x219/0x750 [ttm]
> [10109.861203] ? ttm_bo_alloc_resource+0xa9/0x750 [ttm]
> [10109.861208] ? lock_acquire+0xc4/0x2d0
> [10109.861214] ttm_bo_validate+0x94/0x1c0 [ttm]
> [10109.861218] ? ww_mutex_trylock+0x19d/0x3d0
> [10109.861219] ? _raw_write_unlock+0x22/0x50
> [10109.861223] ttm_bo_init_reserved+0x17d/0x1f0 [ttm]
> [10109.861228] xe_bo_init_locked+0x20a/0x620 [xe]
> [10109.861294] ? __pfx_xe_ttm_bo_destroy+0x10/0x10 [xe]
> [10109.861359] ? mark_held_locks+0x46/0x90
> [10109.861361] ? __create_object+0x68/0xc0
> [10109.861366] __xe_bo_create_locked+0x384/0xa20 [xe]
> [10109.861432] ? lock_acquire+0xc4/0x2d0
> [10109.861434] ? xe_drm_pagemap_populate_mm+0xd3/0x340 [xe]
> [10109.861542] xe_bo_create_locked+0x23/0x40 [xe]
> [10109.861609] xe_drm_pagemap_populate_mm+0x12e/0x340 [xe]
> [10109.861707] ? __lock_acquire+0x43e/0x2930
> [10109.861716] drm_pagemap_populate_mm+0x74/0xe0 [drm_gpusvm_helper]
> [10109.861720] xe_svm_alloc_vram+0xb5/0x2c0 [xe]
> [10109.861817] ? seqcount_lockdep_reader_access.constprop.0+0x9f/0xc0
> [10109.861819] ? ktime_get+0x23/0x130
> [10109.861821] ? trace_hardirqs_on+0x22/0xe0
> [10109.861823] ? seqcount_lockdep_reader_access.constprop.0+0x9f/0xc0
> [10109.861826] __xe_svm_handle_pagefault+0x77d/0xbf0 [xe]
> [10109.861924] ? rwsem_down_write_slowpath+0x43a/0x9a0
> [10109.861926] ? _raw_spin_unlock_irq+0x27/0x70
> [10109.861928] ? rwsem_down_write_slowpath+0x43a/0x9a0
> [10109.861929] ? trace_hardirqs_on+0x22/0xe0
> [10109.861931] ? _raw_spin_unlock_irq+0x27/0x70
> [10109.861933] ? rwsem_down_write_slowpath+0x459/0x9a0
> [10109.861937] xe_svm_handle_pagefault+0x3d/0xb0 [xe]
> [10109.862030] xe_pagefault_queue_work+0x1a9/0x520 [xe]
> [10109.862122] process_one_work+0x239/0x730
> [10109.862127] worker_thread+0x200/0x3f0
> [10109.862130] ? __pfx_worker_thread+0x10/0x10
> [10109.862132] kthread+0x10d/0x150
> [10109.862133] ? __pfx_kthread+0x10/0x10
> [10109.862135] ret_from_fork+0x3bd/0x470
> [10109.862138] ? __pfx_kthread+0x10/0x10
> [10109.862140] ret_from_fork_asm+0x1a/0x30
> [10109.862146] </TASK>
>
> Fixes: 99624bdff867 ("drm/gpusvm: Add support for GPU Shared Virtual Memory")
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Maxime Ripard <mripard@kernel.org>
> Cc: Thomas Zimmermann <tzimmermann@suse.de>
> Cc: David Airlie <airlied@gmail.com>
> Cc: Simona Vetter <simona@ffwll.ch>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Cc: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
> Assisted-by: Claude:claude-opus-4-8
> Signed-off-by: Arvind Yadav <arvind.yadav@intel.com>
> ---
> drivers/gpu/drm/drm_pagemap.c | 143 ++++++++++++++++++++++++++++++++--
> 1 file changed, 136 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c
> index 7a056592ac66..713095e27006 100644
> --- a/drivers/gpu/drm/drm_pagemap.c
> +++ b/drivers/gpu/drm/drm_pagemap.c
> @@ -7,6 +7,7 @@
> #include <linux/dma-mapping.h>
> #include <linux/migrate.h>
> #include <linux/pagemap.h>
> +#include <linux/bitmap.h>
> #include <drm/drm_drv.h>
> #include <drm/drm_pagemap.h>
> #include <drm/drm_pagemap_util.h>
> @@ -66,6 +67,12 @@
> * @refcount: Reference count for the zdd
> * @devmem_allocation: device memory allocation
> * @dpagemap: Refcounted pointer to the underlying struct drm_pagemap.
> + * @range_start: Virtual start address of the device memory allocation. Used to
> + * translate a clipped CPU-fault range into an allocation-relative page offset.
> + * @range_npages: Number of pages covered by the allocation, i.e. the number of
> + * valid bits in @retire_map.
> + * @retire_map: Allocation-relative bitmap. Every base page covered by a
> + * migrated folio remains marked until this mapping generation is destroyed.
> *
> * This structure serves as a generic wrapper installed in
> * page->zone_device_data. It provides infrastructure for looking up a device
> @@ -78,29 +85,38 @@ struct drm_pagemap_zdd {
> struct kref refcount;
> struct drm_pagemap_devmem *devmem_allocation;
> struct drm_pagemap *dpagemap;
> + unsigned long range_start;
Sashiko pointed this out as well: you can't store anything virtual in a
ZDD because it represents a physical object. The range_start may or may
not be the same by the time a fault occurs.
> + unsigned long range_npages;
devmem_allocation->size can derive the number of pages.
> + unsigned long retire_map[];
This won't be needed or any changes to the zdd actually, more below.
> };
>
> /**
> * drm_pagemap_zdd_alloc() - Allocate a zdd structure.
> * @dpagemap: Pointer to the underlying struct drm_pagemap.
> + * @start: Virtual start address of the device memory allocation.
> + * @npages: Number of pages in the device memory allocation.
> *
> * This function allocates and initializes a new zdd structure. It sets up the
> - * reference count and initializes the destroy work.
> + * reference count and a zeroed retirement bitmap sized for @npages.
> *
> - * Return: Pointer to the allocated zdd on success, ERR_PTR() on failure.
> + * Return: Pointer to the allocated zdd on success, NULL on failure.
> */
> static struct drm_pagemap_zdd *
> -drm_pagemap_zdd_alloc(struct drm_pagemap *dpagemap)
> +drm_pagemap_zdd_alloc(struct drm_pagemap *dpagemap, unsigned long start,
> + unsigned long npages)
> {
> struct drm_pagemap_zdd *zdd;
>
> - zdd = kmalloc_obj(*zdd);
> + zdd = kzalloc(struct_size(zdd, retire_map, BITS_TO_LONGS(npages)),
> + GFP_KERNEL);
> if (!zdd)
> return NULL;
>
> kref_init(&zdd->refcount);
> zdd->devmem_allocation = NULL;
> zdd->dpagemap = drm_pagemap_get(dpagemap);
> + zdd->range_start = start;
> + zdd->range_npages = npages;
>
> return zdd;
> }
> @@ -669,7 +685,7 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
> pagemap_addr = buf + (2 * sizeof(*migrate.src) * npages);
> pages = buf + (2 * sizeof(*migrate.src) + sizeof(*pagemap_addr)) * npages;
>
> - zdd = drm_pagemap_zdd_alloc(dpagemap);
> + zdd = drm_pagemap_zdd_alloc(dpagemap, start, npages);
> if (!zdd) {
> err = -ENOMEM;
> kvfree(buf);
> @@ -1102,12 +1118,115 @@ void drm_pagemap_put(struct drm_pagemap *dpagemap)
> }
> EXPORT_SYMBOL(drm_pagemap_put);
>
> +/**
> + * drm_pagemap_is_devmem_page() - Check whether a page is device memory
> + * @page: page to check
> + *
> + * Return: true if @page is device-private or device-coherent
> + */
> +static bool drm_pagemap_is_devmem_page(const struct page *page)
> +{
> + return is_device_private_page(page) || is_device_coherent_page(page);
> +}
> +
> +/**
> + * drm_pagemap_retire_migrated_pages() - Record migrated device folios
> + * @src_pfns: source array after migrate_vma_pages() or migrate_device_pages()
> + * @npages: number of entries in @src_pfns
> + * @first: allocation-relative page offset of @src_pfns[0]
> + *
> + * Record successful migrations in the zdd retirement bitmap before finalize
> + * unlocks the sources. The bit is set only after a confirmed migration, and
> + * test_and_set_bit() is used because concurrent CPU faults on clipped ranges
> + * can update different bits in the same word.
> + */
> +static void drm_pagemap_retire_migrated_pages(unsigned long *src_pfns,
> + unsigned long npages,
> + unsigned long first)
> +{
> + unsigned long i = 0;
> +
> + while (i < npages) {
> + struct page *page = migrate_pfn_to_page(src_pfns[i]);
> + struct drm_pagemap_zdd *zdd;
> + unsigned long bit, j, nr = 1;
> +
> + if (!page) {
> + i++;
> + continue;
> + }
> +
> + nr = folio_nr_pages(page_folio(page));
> +
> + if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) ||
> + !drm_pagemap_is_devmem_page(page))
> + goto next;
> +
> + zdd = drm_pagemap_page_zone_device_data(page);
> + bit = first + i;
> +
> + if (WARN_ON_ONCE(bit >= zdd->range_npages ||
> + nr > zdd->range_npages - bit))
> + goto next;
> +
> + /* Keep later folio splits covered. */
> + for (j = 0; j < nr; j++)
> + WARN_ON_ONCE(test_and_set_bit(bit + j, zdd->retire_map));
> +next:
> + i += nr;
> + }
> +}
> +
> +/**
> + * drm_pagemap_skip_retired_pages() - Drop retired PFNs from a raw-PFN eviction
> + * @src_pfns: source array after migrate_device_pfns() (MIGRATE_PFN encoded)
> + * @npages: number of entries in @src_pfns
> + *
> + * Skip source PFNs already migrated to RAM by either migration path. The
> + * raw-PFN eviction array starts at allocation offset zero, so the array index
> + * is also the retirement bitmap index.
> + */
> +static void drm_pagemap_skip_retired_pages(unsigned long *src_pfns,
> + unsigned long npages)
> +{
> + unsigned long i = 0;
> +
> + while (i < npages) {
> + struct page *page = migrate_pfn_to_page(src_pfns[i]);
> + struct drm_pagemap_zdd *zdd;
> + unsigned long nr = 1;
> +
> + if (!page) {
> + i++;
> + continue;
> + }
> +
> + nr = folio_nr_pages(page_folio(page));
> +
> + if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) ||
> + !drm_pagemap_is_devmem_page(page))
> + goto next;
> +
> + zdd = drm_pagemap_page_zone_device_data(page);
> + if (WARN_ON_ONCE(i >= zdd->range_npages)) {
> + src_pfns[i] &= ~MIGRATE_PFN_MIGRATE;
> + goto next;
> + }
> +
> + if (test_bit(i, zdd->retire_map))
> + src_pfns[i] &= ~MIGRATE_PFN_MIGRATE;
> +next:
> + i += nr;
> + }
> +}
> +
> /**
> * drm_pagemap_evict_to_ram() - Evict GPU SVM range to RAM
> * @devmem_allocation: Pointer to the device memory allocation
> *
> - * Similar to __drm_pagemap_migrate_to_ram but does not require mmap lock and
> - * migration done via migrate_device_* functions.
> + * Similar to __drm_pagemap_migrate_to_ram(), but uses the
> + * migrate_device_* helpers and does not require the mmap lock. Device
> + * PFNs already migrated to RAM by either migration path are skipped.
> *
> * Return: 0 on success, negative error code on failure.
> */
> @@ -1149,6 +1268,8 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
> if (err)
> goto err_free;
>
> + drm_pagemap_skip_retired_pages(src, npages);
> +
> err = drm_pagemap_migrate_populate_ram_pfn(NULL, NULL, npages, &mpages,
> src, dst, 0);
> if (err || !mpages)
> @@ -1179,6 +1300,8 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
> if (err)
> drm_pagemap_migration_unlock_put_pages(npages, dst);
> migrate_device_pages(src, dst, npages);
> + /* Raw-PFN eviction: array starts at allocation offset zero. */
> + drm_pagemap_retire_migrated_pages(src, npages, 0);
> migrate_device_finalize(src, dst, npages);
> drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, dst, npages,
> DMA_FROM_DEVICE, &state);
> @@ -1251,6 +1374,10 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
> if (end > vas->vm_end)
> end = vas->vm_end;
>
> + /* Keep the range within the ZDD allocation so retirement offsets stay valid. */
> + start = max(start, zdd->range_start);
> + end = min(end, zdd->range_start + (zdd->range_npages << PAGE_SHIFT));
Hmm, I guess this tricky if this partially unmapped or meremapped to a
different address, so we need to keep this purely physical.
New idea - store the migration state in folio itself in the private
data in lowest bits of folio->page.zone_device_data (set in
folio_set_zone_device_data) and mask if off in
drm_pagemap_page_zone_device_data.
e.g.,
#define DRM_PAGEMAP_ZDD_FLAG_MIGRATED BIT(0)
#define DRM_PAGEMAP_ZDD_FLAG_MASK 0x1ull /* we can expand this to 0x7 on 64 builds if we need more flags */
static inline struct drm_pagemap_zdd *drm_pagemap_page_zone_device_data(struct page *page)
{
struct folio *folio = page_folio(page);
/* XXX: Plus whatever casting needed */
return folio_zone_device_data(folio) & ~DRM_PAGEMAP_ZDD_FLAG_MASK;
}
static void drm_pagemap_page_set_flags(struct page *page, unsigned long flags)
{
struct folio *folio = page_folio(page);
struct drm_pagemap_zdd *zdd = drm_pagemap_page_zone_device_data(page);
WARN_ON_ONCE(flags & ~DRM_PAGEMAP_ZDD_FLAG_MASK);
folio_set_zone_device_data(folio, zdd | flags);
}
static unsigned long drm_pagemap_page_get_flags(struct page *page)
{
struct folio *folio = page_folio(page);
/* XXX: Plus whatever casting needed */
return folio_zone_device_data(folio) & DRM_PAGEMAP_ZDD_FLAG_MASK;
}
drm_pagemap_retire_migrated_pages()
for_each_page_migrated
drm_pagemap_page_set_flags(page, DRM_PAGEMAP_ZDD_FLAG_MIGRATED);
drm_pagemap_skip_retired_pages gets the flags, skips any folio with
DRM_PAGEMAP_ZDD_FLAG_MIGRATED set.
I think this will work and keep everything in the physical world.
Also btw, some of Sashiko pre-existing which have been flagged are fixed
in this series: https://patchwork.freedesktop.org/series/171651/
Matt
> +
> migrate.start = start;
> migrate.end = end;
> npages = npages_in_range(start, end);
> @@ -1309,6 +1436,8 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
> if (err)
> drm_pagemap_migration_unlock_put_pages(npages, migrate.dst);
> migrate_vma_pages(&migrate);
> + drm_pagemap_retire_migrated_pages(migrate.src, npages,
> + (start - zdd->range_start) >> PAGE_SHIFT);
> migrate_vma_finalize(&migrate);
> if (dev)
> drm_pagemap_migrate_unmap_pages(dev, pagemap_addr, migrate.dst,
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-08-06 7:21 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-06 6:11 [PATCH v2] drm/pagemap: Prevent double migration of device pages Arvind Yadav
2026-08-06 7:21 ` Matthew Brost
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox