All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] drm/xe: Tie BO TTM LRU priority to DONTNEED/WILLNEED madvise state
@ 2026-10-20 16:51 Matthew Brost
  2026-10-20 17:03 ` ✗ CI.KUnit: failure for " Patchwork
                   ` (5 more replies)
  0 siblings, 6 replies; 9+ messages in thread
From: Matthew Brost @ 2026-10-20 16:51 UTC (permalink / raw)
  To: intel-xe

The DRM_XE_MADVISE ioctl lets userspace mark a VMA's BO as DONTNEED,
signalling that its contents are not needed right now and may be
discarded under memory pressure (see madvise_purgeable() in
xe_vm_madvise.c), or restore it back to WILLNEED. This is a stronger,
explicit signal than a plain unbind: userspace has told us the BO is not
just unused for now, but that it is actively fine to lose the data.
There is no reason to keep such a BO competing for LRU space with
actively used buffers, and conversely, once userspace marks it WILLNEED
again it is back in active use and should be reprioritized accordingly.

Give xe two TTM LRU priority bands and tie them to that state:
XE_BO_PRIORITY_LOW for DONTNEED BOs and XE_BO_PRIORITY_NORMAL for
everything else. Since the TTM LRU is walked in ascending priority
order, DONTNEED BOs become the first candidates for eviction and
shrinking. The transition is driven from xe_bo_set_purgeable_state(),
the single point through which a BO's aggregate purgeable state actually
moves between WILLNEED and DONTNEED (PURGED is terminal and left at
XE_BO_PRIORITY_LOW).

Changing bo->ttm.priority cannot be done by a plain store: private, user
BOs have bo->ttm.bulk_move set to their VM's LRU bulk-move range, which
tracks a contiguous [first, last] run of resources per (mem_type,
priority) bucket. Writing the priority while the BO is still attached
would leave the resource linked in the old priority's bucket while the
bulk-move code looks it up under the new one, corrupting both. Add
xe_bo_update_ttm_priority(), which temporarily detaches the BO from its
bulk-move range, updates the priority with a plain non-bulk LRU update,
and reattaches it into the new priority's bucket.
xe_bo_set_purgeable_state() is always called with the BO's dma-resv
held, so this is safe.

With DONTNEED BOs now clustered at XE_BO_PRIORITY_LOW, the shrinker's
purge pass no longer needs to walk each memory type's whole LRU: nothing
beyond XE_BO_PRIORITY_LOW can possibly be purgeable, so continuing past
it just wastes time trylocking BOs that can never be purged in this
pass. Break out of the per-mem_type LRU walk in __xe_shrinker_walk()
once a BO's priority rises above XE_BO_PRIORITY_LOW during the purge
pass. Behavior of the non-purge (regular shrink) pass is unchanged.

Assisted-by: LLM
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c       | 46 ++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo_types.h |  8 +++++-
 drivers/gpu/drm/xe/xe_shrinker.c |  3 +++
 3 files changed, 56 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 6d2ac986e98a..b723addf071d 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1086,6 +1086,38 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
 	}
 }
 
+/*
+ * Change @bo's TTM LRU priority to @priority. @bo's dma-resv must be held.
+ *
+ * Private, user BOs of non-fault-mode VMs have bo->ttm.bulk_move set to their
+ * VM's LRU bulk-move range (see xe_bo_create_user()), which tracks a
+ * contiguous [first, last] run of resources per (mem_type, priority) bucket.
+ * Writing bo->ttm.priority while the BO is still attached to that range would
+ * leave the resource linked in the old priority's bucket while the bulk-move
+ * code looks it up under the new one, corrupting both buckets. Temporarily
+ * detaching the BO from its bulk-move range moves it with a plain, non-bulk
+ * LRU update instead, and reattaching afterwards re-inserts it into the new
+ * priority's bucket.
+ */
+static void xe_bo_update_ttm_priority(struct xe_bo *bo, int priority)
+{
+	struct ttm_lru_bulk_move *bulk = bo->ttm.bulk_move;
+
+	xe_bo_assert_held(bo);
+
+	if (bo->ttm.priority == priority)
+		return;
+
+	if (bulk)
+		ttm_bo_set_bulk_move(&bo->ttm, NULL);
+
+	bo->ttm.priority = priority;
+	ttm_bo_move_to_lru_tail_unlocked(&bo->ttm);
+
+	if (bulk)
+		ttm_bo_set_bulk_move(&bo->ttm, bulk);
+}
+
 /**
  * xe_bo_set_purgeable_state() - Set BO purgeable state with validation
  * @bo: Buffer object
@@ -1095,6 +1127,13 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
  * transitions. Once a BO is PURGED, it cannot transition to any other state.
  * Invalid transitions are caught with xe_assert(). Shrinker page accounting
  * is updated automatically.
+ *
+ * A WILLNEED -> DONTNEED transition also lowers the BO's TTM LRU priority to
+ * XE_BO_PRIORITY_LOW: userspace has signaled it does not need these contents
+ * again soon, so make it one of the first buffers considered for
+ * eviction/shrinking. The shrinker's purge pass relies on this clustering to
+ * bound its LRU walk. Conversely, a DONTNEED -> WILLNEED transition restores
+ * XE_BO_PRIORITY_NORMAL, since the BO is back in active use.
  */
 void xe_bo_set_purgeable_state(struct xe_bo *bo,
 			       enum xe_madv_purgeable_state new_state)
@@ -1112,6 +1151,13 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
 	xe_assert(xe, !(bo->purgeable.state == XE_MADV_PURGEABLE_PURGED &&
 			new_state != XE_MADV_PURGEABLE_PURGED));
 
+	if (new_state == XE_MADV_PURGEABLE_DONTNEED &&
+	    bo->purgeable.state == XE_MADV_PURGEABLE_WILLNEED)
+		xe_bo_update_ttm_priority(bo, XE_BO_PRIORITY_LOW);
+	else if (new_state == XE_MADV_PURGEABLE_WILLNEED &&
+		 bo->purgeable.state == XE_MADV_PURGEABLE_DONTNEED)
+		xe_bo_update_ttm_priority(bo, XE_BO_PRIORITY_NORMAL);
+
 	bo->purgeable.state = new_state;
 	xe_bo_set_purgeable_shrinker(bo, new_state);
 }
diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h
index 8ec4a01a0092..0265fb314c77 100644
--- a/drivers/gpu/drm/xe/xe_bo_types.h
+++ b/drivers/gpu/drm/xe/xe_bo_types.h
@@ -24,7 +24,13 @@ struct xe_exec_queue;
 
 #define XE_BO_MAX_PLACEMENTS	3
 
-/* TODO: To be selected with VM_MADVISE */
+/*
+ * TTM LRU priority bands. The LRU is walked in ascending priority order, so
+ * XE_BO_PRIORITY_LOW objects are the first candidates for eviction and
+ * shrinking. BOs advised DONTNEED through DRM_XE_MADVISE live at
+ * XE_BO_PRIORITY_LOW, everything else at XE_BO_PRIORITY_NORMAL.
+ */
+#define	XE_BO_PRIORITY_LOW	0
 #define	XE_BO_PRIORITY_NORMAL	1
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_shrinker.c b/drivers/gpu/drm/xe/xe_shrinker.c
index 772dab9b3ed7..c86455ea08d1 100644
--- a/drivers/gpu/drm/xe/xe_shrinker.c
+++ b/drivers/gpu/drm/xe/xe_shrinker.c
@@ -133,6 +133,9 @@ static int __xe_shrinker_walk(struct xe_shrinker *shrinker,
 		}
 
 		ttm_bo_lru_for_each_reserved_guarded(&curs, man, &arg, ttm_bo) {
+			if (flags.purge && ttm_bo->priority > XE_BO_PRIORITY_LOW)
+				break;
+
 			if (!ttm_bo_shrink_suitable(ttm_bo, ctx))
 				continue;
 
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-10-20 22:51 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-20 16:51 [PATCH] drm/xe: Tie BO TTM LRU priority to DONTNEED/WILLNEED madvise state Matthew Brost
2026-10-20 17:03 ` ✗ CI.KUnit: failure for " Patchwork
2026-10-20 17:15 ` [PATCH] " Matthew Auld
2026-10-20 17:48 ` ✓ CI.KUnit: success for drm/xe: Tie BO TTM LRU priority to DONTNEED/WILLNEED madvise state (rev2) Patchwork
2026-10-20 18:27 ` ✓ Xe.CI.BAT: " Patchwork
2026-10-20 19:00 ` [PATCH] drm/xe: Tie BO TTM LRU priority to DONTNEED/WILLNEED madvise state Thomas Hellström
2026-10-20 19:13   ` Matthew Brost
2026-10-20 20:57     ` Thomas Hellström
2026-10-20 22:51 ` ✗ Xe.CI.FULL: failure for drm/xe: Tie BO TTM LRU priority to DONTNEED/WILLNEED madvise state (rev2) Patchwork

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.