[PATCH v2 0/4] drm/xe: Prioritize BO eviction based on VM exec queue priority

[PATCH v2 0/4] drm/xe: Prioritize BO eviction based on VM exec queue priority

[Matthew Brost]

This series ties a BO's TTM LRU priority (i.e. how likely it is to be
picked for eviction/shrinking under memory pressure) to how userspace
is actually using the VM and BOs it belongs to, instead of treating
every BO the same.

The reasoning behind the heuristic:

 - Kernel BOs, and any BO not private to a single VM (no vm_id, e.g.
   shared/exported buffers), are given the highest priority so they
   are the least likely to be evicted -- these are foundational or
   shared resources.
 - BOs private to a VM are then prioritized based on the highest
   priority band userspace has assigned to any exec queue attached to
   that VM: a VM with a HIGH priority exec queue keeps its BOs at the
   highest priority (matching kernel/shared BOs), a VM with only
   NORMAL priority queues (or none yet) gets the next priority band
   down, and a VM with only LOW priority queues gets the band below
   that.
 - The lowest priority band is reserved exclusively for BOs (private
   or shared) that have no VMA mappings left anywhere. This lets
   userspace use an unbind as a signal of "drop this from my working
   set, but keep the backing store around in case I need it again" --
   the BO is retained, just made a preferred candidate for
   eviction/shrinking until it is rebound.

Patch 1 adds the accounting: a per-priority-band count of exec queues
attached to each user VM. Patch 2 introduces the actual heuristic and
uses it both to set a BO's initial priority at creation time and to
bulk re-prioritize a VM's existing private BOs when an exec queue is
added or removed. Patch 3 fixes up staleness that these earlier
mechanisms miss: BOs created before any exec queue exists, or bound
late via VM_BIND, and BOs not tied to a single VM that get demoted to
the lowest priority and then rebound. Patch 4 introduces the
lowest-priority demotion itself, applied when a BO loses its last VMA
mapping.

Matt

Matthew Brost (4):
  drm/xe: Track exec queue priority band counts for user VMs
  drm/xe: Re-prioritize VM's private BOs when adding an exec queue
  drm/xe: Fix up BO TTM priority for late VM binds
  drm/xe: Lower BO TTM priority to lowest when fully unmapped from VM

 drivers/gpu/drm/xe/xe_bo.c       |  18 ++-
 drivers/gpu/drm/xe/xe_bo.h       |   3 +-
 drivers/gpu/drm/xe/xe_bo_types.h |   3 +
 drivers/gpu/drm/xe/xe_dma_buf.c  |   2 +-
 drivers/gpu/drm/xe/xe_vm.c       | 205 +++++++++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_vm.h       |   1 +
 drivers/gpu/drm/xe/xe_vm_types.h |   7 ++
 7 files changed, 221 insertions(+), 18 deletions(-)

-- 
2.34.1

[PATCH v2 1/4] drm/xe: Track exec queue priority band counts for user VMs

[Matthew Brost]

Add a per-priority-band count (LOW/NORMAL/HIGH) of exec queues
attached to a user VM in xe_vm_add_exec_queue()/xe_vm_remove_exec_queue(),
protected by vm->exec_queues.lock in write mode. KERNEL priority is
not counted since it is not possible for user exec queues.

This count is needed so a VM's private buffer objects can later be
prioritized for eviction based on the priority userspace has assigned
to the exec queues using that VM (i.e. work submitted at a higher
priority band implies its buffers are more important to keep
resident), rather than treating every VM the same.

Also move the has_ctx_tlb_inval check inside the lock in
xe_vm_add_exec_queue() so the list add/count update is properly
gated under the same critical section, and make
xe_vm_remove_exec_queue() bail out early if q->xef is not set,
rather than relying on callers to check this.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Assisted-by: GitHub_Copilot:claude-sonnet-5
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c       | 31 ++++++++++++++++++++-----------
 drivers/gpu/drm/xe/xe_vm_types.h |  7 +++++++
 2 files changed, 27 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index e915e33ebece..984e76248942 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -4933,8 +4933,9 @@ int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r
  * @vm: The VM.
  * @q: The exec_queue
  *
- * Add exec queue to VM, skipped if the device does not have context based TLB
- * invalidations.
+ * Track exec queue's priority band count for the VM. Also links the exec
+ * queue onto the per-GT list, skipped if the device does not have context
+ * based TLB invalidations.
  */
 void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 {
@@ -4948,12 +4949,14 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 	xe_assert(xe, vm->xef);
 	xe_assert(xe, vm == q->vm);
 
-	if (!xe->info.has_ctx_tlb_inval)
-		return;
-
 	down_write(&vm->exec_queues.lock);
-	list_add(&q->vm_exec_queue_link, &vm->exec_queues.list[q->gt->info.id]);
-	++vm->exec_queues.count[q->gt->info.id];
+	if (xe->info.has_ctx_tlb_inval) {
+		list_add(&q->vm_exec_queue_link, &vm->exec_queues.list[q->gt->info.id]);
+		++vm->exec_queues.count[q->gt->info.id];
+	}
+	if (q->sched_props.priority >= XE_EXEC_QUEUE_PRIORITY_LOW &&
+	    q->sched_props.priority <= XE_EXEC_QUEUE_PRIORITY_HIGH)
+		++vm->exec_queues.priority_count[q->sched_props.priority];
 	up_write(&vm->exec_queues.lock);
 }
 
@@ -4962,18 +4965,24 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
  * @vm: The VM.
  * @q: The exec_queue
  *
- * Remove exec queue from VM, skipped if the device does not have context based
- * TLB invalidations.
+ * Untrack exec queue's priority band count for the VM. Also unlinks the
+ * exec queue from the per-GT list, skipped if the device does not have
+ * context based TLB invalidations. No-op if @q is not a user exec queue,
+ * or is a VM exec queue (VM exec queues are never tracked by
+ * xe_vm_add_exec_queue()).
  */
 void xe_vm_remove_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 {
-	if (!vm->xe->info.has_ctx_tlb_inval)
+	if (!q->xef || (q->flags & EXEC_QUEUE_FLAG_VM))
 		return;
 
 	down_write(&vm->exec_queues.lock);
-	if (!list_empty(&q->vm_exec_queue_link)) {
+	if (vm->xe->info.has_ctx_tlb_inval && !list_empty(&q->vm_exec_queue_link)) {
 		list_del(&q->vm_exec_queue_link);
 		--vm->exec_queues.count[q->gt->info.id];
 	}
+	if (q->sched_props.priority >= XE_EXEC_QUEUE_PRIORITY_LOW &&
+	    q->sched_props.priority <= XE_EXEC_QUEUE_PRIORITY_HIGH)
+		--vm->exec_queues.priority_count[q->sched_props.priority];
 	up_write(&vm->exec_queues.lock);
 }
diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
index 635ed29b9a69..d75546c4eb66 100644
--- a/drivers/gpu/drm/xe/xe_vm_types.h
+++ b/drivers/gpu/drm/xe/xe_vm_types.h
@@ -16,6 +16,7 @@
 #include <linux/scatterlist.h>
 
 #include "xe_device_types.h"
+#include "xe_exec_queue_types.h"
 #include "xe_pt_types.h"
 #include "xe_range_fence.h"
 #include "xe_tlb_inval_types.h"
@@ -342,6 +343,12 @@ struct xe_vm {
 		 * per GT
 		 */
 		int count[XE_MAX_TILES_PER_DEVICE * XE_MAX_GT_PER_TILE];
+		/**
+		 * @exec_queues.priority_count: count of exec queues attached
+		 * to this VM, per priority band (LOW / NORMAL / HIGH only,
+		 * KERNEL priority is not possible for user exec queues)
+		 */
+		int priority_count[XE_EXEC_QUEUE_PRIORITY_HIGH + 1];
 		/** @exec_queues.lock: lock to protect exec_queues list */
 		struct rw_semaphore lock;
 	} exec_queues;
-- 
2.34.1

[PATCH v2 2/4] drm/xe: Re-prioritize VM's private BOs when adding an exec queue

[Matthew Brost]

When a higher (or lower) priority exec queue is added to a VM, the VM's
existing private BOs were left at whatever TTM LRU priority they had
at creation time, even though the newly added queue changes what
xe_vm_bo_priority() would now compute for that VM. This means a VM's
buffers would not reflect the actual importance userspace has now
assigned to work on that VM until new BOs happen to be allocated,
leaving stale, and potentially too-low, eviction priority on
long-lived buffers.

After dropping vm->exec_queues.lock in xe_vm_add_exec_queue(), take the
VM's dma-resv (xe_vm_lock()) and walk all of the VM's private BOs
(bo->vm == vm, skipping extobjs shared with other VMs), updating each
one's TTM priority to the newly computed level and moving it to the
tail of its new priority's LRU list.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Assisted-by: GitHub_Copilot:claude-sonnet-5
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c       |  18 +++--
 drivers/gpu/drm/xe/xe_bo.h       |   3 +-
 drivers/gpu/drm/xe/xe_bo_types.h |   3 +
 drivers/gpu/drm/xe/xe_dma_buf.c  |   2 +-
 drivers/gpu/drm/xe/xe_vm.c       | 126 ++++++++++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_vm.h       |   1 +
 6 files changed, 145 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 85e6d9a0f575..ca2b5e617321 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1346,7 +1346,8 @@ int xe_bo_notifier_prepare_pinned(struct xe_bo *bo)
 		if (bo->flags & XE_BO_FLAG_PINNED_NORESTORE)
 			break;
 
-		backup = xe_bo_init_locked(xe, NULL, NULL, bo->ttm.base.resv, NULL, xe_bo_size(bo),
+		backup = xe_bo_init_locked(xe, NULL, NULL, NULL, bo->ttm.base.resv, NULL,
+					   xe_bo_size(bo),
 					   DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
 					   XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
 					   XE_BO_FLAG_PINNED, &exec);
@@ -1486,7 +1487,7 @@ int xe_bo_evict_pinned(struct xe_bo *bo)
 			break;
 
 		if (!backup) {
-			backup = xe_bo_init_locked(xe, NULL, NULL, bo->ttm.base.resv, NULL,
+			backup = xe_bo_init_locked(xe, NULL, NULL, NULL, bo->ttm.base.resv, NULL,
 						   xe_bo_size(bo),
 						   DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
 						   XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
@@ -2276,6 +2277,9 @@ void xe_bo_free(struct xe_bo *bo)
  * if the function should allocate a new one.
  * @tile: The tile to select for migration of this bo, and the tile used for
  * GGTT binding if any. Only to be non-NULL for ttm_bo_type_kernel bos.
+ * @vm: The VM this bo will be associated with, used to determine the bo's
+ * TTM LRU priority based on the priority band of the VM's exec queues, or
+ * NULL if the bo is not associated with a user VM.
  * @resv: Pointer to a locked shared reservation object to use for this bo,
  * or NULL for the xe_bo to use its own.
  * @bulk: The bulk move to use for LRU bumping, or NULL for external bos.
@@ -2291,7 +2295,8 @@ void xe_bo_free(struct xe_bo *bo)
  * Return: The buffer object on success. Negative error pointer on failure.
  */
 struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
-				struct xe_tile *tile, struct dma_resv *resv,
+				struct xe_tile *tile, struct xe_vm *vm,
+				struct dma_resv *resv,
 				struct ttm_lru_bulk_move *bulk, size_t size,
 				u16 cpu_caching, enum ttm_bo_type type,
 				u32 flags, struct drm_exec *exec)
@@ -2353,7 +2358,10 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 	bo->flags = flags;
 	bo->cpu_caching = cpu_caching;
 	bo->ttm.base.funcs = &xe_gem_object_funcs;
-	bo->ttm.priority = XE_BO_PRIORITY_NORMAL;
+	if (type != ttm_bo_type_device || !vm)
+		bo->ttm.priority = XE_BO_PRIORITY_HIGHEST;
+	else
+		bo->ttm.priority = xe_vm_bo_priority(vm);
 	INIT_LIST_HEAD(&bo->pinned_link);
 #ifdef CONFIG_PROC_FS
 	INIT_LIST_HEAD(&bo->client_link);
@@ -2496,7 +2504,7 @@ __xe_bo_create_locked(struct xe_device *xe,
 		}
 	}
 
-	bo = xe_bo_init_locked(xe, bo, tile, vm ? xe_vm_resv(vm) : NULL,
+	bo = xe_bo_init_locked(xe, bo, tile, vm, vm ? xe_vm_resv(vm) : NULL,
 			       vm && !xe_vm_in_fault_mode(vm) &&
 			       flags & XE_BO_FLAG_USER ?
 			       &vm->lru_bulk_move : NULL, size,
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 6340317f7d2e..12625740a47a 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -115,7 +115,8 @@ struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
 
 struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
-				struct xe_tile *tile, struct dma_resv *resv,
+				struct xe_tile *tile, struct xe_vm *vm,
+				struct dma_resv *resv,
 				struct ttm_lru_bulk_move *bulk, size_t size,
 				u16 cpu_caching, enum ttm_bo_type type,
 				u32 flags, struct drm_exec *exec);
diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h
index fcc63ae3f455..95904b03c71f 100644
--- a/drivers/gpu/drm/xe/xe_bo_types.h
+++ b/drivers/gpu/drm/xe/xe_bo_types.h
@@ -24,7 +24,10 @@ struct xe_vm;
 #define XE_BO_MAX_PLACEMENTS	3
 
 /* TODO: To be selected with VM_MADVISE */
+#define	XE_BO_PRIORITY_LOW	0
 #define	XE_BO_PRIORITY_NORMAL	1
+#define	XE_BO_PRIORITY_HIGH	2
+#define	XE_BO_PRIORITY_HIGHEST	3
 
 /**
  * struct xe_bo - Xe buffer object
diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c b/drivers/gpu/drm/xe/xe_dma_buf.c
index 8a920e58245c..a80831fabd1d 100644
--- a/drivers/gpu/drm/xe/xe_dma_buf.c
+++ b/drivers/gpu/drm/xe/xe_dma_buf.c
@@ -300,7 +300,7 @@ xe_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
 		if (ret)
 			break;
 
-		bo = xe_bo_init_locked(xe, NULL, NULL, resv, NULL, dma_buf->size,
+		bo = xe_bo_init_locked(xe, NULL, NULL, NULL, resv, NULL, dma_buf->size,
 				       0, /* Will require 1way or 2way for vm_bind */
 				       ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, &exec);
 		drm_exec_retry_on_contention(&exec);
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 984e76248942..3c5da9f52f72 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -11,6 +11,7 @@
 #include <drm/drm_drv.h>
 #include <drm/drm_exec.h>
 #include <drm/drm_print.h>
+#include <drm/ttm/ttm_bo.h>
 #include <drm/ttm/ttm_tt.h>
 #include <uapi/drm/xe_drm.h>
 #include <linux/ascii85.h>
@@ -1231,6 +1232,37 @@ static void vma_destroy_cb(struct dma_fence *fence,
 	queue_work(system_dfl_wq, &vma->destroy_work);
 }
 
+/*
+ * Change @bo's TTM LRU priority to @priority. @bo's dma-resv must be held.
+ *
+ * Private, user BOs of non-fault-mode VMs have bo->ttm.bulk_move set to
+ * their VM's LRU bulk-move range (see xe_bo_init_locked()), which tracks a
+ * per-(mem_type, priority) first/last position within that range. Simply
+ * writing bo->ttm.priority and calling ttm_bo_move_to_lru_tail_unlocked()
+ * would make ttm_resource_move_to_lru_tail() look up the bulk-move bucket
+ * for the *new* priority while the resource is still physically linked in
+ * the *old* priority's LRU list, corrupting the bulk-move range (and
+ * potentially crashing on a NULL bucket). Temporarily detaching the BO from
+ * its bulk-move range moves it with a plain, non-bulk LRU update instead,
+ * and reattaching afterwards re-inserts it into the new priority's bucket.
+ */
+static void xe_bo_update_ttm_priority(struct xe_bo *bo, int priority)
+{
+	struct ttm_lru_bulk_move *bulk = bo->ttm.bulk_move;
+
+	xe_bo_assert_held(bo);
+
+	if (bo->ttm.priority == priority)
+		return;
+
+	if (bulk)
+		ttm_bo_set_bulk_move(&bo->ttm, NULL);
+	bo->ttm.priority = priority;
+	ttm_bo_move_to_lru_tail_unlocked(&bo->ttm);
+	if (bulk)
+		ttm_bo_set_bulk_move(&bo->ttm, bulk);
+}
+
 static void xe_vma_destroy(struct xe_vma *vma, struct dma_fence *fence)
 {
 	struct xe_vm *vm = xe_vma_vm(vma);
@@ -4928,6 +4960,87 @@ int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r
 	return xe_vm_alloc_vma(vm, &map_req, false);
 }
 
+/*
+ * Determine the TTM LRU priority to use for BOs private to @vm, based on the
+ * highest priority band among exec queues currently attached to @vm.
+ *
+ * A VM with a HIGH priority exec queue gets XE_BO_PRIORITY_HIGHEST, same as
+ * BOs that are never subject to this heuristic at all (kernel or shared
+ * BOs) -- userspace has indicated its most important work lives here, so
+ * treat these BOs as equally important. Every other band is then shifted up
+ * one level from its "intuitive" mapping: a NORMAL priority exec queue (or no
+ * exec queues at all yet) yields XE_BO_PRIORITY_HIGH, and a LOW priority exec
+ * queue yields XE_BO_PRIORITY_NORMAL. XE_BO_PRIORITY_LOW is intentionally
+ * never returned here -- it is reserved for xe_vma_destroy() to apply once a
+ * BO has no VMAs left, letting userspace signal "drop this from my working
+ * set" via unbind while still retaining the backing store, without this
+ * heuristic immediately promoting it back up on the next bind.
+ *
+ * Must be called with @vm's exec_queues.lock held, in either read or write
+ * mode.
+ */
+static int xe_vm_bo_priority_locked(struct xe_vm *vm)
+{
+	lockdep_assert_held(&vm->exec_queues.lock);
+
+	if (vm->exec_queues.priority_count[XE_EXEC_QUEUE_PRIORITY_HIGH])
+		return XE_BO_PRIORITY_HIGHEST;
+	if (vm->exec_queues.priority_count[XE_EXEC_QUEUE_PRIORITY_LOW])
+		return XE_BO_PRIORITY_NORMAL;
+
+	return XE_BO_PRIORITY_HIGH;
+}
+
+/**
+ * xe_vm_bo_priority() - Determine TTM LRU priority for a VM's private BOs
+ * @vm: The VM.
+ *
+ * Determine the TTM LRU priority to use for BOs private to @vm, based on the
+ * highest priority band among exec queues currently attached to @vm.
+ *
+ * Return: The TTM LRU priority to use for @vm's private BOs.
+ */
+int xe_vm_bo_priority(struct xe_vm *vm)
+{
+	int priority;
+
+	down_read(&vm->exec_queues.lock);
+	priority = xe_vm_bo_priority_locked(vm);
+	up_read(&vm->exec_queues.lock);
+
+	return priority;
+}
+
+/*
+ * Update the TTM LRU priority of all BOs private to @vm to @priority, moving
+ * each to the tail of its new priority level's LRU list. BOs shared with
+ * other VMs (extobjs) are left untouched.
+ */
+static void xe_vm_update_bo_priority(struct xe_vm *vm, int priority)
+{
+	struct drm_gpuva *gpuva;
+
+	/*
+	 * vm->lock is the outer most lock and serializes against concurrent
+	 * VMA insertion/removal (bind/unbind), so it must be held to safely
+	 * walk vm->gpuvm here. xe_vm_lock() (the VM's dma-resv) is taken
+	 * inside it, as it protects each private BO's bo->ttm.priority.
+	 */
+	down_read(&vm->lock);
+	xe_vm_lock(vm, false);
+	drm_gpuvm_for_each_va(gpuva, &vm->gpuvm) {
+		struct xe_vma *vma = gpuva_to_vma(gpuva);
+		struct xe_bo *bo = xe_vma_bo(vma);
+
+		if (!bo || bo->vm != vm || bo->ttm.priority == priority)
+			continue;
+
+		xe_bo_update_ttm_priority(bo, priority);
+	}
+	xe_vm_unlock(vm);
+	up_read(&vm->lock);
+}
+
 /**
  * xe_vm_add_exec_queue() - Add exec queue to VM
  * @vm: The VM.
@@ -4940,6 +5053,7 @@ int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r
 void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 {
 	struct xe_device *xe = vm->xe;
+	int priority;
 
 	/* User VMs and queues only */
 	xe_assert(xe, !(q->flags & EXEC_QUEUE_FLAG_KERNEL));
@@ -4957,7 +5071,10 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 	if (q->sched_props.priority >= XE_EXEC_QUEUE_PRIORITY_LOW &&
 	    q->sched_props.priority <= XE_EXEC_QUEUE_PRIORITY_HIGH)
 		++vm->exec_queues.priority_count[q->sched_props.priority];
+	priority = xe_vm_bo_priority_locked(vm);
 	up_write(&vm->exec_queues.lock);
+
+	xe_vm_update_bo_priority(vm, priority);
 }
 
 /**
@@ -4969,10 +5086,14 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
  * exec queue from the per-GT list, skipped if the device does not have
  * context based TLB invalidations. No-op if @q is not a user exec queue,
  * or is a VM exec queue (VM exec queues are never tracked by
- * xe_vm_add_exec_queue()).
+ * xe_vm_add_exec_queue()). Also re-prioritizes the VM's private BOs in
+ * case removing @q lowered the highest priority band with an attached
+ * exec queue.
  */
 void xe_vm_remove_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 {
+	int priority;
+
 	if (!q->xef || (q->flags & EXEC_QUEUE_FLAG_VM))
 		return;
 
@@ -4984,5 +5105,8 @@ void xe_vm_remove_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
 	if (q->sched_props.priority >= XE_EXEC_QUEUE_PRIORITY_LOW &&
 	    q->sched_props.priority <= XE_EXEC_QUEUE_PRIORITY_HIGH)
 		--vm->exec_queues.priority_count[q->sched_props.priority];
+	priority = xe_vm_bo_priority_locked(vm);
 	up_write(&vm->exec_queues.lock);
+
+	xe_vm_update_bo_priority(vm, priority);
 }
diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
index c5b900f38ded..64d3665bb05a 100644
--- a/drivers/gpu/drm/xe/xe_vm.h
+++ b/drivers/gpu/drm/xe/xe_vm.h
@@ -305,6 +305,7 @@ void xe_vm_kill(struct xe_vm *vm, bool unlocked);
 
 void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q);
 void xe_vm_remove_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q);
+int xe_vm_bo_priority(struct xe_vm *vm);
 
 /**
  * xe_vm_assert_held(vm) - Assert that the vm's reservation object is held.
-- 
2.34.1

[PATCH v2 3/4] drm/xe: Fix up BO TTM priority for late VM binds

[Matthew Brost]

A BO's TTM priority is only ever set once, at creation time, based on
the priority band of exec queues attached to its VM at that moment
(xe_vm_bo_priority()), or XE_BO_PRIORITY_HIGHEST for an extobj not
tied to any single VM. This misses the common flow of VM create -> BO
create -> exec queue create -> VM bind: the BO is created before any
exec queue exists on the VM (so it gets XE_BO_PRIORITY_HIGH, the "no
exec queues yet" default), the exec queue is added afterwards
(xe_vm_add_exec_queue() only re-prioritizes BOs already bound to the
VM at that time), and only then is the BO bound into the VM via
VM_BIND, leaving it stuck at a stale priority that no longer reflects
the VM's exec queues.

It also misses a second flow specific to extobjs: create BO with no
vm_id (bo->vm == NULL) -> bind -> unbind -> rebind. The unbind can
lower the BO's priority to XE_BO_PRIORITY_LOW once it has no VMA
mappings left anywhere (see xe_vma_destroy()), but on rebind nothing
brought it back up, leaving a still-useful, freshly rebound extobj
stuck at the lowest priority, an easy target for eviction/shrinking.

Add op_update_bo_priority(), called from vm_bind_ioctl_ops_fini() for
each DRM_GPUVA_OP_MAP operation. For a BO private to the VM being
bound (bo->vm == vm), recompute the current priority band via
xe_vm_bo_priority() and, if it differs, update bo->ttm.priority and
move the BO to the tail of its new priority's LRU list. For an extobj
(bo->vm == NULL, potentially shared with other VMs), instead restore
it directly to XE_BO_PRIORITY_HIGHEST, since it is now in active use
again and extobjs aren't tied to any single VM's exec queue priority
band. This runs under the VM's dma-resv lock, already held via
drm_exec in vm_bind_ioctl_ops_execute() since private BOs share the
VM's reservation object, and extobjs are locked as part of the same
bind operation.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Assisted-by: GitHub_Copilot:claude-sonnet-5
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 3c5da9f52f72..d42390e7ae2f 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -3554,6 +3554,38 @@ static void op_add_ufence(struct xe_vm *vm, struct xe_vma_op *op,
 	}
 }
 
+/*
+ * A BO's TTM priority is set when it is created, based on the priority band
+ * of @vm's exec queues at that time (for a private BO), or to the highest
+ * level (for an extobj, bo->vm == NULL). If a BO is (re)mapped into @vm
+ * afterwards, its priority may be stale: a private BO's VM may have gained
+ * or lost exec queues since, and an extobj may have been lowered to
+ * XE_BO_PRIORITY_LOW by xe_vma_destroy() after losing all of its VMA
+ * mappings. Bring it in line with the current priority band here.
+ */
+static void op_update_bo_priority(struct xe_vm *vm, struct xe_vma_op *op)
+{
+	struct xe_bo *bo;
+	int priority;
+
+	if (op->base.op != DRM_GPUVA_OP_MAP)
+		return;
+
+	bo = xe_vma_bo(op->map.vma);
+	if (!bo)
+		return;
+
+	/* Private BOs are only ever bound to the VM they were created for. */
+	xe_assert(vm->xe, !bo->vm || bo->vm == vm);
+
+	priority = bo->vm ? xe_vm_bo_priority(vm) : XE_BO_PRIORITY_HIGHEST;
+	if (bo->ttm.priority == priority)
+		return;
+
+	xe_vm_assert_held(vm);
+	xe_bo_update_ttm_priority(bo, priority);
+}
+
 static void vm_bind_ioctl_ops_fini(struct xe_vm *vm, struct xe_vma_ops *vops,
 				   struct dma_fence *fence)
 {
@@ -3566,6 +3598,8 @@ static void vm_bind_ioctl_ops_fini(struct xe_vm *vm, struct xe_vma_ops *vops,
 		if (ufence)
 			op_add_ufence(vm, op, ufence);
 
+		op_update_bo_priority(vm, op);
+
 		if (op->base.op == DRM_GPUVA_OP_UNMAP)
 			xe_vma_destroy(gpuva_to_vma(op->base.unmap.va), fence);
 		else if (op->base.op == DRM_GPUVA_OP_REMAP)
-- 
2.34.1

[PATCH v2 4/4] drm/xe: Lower BO TTM priority to lowest when fully unmapped from VM

[Matthew Brost]

An UNMAP (or the unmap half of a REMAP) can leave a BO with no
remaining VMA mappings anywhere. Such a BO is unlikely to be touched
again soon, so rather than leaving it at whatever priority band it had
while still in active use, drop it to XE_BO_PRIORITY_LOW, the lowest
level, so it becomes one of the first candidates considered for
eviction/shrinking. This applies both to BOs private to a VM and to
extobjs shared across multiple VMs (bo->vm == NULL) -- an extobj with
no bindings left anywhere is just as good an eviction/shrink candidate
as an unmapped private BO. Imported bos are excluded, since their
vma_count is never tracked (xe_bo_vma_count_inc/dec_locked() are
no-ops for them), so it would never reflect anything meaningful.

This has to be done inside xe_vma_destroy() itself, right after
xe_bo_vma_count_dec_locked(), rather than by its callers after the
fact: xe_vma_destroy() may drop the last reference to the BO via
xe_bo_put() in xe_vma_destroy_late() (called either synchronously, or
asynchronously via a fence callback), so a caller that stashed a
pointer to the BO before calling xe_vma_destroy() and then dereferences
it afterwards risks a use-after-free. Doing the check and update
before that possible final put avoids this.

bo->purgeable.vma_count, which xe_bo_vma_count_dec_locked() just
updated, tells us whether the BO has become fully unmapped. A REMAP
that only shrinks a mapping still leaves vma_count > 0 via the
prev/next VMA(s) created earlier in the bind, so those are correctly
left untouched. This runs under the BO's dma-resv, already asserted
held via xe_bo_assert_held() just above.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Assisted-by: GitHub_Copilot:claude-sonnet-5
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index d42390e7ae2f..8f3827f957ea 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -1282,6 +1282,22 @@ static void xe_vma_destroy(struct xe_vma *vma, struct dma_fence *fence)
 		xe_bo_vma_count_dec_locked(bo);
 		if (vma->attr.purgeable_state == XE_MADV_PURGEABLE_WILLNEED)
 			xe_bo_willneed_put_locked(bo);
+
+		/*
+		 * @bo has just lost a VMA mapping. If it now has none left
+		 * anywhere, it is unlikely to be used again soon, so lower
+		 * its TTM priority to the lowest level, making it one of the
+		 * first buffers considered for eviction/shrinking. This
+		 * applies both to BOs private to @vm and to extobjs shared
+		 * across multiple VMs (bo->vm == NULL), but not to imported
+		 * bos, whose vma_count is never tracked. Must be done here,
+		 * rather than by the caller, as this may be the last
+		 * reference to @bo (see xe_vma_destroy_late()).
+		 */
+		if (!drm_gem_is_imported(&bo->ttm.base) &&
+		    !bo->purgeable.vma_count &&
+		    bo->ttm.priority != XE_BO_PRIORITY_LOW)
+			xe_bo_update_ttm_priority(bo, XE_BO_PRIORITY_LOW);
 	}
 
 	xe_vm_assert_held(vm);
-- 
2.34.1

Re: [PATCH v2 3/4] drm/xe: Fix up BO TTM priority for late VM binds

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Data race and TTM LRU corruption due to missing invariant check for cross-VM binds of private BOs.

Pre-existing issues:
- [Medium] Meaningful code and commit message mismatch (Incomplete Implementation of extobj priority lowering).
--

--- Patch [3]: [PATCH v2 3/4] drm/xe: Fix up BO TTM priority for late VM binds ---
commit b5439d40abd0e7b0ac5ecc4fc4045178d3dcfcb7
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Fix up BO TTM priority for late VM binds

This patch updates the BO TTM priority when a BO is (re)mapped into a VM.
It recalculates the priority band based on the VM's current exec queues or
sets it to the highest priority for external objects.

>     A BO's TTM priority is only ever set once, at creation time, based on
>     the priority band of exec queues attached to its VM at that moment
>     (xe_vm_bo_priority()), or XE_BO_PRIORITY_HIGHEST for an extobj not
>     tied to any single VM. This misses the common flow of VM create -> BO
>     create -> exec queue create -> VM bind: the BO is created before any
>     exec queue exists on the VM (so it gets XE_BO_PRIORITY_HIGH, the "no
>     exec queues yet" default), the exec queue is added afterwards
>     (xe_vm_add_exec_queue() only re-prioritizes BOs already bound to the
>     VM at that time), and only then is the BO bound into the VM via
>     VM_BIND, leaving it stuck at a stale priority that no longer reflects
>     the VM's exec queues.
>     
>     It also misses a second flow specific to extobjs: create BO with no
>     vm_id (bo->vm == NULL) -> bind -> unbind -> rebind. The unbind can
>     lower the BO's priority to XE_BO_PRIORITY_LOW once it has no VMA
>     mappings left anywhere (see xe_vma_destroy()), but on rebind nothing

[Severity: Medium]
This is a pre-existing issue, but I don't see any logic in xe_vma_destroy()
or elsewhere in the codebase that actually lowers an extobj's priority to
XE_BO_PRIORITY_LOW.

Because extobjs appear to permanently retain XE_BO_PRIORITY_HIGHEST, won't
the fix in op_update_bo_priority() always hit the early return for extobjs,
missing the intended working set eviction optimization entirely?

>     brought it back up, leaving a still-useful, freshly rebound extobj
>     stuck at the lowest priority, an easy target for eviction/shrinking.
>     
>     Add op_update_bo_priority(), called from vm_bind_ioctl_ops_fini() for

[ ... ]

> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> index 0f2b8ffefb8ec..dd64a46b554ed 100644
> --- a/drivers/gpu/drm/xe/xe_vm.c
> +++ b/drivers/gpu/drm/xe/xe_vm.c

[ ... ]

> @@ -3538,6 +3538,38 @@ static void op_add_ufence(struct xe_vm *vm, struct xe_vma_op *op,
>  	}
>  }
>  
> +/*
> + * A BO's TTM priority is set when it is created, based on the priority band
> + * of @vm's exec queues at that time (for a private BO), or to the highest
> + * level (for an extobj, bo->vm == NULL). If a BO is (re)mapped into @vm
> + * afterwards, its priority may be stale: a private BO's VM may have gained
> + * or lost exec queues since, and an extobj may have been lowered to
> + * XE_BO_PRIORITY_LOW by xe_vma_destroy() after losing all of its VMA
> + * mappings. Bring it in line with the current priority band here.
> + */
> +static void op_update_bo_priority(struct xe_vm *vm, struct xe_vma_op *op)
> +{
> +	struct xe_bo *bo;
> +	int priority;
> +
> +	if (op->base.op != DRM_GPUVA_OP_MAP)
> +		return;
> +
> +	bo = xe_vma_bo(op->map.vma);
> +	if (!bo)
> +		return;
> +
> +	/* Private BOs are only ever bound to the VM they were created for. */
> +	xe_assert(vm->xe, !bo->vm || bo->vm == vm);

[Severity: High]
Could this regression allow a data race and TTM LRU list corruption if
userspace binds a private BO associated with another VM?

If userspace binds a private BO associated with VM_A to VM_B, the xe_assert()
doesn't prevent execution from continuing (and may compile out in non-debug
builds).

During the bind operation, vma_lock_and_validate() skips acquiring the BO's
resv lock because it assumes any BO with a non-NULL bo->vm shares the target
VM's lock:

drivers/gpu/drm/xe/xe_vm.c:vma_lock_and_validate() {
    ...
	if (bo) {
		if (!bo->vm)
			err = drm_exec_lock_obj(exec, &bo->ttm.base);
    ...
}

This means op_update_bo_priority() will call xe_bo_update_ttm_priority() to
mutate bo->ttm.priority and TTM LRU structures without holding the required
bo->ttm.base.resv lock. Can an unprivileged userspace application trigger this
by passing a mismatched BO handle to the bind IOCTL?

> +
> +	priority = bo->vm ? xe_vm_bo_priority(vm) : XE_BO_PRIORITY_HIGHEST;
> +	if (bo->ttm.priority == priority)
> +		return;
> +
> +	xe_vm_assert_held(vm);
> +	xe_bo_update_ttm_priority(bo, priority);
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260715042832.3625096-1-matthew.brost@intel.com?part=3

Re: [PATCH v2 2/4] drm/xe: Re-prioritize VM's private BOs when adding an exec queue

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] Concurrent exec queue additions and removals create a Time-of-Check to Time-of-Use (TOCTOU) race, leaving BOs with stale TTM priorities.
- [Medium] The BO priority evaluation incorrectly prioritizes LOW priority exec queues over NORMAL priority ones, causing unintended priority downgrades.
- [Medium] Unbound private BOs miss dynamic priority updates and permanently retain stale priorities.
- [Medium] The patch omits the intended logic to downgrade completely unmapped BOs to `XE_BO_PRIORITY_LOW` in `xe_vma_destroy()`.
--

--- Patch [2]: [PATCH v2 2/4] drm/xe: Re-prioritize VM's private BOs when adding an exec queue ---
commit 45971462729146ee32e2c293b500a521f745e5fe
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Re-prioritize VM's private BOs when adding an exec queue

This commit updates the TTM LRU eviction priority of a VM's private BOs when
exec queues are added or removed. It iterates over the VM's active VMAs to
apply the new priority based on the highest priority band among the VM's
attached exec queues.

[ ... ]
> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> index 0b315d6120aae..0f2b8ffefb8ec 100644
> --- a/drivers/gpu/drm/xe/xe_vm.c
> +++ b/drivers/gpu/drm/xe/xe_vm.c
[ ... ]
> @@ -4912,6 +4944,87 @@ int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r
>  	return xe_vm_alloc_vma(vm, &map_req, false);
>  }
>  
> +/*
> + * Determine the TTM LRU priority to use for BOs private to @vm, based on the
> + * highest priority band among exec queues currently attached to @vm.
> + *
> + * A VM with a HIGH priority exec queue gets XE_BO_PRIORITY_HIGHEST, same as
> + * BOs that are never subject to this heuristic at all (kernel or shared
> + * BOs) -- userspace has indicated its most important work lives here, so
> + * treat these BOs as equally important. Every other band is then shifted up
> + * one level from its "intuitive" mapping: a NORMAL priority exec queue (or no
> + * exec queues at all yet) yields XE_BO_PRIORITY_HIGH, and a LOW priority exec
> + * queue yields XE_BO_PRIORITY_NORMAL. XE_BO_PRIORITY_LOW is intentionally
> + * never returned here -- it is reserved for xe_vma_destroy() to apply once a
> + * BO has no VMAs left, letting userspace signal "drop this from my working
> + * set" via unbind while still retaining the backing store, without this
> + * heuristic immediately promoting it back up on the next bind.

[Severity: Medium]
Is the logic to downgrade completely unmapped BOs to XE_BO_PRIORITY_LOW
missing from this patch? The comment states it is reserved for
xe_vma_destroy() to apply once a BO has no VMAs left, but the implementation
to actually apply it does not seem to be included in this changeset.

> + *
> + * Must be called with @vm's exec_queues.lock held, in either read or write
> + * mode.
> + */
> +static int xe_vm_bo_priority_locked(struct xe_vm *vm)
> +{
> +	lockdep_assert_held(&vm->exec_queues.lock);
> +
> +	if (vm->exec_queues.priority_count[XE_EXEC_QUEUE_PRIORITY_HIGH])
> +		return XE_BO_PRIORITY_HIGHEST;
> +	if (vm->exec_queues.priority_count[XE_EXEC_QUEUE_PRIORITY_LOW])
> +		return XE_BO_PRIORITY_NORMAL;
> +
> +	return XE_BO_PRIORITY_HIGH;
> +}

[Severity: Medium]
Does this logic incorrectly prioritize LOW priority exec queues over NORMAL
priority ones? If a VM has both a NORMAL and a LOW priority queue, it appears
this will return XE_BO_PRIORITY_NORMAL instead of checking for the presence of
a NORMAL queue first, which might unfairly downgrade the VM's priority band.

> +
> +/**
> + * xe_vm_bo_priority() - Determine TTM LRU priority for a VM's private BOs
[ ... ]
> +/*
> + * Update the TTM LRU priority of all BOs private to @vm to @priority, moving
> + * each to the tail of its new priority level's LRU list. BOs shared with
> + * other VMs (extobjs) are left untouched.
> + */
> +static void xe_vm_update_bo_priority(struct xe_vm *vm, int priority)
> +{
> +	struct drm_gpuva *gpuva;
> +
> +	/*
> +	 * vm->lock is the outer most lock and serializes against concurrent
> +	 * VMA insertion/removal (bind/unbind), so it must be held to safely
> +	 * walk vm->gpuvm here. xe_vm_lock() (the VM's dma-resv) is taken
> +	 * inside it, as it protects each private BO's bo->ttm.priority.
> +	 */
> +	down_read(&vm->lock);
> +	xe_vm_lock(vm, false);
> +	drm_gpuvm_for_each_va(gpuva, &vm->gpuvm) {

[Severity: Medium]
Can unbound private BOs miss dynamic priority updates here? Since this
loop iterates over mapped VMAs using drm_gpuvm_for_each_va(), any allocated
but not yet bound BOs would be skipped and might permanently retain their
stale creation priorities.

> +		struct xe_vma *vma = gpuva_to_vma(gpuva);
> +		struct xe_bo *bo = xe_vma_bo(vma);
> +
> +		if (!bo || bo->vm != vm || bo->ttm.priority == priority)
> +			continue;
> +
> +		xe_bo_update_ttm_priority(bo, priority);
> +	}
> +	xe_vm_unlock(vm);
> +	up_read(&vm->lock);
> +}
[ ... ]
> @@ -4941,7 +5055,10 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)
>  	if (q->sched_props.priority >= XE_EXEC_QUEUE_PRIORITY_LOW &&
>  	    q->sched_props.priority <= XE_EXEC_QUEUE_PRIORITY_HIGH)
>  		++vm->exec_queues.priority_count[q->sched_props.priority];
> +	priority = xe_vm_bo_priority_locked(vm);
>  	up_write(&vm->exec_queues.lock);
> +
> +	xe_vm_update_bo_priority(vm, priority);
>  }

[Severity: Medium]
Is there a race condition between taking the priority snapshot and applying it?

If multiple threads are concurrently adding and removing exec queues, their
execution could interleave between dropping exec_queues.lock and acquiring
vm->lock inside xe_vm_update_bo_priority().

Could the last thread to acquire vm->lock apply a stale priority snapshot
that no longer reflects the current queue state?

>  
>  /**

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260715042832.3625096-1-matthew.brost@intel.com?part=2