[PATCH v2 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

[PATCH v2 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

[Matthew Brost]

The overarching goal of this series is to keep user-facing IOCTLs snappy
by holding the BO dma-resv lock for the shortest possible time, and to
push the expensive, best-effort work - page (re)allocation, DMA mapping
and placement fixups - into the background or out of the locked critical
section entirely.

Two related mechanisms fall out of that goal:

 - Out-of-lock backing. Page allocation and DMA mapping are the dominant
   cost of gem_create and of BO moves, and doing them under the dma-resv
   lock serializes otherwise-independent clients. This series preallocates
   the backing pages (and, where applicable, builds the DMA/IOVA mapping)
   outside the lock, then transfers the result into the BO under a brief
   lock hold. The user gets a fast IOCTL return; the heavy lifting happens
   without contending the reservation.

 - Page defragmentation. A BO allocated under memory pressure keeps its
   scattered, sub-optimally-ordered backing for its entire lifetime,
   costing TLB efficiency forever. TTM grows the plumbing to track
   order-failure and to re-back a populated BO in place at the beneficial
   order, and Xe wires up a background delayed worker that promotes such
   BOs on the GPU once memory is available again - again, off the hot
   path and without stalling the submitting thread.

Since v1 [1] the series has grown considerably. The bulk of the new
material is a direct result of profiling: once the defragmenter was in
place, the remaining dma-resv hold times and the per-BO
allocation/mapping costs showed up clearly, which motivated the
out-of-lock preallocation, the IOVA-based mapping path, and the amdgpu
counterpart. Rather than land the defragmenter alone, v2 folds in these
optimizations since they share the same infrastructure and the same
"hold dma-resv briefly, fix up in the background" architecture.

The series is organized in sections rather than described patch by patch:

 - Patches 1-10 (drm/ttm): core TTM preparation - order-failure tracking,
   the defragmentation move, reclaim backoff, and out-of-lock page
   preallocation plumbing.
 - Patches 11-14: other dependent drm/gpusvm and drm/xe patches this
   series builds on (DMA-mapping accounting, per-order DMA stats, async
   L2 flush, and a VM-teardown ordering fix).
 - Patches 15-23 (drm/xe): the page defragmenter itself - BO tracking,
   the on-GPU defrag copy, xe_bo_move() handling, and the background
   worker with its stats and configuration.
 - Patches 24-25 (drm/xe): out-of-lock system BO backing preallocation
   in gem_create, moving page allocation out of the dma-resv lock.
 - Patches 26-32 (drm/xe): IOVA-based DMA mapping optimizations, building
   and finalizing the mapping outside the lock.
 - Patch 33 (drm/amdgpu): the equivalent out-of-lock system BO
   preallocation for amdgpu, exercising the shared TTM plumbing.

Testing
=======

 - 3D benchmarks on Ubuntu and on Android, with memory intentionally
   fragmented by a separate program at launch (plus beneficial-order
   error injection). BOs are initially backed at a sub-optimal order and
   scores start lower; the background defrag worker then promotes the
   backing to the beneficial order and scores climb back in line with the
   unfragmented baseline.
 - IGT:
   https://patchwork.freedesktop.org/patch/739052/?series=170046&rev=2

Matt

[1] https://patchwork.freedesktop.org/series/169053/

Cc: amd-gfx@lists.freedesktop.org
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>



Matthew Brost (31):
  drm/ttm/pool: Allow backing off reclaim at the beneficial order
  drm/ttm/pool: Add ttm_pool_page_order_nodma() helper
  drm/ttm: Record sub-optimal page order allocations in ttm_tt
  drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()
  drm/ttm: Support defragmentation moves
  drm/ttm: Add fault injection for beneficial-order allocation failures
  drm/ttm: Harvest beneficial-order pages on defragmentation moves
  drm/ttm: Bound page (re)allocation per defragmentation move
  drm/ttm: Preallocate beneficial-order defrag pages outside the lock
  drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()
  drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()
  drm/xe: Destroy page tables after unlinking all VMAs on VM close
  drm/xe: Track BOs backed at a sub-optimal page order
  drm/xe: Back off beneficial-order reclaim under defrag pressure
  drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies
  drm/xe: Handle defrag moves in xe_bo_move()
  drm/xe: Skip self-copies for borrowed pages on defrag moves
  drm/xe: Add a page defragmentation worker
  drm/xe: Add defrag GT stats
  drm/xe: Add Kconfig.profile options for BO defrag configuration
  drm/xe: Defrag using out-of-lock page preallocation
  drm/xe: Add defrag profiling tracepoints
  drm/xe: Preallocate system BO backing outside the dma-resv lock
  drm/xe: Add tracepoint for xe_gem_create_ioctl
  drm/xe: Add IOVA-based xe_res_cursor variant
  drm/xe: Use IOVA-based DMA mapping for eligible tt BOs
  drm/xe: Add per-device dependency scheduler for IOVA defrag finalize
  drm/xe: Add packed copy-step IOVA mapping for defrag
  drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies
  drm/xe: Finalize defrag-IOVA moves with post-copy job
  drm/amdgpu: Preallocate system BO pages outside the reservation lock

Thomas Hellström (2):
  drm/gpusvm: Add a DMA-mapping accounting callback
  drm/xe: Add debugfs stats for DMA-mapped pages per order

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   45 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |    5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |    3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   39 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    3 +
 drivers/gpu/drm/drm_gpusvm.c                  |   17 +-
 drivers/gpu/drm/ttm/ttm_bo.c                  |  101 +-
 drivers/gpu/drm/ttm/ttm_bo_util.c             |   21 +-
 drivers/gpu/drm/ttm/ttm_pool.c                |  931 +++++++++-
 drivers/gpu/drm/ttm/ttm_tt.c                  |   67 +
 drivers/gpu/drm/xe/Kconfig.profile            |   40 +
 drivers/gpu/drm/xe/tests/xe_bo.c              |    8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c         |    2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c         |   12 +-
 drivers/gpu/drm/xe/xe_bo.c                    | 1632 ++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h                    |   15 +-
 drivers/gpu/drm/xe/xe_bo_types.h              |    6 +
 drivers/gpu/drm/xe/xe_debugfs.c               |   26 +
 drivers/gpu/drm/xe/xe_device.c                |   35 +
 drivers/gpu/drm/xe/xe_device_types.h          |   57 +
 drivers/gpu/drm/xe/xe_dma_buf.c               |    2 +-
 drivers/gpu/drm/xe/xe_ggtt.c                  |    2 +-
 drivers/gpu/drm/xe/xe_gt_stats.c              |    5 +
 drivers/gpu/drm/xe/xe_gt_stats_types.h        |   17 +
 drivers/gpu/drm/xe/xe_migrate.c               |  531 +++++-
 drivers/gpu/drm/xe/xe_migrate.h               |   17 +
 drivers/gpu/drm/xe/xe_pt.c                    |    2 +-
 drivers/gpu/drm/xe/xe_res_cursor.h            |   56 +-
 drivers/gpu/drm/xe/xe_svm.c                   |   36 +-
 drivers/gpu/drm/xe/xe_svm.h                   |    3 +-
 drivers/gpu/drm/xe/xe_trace_bo.h              |   90 +
 drivers/gpu/drm/xe/xe_userptr.c               |   55 +
 drivers/gpu/drm/xe/xe_userptr.h               |    1 +
 drivers/gpu/drm/xe/xe_vm.c                    |  149 +-
 drivers/gpu/drm/xe/xe_vm.h                    |    5 +
 include/drm/drm_gpusvm.h                      |   19 +
 include/drm/ttm/ttm_bo.h                      |   64 +
 include/drm/ttm/ttm_pool.h                    |   46 +
 include/drm/ttm/ttm_tt.h                      |   43 +-
 42 files changed, 4029 insertions(+), 189 deletions(-)

-- 
2.34.1

[PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

[Matthew Brost]

Allocating at the pool's beneficial order normally uses
__GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When
beneficial-order allocations have recently been failing, this retry just
burns CPU cycles in reclaim/compaction before falling back to a smaller
order anyway.

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and
plumb it through to ttm_pool_alloc_page(). When set, the beneficial
order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction
entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion
is skipped. Together this makes the allocator back off quickly to a
smaller order instead of stalling.

This is a no-op until a caller opts in.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
 include/drm/ttm/ttm_bo.h       |  7 +++++++
 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 3d5f2ae0a456..dbe977412a81 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && (beneficial_reclaim_backoff ||
+		      (beneficial_order && order != beneficial_order)))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 8310bc3d55f9..fd814b5cf535 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
-- 
2.34.1

[PATCH v2 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

[Matthew Brost]

TTM records the allocation order of a multi-order page in page->private at
allocation time. For pools that do not use dma_alloc the order is stored
directly, while dma_alloc pools store a struct ttm_pool_dma pointer there
instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page
order to walk a populated ttm_tt one chunk at a time, but cannot rely on
folio_order(): TTM allocates high-order pages with alloc_pages_node()
without __GFP_COMP, so they are not compound and folio_order() always
returns 0.

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order
recorded in page->private. It is only valid for pages from a pool that does
not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm
device init time, so callers know from their device configuration that this
helper applies. Use it from the existing internal ttm_pool_page_order() to
keep a single source of truth.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
 include/drm/ttm/ttm_pool.h     |  2 ++
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index dbe977412a81..d34592d4dbc7 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
 	return num_pages;
 }
 
+/**
+ * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
+ * @p: The page to query.
+ *
+ * Return the allocation order that the TTM pool allocator recorded in
+ * @p->private at allocation time. This only works for pages that were
+ * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
+ * alloc_pages_node()), where TTM stores the order directly in page->private.
+ *
+ * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
+ * pointer and this helper must not be used. Whether a TTM device uses
+ * dma_alloc is fixed at ttm device init time, so callers are expected to know
+ * from their TTM device configuration that their pages are not dma_alloc
+ * backed before using this helper.
+ *
+ * Return: The allocation order of the page.
+ */
+unsigned int ttm_pool_page_order_nodma(struct page *p)
+{
+	return p->private;
+}
+EXPORT_SYMBOL(ttm_pool_page_order_nodma);
+
 /* Return the allocation order based for a page */
 static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 {
@@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 		return dma->vaddr & ~PAGE_MASK;
 	}
 
-	return p->private;
+	return ttm_pool_page_order_nodma(p);
 }
 
 /*
diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
index 26ee592e1994..753203980e2c 100644
--- a/include/drm/ttm/ttm_pool.h
+++ b/include/drm/ttm/ttm_pool.h
@@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
 int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			       const struct ttm_operation_ctx *ctx);
 
+unsigned int ttm_pool_page_order_nodma(struct page *p);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
-- 
2.34.1

[PATCH v2 03/33] drm/ttm: Record sub-optimal page order allocations in ttm_tt

[Matthew Brost]

When __ttm_pool_alloc() fails to allocate a chunk at the device's
beneficial order and falls back to a smaller order, the object ends up
backed by a sub-optimal set of pages. Nothing currently records this, so
a driver has no way to know an object would benefit from being re-backed
with beneficial-order pages later.

Add TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED to struct ttm_tt page_flags,
cleared at the start of every fresh allocation (in both ttm_pool_alloc()
and ttm_pool_restore_and_alloc()). Set it in __ttm_pool_alloc() when an
allocation at exactly the beneficial order fails before dropping to a
lower order.

Drivers can use this hint to queue the buffer object for later
defragmentation.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 13 +++++++++++++
 include/drm/ttm/ttm_tt.h       | 24 +++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index d34592d4dbc7..370d991c9882 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -796,6 +796,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
+	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	enum ttm_caching page_caching;
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
@@ -846,6 +847,16 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
 			if (order) {
+				/*
+				 * Failing to allocate at the device's beneficial
+				 * order means we are about to back this object
+				 * with a sub-optimal (smaller order) set of
+				 * pages. Record it so the driver can later try to
+				 * defragment the object back to beneficial order.
+				 */
+				if (beneficial_order && order == beneficial_order)
+					tt->page_flags |=
+						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 				--order;
 				page_caching = tt->caching;
 				allow_pools = true;
@@ -910,6 +921,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
+	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
 
 	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
@@ -942,6 +954,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (!restore) {
 		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
 
+		tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		ttm_pool_alloc_state_init(tt, &alloc);
 		if (ctx->gfp_retry_mayfail)
 			gfp |= __GFP_RETRY_MAYFAIL;
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 406437ad674b..ce7533677d77 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -90,6 +90,14 @@ struct ttm_tt {
 	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
 	 * struct ttm_tt has been (possibly partially) backed up.
 	 *
+	 * TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED: Set by the TTM pool allocator
+	 * when at least one chunk that could have been allocated at the pool's
+	 * beneficial order had to fall back to a smaller order. This indicates
+	 * that a sub-optimal set of pages was chosen for this object, and can
+	 * be used by drivers to queue the buffer object for later
+	 * defragmentation. Drivers should access this via the
+	 * ttm_tt_is_beneficial_order_failed() helper.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -101,8 +109,9 @@ struct ttm_tt {
 #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 #define TTM_TT_FLAG_BACKED_UP	        BIT(5)
+#define TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED	BIT(6)
 
-#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
+#define TTM_TT_FLAG_PRIV_POPULATED	BIT(7)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -179,6 +188,19 @@ static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
 	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 }
 
+/**
+ * ttm_tt_is_beneficial_order_failed() - Whether the tt is backed at a
+ * sub-optimal page order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if the pool allocator had to fall back below the pool's
+ * beneficial order when backing this tt, false otherwise.
+ */
+static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
+{
+	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+}
+
 /**
  * ttm_tt_create
  *
-- 
2.34.1

[PATCH v2 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

[Matthew Brost]

__ttm_pool_alloc() walks the page orders in a single loop whose body is
about to grow several defragmentation phases (reusing old pages, a move
budget, out-of-lock preallocation and fault injection). Left inline these
interleave into one deeply nested block of gotos and combinatorial
conditionals that is hard to follow and extend.

Factor the loop's working state into a struct ttm_pool_alloc_iter and
pull the per-order page acquisition and order-fallback into small phase
helpers, so the main loop reads as a short sequence of named steps. This
is a pure refactor with no functional change; subsequent patches extend
the iterator with their defrag phases rather than nesting more logic into
the loop.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 159 ++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 53 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 370d991c9882..6826c8ae6fc5 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -791,18 +791,102 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/**
+ * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
+ *
+ * Bundles the immutable inputs and the mutable per-order state that the
+ * allocation phase helpers share, so the main loop reads as a short sequence
+ * of named phases rather than one deeply nested block.
+ */
+struct ttm_pool_alloc_iter {
+	/* Immutable for the duration of the allocation. */
+	struct ttm_pool *pool;
+	struct ttm_tt *tt;
+	const struct ttm_operation_ctx *ctx;
+	struct ttm_pool_alloc_state *alloc;
+	struct ttm_pool_tt_restore *restore;
+	unsigned int beneficial_order;
+	gfp_t gfp_flags;
+
+	/* Mutated as the loop walks the orders. */
+	unsigned int order;
+	enum ttm_caching page_caching;
+	bool allow_pools;
+	struct page *p;
+};
+
+/*
+ * Acquire a single page for the current order, leaving it in @it->p (NULL on
+ * failure). Tries a same-order pool page, then a fresh system allocation.
+ */
+static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
+{
+	struct ttm_pool_type *pt;
+
+	/* First, try to allocate a page from a pool if one exists. */
+	it->p = NULL;
+	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
+	if (!it->p && pt && it->allow_pools)
+		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
+
+	/*
+	 * If that fails or previously failed, allocate from system. Note that
+	 * this also disallows additional pool allocations using write-back
+	 * cached pools of the same order.
+	 */
+	if (!it->p) {
+		it->page_caching = ttm_cached;
+		it->allow_pools = false;
+		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
+					    it->ctx->beneficial_reclaim_backoff);
+	}
+}
+
+/*
+ * The current order could not be satisfied. Lower it and retry, recording a
+ * sub-optimal backing (so the driver can later re-defrag) when we drop below
+ * the beneficial order.
+ *
+ * Returns 0 to retry the (lowered) order, or a negative errno.
+ */
+static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
+{
+	if (!it->order)
+		return -ENOMEM;
+
+	/*
+	 * Failing to allocate at the device's beneficial order means we are
+	 * about to back this object with a sub-optimal (smaller order) set of
+	 * pages. Record it so the driver can later try to defragment the object
+	 * back to beneficial order.
+	 */
+	if (it->beneficial_order && it->order == it->beneficial_order)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	it->order--;
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return 0;
+}
+
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    const struct ttm_operation_ctx *ctx,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
-	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
-	enum ttm_caching page_caching;
+	struct ttm_pool_alloc_iter it = {
+		.pool = pool,
+		.tt = tt,
+		.ctx = ctx,
+		.alloc = alloc,
+		.restore = restore,
+		.beneficial_order = ttm_pool_beneficial_order(pool),
+		.page_caching = tt->caching,
+		.allow_pools = true,
+	};
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
-	unsigned int order;
-	bool allow_pools;
-	struct page *p;
 	int r;
 
 	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
@@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	page_caching = tt->caching;
-	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	it.gfp_flags = gfp_flags;
+
+	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, alloc)) {
-		struct ttm_pool_type *pt;
-
-		/* First, try to allocate a page from a pool if one exists. */
-		p = NULL;
-		pt = ttm_pool_select_type(pool, page_caching, order);
-		if (pt && allow_pools)
-			p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
-
-		/*
-		 * If that fails or previously failed, allocate from system.
-		 * Note that this also disallows additional pool allocations using
-		 * write-back cached pools of the same order. Consider removing
-		 * that behaviour.
-		 */
-		if (!p) {
-			page_caching = ttm_cached;
-			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order,
-						ctx->beneficial_reclaim_backoff);
-		}
-		/* If that fails, lower the order if possible and retry. */
-		if (!p) {
-			if (order) {
-				/*
-				 * Failing to allocate at the device's beneficial
-				 * order means we are about to back this object
-				 * with a sub-optimal (smaller order) set of
-				 * pages. Record it so the driver can later try to
-				 * defragment the object back to beneficial order.
-				 */
-				if (beneficial_order && order == beneficial_order)
-					tt->page_flags |=
-						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-				--order;
-				page_caching = tt->caching;
-				allow_pools = true;
-				continue;
-			}
-			r = -ENOMEM;
-			goto error_free_all;
+	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Acquire a page (pool / system) for this order. */
+		ttm_pool_iter_acquire_page(&it);
+		if (!it.p) {
+			r = ttm_pool_iter_lower_order(&it);
+			if (r)
+				goto error_free_all;
+			continue;
 		}
-		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
-					    restore);
+
+		r = ttm_pool_page_allocated(pool, it.order, it.p,
+					    it.page_caching, alloc, restore);
 		if (r)
 			goto error_free_page;
 
 		if (ttm_pool_restore_valid(restore)) {
-			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
+			r = ttm_pool_restore_commit(restore, tt->backup, ctx,
+						    alloc);
 			if (r)
 				goto error_free_all;
 		}
@@ -887,7 +940,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	return 0;
 
 error_free_page:
-	ttm_pool_free_page(pool, page_caching, order, p, false);
+	ttm_pool_free_page(pool, it.page_caching, it.order, it.p, false);
 
 error_free_all:
 	if (tt->restore)
-- 
2.34.1

[PATCH v2 05/33] drm/ttm: Support defragmentation moves

[Matthew Brost]

Add infrastructure to let a driver re-back an already-resident BO with
fresh pages (e.g. at a more beneficial order) by reusing the normal move
machinery.

Add ctx->defrag, which makes ttm_bo_validate() skip the "current
resource already compatible" short-circuit so a move is forced even when
placement is unchanged. In ttm_bo_handle_move_mem(), when defragging a
populated BO, stash the old tt, create and populate a fresh one
(allocating new backing), and hand the old tt to the driver move
callback via bo->defrag_old_tt for copying. On success the old tt is
unpopulated and freed; if populate or the move fails, the freshly
allocated backing is discharged and the BO is restored to its original
tt and placement.

The old tt is handed off through bo->defrag_old_tt (reservation-lock
protected) rather than the operation context. The pointer describes BO
state for the duration of the move, so keeping it on the buffer object
is better layering and lets ttm_bo_move_accel_cleanup() pick it up.

Because a defrag move commonly fails (beneficial-order pages are often
unavailable) and then leaves the BO untouched on its original backing,
defer ttm_bo_unmap_virtual() for the defrag case until the new backing
has been populated and the move is about to commit. Ordinary moves still
unmap up front. This avoids needlessly zapping and refaulting every
userspace mapping on the common failure path where nothing moved.

Teach ttm_bo_move_accel_cleanup()'s ghost path about defrag: instead of
tearing the move's source tt down synchronously, hand bo->defrag_old_tt
to the ghost object so it is kept populated until the copy fence signals
and then unbound and destroyed asynchronously, while the BO keeps its
new tt. This lets a driver pipeline defrag teardown behind the GPU copy
rather than blocking on it.

This is driver-opt-in and a no-op unless ctx->defrag is set.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_bo.c      | 83 ++++++++++++++++++++++++++++---
 drivers/gpu/drm/ttm/ttm_bo_util.c | 16 +++++-
 drivers/gpu/drm/ttm/ttm_pool.c    |  5 +-
 include/drm/ttm/ttm_bo.h          | 17 +++++++
 4 files changed, 111 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 3980f376e3ba..d05da4eb1d8a 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -125,36 +125,93 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 {
 	struct ttm_device *bdev = bo->bdev;
 	bool old_use_tt, new_use_tt;
+	bool defrag;
 	int ret;
 
 	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
 	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
 
-	ttm_bo_unmap_virtual(bo);
+	/*
+	 * Only the BO that is the actual defrag target (moved in place via
+	 * ttm_bo_validate(), evict == false) is being defragmented. The same
+	 * ctx->defrag is threaded into evictions triggered while allocating
+	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
+	 * an unrelated victim BO (evict == true) being misclassified as a
+	 * defrag move and needlessly reallocating its backing.
+	 */
+	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
+		 ttm_tt_is_populated(bo->ttm);
+
+	/*
+	 * Tear down userspace mappings up front for ordinary moves. A defrag
+	 * move is likely to leave the BO in place (beneficial-order allocation
+	 * commonly fails), so defer the unmap until the new backing has been
+	 * populated and the move is about to commit. This avoids a needless
+	 * zap and refault of every userspace mapping on the common failure,
+	 * where the BO keeps its original, still-mapped backing.
+	 */
+	if (!defrag)
+		ttm_bo_unmap_virtual(bo);
 
 	/*
 	 * Create and bind a ttm if required.
 	 */
 
 	if (new_use_tt) {
+		struct ttm_tt *defrag_old_tt = NULL;
+
+		/*
+		 * For a defrag move, stash the current populated tt and force a
+		 * fresh one to be created and populated (at beneficial order).
+		 * The old tt is handed to the driver move callback to copy the
+		 * contents from, and freed once the move completes.
+		 */
+		if (defrag) {
+			defrag_old_tt = bo->ttm;
+			bo->ttm = NULL;
+		}
+
 		/* Zero init the new TTM structure if the old location should
 		 * have used one as well.
 		 */
-		ret = ttm_tt_create(bo, old_use_tt);
-		if (ret)
+		ret = ttm_tt_create(bo, old_use_tt && !defrag_old_tt);
+		if (ret) {
+			if (defrag_old_tt)
+				bo->ttm = defrag_old_tt;
 			goto out_err;
+		}
 
-		if (mem->mem_type != TTM_PL_SYSTEM) {
+		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
 			ret = ttm_bo_populate(bo, ctx);
-			if (ret)
+			if (ret) {
+				/*
+				 * Discharge the freshly allocated backing and
+				 * leave the BO on its original tt/placement.
+				 */
+				if (defrag_old_tt) {
+					ttm_tt_destroy(bo->bdev, bo->ttm);
+					bo->ttm = defrag_old_tt;
+				}
 				goto out_err;
+			}
 		}
+
+		bo->defrag_old_tt = defrag_old_tt;
 	}
 
 	ret = dma_resv_reserve_fences(bo->base.resv, 1);
 	if (ret)
 		goto out_err;
 
+	/*
+	 * A defrag move deferred its virtual unmap until the new backing was
+	 * successfully populated; tear the userspace mappings down now, before
+	 * the move swaps in the new pages, so the next fault re-establishes
+	 * them against the new backing.
+	 */
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
 	if (ret) {
 		if (ret == -EMULTIHOP)
@@ -162,12 +219,24 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		goto out_err;
 	}
 
+	if (bo->defrag_old_tt) {
+		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
+		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
+		bo->defrag_old_tt = NULL;
+	}
+
 	ctx->bytes_moved += bo->base.size;
 	return 0;
 
 out_err:
-	if (!old_use_tt)
+	if (bo->defrag_old_tt) {
+		/* Failed defrag move: restore the original backing. */
+		ttm_bo_tt_destroy(bo);
+		bo->ttm = bo->defrag_old_tt;
+		bo->defrag_old_tt = NULL;
+	} else if (!old_use_tt) {
 		ttm_bo_tt_destroy(bo);
+	}
 
 	return ret;
 }
@@ -835,7 +904,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
 	force_space = false;
 	do {
 		/* Check whether we need to move buffer. */
-		if (bo->resource &&
+		if (bo->resource && !ctx->defrag &&
 		    ttm_resource_compatible(bo->resource, placement,
 					    force_space))
 			return 0;
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 3e3c201a0222..771313162c8d 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -626,10 +626,22 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 	 * bo to be unbound and destroyed.
 	 */
 
-	if (dst_use_tt)
+	if (bo->defrag_old_tt) {
+		/*
+		 * Defrag move: @bo already points at the freshly allocated,
+		 * beneficial-order tt that the move copied into. Hand the old
+		 * tt (the move's source) to the ghost so it is kept populated
+		 * until @fence signals and then unbound and destroyed, instead
+		 * of being torn down synchronously. @bo keeps its new tt.
+		 */
+		ghost_obj->ttm = bo->defrag_old_tt;
+		ghost_obj->defrag_old_tt = NULL;
+		bo->defrag_old_tt = NULL;
+	} else if (dst_use_tt) {
 		ghost_obj->ttm = NULL;
-	else
+	} else {
 		bo->ttm = NULL;
+	}
 
 	dma_resv_unlock(&ghost_obj->base._resv);
 	ttm_bo_put(ghost_obj);
diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 6826c8ae6fc5..340855e9bcfe 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -860,8 +860,11 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order)
+	if (it->beneficial_order && it->order == it->beneficial_order) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		if (it->ctx->defrag)
+			return -ENOMEM;
+	}
 
 	it->order--;
 	it->page_caching = it->tt->caching;
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index fd814b5cf535..f4f8a8c29f04 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -135,6 +135,16 @@ struct ttm_buffer_object {
 	 * reservation lock.
 	 */
 	struct sg_table *sg;
+
+	/**
+	 * @defrag_old_tt: Used internally during a defrag move to hand the BO's
+	 * previous, populated struct ttm_tt to the driver's move callback so
+	 * that it can copy the contents into the freshly allocated
+	 * beneficial-order backing. Set by TTM, consumed by the driver; the old
+	 * tt is freed (or restored on failure) by TTM once the move completes.
+	 * Protected by the reservation lock.
+	 */
+	struct ttm_tt *defrag_old_tt;
 };
 
 #define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -194,6 +204,13 @@ struct ttm_operation_ctx {
 	 * reclaim/compaction before falling back to a smaller order anyway.
 	 */
 	bool beneficial_reclaim_backoff;
+	/**
+	 * @defrag: Defragmentation pass. Attempt a move even when the bo's
+	 * current resource is already compatible with the requested placement,
+	 * so that the backing store can be reallocated at the pool's beneficial
+	 * order.
+	 */
+	bool defrag;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
-- 
2.34.1

[PATCH v2 06/33] drm/ttm: Add fault injection for beneficial-order allocation failures

[Matthew Brost]

Add a beneficial_order_fault_inject debugfs fault attribute, mirroring
backup_fault_inject, that forces allocation at (or above) the device's
beneficial order to fail in __ttm_pool_alloc(). When triggered, both the
pool take and the system page allocation are skipped for that order, so
the allocation falls into the existing lower-the-order path that sets
the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (and bails out with
-ENOMEM for defrag moves).

This lets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tracking and the
driver-side defrag path be exercised deterministically, without having to
drive the system into real memory fragmentation to provoke a sub-optimal
backing.

The knob is registered at
/sys/kernel/debug/ttm/beneficial_order_fault_inject and compiles out
when CONFIG_FAULT_INJECTION is disabled.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 35 ++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 340855e9bcfe..34f533adb8eb 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -53,6 +53,7 @@
 #ifdef CONFIG_FAULT_INJECTION
 #include <linux/fault-inject.h>
 static DECLARE_FAULT_ATTR(backup_fault_inject);
+static DECLARE_FAULT_ATTR(beneficial_order_fault_inject);
 #else
 #define should_fail(...) false
 #endif
@@ -812,21 +813,34 @@ struct ttm_pool_alloc_iter {
 	unsigned int order;
 	enum ttm_caching page_caching;
 	bool allow_pools;
+	bool fail_beneficial;
 	struct page *p;
 };
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation.
+ * failure). Tries a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
 	struct ttm_pool_type *pt;
 
-	/* First, try to allocate a page from a pool if one exists. */
 	it->p = NULL;
+
+	/*
+	 * Fault injection: pretend allocation at (or above) the device's
+	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
+	 * beneficial_order_failed tracking and the driver defrag path without
+	 * driving the system into real fragmentation.
+	 */
+	it->fail_beneficial = IS_ENABLED(CONFIG_FAULT_INJECTION) &&
+		it->beneficial_order && it->order >= it->beneficial_order &&
+		should_fail(&beneficial_order_fault_inject, 1);
+
+	/* First, try to allocate a page from a pool if one exists. */
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
-	if (!it->p && pt && it->allow_pools)
+	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
 
 	/*
@@ -834,7 +848,7 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 	 * this also disallows additional pool allocations using write-back
 	 * cached pools of the same order.
 	 */
-	if (!it->p) {
+	if (!it->p && !it->fail_beneficial) {
 		it->page_caching = ttm_cached;
 		it->allow_pools = false;
 		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
@@ -851,6 +865,9 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
  */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
+	bool at_beneficial = it->beneficial_order &&
+			     it->order == it->beneficial_order;
+
 	if (!it->order)
 		return -ENOMEM;
 
@@ -860,13 +877,16 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order) {
+	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		if (it->ctx->defrag)
 			return -ENOMEM;
 	}
 
-	it->order--;
+	if (it->fail_beneficial)
+		it->order = it->beneficial_order - 1;
+	else
+		it->order--;
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
@@ -1536,6 +1556,9 @@ int ttm_pool_mgr_init(unsigned long num_pages)
 #ifdef CONFIG_FAULT_INJECTION
 	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
 				  &backup_fault_inject);
+	fault_create_debugfs_attr("beneficial_order_fault_inject",
+				  ttm_debugfs_root,
+				  &beneficial_order_fault_inject);
 #endif
 #endif
 
-- 
2.34.1

[PATCH v2 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

[Matthew Brost]

A defragmentation move relocates a populated tt into a freshly allocated
one so that sub-beneficial-order chunks can be upgraded to the device's
beneficial order. Previously, if allocation could not complete entirely
at beneficial order, the move failed and all progress was discarded.

Instead, once allocation has successfully upgraded part of the object,
the remainder of the old tt is harvested to complete the new tt. Thread
the BO's old, still-populated tt into the page allocator via a new
ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding,
any existing chunk already at (or above) the beneficial order is reused
directly instead of being reallocated. If allocation later has to fall
back below beneficial_order after making partial progress, the remainder
of the old tt is harvested instead of failing the move. This preserves
the beneficial-order upgrades already made while leaving the rest of the
object unchanged, allowing subsequent defragmentation attempts to make
additional forward progress.

Harvesting only borrows the pages; they remain owned by the old tt until
the move commits. This keeps every unwind path non-destructive: if the
(re)allocation or the move later fails, the new tt is torn down without
freeing the borrowed pages and the old tt is restored fully intact.
Ownership of the shared pages is transferred to the new tt only once the
move has committed. ttm_tt_defrag_disown_borrowed() implements both
directions of the transfer by clearing a tt's slots for every page it
shares with the other tt (same struct page at the same index), so that
tt's teardown leaves the shared pages alone and each page is freed
exactly once by its owner:

- on the alloc unwind and the failed-move paths, the new tt disowns the
  borrowed pages, leaving the old tt as the sole owner;
- on the successful-move paths (synchronous teardown and the pipelined
  ghost-object teardown), the old tt disowns the now-transferred pages,
  leaving the new tt as the sole owner.

Detecting shared pages by same-index struct page identity needs no extra
per-tt bookkeeping, since two distinct allocations never alias. This is
safe with drivers, such as Xe, that build their own per-tt sg table and
DMA-map it independently: each tt maps and unmaps its own sg, so sharing
the underlying pages between tts is DMA-clean.

This is a no-op unless ctx->defrag_old_tt is set, which only happens on a
defragmentation move.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_bo.c      |  20 ++-
 drivers/gpu/drm/ttm/ttm_bo_util.c |   5 +
 drivers/gpu/drm/ttm/ttm_pool.c    | 220 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c      |  67 +++++++++
 include/drm/ttm/ttm_bo.h          |  12 ++
 include/drm/ttm/ttm_tt.h          |   2 +
 6 files changed, 313 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index d05da4eb1d8a..d5082306ab7c 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		}
 
 		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
+			/*
+			 * Let the page allocator harvest already
+			 * beneficial-order chunks out of the old tt to back
+			 * the new one, rather than reallocating them.
+			 */
+			ctx->defrag_old_tt = defrag_old_tt;
 			ret = ttm_bo_populate(bo, ctx);
+			ctx->defrag_old_tt = NULL;
 			if (ret) {
 				/*
 				 * Discharge the freshly allocated backing and
@@ -220,6 +227,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	}
 
 	if (bo->defrag_old_tt) {
+		/*
+		 * Successful defrag move: any pages the new tt borrowed from the
+		 * old tt are now owned by the new tt. Disown them from the old
+		 * tt before it is torn down so they are not freed.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
 		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
 		bo->defrag_old_tt = NULL;
@@ -230,7 +243,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 
 out_err:
 	if (bo->defrag_old_tt) {
-		/* Failed defrag move: restore the original backing. */
+		/*
+		 * Failed defrag move: drop any pages the new tt borrowed from
+		 * the old tt (still owned by the old tt) before destroying the
+		 * new tt, then restore the original backing.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
 		ttm_bo_tt_destroy(bo);
 		bo->ttm = bo->defrag_old_tt;
 		bo->defrag_old_tt = NULL;
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 771313162c8d..7cb718fc97c0 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 		 * tt (the move's source) to the ghost so it is kept populated
 		 * until @fence signals and then unbound and destroyed, instead
 		 * of being torn down synchronously. @bo keeps its new tt.
+		 *
+		 * Any pages the new tt borrowed from the old tt are now owned by
+		 * the new tt; disown them from the old tt so the ghost teardown
+		 * leaves them alone.
 		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ghost_obj->ttm = bo->defrag_old_tt;
 		ghost_obj->defrag_old_tt = NULL;
 		bo->defrag_old_tt = NULL;
diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 34f533adb8eb..5d2bbcce4eb8 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -792,6 +792,133 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/*
+ * Defragmentation page harvesting.
+ *
+ * A defrag move relocates a populated tt into a freshly allocated one in order
+ * to upgrade sub-beneficial-order chunks to the device's beneficial order.
+ * Chunks in the old tt (ctx->defrag_old_tt) that are ALREADY at the beneficial
+ * order do not need to be reallocated: their physical pages can simply be
+ * transferred ("harvested") into the new tt at the same offset, avoiding the
+ * alloc/free churn that is significant for large buffers.
+ *
+ * Harvesting only borrows the pages: the new tt simply references the old tt's
+ * pages, which remain owned by the old tt. This keeps the unwind paths simple -
+ * if the (re)allocation or the move later fails, the new tt is torn down without
+ * freeing the borrowed pages and the old tt is restored fully intact. Ownership
+ * of the shared pages is transferred to the new tt only once the move has
+ * committed, via ttm_tt_defrag_disown_borrowed() (see ttm_tt.c).
+ */
+
+/*
+ * Return the allocation order of the old tt chunk whose head sits at @off, or 0
+ * if @off is not an old chunk head (it falls inside a chunk, or the slot is
+ * empty / backed up). @cursor monotonically tracks the old chunk boundary and
+ * must be advanced past every offset that has already been queried; a page
+ * order may only be read from a chunk head page, hence the lazy walk.
+ */
+static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
+					      struct ttm_tt *old_tt,
+					      pgoff_t off, pgoff_t *cursor)
+{
+	struct page *p;
+
+	while (*cursor < off) {
+		p = old_tt->pages[*cursor];
+		if (!p || ttm_backup_page_ptr_is_handle(p))
+			return 0;
+		*cursor += 1UL << ttm_pool_page_order(pool, p);
+	}
+
+	if (*cursor != off)
+		return 0;
+
+	p = old_tt->pages[off];
+	if (!p || ttm_backup_page_ptr_is_handle(p))
+		return 0;
+
+	return ttm_pool_page_order(pool, p);
+}
+
+/*
+ * Borrow an already beneficial-order chunk from the old tt into @alloc. The
+ * pages stay owned by the old tt until the move commits.
+ */
+static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
+				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+{
+	pgoff_t nr = 1UL << order;
+	int r;
+
+	/*
+	 * A harvested chunk keeps the (same) bo's caching, so the deferred
+	 * caching range is consistent here; flush it before committing, exactly
+	 * like the caching-consistent path in ttm_pool_page_allocated().
+	 */
+	r = ttm_pool_apply_caching(alloc);
+	if (r)
+		return r;
+
+	/*
+	 * The borrowed chunk's page pointers (consecutive struct pages of one
+	 * compound allocation) and dma addresses already sit contiguously in
+	 * the old tt at @off. Copy the whole run with memcpy() instead of the
+	 * per-page increment loop in ttm_pool_allocated_page_commit(): for a
+	 * large defrag the borrowed pages dominate the new tt, so this turns
+	 * the populate fill from O(num_pages) individual stores into a handful
+	 * of bulk copies.
+	 */
+	memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
+	alloc->pages += nr;
+	alloc->remaining_pages -= nr;
+
+	if (alloc->dma_addr) {
+		memcpy(alloc->dma_addr, &old_tt->dma_address[off],
+		       nr * sizeof(*alloc->dma_addr));
+		alloc->dma_addr += nr;
+	}
+
+	alloc->caching_divide = alloc->pages;
+
+	return 0;
+}
+
+static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
+				      struct ttm_tt *old_tt, pgoff_t off,
+				      struct ttm_pool_alloc_state *alloc)
+{
+	while (alloc->remaining_pages) {
+		struct page *p = old_tt->pages[off];
+		unsigned int order = ttm_pool_page_order(pool, p);
+		pgoff_t nr = 1UL << order;
+		int r;
+
+		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		if (r)
+			return r;
+
+		off += nr;
+	}
+
+	return 0;
+}
+
+/**
+ * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
+ * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
+ * @TTM_POOL_ITER_RETRY: Loop state was adjusted (e.g. order lowered or a chunk
+ *	harvested); re-enter the loop body for the next order.
+ * @TTM_POOL_ITER_STOP: The allocation is complete; stop the loop.
+ *
+ * Phase helpers return one of these (>= 0) on success, or a negative errno on
+ * failure (the caller then unwinds via error_free_all).
+ */
+enum ttm_pool_iter_action {
+	TTM_POOL_ITER_FILL,
+	TTM_POOL_ITER_RETRY,
+	TTM_POOL_ITER_STOP,
+};
+
 /**
  * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
  *
@@ -806,6 +933,7 @@ struct ttm_pool_alloc_iter {
 	const struct ttm_operation_ctx *ctx;
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
+	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 
@@ -814,9 +942,50 @@ struct ttm_pool_alloc_iter {
 	enum ttm_caching page_caching;
 	bool allow_pools;
 	bool fail_beneficial;
+	unsigned int alloc_count;
+	pgoff_t old_cursor;
 	struct page *p;
 };
 
+/*
+ * Defrag move: reuse an already beneficial-order chunk from the old tt at this
+ * offset rather than reallocating it. Harvest whatever order the old chunk has
+ * (>= beneficial order), which may exceed the order the new tt would otherwise
+ * pick. An order below beneficial_order is only reused for the unaligned tail
+ * (where there is no benefit to reallocating). Only borrows the pages; the alloc
+ * unwind path can drop them again.
+ *
+ * Returns TTM_POOL_ITER_RETRY when a chunk was harvested, TTM_POOL_ITER_FILL to
+ * fall through to a fresh allocation, or a negative errno.
+ */
+static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
+{
+	unsigned int harvest_order;
+	pgoff_t off;
+	int r;
+
+	if (!it->defrag_old_tt || !it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	off = it->tt->num_pages - it->alloc->remaining_pages;
+	harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
+						  off, &it->old_cursor);
+
+	if (harvest_order < it->beneficial_order &&
+	    it->order >= it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	it->order = harvest_order;
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	if (r)
+		return r;
+
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return TTM_POOL_ITER_RETRY;
+}
+
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
  * failure). Tries a same-order pool page, then a fresh system allocation. Fault
@@ -859,9 +1028,10 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 /*
  * The current order could not be satisfied. Lower it and retry, recording a
  * sub-optimal backing (so the driver can later re-defrag) when we drop below
- * the beneficial order.
+ * the beneficial order. For a defrag move that has already made partial
+ * progress, harvest the remaining old pages and stop instead.
  *
- * Returns 0 to retry the (lowered) order, or a negative errno.
+ * Returns TTM_POOL_ITER_RETRY, TTM_POOL_ITER_STOP, or a negative errno.
  */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
@@ -871,16 +1041,24 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	if (!it->order)
 		return -ENOMEM;
 
-	/*
-	 * Failing to allocate at the device's beneficial order means we are
-	 * about to back this object with a sub-optimal (smaller order) set of
-	 * pages. Record it so the driver can later try to defragment the object
-	 * back to beneficial order.
-	 */
 	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-		if (it->ctx->defrag)
+
+		if (it->alloc_count && it->defrag_old_tt) {
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+			int r;
+
+			r = ttm_pool_harvest_remaining(it->pool,
+						       it->defrag_old_tt, off,
+						       it->alloc);
+			if (r)
+				return r;
+
+			return TTM_POOL_ITER_STOP;
+		} else if (it->defrag_old_tt) {
 			return -ENOMEM;
+		}
 	}
 
 	if (it->fail_beneficial)
@@ -890,7 +1068,7 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
-	return 0;
+	return TTM_POOL_ITER_RETRY;
 }
 
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
@@ -904,6 +1082,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.ctx = ctx,
 		.alloc = alloc,
 		.restore = restore,
+		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -931,13 +1110,22 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Reuse an already beneficial-order chunk from the old tt. */
+		r = ttm_pool_iter_reuse_old(&it);
+		if (r < 0)
+			goto error_free_all;
+		if (r == TTM_POOL_ITER_RETRY)
+			continue;
+
 		/* Acquire a page (pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
-			if (r)
+			if (r < 0)
 				goto error_free_all;
-			continue;
+			if (r == TTM_POOL_ITER_RETRY)
+				continue;
+			break;	/* TTM_POOL_ITER_STOP */
 		}
 
 		r = ttm_pool_page_allocated(pool, it.order, it.p,
@@ -951,6 +1139,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			if (r)
 				goto error_free_all;
 		}
+
+		it.alloc_count++;
 	}
 
 	r = ttm_pool_apply_caching(alloc);
@@ -969,6 +1159,12 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (tt->restore)
 		return r;
 
+	/*
+	 * Drop any pages borrowed from the old tt during a defrag move: they are
+	 * still owned by the old tt and must not be freed here.
+	 */
+	ttm_tt_defrag_disown_borrowed(tt, it.defrag_old_tt);
+
 	caching_divide = alloc->caching_divide - tt->pages;
 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
 	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index b645a1818184..2388ff794a59 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -450,6 +450,73 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
 }
 EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_unpopulate);
 
+/**
+ * ttm_tt_defrag_disown_borrowed() - Drop borrowed defrag pages from a tt
+ * @tt: The tt to drop the borrowed page references from.
+ * @src: The tt the pages may have been harvested from, or NULL.
+ *
+ * During a defragmentation move the freshly (re)allocated tt may harvest
+ * already beneficial-order chunks from the BO's old tt, sharing the underlying
+ * pages instead of reallocating them. Until the move commits these pages stay
+ * owned by the old tt. Clearing a tt's slots for every page it shares with
+ * @src (the same struct page at the same index) makes that tt's teardown leave
+ * the shared pages alone, so they are freed exactly once by their owner.
+ *
+ * This is used both to unwind a failed defrag move (disown the borrowed pages
+ * from the new tt, leaving the old tt intact) and to commit a successful one
+ * (disown the now-transferred pages from the old tt, leaving them owned by the
+ * new tt).
+ */
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
+{
+	pgoff_t i = 0, run_start = 0;
+	bool in_run = false;
+
+	if (!tt || !src)
+		return;
+
+	/*
+	 * Borrowing happens at chunk granularity: a borrowed chunk shares the
+	 * same struct page on both tts across its whole allocation order, while
+	 * a non-borrowed chunk differs across its whole order. Stride by @src's
+	 * chunk order instead of walking every page, coalescing the borrowed
+	 * (matching) chunks into runs and clearing each run with a single
+	 * memset rather than a per-page store. @src is a fully populated,
+	 * non-dma_alloc tt (the defrag path), so its page order is recorded in
+	 * page->private and read via ttm_pool_page_order_nodma().
+	 */
+	while (i < tt->num_pages) {
+		struct page *sp = src->pages[i];
+		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
+		pgoff_t nr = 1UL << order;
+
+		if (sp && tt->pages[i] == sp) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+		} else if (in_run) {
+			memset(&tt->pages[run_start], 0,
+			       (i - run_start) * sizeof(*tt->pages));
+			if (tt->dma_address)
+				memset(&tt->dma_address[run_start], 0,
+				       (i - run_start) *
+				       sizeof(*tt->dma_address));
+			in_run = false;
+		}
+
+		i += nr;
+	}
+
+	if (in_run) {
+		memset(&tt->pages[run_start], 0,
+		       (i - run_start) * sizeof(*tt->pages));
+		if (tt->dma_address)
+			memset(&tt->dma_address[run_start], 0,
+			       (i - run_start) * sizeof(*tt->dma_address));
+	}
+}
+
 #ifdef CONFIG_DEBUG_FS
 
 /* Test the shrinker functions and dump the result */
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index f4f8a8c29f04..fcaba5fb8bf7 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -211,6 +211,18 @@ struct ttm_operation_ctx {
 	 * order.
 	 */
 	bool defrag;
+	/**
+	 * @defrag_old_tt: During a defragmentation move, the bo's existing
+	 * (populated) tt that is being relocated. The page allocator may
+	 * harvest already beneficial-order chunks from this tt to back the new
+	 * tt instead of reallocating them, avoiding alloc/free churn for large
+	 * buffers. Harvested pages are only borrowed: they stay owned by
+	 * @defrag_old_tt until the move commits, at which point
+	 * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
+	 * transferred pages so they are not freed when the old tt is torn down.
+	 * NULL for non-defrag operations.
+	 */
+	struct ttm_tt *defrag_old_tt;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index ce7533677d77..55c35bcf134d 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
  */
 void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
 
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
+
 /**
  * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
  *
-- 
2.34.1

[PATCH v2 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

[Matthew Brost]

A defragmentation move reallocates an object's backing at the device's
beneficial order and harvests whatever the allocator cannot upgrade from
the old tt. For a very large object this can reallocate (and have the
driver re-copy) the entire backing in one move, which is expensive and
can stall concurrent work for an unbounded stretch.

Add an optional per-move byte budget so a large object can be upgraded in
slices across successive moves. Plumb a new
ttm_operation_ctx::defrag_bytes_remaining through to the pool allocator:
only pages actually (re)allocated at the beneficial order are charged
against it; pages harvested (borrowed) from defrag_old_tt are free and do
not count. Once the budget is exhausted the allocator stops upgrading and
completes the new tt by harvesting the remainder of the old tt as-is,
marking the tt sub-optimal (TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED) so the
driver re-queues it and a later move resumes the upgrade. The slices
already upgraded come back as free already-beneficial harvests on the
next pass, so each move spends budget only on new forward progress.

The allocator decrements the budget as it allocates and writes the unused
remainder back on success, letting the caller compute how many bytes were
actually spent. A value of 0 means unlimited (upgrade the whole object in
one move), preserving the previous behaviour. This is only consulted on a
defragmentation move, i.e. when defrag_old_tt is set.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 101 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  14 +++++
 2 files changed, 112 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 5d2bbcce4eb8..a6847a16c47f 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -77,6 +77,11 @@ struct ttm_pool_dma {
  * @dma_addr: Pointer to the next tt dma_address entry to populate if any.
  * @remaining_pages: Remaining pages to populate.
  * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @defrag_bytes_remaining: Byte budget for newly (re)allocated pages on a
+ * defragmentation move, seeded from ttm_operation_ctx::defrag_bytes_remaining.
+ * Pages harvested (borrowed) from the old tt are free and do not count; once
+ * this drops to zero the allocator stops upgrading and harvests the remainder.
+ * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -84,6 +89,7 @@ struct ttm_pool_alloc_state {
 	dma_addr_t *dma_addr;
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
+	s64 defrag_bytes_remaining;
 };
 
 /**
@@ -780,6 +786,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->dma_addr = tt->dma_address;
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
+	alloc->defrag_bytes_remaining = 0;
 }
 
 /*
@@ -885,14 +892,33 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 				      struct ttm_tt *old_tt, pgoff_t off,
-				      struct ttm_pool_alloc_state *alloc)
+				      struct ttm_pool_alloc_state *alloc,
+				      bool *suboptimal)
 {
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	pgoff_t bnr = 1UL << beneficial;
+
 	while (alloc->remaining_pages) {
 		struct page *p = old_tt->pages[off];
 		unsigned int order = ttm_pool_page_order(pool, p);
 		pgoff_t nr = 1UL << order;
+		pgoff_t num_pages = off + alloc->remaining_pages;
 		int r;
 
+		/*
+		 * Only report a sub-optimal backing when a chunk we cannot
+		 * upgrade (out of budget/prealloc) sits in a beneficial-order
+		 * aligned region that fully fits in the tt: that region could
+		 * have been upgraded with more budget. A chunk below the
+		 * beneficial order is expected for the unaligned tail (the last
+		 * partial beneficial-order region) and for already beneficial-
+		 * order harvests - neither leaves real defrag work, so they must
+		 * not flag the tt for re-queue.
+		 */
+		if (beneficial && order < beneficial &&
+		    round_down(off, bnr) + bnr <= num_pages)
+			*suboptimal = true;
+
 		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
 		if (r)
 			return r;
@@ -936,6 +962,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
+	bool defrag_capped;
 
 	/* Mutated as the loop walks the orders. */
 	unsigned int order;
@@ -947,6 +974,44 @@ struct ttm_pool_alloc_iter {
 	struct page *p;
 };
 
+/*
+ * Defrag move budget exhausted: the upgrade can make no further progress this
+ * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
+ * entry.
+ */
+static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
+{
+	if (!it->defrag_old_tt)
+		return false;
+
+	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+}
+
+/*
+ * Complete the new tt by harvesting the rest of the old tt as-is (borrowing its
+ * pages, including any still below beneficial order). Never fall back to in-lock
+ * reclaim for the shortfall - that would reintroduce the stall we preallocated
+ * to avoid. Only re-queue (flag) the tt when the harvested remainder is
+ * genuinely sub-optimal; if every remaining old chunk was already at beneficial
+ * order the move fully upgraded the object.
+ */
+static int ttm_pool_iter_harvest_rest(struct ttm_pool_alloc_iter *it)
+{
+	pgoff_t off = it->tt->num_pages - it->alloc->remaining_pages;
+	bool suboptimal = false;
+	int r;
+
+	r = ttm_pool_harvest_remaining(it->pool, it->defrag_old_tt, off,
+				       it->alloc, &suboptimal);
+	if (r)
+		return r;
+
+	if (suboptimal)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	return 0;
+}
+
 /*
  * Defrag move: reuse an already beneficial-order chunk from the old tt at this
  * offset rather than reallocating it. Harvest whatever order the old chunk has
@@ -1047,11 +1112,12 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 		if (it->alloc_count && it->defrag_old_tt) {
 			pgoff_t off = it->tt->num_pages -
 				      it->alloc->remaining_pages;
+			bool suboptimal = false;
 			int r;
 
 			r = ttm_pool_harvest_remaining(it->pool,
 						       it->defrag_old_tt, off,
-						       it->alloc);
+						       it->alloc, &suboptimal);
 			if (r)
 				return r;
 
@@ -1106,10 +1172,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;
 
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/*
+		 * Out of defrag budget: harvest the rest of the old tt as-is and
+		 * stop (the tt is re-queued if the remainder is still
+		 * sub-optimal).
+		 */
+		if (ttm_pool_iter_defrag_exhausted(&it)) {
+			r = ttm_pool_iter_harvest_rest(&it);
+			if (r)
+				goto error_free_all;
+			break;
+		}
+
 		/* Reuse an already beneficial-order chunk from the old tt. */
 		r = ttm_pool_iter_reuse_old(&it);
 		if (r < 0)
@@ -1140,6 +1219,15 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				goto error_free_all;
 		}
 
+		/*
+		 * Charge a newly (re)allocated chunk against the defrag move
+		 * budget. Harvested chunks borrow the old tt's pages for free
+		 * and never reach here.
+		 */
+		if (it.defrag_capped)
+			alloc->defrag_bytes_remaining -=
+				(s64)(1UL << it.order) << PAGE_SHIFT;
+
 		it.alloc_count++;
 	}
 
@@ -1189,14 +1277,21 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx)
 {
 	struct ttm_pool_alloc_state alloc;
+	int ret;
 
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
 	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
+	alloc.defrag_bytes_remaining = ctx->defrag_bytes_remaining;
+
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+
+	/* Report the unused defrag budget back to the caller. */
+	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
 
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index fcaba5fb8bf7..6124d2627b47 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -223,6 +223,20 @@ struct ttm_operation_ctx {
 	 * NULL for non-defrag operations.
 	 */
 	struct ttm_tt *defrag_old_tt;
+	/**
+	 * @defrag_bytes_remaining: Byte budget for newly allocated pages during
+	 * a defragmentation move. Only pages that are actually (re)allocated at
+	 * the beneficial order count against it; pages harvested (borrowed) from
+	 * @defrag_old_tt are free and do not. When the budget is exhausted the
+	 * pool allocator stops upgrading and completes the new tt by harvesting
+	 * the remainder of @defrag_old_tt, marking the tt sub-optimal so the
+	 * driver can resume on a later pass. The allocator decrements this as it
+	 * allocates, so on return it holds the unused budget (which the caller
+	 * can use to compute how many bytes were spent). A value of 0 means
+	 * unlimited (upgrade the whole object). Only consulted when
+	 * @defrag_old_tt is set.
+	 */
+	s64 defrag_bytes_remaining;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
-- 
2.34.1

[PATCH v2 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

[Matthew Brost]

Defragmentation moves reallocate a BO's backing at the pool's beneficial
order. On a fragmented system those high-order allocations frequently
stall in direct reclaim/compaction, and they run under the BO dma-resv
lock, holding up concurrent rendering. The defrag worker does not care
how long the work takes; it only cares about minimizing locked time.

Add a preallocation bag so the expensive allocations can be hoisted out
of the lock:

 - struct ttm_pool_prealloc carries beneficial-order pages preallocated
   outside the lock; ttm_pool_prealloc_fill()/_fini() allocate and
   release them. __ttm_pool_alloc() consumes them via ctx->prealloc.
 - Apply the requested cpu-caching to the bag pages during fill, so the
   set_memory_*() cost is paid outside the lock too; consumed pages are
   already cache-consistent and leftovers are restored to write-back on
   free. Bag pages are never highmem so the kernel mapping is always
   valid for set_memory_*().
 - Record the exact number of sub-optimal pages on the tt
   (nr_suboptimal_pages) at populate time so a defrag pass can size its
   prealloc precisely instead of guessing.
 - Once a defrag move exhausts the prealloc bag, harvest the remainder
   of the old tt rather than entering reclaim under the lock, and flag
   the tt sub-optimal so the driver re-queues it for a later pass.

This keeps all reclaim/compaction stalls and cache mode changes out of
the dma-resv critical section: lock-held time then tracks only the bytes
actually copied.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 243 +++++++++++++++++++++++++++++++--
 include/drm/ttm/ttm_bo.h       |  11 ++
 include/drm/ttm/ttm_pool.h     |  29 ++++
 include/drm/ttm/ttm_tt.h       |  17 +++
 4 files changed, 286 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index a6847a16c47f..76e5fc54ea9a 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -82,6 +82,9 @@ struct ttm_pool_dma {
  * Pages harvested (borrowed) from the old tt are free and do not count; once
  * this drops to zero the allocator stops upgrading and harvests the remainder.
  * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
+ * @nr_suboptimal_pages: Number of pages backed below the pool's beneficial
+ * order. Recorded by the pool allocator after populate; a defrag pass reads it
+ * to size its out-of-lock preallocation exactly.
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -90,6 +93,7 @@ struct ttm_pool_alloc_state {
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
 	s64 defrag_bytes_remaining;
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -787,6 +791,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
 	alloc->defrag_bytes_remaining = 0;
+	alloc->nr_suboptimal_pages = 0;
 }
 
 /*
@@ -852,7 +857,8 @@ static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
  * pages stay owned by the old tt until the move commits.
  */
 static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
-				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+				 unsigned int beneficial_order, pgoff_t off,
+				 struct ttm_pool_alloc_state *alloc)
 {
 	pgoff_t nr = 1UL << order;
 	int r;
@@ -887,6 +893,9 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 	alloc->caching_divide = alloc->pages;
 
+	if (order < beneficial_order)
+		alloc->nr_suboptimal_pages += 0x1 << order;
+
 	return 0;
 }
 
@@ -919,7 +928,8 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		    round_down(off, bnr) + bnr <= num_pages)
 			*suboptimal = true;
 
-		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		r = ttm_pool_harvest_page(old_tt, order, beneficial, off,
+					  alloc);
 		if (r)
 			return r;
 
@@ -929,6 +939,172 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}
+
+/*
+ * Apply a tt's cpu-caching to a batch of freshly system-allocated (write-back)
+ * prealloc pages in one shot. @cpages is an unpacked array of @ncpages
+ * individual 4K pages (the constituent pages of the mixed-order chunks, expanded
+ * so set_pages_array_*() sees one entry per page), since the packed prealloc bag
+ * holds multi-order chunks. Prealloc pages are never highmem (see
+ * ttm_pool_prealloc_gfp), so their kernel mapping is valid. No-op for cached
+ * pages and on non-x86. Returns non-zero if the caching change failed (the
+ * caller drops the whole bag and falls back to in-line allocation).
+ */
+static int ttm_pool_prealloc_apply_caching(enum ttm_caching caching,
+					   struct page **cpages,
+					   unsigned int ncpages)
+{
+#ifdef CONFIG_X86
+	if (!ncpages)
+		return 0;
+
+	switch (caching) {
+	case ttm_cached:
+		break;
+	case ttm_write_combined:
+		return set_pages_array_wc(cpages, ncpages);
+	case ttm_uncached:
+		return set_pages_array_uc(cpages, ncpages);
+	}
+#endif
+	return 0;
+}
+
+/* Expand an @order chunk into its constituent 4K pages for bulk caching. */
+static void ttm_pool_prealloc_stage_caching(struct page **cpages,
+					    unsigned int *ncpages,
+					    struct page *p, unsigned int order)
+{
+	unsigned int i, nr = 1u << order;
+
+	for (i = 0; i < nr; i++)
+		cpages[(*ncpages)++] = p + i;
+}
+
+/**
+ * ttm_pool_prealloc_fill() - Preallocate beneficial-order pages outside any lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; @pp->order is set to the beneficial order.
+ * @count: Number of beneficial-order chunks to attempt.
+ *
+ * Allocate up to @count beneficial-order chunks, parking them in @pp for a
+ * later __ttm_pool_alloc() defrag move to consume under the dma-resv lock. May
+ * sleep/reclaim freely as it runs unlocked. A short fill is fine: the pool
+ * falls back to in-line allocation for the shortfall. DMA-alloc pools are not
+ * supported (count stays 0). Returns 0 (release with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count)
+{
+	unsigned int order = ttm_pool_beneficial_order(pool);
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool);
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = order;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+
+	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
+	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	pp->pages = kvzalloc_objs(*pp->pages, count);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Every chunk is a fresh write-back system page of @order that needs a
+	 * caching change; collect their constituent pages into an unpacked
+	 * scratch array and issue a single set_pages_array_*() after the fill.
+	 * Cached tts need no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, (size_t)count << order);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (pp->count < count) {
+		struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);
+
+		if (!p)
+			break;
+
+		pp->pages[pp->count++] = p;
+		if (cpages)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+	}
+
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++)
+			ttm_pool_free_page(pool, tt_caching, order,
+					   pp->pages[i], false);
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill);
+
+/**
+ * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
+ * @pool: The pool the pages came from.
+ * @pp: Prealloc bag to drain. Consumed pages (< @used) are now owned by the tt.
+ */
+void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
+{
+	unsigned int i;
+
+	for (i = pp->used; i < pp->count; ++i)
+		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
+				   false);
+	kvfree(pp->pages);
+	pp->pages = NULL;
+	pp->count = pp->used = 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fini);
+
 /**
  * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
  * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
@@ -960,6 +1136,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
 	struct ttm_tt *defrag_old_tt;
+	struct ttm_pool_prealloc *prealloc;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 	bool defrag_capped;
@@ -975,16 +1152,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
  */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;
 }
 
 /*
@@ -1041,7 +1222,8 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 		return TTM_POOL_ITER_FILL;
 
 	it->order = harvest_order;
-	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order,
+				  it->beneficial_order, off, it->alloc);
 	if (r)
 		return r;
 
@@ -1053,11 +1235,13 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation. Fault
- * injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a beneficial-order page preallocated outside the
+ * dma-resv lock (defrag), a same-order pool page, then a fresh system
+ * allocation. Fault injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
+	struct ttm_pool_prealloc *pp = it->prealloc;
 	struct ttm_pool_type *pt;
 
 	it->p = NULL;
@@ -1072,7 +1256,18 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 		it->beneficial_order && it->order >= it->beneficial_order &&
 		should_fail(&beneficial_order_fault_inject, 1);
 
-	/* First, try to allocate a page from a pool if one exists. */
+	/*
+	 * Defrag move: consume a beneficial-order page preallocated outside the
+	 * dma-resv lock instead of allocating one in-line (which may stall in
+	 * reclaim/compaction). Falls through once the bag is empty.
+	 */
+	if (pp && !it->fail_beneficial && it->order >= it->beneficial_order &&
+	    pp->used < pp->count) {
+		it->order = it->beneficial_order;
+		it->p = pp->pages[pp->used++];
+		it->page_caching = it->tt->caching;
+	}
+
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
 	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
@@ -1149,6 +1344,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.alloc = alloc,
 		.restore = restore,
 		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
+		.prealloc = restore ? NULL : ctx->prealloc,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -1178,8 +1374,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {
@@ -1196,7 +1392,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (r == TTM_POOL_ITER_RETRY)
 			continue;
 
-		/* Acquire a page (pool / system) for this order. */
+		/* Acquire a page (prealloc / pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
@@ -1228,6 +1424,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			alloc->defrag_bytes_remaining -=
 				(s64)(1UL << it.order) << PAGE_SHIFT;
 
+		if (it.order < it.beneficial_order)
+			alloc->nr_suboptimal_pages += 0x1 << it.order;
 		it.alloc_count++;
 	}
 
@@ -1291,6 +1489,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	/* Report the unused defrag budget back to the caller. */
 	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
 	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
@@ -1343,6 +1544,15 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 			if (ret)
 				return ret;
+
+			/*
+			 * __ttm_pool_alloc() counts each freshly (re)allocated
+			 * chunk against nr_suboptimal_pages, but a chunk whose
+			 * backup copy was interrupted and is finished here
+			 * resumes past that loop, so account for it directly.
+			 */
+			if (restore->order < ttm_pool_beneficial_order(pool))
+				alloc.nr_suboptimal_pages += 1U << restore->order;
 		}
 		if (!alloc.remaining_pages) {
 			ret = ttm_pool_apply_caching(&alloc);
@@ -1352,11 +1562,16 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			kfree(tt->restore);
 			tt->restore = NULL;
 
+			tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
 			return 0;
 		}
 	}
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
+	return ret;
 }
 
 /**
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 6124d2627b47..a80304f179ba 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -237,6 +237,17 @@ struct ttm_operation_ctx {
 	 * @defrag_old_tt is set.
 	 */
 	s64 defrag_bytes_remaining;
+	/**
+	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
+	 * move. The pool allocator consumes these instead of allocating fresh
+	 * beneficial-order pages under the lock, moving the (potentially
+	 * reclaim/compaction stalling) high-order allocations out of the
+	 * critical section. NULL means allocate in-line as usual. The pool
+	 * silently falls back to in-line allocation for any shortfall. The
+	 * allocator consumes it for beneficial-order chunks of any populate;
+	 * drivers only set it for a defrag move.
+	 */
+	struct ttm_pool_prealloc *prealloc;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
index 753203980e2c..71670350eb15 100644
--- a/include/drm/ttm/ttm_pool.h
+++ b/include/drm/ttm/ttm_pool.h
@@ -81,6 +81,35 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx);
 void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
+/**
+ * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
+ * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
+ *         short); each entry is the head page of a 1 << @order chunk.
+ * @order: Page order of every preallocated chunk.
+ * @caching: CPU caching applied to the pages, so leftovers can be restored
+ *           to write-back before being freed.
+ * @count: Number of valid entries in @pages.
+ * @used: Number of entries already consumed by the pool allocator.
+ *
+ * Defrag pages are interchangeable, so only a count of beneficial-order chunks
+ * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
+ * __ttm_pool_alloc() drains it; any unused tail is released by
+ * ttm_pool_prealloc_fini().
+ */
+struct ttm_pool_prealloc {
+	struct page **pages;
+	unsigned int order;
+	enum ttm_caching caching;
+	unsigned int count;
+	unsigned int used;
+};
+
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count);
+void ttm_pool_prealloc_fini(struct ttm_pool *pool,
+			    struct ttm_pool_prealloc *pp);
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
+
 void ttm_pool_init(struct ttm_pool *pool, struct device *dev,
 		   int nid, unsigned int alloc_flags);
 void ttm_pool_fini(struct ttm_pool *pool);
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 55c35bcf134d..81a5446ca693 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -135,6 +135,12 @@ struct ttm_tt {
 	enum ttm_caching caching;
 	/** @restore: Partial restoration from backup state. TTM private */
 	struct ttm_pool_tt_restore *restore;
+	/**
+	 * @nr_suboptimal_pages: Number of pages backed below the pool's
+	 * beneficial order. Recorded by the pool allocator after populate; a
+	 * defrag pass reads it to size its out-of-lock preallocation exactly.
+	 */
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -201,6 +207,17 @@ static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 }
 
+/**
+ * ttm_tt_suboptimal_pages() - Pages backed below the pool's beneficial order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: Number of pages a defrag move would (re)allocate at beneficial order.
+ */
+static inline u32 ttm_tt_suboptimal_pages(const struct ttm_tt *tt)
+{
+	return tt->nr_suboptimal_pages;
+}
+
 /**
  * ttm_tt_create
  *
-- 
2.34.1

[PATCH v2 10/33] drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()

[Matthew Brost]

ttm_pool_prealloc_fill() lets a driver preallocate beneficial-order pages
outside the dma-resv lock for a defragmentation move, so the high-order
allocations that may stall in reclaim/compaction happen off the critical
section.  That bag is deliberately partial: it holds only interchangeable
beneficial-order chunks and the allocator consumes it opportunistically,
falling back to in-line allocation for everything else.

An ordinary populate of a system-memory buffer object has the same
problem - the whole backing is allocated under the dma-resv lock - but no
way to move it out of the critical section, because the existing bag does
not cover sub-beneficial orders and does not describe a full tt.

Add a "full" flavour of the same preallocation mechanism:

  - struct ttm_pool_prealloc gains a @full flag.  In full mode the bag is
    an ordered tiling of an entire tt at mixed orders (pool pages
    included), rather than a set of interchangeable beneficial-order
    chunks.  The per-chunk order is recorded on the page itself and read
    back at consume time, so no parallel order array is needed.

  - ttm_pool_prealloc_fill_full() allocates the whole backing for a tt of
    @num_pages pages up front, mirroring the in-line allocator's order
    walk (highest order first, capped by the remaining page count).  It
    reuses the existing unlocked helpers (ttm_pool_prealloc_gfp(),
    ttm_pool_set_caching()) via a new ttm_pool_take_page() helper that
    tries a same-order pool page before a fresh system allocation.

  - ttm_pool_prealloc_fini() frees any unconsumed tail at each chunk's own
    order when @full.

  - ttm_pool_prealloc_fill_full() takes a @beneficial_reclaim_backoff
    argument, plumbed to ttm_pool_alloc_page(), that lets the caller
    allocate higher-order chunks without aggressive reclaim/compaction.
    This suits a driver that upgrades suboptimal pages to beneficial order
    in place later (e.g. via a background defragmenter) and so has no need
    to stall the user context reclaiming for contiguity up front.

  - __ttm_pool_alloc() consumes a full bag with a single early branch in
    ttm_pool_iter_acquire_page(): install the next chunk as-is (order
    taken from the page) and return; once the best-effort bag is drained
    the loop falls through to normal in-line allocation.  The defrag
    harvest and reuse-old phases are inert here as there is no
    defrag_old_tt.

Like the defrag bag, this is best-effort: a short fill is fine (the pool
allocates the shortfall in-line) and DMA-alloc pools are unsupported
(caching is applied on the kernel mapping).  No functional change for
existing users; the new fill routine has no in-tree caller yet.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 232 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  19 +--
 include/drm/ttm/ttm_pool.h     |  27 +++-
 3 files changed, 258 insertions(+), 20 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 76e5fc54ea9a..762ff1c73390 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -1031,6 +1031,7 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 	pp->caching = tt_caching;
 	pp->count = 0;
 	pp->used = 0;
+	pp->full = false;
 
 	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
 	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
@@ -1087,6 +1088,187 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 }
 EXPORT_SYMBOL(ttm_pool_prealloc_fill);
 
+/*
+ * Acquire a single *@order chunk, trying a same-order pool page first (already
+ * carries @caching) then a fresh system allocation (write-back). Runs unlocked.
+ * When @beneficial_reclaim_backoff is set, higher-order system allocations skip
+ * aggressive reclaim/compaction (see ttm_pool_alloc_page()).
+ *
+ * A fresh system page is write-back and needs the tt's caching applied later in
+ * bulk; *@need_caching is set true for it and false for a (pre-cached) pool
+ * page.
+ *
+ * On a genuine failure *@order is left untouched (the caller lowers it by one
+ * and retries). Fault injection instead pretends allocation at (or above) the
+ * beneficial order failed and lowers *@order to the beneficial order, so the
+ * caller's next attempt lands below it - mirroring the in-line acquire path's
+ * fail_beneficial jump and exercising the beneficial_order_failed tracking
+ * without driving the system into real fragmentation.
+ *
+ * Returns NULL if no page of *@order could be obtained.
+ */
+static struct page *ttm_pool_take_page(struct ttm_pool *pool,
+				       enum ttm_caching caching,
+				       unsigned int *order, gfp_t gfp,
+				       bool *need_caching,
+				       bool beneficial_reclaim_backoff)
+{
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	struct ttm_pool_type *pt;
+	struct page *p;
+
+	*need_caching = false;
+
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && beneficial &&
+	    *order >= beneficial &&
+	    should_fail(&beneficial_order_fault_inject, 1)) {
+		*order = beneficial;
+		return NULL;
+	}
+
+	pt = ttm_pool_select_type(pool, caching, *order);
+	if (pt) {
+		p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
+		if (p)
+			return p;	/* pool page already carries @caching */
+	}
+
+	p = ttm_pool_alloc_page(pool, gfp, *order, beneficial_reclaim_backoff);
+	if (!p)
+		return NULL;
+
+	/* Fresh system page is write-back; caller applies @caching in bulk. */
+	*need_caching = true;
+	return p;
+}
+
+/**
+ * ttm_pool_prealloc_fill_full() - Preallocate a whole tt's backing outside any
+ * lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; marked @full and tiled with mixed-order chunks.
+ * @num_pages: Number of 4K pages the tt needs backed.
+ *
+ * Unlike ttm_pool_prealloc_fill() (which parks a bag of interchangeable
+ * beneficial-order defrag chunks), this allocates the ENTIRE backing for a tt
+ * of @num_pages pages up front, at whatever mix of orders is available (pool
+ * pages included), so that the subsequent __ttm_pool_alloc() under the dma-resv
+ * lock merely installs the pages without stalling in reclaim/compaction. The
+ * chunk orders mirror the in-line allocator's order walk; each chunk's order is
+ * recorded on the page itself and read back at consume time.
+ *
+ * May sleep/reclaim freely as it runs unlocked. When
+ * @beneficial_reclaim_backoff is set, higher-order chunks are taken without
+ * aggressive reclaim/compaction: the caller relies on a background
+ * defragmenter to upgrade to beneficial-order pages in place later, so there
+ * is no need to stall the user context reclaiming for contiguity here. A short
+ * fill is fine: the pool falls back to in-line allocation for the shortfall.
+ * DMA-alloc pools are not supported (the bag stays empty). Returns 0 (release
+ * with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff)
+{
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool) | __GFP_ZERO |
+		__GFP_RETRY_MAYFAIL | __GFP_NOWARN;
+	unsigned int highest = MAX_PAGE_ORDER;
+	unsigned int remaining = num_pages;
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = 0;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+	pp->full = true;
+
+	/* Caching is applied on the kernel mapping, so DMA-alloc is unsupported. */
+	if (!num_pages || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	/* Worst case is one order-0 chunk per page. */
+	pp->pages = kvzalloc_objs(*pp->pages, num_pages);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Fresh write-back chunks get their caching changed in a single
+	 * set_pages_array_*() after the fill; collect their constituent pages
+	 * into an unpacked scratch array (one entry per 4K page) since the
+	 * packed bag holds mixed-order chunks. Cached tts and pool pages need
+	 * no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, num_pages);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (remaining) {
+		unsigned int order = min_t(unsigned int, highest,
+					   __fls(remaining));
+		bool need_caching;
+		struct page *p;
+
+		for (;;) {
+			p = ttm_pool_take_page(pool, tt_caching, &order, gfp,
+					       &need_caching,
+					       beneficial_reclaim_backoff);
+			if (p)
+				break;
+			if (!order)
+				goto apply_caching;	/* short fill */
+			/*
+			 * Genuine failure: retry one order down. A fault
+			 * injection lowered @order to the beneficial order
+			 * above, so this drops it below - landing on a
+			 * sub-optimal (defrag-worthy) chunk.
+			 */
+			order--;
+		}
+
+		pp->pages[pp->count++] = p;
+		if (cpages && need_caching)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+		remaining -= 1u << order;
+		highest = order;
+	}
+
+apply_caching:
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++) {
+			struct page *p = pp->pages[i];
+
+			ttm_pool_free_page(pool, tt_caching,
+					   ttm_pool_page_order_nodma(p), p,
+					   false);
+		}
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill_full);
+
 /**
  * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
  * @pool: The pool the pages came from.
@@ -1096,9 +1278,13 @@ void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
 {
 	unsigned int i;
 
-	for (i = pp->used; i < pp->count; ++i)
-		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
-				   false);
+	for (i = pp->used; i < pp->count; ++i) {
+		struct page *p = pp->pages[i];
+		unsigned int order = pp->full ?
+			ttm_pool_page_order_nodma(p) : pp->order;
+
+		ttm_pool_free_page(pool, pp->caching, order, p, false);
+	}
 	kvfree(pp->pages);
 	pp->pages = NULL;
 	pp->count = pp->used = 0;
@@ -1235,9 +1421,10 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries, in order: a beneficial-order page preallocated outside the
- * dma-resv lock (defrag), a same-order pool page, then a fresh system
- * allocation. Fault injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a chunk from a full out-of-lock prealloc (populate
+ * of the whole tt), a beneficial-order page preallocated outside the dma-resv
+ * lock (defrag), a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
@@ -1246,6 +1433,39 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 
 	it->p = NULL;
 
+	/*
+	 * Full prealloc: the entire backing was allocated (pool + system, at
+	 * mixed orders) outside the dma-resv lock. Install the next chunk as-is,
+	 * taking its order from the page. Falls through to in-line allocation
+	 * once the best-effort bag is drained.
+	 */
+	if (pp && pp->full && pp->used < pp->count) {
+		it->fail_beneficial = false;
+		it->p = pp->pages[pp->used++];
+		it->order = ttm_pool_page_order_nodma(it->p);
+		it->page_caching = it->tt->caching;
+
+		/*
+		 * The full prealloc tiles at whatever orders were available
+		 * without stalling in reclaim. A sub-beneficial chunk only
+		 * means a sub-optimal backing (defrag-worthy) when it sits in a
+		 * beneficial-order aligned region that fully fits in the tt -
+		 * that region could have been a beneficial-order page. A low
+		 * order in the unaligned trailing tail is expected and must not
+		 * flag the tt (mirrors ttm_pool_harvest_remaining()).
+		 */
+		if (it->beneficial_order && it->order < it->beneficial_order) {
+			pgoff_t bnr = 1UL << it->beneficial_order;
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+
+			if (round_down(off, bnr) + bnr <= it->tt->num_pages)
+				it->tt->page_flags |=
+					TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		}
+		return;
+	}
+
 	/*
 	 * Fault injection: pretend allocation at (or above) the device's
 	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index a80304f179ba..52ec7e5c0d2d 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -238,14 +238,17 @@ struct ttm_operation_ctx {
 	 */
 	s64 defrag_bytes_remaining;
 	/**
-	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
-	 * move. The pool allocator consumes these instead of allocating fresh
-	 * beneficial-order pages under the lock, moving the (potentially
-	 * reclaim/compaction stalling) high-order allocations out of the
-	 * critical section. NULL means allocate in-line as usual. The pool
-	 * silently falls back to in-line allocation for any shortfall. The
-	 * allocator consumes it for beneficial-order chunks of any populate;
-	 * drivers only set it for a defrag move.
+	 * @prealloc: Pages preallocated outside the dma-resv lock. The pool
+	 * allocator consumes these instead of allocating under the lock, moving
+	 * the (potentially reclaim/compaction stalling) allocations out of the
+	 * critical section. NULL means allocate in-line as usual. Two flavours,
+	 * distinguished by ttm_pool_prealloc::full:
+	 *   - defrag bag (full=false): a bag of interchangeable beneficial-order
+	 *     chunks consumed only for beneficial-order chunks of a defrag move.
+	 *   - full tiling (full=true): the entire backing of the tt, at mixed
+	 *     orders, consumed for every chunk of an ordinary populate.
+	 * The allocator silently falls back to in-line allocation for any
+	 * shortfall.
 	 */
 	struct ttm_pool_prealloc *prealloc;
 	/**
diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
index 71670350eb15..777d5aa08f6a 100644
--- a/include/drm/ttm/ttm_pool.h
+++ b/include/drm/ttm/ttm_pool.h
@@ -83,18 +83,27 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
 /**
  * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
- * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
- *         short); each entry is the head page of a 1 << @order chunk.
- * @order: Page order of every preallocated chunk.
+ * @pages: Array of @count preallocated chunks; each entry is the head page of a
+ *         chunk. In defrag mode every chunk is @order pages; in @full mode the
+ *         chunks tile a whole tt at mixed orders and the per-chunk order is read
+ *         back from the page itself.
+ * @order: Page order of every preallocated chunk (defrag mode only; 0 in @full
+ *         mode where each chunk carries its own order).
  * @caching: CPU caching applied to the pages, so leftovers can be restored
  *           to write-back before being freed.
  * @count: Number of valid entries in @pages.
  * @used: Number of entries already consumed by the pool allocator.
+ * @full: When set the bag holds a full, mixed-order tiling of an entire tt
+ *        (produced by ttm_pool_prealloc_fill_full()) rather than a bag of
+ *        interchangeable beneficial-order defrag chunks. The pool allocator
+ *        then drains the bag in order for every populate, taking each chunk's
+ *        order from the page, and falls back to in-line allocation only for any
+ *        shortfall.
  *
  * Defrag pages are interchangeable, so only a count of beneficial-order chunks
- * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
- * __ttm_pool_alloc() drains it; any unused tail is released by
- * ttm_pool_prealloc_fini().
+ * is needed. ttm_pool_prealloc_fill() / ttm_pool_prealloc_fill_full() populate
+ * this outside the lock and __ttm_pool_alloc() drains it; any unused tail is
+ * released by ttm_pool_prealloc_fini().
  */
 struct ttm_pool_prealloc {
 	struct page **pages;
@@ -102,10 +111,16 @@ struct ttm_pool_prealloc {
 	enum ttm_caching caching;
 	unsigned int count;
 	unsigned int used;
+	bool full;
 };
 
 int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 			   struct ttm_pool_prealloc *pp, unsigned int count);
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff);
 void ttm_pool_prealloc_fini(struct ttm_pool *pool,
 			    struct ttm_pool_prealloc *pp);
 unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
-- 
2.34.1

[PATCH v2 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

[Matthew Brost]

From: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Drivers that use the GPU SVM core to DMA-map system memory for GPU
access may want to keep track of how many pages they have mapped, for
example to expose statistics or to detect leaks. Doing this accounting
in the driver around drm_gpusvm_get_pages() / drm_gpusvm_unmap_pages()
is error prone: the number, order and kind of the dma_addr[] entries can
change between map and unmap (migration, partial unmaps, the iova vs
non-iova paths), which makes symmetric accounting hard to get right.

Add an optional @dma_map_account callback to struct drm_gpusvm_ops and
invoke it once per &drm_pagemap_addr entry at the exact points where the
entry is DMA-mapped (sign == +1) in drm_gpusvm_get_pages() and unmapped
(sign == -1) in __drm_gpusvm_unmap_pages(). Since the map error path and
every unmap/free path funnel through __drm_gpusvm_unmap_pages() for the
entries that were actually mapped, the accounting is symmetric by
construction. The callback receives the full &drm_pagemap_addr so the
driver can use the order and the proto field to distinguish system
memory mappings from device interconnect mappings.

The callback must also be usable by the core drm_gpusvm_pages-only API
(mm == NULL), as used e.g. for userptr-style mappings. Relax
drm_gpusvm_init() to allow such users to pass a restricted @ops that
only implements the page-level hooks; the full-SVM hooks (@invalidate)
and the chunk configuration must still be unset in that mode.

Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
 drivers/gpu/drm/drm_gpusvm.c | 17 +++++++++++++++--
 include/drm/drm_gpusvm.h     | 19 +++++++++++++++++++
 2 files changed, 34 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
index fcfe635bc195..512f52471419 100644
--- a/drivers/gpu/drm/drm_gpusvm.c
+++ b/drivers/gpu/drm/drm_gpusvm.c
@@ -474,8 +474,15 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
 			return -EINVAL;
 		mmgrab(mm);
 	} else {
-		/* No full SVM mode, only core drm_gpusvm_pages API. */
-		if (ops || num_chunks || mm_range || notifier_size)
+		/*
+		 * No full SVM mode, only core drm_gpusvm_pages API. A
+		 * restricted @ops that only implements the page-level hooks
+		 * (e.g. @dma_map_account) is allowed; the full-SVM hooks must
+		 * not be set.
+		 */
+		if (num_chunks || mm_range || notifier_size)
+			return -EINVAL;
+		if (ops && ops->invalidate)
 			return -EINVAL;
 	}
 
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
 			i += 1 << addr->order;
 		}
 
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h
index b7d987bf76aa..36f35816a569 100644
--- a/include/drm/drm_gpusvm.h
+++ b/include/drm/drm_gpusvm.h
@@ -75,6 +75,25 @@ struct drm_gpusvm_ops {
 	void (*invalidate)(struct drm_gpusvm *gpusvm,
 			   struct drm_gpusvm_notifier *notifier,
 			   const struct mmu_notifier_range *mmu_range);
+
+	/**
+	 * @dma_map_account: Account a DMA-mapping change (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @addr: The address descriptor of the chunk being (un)mapped
+	 * @sign: +1 when @addr has just been DMA-mapped, -1 when it is being
+	 *        unmapped
+	 *
+	 * Called once per &drm_pagemap_addr entry, when the entry is
+	 * DMA-mapped by drm_gpusvm_get_pages() (@sign == +1) and when it is
+	 * unmapped (@sign == -1), so the driver can keep symmetric accounting
+	 * of the pages it has mapped for GPU access. The @addr->proto field
+	 * can be used to distinguish system-memory mappings from device
+	 * interconnect mappings. May be provided in both full SVM mode and the
+	 * core drm_gpusvm_pages-only mode.
+	 */
+	void (*dma_map_account)(struct drm_gpusvm *gpusvm,
+				const struct drm_pagemap_addr *addr,
+				int sign);
 };
 
 /**
-- 
2.34.1

[PATCH v2 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

[Matthew Brost]

From: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Expose per-page-order DMA mapping counts for the system memory that the
xe driver maps for GPU access, split into two categories: TTM buffer
objects, and the GPU SVM / userptr ranges (which share a single
drm_gpusvm instance per VM).

The stats are visible at:

  <debugfs>/dri/<N>/dma_mapped_pages

and are broken out into two rows (bo, svm/userptr), one column per page
order, mirroring the layout of the TTM pool page_pool stat.

For TTM BOs the per-order counts are derived from the DMA scatter-gather
table: each mapped segment is physically contiguous, so its page count is
split into the largest power-of-two runs (a segment of N pages contributes
a 2^k chunk for the largest k with 2^k <= N, repeated for the remainder).
Driving the accounting off the sg table rather than the pages[] array keeps
it correct even when pages[] entries are cleared while still DMA-mapped,
as happens on the defrag page-borrowing path.

For the SVM and userptr ranges the accounting is driven by the
drm_gpusvm @dma_map_account callback, which fires once per dma_addr[]
entry at the exact point the entry is DMA-mapped and unmapped. This
makes the accounting symmetric by construction and avoids driver-side
walks that could drift across migration, partial unmaps and the iova vs
non-iova paths. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps
and counted; device interconnect (VRAM, P2P) entries are skipped, so the
counter reflects only system pages actually mapped through the DMA layer.

The callback is shared between the full SVM instance (fault mode) and the
core drm_gpusvm_pages-only instance used for userptr, so userptr
mappings are accounted in both the CONFIG_DRM_XE_GPUSVM=y and =n (but
CONFIG_DRM_GPUSVM=y) configurations, into the same svm/userptr counter.

Since the counts are only ever consumed through debugfs, gate the counter
storage and all accounting on CONFIG_DEBUG_FS so that builds without
debugfs carry no extra atomic traffic or device state; the drm_gpusvm
callback is simply left unregistered in that case.

On device teardown assert that every counter has returned to zero, to
catch any unbalanced accounting (a leaked DMA mapping) during testing.

Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c           | 46 +++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_debugfs.c      | 26 +++++++++++++
 drivers/gpu/drm/xe/xe_device.c       | 24 ++++++++++++
 drivers/gpu/drm/xe/xe_device_types.h | 17 +++++++++
 drivers/gpu/drm/xe/xe_svm.c          | 36 +++++++++++++++++-
 drivers/gpu/drm/xe/xe_svm.h          |  3 +-
 drivers/gpu/drm/xe/xe_userptr.c      | 55 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_userptr.h      |  1 +
 8 files changed, 205 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 85e6d9a0f575..60e0e568aa31 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -367,6 +367,50 @@ struct xe_ttm_tt {
 	bool purgeable;
 };
 
+/*
+ * xe_tt_account_dma_pages - account DMA-mapped tt pages per order
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt whose sg table to walk
+ * @sign: +1 to add, -1 to subtract
+ *
+ * Walk the tt's DMA scatter-gather table and split each (physically
+ * contiguous) segment into the largest power-of-two page runs: a segment of N
+ * pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for
+ * the remainder. Accumulate the count of 2^k pages into
+ * xe->mem.dma_mapped_pages[k]. Driving this off the sg table rather than
+ * tt->pages keeps the accounting correct even when pages[] entries are cleared
+ * while still DMA-mapped (e.g. the defrag page-borrowing path).
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_sg(xe_tt->sg, sg, i) {
+		unsigned long nr_pages = sg->length >> PAGE_SHIFT;
+
+		while (nr_pages) {
+			unsigned int order = min_t(unsigned int,
+						   __fls(nr_pages),
+						   MAX_PAGE_ORDER);
+			unsigned long chunk = 1UL << order;
+
+			atomic_long_add((long)chunk * sign,
+					&xe->mem.dma_mapped_pages[order]);
+
+			nr_pages -= chunk;
+		}
+	}
+}
+#else
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -396,6 +440,7 @@ static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 		return ret;
 	}
 
+	xe_tt_account_dma_pages(xe, xe_tt, 1);
 	return 0;
 }
 
@@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
+		xe_tt_account_dma_pages(xe, xe_tt, -1);
 		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);
diff --git a/drivers/gpu/drm/xe/xe_debugfs.c b/drivers/gpu/drm/xe/xe_debugfs.c
index 8c391c7b017a..97b47c5c4213 100644
--- a/drivers/gpu/drm/xe/xe_debugfs.c
+++ b/drivers/gpu/drm/xe/xe_debugfs.c
@@ -6,6 +6,7 @@
 #include "xe_debugfs.h"
 
 #include <linux/debugfs.h>
+#include <linux/mmzone.h>
 #include <linux/fault-inject.h>
 #include <linux/string_helpers.h>
 
@@ -231,6 +232,28 @@ static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
 	return 0;
 }
 
+static int dma_mapped_pages_show(struct seq_file *m, void *data)
+{
+	struct xe_device *xe = m->private;
+	unsigned int i;
+
+	seq_printf(m, "%-11s ", "");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " ---%2u---", i);
+	seq_printf(m, "\n%-11s:", "bo");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+	seq_printf(m, "\n%-11s:", "svm/userptr");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	seq_puts(m, "\n");
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(dma_mapped_pages);
+
 static const struct drm_info_list debugfs_list[] = {
 	{"info", info, 0},
 	{ .name = "sriov_info", .show = sriov_info, },
@@ -632,6 +655,9 @@ void xe_debugfs_register(struct xe_device *xe)
 	if (man)
 		ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
 
+	debugfs_create_file("dma_mapped_pages", 0444, root, xe,
+			    &dma_mapped_pages_fops);
+
 	for_each_tile(tile, xe, tile_id)
 		xe_tile_debugfs_register(tile);
 
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index ad7f3e61d457..0c4ee874a56d 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -455,10 +455,34 @@ bool xe_device_is_admin_only(const struct xe_device *xe)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+	unsigned int i;
+
+	/*
+	 * All BOs, userptr VMAs and SVM ranges must have been torn down by the
+	 * time the device is destroyed, so every DMA-mapped-pages counter must
+	 * have returned to zero. A non-zero value indicates unbalanced
+	 * accounting, i.e. a missing unmap-side decrement.
+	 */
+	for (i = 0; i < NR_PAGE_ORDERS; i++) {
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	}
+}
+#else
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+}
+#endif
+
 static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
+	xe_device_assert_dma_pages_zero(xe);
+
 	xe_bo_dev_fini(&xe->bo_device);
 
 	if (xe->preempt_fence_wq)
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 022e08205897..95e7cea91cbf 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -6,6 +6,7 @@
 #ifndef _XE_DEVICE_TYPES_H_
 #define _XE_DEVICE_TYPES_H_
 
+#include <linux/mmzone.h>
 #include <linux/pci.h>
 
 #include <drm/drm_device.h>
@@ -279,6 +280,22 @@ struct xe_device {
 		struct xe_shrinker *shrinker;
 		/** @mem.stolen_mgr: stolen memory manager. */
 		struct xe_ttm_stolen_mgr *stolen_mgr;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+		/**
+		 * @mem.dma_mapped_pages: number of DMA-mapped pages per page
+		 * order currently live for this device, for TTM BOs. Only
+		 * accounted when CONFIG_DEBUG_FS is enabled, since it is solely
+		 * exposed through debugfs.
+		 */
+		atomic_long_t dma_mapped_pages[NR_PAGE_ORDERS];
+		/**
+		 * @mem.dma_mapped_pages_svm: number of DMA-mapped pages per
+		 * page order currently live for this device, for SVM ranges
+		 * and userptr VMAs (both share a single drm_gpusvm instance).
+		 * Only accounted when CONFIG_DEBUG_FS is enabled.
+		 */
+		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
+#endif
 	} mem;
 
 	/** @sriov: device level virtualization data */
diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c
index b228a737cfd6..2f639fc0cac4 100644
--- a/drivers/gpu/drm/xe/xe_svm.c
+++ b/drivers/gpu/drm/xe/xe_svm.c
@@ -827,10 +827,42 @@ static int xe_svm_get_pagemaps(struct xe_vm *vm)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_svm_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Maintain per-order counts of the system-memory pages that the full SVM
+ * instance (SVM ranges, and userptr VMAs in fault mode) has DMA-mapped for GPU
+ * access. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_svm_dma_map_account(struct drm_gpusvm *gpusvm,
+				   const struct drm_pagemap_addr *addr,
+				   int sign)
+{
+	struct xe_device *xe = gpusvm_to_vm(gpusvm)->xe;
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	atomic_long_add((long)(1UL << order) * sign,
+			&xe->mem.dma_mapped_pages_svm[order]);
+}
+#endif
+
 static const struct drm_gpusvm_ops gpusvm_ops = {
 	.range_alloc = xe_svm_range_alloc,
 	.range_free = xe_svm_range_free,
 	.invalidate = xe_svm_invalidate,
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	.dma_map_account = xe_svm_dma_map_account,
+#endif
 };
 
 static const unsigned long fault_chunk_sizes[] = {
@@ -925,8 +957,8 @@ int xe_svm_init(struct xe_vm *vm)
 		}
 	} else {
 		err = drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-				      NULL, 0, 0, 0, NULL,
-				      NULL, 0);
+				      NULL, 0, 0, 0,
+				      xe_userptr_gpusvm_ops_get(), NULL, 0);
 	}
 
 	return err;
diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h
index a921556d3466..edd668c41551 100644
--- a/drivers/gpu/drm/xe/xe_svm.h
+++ b/drivers/gpu/drm/xe/xe_svm.h
@@ -236,7 +236,8 @@ int xe_svm_init(struct xe_vm *vm)
 {
 #if IS_ENABLED(CONFIG_DRM_GPUSVM)
 	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-			       NULL, 0, 0, 0, NULL, NULL, 0);
+			       NULL, 0, 0, 0, NULL, NULL,
+			       xe_userptr_gpusvm_ops_get(), 0);
 #else
 	return 0;
 #endif
diff --git a/drivers/gpu/drm/xe/xe_userptr.c b/drivers/gpu/drm/xe/xe_userptr.c
index 8b2d461ea0b2..79c2a81001f2 100644
--- a/drivers/gpu/drm/xe/xe_userptr.c
+++ b/drivers/gpu/drm/xe/xe_userptr.c
@@ -8,9 +8,64 @@
 
 #include <linux/mm.h>
 
+#include <drm/drm_pagemap.h>
+
 #include "xe_tlb_inval.h"
 #include "xe_trace_bo.h"
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_userptr_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Account, per page order, the system-memory pages DMA-mapped for GPU access
+ * through the drm_gpusvm_pages instance shared by userptr (and, in fault mode,
+ * SVM). Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_userptr_dma_map_account(struct drm_gpusvm *gpusvm,
+				       const struct drm_pagemap_addr *addr,
+				       int sign)
+{
+	struct xe_vm *vm = container_of(gpusvm, struct xe_vm, svm.gpusvm);
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	atomic_long_add((long)(1UL << order) * sign,
+			&vm->xe->mem.dma_mapped_pages_svm[order]);
+}
+
+static const struct drm_gpusvm_ops xe_userptr_gpusvm_ops = {
+	.dma_map_account = xe_userptr_dma_map_account,
+};
+#endif
+
+/**
+ * xe_userptr_gpusvm_ops_get() - Accounting ops for the simple gpusvm instance
+ *
+ * The core drm_gpusvm_pages-only instance used for userptr (and, in fault
+ * mode, shared with SVM) is initialised with these ops so that DMA-mapped
+ * system pages are accounted and exposed through debugfs. Returns NULL when
+ * CONFIG_DEBUG_FS is disabled, in which case no accounting is kept and the
+ * instance is initialised without ops.
+ *
+ * Return: Pointer to the restricted &drm_gpusvm_ops, or NULL.
+ */
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void)
+{
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	return &xe_userptr_gpusvm_ops;
+#else
+	return NULL;
+#endif
+}
+
 static void xe_userptr_assert_in_notifier(struct xe_vm *vm)
 {
 	lockdep_assert(lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 0) ||
diff --git a/drivers/gpu/drm/xe/xe_userptr.h b/drivers/gpu/drm/xe/xe_userptr.h
index 2a3cd1b5efbb..1e9601be5713 100644
--- a/drivers/gpu/drm/xe/xe_userptr.h
+++ b/drivers/gpu/drm/xe/xe_userptr.h
@@ -108,6 +108,7 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm);
 int xe_vm_userptr_check_repin(struct xe_vm *vm);
 int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma);
 int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma);
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void);
 #else
 static inline void xe_userptr_remove(struct xe_userptr_vma *uvma) {}
 
-- 
2.34.1

[PATCH v2 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

[Matthew Brost]

On L2-flush-optimized HW with a dma-resv (non-fault) VM, evicting a BO
only needs to flush the L2 cache before the migration copy runs; the
mappings themselves are torn down and rebuilt lazily via
drm_gpuvm_bo_evict() and a subsequent rebind. Today this flush is done
by waiting for the BO to go idle and then issuing a synchronous TLB
invalidation per mapping VMA from inside xe_bo_trigger_rebind(). Both
the idle wait and the synchronous invalidation stall the calling thread
while holding the BO dma-resv lock, serializing the move behind all
in-flight GPU work on the BO.

Replace this with an asynchronous flush. Add xe_vm_flush_vm_bo_tlb_async()
which, for each VMA mapping the BO on each present tile, queues a TLB
invalidation job on the tile migrate (kernel) exec queue. The jobs depend
on the BO's in-flight GPU work, captured once as a singleton over
DMA_RESV_USAGE_BOOKKEEP, so the flush only fires once the GPU is done with
the current mapping. Each job's completion fence is installed into the
BO's dma-resv as a DMA_RESV_USAGE_KERNEL fence, so the migration copy -
which waits on the resv - waits on the flush without stalling this thread.

No PTEs are zapped and vma->tile_invalidated is left untouched: the
mapping stays valid until the lazy rebind, and the only work performed
here is the L2 flush. On any failure the caller falls back to the
existing blocking wait-idle plus xe_vm_invalidate_vma() path.

Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Tejas Upadhyay <tejas.upadhyay@intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c |   7 +++
 drivers/gpu/drm/xe/xe_vm.c | 125 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h |   5 ++
 3 files changed, 137 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 60e0e568aa31..cca617cf34d1 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -732,6 +732,13 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_device_is_l2_flush_optimized(xe))
 				continue;
+
+			/*
+			 * Attempt to flush L2 async, fallback to sync flush on
+			 * failure
+			 */
+			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
+				continue;
 		}
 
 		if (!idle) {
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 73ac031ffb04..4557a8a4d270 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -39,6 +39,7 @@
 #include "xe_sync.h"
 #include "xe_tile.h"
 #include "xe_tlb_inval.h"
+#include "xe_tlb_inval_job.h"
 #include "xe_trace_bo.h"
 #include "xe_vm_madvise.h"
 #include "xe_wa.h"
@@ -4401,6 +4402,130 @@ int xe_vm_invalidate_vma(struct xe_vma *vma)
 	return ret;
 }
 
+/*
+ * xe_vma_tlb_flush_client - Queue an async TLB flush for one VMA on one client
+ *
+ * Create and push a TLB invalidation job on the tile migrate (kernel) exec
+ * queue covering @vma's range, depending on @dep (the BO's in-flight GPU work)
+ * so the flush only fires once the GPU is done with the current mapping. The
+ * job's completion fence is installed into @resv as a KERNEL fence so the
+ * subsequent migration waits on the flush. No PTEs are zapped; this only
+ * flushes L2 via the TLB invalidation.
+ */
+static int xe_vma_tlb_flush_client(struct xe_vm *vm, struct xe_vma *vma,
+				   struct xe_tile *tile, struct xe_gt *gt,
+				   struct dma_resv *resv, struct dma_fence *dep,
+				   int type)
+{
+	struct xe_exec_queue *q = xe_migrate_exec_queue(tile->migrate);
+	struct xe_tlb_inval_job *job;
+	struct dma_fence *fence;
+	int err;
+
+	job = xe_tlb_inval_job_create(q, &gt->tlb_inval,
+				      q->tlb_inval[type].dep_scheduler, vm,
+				      xe_vma_start(vma), xe_vma_end(vma), type);
+	if (IS_ERR(job))
+		return PTR_ERR(job);
+
+	err = xe_tlb_inval_job_alloc_dep(job);
+	if (err)
+		goto out_put;
+
+	err = dma_resv_reserve_fences(resv, 1);
+	if (err)
+		goto out_put;
+
+	/* Cannot fail; consumes a ref on @dep and returns a referenced fence. */
+	fence = xe_tlb_inval_job_push(job, tile->migrate, dep);
+	dma_resv_add_fence(resv, fence, DMA_RESV_USAGE_KERNEL);
+	dma_fence_put(fence);
+
+out_put:
+	/* Drop the creation reference (destroys the job if it was not pushed). */
+	xe_tlb_inval_job_put(job);
+	return err;
+}
+
+/**
+ * xe_vm_flush_vm_bo_tlb_async - Asynchronously flush TLBs for a vm_bo's mappings
+ * @vm: The VM @vm_bo belongs to
+ * @bo: The buffer object being moved
+ * @vm_bo: The gpuvm_bo linking @bo into @vm
+ *
+ * On L2-flush-optimized HW a BO move only needs to flush L2 (via a TLB
+ * invalidation) for the BO's live mappings; the mappings themselves are torn
+ * down and rebuilt lazily via the eviction/rebind path, so no PTEs need to be
+ * zapped here. Rather than blocking the caller on a synchronous invalidation,
+ * issue a TLB invalidation job per VMA per TLB-invalidation client (per present
+ * tile, primary and media GT). Each job waits on the BO's in-flight GPU work
+ * (all dma-resv usages) and its completion fence is installed into the BO's
+ * dma-resv KERNEL slots, so the following migration waits on the flush without
+ * stalling this thread.
+ *
+ * The caller must hold the BO's dma-resv lock and @vm must not be in fault
+ * mode.
+ *
+ * Return: 0 on success, negative error code on failure. On failure the caller
+ * should fall back to the blocking xe_vm_invalidate_vma() path; any jobs
+ * already queued install harmless extra flush fences.
+ */
+int xe_vm_flush_vm_bo_tlb_async(struct xe_vm *vm, struct xe_bo *bo,
+				struct drm_gpuvm_bo *vm_bo)
+{
+	struct xe_device *xe = vm->xe;
+	struct dma_resv *resv = bo->ttm.base.resv;
+	struct dma_fence *dep = NULL;
+	struct drm_gpuva *gpuva;
+	int err;
+
+	dma_resv_assert_held(resv);
+	xe_assert(xe, !xe_vm_in_fault_mode(vm));
+
+	/*
+	 * Single fence capturing all in-flight GPU work on the BO; the TLB
+	 * invalidation jobs depend on it so the flush fires only once the GPU
+	 * is done with the current mapping.
+	 */
+	err = dma_resv_get_singleton(resv, DMA_RESV_USAGE_BOOKKEEP, &dep);
+	if (err)
+		return err;
+	if (!dep)
+		dep = dma_fence_get_stub();
+
+	drm_gpuvm_bo_for_each_va(gpuva, vm_bo) {
+		struct xe_vma *vma = gpuva_to_vma(gpuva);
+		struct xe_tile *tile;
+		u8 id;
+
+		if (xe_vma_is_null(vma) || xe_vma_is_cpu_addr_mirror(vma))
+			continue;
+
+		for_each_tile(tile, xe, id) {
+			if (!(vma->tile_present & BIT(id)))
+				continue;
+
+			err = xe_vma_tlb_flush_client(vm, vma, tile,
+						      tile->primary_gt, resv, dep,
+						      XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT);
+			if (err)
+				goto out;
+
+			if (tile->media_gt) {
+				err = xe_vma_tlb_flush_client(vm, vma, tile,
+							      tile->media_gt, resv, dep,
+							      XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT);
+				if (err)
+					goto out;
+			}
+		}
+	}
+
+out:
+	dma_fence_put(dep);
+	return err;
+}
+
 int xe_vm_validate_protected(struct xe_vm *vm)
 {
 	struct drm_gpuva *gpuva;
diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
index c5b900f38ded..dd5b070eaede 100644
--- a/drivers/gpu/drm/xe/xe_vm.h
+++ b/drivers/gpu/drm/xe/xe_vm.h
@@ -26,6 +26,8 @@ struct ttm_buffer_object;
 
 struct dma_fence;
 
+struct xe_bo;
+struct drm_gpuvm_bo;
 struct xe_exec_queue;
 struct xe_file;
 struct xe_pagefault;
@@ -254,6 +256,9 @@ int xe_vm_invalidate_vma(struct xe_vma *vma);
 
 int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_batch *batch);
 
+int xe_vm_flush_vm_bo_tlb_async(struct xe_vm *vm, struct xe_bo *bo,
+				struct drm_gpuvm_bo *vm_bo);
+
 int xe_vm_validate_protected(struct xe_vm *vm);
 
 static inline void xe_vm_queue_rebind_worker(struct xe_vm *vm)
-- 
2.34.1

[PATCH v2 14/33] drm/xe: Destroy page tables after unlinking all VMAs on VM close

[Matthew Brost]

xe_vm_close_and_put() freed the VM's page-table roots via
xe_vm_pt_destroy() before destroying the "contested" VMAs - those backed
by external (shared) BOs, which are deferred to xe_vma_destroy_unlocked()
because unlinking them requires taking the external BO's dma-resv lock
(via drm_exec), not vm->lock.

Until such a VMA is unlinked it remains reachable from its BO through
drm_gem_for_each_gpuvm_bo(). A concurrent BO move that only holds the
BO's dma-resv lock - TTM eviction, the shrinker, or the defrag worker -
calls xe_bo_move_notify() -> xe_bo_trigger_rebind() ->
xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which dereferences
vm->pt_root[tile->id]. With the page tables already freed this is a
use-after-free / NULL dereference, e.g.:

  Oops: general protection fault [...] kernel NULL pointer dereference
  RIP: xe_pt_zap_ptes+0x7c [xe]
  Call Trace:
   xe_vm_invalidate_vma_submit+0x64/0x130 [xe]
   xe_vm_invalidate_vma+0x27/0x60 [xe]
   xe_bo_move_notify+0x1e7/0x3d0 [xe]
   xe_bo_move+0x62d/0x1620 [xe]
   ttm_bo_handle_move_mem+0x243/0x330 [ttm]
   ttm_bo_validate+0xb4/0x1a0 [ttm]

The race is not closable with a flag or a NULL pt_root check in the
invalidate path: the freeing side holds only vm->lock while the walker
holds only the BO's resv, so there is no common lock to serialise on.

Fix the ordering instead: destroy the page tables only after every VMA -
including the contested external-BO VMAs - has been unlinked. Once the
contested loop completes no BO can reach a VMA of this VM, so freeing the
page tables (under vm->lock, to satisfy xe_vm_assert_held()) is safe.
xe_vm_free_scratch() still runs before the free as it reads
pt_root[]->level.

Fixes: dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 4557a8a4d270..9f55d22cb985 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -1934,15 +1934,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		vma->gpuva.flags |= XE_VMA_DESTROYED;
 	}
 
-	/*
-	 * All vm operations will add shared fences to resv.
-	 * The only exception is eviction for a shared object,
-	 * but even so, the unbind when evicted would still
-	 * install a fence to resv. Hence it's safe to
-	 * destroy the pagetables immediately.
-	 */
 	xe_vm_free_scratch(vm);
-	xe_vm_pt_destroy(vm);
 	xe_vm_unlock(vm);
 
 	/*
@@ -1956,6 +1948,22 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		xe_vma_destroy_unlocked(vma);
 	}
 
+	/*
+	 * Destroy the page tables only after every VMA has been unlinked. The
+	 * contested (external-BO) VMAs above are unlinked from their BO under
+	 * the BO's dma-resv lock, not vm->lock, so they can still be reached
+	 * via drm_gem_for_each_gpuvm_bo() by a concurrent BO move (eviction,
+	 * shrinker or the defrag worker) that only holds the BO's resv. Such a
+	 * move calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which
+	 * dereferences vm->pt_root[]; freeing the page tables while those VMAs
+	 * are still linked is a use-after-free. All vm operations add shared
+	 * fences to the resv (eviction of a shared object still installs an
+	 * unbind fence), so no GPU work outlives this point.
+	 */
+	xe_vm_lock(vm, false);
+	xe_vm_pt_destroy(vm);
+	xe_vm_unlock(vm);
+
 	xe_svm_fini(vm);
 
 	up_write(&vm->lock);
-- 
2.34.1

[PATCH v2 15/33] drm/xe: Track BOs backed at a sub-optimal page order

[Matthew Brost]

Add per-device tracking of BOs whose backing TT pages were allocated
below the pool's beneficial order, as reported by the
TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (read via
ttm_tt_is_beneficial_order_failed()). These BOs are future candidates for
a defragmentation pass that reallocates their pages at the beneficial
order.

Add a mem.defrag sub-struct (lock, list, atomic count) to struct
xe_device and a defrag_link to struct xe_bo, initialised at BO create
and device init. Membership is updated from xe_bo_move() via
xe_bo_defrag_update(): a BO is added when it is a populated, needed,
non-pinned ttm_bo_type_device object flagged
TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED, and removed otherwise. BOs are also
removed on pin, on purge, moved to dontneed, and at destroy.

Only ttm_bo_type_device BOs are tracked.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c           | 115 +++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.h           |   1 +
 drivers/gpu/drm/xe/xe_bo_types.h     |   6 ++
 drivers/gpu/drm/xe/xe_device.c       |   2 +
 drivers/gpu/drm/xe/xe_device_types.h |  19 +++++
 5 files changed, 143 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index cca617cf34d1..d7b0536564ab 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -920,6 +920,9 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
 	}
 }
 
+static void xe_bo_defrag_update(struct xe_bo *bo);
+static void xe_bo_defrag_remove(struct xe_bo *bo);
+
 /**
  * xe_bo_set_purgeable_state() - Set BO purgeable state with validation
  * @bo: Buffer object
@@ -948,6 +951,7 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
 
 	bo->purgeable.state = new_state;
 	xe_bo_set_purgeable_shrinker(bo, new_state);
+	xe_bo_defrag_update(bo);
 }
 
 /**
@@ -997,9 +1001,100 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	/* Commit the state transition only once invalidation was queued */
 	xe_bo_set_purgeable_state(bo, XE_MADV_PURGEABLE_PURGED);
 
+	/*
+	 * The backing store is gone, so the BO no longer needs defrag. Drop it
+	 * from the defrag list here so the count stays accurate even for purge
+	 * paths that don't pass through xe_bo_move() (e.g. swap_notify).
+	 */
+	xe_bo_defrag_remove(bo);
+
 	return 0;
 }
 
+/**
+ * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
+ * @xe: The xe device
+ *
+ * Initialize the list, lock and count used to track BOs whose backing TT
+ * pages were allocated at a sub-optimal order.
+ */
+void xe_bo_defrag_init_early(struct xe_device *xe)
+{
+	spin_lock_init(&xe->mem.defrag.lock);
+	INIT_LIST_HEAD(&xe->mem.defrag.list);
+	atomic_set(&xe->mem.defrag.count, 0);
+}
+
+static bool xe_bo_needs_defrag(struct xe_bo *bo)
+{
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_tt *tt = ttm_bo->ttm;
+
+	xe_bo_assert_held(bo);
+
+	return ttm_bo->type == ttm_bo_type_device && tt &&
+		!xe_bo_madv_is_dontneed(bo) &&
+		ttm_tt_is_populated(tt) &&
+		ttm_tt_is_beneficial_order_failed(tt) &&
+		ttm_bo->resource && ttm_bo->resource->mem_type == XE_PL_TT &&
+		!xe_bo_is_pinned(bo);
+}
+
+static void xe_bo_defrag_add(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	xe_bo_assert_held(bo);
+	xe_assert(xe, xe_bo_needs_defrag(bo));
+
+	scoped_guard(spinlock, &xe->mem.defrag.lock) {
+		if (list_empty(&bo->defrag_link)) {
+			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
+			atomic_inc(&xe->mem.defrag.count);
+		}
+	}
+}
+
+static void __xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	guard(spinlock)(&xe->mem.defrag.lock);
+	if (!list_empty(&bo->defrag_link)) {
+		list_del_init(&bo->defrag_link);
+		atomic_dec(&xe->mem.defrag.count);
+	}
+}
+
+static void xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (list_empty(&bo->defrag_link))
+		return;
+
+	__xe_bo_defrag_remove(bo);
+}
+
+/**
+ * xe_bo_defrag_update() - Update defrag list membership for a BO
+ * @bo: The buffer object
+ *
+ * Add @bo to the device defrag list when it is a ttm_bo_type_device BO resident
+ * in XE_PL_TT with a populated TT whose pages were allocated at a sub-optimal
+ * order (ttm_tt_is_beneficial_order_failed()) and it isn't pinned. Otherwise
+ * ensure it is removed from the list.
+ */
+static void xe_bo_defrag_update(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (xe_bo_needs_defrag(bo))
+		xe_bo_defrag_add(bo);
+	else
+		xe_bo_defrag_remove(bo);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
@@ -1225,6 +1320,8 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		xe_tt_unmap_sg(xe, ttm_bo->ttm);
 	}
 
+	xe_bo_defrag_update(bo);
+
 	return ret;
 }
 
@@ -1261,6 +1358,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 	if (lret > 0) {
 		xe_ttm_tt_account_subtract(xe, bo->ttm);
 		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
+
+		/*
+		 * The pages are gone, so a tracked BO no longer needs defrag.
+		 * This shrinker purge path frees pages without going through
+		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
+		 * bos are never tracked, so guard for real xe bos.
+		 */
+		if (xe_bo_is_xe_bo(bo))
+			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
 	}
 
 	return lret;
@@ -1903,6 +2009,8 @@ static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo)
 		list_del(&bo->vram_userfault_link);
 	mutex_unlock(&xe->mem_access.vram_userfault.lock);
 
+	__xe_bo_defrag_remove(bo);
+
 	kfree(bo);
 }
 
@@ -2408,6 +2516,7 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 	bo->ttm.base.funcs = &xe_gem_object_funcs;
 	bo->ttm.priority = XE_BO_PRIORITY_NORMAL;
 	INIT_LIST_HEAD(&bo->pinned_link);
+	INIT_LIST_HEAD(&bo->defrag_link);
 #ifdef CONFIG_PROC_FS
 	INIT_LIST_HEAD(&bo->client_link);
 #endif
@@ -3014,6 +3123,8 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3072,6 +3183,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3107,6 +3220,7 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -3139,6 +3253,7 @@ void xe_bo_unpin(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 }
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 6340317f7d2e..4186cb51af87 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -224,6 +224,7 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
 int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
+void xe_bo_defrag_init_early(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h
index fcc63ae3f455..9eea9ab380e5 100644
--- a/drivers/gpu/drm/xe/xe_bo_types.h
+++ b/drivers/gpu/drm/xe/xe_bo_types.h
@@ -54,6 +54,12 @@ struct xe_bo {
 	struct ttm_bo_kmap_obj kmap;
 	/** @pinned_link: link to present / evicted list of pinned BO */
 	struct list_head pinned_link;
+	/**
+	 * @defrag_link: link into @xe_device.mem.defrag.list for BOs whose
+	 * backing TT pages were allocated at a sub-optimal order. Protected by
+	 * @xe_device.mem.defrag.lock.
+	 */
+	struct list_head defrag_link;
 #ifdef CONFIG_PROC_FS
 	/**
 	 * @client: @xe_drm_client which created the bo
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index 0c4ee874a56d..d4321419ee35 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -601,6 +601,8 @@ int xe_device_init_early(struct xe_device *xe)
 	if (err)
 		return err;
 
+	xe_bo_defrag_init_early(xe);
+
 	xe->preempt_fence_wq = alloc_ordered_workqueue("xe-preempt-fence-wq",
 						       WQ_MEM_RECLAIM);
 	xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 95e7cea91cbf..9e6244d8f41b 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -296,6 +296,25 @@ struct xe_device {
 		 */
 		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
 #endif
+		/**
+		 * @mem.defrag: Tracking of BOs whose backing TT pages were
+		 * allocated at a sub-optimal (smaller than beneficial) order.
+		 *
+		 * Such BOs are candidates for a future defragmentation pass that
+		 * tries to reallocate their pages at the device's beneficial
+		 * order. Only ttm_bo_type_device BOs are tracked.
+		 */
+		struct {
+			/** @mem.defrag.lock: Protects @mem.defrag.list. */
+			spinlock_t lock;
+			/** @mem.defrag.list: List of struct xe_bo defrag_link. */
+			struct list_head list;
+			/**
+			 * @mem.defrag.count: Number of BOs currently on
+			 * @mem.defrag.list.
+			 */
+			atomic_t count;
+		} defrag;
 	} mem;
 
 	/** @sriov: device level virtualization data */
-- 
2.34.1

[PATCH v2 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure

[Matthew Brost]

Once a meaningful number of BOs are already tracked as being backed at a
sub-optimal order, continuing to push hard for beneficial-order pages
during populate mostly wastes time in reclaim/compaction.

In xe_ttm_tt_populate(), set ctx->beneficial_reclaim_backoff when the
device defrag count reaches XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD (2)
so the TTM pool backs off and allocations make forward progress. The
deferred defrag pass is responsible for upgrading these objects to
beneficial order later.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index d7b0536564ab..21d087170ef3 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -41,6 +41,14 @@
 #include "xe_vm.h"
 #include "xe_vram_types.h"
 
+/*
+ * Once this many BOs are tracked on the device defrag list (i.e. were backed
+ * with a sub-optimal page order), request that the TTM pool backs off from
+ * aggressive reclaim at the beneficial order during populate, so that
+ * allocations make forward progress instead of stalling.
+ */
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",
@@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
 		err = ttm_tt_restore(ttm_dev, tt, ctx);
 	} else {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		if (atomic_read(&xe->mem.defrag.count) >=
+		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+			ctx->beneficial_reclaim_backoff = true;
+
 		ttm_tt_clear_backed_up(tt);
 		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	}
-- 
2.34.1

[PATCH v2 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

[Matthew Brost]

Add xe_migrate_copy_defrag(), a copy helper for the upcoming page
defragmentation path where a single BO has its backing reallocated at a
more beneficial page order and its contents must be relocated from the
old pages to the new ones entirely on the GPU.

The contents are relocated in up to two passes:

 - Data pass (always): copy the data pages verbatim. The pages are
   mapped without the compression PAT, so the (possibly compressed)
   bytes are copied as-is rather than being decompressed and
   recompressed, and no CCS aux copy is appended.

 - CCS pass (need_ccs): copy the CCS aux (compression) state. Both the
   old and new data pages are mapped with the compression PAT so the
   copy engine accesses the aux state indirectly, issuing one
   indirect -> indirect XY_CTRL_SURF_COPY_BLT; the data blit is skipped.

The two passes are issued separately because they require opposite
compression PAT settings on the data page mappings (cleared for the
verbatim data copy, set for the indirect CCS access), which a single
mapping cannot satisfy. Both passes run in order on the same migrate
queue and are captured by the returned fence.

The source pages are passed explicitly via an sg_table argument (the
old, still populated backing) since they no longer match the BO's
current backing. The boolean arguments to __xe_migrate_copy() are folded
into struct xe_migrate_copy_flags to keep the growing set of copy
variants readable.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_migrate.h |   8 ++
 2 files changed, 128 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 92d5e81ceac2..305c6e1942fb 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -858,13 +858,39 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
 	return flush_flags;
 }
 
+/**
+ * struct xe_migrate_copy_flags - Flags for __xe_migrate_copy()
+ * @copy_only_ccs: Only copy the CCS aux state, aliasing the destination data
+ * mapping to the source (used when the data pages are shared between src/dst).
+ * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
+ * compression PAT index for the data copy.
+ * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
+ * two distinct page sets (old @src_sg pages -> new pages). The data blit is
+ * skipped (the data was copied by the @defrag_copy pass) and both the source
+ * and destination data pages are mapped with the compression PAT so the aux
+ * state is copied indirect -> indirect.
+ * @defrag_copy: Defrag move pass that copies the data pages between two
+ * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
+ * aux state. The data pages are mapped without the compression PAT so the
+ * (possibly compressed) bytes are copied verbatim rather than being
+ * decompressed and recompressed, and no CCS aux copy is appended; the aux
+ * state is migrated separately by the @defrag_ccs pass. The two passes need
+ * opposite compression PAT settings, which is why they cannot be combined.
+ */
+struct xe_migrate_copy_flags {
+	u32 copy_only_ccs : 1;
+	u32 is_vram_resolve : 1;
+	u32 defrag_ccs : 1;
+	u32 defrag_copy : 1;
+};
+
 static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct xe_bo *src_bo,
 					   struct xe_bo *dst_bo,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
-					   bool copy_only_ccs,
-					   bool is_vram_resolve)
+					   struct sg_table *src_sg,
+					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
 	struct xe_device *xe = gt_to_xe(gt);
@@ -881,20 +907,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	bool src_is_vram = mem_type_is_vram(src->mem_type);
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 	bool type_device = src_bo->ttm.type == ttm_bo_type_device;
-	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
+	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
+		!flags.defrag_copy;
 	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
-		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
-	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
+		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
+		!flags.defrag_copy;
+	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
+		!flags.defrag_ccs && !flags.defrag_copy;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
 	 * Otherwise, only use the compression PAT index for device memory
 	 * when copying from VRAM to system memory.
 	 */
-	bool use_comp_pat = is_vram_resolve || (type_device &&
+	bool use_comp_pat = flags.is_vram_resolve || (type_device &&
 			    xe_device_has_flat_ccs(xe) &&
 			    GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram);
 
+	/*
+	 * The defrag CCS pass migrates the aux state from the old pages
+	 * (@src_sg) to the new pages with both sides accessed indirectly, so
+	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
+	 */
+
 	/* Copying CCS between two different BOs is not supported yet. */
 	if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
 		return ERR_PTR(-EINVAL);
@@ -903,7 +938,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
@@ -912,7 +947,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
+				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
 
@@ -942,7 +978,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
 					      &src_L0_ofs, &src_L0_pt, 0, 0,
 					      avail_pts);
-		if (copy_only_ccs) {
+		if (flags.copy_only_ccs) {
 			dst_L0_ofs = src_L0_ofs;
 		} else {
 			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
@@ -963,7 +999,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		}
 
 		/* Add copy commands size here */
-		batch_size += ((copy_only_ccs) ? 0 : emit_copy_cmd_len(xe)) +
+		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
+			       emit_copy_cmd_len(xe)) +
 			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
@@ -975,13 +1012,15 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		if (src_is_vram && xe_migrate_allow_identity(src_L0, &src_it))
 			xe_res_next(&src_it, src_L0);
 		else
-			emit_pte(m, bb, src_L0_pt, src_is_vram, copy_system_ccs || use_comp_pat,
+			emit_pte(m, bb, src_L0_pt, src_is_vram,
+				 copy_system_ccs || use_comp_pat || flags.defrag_ccs,
 				 &src_it, src_L0, src);
 
 		if (dst_is_vram && xe_migrate_allow_identity(src_L0, &dst_it))
 			xe_res_next(&dst_it, src_L0);
-		else if (!copy_only_ccs)
-			emit_pte(m, bb, dst_L0_pt, dst_is_vram, copy_system_ccs,
+		else if (!flags.copy_only_ccs)
+			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
+				 copy_system_ccs || flags.defrag_ccs,
 				 &dst_it, src_L0, dst);
 
 		if (copy_system_ccs)
@@ -990,7 +1029,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
 		update_idx = bb->len;
 
-		if (!copy_only_ccs)
+		if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
 		if (needs_ccs_emit)
@@ -1078,7 +1117,69 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs)
 {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, copy_only_ccs, false);
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .copy_only_ccs = copy_only_ccs,
+				 });
+}
+
+/**
+ * xe_migrate_copy_defrag() - Migrate a BO's backing to a reallocated page set.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
+ * @src: The source TTM resource (the old backing).
+ * @dst: The dst TTM resource (the freshly reallocated backing).
+ * @src_sg: Scatter-gather table of the source (old) backing pages.
+ * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
+ *
+ * Used by the defrag path where a single BO has its backing reallocated at a
+ * more beneficial page order. The contents are relocated from the old pages
+ * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
+ * in up to two passes:
+ *
+ *  - Data pass (always): copy the data pages verbatim. The pages are mapped
+ *    without the compression PAT, so the (possibly compressed) bytes are copied
+ *    as-is rather than being decompressed and recompressed, and no CCS aux copy
+ *    is appended.
+ *
+ *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
+ *    and new data pages are mapped with the compression PAT so the copy engine
+ *    accesses the aux state indirectly, issuing one indirect -> indirect
+ *    XY_CTRL_SURF_COPY_BLT; the data blit is skipped.
+ *
+ * The two passes are issued separately because they require opposite
+ * compression PAT settings on the data page mappings (cleared for the verbatim
+ * data copy, set for the indirect CCS access), which a single mapping cannot
+ * satisfy. Both passes run in order on the same migrate queue.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+					 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				   (struct xe_migrate_copy_flags) {
+				   .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
 }
 
 /**
@@ -1096,7 +1197,10 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res)
 {
-	return __xe_migrate_copy(m, bo, bo, res, res, false, true);
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .is_vram_resolve = true,
+				 });
 }
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
index 965c45889c72..e4f996942a6d 100644
--- a/drivers/gpu/drm/xe/xe_migrate.h
+++ b/drivers/gpu/drm/xe/xe_migrate.h
@@ -11,6 +11,7 @@
 struct dma_fence;
 struct drm_pagemap_addr;
 struct iosys_map;
+struct sg_table;
 struct ttm_resource;
 
 struct xe_bo;
@@ -132,6 +133,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs);
+
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res);
-- 
2.34.1

[PATCH v2 18/33] drm/xe: Handle defrag moves in xe_bo_move()

[Matthew Brost]

Teach xe_bo_move() to service a defrag move, where TTM reallocates a
BO's backing at the device's beneficial order and hands the old, still
populated tt to the move via the BO's defrag_old_tt.

A defrag move always has existing contents to copy, so force
move_lacks_source false and fall through the XE_PL_TT -> XE_PL_TT no-op
shortcut so the copy path runs. The relocation is performed entirely on
the GPU via xe_migrate_copy_defrag(), reading the old pages from the
stashed tt's sg table and writing the freshly reallocated backing.

Both copy passes run in order on the same migrate queue and are captured
by the returned fence, so the teardown is pipelined via
ttm_bo_move_accel_cleanup(): the old tt is handed to a ghost object and
only unpopulated and freed once the fence signals, so the move need not
block. The VF CCS attach/detach handling is extended to treat a defrag
move (ctx->defrag) like the XE_PL_SYSTEM <-> XE_PL_TT transitions it
mirrors.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 21d087170ef3..8e9fc8b3dffa 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1164,6 +1164,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	move_lacks_source = !old_mem || (handle_system_ccs ? (!bo->ccs_cleared) :
 					 (!mem_type_is_vram(old_mem_type) && !tt_has_data));
 
+	/*
+	 * A defrag move always copies the existing contents from the stashed
+	 * old tt into the freshly (re)allocated backing.
+	 */
+	if (ttm_bo->defrag_old_tt)
+		move_lacks_source = false;
+
 	needs_clear = (ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) ||
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
@@ -1200,9 +1207,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	/*
 	 * Failed multi-hop where the old_mem is still marked as
 	 * TTM_PL_FLAG_TEMPORARY, should just be a dummy move.
+	 *
+	 * For a defrag move the old and new tt differ, so fallthrough to the
+	 * copy path instead of treating it as a no-op.
 	 */
 	if (old_mem_type == XE_PL_TT &&
-	    new_mem->mem_type == XE_PL_TT) {
+	    new_mem->mem_type == XE_PL_TT && !ttm_bo->defrag_old_tt) {
 		ttm_bo_move_null(ttm_bo, new_mem);
 		goto out;
 	}
@@ -1273,6 +1283,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 			flags |= XE_MIGRATE_CLEAR_FLAG_CCS_DATA;
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
+	} else if (ttm_bo->defrag_old_tt) {
+		struct xe_ttm_tt *old_xe_tt =
+			container_of(ttm_bo->defrag_old_tt, struct xe_ttm_tt, ttm);
+
+		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
+					       old_xe_tt->sg, handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
@@ -1309,13 +1325,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	 * BBs from BO as it is no longer needed.
 	 */
 	if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT &&
-	    new_mem->mem_type == XE_PL_SYSTEM)
+	    (new_mem->mem_type == XE_PL_SYSTEM || ctx->defrag))
 		xe_sriov_vf_ccs_detach_bo(bo);
 
 	if (IS_VF_CCS_READY(xe) &&
 	    ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
-	     (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
-	    handle_system_ccs)
+	     ((old_mem_type == XE_PL_SYSTEM || ctx->defrag) &&
+	      new_mem->mem_type == XE_PL_TT)) && handle_system_ccs)
 		ret = xe_sriov_vf_ccs_attach_bo(bo);
 
 out:
-- 
2.34.1

[PATCH v2 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves

[Matthew Brost]

A defrag move may harvest already beneficial-order chunks from the old
tt, so the freshly reallocated backing borrows (references) some of the
same physical pages. For those pages the source and destination GPU
mappings alias the same memory, making the data blit a pointless
self-copy.

Pass the source (old) tt into __xe_migrate_copy() instead of just its sg
table, so the data pass can compare the src and dst tt page arrays at the
matching offset and only blit runs of pages that actually differ. A new
emit_copy_chunks() helper walks each L0 chunk, skips borrowed-page runs,
and coalesces contiguous to-copy pages into a single emit_copy(). It runs
twice per chunk: once with a NULL batch buffer to count the blits for
batch sizing, then again to emit them.

When an entire L0 chunk turns out to be borrowed, skip it outright: no
batch buffer is allocated and no GPU commands are issued, the cursors are
just advanced past the chunk. If every chunk of the move is borrowed the
copy is a complete no-op, so hand back a signaled stub fence.

The comparison is a cheap per-page pointer test (src page != dst page)
that coalesces contiguous to-copy pages into a single blit. PTEs are
still emitted for partially borrowed chunks, since that setup is cheap and
only the data copies are worth skipping. Only the defrag data pass is
affected; the separate CCS pass still migrates all aux state.

Since the Xe tt sg lives in the xe_ttm_tt subclass rather than the base
ttm_tt, add an xe_tt_sg() helper to fetch it from a struct ttm_tt.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c      |  22 +++++-
 drivers/gpu/drm/xe/xe_bo.h      |   2 +
 drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_migrate.h |   3 +-
 4 files changed, 142 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 8e9fc8b3dffa..3b867c486ea4 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -473,6 +473,22 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
 	return xe_tt->sg;
 }
 
+/**
+ * xe_tt_sg() - Return the DMA scatter-gather table backing a ttm_tt.
+ * @tt: The ttm_tt to query.
+ *
+ * Like xe_bo_sg(), but takes the ttm_tt directly. Used by the defrag copy
+ * path which needs the old (source) tt's sg table without a backing xe_bo.
+ *
+ * Return: The sg table, or NULL if the tt is not currently mapped.
+ */
+struct sg_table *xe_tt_sg(struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	return xe_tt->sg;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
@@ -1284,11 +1300,9 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
 	} else if (ttm_bo->defrag_old_tt) {
-		struct xe_ttm_tt *old_xe_tt =
-			container_of(ttm_bo->defrag_old_tt, struct xe_ttm_tt, ttm);
-
 		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
-					       old_xe_tt->sg, handle_system_ccs);
+					       ttm_bo->defrag_old_tt,
+					       handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 4186cb51af87..32211016a722 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -110,6 +110,7 @@ enum xe_madv_purgeable_state {
 };
 
 struct sg_table;
+struct ttm_tt;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -547,6 +548,7 @@ void xe_bo_dev_init(struct xe_bo_dev *bo_device);
 void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
 
 struct sg_table *xe_bo_sg(struct xe_bo *bo);
+struct sg_table *xe_tt_sg(struct ttm_tt *tt);
 
 /*
  * xe_sg_segment_size() - Provides upper limit for sg segment size.
diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 305c6e1942fb..c1e96f105648 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -811,6 +811,63 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
 		emit_xy_fast_copy(gt, bb, src_ofs, dst_ofs, size, pitch);
 }
 
+/*
+ * emit_copy_chunks() - Emit the data blits for one defrag L0 chunk, skipping
+ * borrowed pages.
+ * @gt: The GT.
+ * @bb: The batch buffer to emit into, or NULL to only count the blits needed.
+ * @src_L0_ofs: GPU offset of the source pages mapped for this L0 chunk.
+ * @dst_L0_ofs: GPU offset of the destination pages mapped for this L0 chunk.
+ * @src_L0: Byte size of the L0 chunk (page aligned).
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ *
+ * A defragmentation move may harvest already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) some of the same physical pages.
+ * For those pages the source and destination GPU mappings alias the same
+ * memory and the data blit is a pointless self-copy. Walk the L0 chunk and
+ * only blit runs of pages that actually differ (src page != dst page),
+ * coalescing contiguous to-copy pages into a single blit.
+ *
+ * Return: The number of blits emitted (or that would be emitted when @bb is
+ * NULL).
+ */
+static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
+			    u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
+			    struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
+			    pgoff_t start_page)
+{
+	u32 npages = src_L0 >> PAGE_SHIFT;
+	u32 i = 0, nblits = 0;
+
+	while (i < npages) {
+		u32 run = 0;
+
+		/* Skip a run of borrowed pages (src and dst alias). */
+		if (src_tt->pages[start_page + i] ==
+		    dst_tt->pages[start_page + i]) {
+			i++;
+			continue;
+		}
+
+		/* Coalesce a run of pages that actually need copying. */
+		while (i + run < npages &&
+		       src_tt->pages[start_page + i + run] !=
+		       dst_tt->pages[start_page + i + run])
+			run++;
+
+		if (bb)
+			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
+				  dst_L0_ofs + (u64)i * PAGE_SIZE,
+				  run * PAGE_SIZE, PAGE_SIZE);
+		nblits++;
+		i += run;
+	}
+
+	return nblits;
+}
+
 static u64 xe_migrate_batch_base(struct xe_migrate *m, bool usm)
 {
 	return usm ? m->usm_batch_base_ofs : m->batch_base_ofs;
@@ -865,13 +922,15 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
  * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
  * compression PAT index for the data copy.
  * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
- * two distinct page sets (old @src_sg pages -> new pages). The data blit is
+ * two distinct page sets (old @src_tt pages -> new pages). The data blit is
  * skipped (the data was copied by the @defrag_copy pass) and both the source
  * and destination data pages are mapped with the compression PAT so the aux
  * state is copied indirect -> indirect.
  * @defrag_copy: Defrag move pass that copies the data pages between two
- * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
- * aux state. The data pages are mapped without the compression PAT so the
+ * distinct page sets (old @src_tt pages -> new pages) without touching the CCS
+ * aux state. Pages the new tt borrowed from @src_tt (harvested already
+ * beneficial-order chunks) alias the same memory on both sides and are skipped.
+ * The data pages are mapped without the compression PAT so the
  * (possibly compressed) bytes are copied verbatim rather than being
  * decompressed and recompressed, and no CCS aux copy is appended; the aux
  * state is migrated separately by the @defrag_ccs pass. The two passes need
@@ -889,7 +948,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct xe_bo *dst_bo,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
-					   struct sg_table *src_sg,
+					   struct ttm_tt *src_tt,
 					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
@@ -914,6 +973,14 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		!flags.defrag_copy;
 	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
 		!flags.defrag_ccs && !flags.defrag_copy;
+	/*
+	 * On the defrag data pass the new (destination) tt may have borrowed
+	 * already beneficial-order pages from the old (source) tt, so skip the
+	 * self-copy for any page they share. Compare the tt page arrays at the
+	 * matching offset.
+	 */
+	struct ttm_tt *dst_tt = dst_bo->ttm.ttm;
+	bool skip_borrowed = flags.defrag_copy && src_tt && dst_tt;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
@@ -926,7 +993,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 
 	/*
 	 * The defrag CCS pass migrates the aux state from the old pages
-	 * (@src_sg) to the new pages with both sides accessed indirectly, so
+	 * (@src_tt) to the new pages with both sides accessed indirectly, so
 	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
 	 */
 
@@ -938,7 +1005,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo), 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
@@ -947,7 +1014,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
+		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo),
 				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
@@ -961,6 +1028,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		u64 ccs_ofs, ccs_size;
 		u32 ccs_pt;
 		u32 pte_flags;
+		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> XE_PTE_SHIFT;
+		u32 ndata_blits = 1;
 
 		bool usm = xe->info.has_usm;
 		u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
@@ -998,9 +1067,30 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 			xe_assert(xe, IS_ALIGNED(ccs_it.start, PAGE_SIZE));
 		}
 
-		/* Add copy commands size here */
+		/*
+		 * Add copy commands size here. On the defrag data pass the blit
+		 * is split into one command per run of non-borrowed pages, so
+		 * size for the number of runs actually needed.
+		 */
+		if (skip_borrowed)
+			ndata_blits = emit_copy_chunks(gt, NULL, 0, 0, src_L0,
+						       src_tt, dst_tt, start_page);
+
+		/*
+		 * If every page in this chunk was borrowed there is nothing to
+		 * copy: don't build a batch or issue any GPU commands, just
+		 * advance the cursors past the chunk. The non-vram tt cursors
+		 * would otherwise be advanced by emit_pte().
+		 */
+		if (skip_borrowed && !ndata_blits) {
+			xe_res_next(&src_it, src_L0);
+			xe_res_next(&dst_it, src_L0);
+			size -= src_L0;
+			continue;
+		}
+
 		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
-			       emit_copy_cmd_len(xe)) +
+			       ndata_blits * emit_copy_cmd_len(xe)) +
 			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
@@ -1029,7 +1119,10 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
 		update_idx = bb->len;
 
-		if (!flags.copy_only_ccs && !flags.defrag_ccs)
+		if (skip_borrowed)
+			emit_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0,
+					 src_tt, dst_tt, start_page);
+		else if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
 		if (needs_ccs_emit)
@@ -1088,6 +1181,14 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(err);
 	}
 
+	/*
+	 * Every chunk of a defrag data pass may have been borrowed (nothing
+	 * copied), leaving no job and a NULL fence. Hand back a signaled stub
+	 * so callers always get a valid fence to wait on.
+	 */
+	if (!fence)
+		return dma_fence_get_stub();
+
 	return fence;
 }
 
@@ -1129,18 +1230,21 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
  * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
  * @src: The source TTM resource (the old backing).
  * @dst: The dst TTM resource (the freshly reallocated backing).
- * @src_sg: Scatter-gather table of the source (old) backing pages.
+ * @src_tt: The source (old) backing tt, providing both the scatter-gather
+ * table and page pointers of the old pages.
  * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
  *
  * Used by the defrag path where a single BO has its backing reallocated at a
  * more beneficial page order. The contents are relocated from the old pages
- * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
+ * (@src_tt) to the new pages (the BO's current backing) entirely on the GPU,
  * in up to two passes:
  *
  *  - Data pass (always): copy the data pages verbatim. The pages are mapped
  *    without the compression PAT, so the (possibly compressed) bytes are copied
  *    as-is rather than being decompressed and recompressed, and no CCS aux copy
- *    is appended.
+ *    is appended. Pages the new backing borrowed from @src_tt (harvested
+ *    already beneficial-order chunks) alias the same memory on both sides and
+ *    are skipped.
  *
  *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
  *    and new data pages are mapped with the compression PAT so the copy engine
@@ -1159,11 +1263,11 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct xe_bo *bo,
 					 struct ttm_resource *src,
 					 struct ttm_resource *dst,
-					 struct sg_table *src_sg,
+					 struct ttm_tt *src_tt,
 					 bool need_ccs)
 {
 	struct dma_fence *fence2, *fence =
-		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt,
 				  (struct xe_migrate_copy_flags) {
 					 .defrag_copy = true,
 					 });
@@ -1171,7 +1275,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	if (IS_ERR(fence) || !need_ccs)
 		return fence;
 
-	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
 				   (struct xe_migrate_copy_flags) {
 				   .defrag_ccs = true,
 				   });
diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
index e4f996942a6d..4094ea1c19c1 100644
--- a/drivers/gpu/drm/xe/xe_migrate.h
+++ b/drivers/gpu/drm/xe/xe_migrate.h
@@ -13,6 +13,7 @@ struct drm_pagemap_addr;
 struct iosys_map;
 struct sg_table;
 struct ttm_resource;
+struct ttm_tt;
 
 struct xe_bo;
 struct xe_gt;
@@ -137,7 +138,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct xe_bo *bo,
 					 struct ttm_resource *src,
 					 struct ttm_resource *dst,
-					 struct sg_table *src_sg,
+					 struct ttm_tt *src_tt,
 					 bool need_ccs);
 
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
-- 
2.34.1

[PATCH v2 20/33] drm/xe: Add a page defragmentation worker

[Matthew Brost]

BOs whose backing TT pages were allocated below the device's beneficial
order (tracked via the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag)
are added to a per-device defrag list. Add a delayed worker that
periodically walks this list and attempts to re-back each BO with
beneficial-order pages, improving TLB efficiency for long-lived objects
that were initially allocated under memory pressure.

The worker is kicked when the list first transitions to non-empty and
reschedules itself while work remains. On a failed pass it backs off
exponentially (XE_BO_DEFRAG_INTERVAL_MS up to
XE_BO_DEFRAG_INTERVAL_MAX_MS) and retries the whole list later; the
interval is reset to the default on the next fresh enqueue. The worker
stops scheduling once the list drains.

A defrag move synchronously reallocates and re-copies a BO's backing
store, which is not free. To avoid stalling concurrent active work when
many BOs (or one very large BO) become eligible at once (e.g. after a
burst of memory pressure, which would otherwise manifest as a large FPS
drop while the worker churns through the whole list), cap the number of
newly (re)allocated bytes processed per run to XE_BO_DEFRAG_SIZE_LIMIT
and reschedule. Only the pages a move actually reallocates are charged -
pages harvested (borrowed) from the old backing are free and do not count
- so the budget tracks real defrag work and an object larger than the
budget is upgraded in budget-sized slices across runs. This spreads the
defrag effort over time, keeping it in the background and yielding to
userspace progress.

Defragmenting a BO forces a TTM move (ctx.defrag) that reallocates the
backing at the beneficial order and relocates the contents on the GPU
(see xe_bo_move()). The per-BO move is bounded by the run's remaining
byte budget via ctx.defrag_bytes_remaining; the pool allocator reports
how much it spent so the worker can charge it accurately. When
beneficial-order pages cannot be obtained the new allocation is
discharged during populate and the BO keeps its original backing, staying
on the list for a later retry.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c           | 277 ++++++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h           |   1 +
 drivers/gpu/drm/xe/xe_device.c       |   9 +
 drivers/gpu/drm/xe/xe_device_types.h |  11 ++
 4 files changed, 294 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 3b867c486ea4..6a847e42946c 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -31,6 +31,7 @@
 #include "xe_pat.h"
 #include "xe_pm.h"
 #include "xe_preempt_fence.h"
+#include "xe_printk.h"
 #include "xe_pxp.h"
 #include "xe_res_cursor.h"
 #include "xe_shrinker.h"
@@ -49,6 +50,35 @@
  */
 #define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
 
+/*
+ * Maximum number of bytes of newly (re)allocated backing the defrag worker will
+ * produce in a single run before yielding and rescheduling itself.
+ *
+ * A defrag move synchronously reallocates and re-copies a BO's backing store,
+ * which is not free. If a large number of BOs (or one very large BO) become
+ * eligible at once (e.g. after a burst of memory pressure), processing them all
+ * in one worker run would hold things up for a long, unbounded stretch and can
+ * visibly starve concurrent active work (e.g. a large FPS drop). Instead, cap
+ * the bytes actually upgraded per run and requeue, spreading the defrag effort
+ * out over time so it stays in the background and yields to userspace progress.
+ * Only pages a move truly reallocates are charged; pages harvested from the old
+ * backing are free, so an object larger than the budget is upgraded in
+ * budget-sized slices across runs.
+ */
+#define XE_BO_DEFRAG_SIZE_LIMIT			SZ_32M
+
+/* Default delay before (re)running the defrag worker, in milliseconds. */
+#define XE_BO_DEFRAG_INTERVAL_MS		25
+
+/*
+ * Upper bound for the (exponentially backed off) defrag worker interval, in
+ * milliseconds, so repeated failures don't push the retry arbitrarily far out.
+ */
+#define XE_BO_DEFRAG_INTERVAL_MAX_MS		15000	/* 15 seconds */
+
+static void xe_bo_defrag_worker(struct work_struct *w);
+static void xe_place_from_ttm_type(u32 mem_type, struct ttm_place *place);
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",
@@ -624,8 +654,10 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	} else {
 		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
 
-		if (atomic_read(&xe->mem.defrag.count) >=
-		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+		if (ctx->defrag)
+			ctx->beneficial_reclaim_backoff = false;
+		else if (atomic_read(&xe->mem.defrag.count) >=
+			 XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
 			ctx->beneficial_reclaim_backoff = true;
 
 		ttm_tt_clear_backed_up(tt);
@@ -1041,18 +1073,54 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	return 0;
 }
 
+static void xe_bo_defrag_fini(void *arg)
+{
+	struct xe_device *xe = arg;
+
+	disable_delayed_work_sync(&xe->mem.defrag.worker);
+}
+
 /**
  * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
  * @xe: The xe device
  *
- * Initialize the list, lock and count used to track BOs whose backing TT
- * pages were allocated at a sub-optimal order.
+ * Initialize the list, lock, count and delayed worker used to track and
+ * defragment BOs whose backing TT pages were allocated at a sub-optimal order.
+ *
+ * The devm action that cancels the worker on teardown is registered separately
+ * by xe_bo_defrag_init(), which must run after the migrate contexts the
+ * worker depends on have been initialized.
  */
 void xe_bo_defrag_init_early(struct xe_device *xe)
 {
 	spin_lock_init(&xe->mem.defrag.lock);
 	INIT_LIST_HEAD(&xe->mem.defrag.list);
 	atomic_set(&xe->mem.defrag.count, 0);
+	xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+	INIT_DELAYED_WORK(&xe->mem.defrag.worker, xe_bo_defrag_worker);
+}
+
+/**
+ * xe_bo_defrag_init() - Register defrag worker teardown
+ * @xe: The xe device
+ *
+ * Register the devm action that disables and drains the defrag worker on
+ * device teardown. This must be called after xe_migrate_init() so that, with
+ * devm's reverse-order cleanup, the worker is stopped before the migrate
+ * contexts it relies on (via xe_bo_defrag_one() -> ttm_bo_validate() ->
+ * xe_bo_move() -> xe_migrate_copy_defrag()) are torn down.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int xe_bo_defrag_init(struct xe_device *xe)
+{
+	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
+}
+
+static void xe_bo_defrag_schedule(struct xe_device *xe)
+{
+	schedule_delayed_work(&xe->mem.defrag.worker,
+			      msecs_to_jiffies(xe->mem.defrag.interval_ms));
 }
 
 static bool xe_bo_needs_defrag(struct xe_bo *bo)
@@ -1073,16 +1141,24 @@ static bool xe_bo_needs_defrag(struct xe_bo *bo)
 static void xe_bo_defrag_add(struct xe_bo *bo)
 {
 	struct xe_device *xe = xe_bo_device(bo);
+	bool kick = false;
 
 	xe_bo_assert_held(bo);
 	xe_assert(xe, xe_bo_needs_defrag(bo));
 
 	scoped_guard(spinlock, &xe->mem.defrag.lock) {
 		if (list_empty(&bo->defrag_link)) {
+			/* Kick the worker when the list transitions to non-empty. */
+			kick = list_empty(&xe->mem.defrag.list);
+			if (kick)
+				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
 			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
 			atomic_inc(&xe->mem.defrag.count);
 		}
 	}
+
+	if (kick)
+		xe_bo_defrag_schedule(xe);
 }
 
 static void __xe_bo_defrag_remove(struct xe_bo *bo)
@@ -1125,6 +1201,199 @@ static void xe_bo_defrag_update(struct xe_bo *bo)
 		xe_bo_defrag_remove(bo);
 }
 
+/*
+ * Attempt to defragment a single BO by forcing a move that reallocates its
+ * backing at the device's beneficial order. @budget bounds how many bytes of
+ * new pages the move may (re)allocate; the remainder of the object is completed
+ * by harvesting the old backing, and the BO stays flagged for a later pass. On
+ * return *@consumed holds the bytes actually (re)allocated (0 on failure or
+ * when nothing needed reallocating). Returns 0 if the BO no longer needs to be
+ * tracked (either defragmented or no longer eligible), or a negative error code
+ * if the attempt should be retried later.
+ */
+static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
+			    u64 budget, u64 *consumed)
+{
+	struct ttm_operation_ctx ctx = {
+		.gfp_retry_mayfail = true,
+		.defrag = true,
+		.defrag_bytes_remaining = budget,
+	};
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_placement placement;
+	struct ttm_place place;
+	int ret = 0;
+
+	*consumed = 0;
+
+	xe_bo_lock(bo, false);
+
+	/* Re-check eligibility under the BO lock. */
+	if (!xe_bo_needs_defrag(bo)) {
+		xe_bo_defrag_remove(bo);
+		goto unlock;
+	}
+
+	xe_place_from_ttm_type(ttm_bo->resource->mem_type, &place);
+	place.flags |= TTM_PL_FLAG_TEMPORARY;
+	placement.num_placement = 1;
+	placement.placement = &place;
+
+	/*
+	 * On success the move reallocates the backing at beneficial order and
+	 * drops the BO from the defrag list. On failure the BO keeps its
+	 * original backing and stays on the list for a later retry.
+	 */
+	ret = ttm_bo_validate(ttm_bo, &placement, &ctx);
+
+	/*
+	 * The pool allocator decremented defrag_bytes_remaining as it
+	 * (re)allocated pages, so the unused budget tells us how much was spent.
+	 * Only meaningful when the move succeeded. A successful move may spend
+	 * nothing - e.g. when the object was already fully upgraded and every
+	 * chunk is harvested (borrowed) from the old backing for free - so a
+	 * zero spend is legitimate and must be charged as zero, not the whole
+	 * budget.
+	 */
+	if (!ret) {
+		s64 spent = (s64)budget - ctx.defrag_bytes_remaining;
+
+		*consumed = spent > 0 ? spent : 0;
+	}
+
+	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
+	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
+
+unlock:
+	xe_bo_unlock(bo);
+	return ret;
+}
+
+static void xe_bo_defrag_worker(struct work_struct *w)
+{
+	struct delayed_work *dwork = to_delayed_work(w);
+	struct xe_device *xe =
+		container_of(dwork, struct xe_device, mem.defrag.worker);
+	u64 defrag_bytes = 0;
+	bool requeue = false;
+	int idx;
+
+	if (!drm_dev_enter(&xe->drm, &idx))
+		return;
+
+	if (!xe_pm_runtime_get_if_in_use(xe)) {
+		drm_dev_exit(idx);
+		return;
+	}
+
+	/*
+	 * Process at most XE_BO_DEFRAG_SIZE_LIMIT bytes of newly (re)allocated
+	 * backing per run rather than draining the whole list in one go. Only
+	 * the pages a move actually reallocates are charged - pages harvested
+	 * (borrowed) from the old backing are free and do not count - so the
+	 * budget tracks real defrag work. A single object larger than the
+	 * budget is upgraded in slices across successive runs. Each defrag is
+	 * synchronous and relatively expensive, so bounding the work per run
+	 * keeps the worker from monopolising resources and lets concurrent
+	 * active work make progress; any remaining work is handled by a
+	 * follow-up run scheduled below.
+	 */
+	while (defrag_bytes < XE_BO_DEFRAG_SIZE_LIMIT) {
+		struct xe_bo *first, *bo;
+		u64 consumed = 0;
+		int ret;
+
+		scoped_guard(spinlock, &xe->mem.defrag.lock) {
+			bool again;
+
+			/* Progress, reset interval */
+			if (defrag_bytes)
+				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+
+			do {
+				first = list_first_entry_or_null(&xe->mem.defrag.list,
+								 struct xe_bo,
+								 defrag_link);
+				bo = first ? xe_bo_get_unless_zero(first) : NULL;
+				again = first && !bo;
+				if (again) {
+					list_del_init(&first->defrag_link);
+					atomic_dec(&xe->mem.defrag.count);
+				}
+			} while (again);
+		}
+
+		if (!bo)
+			break;
+
+		/*
+		 * Bound the per-BO move to the run's remaining byte budget. A
+		 * large object is upgraded in budget-sized slices across runs;
+		 * the slices already upgraded come back as free already-
+		 * beneficial harvests, so each run only spends budget on new
+		 * forward progress.
+		 */
+		ret = xe_bo_defrag_one(xe, bo, XE_BO_DEFRAG_SIZE_LIMIT - defrag_bytes,
+				       &consumed);
+		defrag_bytes += consumed;
+
+		if (ret || ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {
+			scoped_guard(spinlock, &xe->mem.defrag.lock) {
+				if (ret)
+					/*
+					 * Abort the pass and retry the whole
+					 * list later, backing off exponentially
+					 * on every failure.
+					 */
+					xe->mem.defrag.interval_ms =
+						min(xe->mem.defrag.interval_ms * 2,
+						    (unsigned int)XE_BO_DEFRAG_INTERVAL_MAX_MS);
+				else
+					/* Progress, reset interval */
+					xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+			}
+
+			xe_bo_put(bo);
+			requeue = true;
+			break;
+		}
+
+		xe_bo_put(bo);
+	}
+
+	/*
+	 * Decide whether to reschedule:
+	 *
+	 *  - The loop hit its per-run limit (defrag_bytes >=
+	 *    XE_BO_DEFRAG_SIZE_LIMIT): we stopped early to spread the work out,
+	 *    so requeue at the default interval (reset here since this run made
+	 *    progress) whenever the list still has entries.
+	 *
+	 *  - The loop aborted on a failed defrag (requeue): retry later at the
+	 *    backed-off interval computed above.
+	 *
+	 *  - The loop drained the list (broke on the empty list): nothing left to
+	 *    do, so stop scheduling. A fresh enqueue will kick the worker again.
+	 */
+	if (defrag_bytes >= XE_BO_DEFRAG_SIZE_LIMIT) {
+		struct xe_bo *bo;
+
+		scoped_guard(spinlock, &xe->mem.defrag.lock) {
+			bo = list_first_entry_or_null(&xe->mem.defrag.list,
+						      struct xe_bo, defrag_link);
+			xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+		}
+
+		if (bo)
+			xe_bo_defrag_schedule(xe);
+	} else if (requeue) {
+		xe_bo_defrag_schedule(xe);
+	}
+
+	xe_pm_runtime_put(xe);
+	drm_dev_exit(idx);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 32211016a722..38d50f8122cb 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -226,6 +226,7 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
 void xe_bo_defrag_init_early(struct xe_device *xe);
+int xe_bo_defrag_init(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index d4321419ee35..df5bb94f3ecb 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -1063,6 +1063,15 @@ int xe_device_probe(struct xe_device *xe)
 			return err;
 	}
 
+	/*
+	 * Register the defrag worker teardown now that the migrate contexts it
+	 * depends on are initialized, so devm's reverse-order cleanup stops the
+	 * worker before those contexts are torn down.
+	 */
+	err = xe_bo_defrag_init(xe);
+	if (err)
+		return err;
+
 	err = xe_pagefault_init(xe);
 	if (err)
 		return err;
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 9e6244d8f41b..6e68add4d652 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -314,6 +314,17 @@ struct xe_device {
 			 * @mem.defrag.list.
 			 */
 			atomic_t count;
+			/**
+			 * @mem.defrag.worker: Delayed worker that walks
+			 * @mem.defrag.list trying to reallocate BO backing
+			 * store at the device's beneficial order.
+			 */
+			struct delayed_work worker;
+			/**
+			 * @mem.defrag.interval_ms: Reschedule interval for
+			 * @mem.defrag.worker, in milliseconds.
+			 */
+			unsigned int interval_ms;
 		} defrag;
 	} mem;
 
-- 
2.34.1

[PATCH v2 21/33] drm/xe: Add defrag GT stats

[Matthew Brost]

Add observability for the page defragmentation worker.

Introduce four primary-GT statistics counters: defrag_added_count
(BOs queued onto the defrag list after being backed at a sub-optimal
page order), defrag_success_count (BOs fully re-backed at the beneficial
order), defrag_partial_success_count (BOs only partially re-backed at the
beneficial order) defrag_failed_count (attempts that failed and left the
BO on its original backing) and defrag_mb_moved. The
ineligible/already-removed fast path is intentionally not counted as
either success or failure.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c             | 17 +++++++++++++++++
 drivers/gpu/drm/xe/xe_gt_stats.c       |  5 +++++
 drivers/gpu/drm/xe/xe_gt_stats_types.h | 17 +++++++++++++++++
 3 files changed, 39 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 6a847e42946c..c1bbcf0d21ed 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -26,6 +26,7 @@
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
 #include "xe_ggtt.h"
+#include "xe_gt_stats.h"
 #include "xe_map.h"
 #include "xe_migrate.h"
 #include "xe_pat.h"
@@ -1154,6 +1155,8 @@ static void xe_bo_defrag_add(struct xe_bo *bo)
 				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
 			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
 			atomic_inc(&xe->mem.defrag.count);
+			xe_gt_stats_incr(xe_root_mmio_gt(xe),
+					 XE_GT_STATS_ID_DEFRAG_ADDED_COUNT, 1);
 		}
 	}
 
@@ -1264,6 +1267,20 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
 	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
 
+	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm))
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
+				 1);
+	else
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 ret ? XE_GT_STATS_ID_DEFRAG_FAILED_COUNT :
+				       XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT, 1);
+
+	if (!ret)
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_MB_MOVED,
+				 *consumed >> 20);
+
 unlock:
 	xe_bo_unlock(bo);
 	return ret;
diff --git a/drivers/gpu/drm/xe/xe_gt_stats.c b/drivers/gpu/drm/xe/xe_gt_stats.c
index 789397514f3e..68e60d9500c8 100644
--- a/drivers/gpu/drm/xe/xe_gt_stats.c
+++ b/drivers/gpu/drm/xe/xe_gt_stats.c
@@ -154,6 +154,11 @@ static const char *const stat_description[__XE_GT_STATS_NUM_IDS] = {
 	DEF_STAT_STR(PRL_2M_ENTRY_COUNT, "prl_2m_entry_count"),
 	DEF_STAT_STR(PRL_ISSUED_COUNT, "prl_issued_count"),
 	DEF_STAT_STR(PRL_ABORTED_COUNT, "prl_aborted_count"),
+	DEF_STAT_STR(DEFRAG_ADDED_COUNT, "defrag_added_count"),
+	DEF_STAT_STR(DEFRAG_SUCCESS_COUNT, "defrag_success_count"),
+	DEF_STAT_STR(DEFRAG_PARTIAL_SUCCESS_COUNT, "defrag_partial_success_count"),
+	DEF_STAT_STR(DEFRAG_FAILED_COUNT, "defrag_failed_count"),
+	DEF_STAT_STR(DEFRAG_MB_MOVED, "defrag_mb_moved"),
 };
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_gt_stats_types.h b/drivers/gpu/drm/xe/xe_gt_stats_types.h
index 425491bed6c4..c42b494390f8 100644
--- a/drivers/gpu/drm/xe/xe_gt_stats_types.h
+++ b/drivers/gpu/drm/xe/xe_gt_stats_types.h
@@ -122,6 +122,18 @@
  * @XE_GT_STATS_ID_PRL_ABORTED_COUNT: Times the page reclaim process was
  *   aborted.
  *
+ * @XE_GT_STATS_ID_DEFRAG_ADDED_COUNT: Times a buffer object was added to the
+ *   page defragmentation list after being backed at a sub-optimal page order.
+ * @XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT: Times the defrag worker successfully
+ *   re-backed a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT: Times the defrag worker
+ *   partially re-backed a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_FAILED_COUNT: Times the defrag worker failed to
+ *   re-back a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_MB_MOVED: Data (MiB) actually (re)allocated while
+ *   re-backing buffer objects at the beneficial page order. Chunks borrowed
+ *   from the old backing for free are not counted.
+ *
  * @__XE_GT_STATS_NUM_IDS: Number of valid IDs; not a real counter.
  *
  * See Documentation/gpu/xe/xe_gt_stats.rst.
@@ -181,6 +193,11 @@ enum xe_gt_stats_id {
 	XE_GT_STATS_ID_PRL_2M_ENTRY_COUNT,
 	XE_GT_STATS_ID_PRL_ISSUED_COUNT,
 	XE_GT_STATS_ID_PRL_ABORTED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_ADDED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT,
+	XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
+	XE_GT_STATS_ID_DEFRAG_FAILED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_MB_MOVED,
 	/* must be the last entry */
 	__XE_GT_STATS_NUM_IDS,
 };
-- 
2.34.1

[PATCH v2 22/33] drm/xe: Add Kconfig.profile options for BO defrag configuration

[Matthew Brost]

Add Kconfig.profile options to make XE_BO_DEFRAG_* defines configurable:
- DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD: Threshold for TTM reclaim
  backoff
- DRM_XE_BO_DEFRAG_SIZE_LIMIT: Maximum number of bytes to defrag per
  work run
- DRM_XE_BO_DEFRAG_INTERVAL_MS: Default delay before defrag worker runs
- DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS: Upper bound for defrag worker
  interval

Update xe_bo.c to use these Kconfig options as defaults via #ifdef
guards, maintaining backward compatibility with hardcoded values when
not configured.

Additionally, disable defrag completely when XE_BO_DEFRAG_SIZE_LIMIT is
set less than 2M by returning early from xe_bo_defrag_update() (and
xe_bo_defrag_remove()) so BOs are never added to the defrag list or the
worker kicked; xe_bo_defrag_add() and xe_bo_defrag_schedule() assert the
limit is at least 2M.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-haiku-4.5
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/Kconfig.profile | 40 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.c         | 26 +++++++++++++++++++
 2 files changed, 66 insertions(+)

diff --git a/drivers/gpu/drm/xe/Kconfig.profile b/drivers/gpu/drm/xe/Kconfig.profile
index e07517d120e0..e0aade41f53e 100644
--- a/drivers/gpu/drm/xe/Kconfig.profile
+++ b/drivers/gpu/drm/xe/Kconfig.profile
@@ -74,3 +74,43 @@ config DRM_XE_ENABLE_SCHEDTIMEOUT_LIMIT
 	  to apply to applicable user. For elevated user, all above MIN
 	  and MAX values will apply when this configuration is enable to
 	  apply limitation. By default limitation is applied.
+
+config DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+	int "BO defrag reclaim backoff threshold"
+	default 2
+	range 1 1000
+	help
+	  Once this many BOs are tracked on the device defrag list (i.e. were
+	  backed with a sub-optimal page order), request that the TTM pool backs
+	  off from aggressive reclaim at the beneficial order during populate,
+	  so that allocations make forward progress instead of stalling.
+
+config DRM_XE_BO_DEFRAG_SIZE_LIMIT
+	int "Maximum bytes to defrag per work run"
+	default 33554432
+	range 0 1073741824
+	help
+	  Maximum number of bytes the defrag worker will process in a single run
+	  before yielding and rescheduling itself. Set to less than large page
+	  size (2M) to disable defrag. A defrag move synchronously reallocates
+	  and re-copies a BO's backing store, which is not free. If a large
+	  number of BOs become eligible at once, processing them all in one
+	  worker run would hold things up for a long, unbounded stretch.
+	  Instead, cap the work done per run and requeue, spreading the defrag
+	  effort out over time.
+
+config DRM_XE_BO_DEFRAG_INTERVAL_MS
+	int "Default delay before defrag worker run (ms)"
+	default 25
+	range 1 1000
+	help
+	  Default delay before (re)running the defrag worker, in milliseconds.
+
+config DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+	int "Upper bound for defrag worker interval (ms)"
+	default 15000
+	range 100 1000000
+	help
+	  Upper bound for the (exponentially backed off) defrag worker interval,
+	  in milliseconds, so repeated failures don't push the retry arbitrarily
+	  far out. 15000 ms = 15 seconds.
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index c1bbcf0d21ed..405316d0d116 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -49,7 +49,11 @@
  * aggressive reclaim at the beneficial order during populate, so that
  * allocations make forward progress instead of stalling.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	CONFIG_DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+#else
 #define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+#endif
 
 /*
  * Maximum number of bytes of newly (re)allocated backing the defrag worker will
@@ -65,17 +69,31 @@
  * Only pages a move truly reallocates are charged; pages harvested from the old
  * backing are free, so an object larger than the budget is upgraded in
  * budget-sized slices across runs.
+ *
+ * Set to 0 to disable defrag completely.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_SIZE_LIMIT
+#define XE_BO_DEFRAG_SIZE_LIMIT			CONFIG_DRM_XE_BO_DEFRAG_SIZE_LIMIT
+#else
 #define XE_BO_DEFRAG_SIZE_LIMIT			SZ_32M
+#endif
 
 /* Default delay before (re)running the defrag worker, in milliseconds. */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MS
+#define XE_BO_DEFRAG_INTERVAL_MS		CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MS
+#else
 #define XE_BO_DEFRAG_INTERVAL_MS		25
+#endif
 
 /*
  * Upper bound for the (exponentially backed off) defrag worker interval, in
  * milliseconds, so repeated failures don't push the retry arbitrarily far out.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+#define XE_BO_DEFRAG_INTERVAL_MAX_MS		CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+#else
 #define XE_BO_DEFRAG_INTERVAL_MAX_MS		15000	/* 15 seconds */
+#endif
 
 static void xe_bo_defrag_worker(struct work_struct *w);
 static void xe_place_from_ttm_type(u32 mem_type, struct ttm_place *place);
@@ -1120,6 +1138,7 @@ int xe_bo_defrag_init(struct xe_device *xe)
 
 static void xe_bo_defrag_schedule(struct xe_device *xe)
 {
+	xe_assert(xe, XE_BO_DEFRAG_SIZE_LIMIT >= SZ_2M);
 	schedule_delayed_work(&xe->mem.defrag.worker,
 			      msecs_to_jiffies(xe->mem.defrag.interval_ms));
 }
@@ -1146,6 +1165,7 @@ static void xe_bo_defrag_add(struct xe_bo *bo)
 
 	xe_bo_assert_held(bo);
 	xe_assert(xe, xe_bo_needs_defrag(bo));
+	xe_assert(xe, XE_BO_DEFRAG_SIZE_LIMIT >= SZ_2M);
 
 	scoped_guard(spinlock, &xe->mem.defrag.lock) {
 		if (list_empty(&bo->defrag_link)) {
@@ -1179,6 +1199,9 @@ static void xe_bo_defrag_remove(struct xe_bo *bo)
 {
 	xe_bo_assert_held(bo);
 
+	if (XE_BO_DEFRAG_SIZE_LIMIT < SZ_2M)
+		return;
+
 	if (list_empty(&bo->defrag_link))
 		return;
 
@@ -1198,6 +1221,9 @@ static void xe_bo_defrag_update(struct xe_bo *bo)
 {
 	xe_bo_assert_held(bo);
 
+	if (XE_BO_DEFRAG_SIZE_LIMIT < SZ_2M)
+		return;
+
 	if (xe_bo_needs_defrag(bo))
 		xe_bo_defrag_add(bo);
 	else
-- 
2.34.1

[PATCH v2 23/33] drm/xe: Defrag using out-of-lock page preallocation

[Matthew Brost]

Convert xe_bo_defrag_one() to a three-phase scheme that moves the
expensive beneficial-order allocations out of the BO dma-resv lock:

 1. Under the lock, re-check eligibility, capture the tt cpu-caching and
    read the exact sub-optimal page count (ttm_tt_suboptimal_pages()),
    bounded by the run budget.
 2. Drop the lock and preallocate that many beneficial-order chunks via
    ttm_pool_prealloc_fill(); reclaim/compaction stalls and the cpu
    cache mode change both happen here, outside the lock.
 3. Re-take the lock, re-check, and validate consuming the prealloc bag.
    Any shortfall harvests the old backing instead of reclaiming, so the
    move never stalls under the lock; the BO re-queues for a later pass.

With both allocation and caching hoisted out, the BO-lock held time
drops to copy-bound (median ~0.27ms, p95 ~2ms); the remaining tail is
lock-wait behind active rendering, as intended.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 46 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 45 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 405316d0d116..2f014fc9f988 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1249,15 +1249,55 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 		.defrag_bytes_remaining = budget,
 	};
 	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_pool *pool = &ttm_bo->bdev->pool;
+	struct ttm_pool_prealloc pp = {};
 	struct ttm_placement placement;
 	struct ttm_place place;
+	enum ttm_caching tt_caching;
+	unsigned int order, want = 0;
 	int ret = 0;
 
 	*consumed = 0;
 
+	/*
+	 * Phase 1: under the BO lock, re-check eligibility and estimate how
+	 * many beneficial-order chunks the move will (re)allocate, bounded by
+	 * the run's byte budget. Then drop the lock so the high-order, possibly
+	 * reclaim/compaction stalling allocations happen unlocked.
+	 */
+	xe_bo_lock(bo, false);
+	if (!xe_bo_needs_defrag(bo)) {
+		xe_bo_defrag_remove(bo);
+		xe_bo_unlock(bo);
+		goto out;
+	}
+	tt_caching = bo->ttm.ttm->caching;
+	order = ttm_pool_prealloc_order(pool);
+	if (order && ttm_bo->ttm) {
+		u32 suboptimal = ttm_tt_suboptimal_pages(ttm_bo->ttm);
+		u64 cap = min_t(u64, budget, xe_bo_size(bo)) >> PAGE_SHIFT;
+
+		/* Prealloc only the beneficial-order chunks the move replaces. */
+		want = DIV_ROUND_UP(min_t(u64, suboptimal, cap), 1UL << order);
+	}
+	xe_bo_unlock(bo);
+
+	/* Phase 2: preallocate outside the lock; */
+	if (want) {
+		ret = ttm_pool_prealloc_fill(pool, tt_caching, &pp, want);
+		if (ret || !pp.count) {
+			ret = ret ?: -ENOMEM;
+			xe_gt_stats_incr(xe_root_mmio_gt(xe),
+					 XE_GT_STATS_ID_DEFRAG_FAILED_COUNT, 1);
+			goto out_err;
+		}
+	}
+
+	/*
+	 * Phase 3: re-take the lock, re-check, and validate using the prealloc.
+	 */
 	xe_bo_lock(bo, false);
 
-	/* Re-check eligibility under the BO lock. */
 	if (!xe_bo_needs_defrag(bo)) {
 		xe_bo_defrag_remove(bo);
 		goto unlock;
@@ -1267,6 +1307,7 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	place.flags |= TTM_PL_FLAG_TEMPORARY;
 	placement.num_placement = 1;
 	placement.placement = &place;
+	ctx.prealloc = &pp;
 
 	/*
 	 * On success the move reallocates the backing at beneficial order and
@@ -1309,6 +1350,9 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 unlock:
 	xe_bo_unlock(bo);
+out_err:
+	ttm_pool_prealloc_fini(pool, &pp);
+out:
 	return ret;
 }
 
-- 
2.34.1

[PATCH v2 24/33] drm/xe: Add defrag profiling tracepoints

[Matthew Brost]

Add two tracepoints to profile the page defragmentation worker:

 - xe_bo_defrag_worker: emitted on worker entry and exit with the
   current defrag list count and the bytes processed in the run.
 - xe_bo_defrag_one: emitted per BO with the run budget, the prealloc
   fill ratio (got/want) and the lock-held, lock-wait and prealloc
   durations, plus the bytes consumed and the result.

These break down the cost of a defrag attempt into its locked vs
unlocked phases, making it possible to confirm that the expensive
beneficial-order allocation and cpu cache mode change happen outside
the BO dma-resv lock and that the residual lock-held time is copy-bound
(with the tail dominated by lock-wait behind active rendering).

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c       | 18 +++++++++-
 drivers/gpu/drm/xe/xe_trace_bo.h | 59 ++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 2f014fc9f988..a01004265349 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1254,7 +1254,9 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	struct ttm_placement placement;
 	struct ttm_place place;
 	enum ttm_caching tt_caching;
-	unsigned int order, want = 0;
+	unsigned int order, want = 0, got = 0;
+	u64 lock_ns = 0, prealloc_ns = 0, lockwait_ns = 0;
+	ktime_t t0;
 	int ret = 0;
 
 	*consumed = 0;
@@ -1284,6 +1286,7 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 	/* Phase 2: preallocate outside the lock; */
 	if (want) {
+		t0 = ktime_get();
 		ret = ttm_pool_prealloc_fill(pool, tt_caching, &pp, want);
 		if (ret || !pp.count) {
 			ret = ret ?: -ENOMEM;
@@ -1291,12 +1294,16 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 					 XE_GT_STATS_ID_DEFRAG_FAILED_COUNT, 1);
 			goto out_err;
 		}
+		prealloc_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
 	}
 
 	/*
 	 * Phase 3: re-take the lock, re-check, and validate using the prealloc.
 	 */
+	t0 = ktime_get();
 	xe_bo_lock(bo, false);
+	lockwait_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
+	t0 = ktime_get();
 
 	if (!xe_bo_needs_defrag(bo)) {
 		xe_bo_defrag_remove(bo);
@@ -1350,9 +1357,13 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 unlock:
 	xe_bo_unlock(bo);
+	lock_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
+	got = pp.used;
 out_err:
 	ttm_pool_prealloc_fini(pool, &pp);
 out:
+	trace_xe_bo_defrag_one(bo, budget, want, got, lock_ns, prealloc_ns,
+			       lockwait_ns, *consumed, ret);
 	return ret;
 }
 
@@ -1373,6 +1384,9 @@ static void xe_bo_defrag_worker(struct work_struct *w)
 		return;
 	}
 
+	trace_xe_bo_defrag_worker(xe, 0, atomic_read(&xe->mem.defrag.count),
+				  true);
+
 	/*
 	 * Process at most XE_BO_DEFRAG_SIZE_LIMIT bytes of newly (re)allocated
 	 * backing per run rather than draining the whole list in one go. Only
@@ -1479,6 +1493,8 @@ static void xe_bo_defrag_worker(struct work_struct *w)
 
 	xe_pm_runtime_put(xe);
 	drm_dev_exit(idx);
+	trace_xe_bo_defrag_worker(xe, defrag_bytes,
+				  atomic_read(&xe->mem.defrag.count), false);
 }
 
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
diff --git a/drivers/gpu/drm/xe/xe_trace_bo.h b/drivers/gpu/drm/xe/xe_trace_bo.h
index 86323cf3be2c..a169a322464b 100644
--- a/drivers/gpu/drm/xe/xe_trace_bo.h
+++ b/drivers/gpu/drm/xe/xe_trace_bo.h
@@ -85,6 +85,65 @@ TRACE_EVENT(xe_bo_move,
 		      __get_str(new_placement_name), __get_str(device_id))
 );
 
+TRACE_EVENT(xe_bo_defrag_worker,
+	    TP_PROTO(struct xe_device *xe, u64 defrag_bytes, int count,
+		     bool enter),
+	    TP_ARGS(xe, defrag_bytes, count, enter),
+	    TP_STRUCT__entry(
+		     __string(dev, dev_name(xe->drm.dev))
+		     __field(u64, defrag_bytes)
+		     __field(int, count)
+		     __field(bool, enter)
+		     ),
+	    TP_fast_assign(
+		   __assign_str(dev);
+		   __entry->defrag_bytes = defrag_bytes;
+		   __entry->count = count;
+		   __entry->enter = enter;
+		   ),
+	    TP_printk("dev=%s %s defrag_bytes=%llu count=%d",
+		      __get_str(dev), __entry->enter ? "enter" : "exit",
+		      __entry->defrag_bytes, __entry->count)
+);
+
+TRACE_EVENT(xe_bo_defrag_one,
+	    TP_PROTO(struct xe_bo *bo, u64 budget, unsigned int prealloc_want,
+		     unsigned int prealloc_got, u64 lock_ns, u64 prealloc_ns,
+		     u64 lockwait_ns, u64 consumed, int ret),
+	    TP_ARGS(bo, budget, prealloc_want, prealloc_got, lock_ns,
+		    prealloc_ns, lockwait_ns, consumed, ret),
+	    TP_STRUCT__entry(
+		     __string(dev, __dev_name_bo(bo))
+		     __field(size_t, size)
+		     __field(u64, budget)
+		     __field(unsigned int, prealloc_want)
+		     __field(unsigned int, prealloc_got)
+		     __field(u64, lock_ns)
+		     __field(u64, prealloc_ns)
+		     __field(u64, lockwait_ns)
+		     __field(u64, consumed)
+		     __field(int, ret)
+		     ),
+	    TP_fast_assign(
+		   __assign_str(dev);
+		   __entry->size = xe_bo_size(bo);
+		   __entry->budget = budget;
+		   __entry->prealloc_want = prealloc_want;
+		   __entry->prealloc_got = prealloc_got;
+		   __entry->lock_ns = lock_ns;
+		   __entry->prealloc_ns = prealloc_ns;
+		   __entry->lockwait_ns = lockwait_ns;
+		   __entry->consumed = consumed;
+		   __entry->ret = ret;
+		   ),
+	    TP_printk("dev=%s size=%zu budget=%llu prealloc=%u/%u lock_us=%llu lockwait_us=%llu prealloc_us=%llu consumed=%llu ret=%d",
+		      __get_str(dev), __entry->size, __entry->budget,
+		      __entry->prealloc_got, __entry->prealloc_want,
+		      __entry->lock_ns / 1000, __entry->lockwait_ns / 1000,
+		      __entry->prealloc_ns / 1000, __entry->consumed,
+		      __entry->ret)
+);
+
 DECLARE_EVENT_CLASS(xe_vma,
 		    TP_PROTO(struct xe_vma *vma),
 		    TP_ARGS(vma),
-- 
2.34.1

[PATCH v2 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock

[Matthew Brost]

xe_gem_create_ioctl() populates a buffer object's backing store inside
xe_validation_guard(), i.e. under the dma-resv lock.  For a system-memory
buffer that means the whole page allocation - potentially stalling in
reclaim or compaction - happens in the critical section, holding the
reservation and blocking anything that contends on it.

Use the TTM full-preallocation mechanism to move that work out of the
lock.  For a system-only, non-deferred user BO, allocate the entire tt
backing up front, before entering the validation guard, and hand the bag
down to the populate so it merely installs the pages:

  - Thread an optional struct ttm_pool_prealloc through xe_bo_create_user()
    -> __xe_bo_create_locked() -> xe_bo_init_locked(), which stashes it in
    ttm_operation_ctx::prealloc.  All other callers pass NULL.

  - In xe_gem_create_ioctl(), for XE_BO_FLAG_SYSTEM BOs that are not VRAM,
    stolen or deferred-backing, call ttm_pool_prealloc_fill_full() before
    the guard and ttm_pool_prealloc_fini() after it.  The bag size covers
    both the data pages and the appended CCS backup pages, matching the tt
    layout in xe_ttm_tt_create(), and the caching mirrors the same path.

The fill is best-effort: on a short fill (or if it is skipped) the pool
allocates the remainder in-line as before, so this is purely an
optimisation with no behavioural change to the created BO.  fini() only
releases the unconsumed tail, so it is safe on every success, error and
OOM-retry path (consumed pages are owned by the tt).

The fill requests reclaim backoff for higher-order chunks
(@beneficial_reclaim_backoff): Xe runs a background defragmenter that
upgrades suboptimal pages to beneficial order in place, so there is no
need to stall this user context reclaiming/compacting for contiguity.

To size the preallocation before the xe_bo exists, factor the CCS-pages
decision in xe_bo_needs_ccs_pages() into a parameterised helper,
__xe_bo_needs_ccs_pages(), and query it directly from the ioctl.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/tests/xe_bo.c      |   8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c |   2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c |   6 +-
 drivers/gpu/drm/xe/xe_bo.c            | 120 ++++++++++++++++++++------
 drivers/gpu/drm/xe/xe_bo.h            |   8 +-
 drivers/gpu/drm/xe/xe_dma_buf.c       |   2 +-
 6 files changed, 109 insertions(+), 37 deletions(-)

diff --git a/drivers/gpu/drm/xe/tests/xe_bo.c b/drivers/gpu/drm/xe/tests/xe_bo.c
index 49c95ed67d7e..78781317b420 100644
--- a/drivers/gpu/drm/xe/tests/xe_bo.c
+++ b/drivers/gpu/drm/xe/tests/xe_bo.c
@@ -141,7 +141,7 @@ static void ccs_test_run_tile(struct xe_device *xe, struct xe_tile *tile,
 		kunit_info(test, "Testing system memory\n");
 
 	bo = xe_bo_create_user(xe, NULL, SZ_1M, DRM_XE_GEM_CPU_CACHING_WC,
-			       bo_flags, exec);
+			       bo_flags, NULL, exec);
 	if (IS_ERR(bo)) {
 		KUNIT_FAIL(test, "Failed to create bo.\n");
 		return;
@@ -220,7 +220,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
 		xe_vm_lock(vm, false);
 		bo = xe_bo_create_user(xe, vm, 0x10000,
 				       DRM_XE_GEM_CPU_CACHING_WC,
-				       bo_flags, exec);
+				       bo_flags, NULL, exec);
 		xe_vm_unlock(vm);
 		if (IS_ERR(bo)) {
 			KUNIT_FAIL(test, "bo create err=%pe\n", bo);
@@ -229,7 +229,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
 
 		external = xe_bo_create_user(xe, NULL, 0x10000,
 					     DRM_XE_GEM_CPU_CACHING_WC,
-					     bo_flags, NULL);
+					     bo_flags, NULL, NULL);
 		if (IS_ERR(external)) {
 			KUNIT_FAIL(test, "external bo create err=%pe\n", external);
 			goto cleanup_bo;
@@ -494,7 +494,7 @@ static int shrink_test_run_device(struct xe_device *xe)
 		/* We can create bos using WC caching here. But it is slower. */
 		bo = xe_bo_create_user(xe, NULL, XE_BO_SHRINK_SIZE,
 				       DRM_XE_GEM_CPU_CACHING_WB,
-				       XE_BO_FLAG_SYSTEM, NULL);
+				       XE_BO_FLAG_SYSTEM, NULL, NULL);
 		if (IS_ERR(bo)) {
 			if (bo != ERR_PTR(-ENOMEM) && bo != ERR_PTR(-ENOSPC) &&
 			    bo != ERR_PTR(-EINTR) && bo != ERR_PTR(-ERESTARTSYS))
diff --git a/drivers/gpu/drm/xe/tests/xe_dma_buf.c b/drivers/gpu/drm/xe/tests/xe_dma_buf.c
index 0be8440b3976..e4fdc247c0c7 100644
--- a/drivers/gpu/drm/xe/tests/xe_dma_buf.c
+++ b/drivers/gpu/drm/xe/tests/xe_dma_buf.c
@@ -127,7 +127,7 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
 
 	kunit_info(test, "running %s\n", __func__);
 	bo = xe_bo_create_user(xe, NULL, size, DRM_XE_GEM_CPU_CACHING_WC,
-			       params->mem_mask, NULL);
+			       params->mem_mask, NULL, NULL);
 	if (IS_ERR(bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
 			   PTR_ERR(bo));
diff --git a/drivers/gpu/drm/xe/tests/xe_migrate.c b/drivers/gpu/drm/xe/tests/xe_migrate.c
index 3c1be809be82..41de648b2c86 100644
--- a/drivers/gpu/drm/xe/tests/xe_migrate.c
+++ b/drivers/gpu/drm/xe/tests/xe_migrate.c
@@ -647,7 +647,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 				   DRM_XE_GEM_CPU_CACHING_WC,
 				   XE_BO_FLAG_SYSTEM |
 				   XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				   XE_BO_FLAG_PINNED, NULL);
+				   XE_BO_FLAG_PINNED, NULL, NULL);
 
 	if (IS_ERR(sys_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@@ -673,7 +673,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 	ccs_bo = xe_bo_create_user(xe, NULL, SZ_4M,
 				   DRM_XE_GEM_CPU_CACHING_WC,
 				   bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				   XE_BO_FLAG_PINNED, NULL);
+				   XE_BO_FLAG_PINNED, NULL, NULL);
 
 	if (IS_ERR(ccs_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@@ -698,7 +698,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 	vram_bo = xe_bo_create_user(xe, NULL, SZ_4M,
 				    DRM_XE_GEM_CPU_CACHING_WC,
 				    bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				    XE_BO_FLAG_PINNED, NULL);
+				    XE_BO_FLAG_PINNED, NULL, NULL);
 	if (IS_ERR(vram_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
 			   PTR_ERR(vram_bo));
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index a01004265349..d9226a3995fe 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1924,7 +1924,7 @@ int xe_bo_notifier_prepare_pinned(struct xe_bo *bo)
 		backup = xe_bo_init_locked(xe, NULL, NULL, bo->ttm.base.resv, NULL, xe_bo_size(bo),
 					   DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
 					   XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-					   XE_BO_FLAG_PINNED, &exec);
+					   XE_BO_FLAG_PINNED, NULL, &exec);
 		if (IS_ERR(backup)) {
 			drm_exec_retry_on_contention(&exec);
 			ret = PTR_ERR(backup);
@@ -2065,7 +2065,7 @@ int xe_bo_evict_pinned(struct xe_bo *bo)
 						   xe_bo_size(bo),
 						   DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
 						   XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-						   XE_BO_FLAG_PINNED, &exec);
+						   XE_BO_FLAG_PINNED, NULL, &exec);
 			if (IS_ERR(backup)) {
 				drm_exec_retry_on_contention(&exec);
 				ret = PTR_ERR(backup);
@@ -2860,6 +2860,8 @@ void xe_bo_free(struct xe_bo *bo)
  * @cpu_caching: The cpu caching used for system memory backing store.
  * @type: The TTM buffer object type.
  * @flags: XE_BO_FLAG_ flags.
+ * @prealloc: Optional pages preallocated outside the dma-resv lock to back the
+ * bo (see ttm_operation_ctx::prealloc), or NULL.
  * @exec: The drm_exec transaction to use for exhaustive eviction.
  *
  * Initialize or create an xe buffer object. On failure, any allocated buffer
@@ -2871,12 +2873,14 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 				struct xe_tile *tile, struct dma_resv *resv,
 				struct ttm_lru_bulk_move *bulk, size_t size,
 				u16 cpu_caching, enum ttm_bo_type type,
-				u32 flags, struct drm_exec *exec)
+				u32 flags, struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec)
 {
 	struct ttm_operation_ctx ctx = {
 		.interruptible = true,
 		.no_wait_gpu = false,
 		.gfp_retry_mayfail = true,
+		.prealloc = prealloc,
 	};
 	struct ttm_placement *placement;
 	uint32_t alignment;
@@ -3053,7 +3057,8 @@ __xe_bo_create_locked(struct xe_device *xe,
 		      struct xe_tile *tile, struct xe_vm *vm,
 		      size_t size, u64 start, u64 end,
 		      u16 cpu_caching, enum ttm_bo_type type, u32 flags,
-		      u64 alignment, struct drm_exec *exec)
+		      u64 alignment, struct ttm_pool_prealloc *prealloc,
+		      struct drm_exec *exec)
 {
 	struct xe_bo *bo = NULL;
 	int err;
@@ -3078,7 +3083,7 @@ __xe_bo_create_locked(struct xe_device *xe,
 			       vm && !xe_vm_in_fault_mode(vm) &&
 			       flags & XE_BO_FLAG_USER ?
 			       &vm->lru_bulk_move : NULL, size,
-			       cpu_caching, type, flags, exec);
+			       cpu_caching, type, flags, prealloc, exec);
 	if (IS_ERR(bo))
 		return bo;
 
@@ -3153,7 +3158,7 @@ struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile,
 				  struct drm_exec *exec)
 {
 	return __xe_bo_create_locked(xe, tile, vm, size, 0, ~0ULL, 0, type,
-				     flags, 0, exec);
+				     flags, 0, NULL, exec);
 }
 
 static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *tile,
@@ -3169,7 +3174,7 @@ static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *til
 	xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {.interruptible = intr},
 			    ret) {
 		bo = __xe_bo_create_locked(xe, tile, NULL, size, 0, ~0ULL,
-					   cpu_caching, type, flags, alignment, &exec);
+					   cpu_caching, type, flags, alignment, NULL, &exec);
 		drm_exec_retry_on_contention(&exec);
 		if (IS_ERR(bo)) {
 			ret = PTR_ERR(bo);
@@ -3189,6 +3194,9 @@ static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *til
  * @size: The storage size to use for the bo.
  * @cpu_caching: The caching mode to be used for system backing store.
  * @flags: XE_BO_FLAG_ flags.
+ * @prealloc: Optional pages preallocated outside the dma-resv lock to back the
+ * bo (see ttm_operation_ctx::prealloc), or NULL. Only consumed on the locked
+ * (@vm or @exec) path.
  * @exec: The drm_exec transaction to use for exhaustive eviction, or NULL
  * if such a transaction should be initiated by the call.
  *
@@ -3199,7 +3207,8 @@ static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *til
 struct xe_bo *xe_bo_create_user(struct xe_device *xe,
 				struct xe_vm *vm, size_t size,
 				u16 cpu_caching,
-				u32 flags, struct drm_exec *exec)
+				u32 flags, struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec)
 {
 	struct xe_bo *bo;
 
@@ -3209,7 +3218,7 @@ struct xe_bo *xe_bo_create_user(struct xe_device *xe,
 		xe_assert(xe, exec);
 		bo = __xe_bo_create_locked(xe, NULL, vm, size, 0, ~0ULL,
 					   cpu_caching, ttm_bo_type_device,
-					   flags, 0, exec);
+					   flags, 0, prealloc, exec);
 		if (!IS_ERR(bo))
 			xe_bo_unlock_vm_held(bo);
 	} else {
@@ -3248,7 +3257,7 @@ struct xe_bo *xe_bo_create_pin_range_novm(struct xe_device *xe, struct xe_tile *
 
 	xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, err) {
 		bo = __xe_bo_create_locked(xe, tile, NULL, size, start, end,
-					   0, type, flags, 0, &exec);
+					   0, type, flags, 0, NULL, &exec);
 		if (IS_ERR(bo)) {
 			drm_exec_retry_on_contention(&exec);
 			err = PTR_ERR(bo);
@@ -3287,7 +3296,7 @@ static struct xe_bo *xe_bo_create_pin_map_at_aligned(struct xe_device *xe,
 
 	bo = __xe_bo_create_locked(xe, tile, vm, size, start, end, 0, type,
 				   flags | XE_BO_FLAG_NEEDS_CPU_ACCESS | XE_BO_FLAG_PINNED,
-				   alignment, exec);
+				   alignment, NULL, exec);
 	if (IS_ERR(bo))
 		return bo;
 
@@ -3902,6 +3911,10 @@ static int gem_create_user_extensions(struct xe_device *xe, struct xe_bo *bo,
 	return 0;
 }
 
+static bool __xe_bo_needs_ccs_pages(struct xe_device *xe,
+				    enum ttm_bo_type type, u32 flags,
+				    u16 cpu_caching);
+
 int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			struct drm_file *file)
 {
@@ -3912,6 +3925,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	struct drm_exec exec;
 	struct xe_vm *vm = NULL;
 	struct xe_bo *bo;
+	struct ttm_pool_prealloc prealloc = {};
+	bool have_prealloc = false;
 	unsigned int bo_flags;
 	u32 handle;
 	int err;
@@ -3994,6 +4009,45 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			return -ENOENT;
 	}
 
+	/*
+	 * For system-only BOs, preallocate the whole backing outside the
+	 * dma-resv lock so the populate under the validation guard does not
+	 * stall in reclaim/compaction. The caching mirrors the system-BO path
+	 * in xe_ttm_tt_create(); the pool falls back to in-line allocation for
+	 * any shortfall, so a failed fill is harmless.
+	 */
+	if ((bo_flags & XE_BO_FLAG_SYSTEM) &&
+	    !(bo_flags & (XE_BO_FLAG_DEFER_BACKING | XE_BO_FLAG_VRAM_MASK |
+			  XE_BO_FLAG_STOLEN))) {
+		enum ttm_caching caching = ttm_cached;
+		unsigned long num_pages = args->size >> PAGE_SHIFT;
+
+		if (!IS_DGFX(xe) &&
+		    args->cpu_caching == DRM_XE_GEM_CPU_CACHING_WC)
+			caching = ttm_write_combined;
+
+		/*
+		 * Cover the CCS backup pages appended to the tt (see
+		 * xe_ttm_tt_create()) so the whole tt is preallocated.
+		 */
+		if (__xe_bo_needs_ccs_pages(xe, ttm_bo_type_device, bo_flags,
+					    args->cpu_caching))
+			num_pages += DIV_ROUND_UP(xe_device_ccs_bytes(xe,
+								      args->size),
+						  PAGE_SIZE);
+
+		/*
+		 * Back off aggressive reclaim/compaction for higher-order
+		 * chunks here: Xe runs a background defragmenter that upgrades
+		 * suboptimal pages to beneficial order in place later, so there
+		 * is no need to stall this user context reclaiming for
+		 * contiguity.
+		 */
+		ttm_pool_prealloc_fill_full(&xe->ttm.pool, caching, &prealloc,
+					    num_pages, true);
+		have_prealloc = true;
+	}
+
 	err = 0;
 	xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {.interruptible = true},
 			    err) {
@@ -4004,7 +4058,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 				break;
 		}
 		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
-				       bo_flags, &exec);
+				       bo_flags, have_prealloc ? &prealloc : NULL,
+				       &exec);
 		drm_exec_retry_on_contention(&exec);
 		if (IS_ERR(bo)) {
 			err = PTR_ERR(bo);
@@ -4012,6 +4067,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			break;
 		}
 	}
+	if (have_prealloc)
+		ttm_pool_prealloc_fini(&xe->ttm.pool, &prealloc);
 	if (err)
 		goto out_vm;
 
@@ -4307,21 +4364,19 @@ int xe_bo_evict(struct xe_bo *bo, struct drm_exec *exec)
 	return 0;
 }
 
-/**
- * xe_bo_needs_ccs_pages - Whether a bo needs to back up CCS pages when
- * placed in system memory.
- * @bo: The xe_bo
- *
- * Return: true if extra pages need to be allocated, false otherwise.
+/*
+ * Whether a bo with the given attributes needs to back up CCS pages when placed
+ * in system memory. Split out so callers can query before the xe_bo exists (see
+ * xe_gem_create_ioctl() preallocation).
  */
-bool xe_bo_needs_ccs_pages(struct xe_bo *bo)
+static bool __xe_bo_needs_ccs_pages(struct xe_device *xe,
+				    enum ttm_bo_type type, u32 flags,
+				    u16 cpu_caching)
 {
-	struct xe_device *xe = xe_bo_device(bo);
-
 	if (GRAPHICS_VER(xe) >= 20 && IS_DGFX(xe))
 		return false;
 
-	if (!xe_device_has_flat_ccs(xe) || bo->ttm.type != ttm_bo_type_device)
+	if (!xe_device_has_flat_ccs(xe) || type != ttm_bo_type_device)
 		return false;
 
 	/* On discrete GPUs, if the GPU can access this buffer from
@@ -4329,24 +4384,37 @@ bool xe_bo_needs_ccs_pages(struct xe_bo *bo)
 	 * can't be used since there's no CCS storage associated with
 	 * non-VRAM addresses.
 	 */
-	if (IS_DGFX(xe) && (bo->flags & XE_BO_FLAG_SYSTEM))
+	if (IS_DGFX(xe) && (flags & XE_BO_FLAG_SYSTEM))
 		return false;
 
 	/* Check if userspace explicitly requested no compression */
-	if (bo->flags & XE_BO_FLAG_NO_COMPRESSION)
+	if (flags & XE_BO_FLAG_NO_COMPRESSION)
 		return false;
 
 	/*
 	 * For WB (Write-Back) CPU caching mode, check if the device
 	 * supports WB compression with coherency.
 	 */
-	if (bo->cpu_caching == DRM_XE_GEM_CPU_CACHING_WB &&
+	if (cpu_caching == DRM_XE_GEM_CPU_CACHING_WB &&
 	    xe->pat.idx[XE_CACHE_WB_COMPRESSION] == XE_PAT_INVALID_IDX)
 		return false;
 
 	return true;
 }
 
+/**
+ * xe_bo_needs_ccs_pages - Whether a bo needs to back up CCS pages when
+ * placed in system memory.
+ * @bo: The xe_bo
+ *
+ * Return: true if extra pages need to be allocated, false otherwise.
+ */
+bool xe_bo_needs_ccs_pages(struct xe_bo *bo)
+{
+	return __xe_bo_needs_ccs_pages(xe_bo_device(bo), bo->ttm.type,
+				       bo->flags, bo->cpu_caching);
+}
+
 /**
  * __xe_bo_release_dummy() - Dummy kref release function
  * @kref: The embedded struct kref.
@@ -4453,7 +4521,7 @@ int xe_bo_dumb_create(struct drm_file *file_priv,
 			       DRM_XE_GEM_CPU_CACHING_WC,
 			       XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
 			       XE_BO_FLAG_FORCE_WC |
-			       XE_BO_FLAG_NEEDS_CPU_ACCESS, NULL);
+			       XE_BO_FLAG_NEEDS_CPU_ACCESS, NULL, NULL);
 	if (IS_ERR(bo))
 		return PTR_ERR(bo);
 
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 38d50f8122cb..18fcccd082b7 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -111,6 +111,7 @@ enum xe_madv_purgeable_state {
 
 struct sg_table;
 struct ttm_tt;
+struct ttm_pool_prealloc;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -119,13 +120,16 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 				struct xe_tile *tile, struct dma_resv *resv,
 				struct ttm_lru_bulk_move *bulk, size_t size,
 				u16 cpu_caching, enum ttm_bo_type type,
-				u32 flags, struct drm_exec *exec);
+				u32 flags, struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec);
 struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile,
 				  struct xe_vm *vm, size_t size,
 				  enum ttm_bo_type type, u32 flags,
 				  struct drm_exec *exec);
 struct xe_bo *xe_bo_create_user(struct xe_device *xe, struct xe_vm *vm, size_t size,
-				u16 cpu_caching, u32 flags, struct drm_exec *exec);
+				u16 cpu_caching, u32 flags,
+				struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec);
 struct xe_bo *xe_bo_create_pin_map(struct xe_device *xe, struct xe_tile *tile,
 				   struct xe_vm *vm, size_t size,
 				   enum ttm_bo_type type, u32 flags,
diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c b/drivers/gpu/drm/xe/xe_dma_buf.c
index 8a920e58245c..ec947d86a1ce 100644
--- a/drivers/gpu/drm/xe/xe_dma_buf.c
+++ b/drivers/gpu/drm/xe/xe_dma_buf.c
@@ -302,7 +302,7 @@ xe_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
 
 		bo = xe_bo_init_locked(xe, NULL, NULL, resv, NULL, dma_buf->size,
 				       0, /* Will require 1way or 2way for vm_bind */
-				       ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, &exec);
+				       ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, NULL, &exec);
 		drm_exec_retry_on_contention(&exec);
 		if (IS_ERR(bo)) {
 			ret = PTR_ERR(bo);
-- 
2.34.1

[PATCH v2 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl

[Matthew Brost]

Add a xe_gem_create_ioctl tracepoint to help profile GEM creation, in
particular the cost of populating a BO while holding the dma-resv lock
versus the total IOCTL cost. This is useful for evaluating the
out-of-lock preallocation path, which aims to shrink the time spent
under the lock.

The tracepoint records the BO size, the requested placement mask, the
CPU caching mode, the VM asid (0 when no VM is bound), the total time
spent in the IOCTL, and the time spent under the dma-resv lock (the
xe_validation_guard region where the lock is held).

The asid is captured at VM lookup time rather than at trace emission,
since the VM reference is dropped before the tracepoint fires.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c       | 16 ++++++++++++++++
 drivers/gpu/drm/xe/xe_trace_bo.h | 31 +++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index d9226a3995fe..c096f3dbd528 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -3929,6 +3929,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	bool have_prealloc = false;
 	unsigned int bo_flags;
 	u32 handle;
+	ktime_t ioctl_start = ktime_get();
+	ktime_t lock_start = ktime_set(0, 0);
+	u64 lock_ns = 0;
+	u32 asid = 0;
 	int err;
 
 	if (XE_IOCTL_DBG(xe, args->pad[0] || args->pad[1] || args->pad[2]) ||
@@ -4007,6 +4011,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 		vm = xe_vm_lookup(xef, args->vm_id);
 		if (XE_IOCTL_DBG(xe, !vm))
 			return -ENOENT;
+		asid = vm->usm.asid;
 	}
 
 	/*
@@ -4057,6 +4062,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			if (err)
 				break;
 		}
+
+		if (lock_start == ktime_set(0, 0))
+			lock_start = ktime_get();
+
 		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
 				       bo_flags, have_prealloc ? &prealloc : NULL,
 				       &exec);
@@ -4067,6 +4076,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			break;
 		}
 	}
+	lock_ns = ktime_to_ns(ktime_sub(ktime_get(), lock_start));
 	if (have_prealloc)
 		ttm_pool_prealloc_fini(&xe->ttm.pool, &prealloc);
 	if (err)
@@ -4097,6 +4107,12 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	if (vm)
 		xe_vm_put(vm);
 
+	trace_xe_gem_create_ioctl(xe, args->size, args->placement,
+				  args->cpu_caching, asid,
+				  ktime_to_ns(ktime_sub(ktime_get(),
+							ioctl_start)) / 1000,
+				  lock_ns / 1000);
+
 	return err;
 }
 
diff --git a/drivers/gpu/drm/xe/xe_trace_bo.h b/drivers/gpu/drm/xe/xe_trace_bo.h
index a169a322464b..b2dacb5fa3f7 100644
--- a/drivers/gpu/drm/xe/xe_trace_bo.h
+++ b/drivers/gpu/drm/xe/xe_trace_bo.h
@@ -144,6 +144,37 @@ TRACE_EVENT(xe_bo_defrag_one,
 		      __entry->ret)
 );
 
+TRACE_EVENT(xe_gem_create_ioctl,
+	    TP_PROTO(struct xe_device *xe, u64 size, u32 placement,
+		     u16 caching, u32 asid, u64 ioctl_us, u64 lock_us),
+	    TP_ARGS(xe, size, placement, caching, asid, ioctl_us, lock_us),
+
+	    TP_STRUCT__entry(
+		     __string(dev, dev_name(xe->drm.dev))
+		     __field(u64, size)
+		     __field(u32, placement)
+		     __field(u16, caching)
+		     __field(u32, asid)
+		     __field(u64, ioctl_us)
+		     __field(u64, lock_us)
+		     ),
+
+	    TP_fast_assign(
+		   __assign_str(dev);
+		   __entry->size = size;
+		   __entry->placement = placement;
+		   __entry->caching = caching;
+		   __entry->asid = asid;
+		   __entry->ioctl_us = ioctl_us;
+		   __entry->lock_us = lock_us;
+		   ),
+
+	    TP_printk("dev=%s, size=%llu, placement=0x%x, caching=%u, asid=%u, ioctl_us=%llu, lock_us=%llu",
+		      __get_str(dev), __entry->size, __entry->placement,
+		      __entry->caching, __entry->asid, __entry->ioctl_us,
+		      __entry->lock_us)
+);
+
 DECLARE_EVENT_CLASS(xe_vma,
 		    TP_PROTO(struct xe_vma *vma),
 		    TP_ARGS(vma),
-- 
2.34.1

[PATCH v2 27/33] drm/xe: Add IOVA-based xe_res_cursor variant

[Matthew Brost]

Introduce xe_res_first_iova() to walk a buffer that was DMA mapped using
the IOVA-based DMA API (dma_iova_try_alloc() and friends). Such a mapping
is described by a struct dma_iova_state and is a single contiguous IOVA
range, so the cursor walks it as one segment.

Add an is_iova flag to struct xe_res_cursor to distinguish this case.
When set, xe_res_dma() returns the IOVA backing the current position
(state->addr + cursor offset), xe_res_next() advances within the single
contiguous segment, and xe_res_is_vram() returns false since IOVA
mappings never point to same-device VRAM.

The flag is explicitly cleared in the existing xe_res_first(),
xe_res_first_sg() and xe_res_first_dma() initializers so stale stack
values cannot leak into the new path.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_res_cursor.h | 56 +++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_res_cursor.h b/drivers/gpu/drm/xe/xe_res_cursor.h
index 0522caafd89d..f0f1d699e16e 100644
--- a/drivers/gpu/drm/xe/xe_res_cursor.h
+++ b/drivers/gpu/drm/xe/xe_res_cursor.h
@@ -24,6 +24,7 @@
 #ifndef _XE_RES_CURSOR_H_
 #define _XE_RES_CURSOR_H_
 
+#include <linux/dma-mapping.h>
 #include <linux/scatterlist.h>
 
 #include <drm/drm_pagemap.h>
@@ -67,6 +68,12 @@ struct xe_res_cursor {
 	u64 dma_start;
 	/** @dma_seg_size: Size of the current DMA segment. */
 	u64 dma_seg_size;
+	/**
+	 * @is_iova: Whether the cursor walks a single contiguous IOVA
+	 * mapping described by a struct dma_iova_state. When true, the
+	 * DMA address is @dma_start + @start and never points to VRAM.
+	 */
+	bool is_iova;
 };
 
 static struct gpu_buddy *xe_res_get_buddy(struct ttm_resource *res)
@@ -93,6 +100,7 @@ static inline void xe_res_first(struct ttm_resource *res,
 {
 	cur->sgl = NULL;
 	cur->dma_addr = NULL;
+	cur->is_iova = false;
 	if (!res)
 		goto fallback;
 
@@ -224,6 +232,7 @@ static inline void xe_res_first_sg(const struct sg_table *sg,
 	cur->dma_addr = NULL;
 	cur->sgl = sg->sgl;
 	cur->mem_type = XE_PL_TT;
+	cur->is_iova = false;
 	__xe_res_sg_next(cur);
 }
 
@@ -255,6 +264,41 @@ static inline void xe_res_first_dma(const struct drm_pagemap_addr *dma_addr,
 	__xe_res_dma_next(cur);
 	cur->sgl = NULL;
 	cur->mem_type = XE_PL_TT;
+	cur->is_iova = false;
+}
+
+/**
+ * xe_res_first_iova() - initialize a xe_res_cursor with a dma_iova_state
+ *
+ * @state: struct dma_iova_state describing a single contiguous IOVA mapping
+ * @start: Start of the range
+ * @size: Size of the range
+ * @cur: cursor object to initialize
+ *
+ * Start walking over the range of an allocation that was DMA mapped using the
+ * IOVA-based DMA API (see dma_iova_try_alloc()). Such a mapping is a single
+ * contiguous IOVA range, so the cursor walks it as one segment. Subsequent
+ * calls to xe_res_next() advance within that range and xe_res_dma() returns
+ * the IOVA backing the current position. xe_res_is_vram() always returns false
+ * for such a cursor.
+ */
+static inline void xe_res_first_iova(struct dma_iova_state *state,
+				     u64 start, u64 size,
+				     struct xe_res_cursor *cur)
+{
+	XE_WARN_ON(!state);
+	XE_WARN_ON(start + size > dma_iova_size(state));
+
+	cur->node = NULL;
+	cur->start = start;
+	cur->remaining = size;
+	cur->size = size;
+	cur->dma_addr = NULL;
+	cur->sgl = NULL;
+	cur->dma_start = state->addr;
+	cur->dma_seg_size = dma_iova_size(state);
+	cur->mem_type = XE_PL_TT;
+	cur->is_iova = true;
 }
 
 /**
@@ -283,6 +327,13 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
 		return;
 	}
 
+	if (cur->is_iova) {
+		/* Single contiguous IOVA segment. */
+		cur->start += size;
+		cur->size = cur->remaining;
+		return;
+	}
+
 	if (cur->dma_addr) {
 		cur->start += size;
 		__xe_res_dma_next(cur);
@@ -334,7 +385,7 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
  */
 static inline u64 xe_res_dma(const struct xe_res_cursor *cur)
 {
-	if (cur->dma_addr)
+	if (cur->is_iova || cur->dma_addr)
 		return cur->dma_start + cur->start;
 	else if (cur->sgl)
 		return sg_dma_address(cur->sgl) + cur->start;
@@ -351,6 +402,9 @@ static inline u64 xe_res_dma(const struct xe_res_cursor *cur)
  */
 static inline bool xe_res_is_vram(const struct xe_res_cursor *cur)
 {
+	if (cur->is_iova)
+		return false;
+
 	if (cur->dma_addr)
 		return cur->dma_addr->proto == XE_INTERCONNECT_VRAM;
 
-- 
2.34.1

[PATCH v2 28/33] drm/xe: Use IOVA-based DMA mapping for eligible tt BOs

[Matthew Brost]

Add support for mapping a ttm_tt's pages using the IOVA-based DMA API
(dma_iova_try_alloc()/link()/sync()/unlink()/free()) as an alternative
to the scatter-gather list. An IOVA mapping is a single contiguous DMA
range, which is cheaper to walk and program than an sg list.

Whether the IOVA API can be used depends on the IOMMU mode, so the
decision is made once per tt at xe_ttm_tt_create() time and recorded in
struct xe_ttm_tt::use_iova. To keep the scope limited, only device BOs
that are not GGTT-mapped are eligible: dma-buf imports manage their own
mapping and GGTT bindings are programmed directly from the sg list.
dma_iova_try_alloc() may still fail, in which case the tt transparently
falls back to the sg list, which remains the default.

The IOVA reservation is allocated at tt creation and freed at tt
destroy. xe_tt_map_iova() links the tt pages (coalescing physically
contiguous runs) and syncs the IOTLB; xe_tt_unmap_iova() unlinks them
but keeps the reservation so the tt can be remapped without
re-allocating IOVA space. xe_tt_map()/xe_tt_unmap() dispatch between the
IOVA and sg backends, and xe_bo_dma_unmap_pinned() learns about the IOVA
case.

Consumers that walked the sg list via xe_res_first_sg(xe_bo_sg()) now go
through the new xe_res_first_tt() helper, which initializes either an
IOVA cursor (xe_res_first_iova()) or an sg cursor depending on the tt's
mapping mode.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/tests/xe_migrate.c |   6 +-
 drivers/gpu/drm/xe/xe_bo.c            | 239 ++++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_bo.h            |   5 +-
 drivers/gpu/drm/xe/xe_ggtt.c          |   2 +-
 drivers/gpu/drm/xe/xe_migrate.c       |  14 +-
 drivers/gpu/drm/xe/xe_pt.c            |   2 +-
 6 files changed, 237 insertions(+), 31 deletions(-)

diff --git a/drivers/gpu/drm/xe/tests/xe_migrate.c b/drivers/gpu/drm/xe/tests/xe_migrate.c
index 41de648b2c86..119134e67c89 100644
--- a/drivers/gpu/drm/xe/tests/xe_migrate.c
+++ b/drivers/gpu/drm/xe/tests/xe_migrate.c
@@ -253,7 +253,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test,
 	if (xe_bo_is_vram(pt))
 		xe_res_first(pt->ttm.resource, 0, xe_bo_size(pt), &src_it);
 	else
-		xe_res_first_sg(xe_bo_sg(pt), 0, xe_bo_size(pt), &src_it);
+		xe_res_first_tt(pt->ttm.ttm, 0, xe_bo_size(pt), &src_it);
 
 	emit_pte(m, bb, NUM_KERNEL_PDE - 1, xe_bo_is_vram(pt), false,
 		 &src_it, XE_PAGE_SIZE, pt->ttm.resource);
@@ -385,12 +385,12 @@ static struct dma_fence *blt_copy(struct xe_tile *tile,
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 
 	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_tt(src_bo->ttm.ttm, 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 
 	if (!dst_is_vram)
-		xe_res_first_sg(xe_bo_sg(dst_bo), 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index c096f3dbd528..f3d76ad9a105 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -6,6 +6,7 @@
 #include "xe_bo.h"
 
 #include <linux/dma-buf.h>
+#include <linux/dma-mapping.h>
 #include <linux/nospec.h>
 
 #include <drm/drm_drv.h>
@@ -15,6 +16,7 @@
 #include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_device.h>
 #include <drm/ttm/ttm_placement.h>
+#include <drm/ttm/ttm_pool.h>
 #include <drm/ttm/ttm_tt.h>
 #include <uapi/drm/xe_drm.h>
 
@@ -420,6 +422,32 @@ struct xe_ttm_tt {
 	struct ttm_tt ttm;
 	struct sg_table sgt;
 	struct sg_table *sg;
+	/**
+	 * @iova: IOVA state describing the single contiguous IOVA mapping of
+	 * the tt pages. Only valid when @use_iova is true.
+	 */
+	struct dma_iova_state iova;
+	/**
+	 * @use_iova: Whether this tt is DMA mapped using the IOVA-based DMA
+	 * API instead of a scatter-gather list. Decided once at tt creation
+	 * time and stable for the lifetime of the tt.
+	 */
+	bool use_iova;
+	/**
+	 * @iova_linked: Whether the tt pages are currently linked into the
+	 * IOVA mapping. Only meaningful when @use_iova is true.
+	 */
+	bool iova_linked;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	/**
+	 * @iova_dma_pages: Per-order count of DMA-mapped pages this tt
+	 * accounted into xe->mem.dma_mapped_pages[] for its IOVA mapping,
+	 * recorded at link time so the unlink can apply an exactly matching
+	 * decrement without re-walking tt->pages (which TTM may mutate, e.g.
+	 * during defrag). Counter ownership stays isolated in xe_bo.c.
+	 */
+	unsigned long iova_dma_pages[NR_PAGE_ORDERS];
+#endif
 	/** @purgeable: Whether the content of the pages of @ttm is purgeable. */
 	bool purgeable;
 };
@@ -468,6 +496,52 @@ static void xe_tt_account_dma_pages(struct xe_device *xe,
 }
 #endif
 
+/*
+ * xe_tt_account_iova_add - account one freshly linked IOVA folio
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt being mapped
+ * @order: page order of the linked folio
+ *
+ * Add the folio's 2^order pages to xe->mem.dma_mapped_pages[order] and record
+ * the same amount in @xe_tt so xe_tt_unaccount_iova() can later subtract an
+ * exactly matching count without re-walking tt->pages.
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_iova_add(struct xe_device *xe,
+				   struct xe_ttm_tt *xe_tt, unsigned int order)
+{
+	unsigned long chunk = 1UL << order;
+
+	xe_tt->iova_dma_pages[order] += chunk;
+	atomic_long_add(chunk, &xe->mem.dma_mapped_pages[order]);
+}
+
+static void xe_tt_unaccount_iova(struct xe_device *xe,
+				 struct xe_ttm_tt *xe_tt)
+{
+	unsigned int order;
+
+	for (order = 0; order < NR_PAGE_ORDERS; order++) {
+		unsigned long pages = xe_tt->iova_dma_pages[order];
+
+		if (pages) {
+			atomic_long_sub(pages, &xe->mem.dma_mapped_pages[order]);
+			xe_tt->iova_dma_pages[order] = 0;
+		}
+	}
+}
+#else
+static void xe_tt_account_iova_add(struct xe_device *xe,
+				   struct xe_ttm_tt *xe_tt, unsigned int order)
+{
+}
+
+static void xe_tt_unaccount_iova(struct xe_device *xe,
+				 struct xe_ttm_tt *xe_tt)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -514,28 +588,129 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	}
 }
 
-struct sg_table *xe_bo_sg(struct xe_bo *bo)
+/*
+ * xe_tt_map_iova - DMA map a tt using the IOVA-based DMA API.
+ *
+ * Link all of the tt's pages into the IOVA range reserved at tt creation
+ * time, one folio at a time, and sync the IOTLB once at the end. The IOVA
+ * reservation itself is kept across map/unmap cycles and only released when
+ * the tt is destroyed.
+ */
+static int xe_tt_map_iova(struct xe_device *xe, struct ttm_tt *tt)
 {
-	struct ttm_tt *tt = bo->ttm.ttm;
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+	struct device *dev = xe->drm.dev;
+	unsigned long num_pages = tt->num_pages;
+	unsigned long i = 0;
+	size_t mapped = 0;
+	int ret;
+
+	if (xe_tt->iova_linked)
+		return 0;
+
+	while (i < num_pages) {
+		struct page *page = tt->pages[i];
+		unsigned int order = ttm_pool_page_order_nodma(page);
+		size_t len = PAGE_SIZE << order;
+
+		ret = dma_iova_link(dev, &xe_tt->iova, page_to_phys(page),
+				    mapped, len, DMA_BIDIRECTIONAL,
+				    DMA_ATTR_SKIP_CPU_SYNC);
+		if (ret)
+			goto err_unlink;
 
-	return xe_tt->sg;
+		xe_tt_account_iova_add(xe, xe_tt, order);
+		mapped += len;
+		i += 1UL << order;
+	}
+
+	ret = dma_iova_sync(dev, &xe_tt->iova, 0, mapped);
+	if (ret)
+		goto err_unlink;
+
+	xe_tt->iova_linked = true;
+	return 0;
+
+err_unlink:
+	if (mapped) {
+		dma_iova_unlink(dev, &xe_tt->iova, 0, mapped,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+		xe_tt_unaccount_iova(xe, xe_tt);
+	}
+	return ret;
 }
 
-/**
- * xe_tt_sg() - Return the DMA scatter-gather table backing a ttm_tt.
- * @tt: The ttm_tt to query.
+/*
+ * xe_tt_unmap_iova - Unlink a tt's pages from its IOVA mapping.
  *
- * Like xe_bo_sg(), but takes the ttm_tt directly. Used by the defrag copy
- * path which needs the old (source) tt's sg table without a backing xe_bo.
+ * Unlink the pages but keep the IOVA reservation so the tt can be remapped
+ * later without re-allocating IOVA space.
+ */
+static void xe_tt_unmap_iova(struct xe_device *xe, struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->iova_linked) {
+		/*
+		 * Unmap with attrs 0 (matching the sg unmap path) so that the
+		 * CPU copy-back sync is performed for non-coherent or bounced
+		 * (SWIOTLB) mappings, even though the link was done with
+		 * DMA_ATTR_SKIP_CPU_SYNC.
+		 */
+		dma_iova_unlink(xe->drm.dev, &xe_tt->iova, 0,
+				dma_iova_size(&xe_tt->iova),
+				DMA_BIDIRECTIONAL, 0);
+		xe_tt_unaccount_iova(xe, xe_tt);
+		xe_tt->iova_linked = false;
+	}
+}
+
+/*
+ * xe_tt_map - DMA map a tt, using IOVA when available and sg otherwise.
+ */
+static int xe_tt_map(struct xe_device *xe, struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->use_iova)
+		return xe_tt_map_iova(xe, tt);
+
+	return xe_tt_map_sg(xe, tt);
+}
+
+/*
+ * xe_tt_unmap - Undo xe_tt_map().
+ */
+static void xe_tt_unmap(struct xe_device *xe, struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->use_iova)
+		xe_tt_unmap_iova(xe, tt);
+	else
+		xe_tt_unmap_sg(xe, tt);
+}
+
+/**
+ * xe_res_first_tt() - Initialize a resource cursor over a ttm_tt's DMA mapping.
+ * @tt: The ttm_tt to walk.
+ * @start: Start of the range, in bytes.
+ * @size: Size of the range, in bytes.
+ * @cur: The cursor to initialize.
  *
- * Return: The sg table, or NULL if the tt is not currently mapped.
+ * Initialize @cur to walk the DMA addresses backing @tt, transparently
+ * handling both the IOVA-based mapping (single contiguous range) and the
+ * scatter-gather list fallback.
  */
-struct sg_table *xe_tt_sg(struct ttm_tt *tt)
+void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
+		     struct xe_res_cursor *cur)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
-	return xe_tt->sg;
+	if (xe_tt->use_iova)
+		xe_res_first_iova(&xe_tt->iova, start, size, cur);
+	else
+		xe_res_first_sg(xe_tt->sg, start, size, cur);
 }
 
 /*
@@ -651,6 +826,21 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
 		}
 	}
 
+	/*
+	 * Eligible BOs are DMA mapped using the IOVA-based DMA API, which
+	 * yields a single contiguous IOVA range instead of a scatter-gather
+	 * list. Only device BOs that are not GGTT-mapped qualify: dma-buf
+	 * imports manage their own mapping and GGTT bindings are programmed
+	 * directly from the sg list. dma_iova_try_alloc() may still fail
+	 * depending on the IOMMU mode, in which case we transparently fall
+	 * back to the sg list. The reservation lives for the tt's lifetime.
+	 */
+	if (ttm_bo->type != ttm_bo_type_sg &&
+	    !(bo->flags & (XE_BO_FLAG_GGTT | XE_BO_FLAG_GGTT_ALL)) &&
+	    dma_iova_try_alloc(xe->drm.dev, &xe_tt->iova, 0,
+			       (size_t)tt->num_pages << PAGE_SHIFT))
+		xe_tt->use_iova = true;
+
 	return tt;
 }
 
@@ -700,7 +890,7 @@ static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
 		return;
 
-	xe_tt_unmap_sg(xe, tt);
+	xe_tt_unmap(xe, tt);
 
 	ttm_pool_free(&ttm_dev->pool, tt);
 	xe_ttm_tt_account_subtract(xe, tt);
@@ -709,6 +899,19 @@ static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 
 static void xe_ttm_tt_destroy(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 {
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	/*
+	 * Release the IOVA reservation made in xe_ttm_tt_create(). The pages
+	 * must already have been unlinked by xe_tt_unmap().
+	 */
+	if (xe_tt->use_iova) {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		xe_assert(xe, !xe_tt->iova_linked);
+		dma_iova_free(xe->drm.dev, &xe_tt->iova);
+	}
+
 	ttm_tt_fini(tt);
 	kfree(tt);
 }
@@ -1533,7 +1736,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	/* Bo creation path, moving to system or TT. */
 	if ((!old_mem && ttm) && !handle_system_ccs) {
 		if (new_mem->mem_type == XE_PL_TT)
-			ret = xe_tt_map_sg(xe, ttm);
+			ret = xe_tt_map(xe, ttm);
 		if (!ret)
 			ttm_bo_move_null(ttm_bo, new_mem);
 		goto out;
@@ -1563,7 +1766,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
 	if (new_mem->mem_type == XE_PL_TT) {
-		ret = xe_tt_map_sg(xe, ttm);
+		ret = xe_tt_map(xe, ttm);
 		if (ret)
 			goto out;
 	}
@@ -1733,7 +1936,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		if (IS_VF_CCS_READY(xe))
 			xe_sriov_vf_ccs_detach_bo(bo);
 
-		xe_tt_unmap_sg(xe, ttm_bo->ttm);
+		xe_tt_unmap(xe, ttm_bo->ttm);
 	}
 
 	xe_bo_defrag_update(bo);
@@ -1761,7 +1964,7 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 		if (lret)
 			return lret;
 
-		xe_tt_unmap_sg(xe, bo->ttm);
+		xe_tt_unmap(xe, bo->ttm);
 		ttm_bo_move_null(bo, new_resource);
 	}
 
@@ -2188,6 +2391,8 @@ int xe_bo_dma_unmap_pinned(struct xe_bo *bo)
 						 DMA_BIDIRECTIONAL);
 			ttm_bo->sg = NULL;
 			xe_tt->sg = NULL;
+		} else if (xe_tt->use_iova) {
+			xe_tt_unmap_iova(ttm_to_xe_device(ttm_bo->bdev), tt);
 		} else if (xe_tt->sg) {
 			dma_unmap_sgtable(ttm_to_xe_device(ttm_bo->bdev)->drm.dev,
 					  xe_tt->sg,
@@ -3755,7 +3960,7 @@ dma_addr_t __xe_bo_addr(struct xe_bo *bo, u64 offset, size_t page_size)
 	if (!xe_bo_is_vram(bo) && !xe_bo_is_stolen(bo)) {
 		xe_assert(xe, bo->ttm.ttm);
 
-		xe_res_first_sg(xe_bo_sg(bo), page << PAGE_SHIFT,
+		xe_res_first_tt(bo->ttm.ttm, page << PAGE_SHIFT,
 				page_size, &cur);
 		return xe_res_dma(&cur) + offset;
 	} else {
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 18fcccd082b7..3e2eaefecb8f 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -552,8 +552,9 @@ void xe_bo_dev_init(struct xe_bo_dev *bo_device);
 
 void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
 
-struct sg_table *xe_bo_sg(struct xe_bo *bo);
-struct sg_table *xe_tt_sg(struct ttm_tt *tt);
+struct xe_res_cursor;
+void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
+		     struct xe_res_cursor *cur);
 
 /*
  * xe_sg_segment_size() - Provides upper limit for sg segment size.
diff --git a/drivers/gpu/drm/xe/xe_ggtt.c b/drivers/gpu/drm/xe/xe_ggtt.c
index 8ec23862477f..4084bc183e06 100644
--- a/drivers/gpu/drm/xe/xe_ggtt.c
+++ b/drivers/gpu/drm/xe/xe_ggtt.c
@@ -699,7 +699,7 @@ static void xe_ggtt_map_bo(struct xe_ggtt *ggtt, struct xe_ggtt_node *node,
 	if (!xe_bo_is_vram(bo) && !xe_bo_is_stolen(bo)) {
 		xe_assert(xe_bo_device(bo), bo->ttm.ttm);
 
-		for (xe_res_first_sg(xe_bo_sg(bo), 0, xe_bo_size(bo), &cur);
+		for (xe_res_first_tt(bo->ttm.ttm, 0, xe_bo_size(bo), &cur);
 		     cur.remaining; xe_res_next(&cur, XE_PAGE_SIZE))
 			ggtt->pt_ops->ggtt_set_pte(ggtt, end - cur.remaining,
 						   pte | xe_res_dma(&cur));
diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index c1e96f105648..80fff1b5b3aa 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -1005,16 +1005,16 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_tt(src_tt ? src_tt : src_bo->ttm.ttm, 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
-		xe_res_first_sg(xe_bo_sg(dst_bo), 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo),
+		xe_res_first_tt(src_tt ? src_tt : src_bo->ttm.ttm,
 				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
@@ -1383,9 +1383,9 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
 
 	ctx = &xe->sriov.vf.ccs.contexts[read_write];
 
-	xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+	xe_res_first_tt(src_bo->ttm.ttm, 0, size, &src_it);
 
-	xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+	xe_res_first_tt(src_bo->ttm.ttm, xe_bo_ccs_pages_start(src_bo),
 			PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 			&ccs_it);
 
@@ -1588,7 +1588,7 @@ struct dma_fence *xe_migrate_vram_copy_chunk(struct xe_bo *vram_bo, u64 vram_off
 	xe_assert(xe, !range_overflows(sysmem_offset, size, (u64)sysmem_bo->ttm.base.size));
 
 	xe_res_first(vram, vram_offset, size, &vram_it);
-	xe_res_first_sg(xe_bo_sg(sysmem_bo), sysmem_offset, size, &sysmem_it);
+	xe_res_first_tt(sysmem_bo->ttm.ttm, sysmem_offset, size, &sysmem_it);
 
 	while (size) {
 		u32 pte_flags = PTE_UPDATE_FLAG_IS_VRAM;
@@ -1800,7 +1800,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,
 		clear_only_system_ccs = true;
 
 	if (!clear_vram)
-		xe_res_first_sg(xe_bo_sg(bo), 0, xe_bo_size(bo), &src_it);
+		xe_res_first_tt(bo->ttm.ttm, 0, xe_bo_size(bo), &src_it);
 	else
 		xe_res_first(src, 0, xe_bo_size(bo), &src_it);
 
diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c
index 7111b5d0f6aa..185d15fb0a97 100644
--- a/drivers/gpu/drm/xe/xe_pt.c
+++ b/drivers/gpu/drm/xe/xe_pt.c
@@ -797,7 +797,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,
 			xe_res_first(bo->ttm.resource, xe_vma_bo_offset(vma),
 				     xe_vma_size(vma), &curs);
 		else
-			xe_res_first_sg(xe_bo_sg(bo), xe_vma_bo_offset(vma),
+			xe_res_first_tt(bo->ttm.ttm, xe_vma_bo_offset(vma),
 					xe_vma_size(vma), &curs);
 	} else if (!range) {
 		curs.size = xe_vma_size(vma);
-- 
2.34.1

[PATCH v2 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

[Matthew Brost]

A future patch performs the destination DMA mapping of a defrag move in
two stages when the BO is IOVA mapped: a temporary contiguous mapping of
just tht changed pages used for the GPU copy, followed by a post-copy
finalize stet that tears that mapping down and links the new backing
into the (transferred) IOVA reservation. The finalize step must run only
after the GPU copy it depends on has completed, so it is dispatched as
an xe_dep_job,

Add a per-device xe_dep_scheduler dedicated to these finalize jobs,
created in xe_bo_defrag_init() (after the migrate contexts are up) and
torn down in xe_bo_defrag_fini(). No users yet; the job itself is added
in a later patch.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c           | 15 +++++++++++++++
 drivers/gpu/drm/xe/xe_device_types.h | 10 ++++++++++
 2 files changed, 25 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index f3d76ad9a105..ec034518e2ea 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -25,6 +25,7 @@
 #include <trace/events/gpu_mem.h>
 
 #include "xe_device.h"
+#include "xe_dep_scheduler.h"
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
 #include "xe_ggtt.h"
@@ -1300,6 +1301,10 @@ static void xe_bo_defrag_fini(void *arg)
 	struct xe_device *xe = arg;
 
 	disable_delayed_work_sync(&xe->mem.defrag.worker);
+	if (xe->mem.defrag.iova_sched) {
+		xe_dep_scheduler_fini(xe->mem.defrag.iova_sched);
+		xe->mem.defrag.iova_sched = NULL;
+	}
 }
 
 /**
@@ -1336,6 +1341,16 @@ void xe_bo_defrag_init_early(struct xe_device *xe)
  */
 int xe_bo_defrag_init(struct xe_device *xe)
 {
+#define XE_BO_MAX_IOVA_DEFRAG_JOBS	16	/* Picking a reasonable value */
+	struct xe_dep_scheduler *iova_sched;
+
+	iova_sched = xe_dep_scheduler_create(xe, NULL, "xe_iova_defrag",
+					     XE_BO_MAX_IOVA_DEFRAG_JOBS);
+	if (IS_ERR(iova_sched))
+		return PTR_ERR(iova_sched);
+	xe->mem.defrag.iova_sched = iova_sched;
+#undef XE_BO_MAX_IOVA_DEFRAG_JOBS
+
 	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
 }
 
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 6e68add4d652..b55d549d5e30 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -38,6 +38,7 @@
 
 struct drm_pagemap_shrinker;
 struct intel_display;
+struct xe_dep_scheduler;
 struct intel_dg_nvm_dev;
 struct xe_ggtt;
 struct xe_i2c;
@@ -325,6 +326,15 @@ struct xe_device {
 			 * @mem.defrag.worker, in milliseconds.
 			 */
 			unsigned int interval_ms;
+			/**
+			 * @mem.defrag.iova_sched: Dependency scheduler used to
+			 * run the post-copy IOVA finalize job for defrag moves
+			 * of IOVA-mapped BOs. The job tears down the temporary
+			 * copy-step IOVA mapping and links the new backing into
+			 * the (transferred) reservation once the GPU copy that
+			 * its dependencies gate has completed.
+			 */
+			struct xe_dep_scheduler *iova_sched;
 		} defrag;
 	} mem;
 
-- 
2.34.1

[PATCH v2 30/33] drm/xe: Add packed copy-step IOVA mapping for defrag

[Matthew Brost]

Add machinery to map only changed pages during defrag when the
destination TT is IOVA-mapped. Only the non-borrowed (freshly
allocated) pages need copying; borrowed pages are aliases from
the old TT.

Add xe_tt_iova_copy_init() to diff new vs old TT folios and
record changed-page runs. Add xe_tt_map_iova_copy() to link
those changed pages packed contiguously at the front of the
new TT's IOVA reservation, synced and ready for GPU blits.

No users yet; wired up in later patches.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 364 +++++++++++++++++++++++++++++++++++++
 1 file changed, 364 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index ec034518e2ea..16d9e135cba2 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -531,6 +531,31 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 		}
 	}
 }
+
+/*
+ * xe_tt_account_iova_pages - account a tt's pages from a full folio walk
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt to account (its @iova_dma_pages must start zeroed)
+ *
+ * Walk @xe_tt's pages one folio at a time (the same head-page order walk used
+ * by xe_tt_map_iova()) and account each into xe->mem.dma_mapped_pages[] and
+ * @xe_tt->iova_dma_pages[]. Used to (re)derive accounting that exactly matches
+ * the pages a tt is actually mapped with.
+ */
+__maybe_unused
+static void xe_tt_account_iova_pages(struct xe_device *xe,
+				     struct xe_ttm_tt *xe_tt)
+{
+	struct ttm_tt *tt = &xe_tt->ttm;
+	unsigned long i = 0;
+
+	while (i < tt->num_pages) {
+		unsigned int order = ttm_pool_page_order_nodma(tt->pages[i]);
+
+		xe_tt_account_iova_add(xe, xe_tt, order);
+		i += 1UL << order;
+	}
+}
 #else
 static void xe_tt_account_iova_add(struct xe_device *xe,
 				   struct xe_ttm_tt *xe_tt, unsigned int order)
@@ -541,6 +566,12 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 				 struct xe_ttm_tt *xe_tt)
 {
 }
+
+__maybe_unused
+static void xe_tt_account_iova_pages(struct xe_device *xe,
+				     struct xe_ttm_tt *xe_tt)
+{
+}
 #endif
 
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
@@ -714,6 +745,339 @@ void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
 		xe_res_first_sg(xe_tt->sg, start, size, cur);
 }
 
+/**
+ * struct xe_tt_iova_run - A contiguous run of changed pages in a defrag move
+ * @start: First page offset (natural ttm_tt index) of the run.
+ * @count: Number of pages in the run.
+ *
+ * Runs are folio aligned: defrag borrows whole beneficial-order chunks, so a
+ * folio is either entirely borrowed (unchanged) or entirely freshly allocated
+ * (changed), never split across the boundary.
+ */
+struct xe_tt_iova_run {
+	pgoff_t start;
+	pgoff_t count;
+};
+
+/**
+ * struct xe_tt_iova_link - A snapshot folio link descriptor for defrag relink
+ * @phys: Physical address of the folio's head page.
+ * @offset: Natural byte offset into the reservation to link the folio at.
+ * @len: Byte length of the folio (PAGE_SIZE << order).
+ *
+ * Captured at copy-init time while the destination tt's page array is valid and
+ * stable under the BO lock. The post-copy finalize job runs asynchronously,
+ * after TTM may have mutated (or torn down) the destination tt page array, so
+ * it relinks purely from these snapshots and never dereferences that array.
+ */
+struct xe_tt_iova_link {
+	phys_addr_t phys;
+	size_t offset;
+	size_t len;
+};
+
+/**
+ * struct xe_tt_iova_copy - Changed-page subset of an IOVA defrag move
+ * @runs: Array of changed-page runs, in ascending natural tt offset.
+ * @nr_runs: Number of entries in @runs.
+ * @links: Per-folio link snapshots covering the changed runs, in ascending
+ *         natural offset. Drives the asynchronous finalize relink so it never
+ *         touches the destination tt's (possibly mutated) page array.
+ * @nr_links: Number of entries in @links.
+ * @nr_changed: Total number of changed pages (sum of the run counts). This is
+ *              also the size, in pages, of the packed copy-step IOVA mapping
+ *              produced by xe_tt_map_iova_copy().
+ */
+struct xe_tt_iova_copy {
+	struct xe_tt_iova_run *runs;
+	unsigned int nr_runs;
+	struct xe_tt_iova_link *links;
+	unsigned int nr_links;
+	pgoff_t nr_changed;
+};
+
+/*
+ * xe_tt_iova_walk_changed - Walk a defrag move's changed-page runs.
+ *
+ * Iterate @new_tt one folio at a time (folio order read from TTM via
+ * ttm_pool_page_order_nodma()) and compare each folio's head page against
+ * @old_tt at the same natural offset. Pages the new backing borrowed from the
+ * old tt alias the same struct page and are unchanged; freshly allocated pages
+ * differ. Coalesce contiguous changed folios into runs and, for each run,
+ * invoke @fn. Returns the total number of changed pages.
+ */
+__maybe_unused
+static pgoff_t xe_tt_iova_walk_changed(struct ttm_tt *new_tt,
+				       struct ttm_tt *old_tt,
+				       void (*fn)(pgoff_t start, pgoff_t count,
+						  void *arg),
+				       void *arg)
+{
+	pgoff_t i = 0, n = new_tt->num_pages;
+	pgoff_t run_start = 0, nr_changed = 0;
+	bool in_run = false;
+
+	while (i < n) {
+		unsigned int order = ttm_pool_page_order_nodma(new_tt->pages[i]);
+		pgoff_t nr = 1UL << order;
+		bool changed = new_tt->pages[i] != old_tt->pages[i];
+
+		if (changed) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+			nr_changed += nr;
+		} else if (in_run) {
+			if (fn)
+				fn(run_start, i - run_start, arg);
+			in_run = false;
+		}
+
+		i += nr;
+	}
+
+	if (in_run && fn)
+		fn(run_start, n - run_start, arg);
+
+	return nr_changed;
+}
+
+struct xe_tt_iova_run_collect {
+	struct ttm_tt *new_tt;
+	struct xe_tt_iova_run *runs;
+	unsigned int nr_runs;
+	unsigned int runs_cap;
+	struct xe_tt_iova_link *links;
+	unsigned int nr_links;
+	unsigned int links_cap;
+	int err;
+};
+
+/* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
+__maybe_unused
+static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
+			   size_t elem)
+{
+	unsigned int new_cap;
+	void *p;
+
+	if (need <= *cap)
+		return 0;
+
+	new_cap = *cap ? *cap * 2 : 64;
+	if (new_cap < need)
+		new_cap = need;
+
+	p = krealloc_array(*arr, new_cap, elem, GFP_KERNEL);
+	if (!p)
+		return -ENOMEM;
+
+	*arr = p;
+	*cap = new_cap;
+	return 0;
+}
+
+__maybe_unused
+static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
+{
+	struct xe_tt_iova_run_collect *c = arg;
+	pgoff_t i = start, end = start + count;
+
+	if (c->err)
+		return;
+
+	if (xe_tt_iova_grow((void **)&c->runs, &c->runs_cap, c->nr_runs + 1,
+			    sizeof(*c->runs))) {
+		c->err = -ENOMEM;
+		return;
+	}
+	c->runs[c->nr_runs].start = start;
+	c->runs[c->nr_runs].count = count;
+	c->nr_runs++;
+
+	while (i < end) {
+		struct page *page = c->new_tt->pages[i];
+		unsigned int order = ttm_pool_page_order_nodma(page);
+
+		if (xe_tt_iova_grow((void **)&c->links, &c->links_cap,
+				    c->nr_links + 1, sizeof(*c->links))) {
+			c->err = -ENOMEM;
+			return;
+		}
+		c->links[c->nr_links].phys = page_to_phys(page);
+		c->links[c->nr_links].offset = (size_t)i << PAGE_SHIFT;
+		c->links[c->nr_links].len = PAGE_SIZE << order;
+		c->nr_links++;
+
+		i += 1UL << order;
+	}
+}
+
+/**
+ * xe_tt_iova_copy_init() - Compute the changed-page subset of a defrag move
+ * @new_tt: The freshly (re)allocated destination tt.
+ * @old_tt: The stashed source tt (bo->defrag_old_tt).
+ * @copy: Output description of the changed pages.
+ *
+ * Diff @new_tt against @old_tt and record the runs of changed (non-borrowed)
+ * pages into @copy, along with a per-folio snapshot of their physical
+ * addresses, natural offsets and lengths. The result drives both the packed
+ * copy-step IOVA mapping (xe_tt_map_iova_copy()) and the post-copy finalize
+ * that relinks the new backing into the IOVA reservation. The finalize runs
+ * asynchronously, after TTM may have mutated @new_tt->pages, so it consumes the
+ * snapshot rather than re-reading the page array. @new_tt and @old_tt must
+ * describe the same BO (equal num_pages) and both be populated.
+ *
+ * The caller owns @copy and must release it with xe_tt_iova_copy_fini().
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+__maybe_unused
+static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
+				struct xe_tt_iova_copy *copy)
+{
+	struct xe_tt_iova_run_collect c = { .new_tt = new_tt };
+
+	memset(copy, 0, sizeof(*copy));
+
+	/*
+	 * Single pass: collect the changed runs and per-folio link snapshots
+	 * directly, growing the arrays on demand. Changed runs are few (the
+	 * borrowed majority is skipped), so the geometric reallocs are rare and
+	 * this avoids a second full O(num_pages) walk just to size the arrays.
+	 */
+	copy->nr_changed = xe_tt_iova_walk_changed(new_tt, old_tt,
+						   xe_tt_iova_store_run, &c);
+	if (c.err) {
+		kfree(c.runs);
+		kfree(c.links);
+		return c.err;
+	}
+	if (!c.nr_runs)
+		return 0;
+
+	copy->runs = c.runs;
+	copy->nr_runs = c.nr_runs;
+	copy->links = c.links;
+	copy->nr_links = c.nr_links;
+
+	return 0;
+}
+
+/**
+ * xe_tt_iova_copy_fini() - Release a struct xe_tt_iova_copy
+ * @copy: The copy description to release.
+ */
+__maybe_unused
+static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
+{
+	kfree(copy->runs);
+	kfree(copy->links);
+	copy->runs = NULL;
+	copy->links = NULL;
+	copy->nr_runs = 0;
+	copy->nr_links = 0;
+	copy->nr_changed = 0;
+}
+
+/**
+ * xe_tt_map_iova_copy() - DMA map the changed-page subset for a defrag copy
+ * @xe: The xe device.
+ * @new_tt: The destination tt (must be IOVA eligible, use_iova == true).
+ * @copy: The changed-page description from xe_tt_iova_copy_init().
+ *
+ * Link only the changed pages of @new_tt into its IOVA reservation, packed
+ * contiguously at the front of the range ([0, @copy->nr_changed) pages). This
+ * provides a compact, contiguous destination for the GPU defrag copy without
+ * mapping the (majority) borrowed pages, and lets the whole temporary mapping
+ * later be torn down with a single dma_iova_destroy().
+ *
+ * The reservation's remaining tail is left unlinked. This does not establish
+ * the tt's final mapping (@iova_linked stays false); the post-copy finalize is
+ * responsible for that.
+ *
+ * Return: 0 on success, negative error code on failure (any partial linkage is
+ * unwound).
+ */
+__maybe_unused
+static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
+			       const struct xe_tt_iova_copy *copy)
+{
+	struct xe_ttm_tt *xe_tt = container_of(new_tt, struct xe_ttm_tt, ttm);
+	struct device *dev = xe->drm.dev;
+	size_t mapped = 0;
+	unsigned int r;
+	int ret;
+
+	xe_assert(xe, xe_tt->use_iova);
+	xe_assert(xe, !xe_tt->iova_linked);
+
+	for (r = 0; r < copy->nr_runs; r++) {
+		pgoff_t i = copy->runs[r].start;
+		pgoff_t end = i + copy->runs[r].count;
+
+		while (i < end) {
+			struct page *page = new_tt->pages[i];
+			unsigned int order = ttm_pool_page_order_nodma(page);
+			size_t len = PAGE_SIZE << order;
+
+			ret = dma_iova_link(dev, &xe_tt->iova,
+					    page_to_phys(page), mapped, len,
+					    DMA_BIDIRECTIONAL,
+					    DMA_ATTR_SKIP_CPU_SYNC);
+			if (ret)
+				goto err_unlink;
+
+			mapped += len;
+			i += 1UL << order;
+		}
+	}
+
+	ret = dma_iova_sync(dev, &xe_tt->iova, 0, mapped);
+	if (ret)
+		goto err_unlink;
+
+	return 0;
+
+err_unlink:
+	if (mapped)
+		dma_iova_unlink(dev, &xe_tt->iova, 0, mapped,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+	return ret;
+}
+
+/*
+ * xe_tt_iova_transfer - Transfer a full IOVA reservation old_tt -> new_tt.
+ *
+ * Move the (fully linked) IOVA reservation from the old tt to the new tt,
+ * leaving the old tt without a reservation so its destroy path does not double
+ * free. The new tt keeps its existing reservation snapshotted in the finalize
+ * job for teardown. Called at the defrag finalize job's point of no failure.
+ *
+ * Also rebalance the per-order DMA-page accounting: drop the old tt's
+ * accounting and account the pages the new tt actually ends up mapped with.
+ * The finalize relink leaves the new tt's mapping equal to new_tt->pages
+ * (borrowed pages alias the old tt; changed runs use the new pages, which may
+ * have a different order distribution), so deriving the accounting from a full
+ * folio walk of the new tt keeps xe->mem.dma_mapped_pages[] exact while the new
+ * mapping lives, rather than carrying the old tt's stale distribution.
+ */
+__maybe_unused
+static void xe_tt_iova_transfer(struct xe_device *xe,
+				struct xe_ttm_tt *new_tt,
+				struct xe_ttm_tt *old_tt)
+{
+	new_tt->iova = old_tt->iova;
+	new_tt->iova_linked = true;
+
+	xe_tt_unaccount_iova(xe, old_tt);
+	xe_tt_account_iova_pages(xe, new_tt);
+
+	old_tt->iova_linked = false;
+	old_tt->use_iova = false;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
-- 
2.34.1

[PATCH v2 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

[Matthew Brost]

A defrag move with IOVA enabled only DMA-maps the changed (non-borrowed)
pages of the new tt, packed contiguously into the front of its IOVA
reservation (xe_tt_map_iova_copy()). The GPU copy therefore has an
asymmetric geometry: the source is read from the old tt at its natural
page offsets (via the old tt's full IOVA), while the destination only
exposes the changed pages, packed.

Teach __xe_migrate_copy() about this packed-destination mode:

- Add a @dst_packed_pages argument. When non-zero the destination tt
  cursor walks just that many packed pages and the source (not the
  destination) drives the chunk size, so the smaller packed prefix never
  clamps the source walk.

- Size the destination PTE update for each chunk from the changed-page
  count (xe_migrate_count_changed_pages()) rather than the full chunk,
  and skip the destination PTEs entirely for an all-borrowed chunk.

- Extend emit_copy_chunks() with a @pack_dst flag so each emitted run is
  blitted from its natural source offset into a running packed
  destination offset that only advances for changed runs. The
  all-borrowed skip leaves the packed destination cursor untouched.

Expose this via xe_migrate_copy_defrag_iova(). It is only valid when the
BO needs no CCS pass, since the separate CCS pass migrates aux state for
all pages and would require the full destination mapping. The existing
xe_migrate_copy_defrag() (full destination mapping) remains the path for
CCS BOs and the non-IOVA fallback.

No users yet; xe_bo_move() is wired to this path in a later patch.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_migrate.c | 307 ++++++++++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_migrate.h |   8 +
 2 files changed, 296 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 80fff1b5b3aa..f285a2a5a075 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -811,6 +811,64 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
 		emit_xy_fast_copy(gt, bb, src_ofs, dst_ofs, size, pitch);
 }
 
+/*
+ * xe_migrate_count_changed_pages() - Count non-borrowed pages in a defrag chunk.
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ * @npages: Number of pages in the L0 chunk.
+ *
+ * A defragmentation move may harvest already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) some of the same physical pages.
+ * Count the pages that actually differ (src page != dst page) within the chunk.
+ *
+ * Return: The number of non-borrowed (changed) pages.
+ */
+static u32 xe_migrate_count_changed_pages(struct ttm_tt *src_tt,
+					  struct ttm_tt *dst_tt,
+					  pgoff_t start_page, u32 npages)
+{
+	u32 i, nchg = 0;
+
+	for (i = 0; i < npages; i++)
+		if (src_tt->pages[start_page + i] !=
+		    dst_tt->pages[start_page + i])
+			nchg++;
+
+	return nchg;
+}
+
+/*
+ * xe_migrate_leading_borrowed_pages() - Count the run of borrowed pages at the
+ * head of the remaining copy.
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of the next page to copy within the tt page arrays.
+ * @max_pages: Pages left to copy.
+ *
+ * A defragmentation move borrows the already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) the same physical pages for the
+ * unchanged majority of a large object. Those pages alias on both the source
+ * and destination GPU mappings and need no copy. Count how many consecutive
+ * pages starting at @start_page are borrowed so the caller can advance past the
+ * whole run at once instead of rediscovering it one L0 chunk at a time.
+ *
+ * Return: The number of leading borrowed (src page == dst page) pages, capped
+ * at @max_pages.
+ */
+static u32 xe_migrate_leading_borrowed_pages(struct ttm_tt *src_tt,
+					     struct ttm_tt *dst_tt,
+					     pgoff_t start_page, u32 max_pages)
+{
+	u32 n = 0;
+
+	while (n < max_pages &&
+	       src_tt->pages[start_page + n] == dst_tt->pages[start_page + n])
+		n++;
+
+	return n;
+}
+
 /*
  * emit_copy_chunks() - Emit the data blits for one defrag L0 chunk, skipping
  * borrowed pages.
@@ -822,6 +880,9 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
  * @src_tt: Source (old) tt providing the page pointers.
  * @dst_tt: Destination (new) tt providing the page pointers.
  * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ * @pack_dst: If true the destination only maps the changed pages packed
+ *	      contiguously, so the destination offset advances by changed-page
+ *	      runs instead of tracking the source page index.
  *
  * A defragmentation move may harvest already beneficial-order chunks from the
  * old tt, so the new tt references (borrows) some of the same physical pages.
@@ -830,16 +891,21 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
  * only blit runs of pages that actually differ (src page != dst page),
  * coalescing contiguous to-copy pages into a single blit.
  *
+ * When @pack_dst is set the source pages are read from their natural offsets
+ * (@src_L0_ofs + page index) while the destination pages were packed into a
+ * contiguous prefix, so the destination offset is a running counter that only
+ * advances for emitted (changed) runs.
+ *
  * Return: The number of blits emitted (or that would be emitted when @bb is
  * NULL).
  */
 static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
 			    u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
 			    struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
-			    pgoff_t start_page)
+			    pgoff_t start_page, bool pack_dst)
 {
 	u32 npages = src_L0 >> PAGE_SHIFT;
-	u32 i = 0, nblits = 0;
+	u32 i = 0, pack = 0, nblits = 0;
 
 	while (i < npages) {
 		u32 run = 0;
@@ -859,9 +925,71 @@ static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
 
 		if (bb)
 			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
-				  dst_L0_ofs + (u64)i * PAGE_SIZE,
+				  dst_L0_ofs +
+				  (u64)(pack_dst ? pack : i) * PAGE_SIZE,
 				  run * PAGE_SIZE, PAGE_SIZE);
 		nblits++;
+		pack += run;
+		i += run;
+	}
+
+	return nblits;
+}
+
+/*
+ * emit_ccs_copy_chunks() - Emit the indirect CCS aux copies for one packed
+ * defrag L0 chunk, skipping borrowed pages.
+ * @gt: The GT.
+ * @bb: The batch buffer to emit into, or NULL to only count the copies needed.
+ * @src_L0_ofs: GPU offset of the source pages mapped (comp PAT) for this chunk.
+ * @dst_L0_ofs: GPU offset of the destination pages mapped (comp PAT) for this
+ *	        chunk (the packed changed-page prefix).
+ * @src_L0: Byte size of the L0 chunk (page aligned).
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ *
+ * Companion to emit_copy_chunks() for the packed defrag CCS pass. The flat-CCS
+ * aux state is migrated indirectly through the data-page mappings, so it
+ * follows the physical page regardless of the (packed) GPU VA layout. Borrowed
+ * pages alias the same physical page on both sides, so their aux is already
+ * identical and is skipped; only the changed runs are copied, from the source
+ * at its natural offset to the destination at its packed offset.
+ *
+ * Return: The number of CCS copies emitted (or that would be emitted when @bb
+ * is NULL).
+ */
+static u32 emit_ccs_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
+				u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
+				struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
+				pgoff_t start_page)
+{
+	u32 npages = src_L0 >> PAGE_SHIFT;
+	u32 i = 0, pack = 0, nblits = 0;
+
+	while (i < npages) {
+		u32 run = 0;
+
+		/* Skip a run of borrowed pages (src and dst alias). */
+		if (src_tt->pages[start_page + i] ==
+		    dst_tt->pages[start_page + i]) {
+			i++;
+			continue;
+		}
+
+		/* Coalesce a run of pages whose aux must be migrated. */
+		while (i + run < npages &&
+		       src_tt->pages[start_page + i + run] !=
+		       dst_tt->pages[start_page + i + run])
+			run++;
+
+		if (bb)
+			emit_copy_ccs(gt, bb,
+				      dst_L0_ofs + (u64)pack * PAGE_SIZE, true,
+				      src_L0_ofs + (u64)i * PAGE_SIZE, true,
+				      run * PAGE_SIZE);
+		nblits++;
+		pack += run;
 		i += run;
 	}
 
@@ -949,6 +1077,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
 					   struct ttm_tt *src_tt,
+					   pgoff_t dst_packed_pages,
 					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
@@ -981,6 +1110,23 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	 */
 	struct ttm_tt *dst_tt = dst_bo->ttm.ttm;
 	bool skip_borrowed = flags.defrag_copy && src_tt && dst_tt;
+	/*
+	 * Defrag-IOVA copy: the destination tt only maps the changed
+	 * (non-borrowed) pages, packed contiguously into the first
+	 * @dst_packed_pages pages of its IOVA. The source is read from the old
+	 * tt at natural offsets and each changed run is blitted into the packed
+	 * destination prefix.
+	 */
+	bool pack_dst = dst_packed_pages != 0;
+	/*
+	 * Packed defrag CCS pass: migrate the flat-CCS aux state of the changed
+	 * pages only, from the source at its natural offset to the destination
+	 * packed prefix. Borrowed pages share the physical page (and thus its
+	 * aux) on both sides, so they are skipped exactly like the data pass.
+	 */
+	bool ccs_pack = flags.defrag_ccs && pack_dst && src_tt && dst_tt;
+	/* Per-run, borrowed-skipping defrag emission (data or packed CCS). */
+	bool defrag_runs = skip_borrowed || ccs_pack;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
@@ -1009,7 +1155,9 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
-		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0,
+				pack_dst ? (u64)dst_packed_pages << PAGE_SHIFT :
+				size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
@@ -1030,29 +1178,86 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		u32 pte_flags;
 		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> XE_PTE_SHIFT;
 		u32 ndata_blits = 1;
+		u32 nchg = 0;
+		u64 dst_size;
 
 		bool usm = xe->info.has_usm;
 		u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
 
+		/*
+		 * Defrag fast-forward: the new (destination) tt borrows the
+		 * unchanged beneficial-order chunks from the old tt, so for a
+		 * large object the bulk of the copy is borrowed pages that alias
+		 * the same memory on both sides. Skip the entire leading
+		 * borrowed run in a single cursor stride rather than discovering
+		 * it one (8 MB) L0 chunk at a time - each such pass would
+		 * otherwise pay xe_migrate_res_sizes() + pte_update_size() + a
+		 * full chunk walk only to copy nothing. The non-vram tt cursors
+		 * are normally advanced by emit_pte(); replicate that with
+		 * xe_res_next(). In pack_dst mode the destination only maps the
+		 * changed pages, so its cursor must not advance for borrowed
+		 * pages.
+		 */
+		if (defrag_runs) {
+			u32 brun = xe_migrate_leading_borrowed_pages(src_tt,
+								     dst_tt,
+								     start_page,
+								     size >> PAGE_SHIFT);
+
+			if (brun) {
+				u64 brun_bytes = (u64)brun << PAGE_SHIFT;
+
+				xe_res_next(&src_it, brun_bytes);
+				if (!pack_dst)
+					xe_res_next(&dst_it, brun_bytes);
+				size -= brun_bytes;
+				continue;
+			}
+		}
+
 		src_L0 = xe_migrate_res_sizes(m, &src_it);
 		dst_L0 = xe_migrate_res_sizes(m, &dst_it);
 
 		drm_dbg(&xe->drm, "Pass %u, sizes: %llu & %llu\n",
 			pass++, src_L0, dst_L0);
 
-		src_L0 = min(src_L0, dst_L0);
+		/*
+		 * In pack_dst mode the destination is the (smaller) packed
+		 * prefix, so it must not clamp the source chunk size; the
+		 * source drives the walk and the destination is sized
+		 * independently from the changed-page count below.
+		 */
+		if (!pack_dst)
+			src_L0 = min(src_L0, dst_L0);
 
 		pte_flags = src_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
 		pte_flags |= use_comp_pat ? PTE_UPDATE_FLAG_IS_COMP_PTE : 0;
 		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
 					      &src_L0_ofs, &src_L0_pt, 0, 0,
 					      avail_pts);
+		/*
+		 * The source chunk size may have been clamped above, so derive
+		 * the packed destination size from the changed-page count of
+		 * the final chunk.
+		 */
+		if (pack_dst) {
+			nchg = xe_migrate_count_changed_pages(src_tt, dst_tt,
+							      start_page,
+							      src_L0 >> PAGE_SHIFT);
+			dst_size = (u64)nchg << PAGE_SHIFT;
+		} else {
+			dst_size = src_L0;
+		}
 		if (flags.copy_only_ccs) {
 			dst_L0_ofs = src_L0_ofs;
+		} else if (pack_dst && !nchg) {
+			/* All borrowed: no destination PTEs for this chunk. */
+			dst_L0_ofs = 0;
+			dst_L0_pt = 0;
 		} else {
 			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
 			batch_size += pte_update_size(m, pte_flags, dst,
-						      &dst_it, &src_L0,
+						      &dst_it, &dst_size,
 						      &dst_L0_ofs, &dst_L0_pt,
 						      0, avail_pts, avail_pts);
 		}
@@ -1072,26 +1277,31 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		 * is split into one command per run of non-borrowed pages, so
 		 * size for the number of runs actually needed.
 		 */
-		if (skip_borrowed)
+		if (defrag_runs)
 			ndata_blits = emit_copy_chunks(gt, NULL, 0, 0, src_L0,
-						       src_tt, dst_tt, start_page);
+						       src_tt, dst_tt, start_page,
+						       pack_dst);
 
 		/*
 		 * If every page in this chunk was borrowed there is nothing to
 		 * copy: don't build a batch or issue any GPU commands, just
 		 * advance the cursors past the chunk. The non-vram tt cursors
-		 * would otherwise be advanced by emit_pte().
+		 * would otherwise be advanced by emit_pte(). In pack_dst mode
+		 * the destination consumed no packed pages, so leave its cursor
+		 * untouched.
 		 */
-		if (skip_borrowed && !ndata_blits) {
+		if (defrag_runs && !ndata_blits) {
 			xe_res_next(&src_it, src_L0);
-			xe_res_next(&dst_it, src_L0);
+			if (!pack_dst)
+				xe_res_next(&dst_it, src_L0);
 			size -= src_L0;
 			continue;
 		}
 
 		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
 			       ndata_blits * emit_copy_cmd_len(xe)) +
-			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
+			(ccs_pack ? ndata_blits * EMIT_COPY_CCS_DW :
+			 (needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
 		if (IS_ERR(bb)) {
@@ -1111,7 +1321,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		else if (!flags.copy_only_ccs)
 			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
 				 copy_system_ccs || flags.defrag_ccs,
-				 &dst_it, src_L0, dst);
+				 &dst_it, dst_size, dst);
 
 		if (copy_system_ccs)
 			emit_pte(m, bb, ccs_pt, false, false, &ccs_it, ccs_size, src);
@@ -1121,16 +1331,22 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 
 		if (skip_borrowed)
 			emit_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0,
-					 src_tt, dst_tt, start_page);
+					 src_tt, dst_tt, start_page, pack_dst);
 		else if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
-		if (needs_ccs_emit)
+		if (ccs_pack) {
+			if (emit_ccs_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs,
+						 src_L0, src_tt, dst_tt,
+						 start_page))
+				flush_flags = MI_FLUSH_DW_CCS;
+		} else if (needs_ccs_emit) {
 			flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
 							  IS_DGFX(xe) ? src_is_vram : src_is_pltt,
 							  dst_L0_ofs,
 							  IS_DGFX(xe) ? dst_is_vram : dst_is_pltt,
 							  src_L0, ccs_ofs, copy_ccs);
+		}
 
 		job = xe_bb_create_migration_job(m->q, bb,
 						 xe_migrate_batch_base(m, usm),
@@ -1218,7 +1434,7 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs)
 {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL, 0,
 				 (struct xe_migrate_copy_flags) {
 					 .copy_only_ccs = copy_only_ccs,
 				 });
@@ -1267,7 +1483,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 bool need_ccs)
 {
 	struct dma_fence *fence2, *fence =
-		__xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt, 0,
 				  (struct xe_migrate_copy_flags) {
 					 .defrag_copy = true,
 					 });
@@ -1275,7 +1491,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	if (IS_ERR(fence) || !need_ccs)
 		return fence;
 
-	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt, 0,
 				   (struct xe_migrate_copy_flags) {
 				   .defrag_ccs = true,
 				   });
@@ -1286,6 +1502,59 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	return fence2;
 }
 
+/**
+ * xe_migrate_copy_defrag_iova() - Copy a defrag move into a packed IOVA dst.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented.
+ * @src: The source (old) TTM resource.
+ * @dst: The destination (new) TTM resource.
+ * @src_tt: The stashed source (old) tt providing the page pointers, fully
+ *	    IOVA-mapped at natural offsets.
+ * @dst_packed_pages: Number of changed pages packed into the destination tt's
+ *	    IOVA prefix by xe_tt_map_iova_copy().
+ * @need_ccs: Whether the BO has flat-CCS aux state that must be migrated too.
+ *
+ * Defrag copy variant for IOVA-mapped BOs where the destination only maps the
+ * changed (non-borrowed) pages, packed contiguously into the first
+ * @dst_packed_pages of its IOVA. The source is read from the old tt at its
+ * natural offsets and each changed run is blitted into the packed destination
+ * prefix. When @need_ccs is set a second pass migrates the flat-CCS aux state
+ * of the same changed pages indirectly through their (packed) data mappings;
+ * borrowed pages share the physical page and its aux, so only the changed runs
+ * are touched.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag_iova(struct xe_migrate *m,
+					      struct xe_bo *bo,
+					      struct ttm_resource *src,
+					      struct ttm_resource *dst,
+					      struct ttm_tt *src_tt,
+					      pgoff_t dst_packed_pages,
+					      bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt, dst_packed_pages,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+				 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+				   dst_packed_pages,
+				   (struct xe_migrate_copy_flags) {
+					 .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
+}
+
 /**
  * xe_migrate_resolve() - Resolve and decompress a buffer object if required.
  * @m: The migrate context
@@ -1301,7 +1570,7 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res)
 {
-	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL, 0,
 				 (struct xe_migrate_copy_flags) {
 					 .is_vram_resolve = true,
 				 });
diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
index 4094ea1c19c1..da066cdf01bf 100644
--- a/drivers/gpu/drm/xe/xe_migrate.h
+++ b/drivers/gpu/drm/xe/xe_migrate.h
@@ -141,6 +141,14 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct ttm_tt *src_tt,
 					 bool need_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag_iova(struct xe_migrate *m,
+					      struct xe_bo *bo,
+					      struct ttm_resource *src,
+					      struct ttm_resource *dst,
+					      struct ttm_tt *src_tt,
+					      pgoff_t dst_packed_pages,
+					      bool need_ccs);
+
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res);
-- 
2.34.1

[PATCH v2 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job

[Matthew Brost]

Wire up end-to-end defrag for IOVA-mapped BOs. A defrag move
is eligible whenever the destination TT is IOVA mapped. The
packed-copy step moves only changed pages; the IOVA reservation
is transferred to the new backing so device-visible addresses
are preserved, avoiding GPU rebind overhead.

Add xe_bo_defrag_use_iova() to select the IOVA defrag path.
Add xe_bo_defrag_iova_copy() to drive the move: diff new vs
stashed old TT, link changed pages packed, blit them, then
queue a post-copy finalize job.

The finalize job tears down the temporary packed mapping,
transfers the full old TT reservation to the new one, and
relinks the copied pages at natural offsets.

Borrowed pages skip the copy and simply adopt the old
reservation.

Since reservation transfer preserves device-visible addresses,
xe_bo_trigger_rebind() skips drm_gpuvm_bo_evict() marking;
GPU PTEs stay valid.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 302 +++++++++++++++++++++++++++++++++++--
 1 file changed, 288 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 16d9e135cba2..0eee53f4d653 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -13,6 +13,7 @@
 #include <drm/drm_dumb_buffers.h>
 #include <drm/drm_gem_ttm_helper.h>
 #include <drm/drm_managed.h>
+#include <drm/gpu_scheduler.h>
 #include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_device.h>
 #include <drm/ttm/ttm_placement.h>
@@ -25,6 +26,7 @@
 #include <trace/events/gpu_mem.h>
 
 #include "xe_device.h"
+#include "xe_dep_job_types.h"
 #include "xe_dep_scheduler.h"
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
@@ -542,7 +544,6 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
  * @xe_tt->iova_dma_pages[]. Used to (re)derive accounting that exactly matches
  * the pages a tt is actually mapped with.
  */
-__maybe_unused
 static void xe_tt_account_iova_pages(struct xe_device *xe,
 				     struct xe_ttm_tt *xe_tt)
 {
@@ -567,7 +568,6 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 {
 }
 
-__maybe_unused
 static void xe_tt_account_iova_pages(struct xe_device *xe,
 				     struct xe_ttm_tt *xe_tt)
 {
@@ -806,7 +806,6 @@ struct xe_tt_iova_copy {
  * differ. Coalesce contiguous changed folios into runs and, for each run,
  * invoke @fn. Returns the total number of changed pages.
  */
-__maybe_unused
 static pgoff_t xe_tt_iova_walk_changed(struct ttm_tt *new_tt,
 				       struct ttm_tt *old_tt,
 				       void (*fn)(pgoff_t start, pgoff_t count,
@@ -855,7 +854,6 @@ struct xe_tt_iova_run_collect {
 };
 
 /* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
-__maybe_unused
 static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 			   size_t elem)
 {
@@ -878,7 +876,6 @@ static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 	return 0;
 }
 
-__maybe_unused
 static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
 {
 	struct xe_tt_iova_run_collect *c = arg;
@@ -933,7 +930,6 @@ static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
  *
  * Return: 0 on success, -ENOMEM on allocation failure.
  */
-__maybe_unused
 static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
 				struct xe_tt_iova_copy *copy)
 {
@@ -969,7 +965,6 @@ static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
  * xe_tt_iova_copy_fini() - Release a struct xe_tt_iova_copy
  * @copy: The copy description to release.
  */
-__maybe_unused
 static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
 {
 	kfree(copy->runs);
@@ -1000,7 +995,6 @@ static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
  * Return: 0 on success, negative error code on failure (any partial linkage is
  * unwound).
  */
-__maybe_unused
 static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
 			       const struct xe_tt_iova_copy *copy)
 {
@@ -1047,6 +1041,129 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
 	return ret;
 }
 
+/**
+ * struct xe_iova_defrag_job - Post-copy finalize job for an IOVA defrag move.
+ *
+ * After the GPU defrag copy into the packed copy-step mapping completes, this
+ * dependency job runs the CPU-side IOMMU fixups at the point of no failure:
+ * tear down the temporary packed mapping, then per changed run unlink the old
+ * pages and link the new pages into the (transferred) reservation at their
+ * natural offset.
+ */
+struct xe_iova_defrag_job {
+	/** @dep: base generic dependency Xe job. */
+	struct xe_dep_job dep;
+	/** @xe: The xe device. */
+	struct xe_device *xe;
+	/** @new_tt: The destination tt being finalized (owns the reservation). */
+	struct ttm_tt *new_tt;
+	/**
+	 * @copy_iova: Snapshot of the new tt's original IOVA reservation that
+	 * held the temporary packed copy-step mapping. Destroyed by the job
+	 * (the new tt itself adopts the old tt's reservation at job creation).
+	 */
+	struct dma_iova_state copy_iova;
+	/** @copy_bytes: Byte length of the packed copy-step mapping. */
+	size_t copy_bytes;
+	/** @copy: Changed-page description; the job owns and frees @copy.runs. */
+	struct xe_tt_iova_copy copy;
+};
+
+static struct dma_fence *xe_iova_defrag_job_run(struct xe_dep_job *dep_job)
+{
+	struct xe_iova_defrag_job *job =
+		container_of(dep_job, typeof(*job), dep);
+	struct xe_device *xe = job->xe;
+	struct device *dev = xe->drm.dev;
+	struct xe_ttm_tt *xe_tt =
+		container_of(job->new_tt, struct xe_ttm_tt, ttm);
+	unsigned int r, l;
+
+	/*
+	 * Tear down the temporary packed copy-step mapping and free its
+	 * reservation. The new tt has already adopted the old tt's full
+	 * reservation (transferred at job creation), so this only touches the
+	 * snapshot taken before the transfer.
+	 *
+	 * This mapping was the GPU blit's write destination, so tear it down
+	 * with attrs 0 (not DMA_ATTR_SKIP_CPU_SYNC): for non-coherent or
+	 * bounced (SWIOTLB) mappings the device->CPU copy-back / cache sync of
+	 * the freshly copied data happens during unlink, exactly as
+	 * xe_tt_unmap_iova() does. Skipping it would silently drop the
+	 * defragmented data on those configs.
+	 */
+	if (job->copy_bytes)
+		dma_iova_destroy(dev, &job->copy_iova, job->copy_bytes,
+				 DMA_BIDIRECTIONAL, 0);
+	else
+		dma_iova_free(dev, &job->copy_iova);
+
+	/*
+	 * For each changed run, unlink the old pages (still linked from the
+	 * adopted reservation).
+	 */
+	for (r = 0; r < job->copy.nr_runs; r++) {
+		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
+		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
+
+		dma_iova_unlink(dev, &xe_tt->iova, off, len,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+	}
+
+	/*
+	 * Relink the freshly copied new pages from the snapshot captured at job
+	 * creation. The destination tt's page array may have been mutated (or
+	 * torn down) by TTM since then, so it is never dereferenced here.
+	 * Borrowed pages are shared with the old tt and stay correctly linked
+	 * for free, so the snapshot only covers the changed runs.
+	 */
+	for (l = 0; l < job->copy.nr_links; l++) {
+		int ret = dma_iova_link(dev, &xe_tt->iova,
+					job->copy.links[l].phys,
+					job->copy.links[l].offset,
+					job->copy.links[l].len,
+					DMA_BIDIRECTIONAL,
+					DMA_ATTR_SKIP_CPU_SYNC);
+		/*
+		 * Point of no failure: the data is already copied and the
+		 * reservation ownership has been transferred. A relink failure
+		 * leaves a hole in the mapping; there is no safe recovery, so
+		 * just warn.
+		 */
+		drm_WARN_ON(&xe->drm, ret);
+	}
+
+	/* Sync the relinked changed runs into the device's view. */
+	for (r = 0; r < job->copy.nr_runs; r++) {
+		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
+		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
+
+		dma_iova_sync(dev, &xe_tt->iova, off, len);
+	}
+
+	/*
+	 * All work is synchronous CPU-side IOMMU fixup, so there is no hardware
+	 * fence to wait on; returning NULL makes the scheduler signal the job's
+	 * finished fence immediately.
+	 */
+	return NULL;
+}
+
+static void xe_iova_defrag_job_free(struct xe_dep_job *dep_job)
+{
+	struct xe_iova_defrag_job *job =
+		container_of(dep_job, typeof(*job), dep);
+
+	xe_tt_iova_copy_fini(&job->copy);
+	drm_sched_job_cleanup(&job->dep.drm);
+	kfree(job);
+}
+
+static const struct xe_dep_job_ops xe_iova_defrag_job_ops = {
+	.run_job = xe_iova_defrag_job_run,
+	.free_job = xe_iova_defrag_job_free,
+};
+
 /*
  * xe_tt_iova_transfer - Transfer a full IOVA reservation old_tt -> new_tt.
  *
@@ -1063,7 +1180,6 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
  * folio walk of the new tt keeps xe->mem.dma_mapped_pages[] exact while the new
  * mapping lives, rather than carrying the old tt's stale distribution.
  */
-__maybe_unused
 static void xe_tt_iova_transfer(struct xe_device *xe,
 				struct xe_ttm_tt *new_tt,
 				struct xe_ttm_tt *old_tt)
@@ -1078,6 +1194,135 @@ static void xe_tt_iova_transfer(struct xe_device *xe,
 	old_tt->use_iova = false;
 }
 
+/*
+ * xe_bo_defrag_iova_copy - Run an IOVA defrag move with a packed copy mapping.
+ *
+ * Only the changed (non-borrowed) pages of @new_tt are DMA mapped, packed
+ * contiguously into the front of its reservation, the GPU copy blits them from
+ * the old tt at natural offsets, and a post-copy finalize job adopts the old
+ * tt's full reservation and relinks the changed runs. Returns the finalize
+ * job's finished fence (the fence the move waits on), or an ERR_PTR on failure
+ * with all partial state unwound.
+ */
+static struct dma_fence *
+xe_bo_defrag_iova_copy(struct xe_device *xe, struct xe_migrate *migrate,
+		       struct xe_bo *bo, struct ttm_resource *old_mem,
+		       struct ttm_resource *new_mem, struct ttm_tt *old_tt,
+		       bool handle_system_ccs)
+{
+	struct ttm_tt *new_tt = bo->ttm.ttm;
+	struct xe_ttm_tt *new_xe_tt = container_of(new_tt, struct xe_ttm_tt, ttm);
+	struct xe_ttm_tt *old_xe_tt = container_of(old_tt, struct xe_ttm_tt, ttm);
+	struct drm_sched_entity *entity =
+		xe_dep_scheduler_entity(xe->mem.defrag.iova_sched);
+	struct xe_iova_defrag_job *job;
+	struct dma_fence *copy_fence;
+	struct dma_fence *fence;
+	struct xe_tt_iova_copy copy;
+	bool packed = false;
+	int err;
+
+	xe_assert(xe, new_xe_tt->use_iova && !new_xe_tt->iova_linked);
+	xe_assert(xe, old_xe_tt->use_iova && old_xe_tt->iova_linked);
+
+	err = xe_tt_iova_copy_init(new_tt, old_tt, &copy);
+	if (err)
+		return ERR_PTR(err);
+
+	if (copy.nr_changed) {
+		err = xe_tt_map_iova_copy(xe, new_tt, &copy);
+		if (err)
+			goto err_copy;
+		packed = true;
+
+		copy_fence = xe_migrate_copy_defrag_iova(migrate, bo, old_mem,
+							 new_mem, old_tt,
+							 copy.nr_changed,
+							 handle_system_ccs);
+	} else {
+		copy_fence = dma_fence_get_stub();
+	}
+	if (IS_ERR(copy_fence)) {
+		err = PTR_ERR(copy_fence);
+		goto err_unmap;
+	}
+
+	job = kzalloc(sizeof(*job), GFP_KERNEL);
+	if (!job) {
+		err = -ENOMEM;
+		goto err_fence;
+	}
+
+	job->dep.ops = &xe_iova_defrag_job_ops;
+	job->xe = xe;
+	job->new_tt = new_tt;
+	job->copy_iova = new_xe_tt->iova;
+	job->copy_bytes = (size_t)copy.nr_changed << PAGE_SHIFT;
+	job->copy = copy;
+
+	err = drm_sched_job_init(&job->dep.drm, entity, 1, NULL, 0);
+	if (err)
+		goto err_job;
+
+	err = drm_sched_job_add_dependency(&job->dep.drm,
+					   dma_fence_get(copy_fence));
+	if (err)
+		goto err_job_init;
+
+	/*
+	 * Point of no failure: transfer the old tt's full reservation to the
+	 * new tt. The finalize job (already armed below) destroys the packed
+	 * copy mapping snapshot and relinks the changed runs.
+	 */
+	xe_tt_iova_transfer(xe, new_xe_tt, old_xe_tt);
+	memset(&copy, 0, sizeof(copy)); /* ownership moved to the job */
+
+	drm_sched_job_arm(&job->dep.drm);
+	fence = dma_fence_get(&job->dep.drm.s_fence->finished);
+	drm_sched_entity_push_job(&job->dep.drm);
+
+	dma_fence_put(copy_fence);
+
+	return fence;
+
+err_job_init:
+	drm_sched_job_cleanup(&job->dep.drm);
+err_job:
+	kfree(job);
+err_fence:
+	dma_fence_put(copy_fence);
+err_unmap:
+	if (packed)
+		dma_iova_unlink(xe->drm.dev, &new_xe_tt->iova, 0,
+				(size_t)copy.nr_changed << PAGE_SHIFT,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+err_copy:
+	xe_tt_iova_copy_fini(&copy);
+	return ERR_PTR(err);
+}
+
+/*
+ * xe_bo_defrag_use_iova - Whether a defrag move should use the packed IOVA path.
+ *
+ * Eligible when the move is a defrag (stashed old tt) and the destination tt is
+ * IOVA mapped. The IOVA reservation is transferred to the new backing, so the
+ * device-visible addresses are preserved and no GPU rebind is needed (see
+ * xe_bo_trigger_rebind()); this makes the path worthwhile regardless of BO
+ * size. A flat-CCS aux pass, if needed, is migrated packed alongside the data
+ * (changed pages only), so it does not disqualify the path. Otherwise the
+ * full-mapping defrag path is used.
+ */
+static bool xe_bo_defrag_use_iova(struct ttm_buffer_object *ttm_bo)
+{
+	struct xe_ttm_tt *xe_tt;
+
+	if (!ttm_bo->defrag_old_tt || !ttm_bo->ttm)
+		return false;
+
+	xe_tt = container_of(ttm_bo->ttm, struct xe_ttm_tt, ttm);
+	return xe_tt->use_iova;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
@@ -1357,6 +1602,7 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 	struct drm_gem_object *obj = &bo->ttm.base;
 	struct drm_gpuvm_bo *vm_bo;
 	bool idle = false;
+	bool defrag_iova;
 	int ret = 0;
 
 	dma_resv_assert_held(bo->ttm.base.resv);
@@ -1369,12 +1615,25 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 		dma_resv_iter_end(&cursor);
 	}
 
+	/*
+	 * A defrag move that reallocates IOVA-mapped backing preserves the
+	 * device-visible addresses: the IOVA reservation is transferred from
+	 * the old to the new tt, so the existing GPU PTEs stay valid and the
+	 * BO must not be marked evicted (the rebind paths in xe_exec_ioctl()
+	 * and preempt_rebind_work_func() would otherwise needlessly rebind).
+	 * Preempt fences still need to be triggered, and fault-mode / L2
+	 * flush-optimized configs still need a TLB invalidation below, so only
+	 * the eviction marking is skipped.
+	 */
+	defrag_iova = xe_bo_defrag_use_iova(&bo->ttm);
+
 	drm_gem_for_each_gpuvm_bo(vm_bo, obj) {
 		struct xe_vm *vm = gpuvm_to_vm(vm_bo->vm);
 		struct drm_gpuva *gpuva;
 
 		if (!xe_vm_in_fault_mode(vm)) {
-			drm_gpuvm_bo_evict(vm_bo, true);
+			if (!defrag_iova)
+				drm_gpuvm_bo_evict(vm_bo, true);
 			/*
 			 * L2 cache may not be flushed, so ensure that is done in
 			 * xe_vm_invalidate_vma() below
@@ -1388,6 +1647,8 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
 				continue;
+			else if (defrag_iova)
+				drm_gpuvm_bo_evict(vm_bo, true);
 		}
 
 		if (!idle) {
@@ -2096,6 +2357,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	bool needs_clear;
 	bool handle_system_ccs = (!IS_DGFX(xe) && xe_bo_needs_ccs_pages(bo) &&
 				  ttm && ttm_tt_is_populated(ttm)) ? true : false;
+	bool defrag_iova = xe_bo_defrag_use_iova(ttm_bo);
 	int ret = 0;
 
 	/*
@@ -2144,7 +2406,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	needs_clear = (ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) ||
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
-	if (new_mem->mem_type == XE_PL_TT) {
+	/*
+	 * For a defrag move using the packed IOVA path the destination tt is
+	 * not fully mapped here; only the changed pages are mapped (packed) for
+	 * the copy, and the finalize job establishes the full mapping.
+	 */
+	if (new_mem->mem_type == XE_PL_TT && !defrag_iova) {
 		ret = xe_tt_map(xe, ttm);
 		if (ret)
 			goto out;
@@ -2254,9 +2521,16 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
 	} else if (ttm_bo->defrag_old_tt) {
-		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
-					       ttm_bo->defrag_old_tt,
-					       handle_system_ccs);
+		if (defrag_iova)
+			fence = xe_bo_defrag_iova_copy(xe, migrate, bo, old_mem,
+						       new_mem,
+						       ttm_bo->defrag_old_tt,
+						       handle_system_ccs);
+		else
+			fence = xe_migrate_copy_defrag(migrate, bo, old_mem,
+						       new_mem,
+						       ttm_bo->defrag_old_tt,
+						       handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
-- 
2.34.1

[PATCH v2 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock

[Matthew Brost]

Populating a GTT (system) buffer object under the reservation lock can
stall in reclaim and compaction while trying to satisfy beneficial-order
allocations, holding the lock for the duration.

Mirror the Xe out-of-lock preallocation for pure system BOs: when a
gem_create request targets AMDGPU_GEM_DOMAIN_GTT only, fill the full page
backing up front via ttm_pool_prealloc_fill_full() before taking the
reservation lock. The populate under the lock then simply installs these
pages instead of reclaiming and compacting in the critical section. The
fill is best-effort - a short fill falls back to the normal in-lock
allocation for the remaining pages, and any leftover pages are released
by ttm_pool_prealloc_fini() on all exit paths.

Unlike Xe, amdgpu has no background defragmenter, so the higher-order
reclaim backoff is left disabled (normal reclaim). If the pool uses
dma-alloc (swiotlb), the fill bails and the feature is a silent no-op.

Factor the tt pool selection into amdgpu_ttm_tt_pool_id() and
amdgpu_ttm_pool() so the preallocation targets the exact pool the
populate will consume, and thread an optional prealloc bag through
amdgpu_gem_object_create() and amdgpu_bo_param into the populate ctx.

Cc: amd-gfx@lists.freedesktop.org
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 45 +++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |  5 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |  3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  4 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 39 +++++++++++-----
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |  3 ++
 8 files changed, 80 insertions(+), 25 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 20831dbebc31..940f58848a97 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -345,7 +345,7 @@ create_dmamap_sg_bo(struct amdgpu_device *adev,
 
 	ret = amdgpu_gem_object_create(adev, mem->bo->tbo.base.size, 1,
 			AMDGPU_GEM_DOMAIN_CPU, AMDGPU_GEM_CREATE_PREEMPTIBLE | flags,
-			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0);
+			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0, NULL);
 
 	amdgpu_bo_unreserve(mem->bo);
 
@@ -1811,7 +1811,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
 		 domain_string(alloc_domain), xcp_id);
 
 	ret = amdgpu_gem_object_create(adev, aligned_size, 1, alloc_domain, alloc_flags,
-				       bo_type, NULL, &gobj, xcp_id + 1);
+				       bo_type, NULL, &gobj, xcp_id + 1, NULL);
 	if (ret) {
 		pr_debug("Failed to create BO on domain %s. ret %d\n",
 			 domain_string(alloc_domain), ret);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index b33c300e26e2..51510e831129 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -435,7 +435,7 @@ amdgpu_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
 
 	ret = amdgpu_gem_object_create(adev, dma_buf->size, PAGE_SIZE,
 				       AMDGPU_GEM_DOMAIN_CPU, flags,
-				       ttm_bo_type_sg, resv, &gobj, 0);
+				       ttm_bo_type_sg, resv, &gobj, 0, NULL);
 	if (ret)
 		goto error;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index 6a0699746fbc..bef8b5bc8b89 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -39,6 +39,7 @@
 #include <drm/drm_exec.h>
 #include <drm/drm_gem_ttm_helper.h>
 #include <drm/ttm/ttm_tt.h>
+#include <drm/ttm/ttm_pool.h>
 #include <drm/drm_syncobj.h>
 
 #include "amdgpu.h"
@@ -168,7 +169,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
 			     int alignment, u32 initial_domain,
 			     u64 flags, enum ttm_bo_type type,
 			     struct dma_resv *resv,
-			     struct drm_gem_object **obj, int8_t xcp_id_plus1)
+			     struct drm_gem_object **obj, int8_t xcp_id_plus1,
+			     struct ttm_pool_prealloc *prealloc)
 {
 	struct amdgpu_bo *bo;
 	struct amdgpu_bo_user *ubo;
@@ -188,6 +190,7 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
 	bp.domain = initial_domain;
 	bp.bo_ptr_size = sizeof(struct amdgpu_bo);
 	bp.xcp_id_plus1 = xcp_id_plus1;
+	bp.prealloc = prealloc;
 
 	r = amdgpu_bo_create_user(adev, &bp, &ubo);
 	if (r)
@@ -412,6 +415,8 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 	struct dma_resv *resv = NULL;
 	struct drm_gem_object *gobj;
 	uint32_t handle, initial_domain;
+	struct ttm_pool_prealloc prealloc = {};
+	struct ttm_pool *prealloc_pool = NULL;
 	int r;
 
 	/* reject invalid gem flags */
@@ -443,10 +448,29 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 		flags |= AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
 	}
 
+	/*
+	 * For system-only (pure GTT) BOs, preallocate the whole page backing
+	 * up front, outside the reservation lock. Populate under the lock then
+	 * just installs these pages instead of reclaiming/compacting in the
+	 * critical section. Best-effort: a short fill falls back to the normal
+	 * in-lock allocation for the missing pages.
+	 */
+	if (args->in.domains == AMDGPU_GEM_DOMAIN_GTT) {
+		int32_t xcp_id = adev->gmc.mem_partitions ? fpriv->xcp_id : 0;
+		int32_t pool_id = amdgpu_ttm_tt_pool_id(adev, xcp_id);
+		enum ttm_caching caching =
+			(flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC) ?
+			ttm_write_combined : ttm_cached;
+
+		prealloc_pool = amdgpu_ttm_pool(adev, pool_id);
+		ttm_pool_prealloc_fill_full(prealloc_pool, caching, &prealloc,
+					    PFN_UP(size), false);
+	}
+
 	if (flags & AMDGPU_GEM_CREATE_VM_ALWAYS_VALID) {
 		r = amdgpu_bo_reserve(vm->root.bo, false);
 		if (r)
-			return r;
+			goto out_prealloc;
 
 		resv = vm->root.bo->tbo.base.resv;
 	}
@@ -455,7 +479,8 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 retry:
 	r = amdgpu_gem_object_create(adev, size, args->in.alignment,
 				     initial_domain,
-				     flags, ttm_bo_type_device, resv, &gobj, fpriv->xcp_id + 1);
+				     flags, ttm_bo_type_device, resv, &gobj,
+				     fpriv->xcp_id + 1, prealloc_pool ? &prealloc : NULL);
 	if (r && r != -ERESTARTSYS) {
 		if (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) {
 			flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
@@ -479,17 +504,21 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 		amdgpu_bo_unreserve(vm->root.bo);
 	}
 	if (r)
-		return r;
+		goto out_prealloc;
 
 	r = drm_gem_handle_create(filp, gobj, &handle);
 	/* drop reference from allocate - handle holds it now */
 	drm_gem_object_put(gobj);
 	if (r)
-		return r;
+		goto out_prealloc;
 
 	memset(args, 0, sizeof(*args));
 	args->out.handle = handle;
-	return 0;
+
+out_prealloc:
+	if (prealloc_pool)
+		ttm_pool_prealloc_fini(prealloc_pool, &prealloc);
+	return r;
 }
 
 int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,
@@ -528,7 +557,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,
 
 	/* create a gem object to contain this object in */
 	r = amdgpu_gem_object_create(adev, args->size, 0, AMDGPU_GEM_DOMAIN_CPU,
-				     0, ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1);
+				     0, ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1, NULL);
 	if (r)
 		return r;
 
@@ -1298,7 +1327,7 @@ int amdgpu_mode_dumb_create(struct drm_file *file_priv,
 	domain = amdgpu_bo_get_preferred_domain(adev,
 				amdgpu_display_supported_domains(adev, flags));
 	r = amdgpu_gem_object_create(adev, args->size, 0, domain, flags,
-				     ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1);
+				     ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1, NULL);
 	if (r)
 		return -ENOMEM;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
index b558336bc4c6..706aeca011f1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
@@ -35,6 +35,8 @@
 
 extern const struct drm_gem_object_funcs amdgpu_gem_object_funcs;
 
+struct ttm_pool_prealloc;
+
 unsigned long amdgpu_gem_timeout(uint64_t timeout_ns);
 
 /*
@@ -45,7 +47,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
 			     int alignment, u32 initial_domain,
 			     u64 flags, enum ttm_bo_type type,
 			     struct dma_resv *resv,
-			     struct drm_gem_object **obj, int8_t xcp_id_plus1);
+			     struct drm_gem_object **obj, int8_t xcp_id_plus1,
+			     struct ttm_pool_prealloc *prealloc);
 int amdgpu_mode_dumb_create(struct drm_file *file_priv,
 			    struct drm_device *dev,
 			    struct drm_mode_create_dumb *args);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index f98bfba59a2c..18c4cf3f35a5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -632,7 +632,8 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 		/* We opt to avoid OOM on system pages allocations */
 		.gfp_retry_mayfail = true,
 		.allow_res_evict = bp->type != ttm_bo_type_kernel,
-		.resv = bp->resv
+		.resv = bp->resv,
+		.prealloc = bp->prealloc,
 	};
 	struct amdgpu_bo *bo;
 	unsigned long page_align, size = bp->size;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index ff11a0903499..11f1d403f152 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -45,6 +45,8 @@
 #define to_amdgpu_bo_user(abo) container_of((abo), struct amdgpu_bo_user, bo)
 #define to_amdgpu_bo_vm(abo) container_of((abo), struct amdgpu_bo_vm, bo)
 
+struct ttm_pool_prealloc;
+
 struct amdgpu_bo_param {
 	unsigned long			size;
 	int				byte_align;
@@ -58,6 +60,8 @@ struct amdgpu_bo_param {
 	void				(*destroy)(struct ttm_buffer_object *bo);
 	/* xcp partition number plus 1, 0 means any partition */
 	int8_t				xcp_id_plus1;
+	/* optional out-of-lock preallocated backing (system/GTT only) */
+	struct ttm_pool_prealloc	*prealloc;
 };
 
 /* bo virtual addresses in a vm */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index b10b0878df37..c9cd4714f515 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1173,6 +1173,30 @@ void amdgpu_ttm_mmio_remap_free_sgt(struct device *dev,
 	kfree(sgt);
 }
 
+/*
+ * amdgpu_ttm_tt_pool_id - compute the ttm pool id backing a given xcp
+ *
+ * Mirrors the mapping used when creating the gtt ttm_tt, so callers that only
+ * have an xcp id (e.g. an out-of-lock preallocation before the bo exists) pick
+ * the same pool the populate will use.
+ */
+int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id)
+{
+	if (adev->gmc.mem_partitions && xcp_id >= 0)
+		return KFD_XCP_MEM_ID(adev, xcp_id);
+
+	return xcp_id;
+}
+
+/* amdgpu_ttm_pool - select the ttm pool for a given pool id */
+struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id)
+{
+	if (adev->mman.ttm_pools && pool_id >= 0)
+		return &adev->mman.ttm_pools[pool_id];
+
+	return &adev->mman.bdev.pool;
+}
+
 /**
  * amdgpu_ttm_tt_create - Create a ttm_tt object for a given BO
  *
@@ -1194,10 +1218,7 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo,
 		return NULL;
 
 	gtt->gobj = &bo->base;
-	if (adev->gmc.mem_partitions && abo->xcp_id >= 0)
-		gtt->pool_id = KFD_XCP_MEM_ID(adev, abo->xcp_id);
-	else
-		gtt->pool_id = abo->xcp_id;
+	gtt->pool_id = amdgpu_ttm_tt_pool_id(adev, abo->xcp_id);
 
 	if (abo->flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC)
 		caching = ttm_write_combined;
@@ -1239,10 +1260,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev,
 	if (ttm->page_flags & TTM_TT_FLAG_EXTERNAL)
 		return 0;
 
-	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
-		pool = &adev->mman.ttm_pools[gtt->pool_id];
-	else
-		pool = &adev->mman.bdev.pool;
+	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
 	ret = ttm_pool_alloc(pool, ttm, ctx);
 	if (ret)
 		return ret;
@@ -1284,10 +1302,7 @@ static void amdgpu_ttm_tt_unpopulate(struct ttm_device *bdev,
 
 	adev = amdgpu_ttm_adev(bdev);
 
-	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
-		pool = &adev->mman.ttm_pools[gtt->pool_id];
-	else
-		pool = &adev->mman.bdev.pool;
+	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
 
 	return ttm_pool_free(pool, ttm);
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
index ff9e2e346609..f90844937b73 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
@@ -207,6 +207,9 @@ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo);
 void amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo);
 uint64_t amdgpu_ttm_domain_start(struct amdgpu_device *adev, uint32_t type);
 
+int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id);
+struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id);
+
 #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
 int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo,
 				 struct amdgpu_hmm_range *range);
-- 
2.34.1

Re: [PATCH v2 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Calling `__fls()` with 0 causes undefined behavior when an allocation fulfills the exact remaining pages, potentially causing an infinite loop.
--

--- Patch [4]: [PATCH v2 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() ---
commit 5c371f076caffdc9b759a99aaeaf1a30e3ac5dd7
Author: Matthew Brost <matthew.brost@intel.com>

drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()
    
This commit factors the working state of the __ttm_pool_alloc loop into a
struct ttm_pool_alloc_iter. It pulls the per-order page acquisition into 
smaller helpers to simplify the nested block and improve readability.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index 370d991c9882b..6826c8ae6fc56 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c

[ ... ]

> @@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>  	else
>  		gfp_flags |= GFP_HIGHUSER;
>  
> -	page_caching = tt->caching;
> -	allow_pools = true;
> -	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
> +	it.gfp_flags = gfp_flags;
> +
> +	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
>  	     alloc->remaining_pages;
> -	     order = ttm_pool_alloc_find_order(order, alloc)) {
> +	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {

[Severity: High]
This is a pre-existing issue, but does this loop increment expression
evaluate __fls(0) when the allocation exactly fulfills the request?

If the loop successfully allocates the final needed pages during an iteration,
alloc->remaining_pages drops to 0 inside the loop body. The C for-loop
evaluates the increment expression before checking the loop condition, which
calls ttm_pool_alloc_find_order().

Inside ttm_pool_alloc_find_order(), this directly executes:
min_t(unsigned int, highest, __fls(alloc->remaining_pages))

Calling __fls(0) translates to __builtin_clzl(0), which is explicitly
undefined behavior in GCC and Clang. 

Could this undefined behavior allow the compiler to assume
alloc->remaining_pages is always non-zero because it is passed to __fls, 
potentially optimizing away the loop termination condition and causing an 
infinite loop?

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=4

Re: [PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] `beneficial_reclaim_backoff` unintentionally disables reclaim for all non-zero orders if the pool has no beneficial order.
--

--- Patch [1]: [PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order ---
commit 33813073a08d456967f60dbd0f79ceb761550bda
Author: Matthew Brost <matthew.brost@intel.com>

drm/ttm/pool: Allow backing off reclaim at the beneficial order

This commit adds a beneficial_reclaim_backoff flag to struct
ttm_operation_ctx. When set, it allows the allocator to skip reclaim and
compaction at the pool's beneficial order, falling back to smaller orders
quickly instead of stalling.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index 3d5f2ae0a4564..dbe977412a811 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
[ ... ]
> @@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
>  	 * Do not add latency to the allocation path for allocations orders
>  	 * device tolds us do not bring them additional performance gains.
>  	 */
> -	if (order && beneficial_order && order != beneficial_order)
> +	if (order && (beneficial_reclaim_backoff ||
> +		      (beneficial_order && order != beneficial_order)))
>  		gfp_flags &= ~__GFP_RECLAIM;

[Severity: Medium]
Does this unintentionally disable reclaim for all non-zero orders if the
pool has no configured beneficial order?

If a caller sets ctx->beneficial_reclaim_backoff = true for an allocation
on a TTM pool where ttm_pool_beneficial_order(pool) == 0, the condition
above evaluates to true for any order > 0. This unconditionally clears
__GFP_RECLAIM.

High-order page allocations would then completely skip direct and kswapd
reclaim, causing them to fail prematurely and fall back to order-0
allocations. This seems to defeat the purpose of high-order pools when
no beneficial order is specified.

Should this condition explicitly require beneficial_order to be non-zero
when evaluating beneficial_reclaim_backoff?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=1

Re: [PATCH v2 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Harvesting tail pages of a bisected chunk during defragmentation leads to Use-After-Free.
- [High] Missing __GFP_NOWARN and __GFP_RETRY_MAYFAIL in best-effort preallocation causes log spam and potential OOM killer invocation.
--

--- Patch [9]: [PATCH v2 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock ---
commit d825f68af024a3a3838c6e5d1ecfcf5ee8717374
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/ttm: Preallocate beneficial-order defrag pages outside the lock

This commit introduces a mechanism to preallocate high-order pages outside
the DMA reservation lock during defragmentation moves. By using a prealloc
bag, it avoids stalling in direct reclaim or compaction while holding the
lock, improving concurrent rendering performance.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
[ ... ]
> @@ -929,6 +939,172 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
>  	return 0;
>  }
>  
> +unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
> +{
> +	return ttm_pool_beneficial_order(pool);
> +}
> +EXPORT_SYMBOL(ttm_pool_prealloc_order);
> +
> +/*
> + * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
> + * order page allocations, matching the in-line defrag alloc path.
> + */
> +static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
> +{
> +	gfp_t gfp = GFP_USER;
> +
> +	/*
> +	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
> +	 * on the kernel mapping, so the pages must be permanently mapped.
> +	 */
> +	if (ttm_pool_uses_dma32(pool))
> +		gfp |= GFP_DMA32;
> +
> +	return gfp;
> +}

[Severity: High]
Does this base GFP flag configuration risk invoking the OOM killer on a
fragmented system?

The preallocation performed in ttm_pool_prealloc_fill() appears to be an
opportunistic, best-effort operation:

ttm_pool_prealloc_fill() {
        ...
        struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);

Without including __GFP_NOWARN and __GFP_RETRY_MAYFAIL here, a failure to
allocate these high-order pages will trigger aggressive reclaim. This could
unnecessarily invoke the OOM killer and spam the logs, even though the pool
can safely fall back to in-line allocation.

Should these suppression flags be added to the preallocation gfp mask?

[ ... ]
> @@ -1178,8 +1374,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>  	     alloc->remaining_pages;
>  	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
>  		/*
> -		 * Out of defrag budget: harvest the rest of the old tt as-is and
> -		 * stop (the tt is re-queued if the remainder is still
> +		 * Out of defrag budget/prealloc: harvest the rest of the old tt
> +		 * as-is and stop (the tt is re-queued if the remainder is still
>  		 * sub-optimal).
>  		 */
>  		if (ttm_pool_iter_defrag_exhausted(&it)) {

[Severity: Critical]
Can this lead to memory corruption if the budget is exhausted while the
offset is misaligned with the chunks in the old tt?

If the budget is exhausted here, ttm_pool_iter_harvest_rest() will
call ttm_pool_harvest_remaining(), which reads the order from the
current page:

ttm_pool_harvest_remaining() {
        ...
        struct page *p = old_tt->pages[off];
        unsigned int order = ttm_pool_page_order(pool, p);

If off points to a tail page of a previously allocated larger chunk,
ttm_pool_page_order() will return 0 because the private field is 0 on tail
pages. This causes the code to falsely harvest it as an order-0 chunk. When
the old tt is later freed, it frees the entire large chunk from the head
page, leaving the new tt with pointers to freed memory.

> +			r = ttm_pool_iter_harvest_rest(&it);
> +			if (r)
> +				goto error_free_all;
> +			break;
> +		}
[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=9

Re: [PATCH v2 05/33] drm/ttm: Support defragmentation moves

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] A defragmentation move that requires an intermediate bounce buffer (returns -EMULTIHOP) permanently leaks the original ttm_tt, loses user data, and exposes uninitialized memory.
--

--- Patch [5]: [PATCH v2 05/33] drm/ttm: Support defragmentation moves ---
commit 4cef86e1c4adcce1b49f5be21dce811803bbfbeb
Author: Matthew Brost <matthew.brost@intel.com>

drm/ttm: Support defragmentation moves

Add infrastructure to let a driver re-back an already-resident BO with
fresh pages (e.g. at a more beneficial order) by reusing the normal move
machinery.

> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index 3980f376e3ba4..d05da4eb1d8a4 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c

[ ... ]

> @@ -125,36 +125,93 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,

[ ... ]

> +	if (defrag)
> +		ttm_bo_unmap_virtual(bo);
> +
>  	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
>  	if (ret) {
>  		if (ret == -EMULTIHOP)
>  			return ret;

[Severity: Critical]
Does returning here directly skip restoring the original backing?

If bdev->funcs->move() returns -EMULTIHOP during a defragmentation move, we
return immediately instead of going to out_err. This leaves bo->ttm pointing
to the new unpopulated TT, and bo->defrag_old_tt holding the original
populated TT.

When ttm_bo_validate() subsequently handles the multi-hop by calling
ttm_bo_bounce_temp_buffer(), it passes the same ctx where ctx->defrag is
still true.

Will the nested ttm_bo_handle_move_mem() misclassify this bounce as another
defrag move and stash the empty bo->ttm into bo->defrag_old_tt? This appears
to overwrite the original TT, permanently leaking it and resulting in the
loss of user data while exposing uninitialized memory.

>  		goto out_err;
>  	}
>  
> +	if (bo->defrag_old_tt) {
> +		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
> +		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
> +		bo->defrag_old_tt = NULL;
> +	}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=5

Re: [PATCH v2 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [High] Device memory mappings are leaked on the error path because `dpagemap` is not yet stored in `svm_pages`.
- [High] Reading uninitialized memory (`svm_pages->dma_addr[0].dir`) when unwinding an IOVA allocation failure.
- [High] System pages are incorrectly routed to `device_unmap` during unmapping of mixed ranges when IOVA is used.
--

--- Patch [11]: [PATCH v2 11/33] drm/gpusvm: Add a DMA-mapping accounting callback ---
commit 98db73f178129cc5ccc6277eaada197044dfbd3b
Author: Thomas Hellström <thomas.hellstrom@linux.intel.com>

drm/gpusvm: Add a DMA-mapping accounting callback

This commit adds an optional @dma_map_account callback to struct
drm_gpusvm_ops to track mapped pages. It is invoked during mapping and
unmapping to keep symmetric accounting. It also relaxes drm_gpusvm_init() to
allow restricted ops without full-SVM mode.

> diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
> index fcfe635bc1956..512f52471419e 100644
> --- a/drivers/gpu/drm/drm_gpusvm.c
> +++ b/drivers/gpu/drm/drm_gpusvm.c

[ ... ]

> @@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
>  			else if (dpagemap && dpagemap->ops->device_unmap)
>  				dpagemap->ops->device_unmap(dpagemap,
>  							    dev, addr);

[Severity: High]
This isn't a bug introduced by this patch, but can this leak device memory
mappings on the error path?

In drm_gpusvm_get_pages(), if an error occurs after successfully mapping
one or more device pages, the code jumps to err_unmap and calls
__drm_gpusvm_unmap_pages().

However, svm_pages->dpagemap is purposefully only updated after the mapping
loop completes successfully. During the early error unwind, it remains NULL.

Because of this, in __drm_gpusvm_unmap_pages():

    struct drm_pagemap *dpagemap = svm_pages->dpagemap;

The dpagemap evaluated here will be NULL, causing this else if condition to
fail and completely bypassing the unmap callback for all successfully mapped
device pages in that chunk.

[Severity: High]
This is also a pre-existing issue, but could this incorrectly unmap system
pages as device pages in a mixed chunk?

In drm_gpusvm_get_pages(), if a chunk has a mix of device and system pages
(ctx->allow_mixed is true) and IOVA is used:

When iterating over the pages in __drm_gpusvm_unmap_pages(), system pages will
have addr->proto == DRM_INTERCONNECT_SYSTEM. However, if use_iova is true, the
previous condition:

    if (!use_iova && addr->proto == DRM_INTERCONNECT_SYSTEM)
        dma_unmap_page(...)

will evaluate to false.

Execution then falls through to this else if branch. If a previous device page
set dpagemap, it will inappropriately call dpagemap->ops->device_unmap() on the
system page.

> +			if (gpusvm->ops && gpusvm->ops->dma_map_account)
> +				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
>  			i += 1 << addr->order;
>  		}
>  

[ ... ]

> @@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
>  				(addr, DRM_INTERCONNECT_SYSTEM, order,
>  				 dma_dir);
>  		}
> +		if (gpusvm->ops && gpusvm->ops->dma_map_account)
> +			gpusvm->ops->dma_map_account(gpusvm,
> +						     &svm_pages->dma_addr[j],
> +						     1);

[Severity: High]
This is a pre-existing issue, but is it possible to read uninitialized memory
here if the first page mapping fails?

In drm_gpusvm_get_pages(), svm_pages->dma_addr is allocated via
kvmalloc_objs() without zero-initialization.

If an error occurs on the very first page (i = 0), such as an IOVA allocation
failure, the code jumps to err_unmap with num_dma_mapped = 0, and
__drm_gpusvm_unmap_pages() is called with npages = 0.

Inside __drm_gpusvm_unmap_pages(), if dma_use_iova(&svm_pages->state) is true,
it will unconditionally call:

    dma_iova_destroy(dev, &svm_pages->state,
             svm_pages->state_offset,
             svm_pages->dma_addr[0].dir, 0);

Since npages = 0, svm_pages->dma_addr[0] was never initialized, leading to
svm_pages->dma_addr[0].dir being read from uninitialized heap memory.

>  		i += 1 << order;
>  		num_dma_mapped = i;
>  		flags.has_dma_mapping = true;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=11

Re: [PATCH v2 21/33] drm/xe: Add defrag GT stats

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The statistic `XE_GT_STATS_ID_DEFRAG_MB_MOVED` systematically undercounts data moved because it truncates the byte count to megabytes *before* accumulating it into the global counter. Any defragmentation move smaller than 1 MiB (e.g., re-backing a 64 KiB page) will evaluate to 0 and not be counted at all. Additionally, the sub-megabyte remainders of larger moves are permanently lost.
--

--- Patch [21]: [PATCH v2 21/33] drm/xe: Add defrag GT stats ---
commit a152b345237256025e6e622bf740f0941373de67
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add defrag GT stats

This commit introduces observability for the page defragmentation worker by
adding several primary-GT statistics counters. These counters track the
success, failure, and amount of data moved during defragmentation attempts.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 6a847e42946ce..c1bbcf0d21edb 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -1264,6 +1267,20 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
>  	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
>  	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
>  
> +	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm))
> +		xe_gt_stats_incr(xe_root_mmio_gt(xe),
> +				 XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
> +				 1);
> +	else
> +		xe_gt_stats_incr(xe_root_mmio_gt(xe),
> +				 ret ? XE_GT_STATS_ID_DEFRAG_FAILED_COUNT :
> +				       XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT, 1);
> +
> +	if (!ret)
> +		xe_gt_stats_incr(xe_root_mmio_gt(xe),
> +				 XE_GT_STATS_ID_DEFRAG_MB_MOVED,
> +				 *consumed >> 20);

[Severity: Medium]
Will this undercount the data moved if we truncate to megabytes before
accumulation? 

If the consumed amount is less than 1 MiB (e.g. re-backing a 64 KiB page),
this evaluates to 0 and nothing is added to the global counter. The
sub-megabyte remainders of larger moves also appear to be permanently lost
on every function call.

Would it be better to accumulate the exact value in bytes (or kilobytes)
and only perform the division when presenting the statistic to user-space?

>  unlock:
>  	xe_bo_unlock(bo);
>  	return ret;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=21

Re: [PATCH v2 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Uninitialized variable `ccs_ofs` and kernel assertion failure in `xe_migrate_ccs_copy()` during `defrag_ccs` pass for TT memory on DGFX.
- [High] Defragmentation bypasses mandatory CCS metadata clearing for BOs without CCS, causing a security/information leak.
--

--- Patch [17]: [PATCH v2 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies ---
commit b14779b7d60fb7520bb68944732a8ae6ceb089e7
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

Add xe_migrate_copy_defrag(), a copy helper for the upcoming page
defragmentation path where a single BO has its backing reallocated at a
more beneficial page order. The contents are relocated in up to two
passes (data and CCS) because they require opposite compression PAT
settings.

> diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
> index 9428dd5e7760c..956fd75454fa0 100644
> --- a/drivers/gpu/drm/xe/xe_migrate.c
> +++ b/drivers/gpu/drm/xe/xe_migrate.c
[ ... ]
> @@ -905,20 +931,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
>  	bool src_is_vram = mem_type_is_vram(src->mem_type);
>  	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
>  	bool type_device = src_bo->ttm.type == ttm_bo_type_device;
> -	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
> +	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
> +		!flags.defrag_copy;
> +	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
> -		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
> -	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
> +		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
> +		!flags.defrag_copy;
> +	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
> +		!flags.defrag_ccs && !flags.defrag_copy;

[Severity: High]
If a BO does not use CCS, does forcing needs_ccs_emit to false during the
data pass bypass the mandatory security clearing of its CCS metadata?

In xe_migrate_copy_defrag(), if need_ccs is false, it performs the data pass
and returns early:

    struct dma_fence *fence =
        __xe_migrate_copy(..., (struct xe_migrate_copy_flags) {
            .defrag_copy = true,
        });

    if (IS_ERR(fence) || !need_ccs)
        return fence;

Since needs_ccs_emit is false when flags.defrag_copy is true,
xe_migrate_ccs_copy() is never called to clear the CCS metadata. Could this
leave stale cross-process data in the newly allocated destination pages?

[ ... ]
> @@ -1014,7 +1053,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
>  		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
>  		update_idx = bb->len;
>  
> -		if (!copy_only_ccs)
> +		if (!flags.copy_only_ccs && !flags.defrag_ccs)
>  			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
>  
>  		if (needs_ccs_emit)

[Severity: Critical]
Can this pass an uninitialized ccs_ofs to xe_migrate_ccs_copy() and trigger
a kernel assertion failure on DGFX?

During a defrag_ccs pass for a TT memory BO, flags.defrag_ccs is true, which
forces copy_system_ccs to false:

    bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
        !flags.defrag_ccs && !flags.defrag_copy;

Because copy_system_ccs is false, the loop skips the initialization of ccs_ofs:

    if (copy_system_ccs) {
        ...
        batch_size += pte_update_size(..., &ccs_ofs, ...);
    }

However, needs_ccs_emit remains true (since flags.defrag_copy is false). This
causes xe_migrate_ccs_copy() to be called with the uninitialized ccs_ofs.

Furthermore, for TT memory on DGFX, both src_is_vram and dst_is_vram are false.
These are passed as the indirect flags to xe_migrate_ccs_copy():

    flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
                      IS_DGFX(xe) ? src_is_vram : src_is_pltt,
                      dst_L0_ofs,
                      IS_DGFX(xe) ? dst_is_vram : dst_is_pltt,
                      src_L0, ccs_ofs, copy_ccs);

Inside xe_migrate_ccs_copy(), this will trigger the assertion because both
indirect flags are false:

    xe_gt_assert(gt, src_is_indirect || dst_is_indirect);
    if (!src_is_indirect)
        src_ofs = ccs_ofs;

And then it programs the GPU copy engine with the uninitialized ccs_ofs,
which could lead to arbitrary GPU memory access or data corruption.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=17

Re: [PATCH v2 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] `ttm_pool_harvest_remaining()` fetches uninitialized `page->private` on tail pages when the defragmentation budget is exhausted, causing a NULL pointer dereference or buddy allocator corruption.
- [Critical] Double-free and Undefined Behavior in `ttm_tt_defrag_disown_borrowed()` when defragmenting pools that use `dma_alloc`.
- [Medium] Negative `defrag_bytes_remaining` disables the defragmentation budget instead of treating it as exhausted, breaking batch budget tracking.
--

--- Patch [8]: [PATCH v2 08/33] drm/ttm: Bound page (re)allocation per defragmentation move ---
commit 9447c53b431957799260b5fae41a105a050c3394
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/ttm: Bound page (re)allocation per defragmentation move

This commit bounds page reallocations per defragmentation move to avoid
stalling concurrent work for unbounded stretches when dealing with very large
objects. It introduces an optional per-move byte budget to allow a large
object to be upgraded in slices across successive moves.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index 5d2bbcce4eb89..a6847a16c47f1 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
[ ... ]
> @@ -885,14 +892,33 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
>  
>  static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
>  				      struct ttm_tt *old_tt, pgoff_t off,
> -				      struct ttm_pool_alloc_state *alloc)
> +				      struct ttm_pool_alloc_state *alloc,
> +				      bool *suboptimal)
>  {
> +	unsigned int beneficial = ttm_pool_beneficial_order(pool);
> +	pgoff_t bnr = 1UL << beneficial;
> +
>  	while (alloc->remaining_pages) {
>  		struct page *p = old_tt->pages[off];
>  		unsigned int order = ttm_pool_page_order(pool, p);

[Severity: Critical]
Does this code safely handle the case where off falls inside a multi-page
chunk rather than exactly on a chunk head?

During defragmentation, new_tt allocates chunks independently of old_tt's
chunk boundaries. When the budget is exhausted, ttm_pool_harvest_remaining()
is called with an off that may land on a tail page.

If this fetches a tail page, ttm_pool_page_order() will read its uninitialized
private field. For DMA pools, this uninitialized private field is cast to a
struct ttm_pool_dma pointer and dereferenced, causing a kernel panic. For
non-DMA pools, it evaluates to order 0, corrupting the chunk boundaries and
potentially leading to a double-free.

>  		pgoff_t nr = 1UL << order;
[ ... ]
> @@ -1106,10 +1172,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>  		gfp_flags |= GFP_HIGHUSER;
>  
>  	it.gfp_flags = gfp_flags;
> +	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;

[Severity: Medium]
Will this properly cap the allocation if the budget is overdrawn and becomes
negative?

When the allocator decrements defrag_bytes_remaining by the size of the newly
allocated chunks, a chunk can exceed the exact remaining budget, causing the
value to drop below zero.

If this overdrawn negative budget is passed to the next BO in the batch,
alloc->defrag_bytes_remaining > 0 will evaluate to false. The allocator will
then treat the negative budget as unlimited rather than exhausted, bypassing
the cap entirely and stalling concurrent work.

>  
>  	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
[ ... ]

Additionally, looking at ttm_tt_defrag_disown_borrowed() in
drivers/gpu/drm/ttm/ttm_tt.c:

drivers/gpu/drm/ttm/ttm_tt.c:ttm_tt_defrag_disown_borrowed() {
    ...
    unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
    pgoff_t nr = 1UL << order;
    ...
}

[Severity: Critical]
Is it safe to hardcode ttm_pool_page_order_nodma() here when defragmenting
pools that use dma_alloc?

Drivers that use dma_alloc pools (e.g., amdgpu, vmwgfx) store a pointer to a
struct ttm_pool_dma in the page->private field. Casting this pointer to an
integer order results in a huge bit shift, which destroys the array stride.
This silently fails to disown the shared pages, leading to an exploitable
use-after-free or double-free when the old tt is destroyed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=8

Re: [PATCH v2 24/33] drm/xe: Add defrag profiling tracepoints

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] 64-bit integer division in TP_printk causes a link error (__udivdi3 undefined) on 32-bit architectures.
--

--- Patch [24]: [PATCH v2 24/33] drm/xe: Add defrag profiling tracepoints ---
commit 414478b1f94933fc988a5668d191a4c7976fd319
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add defrag profiling tracepoints

This commit adds two tracepoints, xe_bo_defrag_worker and xe_bo_defrag_one,
to profile the page defragmentation worker and break down the cost of a
defrag attempt.

> diff --git a/drivers/gpu/drm/xe/xe_trace_bo.h b/drivers/gpu/drm/xe/xe_trace_bo.h
> index 86323cf3be2c0..a169a322464b2 100644
> --- a/drivers/gpu/drm/xe/xe_trace_bo.h
> +++ b/drivers/gpu/drm/xe/xe_trace_bo.h

[ ... ]

> +TRACE_EVENT(xe_bo_defrag_one,

[ ... ]

> +	    TP_printk("dev=%s size=%zu budget=%llu prealloc=%u/%u lock_us=%llu lockwait_us=%llu prealloc_us=%llu consumed=%llu ret=%d",
> +		      __get_str(dev), __entry->size, __entry->budget,
> +		      __entry->prealloc_got, __entry->prealloc_want,
> +		      __entry->lock_ns / 1000, __entry->lockwait_ns / 1000,
> +		      __entry->prealloc_ns / 1000, __entry->consumed,
> +		      __entry->ret)
> +);

[Severity: Low]
Will doing 64-bit integer divisions (__entry->lock_ns / 1000, etc.) in
TP_printk cause build failures on 32-bit architectures?

The C code generated by TP_printk for TRACE_EVENT(xe_bo_defrag_one) performs
direct u64 division. On 32-bit platforms, this translates to a dependency on
the __udivdi3 libgcc function. Since the kernel does not export this function,
this will result in a linking error during 32-bit builds (such as when
COMPILE_TEST is enabled).

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=24

Re: [PATCH v2 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] GFP_KERNEL allocation in shrinker and dma-resv critical section causes deadlocks and lockdep splats
--

--- Patch [13]: [PATCH v2 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind() ---
commit 985db7b22213ce9503872b19a27fddeba9fe24ff
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

This commit replaces synchronous TLB invalidations during buffer object
eviction with asynchronous flushes on L2-flush-optimized hardware. It
queues a TLB invalidation job per VMA, which waits on the buffer's
in-flight GPU work before flushing.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -732,6 +732,13 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
>  			 */
>  			if (!xe_device_is_l2_flush_optimized(xe))
>  				continue;
> +
> +			/*
> +			 * Attempt to flush L2 async, fallback to sync flush on
> +			 * failure
> +			 */
> +			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
> +				continue;

[Severity: High]
Does this create a deadlock regression when reached from the shrinker path?

The memory shrinker can call into this path through xe_shrinker_scan() ->
xe_bo_shrink() -> xe_bo_move_notify() -> xe_bo_trigger_rebind().

In this path, the shrinker operates under the fs_reclaim pseudo-lock and
holds the buffer object's dma-resv lock.

> diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
> --- a/drivers/gpu/drm/xe/xe_vm.c
> +++ b/drivers/gpu/drm/xe/xe_vm.c
[ ... ]
> +static int xe_vma_tlb_flush_client(struct xe_vm *vm, struct xe_vma *vma,
> +				   struct xe_tile *tile, struct xe_gt *gt,
> +				   struct dma_resv *resv, struct dma_fence *dep,
> +				   int type)
> +{
> +	struct xe_exec_queue *q = xe_migrate_exec_queue(tile->migrate);
> +	struct xe_tlb_inval_job *job;
> +	struct dma_fence *fence;
> +	int err;
> +
> +	job = xe_tlb_inval_job_create(q, &gt->tlb_inval,
> +				      q->tlb_inval[type].dep_scheduler, vm,
> +				      xe_vma_start(vma), xe_vma_end(vma), type);

[Severity: High]
Could this allocation recurse into memory reclaim and attempt to acquire the
same dma-resv lock, or trigger a lockdep splat for context violations?

The xe_tlb_inval_job_create() function allocates the job using a default
GFP_KERNEL allocation. Furthermore, the subsequent call to
xe_tlb_inval_job_alloc_dep() explicitly calls might_alloc(GFP_KERNEL).

Since this can be reached while holding the dma-resv lock or while in the
shrinker reclaim path, these GFP_KERNEL allocations appear to violate the
context constraints.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=13

Re: [PATCH v2 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Out-of-bounds array access due to incorrect page shift in start_page calculation
- [High] Hardware assertion failure and command stream corruption due to using PAGE_SIZE as pitch
--

--- Patch [19]: [PATCH v2 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves ---
commit 0cb7423f91b7764bc47ff13572694c591c1f58e2
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Skip self-copies for borrowed pages on defrag moves

A defrag move may harvest already beneficial-order chunks from the old
tt, so the freshly reallocated backing borrows (references) some of the
same physical pages.

> diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
> index 956fd75454fa0..450ca8bf4f3b6 100644
> --- a/drivers/gpu/drm/xe/xe_migrate.c
> +++ b/drivers/gpu/drm/xe/xe_migrate.c

[ ... ]

> @@ -835,6 +835,63 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
>  		emit_xy_fast_copy(gt, bb, src_ofs, dst_ofs, size, pitch);
>  }
>  
> +/*
> + * emit_copy_chunks() - Emit the data blits for one defrag L0 chunk, skipping

[ ... ]

> +		/* Coalesce a run of pages that actually need copying. */
> +		while (i + run < npages &&
> +		       src_tt->pages[start_page + i + run] !=
> +		       dst_tt->pages[start_page + i + run])
> +			run++;
> +
> +		if (bb)
> +			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
> +				  dst_L0_ofs + (u64)i * PAGE_SIZE,
> +				  run * PAGE_SIZE, PAGE_SIZE);

[Severity: High]
Can the use of PAGE_SIZE here cause a hardware assertion failure and command
stream corruption on systems with 64KB pages?

The PAGE_SIZE is passed as the pitch parameter to emit_copy(), which flows into
emit_xy_fast_copy() where there is a hardware limit check:

xe_gt_assert(gt, pitch <= U16_MAX);

On 64KB page systems, PAGE_SIZE is 65536, which exceeds U16_MAX (65535). This
will unconditionally trigger the driver assertion. Furthermore, because the
pitch is bitwise OR'ed into the BLT command instruction, won't this oversized
pitch overflow its bit field and corrupt adjacent fields in the GPU command?

Should this use XE_PAGE_SIZE (4096) instead, similar to the fallback path
in normal emit_copy() usage?

[ ... ]

> @@ -985,6 +1052,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
>  		u64 ccs_ofs, ccs_size;
>  		u32 ccs_pt;
>  		u32 pte_flags;
> +		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> XE_PTE_SHIFT;
> +		u32 ndata_blits = 1;
>  
>  		bool usm = xe->info.has_usm;

[Severity: High]
Is there a risk of an out-of-bounds array access here on systems with a
page size larger than 4KB (like 64KB on ARM64 or PPC64)?

By shifting with XE_PTE_SHIFT (12), the index is computed in terms of 4KB
GPU pages. However, this index is later passed to emit_copy_chunks() to
index the src_tt->pages and dst_tt->pages arrays:

if (src_tt->pages[start_page + i] == dst_tt->pages[start_page + i]) {

Since the ttm_tt page arrays are sized based on the system's PAGE_SHIFT,
wouldn't an index calculated with XE_PTE_SHIFT grow much faster than the
array bounds on a 64KB page system, leading to a memory out-of-bounds read?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=19

Re: [PATCH v2 15/33] drm/xe: Track BOs backed at a sub-optimal page order

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] BOs moved to SYSTEM memory during a failed shrinker purge are incorrectly left on the defrag list.
- [Medium] Framebuffer pinning bypasses defrag list tracking updates, violating the invariant that pinned BOs are removed from the list.
--

--- Patch [15]: [PATCH v2 15/33] drm/xe: Track BOs backed at a sub-optimal page order ---
commit d8c62c56c566775d903f43756778e071af0acb80
Author: Matthew Brost <matthew.brost@intel.com>

    drm/xe: Track BOs backed at a sub-optimal page order

    Add per-device tracking of BOs whose backing TT pages were allocated
    below the pool's beneficial order, as reported by the
    TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag. These BOs are future
    candidates for a defragmentation pass that reallocates their pages at the
    beneficial order.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index cca617cf34d13..d7b0536564ab2 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -1261,6 +1358,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
>  	if (lret > 0) {
>  		xe_ttm_tt_account_subtract(xe, bo->ttm);
>  		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
> +
> +		/*
> +		 * The pages are gone, so a tracked BO no longer needs defrag.
> +		 * This shrinker purge path frees pages without going through
> +		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
> +		 * bos are never tracked, so guard for real xe bos.
> +		 */
> +		if (xe_bo_is_xe_bo(bo))
> +			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
>  	}
>  
>  	return lret;

[Severity: Medium]
When xe_bo_shrink_purge() attempts to purge a BO, it uses ttm_bo_move_null() to
move it to XE_PL_SYSTEM, which bypasses xe_bo_defrag_update(). 

If the subsequent ttm_bo_shrink() call fails (lret <= 0), this explicit
xe_bo_defrag_remove() is skipped.

Could this leave a tracked BO in SYSTEM memory while incorrectly remaining on
the defrag list, violating the tracking invariant and wasting worker cycles?

[ ... ]
> @@ -3072,6 +3183,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
>  	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
>  		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
>  
> +	xe_bo_defrag_remove(bo);
> +
>  	/*
>  	 * FIXME: If we always use the reserve / unreserve functions for locking
>  	 * we do not need this.

[Severity: Medium]
Will this miss defrag list tracking updates for framebuffer BOs?

xe_fb_pin() and __xe_unpin_fb_vma() in drivers/gpu/drm/xe/display/xe_fb_pin.c
call ttm_bo_pin() and ttm_bo_unpin() directly rather than using this
xe_bo_pin() wrapper.

If a framebuffer BO resides in XE_PL_TT with sub-optimal page allocations,
could it remain on the defrag list even when pinned, causing the defrag
worker to incorrectly process pinned BOs?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=15

Re: [PATCH v2 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Defragmentation of DMA-allocated or external TTs causes massive out-of-bounds memory writes due to misinterpreting page private data as an allocation order.
- [Critical] Misaligned chunks between the old and new TT lead to partial chunk harvesting, resulting in a severe use-after-free.
- [High] If the driver's move callback returns `-EMULTIHOP`, the defragmentation rollback is bypassed, causing a permanent memory leak of the original buffer object TT.
--

--- Patch [7]: [PATCH v2 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves ---
commit 74e91ed102adee47c82989ab7b4f83fd70da47c2
Author: Matthew Brost <matthew.brost@intel.com>
drm/ttm: Harvest beneficial-order pages on defragmentation moves

A defragmentation move relocates a populated tt into a freshly allocated
one so that sub-beneficial-order chunks can be upgraded to the device's
beneficial order. Previously, if allocation could not complete entirely
at beneficial order, the move failed and all progress was discarded.
Instead, once allocation has successfully upgraded part of the object,
the remainder of the old tt is harvested to complete the new tt. Thread
the BO's old, still-populated tt into the page allocator via a new
ttm_operation_ctx::defrag_old_tt.

> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
[ ... ]
> @@ -218,10 +225,18 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
>  	if (defrag)
>  		ttm_bo_unmap_virtual(bo);
>  
>  	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
>  	if (ret) {
>  		if (ret == -EMULTIHOP)
>  			return ret;
>  		goto out_err;
>  	}
>  
>  	if (bo->defrag_old_tt) {

[Severity: High]
If bdev->funcs->move() returns -EMULTIHOP, does returning immediately bypass
the out_err rollback path?

It looks like the original TT remains stashed in bo->defrag_old_tt instead of
being restored.

When the move is retried, it appears bo->defrag_old_tt would be overwritten
or cleared, causing the original buffer object TT and its unborrowed pages
to be permanently leaked.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
[ ... ]
> +static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
> +{
> +	unsigned int harvest_order;
> +	pgoff_t off;
> +	int r;
> +
> +	if (!it->defrag_old_tt || !it->beneficial_order)
> +		return TTM_POOL_ITER_FILL;
> +
> +	off = it->tt->num_pages - it->alloc->remaining_pages;
> +	harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
> +						  off, &it->old_cursor);
> +
> +	if (harvest_order < it->beneficial_order &&
> +	    it->order >= it->beneficial_order)
> +		return TTM_POOL_ITER_FILL;
> +
> +	it->order = harvest_order;
> +	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);

[Severity: Critical]
What happens if the new TT allocation size is unaligned, causing the offset
into the old TT to land in the middle of a large chunk?

Could ttm_pool_iter_reuse_old() or ttm_pool_harvest_remaining() harvest a
partial chunk, such as just the tail page?

If it harvests a tail page, the page order would be read as 0 since the
private data is only populated on the head page.

> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
[ ... ]
> +void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
> +{
> +	pgoff_t i = 0, run_start = 0;
> +	bool in_run = false;
[ ... ]
> +	while (i < tt->num_pages) {
> +		struct page *sp = src->pages[i];
> +		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
> +		pgoff_t nr = 1UL << order;

[Severity: Critical]
Can defragmentation be initiated on a buffer object that uses a DMA-allocated
or external TT?

Looking at ttm_bo_handle_move_mem(), it does not appear to restrict
defragmentation to non-DMA pools.

If a DMA-allocated TT is defragmented, wouldn't ttm_pool_page_order_nodma()
incorrectly read the DMA pointer stored in the page private data as an
integer order?

This would result in a huge bitshift when calculating nr, which is then
used in the loop increment or the memset() calls, potentially causing severe
out-of-bounds memory writes.

> +		if (sp && tt->pages[i] == sp) {
> +			if (!in_run) {
> +				run_start = i;
> +				in_run = true;
> +			}
> +		} else if (in_run) {
> +			memset(&tt->pages[run_start], 0,
> +			       (i - run_start) * sizeof(*tt->pages));

[Severity: Critical]
Following up on the potential for harvesting an unaligned tail page (noted in
ttm_pool.c), if ttm_pool_iter_reuse_old() harvested a tail page, wouldn't
this logic clear the pointer in the old TT for that tail page, but leave the
head page pointer intact?

When the old TT is eventually destroyed, wouldn't ttm_pool_free_range() free
the entire chunk back to the system (since the head page still indicates a
large order) while the tail page is still actively mapped and used by the new
TT, leading to a severe use-after-free regression?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=7

Re: [PATCH v2 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Tracepoint metric `lock_us` incorrectly reports system uptime on early error paths and includes lock-dropped backoff time during contention retries.
--

--- Patch [26]: [PATCH v2 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl ---
commit 245e602165eaf1749a023b1941ce834af1e88d5e
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add tracepoint for xe_gem_create_ioctl

This commit adds a tracepoint to xe_gem_create_ioctl to profile GEM creation,
particularly tracking the time spent under the dma-resv lock versus total IOCTL
time. The asid is captured at VM lookup time since the VM reference is dropped
before the tracepoint fires.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index d9226a3995fe4..c096f3dbd5289 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -4057,6 +4062,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
>  			if (err)
>  				break;
>  		}
> +
> +		if (lock_start == ktime_set(0, 0))
> +			lock_start = ktime_get();

[Severity: Medium]
Will this include lock-dropped backoff time if there is lock contention? 

If a retry occurs and the validation loop restarts, lock_start is not reset 
to the actual lock acquisition time because it is no longer 0.

> +
>  		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
>  				       bo_flags, have_prealloc ? &prealloc : NULL,
>  				       &exec);
> @@ -4067,6 +4076,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
>  			break;
>  		}
>  	}
> +	lock_ns = ktime_to_ns(ktime_sub(ktime_get(), lock_start));

[Severity: Medium]
Does this calculate a duration based on system uptime on early error paths? 

If the validation loop breaks early (e.g. from an -EINTR in xe_vm_drm_exec_lock) 
before lock_start is set, lock_start will still be 0. 

>  	if (have_prealloc)
>  		ttm_pool_prealloc_fini(&xe->ttm.pool, &prealloc);
>  	if (err)

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=26

Re: [PATCH v2 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Integer truncation in xe_device_ccs_bytes() returning u32 causes buffer overflows and physical memory corruption/panics for BOs >= 1TB.
--

--- Patch [25]: [PATCH v2 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock ---
commit aa051bafd6b83b01e59b139bb60870dfa1044d22
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Preallocate system BO backing outside the dma-resv lock

xe_gem_create_ioctl() populates a buffer object's backing store inside
xe_validation_guard(), i.e. under the dma-resv lock. This patch uses the
TTM full-preallocation mechanism to move that work out of the lock.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index a010042653495..d9226a3995fe4 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -3994,6 +4009,45 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
>  			return -ENOENT;
>  	}
>  
> +	/*
> +	 * For system-only BOs, preallocate the whole backing outside the
> +	 * dma-resv lock so the populate under the validation guard does not
> +	 * stall in reclaim/compaction. The caching mirrors the system-BO path
> +	 * in xe_ttm_tt_create(); the pool falls back to in-line allocation for
> +	 * any shortfall, so a failed fill is harmless.
> +	 */
> +	if ((bo_flags & XE_BO_FLAG_SYSTEM) &&
> +	    !(bo_flags & (XE_BO_FLAG_DEFER_BACKING | XE_BO_FLAG_VRAM_MASK |
> +			  XE_BO_FLAG_STOLEN))) {
> +		enum ttm_caching caching = ttm_cached;
> +		unsigned long num_pages = args->size >> PAGE_SHIFT;
> +
> +		if (!IS_DGFX(xe) &&
> +		    args->cpu_caching == DRM_XE_GEM_CPU_CACHING_WC)
> +			caching = ttm_write_combined;
> +
> +		/*
> +		 * Cover the CCS backup pages appended to the tt (see
> +		 * xe_ttm_tt_create()) so the whole tt is preallocated.
> +		 */
> +		if (__xe_bo_needs_ccs_pages(xe, ttm_bo_type_device, bo_flags,
> +					    args->cpu_caching))
> +			num_pages += DIV_ROUND_UP(xe_device_ccs_bytes(xe,
> +								      args->size),
> +						  PAGE_SIZE);

[Severity: High]
This is a pre-existing issue, but can this lead to an integer truncation for
very large buffer sizes?

The function xe_device_ccs_bytes() in drivers/gpu/drm/xe/xe_device.c returns
a u32, but takes a u64 size parameter. If a user requests a BO allocation of
>= 1TB via DRM_IOCTL_XE_GEM_CREATE with CCS enabled, the required CCS size
would be 1TB / 256 = 4GB, which silently truncates to 0 in the u32 return
type.

Because this patch propagates the pattern by calling xe_device_ccs_bytes() on
the user-controlled args->size, the extra_pages for the CCS backup could be
calculated as 0, under-allocating the BO's SG table. Later, when
xe_migrate_copy() processes the BO in chunks, it could advance the CCS cursor
past the end of the under-allocated SG table, resulting in a NULL pointer
dereference and kernel panic.

Should xe_device_ccs_bytes() be updated to return a u64, or should there be
size limits enforced earlier in the ioctl?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=25

Re: [PATCH v2 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

New issues:
- [High] xe_device_assert_dma_pages_zero() is called too early in xe_device_destroy(), leading to spurious kernel assertion failures.
- [High] Uncapped `addr->order` in SVM and userptr DMA accounting leads to out-of-bounds array writes.
- [Low] Too many arguments to `drm_gpusvm_init` in `xe_svm.h` causes a compilation error when `CONFIG_DRM_GPUSVM` is enabled but `CONFIG_DRM_XE_GPUSVM` is disabled.

Pre-existing issues:
- [Critical] Asynchronous execution of `vm_destroy_work_func` can cause a Use-After-Free of the `xe` device structure.
--

--- Patch [12]: [PATCH v2 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order ---
commit 808c14d37a18acfee1cc7559a4568116c9d63f8b
Author: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Subject: drm/xe: Add debugfs stats for DMA-mapped pages per order

This commit adds per-page-order DMA mapping counters for system memory
accessible by the GPU. The stats cover both TTM buffer objects and SVM/userptr
ranges, exposing them through debugfs.

> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> index ad7f3e61d4579..0c4ee874a56d7 100644
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
[ ... ]
> @@ -475,8 +499,10 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)
>  {
>  	struct xe_device *xe = to_xe_device(dev);
>  
> +	xe_device_assert_dma_pages_zero(xe);
> +
>  	xe_bo_dev_fini(&xe->bo_device);

[Severity: High]
Does this assertion trigger prematurely if the asynchronous BO destruction
worker has pending buffers to free?

In xe_device_destroy(), xe_device_assert_dma_pages_zero() is called
before xe_bo_dev_fini() flushes the async_free worker. If there are
delayed destructions, wouldn't the DMA mappings still be active, causing
the counters to be non-zero and triggering a kernel warning?

> diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c
> index b228a737cfd6d..2f639fc0cac4e 100644
> --- a/drivers/gpu/drm/xe/xe_svm.c
> +++ b/drivers/gpu/drm/xe/xe_svm.c
[ ... ]
> +static void xe_svm_dma_map_account(struct drm_gpusvm *gpusvm,
> +				   const struct drm_pagemap_addr *addr,
> +				   int sign)
> +{
> +	struct xe_device *xe = gpusvm_to_vm(gpusvm)->xe;
> +	unsigned int order = addr->order;
> +
> +	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
> +		return;
> +
> +	atomic_long_add((long)(1UL << order) * sign,
> +			&xe->mem.dma_mapped_pages_svm[order]);

[Severity: High]
Can this cause an out-of-bounds write if addr->order exceeds the array
bounds of dma_mapped_pages_svm?

In xe_svm_dma_map_account(), addr->order is used directly as an array
index. If an unprivileged user maps a large huge page (such as a 1GB page
with an order of 18) and faults it into the GPU, wouldn't this exceed the
NR_PAGE_ORDERS limit and corrupt adjacent kernel memory?

> +}
> +#endif

> diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h
> index a921556d34662..edd668c415512 100644
> --- a/drivers/gpu/drm/xe/xe_svm.h
> +++ b/drivers/gpu/drm/xe/xe_svm.h
[ ... ]
> @@ -236,7 +236,8 @@ int xe_svm_init(struct xe_vm *vm)
>  {
>  #if IS_ENABLED(CONFIG_DRM_GPUSVM)
>  	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
> -			       NULL, 0, 0, 0, NULL, NULL, 0);
> +			       NULL, 0, 0, 0, NULL, NULL,
> +			       xe_userptr_gpusvm_ops_get(), 0);

[Severity: Low]
Does this function call pass too many arguments to drm_gpusvm_init()?

In the xe_svm_init() inline stub, the xe_userptr_gpusvm_ops_get() argument
was added without removing one of the NULL placeholders. Since
drm_gpusvm_init() expects 9 arguments and this now provides 10, won't this
cause a compilation error when CONFIG_DRM_GPUSVM is enabled but
CONFIG_DRM_XE_GPUSVM is disabled?

>  #else
>  	return 0;
>  #endif

> diff --git a/drivers/gpu/drm/xe/xe_userptr.c b/drivers/gpu/drm/xe/xe_userptr.c
> index 8b2d461ea0b28..79c2a81001f2e 100644
> --- a/drivers/gpu/drm/xe/xe_userptr.c
> +++ b/drivers/gpu/drm/xe/xe_userptr.c
[ ... ]
> +static void xe_userptr_dma_map_account(struct drm_gpusvm *gpusvm,
> +				       const struct drm_pagemap_addr *addr,
> +				       int sign)
> +{
> +	struct xe_vm *vm = container_of(gpusvm, struct xe_vm, svm.gpusvm);
> +	unsigned int order = addr->order;
> +
> +	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
> +		return;
> +
> +	atomic_long_add((long)(1UL << order) * sign,
> +			&vm->xe->mem.dma_mapped_pages_svm[order]);

[Severity: High]
Similarly to xe_svm_dma_map_account(), can this cause an out-of-bounds
write if addr->order is larger than NR_PAGE_ORDERS?

Using addr->order directly as an array index without capping it could allow
unprivileged user-space applications mapping large huge pages to corrupt
memory when invoking xe_userptr_dma_map_account().

> +}


While reviewing the VM lifecycle code related to this patch, I noticed a
separate pre-existing issue in drivers/gpu/drm/xe/xe_vm.c.

[Severity: Critical]
This is a pre-existing issue, but does queueing the destroy work
asynchronously lead to a use-after-free if the last reference to the DRM
device is dropped before the worker executes?

In xe_vm_free(), the destruction work is queued to system_dfl_wq:

	queue_work(system_dfl_wq, &vm->destroy_work);

If this drops the last device reference, the xe device could be freed via
RCU before vm_destroy_work_func() runs. Since vm_destroy_work_func()
dereferences the xe pointer (for example, when calling
xe_pm_runtime_put()), wouldn't this result in a use-after-free?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=12

Re: [PATCH v2 27/33] drm/xe: Add IOVA-based xe_res_cursor variant

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unreachable dead code block added for IOVA cursor segment advancement in xe_res_next().
--

--- Patch [27]: [PATCH v2 27/33] drm/xe: Add IOVA-based xe_res_cursor variant ---
commit 8700355546957aacaa7e49fc841d6e860b3ed9e7
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add IOVA-based xe_res_cursor variant

Introduce xe_res_first_iova() to walk a buffer that was DMA mapped using
the IOVA-based DMA API. Such a mapping is described by a struct
dma_iova_state and is a single contiguous IOVA range, so the cursor walks
it as one segment.

> diff --git a/drivers/gpu/drm/xe/xe_res_cursor.h b/drivers/gpu/drm/xe/xe_res_cursor.h
> index 0522caafd89d7..f0f1d699e16e5 100644
> --- a/drivers/gpu/drm/xe/xe_res_cursor.h
> +++ b/drivers/gpu/drm/xe/xe_res_cursor.h

[ ... ]

> @@ -255,6 +264,41 @@ static inline void xe_res_first_dma(const struct drm_pagemap_addr *dma_addr,
> +static inline void xe_res_first_iova(struct dma_iova_state *state,
> +				     u64 start, u64 size,
> +				     struct xe_res_cursor *cur)
> +{
> +	XE_WARN_ON(!state);
> +	XE_WARN_ON(start + size > dma_iova_size(state));
> +
> +	cur->node = NULL;
> +	cur->start = start;
> +	cur->remaining = size;
> +	cur->size = size;
> +	cur->dma_addr = NULL;
> +	cur->sgl = NULL;
> +	cur->dma_start = state->addr;
> +	cur->dma_seg_size = dma_iova_size(state);
> +	cur->mem_type = XE_PL_TT;
> +	cur->is_iova = true;
>  }

[ ... ]

> @@ -283,6 +327,13 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
>  	if (cur->size > size) {
>  		cur->size -= size;
>  		cur->start += size;
>  		return;
>  	}
>  
> +	if (cur->is_iova) {
> +		/* Single contiguous IOVA segment. */
> +		cur->start += size;
> +		cur->size = cur->remaining;
> +		return;
> +	}
> +

[Severity: Low]
Is this block unreachable dead code?

In xe_res_first_iova(), the cursor is initialized as a single segment where
cur->size is equal to cur->remaining:

    cur->remaining = size;
    cur->size = size;

When xe_res_next(cur, size) is called, it first subtracts size from
cur->remaining. If the entire remaining segment is consumed, it returns
early via if (!cur->remaining).

If only a portion of the segment is consumed, size is strictly less than
cur->size. This causes the function to always take the early return path
immediately above this newly added block:

    if (cur->size > size) {
        cur->size -= size;
        cur->start += size;
        return;
    }

Because an IOVA cursor is a single contiguous segment where cur->size and
cur->remaining start perfectly synchronized, does it ever need to bypass
these early returns and evaluate the if (cur->is_iova) block?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=27

Re: [PATCH v2 20/33] drm/xe: Add a page defragmentation worker

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] Accessing `bo->ttm.ttm` outside of the BO's dma-resv lock can cause a NULL pointer dereference or data race.
- [Critical] The defrag delayed worker can be scheduled during device probe before its teardown is registered, leading to a UAF if the probe fails.
- [High] The defrag worker permanently stalls if the device enters runtime suspend.
- [High] The defrag worker fails to apply exponential backoff on sub-optimal fallback, causing an infinite spin loop.
--

--- Patch [20]: [PATCH v2 20/33] drm/xe: Add a page defragmentation worker ---
commit 7f5402dce2516caf0e93f47b50a3f18b01fe034e
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add a page defragmentation worker
    
This patch adds a per-device defrag list and delayed worker to track and
reallocate BOs whose backing TT pages were allocated below the device's
beneficial order. The worker runs periodically and attempts to re-back BOs
with beneficial-order pages, up to a maximum size budget per run.

> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -1271,6 +1364,96 @@ static void xe_bo_defrag_worker(struct work_struct *w)
>  	struct delayed_work *dwork = to_delayed_work(w);
>  	struct xe_device *xe =
>  		container_of(dwork, struct xe_device, mem.defrag.worker);
>  	u64 defrag_bytes = 0;
>  	bool requeue = false;
>  	int idx;
>  
>  	if (!drm_dev_enter(&xe->drm, &idx))
>  		return;
>  
>  	if (!xe_pm_runtime_get_if_in_use(xe)) {
>  		drm_dev_exit(idx);
>  		return;
>  	}

[Severity: High]
Does the defrag worker permanently stall if the device enters runtime suspend?

If the worker executes while the device is suspending,
xe_pm_runtime_get_if_in_use() returns false and the worker exits immediately
without rescheduling itself.

Because the list is not emptied, subsequent additions to the list in
xe_bo_defrag_add() will not see an empty-to-non-empty transition, meaning
xe_bo_defrag_schedule() will never be called again.

[ ... ]
> +		ret = xe_bo_defrag_one(xe, bo, XE_BO_DEFRAG_SIZE_LIMIT - defrag_bytes,
> +				       &consumed);
> +		defrag_bytes += consumed;
> +
> +		if (ret || ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {

[Severity: Critical]
Can accessing bo->ttm.ttm here cause a NULL pointer dereference?

The xe_bo_defrag_one() function releases the BO's dma-resv lock before it
returns. If a concurrent operation, such as evicting to VRAM or swapping out,
destroys the TT while the BO is unlocked, checking
ttm_tt_is_beneficial_order_failed(bo->ttm.ttm) might dereference a NULL
pointer.

> +			scoped_guard(spinlock, &xe->mem.defrag.lock) {
> +				if (ret)
> +					/*
> +					 * Abort the pass and retry the whole
> +					 * list later, backing off exponentially
> +					 * on every failure.
> +					 */
> +					xe->mem.defrag.interval_ms =
> +						min(xe->mem.defrag.interval_ms * 2,
> +						    (unsigned int)XE_BO_DEFRAG_INTERVAL_MAX_MS);
> +				else
> +					/* Progress, reset interval */
> +					xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
> +			}

[Severity: High]
Does the backoff logic fail to apply when a fallback allocation succeeds?

If system memory is highly fragmented, the TTM pool allocator might fail to
allocate beneficial-order pages but successfully fall back to a smaller order.
In this case, xe_bo_defrag_one() returns 0 for success, but
ttm_tt_is_beneficial_order_failed() remains true.

The backoff block checks if (ret). Since ret is 0, it takes the else branch,
interprets the run as progress, and resets the interval to 25ms. This could
result in a tight loop of repeated failed upgrades that wastes CPU time without
ever backing off.

> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
[ ... ]
> @@ -1063,6 +1063,15 @@ int xe_device_probe(struct xe_device *xe)
>  			return err;
>  	}
>  
> +	/*
> +	 * Register the defrag worker teardown now that the migrate contexts it
> +	 * depends on are initialized, so devm's reverse-order cleanup stops the
> +	 * worker before those contexts are torn down.
> +	 */
> +	err = xe_bo_defrag_init(xe);

[Severity: Critical]
Can the delayed worker outlive the device if probe fails before the teardown
is registered?

During early probe, components might allocate BOs. If these BOs fail to get
beneficial order pages, they are added to the defrag list, which schedules
the delayed worker via xe_bo_defrag_schedule().

If a probe step fails before reaching this late xe_bo_defrag_init() call,
the devm cleanup will roll back and free the device without cancelling the
worker. When the timer expires, the worker will access freed memory.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=20

Re: [PATCH v2 18/33] drm/xe: Handle defrag moves in xe_bo_move()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] Returning an error from xe_bo_move() after the move has been successfully committed causes a Use-After-Free of the new resource in TTM core.
- [High] Live migration during an asynchronous move causes silent loss of VF CCS metadata.
--

--- Patch [18]: [PATCH v2 18/33] drm/xe: Handle defrag moves in xe_bo_move() ---
commit dad45636f401d271e91aa77826c4290b4048e3c7
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Handle defrag moves in xe_bo_move()

This commit adds support for defrag moves in xe_bo_move(). It allows TTM
to reallocate a buffer object's backing at a beneficial order and handles
the relocation entirely on the GPU via xe_migrate_copy_defrag().

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 21d087170ef36..8e9fc8b3dffaf 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -1309,13 +1325,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
>  	 * BBs from BO as it is no longer needed.
>  	 */
>  	if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT &&
> -	    new_mem->mem_type == XE_PL_SYSTEM)
> +	    (new_mem->mem_type == XE_PL_SYSTEM || ctx->defrag))
>  		xe_sriov_vf_ccs_detach_bo(bo);

[Severity: High]
This isn't a bug introduced by this patch, but if a live migration is
triggered concurrently with a BO defragmentation or eviction, does this
synchronous CPU detachment cause a loss of CCS metadata?

Because xe_sriov_vf_ccs_detach_bo() and xe_sriov_vf_ccs_attach_bo() are
called synchronously on the CPU before the GPU has finished copying the data,
they immediately point the host to the new memory for CCS save and restore
operations.

If a live migration occurs before the GPU copy finishes, the host might save
uninitialized CCS data from the new memory, and fail to save the valid CCS
data from the old memory. Upon VM restore, the GPU migrate job resumes and
copies the corrupted old CCS to the new memory.

>  
>  	if (IS_VF_CCS_READY(xe) &&
>  	    ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
> -	     (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
> -	    handle_system_ccs)
> +	     ((old_mem_type == XE_PL_SYSTEM || ctx->defrag) &&
> +	      new_mem->mem_type == XE_PL_TT)) && handle_system_ccs)
>  		ret = xe_sriov_vf_ccs_attach_bo(bo);

[Severity: High]
This is a pre-existing issue, but can returning an error code here trigger
a use-after-free in the TTM core?

If xe_bo_move() commits the memory move earlier in the function by calling
ttm_bo_move_accel_cleanup() or ttm_bo_move_null(), new_mem is assigned to
bo->resource and the old memory is handed to a ghost object.

If xe_sriov_vf_ccs_attach_bo(bo) fails with an error like -ENOMEM, or if
any subsequent timeout wait fails, the error code is returned to the TTM
core.

The TTM core assumes a failed move means the new resource was not adopted:

ttm_bo_validate()
  ttm_bo_handle_move_mem()
    ret = ttm_bo_move()
    if (ret) {
        ...
        ttm_resource_free(bo, &res);
    }

Since bo->resource already points to the new resource, would this leave
bo->resource pointing to freed memory?

>  
>  out:

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=18

Re: [PATCH v2 23/33] drm/xe: Defrag using out-of-lock page preallocation

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Low] 64-bit division using the '/' operator causes a link-time build failure on 32-bit architectures (missing __udivdi3).

Pre-existing issues:
- [High] Accessing `bo->ttm.ttm` without holding the BO's dma-resv lock can lead to a NULL pointer dereference or UAF if the BO is concurrently evicted or destroyed.
--

--- Patch [23]: [PATCH v2 23/33] drm/xe: Defrag using out-of-lock page preallocation ---
commit d11ab210d871e5fedb57edb2f956ed1814f6d5a4
Author: Matthew Brost <matthew.brost@intel.com>
Date:   Fri Jul 10 21:54:32 2026 +0000

    drm/xe: Defrag using out-of-lock page preallocation
    
    Convert xe_bo_defrag_one() to a three-phase scheme that moves the
    expensive beneficial-order allocations out of the BO dma-resv lock.
    This hoists allocation and caching out to reduce BO-lock held time.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 405316d0d116b..2f014fc9f988e 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -1249,15 +1249,55 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,

[ ... ]

> +	if (order && ttm_bo->ttm) {
> +		u32 suboptimal = ttm_tt_suboptimal_pages(ttm_bo->ttm);
> +		u64 cap = min_t(u64, budget, xe_bo_size(bo)) >> PAGE_SHIFT;
> +
> +		/* Prealloc only the beneficial-order chunks the move replaces. */
> +		want = DIV_ROUND_UP(min_t(u64, suboptimal, cap), 1UL << order);
                 ^^^^^^^^^^^^

[Severity: Low]
Does this code cause a link-time build failure on 32-bit architectures?

The first argument to DIV_ROUND_UP() is a u64, and the second evaluates
to an unsigned long (32-bit integer) and is not a compile-time constant.
This results in a 64-bit division using the '/' operator, which emits a
call to __udivdi3 on 32-bit GCC. Because the kernel doesn't provide this
intrinsic, it will break the build.

Could DIV_ROUND_UP_ULL() be used here instead?

> +	}
> +	xe_bo_unlock(bo);

[ ... ]

This is a pre-existing issue, but I noticed a potential problem in
xe_bo_defrag_worker() regarding how the BO is accessed after unlocking:

drivers/gpu/drm/xe/xe_bo.c:xe_bo_defrag_worker() {
	...
		ret = xe_bo_defrag_one(xe, bo, XE_BO_DEFRAG_SIZE_LIMIT - defrag_bytes,
				       &consumed);
		defrag_bytes += consumed;

		if (ret || ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {
	...
}

[Severity: High]
Can accessing bo->ttm.ttm here without holding the BO's dma-resv lock lead
to a NULL pointer dereference or use-after-free?

xe_bo_defrag_worker() calls xe_bo_defrag_one(), which acquires the BO lock,
performs the move, and then unlocks the BO before returning. After returning,
the worker evaluates ttm_tt_is_beneficial_order_failed(bo->ttm.ttm) without
holding the BO lock.

If another thread concurrently evicts the BO to VRAM, the system memory
ttm_tt is destroyed and bo->ttm.ttm is set to NULL. Additionally, if
xe_bo_defrag_one() finds the BO no longer needs defragmentation, it removes
it from the list and returns 0, and bo->ttm.ttm could also be NULL at this
point.

Should this check be performed while the BO lock is still held?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=23

Re: [PATCH v2 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] If `xe_dep_scheduler_create()` fails, `xe_bo_defrag_init()` returns early and skips registering the `xe_bo_defrag_fini()` devm action. This leaves the `xe->mem.defrag.worker` active, which can result in a use-after-free when the device struct is freed on probe failure.
--

--- Patch [29]: [PATCH v2 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize ---
commit 5ebeeff5cf94ad2658593973ac47acd35100efa7
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

This patch adds a per-device xe_dep_scheduler dedicated to IOVA defrag
finalize jobs. It initializes this scheduler in xe_bo_defrag_init() and
cleans it up in xe_bo_defrag_fini().

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index f3d76ad9a1050..ec034518e2eae 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -1336,6 +1341,16 @@ void xe_bo_defrag_init_early(struct xe_device *xe)
>   */
>  int xe_bo_defrag_init(struct xe_device *xe)
>  {
> +#define XE_BO_MAX_IOVA_DEFRAG_JOBS	16	/* Picking a reasonable value */
> +	struct xe_dep_scheduler *iova_sched;
> +
> +	iova_sched = xe_dep_scheduler_create(xe, NULL, "xe_iova_defrag",
> +					     XE_BO_MAX_IOVA_DEFRAG_JOBS);
> +	if (IS_ERR(iova_sched))
> +		return PTR_ERR(iova_sched);

[Severity: High]
If xe_dep_scheduler_create() fails, returning early here skips the
registration of the xe_bo_defrag_fini() devm action below.

Does this leave the xe->mem.defrag.worker active on this probe error path?
If the worker was already queued during earlier device initialization, it
appears this could result in a use-after-free when the timer fires and the
worker accesses the freed xe_device structure.

> +	xe->mem.defrag.iova_sched = iova_sched;
> +#undef XE_BO_MAX_IOVA_DEFRAG_JOBS
> +
>  	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=29

Re: [PATCH v2 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Preallocated pages can be configured with Write-Combined (WC) caching but assigned to a Write-Back (WB) TTM buffer object if the platform rejects USWC, causing PAT aliasing and TTM pool corruption.
- [High] Unbounded memory preallocation allows unprivileged users to cause massive memory pressure and kernel stalls before BO size limits are checked.
--

--- Patch [33]: [PATCH v2 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock ---
commit 4ff1d390d0b654be970d2510b2e5cd29bc266121
Author: Matthew Brost <matthew.brost@intel.com>

drm/amdgpu: Preallocate system BO pages outside the reservation lock

This commit mirrors Xe's out-of-lock preallocation for pure system BOs in
amdgpu. When a gem_create request targets only AMDGPU_GEM_DOMAIN_GTT, it
fills the full page backing up front before taking the reservation lock,
to prevent stalls in reclaim and compaction while holding the lock.

> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> index 6a0699746fbcd..bef8b5bc8b892 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c

[ ... ]

> @@ -443,10 +448,29 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
>  		flags |= AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
>  	}
>  
> +	/*
> +	 * For system-only (pure GTT) BOs, preallocate the whole page backing
> +	 * up front, outside the reservation lock. Populate under the lock then
> +	 * just installs these pages instead of reclaiming/compacting in the
> +	 * critical section. Best-effort: a short fill falls back to the normal
> +	 * in-lock allocation for the missing pages.
> +	 */
> +	if (args->in.domains == AMDGPU_GEM_DOMAIN_GTT) {
> +		int32_t xcp_id = adev->gmc.mem_partitions ? fpriv->xcp_id : 0;
> +		int32_t pool_id = amdgpu_ttm_tt_pool_id(adev, xcp_id);
> +		enum ttm_caching caching =
> +			(flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC) ?
> +			ttm_write_combined : ttm_cached;

[Severity: Critical]
Is it possible for the preallocated pages to be configured with write-combined
caching here, but later assigned to a write-back TTM buffer object?

If an unprivileged user requests AMDGPU_GEM_CREATE_CPU_GTT_USWC on a platform
where amdgpu_bo_support_uswc() returns false, the code preallocates the
pages as ttm_write_combined.

Later, amdgpu_bo_create() in drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
strips the unsupported flag:

amdgpu_bo_create() {
    ...
    if (!amdgpu_bo_support_uswc(bo->flags))
        bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
    ...
}

This creates a ttm_tt initialized as ttm_cached (write-back). When the
object is eventually destroyed, ttm_pool_free() will use the write-back
caching state, bypassing the required PAT restoration.

Does this leak the write-combined physical pages back into the kernel
allocator and cause PAT aliasing or machine check exceptions?

> +
> +		prealloc_pool = amdgpu_ttm_pool(adev, pool_id);
> +		ttm_pool_prealloc_fill_full(prealloc_pool, caching, &prealloc,
> +					    PFN_UP(size), false);

[Severity: High]
Could this allow unprivileged users to trigger massive memory pressure before
buffer object size limits are checked?

The user-provided size is passed directly to ttm_pool_prealloc_fill_full()
here, but the size validation against domain limits does not happen until
later in the call chain:

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:amdgpu_bo_create() {
    ...
    if (!amdgpu_bo_validate_size(adev, size, bp->domain))
        return -ENOMEM;
    ...
}

If an unprivileged user makes an ioctl request with an excessively large size
(e.g., 100 GB), ttm_pool_prealloc_fill_full() will attempt to satisfy the
request, triggering intensive reclaim and compaction before the size is ever
validated.

Would it be safer to validate the size against domain limits before
attempting the preallocation?

> +	}
> +
>  	if (flags & AMDGPU_GEM_CREATE_VM_ALWAYS_VALID) {
>  		r = amdgpu_bo_reserve(vm->root.bo, false);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=33

Re: [PATCH v2 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Overloading `dst_packed_pages` as a boolean flag causes GPU IOMMU faults when defragmentation yields 0 changed pages.

Pre-existing issues:
- [High] Mismatch between XE_PTE_SHIFT and PAGE_SHIFT causes out-of-bounds array indexing and assertion failures on architectures with 64K pages.
--

--- Patch [31]: [PATCH v2 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies ---
commit b1d39bbd2548d9ad82a4ac12f4936dc748d93736
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

This commit adds a packed-destination mode to __xe_migrate_copy for defrag
moves with IOVA enabled. It allows reading from the old tt at natural offsets
and writing to the new tt at packed contiguous offsets. A new
xe_migrate_copy_defrag_iova entry point is exposed for this functionality.

> diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
> index 0a3a1c5ca1938..35fd0253fef30 100644
> --- a/drivers/gpu/drm/xe/xe_migrate.c
> +++ b/drivers/gpu/drm/xe/xe_migrate.c
[ ... ]
> @@ -883,9 +949,71 @@ static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
>  
>  		if (bb)
>  			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
> -				  dst_L0_ofs + (u64)i * PAGE_SIZE,
> +				  dst_L0_ofs +
> +				  (u64)(pack_dst ? pack : i) * PAGE_SIZE,
>  				  run * PAGE_SIZE, PAGE_SIZE);

[Severity: High]
This isn't a bug introduced by this patch, but does passing PAGE_SIZE as the
pitch to emit_copy overflow the 16-bit pitch limit on systems with 64KB CPU
pages? The xe_gt_assert condition in emit_xy_fast_copy expects the pitch to be
<= U16_MAX, which could cause a crash on architectures like ARM64 when
defragmenting BOs.

>  		nblits++;
> +		pack += run;
> +		i += run;
> +	}
[ ... ]
> @@ -1005,6 +1134,23 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
>  	 */
>  	struct ttm_tt *dst_tt = dst_bo->ttm.ttm;
>  	bool skip_borrowed = flags.defrag_copy && src_tt && dst_tt;
> +	/*
> +	 * Defrag-IOVA copy: the destination tt only maps the changed
> +	 * (non-borrowed) pages, packed contiguously into the first
> +	 * @dst_packed_pages pages of its IOVA. The source is read from the old
> +	 * tt at natural offsets and each changed run is blitted into the packed
> +	 * destination prefix.
> +	 */
> +	bool pack_dst = dst_packed_pages != 0;

[Severity: High]
Can evaluating pack_dst this way lead to GPU IOMMU faults when a
defragmentation move yields 0 changed pages?

If a BO defragmentation successfully borrows all pages, dst_packed_pages will
be 0. This makes pack_dst false, which disables the packed CCS logic. During
the CCS pass, the driver might then fall back to a full BO CCS copy via
xe_migrate_ccs_copy(), assuming the destination is fully mapped. However,
because the destination IOVA mapping only covers the 0 changed pages, the GPU
would write CCS data to unmapped IOVA addresses.

> +	/*
> +	 * Packed defrag CCS pass: migrate the flat-CCS aux state of the changed
> +	 * pages only, from the source at its natural offset to the destination
> +	 * packed prefix. Borrowed pages share the physical page (and thus its
> +	 * aux) on both sides, so they are skipped exactly like the data pass.
> +	 */
> +	bool ccs_pack = flags.defrag_ccs && pack_dst && src_tt && dst_tt;
[ ... ]
> @@ -1054,29 +1202,86 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
>  		u32 pte_flags;
>  		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> XE_PTE_SHIFT;

[Severity: High]
This isn't a bug introduced by this patch, but could the start_page
calculation cause out-of-bounds array indexing on architectures with
64KB pages? 

The index start_page is computed using XE_PTE_SHIFT (which is 4KB), but
it is later used to index ttm_tt->pages which is sized based on
PAGE_SHIFT. On a 64KB page system, the calculated index would be 16x
larger than the array bounds, potentially leading to memory corruption
during defragmentation.

>  		u32 ndata_blits = 1;
> +		u32 nchg = 0;
> +		u64 dst_size;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=31

Re: [PATCH v2 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [Critical] Error paths tear down the destination IOVA mapping without waiting for the submitted asynchronous GPU copy job, leading to IOMMU faults and Write-After-Free.
- [High] xe_bo_defrag_use_iova() checks only the new_tt for IOVA support, ignoring whether old_tt actually uses IOVA, leading to invalid IOVA transfer and IOMMU corruption.
- [High] TLB/L2 cache invalidation for the IOVA defrag is scheduled too early, leading to stale entries in the GPU TLB after the IOMMU update completes.
- [High] Allocating memory with GFP_KERNEL while holding the BO's dma_resv lock creates an AB-BA deadlock risk with the DRM memory shrinker.
- [High] Incomplete SWIOTLB bounce buffer handling causes the GPU to read uninitialized memory after defragmentation.
- [Medium] The return value of dma_iova_sync() is ignored in the defrag finalize job, potentially leaving the IOTLB/cache unflushed.
--

--- Patch [32]: [PATCH v2 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job ---
commit 23b117a272c85c9d94147e6761afd74a17131d1f
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Finalize defrag-IOVA moves with post-copy job

This commit wires up end-to-end defrag for IOVA-mapped BOs. A defrag move
is eligible whenever the destination TT is IOVA mapped. The packed-copy
step moves only changed pages, and a post-copy finalize job tears down the
temporary packed mapping and relinks the copied pages.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 16d9e135cba2e..0eee53f4d653f 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -855,7 +854,6 @@ struct xe_tt_iova_run_collect {
>  };
>  
>  /* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
> -__maybe_unused
>  static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
>  			   size_t elem)
>  {
> @@ -866,13 +864,12 @@ static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
>  		return 0;
>  
>  	new_cap = *cap ? *cap * 2 : 64;
>  	if (new_cap < need)
>  		new_cap = need;
>  
>  	p = krealloc_array(*arr, new_cap, elem, GFP_KERNEL);

[Severity: High]
Is it safe to use GFP_KERNEL here? This is called from xe_bo_move() while
holding the buffer object's dma_resv lock.

If the memory allocator triggers direct reclaim, it can invoke the DRM
memory shrinker, which will attempt to acquire dma_resv locks to evict
buffer objects, leading to an AB-BA deadlock.

>  	if (!p)
>  		return -ENOMEM;

[ ... ]

> @@ -1047,6 +1041,129 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
> +static struct dma_fence *xe_iova_defrag_job_run(struct xe_dep_job *dep_job)
> +{

[ ... ]

> +	/*
> +	 * Relink the freshly copied new pages from the snapshot captured at job
> +	 * creation. The destination tt's page array may have been mutated (or
> +	 * torn down) by TTM since then, so it is never dereferenced here.
> +	 * Borrowed pages are shared with the old tt and stay correctly linked
> +	 * for free, so the snapshot only covers the changed runs.
> +	 */
> +	for (l = 0; l < job->copy.nr_links; l++) {
> +		int ret = dma_iova_link(dev, &xe_tt->iova,
> +					job->copy.links[l].phys,
> +					job->copy.links[l].offset,
> +					job->copy.links[l].len,
> +					DMA_BIDIRECTIONAL,
> +					DMA_ATTR_SKIP_CPU_SYNC);

[Severity: High]
Does skipping CPU sync here break configurations using SWIOTLB?

When using DMA_ATTR_SKIP_CPU_SYNC, the DMA API allocates a new bounce
buffer for the physical page but bypasses copying the physical page
contents into it.

This would cause the new IOVA mapping to point to an uninitialized bounce
buffer, leading to the GPU reading uninitialized memory.

> +		/*
> +		 * Point of no failure: the data is already copied and the
> +		 * reservation ownership has been transferred. A relink failure
> +		 * leaves a hole in the mapping; there is no safe recovery, so
> +		 * just warn.
> +		 */
> +		drm_WARN_ON(&xe->drm, ret);
> +	}
> +
> +	/* Sync the relinked changed runs into the device's view. */
> +	for (r = 0; r < job->copy.nr_runs; r++) {
> +		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
> +		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
> +
> +		dma_iova_sync(dev, &xe_tt->iova, off, len);

[Severity: Medium]
Should the return value of dma_iova_sync() be checked here?

While this is a point of no return and the job cannot be aborted gracefully,
silently ignoring the error hides IOMMU cache sync failures.

A failure could result in stale IOMMU caches and GPU data corruption, so
perhaps at least a drm_WARN_ON() is warranted, similar to the
dma_iova_link() call above.

> +	}

[ ... ]

> @@ -1063,7 +1180,6 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
> -__maybe_unused
>  static void xe_tt_iova_transfer(struct xe_device *xe,
>  				struct xe_ttm_tt *new_tt,
>  				struct xe_ttm_tt *old_tt)
>  {
>  	new_tt->iova = old_tt->iova;

[Severity: High]
What happens if the original old_tt failed to allocate an IOVA mapping and
fell back to an SG mapping?

If old_tt->use_iova is false, old_tt->iova would be uninitialized or empty.
Copying it here would overwrite the valid new_tt->iova state, which could
lead to IOMMU corruption when the finalize job attempts to link pages.

>  	new_tt->iova_linked = true;

[ ... ]

> +static struct dma_fence *
> +xe_bo_defrag_iova_copy(struct xe_device *xe, struct xe_migrate *migrate,
> +		       struct xe_bo *bo, struct ttm_resource *old_mem,
> +		       struct ttm_resource *new_mem, struct ttm_tt *old_tt,
> +		       bool handle_system_ccs)
> +{

[ ... ]

> +	job = kzalloc(sizeof(*job), GFP_KERNEL);

[Severity: High]
Could this GFP_KERNEL allocation cause a deadlock?

Just like the allocation in xe_tt_iova_grow(), this executes under the
buffer object's dma_resv lock during xe_bo_move(). If memory pressure
triggers the shrinker, it might try to acquire the same dma_resv lock.

> +	if (!job) {
> +		err = -ENOMEM;
> +		goto err_fence;
> +	}

[ ... ]

> +err_job_init:
> +	drm_sched_job_cleanup(&job->dep.drm);
> +err_job:
> +	kfree(job);
> +err_fence:
> +	dma_fence_put(copy_fence);
> +err_unmap:
> +	if (packed)
> +		dma_iova_unlink(xe->drm.dev, &new_xe_tt->iova, 0,
> +				(size_t)copy.nr_changed << PAGE_SHIFT,
> +				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);

[Severity: Critical]
Is it safe to unlink the IOVA mapping here without waiting for copy_fence
to signal?

By destroying the destination IOVA page tables immediately, the asynchronous
GPU migration job might still be actively writing to those addresses.

This could result in IOMMU faults or write-after-free corruption if TTM
aborts the move and frees the physical pages while the GPU blit is still
running.

> +err_copy:
> +	xe_tt_iova_copy_fini(&copy);
> +	return ERR_PTR(err);
> +}

[ ... ]

> +/*
> + * xe_bo_defrag_use_iova - Whether a defrag move should use the packed IOVA path.
> + */
> +static bool xe_bo_defrag_use_iova(struct ttm_buffer_object *ttm_bo)
> +{
> +	struct xe_ttm_tt *xe_tt;
> +
> +	if (!ttm_bo->defrag_old_tt || !ttm_bo->ttm)
> +		return false;
> +
> +	xe_tt = container_of(ttm_bo->ttm, struct xe_ttm_tt, ttm);
> +	return xe_tt->use_iova;

[Severity: High]
Does this check adequately ensure the old_tt is actually using IOVA?

The function verifies new_tt->use_iova, but omits checking
old_tt->use_iova.

If the initial old_tt fell back to an SG mapping, this path would still
return true as long as new_tt is IOVA-eligible. This would then misroute
the move to xe_bo_defrag_iova_copy() and trigger the uninitialized IOVA
transfer mentioned earlier.

> +}

[ ... ]

> @@ -1388,6 +1647,8 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
>  			 */
>  			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
>  				continue;
> +			else if (defrag_iova)
> +				drm_gpuvm_bo_evict(vm_bo, true);

[Severity: High]
Can this sequence cause the GPU to access freed physical pages?

The function xe_vm_flush_vm_bo_tlb_async() flushes the TLB asynchronously
based on the current fences in the reservation. However, this is executed
before the migration and finalize jobs are added to the buffer object's
reservation.

Because the eviction marking is skipped for defrag_iova, the userspace GPU
context is not stopped. It can access the buffer object and repopulate the
GPU TLB with the old IOMMU translations after this premature flush
completes, but before the finalize job actually updates the IOMMU.

>  		}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost@intel.com?part=32

Re: [PATCH v2 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

[Christian König]

On 7/10/26 23:54, Matthew Brost wrote:
> The overarching goal of this series is to keep user-facing IOCTLs snappy
> by holding the BO dma-resv lock for the shortest possible time, and to
> push the expensive, best-effort work - page (re)allocation, DMA mapping
> and placement fixups - into the background or out of the locked critical
> section entirely.
> 
> Two related mechanisms fall out of that goal:
> 
>  - Out-of-lock backing. Page allocation and DMA mapping are the dominant
>    cost of gem_create and of BO moves, and doing them under the dma-resv
>    lock serializes otherwise-independent clients.

Saying that doesn't make sense.

The DMA-resv lock is either per BO or per VM and so the clients are not even remotely independent of that lock.

>    This series preallocates
>    the backing pages (and, where applicable, builds the DMA/IOVA mapping)
>    outside the lock, then transfers the result into the BO under a brief
>    lock hold. The user gets a fast IOCTL return; the heavy lifting happens
>    without contending the reservation.

Ok, that sounds like a really bad idea.

Why in the world would you do this? The dma_resv lock is exactly what is used to prevent something like that.

> 
>  - Page defragmentation. A BO allocated under memory pressure keeps its
>    scattered, sub-optimally-ordered backing for its entire lifetime,
>    costing TLB efficiency forever. TTM grows the plumbing to track
>    order-failure and to re-back a populated BO in place at the beneficial
>    order, and Xe wires up a background delayed worker that promotes such
>    BOs on the GPU once memory is available again - again, off the hot
>    path and without stalling the submitting thread.

That could be quite beneficial, but so far falling back to low order allocation was only seen as last resort to avoid OOM.

Regards,
Christian.

> 
> Since v1 [1] the series has grown considerably. The bulk of the new
> material is a direct result of profiling: once the defragmenter was in
> place, the remaining dma-resv hold times and the per-BO
> allocation/mapping costs showed up clearly, which motivated the
> out-of-lock preallocation, the IOVA-based mapping path, and the amdgpu
> counterpart. Rather than land the defragmenter alone, v2 folds in these
> optimizations since they share the same infrastructure and the same
> "hold dma-resv briefly, fix up in the background" architecture.
> 
> The series is organized in sections rather than described patch by patch:
> 
>  - Patches 1-10 (drm/ttm): core TTM preparation - order-failure tracking,
>    the defragmentation move, reclaim backoff, and out-of-lock page
>    preallocation plumbing.
>  - Patches 11-14: other dependent drm/gpusvm and drm/xe patches this
>    series builds on (DMA-mapping accounting, per-order DMA stats, async
>    L2 flush, and a VM-teardown ordering fix).
>  - Patches 15-23 (drm/xe): the page defragmenter itself - BO tracking,
>    the on-GPU defrag copy, xe_bo_move() handling, and the background
>    worker with its stats and configuration.
>  - Patches 24-25 (drm/xe): out-of-lock system BO backing preallocation
>    in gem_create, moving page allocation out of the dma-resv lock.
>  - Patches 26-32 (drm/xe): IOVA-based DMA mapping optimizations, building
>    and finalizing the mapping outside the lock.
>  - Patch 33 (drm/amdgpu): the equivalent out-of-lock system BO
>    preallocation for amdgpu, exercising the shared TTM plumbing.
> 
> Testing
> =======
> 
>  - 3D benchmarks on Ubuntu and on Android, with memory intentionally
>    fragmented by a separate program at launch (plus beneficial-order
>    error injection). BOs are initially backed at a sub-optimal order and
>    scores start lower; the background defrag worker then promotes the
>    backing to the beneficial order and scores climb back in line with the
>    unfragmented baseline.
>  - IGT:
>    https://patchwork.freedesktop.org/patch/739052/?series=170046&rev=2
> 
> Matt
> 
> [1] https://patchwork.freedesktop.org/series/169053/
> 
> Cc: amd-gfx@lists.freedesktop.org
> Cc: Alex Deucher <alexander.deucher@amd.com>
> Cc: Carlos Santa <carlos.santa@intel.com>
> Cc: Ryan Neph <ryanneph@google.com>
> Cc: Christian Koenig <christian.koenig@amd.com>
> Cc: Huang Rui <ray.huang@amd.com>
> Cc: Matthew Auld <matthew.auld@intel.com>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Maxime Ripard <mripard@kernel.org>
> Cc: Thomas Zimmermann <tzimmermann@suse.de>
> Cc: David Airlie <airlied@gmail.com>
> Cc: Simona Vetter <simona@ffwll.ch>
> Cc: dri-devel@lists.freedesktop.org
> Cc: linux-kernel@vger.kernel.org
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> 
> 
> 
> Matthew Brost (31):
>   drm/ttm/pool: Allow backing off reclaim at the beneficial order
>   drm/ttm/pool: Add ttm_pool_page_order_nodma() helper
>   drm/ttm: Record sub-optimal page order allocations in ttm_tt
>   drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()
>   drm/ttm: Support defragmentation moves
>   drm/ttm: Add fault injection for beneficial-order allocation failures
>   drm/ttm: Harvest beneficial-order pages on defragmentation moves
>   drm/ttm: Bound page (re)allocation per defragmentation move
>   drm/ttm: Preallocate beneficial-order defrag pages outside the lock
>   drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()
>   drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()
>   drm/xe: Destroy page tables after unlinking all VMAs on VM close
>   drm/xe: Track BOs backed at a sub-optimal page order
>   drm/xe: Back off beneficial-order reclaim under defrag pressure
>   drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies
>   drm/xe: Handle defrag moves in xe_bo_move()
>   drm/xe: Skip self-copies for borrowed pages on defrag moves
>   drm/xe: Add a page defragmentation worker
>   drm/xe: Add defrag GT stats
>   drm/xe: Add Kconfig.profile options for BO defrag configuration
>   drm/xe: Defrag using out-of-lock page preallocation
>   drm/xe: Add defrag profiling tracepoints
>   drm/xe: Preallocate system BO backing outside the dma-resv lock
>   drm/xe: Add tracepoint for xe_gem_create_ioctl
>   drm/xe: Add IOVA-based xe_res_cursor variant
>   drm/xe: Use IOVA-based DMA mapping for eligible tt BOs
>   drm/xe: Add per-device dependency scheduler for IOVA defrag finalize
>   drm/xe: Add packed copy-step IOVA mapping for defrag
>   drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies
>   drm/xe: Finalize defrag-IOVA moves with post-copy job
>   drm/amdgpu: Preallocate system BO pages outside the reservation lock
> 
> Thomas Hellström (2):
>   drm/gpusvm: Add a DMA-mapping accounting callback
>   drm/xe: Add debugfs stats for DMA-mapped pages per order
> 
>  .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    4 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   45 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |    5 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |    3 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    4 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   39 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    3 +
>  drivers/gpu/drm/drm_gpusvm.c                  |   17 +-
>  drivers/gpu/drm/ttm/ttm_bo.c                  |  101 +-
>  drivers/gpu/drm/ttm/ttm_bo_util.c             |   21 +-
>  drivers/gpu/drm/ttm/ttm_pool.c                |  931 +++++++++-
>  drivers/gpu/drm/ttm/ttm_tt.c                  |   67 +
>  drivers/gpu/drm/xe/Kconfig.profile            |   40 +
>  drivers/gpu/drm/xe/tests/xe_bo.c              |    8 +-
>  drivers/gpu/drm/xe/tests/xe_dma_buf.c         |    2 +-
>  drivers/gpu/drm/xe/tests/xe_migrate.c         |   12 +-
>  drivers/gpu/drm/xe/xe_bo.c                    | 1632 ++++++++++++++++-
>  drivers/gpu/drm/xe/xe_bo.h                    |   15 +-
>  drivers/gpu/drm/xe/xe_bo_types.h              |    6 +
>  drivers/gpu/drm/xe/xe_debugfs.c               |   26 +
>  drivers/gpu/drm/xe/xe_device.c                |   35 +
>  drivers/gpu/drm/xe/xe_device_types.h          |   57 +
>  drivers/gpu/drm/xe/xe_dma_buf.c               |    2 +-
>  drivers/gpu/drm/xe/xe_ggtt.c                  |    2 +-
>  drivers/gpu/drm/xe/xe_gt_stats.c              |    5 +
>  drivers/gpu/drm/xe/xe_gt_stats_types.h        |   17 +
>  drivers/gpu/drm/xe/xe_migrate.c               |  531 +++++-
>  drivers/gpu/drm/xe/xe_migrate.h               |   17 +
>  drivers/gpu/drm/xe/xe_pt.c                    |    2 +-
>  drivers/gpu/drm/xe/xe_res_cursor.h            |   56 +-
>  drivers/gpu/drm/xe/xe_svm.c                   |   36 +-
>  drivers/gpu/drm/xe/xe_svm.h                   |    3 +-
>  drivers/gpu/drm/xe/xe_trace_bo.h              |   90 +
>  drivers/gpu/drm/xe/xe_userptr.c               |   55 +
>  drivers/gpu/drm/xe/xe_userptr.h               |    1 +
>  drivers/gpu/drm/xe/xe_vm.c                    |  149 +-
>  drivers/gpu/drm/xe/xe_vm.h                    |    5 +
>  include/drm/drm_gpusvm.h                      |   19 +
>  include/drm/ttm/ttm_bo.h                      |   64 +
>  include/drm/ttm/ttm_pool.h                    |   46 +
>  include/drm/ttm/ttm_tt.h                      |   43 +-
>  42 files changed, 4029 insertions(+), 189 deletions(-)
> 

Re: [PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

[Christian König]

On 7/10/26 23:54, Matthew Brost wrote:
> Allocating at the pool's beneficial order normally uses
> __GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When
> beneficial-order allocations have recently been failing, this retry just
> burns CPU cycles in reclaim/compaction before falling back to a smaller
> order anyway.
> 
> Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and
> plumb it through to ttm_pool_alloc_page(). When set, the beneficial
> order is treated like the orders we already consider not worth stalling
> for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction
> entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion
> is skipped. Together this makes the allocator back off quickly to a
> smaller order instead of stalling.

That makes no sense at all.

You first give a beneficial order from the driver and then later overwrite it.

What could be done is to make the beneficial order part of the context in the first place, but I don't get what the use case for this should be.

It looks like you propose that TTM should accept smaller chunks of memory initially and then comes back later on and re-allocates things in huge pages.

If that is really the case then I would clearly reject the whole idea. De-fragmentation of smaller pages into larger ones is the job of the core memory management and *not* TTM.

Regards,
Christian.

> 
> This is a no-op until a caller opts in.
> 
> Cc: Carlos Santa <carlos.santa@intel.com>
> Cc: Ryan Neph <ryanneph@google.com>
> Cc: Christian Koenig <christian.koenig@amd.com>
> Cc: Huang Rui <ray.huang@amd.com>
> Cc: Matthew Auld <matthew.auld@intel.com>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Maxime Ripard <mripard@kernel.org>
> Cc: Thomas Zimmermann <tzimmermann@suse.de>
> Cc: David Airlie <airlied@gmail.com>
> Cc: Simona Vetter <simona@ffwll.ch>
> Cc: dri-devel@lists.freedesktop.org
> Cc: linux-kernel@vger.kernel.org
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Assisted-by: GitHub_Copilot:claude-opus-4.8
> Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> ---
>  drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
>  include/drm/ttm/ttm_bo.h       |  7 +++++++
>  2 files changed, 15 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index 3d5f2ae0a456..dbe977412a81 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> @@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
>  
>  /* Allocate pages of size 1 << order with the given gfp_flags */
>  static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
> -					unsigned int order)
> +					unsigned int order,
> +					bool beneficial_reclaim_backoff)
>  {
>  	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
>  	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
> @@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
>  	 * Do not add latency to the allocation path for allocations orders
>  	 * device tolds us do not bring them additional performance gains.
>  	 */
> -	if (order && beneficial_order && order != beneficial_order)
> +	if (order && (beneficial_reclaim_backoff ||
> +		      (beneficial_order && order != beneficial_order)))
>  		gfp_flags &= ~__GFP_RECLAIM;
>  
> -	if (beneficial_order && order == beneficial_order) {
> +	if (!beneficial_reclaim_backoff &&
> +	    beneficial_order && order == beneficial_order) {
>  		gfp_flags &= ~__GFP_NORETRY;
>  		gfp_flags |= __GFP_RETRY_MAYFAIL;
>  	}
> @@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>  		if (!p) {
>  			page_caching = ttm_cached;
>  			allow_pools = false;
> -			p = ttm_pool_alloc_page(pool, gfp_flags, order);
> +			p = ttm_pool_alloc_page(pool, gfp_flags, order,
> +						ctx->beneficial_reclaim_backoff);
>  		}
>  		/* If that fails, lower the order if possible and retry. */
>  		if (!p) {
> diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
> index 8310bc3d55f9..fd814b5cf535 100644
> --- a/include/drm/ttm/ttm_bo.h
> +++ b/include/drm/ttm/ttm_bo.h
> @@ -187,6 +187,13 @@ struct ttm_operation_ctx {
>  	 * when multiple BOs share the same reservation object @resv.
>  	 */
>  	bool allow_res_evict;
> +	/**
> +	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
> +	 * pool's beneficial order. Useful when beneficial-order allocation has
> +	 * recently been failing, to avoid burning CPU cycles in
> +	 * reclaim/compaction before falling back to a smaller order anyway.
> +	 */
> +	bool beneficial_reclaim_backoff;
>  	/**
>  	 * @resv: Reservation object to be used together with
>  	 * @allow_res_evict.

Re: [PATCH v2 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

[Christian König]

On 7/10/26 23:54, Matthew Brost wrote:
> TTM records the allocation order of a multi-order page in page->private at
> allocation time. For pools that do not use dma_alloc the order is stored
> directly, while dma_alloc pools store a struct ttm_pool_dma pointer there
> instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page
> order to walk a populated ttm_tt one chunk at a time, but cannot rely on
> folio_order(): TTM allocates high-order pages with alloc_pages_node()
> without __GFP_COMP, so they are not compound and folio_order() always
> returns 0.

Well what is the justification of XE doing the DMA mapping themselves?

Regards,
Christian.

> 
> Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order
> recorded in page->private. It is only valid for pages from a pool that does
> not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm
> device init time, so callers know from their device configuration that this
> helper applies. Use it from the existing internal ttm_pool_page_order() to
> keep a single source of truth.
> 
> Cc: Carlos Santa <carlos.santa@intel.com>
> Cc: Ryan Neph <ryanneph@google.com>
> Cc: Christian Koenig <christian.koenig@amd.com>
> Cc: Huang Rui <ray.huang@amd.com>
> Cc: Matthew Auld <matthew.auld@intel.com>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Maxime Ripard <mripard@kernel.org>
> Cc: Thomas Zimmermann <tzimmermann@suse.de>
> Cc: David Airlie <airlied@gmail.com>
> Cc: Simona Vetter <simona@ffwll.ch>
> Cc: dri-devel@lists.freedesktop.org
> Cc: linux-kernel@vger.kernel.org
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Assisted-by: GitHub_Copilot:claude-opus-4.8
> Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> ---
>  drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
>  include/drm/ttm/ttm_pool.h     |  2 ++
>  2 files changed, 26 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index dbe977412a81..d34592d4dbc7 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c
> @@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
>  	return num_pages;
>  }
>  
> +/**
> + * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
> + * @p: The page to query.
> + *
> + * Return the allocation order that the TTM pool allocator recorded in
> + * @p->private at allocation time. This only works for pages that were
> + * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
> + * alloc_pages_node()), where TTM stores the order directly in page->private.
> + *
> + * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
> + * pointer and this helper must not be used. Whether a TTM device uses
> + * dma_alloc is fixed at ttm device init time, so callers are expected to know
> + * from their TTM device configuration that their pages are not dma_alloc
> + * backed before using this helper.
> + *
> + * Return: The allocation order of the page.
> + */
> +unsigned int ttm_pool_page_order_nodma(struct page *p)
> +{
> +	return p->private;
> +}
> +EXPORT_SYMBOL(ttm_pool_page_order_nodma);
> +
>  /* Return the allocation order based for a page */
>  static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
>  {
> @@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
>  		return dma->vaddr & ~PAGE_MASK;
>  	}
>  
> -	return p->private;
> +	return ttm_pool_page_order_nodma(p);
>  }
>  
>  /*
> diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
> index 26ee592e1994..753203980e2c 100644
> --- a/include/drm/ttm/ttm_pool.h
> +++ b/include/drm/ttm/ttm_pool.h
> @@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
>  int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>  			       const struct ttm_operation_ctx *ctx);
>  
> +unsigned int ttm_pool_page_order_nodma(struct page *p);
> +
>  int ttm_pool_mgr_init(unsigned long num_pages);
>  void ttm_pool_mgr_fini(void);
>