[PATCH v3 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

[PATCH v3 00/33] drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

[Matthew Brost]

The overarching goal of this series is to keep user-facing IOCTLs snappy
by holding the BO dma-resv lock for the shortest possible time, and to
push the expensive, best-effort work - page (re)allocation, DMA mapping
and placement fixups - into the background or out of the locked critical
section entirely.

Two related mechanisms fall out of that goal:

 - Out-of-lock backing. Page allocation and DMA mapping are the dominant
   cost of gem_create and of BO moves, and doing them under the dma-resv
   lock serializes otherwise-independent clients. This series preallocates
   the backing pages (and, where applicable, builds the DMA/IOVA mapping)
   outside the lock, then transfers the result into the BO under a brief
   lock hold. The user gets a fast IOCTL return; the heavy lifting happens
   without contending the reservation.

 - Page defragmentation. A BO allocated under memory pressure keeps its
   scattered, sub-optimally-ordered backing for its entire lifetime,
   costing TLB efficiency forever. TTM grows the plumbing to track
   order-failure and to re-back a populated BO in place at the beneficial
   order, and Xe wires up a background delayed worker that promotes such
   BOs on the GPU once memory is available again - again, off the hot
   path and without stalling the submitting thread.

Since v1 [1] the series has grown considerably. The bulk of the new
material is a direct result of profiling: once the defragmenter was in
place, the remaining dma-resv hold times and the per-BO
allocation/mapping costs showed up clearly, which motivated the
out-of-lock preallocation, the IOVA-based mapping path, and the amdgpu
counterpart. Rather than land the defragmenter alone, v2 folds in these
optimizations since they share the same infrastructure and the same
"hold dma-resv briefly, fix up in the background" architecture.

The series is organized in sections rather than described patch by patch:

 - Patches 1-10 (drm/ttm): core TTM preparation - order-failure tracking,
   the defragmentation move, reclaim backoff, and out-of-lock page
   preallocation plumbing.
 - Patches 11-14: other dependent drm/gpusvm and drm/xe patches this
   series builds on (DMA-mapping accounting, per-order DMA stats, async
   L2 flush, and a VM-teardown ordering fix).
 - Patches 15-23 (drm/xe): the page defragmenter itself - BO tracking,
   the on-GPU defrag copy, xe_bo_move() handling, and the background
   worker with its stats and configuration.
 - Patches 24-25 (drm/xe): out-of-lock system BO backing preallocation
   in gem_create, moving page allocation out of the dma-resv lock.
 - Patches 26-32 (drm/xe): IOVA-based DMA mapping optimizations, building
   and finalizing the mapping outside the lock.
 - Patch 33 (drm/amdgpu): the equivalent out-of-lock system BO
   preallocation for amdgpu, exercising the shared TTM plumbing.

Testing
=======

 - 3D benchmarks on Ubuntu and on Android, with memory intentionally
   fragmented by a separate program at launch (plus beneficial-order
   error injection). BOs are initially backed at a sub-optimal order and
   scores start lower; the background defrag worker then promotes the
   backing to the beneficial order and scores climb back in line with the
   unfragmented baseline.
 - IGT:
   https://patchwork.freedesktop.org/patch/739052/?series=170046&rev=2

Matt

v3:
 - Fix valid raised raised by Sashiko on v2 [2]

[1] https://patchwork.freedesktop.org/series/169053/
[2] https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost%40intel.com

Cc: amd-gfx@lists.freedesktop.org
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Matthew Brost (31):
  drm/ttm/pool: Allow backing off reclaim at the beneficial order
  drm/ttm/pool: Add ttm_pool_page_order_nodma() helper
  drm/ttm: Record sub-optimal page order allocations in ttm_tt
  drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()
  drm/ttm: Support defragmentation moves
  drm/ttm: Add fault injection for beneficial-order allocation failures
  drm/ttm: Harvest beneficial-order pages on defragmentation moves
  drm/ttm: Bound page (re)allocation per defragmentation move
  drm/ttm: Preallocate beneficial-order defrag pages outside the lock
  drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()
  drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()
  drm/xe: Destroy page tables after unlinking all VMAs on VM close
  drm/xe: Track BOs backed at a sub-optimal page order
  drm/xe: Back off beneficial-order reclaim under defrag pressure
  drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies
  drm/xe: Handle defrag moves in xe_bo_move()
  drm/xe: Skip self-copies for borrowed pages on defrag moves
  drm/xe: Add a page defragmentation worker
  drm/xe: Add defrag GT stats
  drm/xe: Add Kconfig.profile options for BO defrag configuration
  drm/xe: Defrag using out-of-lock page preallocation
  drm/xe: Add defrag profiling tracepoints
  drm/xe: Preallocate system BO backing outside the dma-resv lock
  drm/xe: Add tracepoint for xe_gem_create_ioctl
  drm/xe: Add IOVA-based xe_res_cursor variant
  drm/xe: Use IOVA-based DMA mapping for eligible tt BOs
  drm/xe: Add per-device dependency scheduler for IOVA defrag finalize
  drm/xe: Add packed copy-step IOVA mapping for defrag
  drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies
  drm/xe: Finalize defrag-IOVA moves with post-copy job
  drm/amdgpu: Preallocate system BO pages outside the reservation lock

Thomas Hellström (2):
  drm/gpusvm: Add a DMA-mapping accounting callback
  drm/xe: Add debugfs stats for DMA-mapped pages per order

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   73 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |    5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |    3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   39 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    3 +
 drivers/gpu/drm/drm_gpusvm.c                  |   17 +-
 drivers/gpu/drm/ttm/ttm_bo.c                  |  118 +-
 drivers/gpu/drm/ttm/ttm_bo_util.c             |   21 +-
 drivers/gpu/drm/ttm/ttm_pool.c                |  947 ++++++++-
 drivers/gpu/drm/ttm/ttm_tt.c                  |   67 +
 drivers/gpu/drm/xe/Kconfig.profile            |   40 +
 drivers/gpu/drm/xe/tests/xe_bo.c              |    8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c         |    2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c         |   12 +-
 drivers/gpu/drm/xe/xe_bo.c                    | 1699 ++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h                    |   15 +-
 drivers/gpu/drm/xe/xe_bo_types.h              |    6 +
 drivers/gpu/drm/xe/xe_debugfs.c               |   26 +
 drivers/gpu/drm/xe/xe_device.c                |   41 +
 drivers/gpu/drm/xe/xe_device_types.h          |   57 +
 drivers/gpu/drm/xe/xe_dma_buf.c               |    2 +-
 drivers/gpu/drm/xe/xe_ggtt.c                  |    2 +-
 drivers/gpu/drm/xe/xe_gt_stats.c              |    5 +
 drivers/gpu/drm/xe/xe_gt_stats_types.h        |   17 +
 drivers/gpu/drm/xe/xe_migrate.c               |  531 +++++-
 drivers/gpu/drm/xe/xe_migrate.h               |   17 +
 drivers/gpu/drm/xe/xe_pt.c                    |    2 +-
 drivers/gpu/drm/xe/xe_res_cursor.h            |   56 +-
 drivers/gpu/drm/xe/xe_svm.c                   |   44 +-
 drivers/gpu/drm/xe/xe_svm.h                   |    4 +-
 drivers/gpu/drm/xe/xe_trace_bo.h              |   90 +
 drivers/gpu/drm/xe/xe_userptr.c               |   63 +
 drivers/gpu/drm/xe/xe_userptr.h               |    1 +
 drivers/gpu/drm/xe/xe_vm.c                    |  149 +-
 drivers/gpu/drm/xe/xe_vm.h                    |    5 +
 include/drm/drm_gpusvm.h                      |   19 +
 include/drm/ttm/ttm_bo.h                      |   64 +
 include/drm/ttm/ttm_pool.h                    |   46 +
 include/drm/ttm/ttm_tt.h                      |   43 +-
 42 files changed, 4179 insertions(+), 190 deletions(-)

-- 
2.34.1

[PATCH v3 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

[Matthew Brost]

Allocating at the pool's beneficial order normally uses
__GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When
beneficial-order allocations have recently been failing, this retry just
burns CPU cycles in reclaim/compaction before falling back to a smaller
order anyway.

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and
plumb it through to ttm_pool_alloc_page(). When set, the beneficial
order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction
entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion
is skipped. Together this makes the allocator back off quickly to a
smaller order instead of stalling.

This is a no-op until a caller opts in.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Require a valid beneficial order before backing off reclaim in
   ttm_pool_alloc_page() (Sashiko)
---
 drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
 include/drm/ttm/ttm_bo.h       |  7 +++++++
 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 3d5f2ae0a456..ab242ad4339e 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && beneficial_order &&
+	    (beneficial_reclaim_backoff || order != beneficial_order))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 8310bc3d55f9..fd814b5cf535 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
-- 
2.34.1

[PATCH v3 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

[Matthew Brost]

TTM records the allocation order of a multi-order page in page->private at
allocation time. For pools that do not use dma_alloc the order is stored
directly, while dma_alloc pools store a struct ttm_pool_dma pointer there
instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page
order to walk a populated ttm_tt one chunk at a time, but cannot rely on
folio_order(): TTM allocates high-order pages with alloc_pages_node()
without __GFP_COMP, so they are not compound and folio_order() always
returns 0.

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order
recorded in page->private. It is only valid for pages from a pool that does
not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm
device init time, so callers know from their device configuration that this
helper applies. Use it from the existing internal ttm_pool_page_order() to
keep a single source of truth.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
 include/drm/ttm/ttm_pool.h     |  2 ++
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index ab242ad4339e..5b7d7f5ae53d 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
 	return num_pages;
 }
 
+/**
+ * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
+ * @p: The page to query.
+ *
+ * Return the allocation order that the TTM pool allocator recorded in
+ * @p->private at allocation time. This only works for pages that were
+ * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
+ * alloc_pages_node()), where TTM stores the order directly in page->private.
+ *
+ * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
+ * pointer and this helper must not be used. Whether a TTM device uses
+ * dma_alloc is fixed at ttm device init time, so callers are expected to know
+ * from their TTM device configuration that their pages are not dma_alloc
+ * backed before using this helper.
+ *
+ * Return: The allocation order of the page.
+ */
+unsigned int ttm_pool_page_order_nodma(struct page *p)
+{
+	return p->private;
+}
+EXPORT_SYMBOL(ttm_pool_page_order_nodma);
+
 /* Return the allocation order based for a page */
 static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 {
@@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 		return dma->vaddr & ~PAGE_MASK;
 	}
 
-	return p->private;
+	return ttm_pool_page_order_nodma(p);
 }
 
 /*
diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
index 26ee592e1994..753203980e2c 100644
--- a/include/drm/ttm/ttm_pool.h
+++ b/include/drm/ttm/ttm_pool.h
@@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
 int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			       const struct ttm_operation_ctx *ctx);
 
+unsigned int ttm_pool_page_order_nodma(struct page *p);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
-- 
2.34.1

[PATCH v3 03/33] drm/ttm: Record sub-optimal page order allocations in ttm_tt

[Matthew Brost]

When __ttm_pool_alloc() fails to allocate a chunk at the device's
beneficial order and falls back to a smaller order, the object ends up
backed by a sub-optimal set of pages. Nothing currently records this, so
a driver has no way to know an object would benefit from being re-backed
with beneficial-order pages later.

Add TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED to struct ttm_tt page_flags,
cleared at the start of every fresh allocation (in both ttm_pool_alloc()
and ttm_pool_restore_and_alloc()). Set it in __ttm_pool_alloc() when an
allocation at exactly the beneficial order fails before dropping to a
lower order.

Drivers can use this hint to queue the buffer object for later
defragmentation.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 13 +++++++++++++
 include/drm/ttm/ttm_tt.h       | 24 +++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 5b7d7f5ae53d..e9175c0290d7 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -796,6 +796,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
+	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	enum ttm_caching page_caching;
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
@@ -846,6 +847,16 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
 			if (order) {
+				/*
+				 * Failing to allocate at the device's beneficial
+				 * order means we are about to back this object
+				 * with a sub-optimal (smaller order) set of
+				 * pages. Record it so the driver can later try to
+				 * defragment the object back to beneficial order.
+				 */
+				if (beneficial_order && order == beneficial_order)
+					tt->page_flags |=
+						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 				--order;
 				page_caching = tt->caching;
 				allow_pools = true;
@@ -910,6 +921,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
+	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
 
 	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
@@ -942,6 +954,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (!restore) {
 		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
 
+		tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		ttm_pool_alloc_state_init(tt, &alloc);
 		if (ctx->gfp_retry_mayfail)
 			gfp |= __GFP_RETRY_MAYFAIL;
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 406437ad674b..ce7533677d77 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -90,6 +90,14 @@ struct ttm_tt {
 	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
 	 * struct ttm_tt has been (possibly partially) backed up.
 	 *
+	 * TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED: Set by the TTM pool allocator
+	 * when at least one chunk that could have been allocated at the pool's
+	 * beneficial order had to fall back to a smaller order. This indicates
+	 * that a sub-optimal set of pages was chosen for this object, and can
+	 * be used by drivers to queue the buffer object for later
+	 * defragmentation. Drivers should access this via the
+	 * ttm_tt_is_beneficial_order_failed() helper.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -101,8 +109,9 @@ struct ttm_tt {
 #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 #define TTM_TT_FLAG_BACKED_UP	        BIT(5)
+#define TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED	BIT(6)
 
-#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
+#define TTM_TT_FLAG_PRIV_POPULATED	BIT(7)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -179,6 +188,19 @@ static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
 	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 }
 
+/**
+ * ttm_tt_is_beneficial_order_failed() - Whether the tt is backed at a
+ * sub-optimal page order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if the pool allocator had to fall back below the pool's
+ * beneficial order when backing this tt, false otherwise.
+ */
+static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
+{
+	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+}
+
 /**
  * ttm_tt_create
  *
-- 
2.34.1

[PATCH v3 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

[Matthew Brost]

__ttm_pool_alloc() walks the page orders in a single loop whose body is
about to grow several defragmentation phases (reusing old pages, a move
budget, out-of-lock preallocation and fault injection). Left inline these
interleave into one deeply nested block of gotos and combinatorial
conditionals that is hard to follow and extend.

Factor the loop's working state into a struct ttm_pool_alloc_iter and
pull the per-order page acquisition and order-fallback into small phase
helpers, so the main loop reads as a short sequence of named steps. This
is a pure refactor with no functional change; subsequent patches extend
the iterator with their defrag phases rather than nesting more logic into
the loop.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 159 ++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 53 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index e9175c0290d7..0ca63f5ce73c 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -791,18 +791,102 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/**
+ * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
+ *
+ * Bundles the immutable inputs and the mutable per-order state that the
+ * allocation phase helpers share, so the main loop reads as a short sequence
+ * of named phases rather than one deeply nested block.
+ */
+struct ttm_pool_alloc_iter {
+	/* Immutable for the duration of the allocation. */
+	struct ttm_pool *pool;
+	struct ttm_tt *tt;
+	const struct ttm_operation_ctx *ctx;
+	struct ttm_pool_alloc_state *alloc;
+	struct ttm_pool_tt_restore *restore;
+	unsigned int beneficial_order;
+	gfp_t gfp_flags;
+
+	/* Mutated as the loop walks the orders. */
+	unsigned int order;
+	enum ttm_caching page_caching;
+	bool allow_pools;
+	struct page *p;
+};
+
+/*
+ * Acquire a single page for the current order, leaving it in @it->p (NULL on
+ * failure). Tries a same-order pool page, then a fresh system allocation.
+ */
+static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
+{
+	struct ttm_pool_type *pt;
+
+	/* First, try to allocate a page from a pool if one exists. */
+	it->p = NULL;
+	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
+	if (!it->p && pt && it->allow_pools)
+		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
+
+	/*
+	 * If that fails or previously failed, allocate from system. Note that
+	 * this also disallows additional pool allocations using write-back
+	 * cached pools of the same order.
+	 */
+	if (!it->p) {
+		it->page_caching = ttm_cached;
+		it->allow_pools = false;
+		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
+					    it->ctx->beneficial_reclaim_backoff);
+	}
+}
+
+/*
+ * The current order could not be satisfied. Lower it and retry, recording a
+ * sub-optimal backing (so the driver can later re-defrag) when we drop below
+ * the beneficial order.
+ *
+ * Returns 0 to retry the (lowered) order, or a negative errno.
+ */
+static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
+{
+	if (!it->order)
+		return -ENOMEM;
+
+	/*
+	 * Failing to allocate at the device's beneficial order means we are
+	 * about to back this object with a sub-optimal (smaller order) set of
+	 * pages. Record it so the driver can later try to defragment the object
+	 * back to beneficial order.
+	 */
+	if (it->beneficial_order && it->order == it->beneficial_order)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	it->order--;
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return 0;
+}
+
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    const struct ttm_operation_ctx *ctx,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
-	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
-	enum ttm_caching page_caching;
+	struct ttm_pool_alloc_iter it = {
+		.pool = pool,
+		.tt = tt,
+		.ctx = ctx,
+		.alloc = alloc,
+		.restore = restore,
+		.beneficial_order = ttm_pool_beneficial_order(pool),
+		.page_caching = tt->caching,
+		.allow_pools = true,
+	};
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
-	unsigned int order;
-	bool allow_pools;
-	struct page *p;
 	int r;
 
 	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
@@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	page_caching = tt->caching;
-	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	it.gfp_flags = gfp_flags;
+
+	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, alloc)) {
-		struct ttm_pool_type *pt;
-
-		/* First, try to allocate a page from a pool if one exists. */
-		p = NULL;
-		pt = ttm_pool_select_type(pool, page_caching, order);
-		if (pt && allow_pools)
-			p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
-
-		/*
-		 * If that fails or previously failed, allocate from system.
-		 * Note that this also disallows additional pool allocations using
-		 * write-back cached pools of the same order. Consider removing
-		 * that behaviour.
-		 */
-		if (!p) {
-			page_caching = ttm_cached;
-			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order,
-						ctx->beneficial_reclaim_backoff);
-		}
-		/* If that fails, lower the order if possible and retry. */
-		if (!p) {
-			if (order) {
-				/*
-				 * Failing to allocate at the device's beneficial
-				 * order means we are about to back this object
-				 * with a sub-optimal (smaller order) set of
-				 * pages. Record it so the driver can later try to
-				 * defragment the object back to beneficial order.
-				 */
-				if (beneficial_order && order == beneficial_order)
-					tt->page_flags |=
-						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-				--order;
-				page_caching = tt->caching;
-				allow_pools = true;
-				continue;
-			}
-			r = -ENOMEM;
-			goto error_free_all;
+	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Acquire a page (pool / system) for this order. */
+		ttm_pool_iter_acquire_page(&it);
+		if (!it.p) {
+			r = ttm_pool_iter_lower_order(&it);
+			if (r)
+				goto error_free_all;
+			continue;
 		}
-		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
-					    restore);
+
+		r = ttm_pool_page_allocated(pool, it.order, it.p,
+					    it.page_caching, alloc, restore);
 		if (r)
 			goto error_free_page;
 
 		if (ttm_pool_restore_valid(restore)) {
-			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
+			r = ttm_pool_restore_commit(restore, tt->backup, ctx,
+						    alloc);
 			if (r)
 				goto error_free_all;
 		}
@@ -887,7 +940,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	return 0;
 
 error_free_page:
-	ttm_pool_free_page(pool, page_caching, order, p, false);
+	ttm_pool_free_page(pool, it.page_caching, it.order, it.p, false);
 
 error_free_all:
 	if (tt->restore)
-- 
2.34.1

[PATCH v3 05/33] drm/ttm: Support defragmentation moves

[Matthew Brost]

Add infrastructure to let a driver re-back an already-resident BO with
fresh pages (e.g. at a more beneficial order) by reusing the normal move
machinery.

Add ctx->defrag, which makes ttm_bo_validate() skip the "current
resource already compatible" short-circuit so a move is forced even when
placement is unchanged. In ttm_bo_handle_move_mem(), when defragging a
populated BO, stash the old tt, create and populate a fresh one
(allocating new backing), and hand the old tt to the driver move
callback via bo->defrag_old_tt for copying. On success the old tt is
unpopulated and freed; if populate or the move fails, the freshly
allocated backing is discharged and the BO is restored to its original
tt and placement.

The old tt is handed off through bo->defrag_old_tt (reservation-lock
protected) rather than the operation context. The pointer describes BO
state for the duration of the move, so keeping it on the buffer object
is better layering and lets ttm_bo_move_accel_cleanup() pick it up.

Because a defrag move commonly fails (beneficial-order pages are often
unavailable) and then leaves the BO untouched on its original backing,
defer ttm_bo_unmap_virtual() for the defrag case until the new backing
has been populated and the move is about to commit. Ordinary moves still
unmap up front. This avoids needlessly zapping and refaulting every
userspace mapping on the common failure path where nothing moved.

Teach ttm_bo_move_accel_cleanup()'s ghost path about defrag: instead of
tearing the move's source tt down synchronously, hand bo->defrag_old_tt
to the ghost object so it is kept populated until the copy fence signals
and then unbound and destroyed asynchronously, while the BO keeps its
new tt. This lets a driver pipeline defrag teardown behind the GPU copy
rather than blocking on it.

This is driver-opt-in and a no-op unless ctx->defrag is set.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Reject defrag moves needing a multihop bounce: WARN, restore backing,
   -EINVAL (Sashiko)
---
 drivers/gpu/drm/ttm/ttm_bo.c      | 100 +++++++++++++++++++++++++++---
 drivers/gpu/drm/ttm/ttm_bo_util.c |  16 ++++-
 drivers/gpu/drm/ttm/ttm_pool.c    |   5 +-
 include/drm/ttm/ttm_bo.h          |  17 +++++
 4 files changed, 127 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 3980f376e3ba..c1629b4c4f0f 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -125,49 +125,133 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 {
 	struct ttm_device *bdev = bo->bdev;
 	bool old_use_tt, new_use_tt;
+	bool defrag;
 	int ret;
 
 	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
 	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
 
-	ttm_bo_unmap_virtual(bo);
+	/*
+	 * Only the BO that is the actual defrag target (moved in place via
+	 * ttm_bo_validate(), evict == false) is being defragmented. The same
+	 * ctx->defrag is threaded into evictions triggered while allocating
+	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
+	 * an unrelated victim BO (evict == true) being misclassified as a
+	 * defrag move and needlessly reallocating its backing.
+	 */
+	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
+		 ttm_tt_is_populated(bo->ttm);
+
+	/*
+	 * Tear down userspace mappings up front for ordinary moves. A defrag
+	 * move is likely to leave the BO in place (beneficial-order allocation
+	 * commonly fails), so defer the unmap until the new backing has been
+	 * populated and the move is about to commit. This avoids a needless
+	 * zap and refault of every userspace mapping on the common failure,
+	 * where the BO keeps its original, still-mapped backing.
+	 */
+	if (!defrag)
+		ttm_bo_unmap_virtual(bo);
 
 	/*
 	 * Create and bind a ttm if required.
 	 */
 
 	if (new_use_tt) {
+		struct ttm_tt *defrag_old_tt = NULL;
+
+		/*
+		 * For a defrag move, stash the current populated tt and force a
+		 * fresh one to be created and populated (at beneficial order).
+		 * The old tt is handed to the driver move callback to copy the
+		 * contents from, and freed once the move completes.
+		 */
+		if (defrag) {
+			defrag_old_tt = bo->ttm;
+			bo->ttm = NULL;
+		}
+
 		/* Zero init the new TTM structure if the old location should
 		 * have used one as well.
 		 */
-		ret = ttm_tt_create(bo, old_use_tt);
-		if (ret)
+		ret = ttm_tt_create(bo, old_use_tt && !defrag_old_tt);
+		if (ret) {
+			if (defrag_old_tt)
+				bo->ttm = defrag_old_tt;
 			goto out_err;
+		}
 
-		if (mem->mem_type != TTM_PL_SYSTEM) {
+		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
 			ret = ttm_bo_populate(bo, ctx);
-			if (ret)
+			if (ret) {
+				/*
+				 * Discharge the freshly allocated backing and
+				 * leave the BO on its original tt/placement.
+				 */
+				if (defrag_old_tt) {
+					ttm_tt_destroy(bo->bdev, bo->ttm);
+					bo->ttm = defrag_old_tt;
+				}
 				goto out_err;
+			}
 		}
+
+		bo->defrag_old_tt = defrag_old_tt;
 	}
 
 	ret = dma_resv_reserve_fences(bo->base.resv, 1);
 	if (ret)
 		goto out_err;
 
+	/*
+	 * A defrag move deferred its virtual unmap until the new backing was
+	 * successfully populated; tear the userspace mappings down now, before
+	 * the move swaps in the new pages, so the next fault re-establishes
+	 * them against the new backing.
+	 */
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
 	if (ret) {
-		if (ret == -EMULTIHOP)
+		if (ret == -EMULTIHOP) {
+			/*
+			 * A defrag move targets the BO's current placement and
+			 * must complete in a single hop. A driver requesting a
+			 * bounce here is unsupported: restore the original
+			 * backing and fail the defrag rather than retrying the
+			 * multihop with ctx->defrag still set, which would
+			 * misclassify the bounce as another defrag move and
+			 * overwrite/leak bo->defrag_old_tt.
+			 */
+			if (unlikely(defrag)) {
+				WARN_ON_ONCE(1);
+				ret = -EINVAL;
+				goto out_err;
+			}
 			return ret;
+		}
 		goto out_err;
 	}
 
+	if (bo->defrag_old_tt) {
+		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
+		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
+		bo->defrag_old_tt = NULL;
+	}
+
 	ctx->bytes_moved += bo->base.size;
 	return 0;
 
 out_err:
-	if (!old_use_tt)
+	if (bo->defrag_old_tt) {
+		/* Failed defrag move: restore the original backing. */
+		ttm_bo_tt_destroy(bo);
+		bo->ttm = bo->defrag_old_tt;
+		bo->defrag_old_tt = NULL;
+	} else if (!old_use_tt) {
 		ttm_bo_tt_destroy(bo);
+	}
 
 	return ret;
 }
@@ -835,7 +919,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
 	force_space = false;
 	do {
 		/* Check whether we need to move buffer. */
-		if (bo->resource &&
+		if (bo->resource && !ctx->defrag &&
 		    ttm_resource_compatible(bo->resource, placement,
 					    force_space))
 			return 0;
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 3e3c201a0222..771313162c8d 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -626,10 +626,22 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 	 * bo to be unbound and destroyed.
 	 */
 
-	if (dst_use_tt)
+	if (bo->defrag_old_tt) {
+		/*
+		 * Defrag move: @bo already points at the freshly allocated,
+		 * beneficial-order tt that the move copied into. Hand the old
+		 * tt (the move's source) to the ghost so it is kept populated
+		 * until @fence signals and then unbound and destroyed, instead
+		 * of being torn down synchronously. @bo keeps its new tt.
+		 */
+		ghost_obj->ttm = bo->defrag_old_tt;
+		ghost_obj->defrag_old_tt = NULL;
+		bo->defrag_old_tt = NULL;
+	} else if (dst_use_tt) {
 		ghost_obj->ttm = NULL;
-	else
+	} else {
 		bo->ttm = NULL;
+	}
 
 	dma_resv_unlock(&ghost_obj->base._resv);
 	ttm_bo_put(ghost_obj);
diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 0ca63f5ce73c..d045105afc24 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -860,8 +860,11 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order)
+	if (it->beneficial_order && it->order == it->beneficial_order) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		if (it->ctx->defrag)
+			return -ENOMEM;
+	}
 
 	it->order--;
 	it->page_caching = it->tt->caching;
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index fd814b5cf535..f4f8a8c29f04 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -135,6 +135,16 @@ struct ttm_buffer_object {
 	 * reservation lock.
 	 */
 	struct sg_table *sg;
+
+	/**
+	 * @defrag_old_tt: Used internally during a defrag move to hand the BO's
+	 * previous, populated struct ttm_tt to the driver's move callback so
+	 * that it can copy the contents into the freshly allocated
+	 * beneficial-order backing. Set by TTM, consumed by the driver; the old
+	 * tt is freed (or restored on failure) by TTM once the move completes.
+	 * Protected by the reservation lock.
+	 */
+	struct ttm_tt *defrag_old_tt;
 };
 
 #define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -194,6 +204,13 @@ struct ttm_operation_ctx {
 	 * reclaim/compaction before falling back to a smaller order anyway.
 	 */
 	bool beneficial_reclaim_backoff;
+	/**
+	 * @defrag: Defragmentation pass. Attempt a move even when the bo's
+	 * current resource is already compatible with the requested placement,
+	 * so that the backing store can be reallocated at the pool's beneficial
+	 * order.
+	 */
+	bool defrag;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
-- 
2.34.1

[PATCH v3 06/33] drm/ttm: Add fault injection for beneficial-order allocation failures

[Matthew Brost]

Add a beneficial_order_fault_inject debugfs fault attribute, mirroring
backup_fault_inject, that forces allocation at (or above) the device's
beneficial order to fail in __ttm_pool_alloc(). When triggered, both the
pool take and the system page allocation are skipped for that order, so
the allocation falls into the existing lower-the-order path that sets
the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (and bails out with
-ENOMEM for defrag moves).

This lets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tracking and the
driver-side defrag path be exercised deterministically, without having to
drive the system into real memory fragmentation to provoke a sub-optimal
backing.

The knob is registered at
/sys/kernel/debug/ttm/beneficial_order_fault_inject and compiles out
when CONFIG_FAULT_INJECTION is disabled.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 35 ++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index d045105afc24..2598d2f33767 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -53,6 +53,7 @@
 #ifdef CONFIG_FAULT_INJECTION
 #include <linux/fault-inject.h>
 static DECLARE_FAULT_ATTR(backup_fault_inject);
+static DECLARE_FAULT_ATTR(beneficial_order_fault_inject);
 #else
 #define should_fail(...) false
 #endif
@@ -812,21 +813,34 @@ struct ttm_pool_alloc_iter {
 	unsigned int order;
 	enum ttm_caching page_caching;
 	bool allow_pools;
+	bool fail_beneficial;
 	struct page *p;
 };
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation.
+ * failure). Tries a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
 	struct ttm_pool_type *pt;
 
-	/* First, try to allocate a page from a pool if one exists. */
 	it->p = NULL;
+
+	/*
+	 * Fault injection: pretend allocation at (or above) the device's
+	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
+	 * beneficial_order_failed tracking and the driver defrag path without
+	 * driving the system into real fragmentation.
+	 */
+	it->fail_beneficial = IS_ENABLED(CONFIG_FAULT_INJECTION) &&
+		it->beneficial_order && it->order >= it->beneficial_order &&
+		should_fail(&beneficial_order_fault_inject, 1);
+
+	/* First, try to allocate a page from a pool if one exists. */
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
-	if (!it->p && pt && it->allow_pools)
+	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
 
 	/*
@@ -834,7 +848,7 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 	 * this also disallows additional pool allocations using write-back
 	 * cached pools of the same order.
 	 */
-	if (!it->p) {
+	if (!it->p && !it->fail_beneficial) {
 		it->page_caching = ttm_cached;
 		it->allow_pools = false;
 		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
@@ -851,6 +865,9 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
  */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
+	bool at_beneficial = it->beneficial_order &&
+			     it->order == it->beneficial_order;
+
 	if (!it->order)
 		return -ENOMEM;
 
@@ -860,13 +877,16 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order) {
+	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		if (it->ctx->defrag)
 			return -ENOMEM;
 	}
 
-	it->order--;
+	if (it->fail_beneficial)
+		it->order = it->beneficial_order - 1;
+	else
+		it->order--;
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
@@ -1536,6 +1556,9 @@ int ttm_pool_mgr_init(unsigned long num_pages)
 #ifdef CONFIG_FAULT_INJECTION
 	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
 				  &backup_fault_inject);
+	fault_create_debugfs_attr("beneficial_order_fault_inject",
+				  ttm_debugfs_root,
+				  &beneficial_order_fault_inject);
 #endif
 #endif
 
-- 
2.34.1

[PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

[Matthew Brost]

A defragmentation move relocates a populated tt into a freshly allocated
one so that sub-beneficial-order chunks can be upgraded to the device's
beneficial order. Previously, if allocation could not complete entirely
at beneficial order, the move failed and all progress was discarded.

Instead, once allocation has successfully upgraded part of the object,
the remainder of the old tt is harvested to complete the new tt. Thread
the BO's old, still-populated tt into the page allocator via a new
ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding,
any existing chunk already at (or above) the beneficial order is reused
directly instead of being reallocated. If allocation later has to fall
back below beneficial_order after making partial progress, the remainder
of the old tt is harvested instead of failing the move. This preserves
the beneficial-order upgrades already made while leaving the rest of the
object unchanged, allowing subsequent defragmentation attempts to make
additional forward progress.

Harvesting only borrows the pages; they remain owned by the old tt until
the move commits. This keeps every unwind path non-destructive: if the
(re)allocation or the move later fails, the new tt is torn down without
freeing the borrowed pages and the old tt is restored fully intact.
Ownership of the shared pages is transferred to the new tt only once the
move has committed. ttm_tt_defrag_disown_borrowed() implements both
directions of the transfer by clearing a tt's slots for every page it
shares with the other tt (same struct page at the same index), so that
tt's teardown leaves the shared pages alone and each page is freed
exactly once by its owner:

- on the alloc unwind and the failed-move paths, the new tt disowns the
  borrowed pages, leaving the old tt as the sole owner;
- on the successful-move paths (synchronous teardown and the pipelined
  ghost-object teardown), the old tt disowns the now-transferred pages,
  leaving the new tt as the sole owner.

Detecting shared pages by same-index struct page identity needs no extra
per-tt bookkeeping, since two distinct allocations never alias. This is
safe with drivers, such as Xe, that build their own per-tt sg table and
DMA-map it independently: each tt maps and unmaps its own sg, so sharing
the underlying pages between tts is DMA-clean.

This is a no-op unless ctx->defrag_old_tt is set, which only happens on a
defragmentation move.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Document the chunk-head invariant read by
   ttm_pool_harvest_remaining() (Sashiko)
---
 drivers/gpu/drm/ttm/ttm_bo.c      |  20 ++-
 drivers/gpu/drm/ttm/ttm_bo_util.c |   5 +
 drivers/gpu/drm/ttm/ttm_pool.c    | 235 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c      |  67 +++++++++
 include/drm/ttm/ttm_bo.h          |  12 ++
 include/drm/ttm/ttm_tt.h          |   2 +
 6 files changed, 328 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index c1629b4c4f0f..ab2e637a017f 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		}
 
 		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
+			/*
+			 * Let the page allocator harvest already
+			 * beneficial-order chunks out of the old tt to back
+			 * the new one, rather than reallocating them.
+			 */
+			ctx->defrag_old_tt = defrag_old_tt;
 			ret = ttm_bo_populate(bo, ctx);
+			ctx->defrag_old_tt = NULL;
 			if (ret) {
 				/*
 				 * Discharge the freshly allocated backing and
@@ -235,6 +242,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	}
 
 	if (bo->defrag_old_tt) {
+		/*
+		 * Successful defrag move: any pages the new tt borrowed from the
+		 * old tt are now owned by the new tt. Disown them from the old
+		 * tt before it is torn down so they are not freed.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
 		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
 		bo->defrag_old_tt = NULL;
@@ -245,7 +258,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 
 out_err:
 	if (bo->defrag_old_tt) {
-		/* Failed defrag move: restore the original backing. */
+		/*
+		 * Failed defrag move: drop any pages the new tt borrowed from
+		 * the old tt (still owned by the old tt) before destroying the
+		 * new tt, then restore the original backing.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
 		ttm_bo_tt_destroy(bo);
 		bo->ttm = bo->defrag_old_tt;
 		bo->defrag_old_tt = NULL;
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 771313162c8d..7cb718fc97c0 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 		 * tt (the move's source) to the ghost so it is kept populated
 		 * until @fence signals and then unbound and destroyed, instead
 		 * of being torn down synchronously. @bo keeps its new tt.
+		 *
+		 * Any pages the new tt borrowed from the old tt are now owned by
+		 * the new tt; disown them from the old tt so the ghost teardown
+		 * leaves them alone.
 		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ghost_obj->ttm = bo->defrag_old_tt;
 		ghost_obj->defrag_old_tt = NULL;
 		bo->defrag_old_tt = NULL;
diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 2598d2f33767..52f5e64f05dd 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -792,6 +792,148 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/*
+ * Defragmentation page harvesting.
+ *
+ * A defrag move relocates a populated tt into a freshly allocated one in order
+ * to upgrade sub-beneficial-order chunks to the device's beneficial order.
+ * Chunks in the old tt (ctx->defrag_old_tt) that are ALREADY at the beneficial
+ * order do not need to be reallocated: their physical pages can simply be
+ * transferred ("harvested") into the new tt at the same offset, avoiding the
+ * alloc/free churn that is significant for large buffers.
+ *
+ * Harvesting only borrows the pages: the new tt simply references the old tt's
+ * pages, which remain owned by the old tt. This keeps the unwind paths simple -
+ * if the (re)allocation or the move later fails, the new tt is torn down without
+ * freeing the borrowed pages and the old tt is restored fully intact. Ownership
+ * of the shared pages is transferred to the new tt only once the move has
+ * committed, via ttm_tt_defrag_disown_borrowed() (see ttm_tt.c).
+ */
+
+/*
+ * Return the allocation order of the old tt chunk whose head sits at @off, or 0
+ * if @off is not an old chunk head (it falls inside a chunk, or the slot is
+ * empty / backed up). @cursor monotonically tracks the old chunk boundary and
+ * must be advanced past every offset that has already been queried; a page
+ * order may only be read from a chunk head page, hence the lazy walk.
+ */
+static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
+					      struct ttm_tt *old_tt,
+					      pgoff_t off, pgoff_t *cursor)
+{
+	struct page *p;
+
+	while (*cursor < off) {
+		p = old_tt->pages[*cursor];
+		if (!p || ttm_backup_page_ptr_is_handle(p))
+			return 0;
+		*cursor += 1UL << ttm_pool_page_order(pool, p);
+	}
+
+	if (*cursor != off)
+		return 0;
+
+	p = old_tt->pages[off];
+	if (!p || ttm_backup_page_ptr_is_handle(p))
+		return 0;
+
+	return ttm_pool_page_order(pool, p);
+}
+
+/*
+ * Borrow an already beneficial-order chunk from the old tt into @alloc. The
+ * pages stay owned by the old tt until the move commits.
+ */
+static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
+				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+{
+	pgoff_t nr = 1UL << order;
+	int r;
+
+	/*
+	 * A harvested chunk keeps the (same) bo's caching, so the deferred
+	 * caching range is consistent here; flush it before committing, exactly
+	 * like the caching-consistent path in ttm_pool_page_allocated().
+	 */
+	r = ttm_pool_apply_caching(alloc);
+	if (r)
+		return r;
+
+	/*
+	 * The borrowed chunk's page pointers (consecutive struct pages of one
+	 * compound allocation) and dma addresses already sit contiguously in
+	 * the old tt at @off. Copy the whole run with memcpy() instead of the
+	 * per-page increment loop in ttm_pool_allocated_page_commit(): for a
+	 * large defrag the borrowed pages dominate the new tt, so this turns
+	 * the populate fill from O(num_pages) individual stores into a handful
+	 * of bulk copies.
+	 */
+	memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
+	alloc->pages += nr;
+	alloc->remaining_pages -= nr;
+
+	if (alloc->dma_addr) {
+		memcpy(alloc->dma_addr, &old_tt->dma_address[off],
+		       nr * sizeof(*alloc->dma_addr));
+		alloc->dma_addr += nr;
+	}
+
+	alloc->caching_divide = alloc->pages;
+
+	return 0;
+}
+
+static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
+				      struct ttm_tt *old_tt, pgoff_t off,
+				      struct ttm_pool_alloc_state *alloc)
+{
+	/*
+	 * @off is always an old-tt chunk head here, never a tail page, so
+	 * reading the per-chunk order below is safe. This holds because:
+	 *  - Old chunks are self-aligned: greedy largest-power-of-two fill
+	 *    places an order-k chunk at a 2^k-aligned offset, so any
+	 *    beneficial-order-aligned offset is a chunk boundary.
+	 *  - Fresh defrag allocations are capped at the beneficial order (see
+	 *    ttm_pool_iter_acquire_page()), so they advance @off in beneficial
+	 *    steps and never overshoot into the middle of a larger old chunk;
+	 *    super-beneficial old chunks are always harvested whole by
+	 *    ttm_pool_iter_reuse_old().
+	 *  - We are only reached once the defrag budget/prealloc is exhausted,
+	 *    which in the capped/prealloc config leaves @off beneficial-aligned
+	 *    (or, in the sub-beneficial tail, already walking old boundaries).
+	 */
+	while (alloc->remaining_pages) {
+		struct page *p = old_tt->pages[off];
+		unsigned int order = ttm_pool_page_order(pool, p);
+		pgoff_t nr = 1UL << order;
+		int r;
+
+		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		if (r)
+			return r;
+
+		off += nr;
+	}
+
+	return 0;
+}
+
+/**
+ * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
+ * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
+ * @TTM_POOL_ITER_RETRY: Loop state was adjusted (e.g. order lowered or a chunk
+ *	harvested); re-enter the loop body for the next order.
+ * @TTM_POOL_ITER_STOP: The allocation is complete; stop the loop.
+ *
+ * Phase helpers return one of these (>= 0) on success, or a negative errno on
+ * failure (the caller then unwinds via error_free_all).
+ */
+enum ttm_pool_iter_action {
+	TTM_POOL_ITER_FILL,
+	TTM_POOL_ITER_RETRY,
+	TTM_POOL_ITER_STOP,
+};
+
 /**
  * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
  *
@@ -806,6 +948,7 @@ struct ttm_pool_alloc_iter {
 	const struct ttm_operation_ctx *ctx;
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
+	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 
@@ -814,9 +957,50 @@ struct ttm_pool_alloc_iter {
 	enum ttm_caching page_caching;
 	bool allow_pools;
 	bool fail_beneficial;
+	unsigned int alloc_count;
+	pgoff_t old_cursor;
 	struct page *p;
 };
 
+/*
+ * Defrag move: reuse an already beneficial-order chunk from the old tt at this
+ * offset rather than reallocating it. Harvest whatever order the old chunk has
+ * (>= beneficial order), which may exceed the order the new tt would otherwise
+ * pick. An order below beneficial_order is only reused for the unaligned tail
+ * (where there is no benefit to reallocating). Only borrows the pages; the alloc
+ * unwind path can drop them again.
+ *
+ * Returns TTM_POOL_ITER_RETRY when a chunk was harvested, TTM_POOL_ITER_FILL to
+ * fall through to a fresh allocation, or a negative errno.
+ */
+static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
+{
+	unsigned int harvest_order;
+	pgoff_t off;
+	int r;
+
+	if (!it->defrag_old_tt || !it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	off = it->tt->num_pages - it->alloc->remaining_pages;
+	harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
+						  off, &it->old_cursor);
+
+	if (harvest_order < it->beneficial_order &&
+	    it->order >= it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	it->order = harvest_order;
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	if (r)
+		return r;
+
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return TTM_POOL_ITER_RETRY;
+}
+
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
  * failure). Tries a same-order pool page, then a fresh system allocation. Fault
@@ -859,9 +1043,10 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 /*
  * The current order could not be satisfied. Lower it and retry, recording a
  * sub-optimal backing (so the driver can later re-defrag) when we drop below
- * the beneficial order.
+ * the beneficial order. For a defrag move that has already made partial
+ * progress, harvest the remaining old pages and stop instead.
  *
- * Returns 0 to retry the (lowered) order, or a negative errno.
+ * Returns TTM_POOL_ITER_RETRY, TTM_POOL_ITER_STOP, or a negative errno.
  */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
@@ -871,16 +1056,24 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	if (!it->order)
 		return -ENOMEM;
 
-	/*
-	 * Failing to allocate at the device's beneficial order means we are
-	 * about to back this object with a sub-optimal (smaller order) set of
-	 * pages. Record it so the driver can later try to defragment the object
-	 * back to beneficial order.
-	 */
 	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-		if (it->ctx->defrag)
+
+		if (it->alloc_count && it->defrag_old_tt) {
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+			int r;
+
+			r = ttm_pool_harvest_remaining(it->pool,
+						       it->defrag_old_tt, off,
+						       it->alloc);
+			if (r)
+				return r;
+
+			return TTM_POOL_ITER_STOP;
+		} else if (it->defrag_old_tt) {
 			return -ENOMEM;
+		}
 	}
 
 	if (it->fail_beneficial)
@@ -890,7 +1083,7 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
-	return 0;
+	return TTM_POOL_ITER_RETRY;
 }
 
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
@@ -904,6 +1097,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.ctx = ctx,
 		.alloc = alloc,
 		.restore = restore,
+		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -931,13 +1125,22 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Reuse an already beneficial-order chunk from the old tt. */
+		r = ttm_pool_iter_reuse_old(&it);
+		if (r < 0)
+			goto error_free_all;
+		if (r == TTM_POOL_ITER_RETRY)
+			continue;
+
 		/* Acquire a page (pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
-			if (r)
+			if (r < 0)
 				goto error_free_all;
-			continue;
+			if (r == TTM_POOL_ITER_RETRY)
+				continue;
+			break;	/* TTM_POOL_ITER_STOP */
 		}
 
 		r = ttm_pool_page_allocated(pool, it.order, it.p,
@@ -951,6 +1154,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			if (r)
 				goto error_free_all;
 		}
+
+		it.alloc_count++;
 	}
 
 	r = ttm_pool_apply_caching(alloc);
@@ -969,6 +1174,12 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (tt->restore)
 		return r;
 
+	/*
+	 * Drop any pages borrowed from the old tt during a defrag move: they are
+	 * still owned by the old tt and must not be freed here.
+	 */
+	ttm_tt_defrag_disown_borrowed(tt, it.defrag_old_tt);
+
 	caching_divide = alloc->caching_divide - tt->pages;
 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
 	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index b645a1818184..2388ff794a59 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -450,6 +450,73 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
 }
 EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_unpopulate);
 
+/**
+ * ttm_tt_defrag_disown_borrowed() - Drop borrowed defrag pages from a tt
+ * @tt: The tt to drop the borrowed page references from.
+ * @src: The tt the pages may have been harvested from, or NULL.
+ *
+ * During a defragmentation move the freshly (re)allocated tt may harvest
+ * already beneficial-order chunks from the BO's old tt, sharing the underlying
+ * pages instead of reallocating them. Until the move commits these pages stay
+ * owned by the old tt. Clearing a tt's slots for every page it shares with
+ * @src (the same struct page at the same index) makes that tt's teardown leave
+ * the shared pages alone, so they are freed exactly once by their owner.
+ *
+ * This is used both to unwind a failed defrag move (disown the borrowed pages
+ * from the new tt, leaving the old tt intact) and to commit a successful one
+ * (disown the now-transferred pages from the old tt, leaving them owned by the
+ * new tt).
+ */
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
+{
+	pgoff_t i = 0, run_start = 0;
+	bool in_run = false;
+
+	if (!tt || !src)
+		return;
+
+	/*
+	 * Borrowing happens at chunk granularity: a borrowed chunk shares the
+	 * same struct page on both tts across its whole allocation order, while
+	 * a non-borrowed chunk differs across its whole order. Stride by @src's
+	 * chunk order instead of walking every page, coalescing the borrowed
+	 * (matching) chunks into runs and clearing each run with a single
+	 * memset rather than a per-page store. @src is a fully populated,
+	 * non-dma_alloc tt (the defrag path), so its page order is recorded in
+	 * page->private and read via ttm_pool_page_order_nodma().
+	 */
+	while (i < tt->num_pages) {
+		struct page *sp = src->pages[i];
+		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
+		pgoff_t nr = 1UL << order;
+
+		if (sp && tt->pages[i] == sp) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+		} else if (in_run) {
+			memset(&tt->pages[run_start], 0,
+			       (i - run_start) * sizeof(*tt->pages));
+			if (tt->dma_address)
+				memset(&tt->dma_address[run_start], 0,
+				       (i - run_start) *
+				       sizeof(*tt->dma_address));
+			in_run = false;
+		}
+
+		i += nr;
+	}
+
+	if (in_run) {
+		memset(&tt->pages[run_start], 0,
+		       (i - run_start) * sizeof(*tt->pages));
+		if (tt->dma_address)
+			memset(&tt->dma_address[run_start], 0,
+			       (i - run_start) * sizeof(*tt->dma_address));
+	}
+}
+
 #ifdef CONFIG_DEBUG_FS
 
 /* Test the shrinker functions and dump the result */
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index f4f8a8c29f04..fcaba5fb8bf7 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -211,6 +211,18 @@ struct ttm_operation_ctx {
 	 * order.
 	 */
 	bool defrag;
+	/**
+	 * @defrag_old_tt: During a defragmentation move, the bo's existing
+	 * (populated) tt that is being relocated. The page allocator may
+	 * harvest already beneficial-order chunks from this tt to back the new
+	 * tt instead of reallocating them, avoiding alloc/free churn for large
+	 * buffers. Harvested pages are only borrowed: they stay owned by
+	 * @defrag_old_tt until the move commits, at which point
+	 * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
+	 * transferred pages so they are not freed when the old tt is torn down.
+	 * NULL for non-defrag operations.
+	 */
+	struct ttm_tt *defrag_old_tt;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index ce7533677d77..55c35bcf134d 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
  */
 void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
 
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
+
 /**
  * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
  *
-- 
2.34.1

[PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

[Matthew Brost]

A defragmentation move reallocates an object's backing at the device's
beneficial order and harvests whatever the allocator cannot upgrade from
the old tt. For a very large object this can reallocate (and have the
driver re-copy) the entire backing in one move, which is expensive and
can stall concurrent work for an unbounded stretch.

Add an optional per-move byte budget so a large object can be upgraded in
slices across successive moves. Plumb a new
ttm_operation_ctx::defrag_bytes_remaining through to the pool allocator:
only pages actually (re)allocated at the beneficial order are charged
against it; pages harvested (borrowed) from defrag_old_tt are free and do
not count. Once the budget is exhausted the allocator stops upgrading and
completes the new tt by harvesting the remainder of the old tt as-is,
marking the tt sub-optimal (TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED) so the
driver re-queues it and a later move resumes the upgrade. The slices
already upgraded come back as free already-beneficial harvests on the
next pass, so each move spends budget only on new forward progress.

The allocator decrements the budget as it allocates and writes the unused
remainder back on success, letting the caller compute how many bytes were
actually spent. A value of 0 means unlimited (upgrade the whole object in
one move), preserving the previous behaviour. This is only consulted on a
defragmentation move, i.e. when defrag_old_tt is set.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 101 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  14 +++++
 2 files changed, 112 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 52f5e64f05dd..1673e81b8df8 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -77,6 +77,11 @@ struct ttm_pool_dma {
  * @dma_addr: Pointer to the next tt dma_address entry to populate if any.
  * @remaining_pages: Remaining pages to populate.
  * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @defrag_bytes_remaining: Byte budget for newly (re)allocated pages on a
+ * defragmentation move, seeded from ttm_operation_ctx::defrag_bytes_remaining.
+ * Pages harvested (borrowed) from the old tt are free and do not count; once
+ * this drops to zero the allocator stops upgrading and harvests the remainder.
+ * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -84,6 +89,7 @@ struct ttm_pool_alloc_state {
 	dma_addr_t *dma_addr;
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
+	s64 defrag_bytes_remaining;
 };
 
 /**
@@ -780,6 +786,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->dma_addr = tt->dma_address;
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
+	alloc->defrag_bytes_remaining = 0;
 }
 
 /*
@@ -885,8 +892,12 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 				      struct ttm_tt *old_tt, pgoff_t off,
-				      struct ttm_pool_alloc_state *alloc)
+				      struct ttm_pool_alloc_state *alloc,
+				      bool *suboptimal)
 {
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	pgoff_t bnr = 1UL << beneficial;
+
 	/*
 	 * @off is always an old-tt chunk head here, never a tail page, so
 	 * reading the per-chunk order below is safe. This holds because:
@@ -906,8 +917,23 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		struct page *p = old_tt->pages[off];
 		unsigned int order = ttm_pool_page_order(pool, p);
 		pgoff_t nr = 1UL << order;
+		pgoff_t num_pages = off + alloc->remaining_pages;
 		int r;
 
+		/*
+		 * Only report a sub-optimal backing when a chunk we cannot
+		 * upgrade (out of budget/prealloc) sits in a beneficial-order
+		 * aligned region that fully fits in the tt: that region could
+		 * have been upgraded with more budget. A chunk below the
+		 * beneficial order is expected for the unaligned tail (the last
+		 * partial beneficial-order region) and for already beneficial-
+		 * order harvests - neither leaves real defrag work, so they must
+		 * not flag the tt for re-queue.
+		 */
+		if (beneficial && order < beneficial &&
+		    round_down(off, bnr) + bnr <= num_pages)
+			*suboptimal = true;
+
 		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
 		if (r)
 			return r;
@@ -951,6 +977,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
+	bool defrag_capped;
 
 	/* Mutated as the loop walks the orders. */
 	unsigned int order;
@@ -962,6 +989,44 @@ struct ttm_pool_alloc_iter {
 	struct page *p;
 };
 
+/*
+ * Defrag move budget exhausted: the upgrade can make no further progress this
+ * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
+ * entry.
+ */
+static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
+{
+	if (!it->defrag_old_tt)
+		return false;
+
+	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+}
+
+/*
+ * Complete the new tt by harvesting the rest of the old tt as-is (borrowing its
+ * pages, including any still below beneficial order). Never fall back to in-lock
+ * reclaim for the shortfall - that would reintroduce the stall we preallocated
+ * to avoid. Only re-queue (flag) the tt when the harvested remainder is
+ * genuinely sub-optimal; if every remaining old chunk was already at beneficial
+ * order the move fully upgraded the object.
+ */
+static int ttm_pool_iter_harvest_rest(struct ttm_pool_alloc_iter *it)
+{
+	pgoff_t off = it->tt->num_pages - it->alloc->remaining_pages;
+	bool suboptimal = false;
+	int r;
+
+	r = ttm_pool_harvest_remaining(it->pool, it->defrag_old_tt, off,
+				       it->alloc, &suboptimal);
+	if (r)
+		return r;
+
+	if (suboptimal)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	return 0;
+}
+
 /*
  * Defrag move: reuse an already beneficial-order chunk from the old tt at this
  * offset rather than reallocating it. Harvest whatever order the old chunk has
@@ -1062,11 +1127,12 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 		if (it->alloc_count && it->defrag_old_tt) {
 			pgoff_t off = it->tt->num_pages -
 				      it->alloc->remaining_pages;
+			bool suboptimal = false;
 			int r;
 
 			r = ttm_pool_harvest_remaining(it->pool,
 						       it->defrag_old_tt, off,
-						       it->alloc);
+						       it->alloc, &suboptimal);
 			if (r)
 				return r;
 
@@ -1121,10 +1187,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;
 
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/*
+		 * Out of defrag budget: harvest the rest of the old tt as-is and
+		 * stop (the tt is re-queued if the remainder is still
+		 * sub-optimal).
+		 */
+		if (ttm_pool_iter_defrag_exhausted(&it)) {
+			r = ttm_pool_iter_harvest_rest(&it);
+			if (r)
+				goto error_free_all;
+			break;
+		}
+
 		/* Reuse an already beneficial-order chunk from the old tt. */
 		r = ttm_pool_iter_reuse_old(&it);
 		if (r < 0)
@@ -1155,6 +1234,15 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				goto error_free_all;
 		}
 
+		/*
+		 * Charge a newly (re)allocated chunk against the defrag move
+		 * budget. Harvested chunks borrow the old tt's pages for free
+		 * and never reach here.
+		 */
+		if (it.defrag_capped)
+			alloc->defrag_bytes_remaining -=
+				(s64)(1UL << it.order) << PAGE_SHIFT;
+
 		it.alloc_count++;
 	}
 
@@ -1204,14 +1292,21 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx)
 {
 	struct ttm_pool_alloc_state alloc;
+	int ret;
 
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
 	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
+	alloc.defrag_bytes_remaining = ctx->defrag_bytes_remaining;
+
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+
+	/* Report the unused defrag budget back to the caller. */
+	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
 
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index fcaba5fb8bf7..6124d2627b47 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -223,6 +223,20 @@ struct ttm_operation_ctx {
 	 * NULL for non-defrag operations.
 	 */
 	struct ttm_tt *defrag_old_tt;
+	/**
+	 * @defrag_bytes_remaining: Byte budget for newly allocated pages during
+	 * a defragmentation move. Only pages that are actually (re)allocated at
+	 * the beneficial order count against it; pages harvested (borrowed) from
+	 * @defrag_old_tt are free and do not. When the budget is exhausted the
+	 * pool allocator stops upgrading and completes the new tt by harvesting
+	 * the remainder of @defrag_old_tt, marking the tt sub-optimal so the
+	 * driver can resume on a later pass. The allocator decrements this as it
+	 * allocates, so on return it holds the unused budget (which the caller
+	 * can use to compute how many bytes were spent). A value of 0 means
+	 * unlimited (upgrade the whole object). Only consulted when
+	 * @defrag_old_tt is set.
+	 */
+	s64 defrag_bytes_remaining;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
-- 
2.34.1

[PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

[Matthew Brost]

Defragmentation moves reallocate a BO's backing at the pool's beneficial
order. On a fragmented system those high-order allocations frequently
stall in direct reclaim/compaction, and they run under the BO dma-resv
lock, holding up concurrent rendering. The defrag worker does not care
how long the work takes; it only cares about minimizing locked time.

Add a preallocation bag so the expensive allocations can be hoisted out
of the lock:

 - struct ttm_pool_prealloc carries beneficial-order pages preallocated
   outside the lock; ttm_pool_prealloc_fill()/_fini() allocate and
   release them. __ttm_pool_alloc() consumes them via ctx->prealloc.
 - Apply the requested cpu-caching to the bag pages during fill, so the
   set_memory_*() cost is paid outside the lock too; consumed pages are
   already cache-consistent and leftovers are restored to write-back on
   free. Bag pages are never highmem so the kernel mapping is always
   valid for set_memory_*().
 - Record the exact number of sub-optimal pages on the tt
   (nr_suboptimal_pages) at populate time so a defrag pass can size its
   prealloc precisely instead of guessing.
 - Once a defrag move exhausts the prealloc bag, harvest the remainder
   of the old tt rather than entering reclaim under the lock, and flag
   the tt sub-optimal so the driver re-queues it for a later pass.

This keeps all reclaim/compaction stalls and cache mode changes out of
the dma-resv critical section: lock-held time then tracks only the bytes
actually copied.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 244 +++++++++++++++++++++++++++++++--
 include/drm/ttm/ttm_bo.h       |  11 ++
 include/drm/ttm/ttm_pool.h     |  29 ++++
 include/drm/ttm/ttm_tt.h       |  17 +++
 4 files changed, 287 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index 1673e81b8df8..da7d132c8297 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -82,6 +82,9 @@ struct ttm_pool_dma {
  * Pages harvested (borrowed) from the old tt are free and do not count; once
  * this drops to zero the allocator stops upgrading and harvests the remainder.
  * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
+ * @nr_suboptimal_pages: Number of pages backed below the pool's beneficial
+ * order. Recorded by the pool allocator after populate; a defrag pass reads it
+ * to size its out-of-lock preallocation exactly.
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -90,6 +93,7 @@ struct ttm_pool_alloc_state {
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
 	s64 defrag_bytes_remaining;
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -787,6 +791,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
 	alloc->defrag_bytes_remaining = 0;
+	alloc->nr_suboptimal_pages = 0;
 }
 
 /*
@@ -852,7 +857,8 @@ static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
  * pages stay owned by the old tt until the move commits.
  */
 static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
-				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+				 unsigned int beneficial_order, pgoff_t off,
+				 struct ttm_pool_alloc_state *alloc)
 {
 	pgoff_t nr = 1UL << order;
 	int r;
@@ -887,6 +893,9 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 	alloc->caching_divide = alloc->pages;
 
+	if (order < beneficial_order)
+		alloc->nr_suboptimal_pages += 0x1 << order;
+
 	return 0;
 }
 
@@ -934,7 +943,8 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		    round_down(off, bnr) + bnr <= num_pages)
 			*suboptimal = true;
 
-		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		r = ttm_pool_harvest_page(old_tt, order, beneficial, off,
+					  alloc);
 		if (r)
 			return r;
 
@@ -944,6 +954,173 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}
+
+/*
+ * Apply a tt's cpu-caching to a batch of freshly system-allocated (write-back)
+ * prealloc pages in one shot. @cpages is an unpacked array of @ncpages
+ * individual 4K pages (the constituent pages of the mixed-order chunks, expanded
+ * so set_pages_array_*() sees one entry per page), since the packed prealloc bag
+ * holds multi-order chunks. Prealloc pages are never highmem (see
+ * ttm_pool_prealloc_gfp), so their kernel mapping is valid. No-op for cached
+ * pages and on non-x86. Returns non-zero if the caching change failed (the
+ * caller drops the whole bag and falls back to in-line allocation).
+ */
+static int ttm_pool_prealloc_apply_caching(enum ttm_caching caching,
+					   struct page **cpages,
+					   unsigned int ncpages)
+{
+#ifdef CONFIG_X86
+	if (!ncpages)
+		return 0;
+
+	switch (caching) {
+	case ttm_cached:
+		break;
+	case ttm_write_combined:
+		return set_pages_array_wc(cpages, ncpages);
+	case ttm_uncached:
+		return set_pages_array_uc(cpages, ncpages);
+	}
+#endif
+	return 0;
+}
+
+/* Expand an @order chunk into its constituent 4K pages for bulk caching. */
+static void ttm_pool_prealloc_stage_caching(struct page **cpages,
+					    unsigned int *ncpages,
+					    struct page *p, unsigned int order)
+{
+	unsigned int i, nr = 1u << order;
+
+	for (i = 0; i < nr; i++)
+		cpages[(*ncpages)++] = p + i;
+}
+
+/**
+ * ttm_pool_prealloc_fill() - Preallocate beneficial-order pages outside any lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; @pp->order is set to the beneficial order.
+ * @count: Number of beneficial-order chunks to attempt.
+ *
+ * Allocate up to @count beneficial-order chunks, parking them in @pp for a
+ * later __ttm_pool_alloc() defrag move to consume under the dma-resv lock. May
+ * sleep/reclaim freely as it runs unlocked. A short fill is fine: the pool
+ * falls back to in-line allocation for the shortfall. DMA-alloc pools are not
+ * supported (count stays 0). Returns 0 (release with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count)
+{
+	unsigned int order = ttm_pool_beneficial_order(pool);
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool);
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = order;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+
+	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
+	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	pp->pages = kvzalloc_objs(*pp->pages, count);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Every chunk is a fresh write-back system page of @order that needs a
+	 * caching change; collect their constituent pages into an unpacked
+	 * scratch array and issue a single set_pages_array_*() after the fill.
+	 * Cached tts need no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, (size_t)count << order);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (pp->count < count) {
+		struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);
+
+		if (!p)
+			break;
+
+		pp->pages[pp->count++] = p;
+		if (cpages)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+	}
+
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++)
+			ttm_pool_free_page(pool, tt_caching, order,
+					   pp->pages[i], false);
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill);
+
+/**
+ * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
+ * @pool: The pool the pages came from.
+ * @pp: Prealloc bag to drain. Consumed pages (< @used) are now owned by the tt.
+ */
+void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
+{
+	unsigned int i;
+
+	for (i = pp->used; i < pp->count; ++i)
+		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
+				   false);
+	kvfree(pp->pages);
+	pp->pages = NULL;
+	pp->count = 0;
+	pp->used = 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fini);
+
 /**
  * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
  * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
@@ -975,6 +1152,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
 	struct ttm_tt *defrag_old_tt;
+	struct ttm_pool_prealloc *prealloc;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 	bool defrag_capped;
@@ -990,16 +1168,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
  */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;
 }
 
 /*
@@ -1056,7 +1238,8 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 		return TTM_POOL_ITER_FILL;
 
 	it->order = harvest_order;
-	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order,
+				  it->beneficial_order, off, it->alloc);
 	if (r)
 		return r;
 
@@ -1068,11 +1251,13 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation. Fault
- * injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a beneficial-order page preallocated outside the
+ * dma-resv lock (defrag), a same-order pool page, then a fresh system
+ * allocation. Fault injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
+	struct ttm_pool_prealloc *pp = it->prealloc;
 	struct ttm_pool_type *pt;
 
 	it->p = NULL;
@@ -1087,7 +1272,18 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 		it->beneficial_order && it->order >= it->beneficial_order &&
 		should_fail(&beneficial_order_fault_inject, 1);
 
-	/* First, try to allocate a page from a pool if one exists. */
+	/*
+	 * Defrag move: consume a beneficial-order page preallocated outside the
+	 * dma-resv lock instead of allocating one in-line (which may stall in
+	 * reclaim/compaction). Falls through once the bag is empty.
+	 */
+	if (pp && !it->fail_beneficial && it->order >= it->beneficial_order &&
+	    pp->used < pp->count) {
+		it->order = it->beneficial_order;
+		it->p = pp->pages[pp->used++];
+		it->page_caching = it->tt->caching;
+	}
+
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
 	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
@@ -1164,6 +1360,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.alloc = alloc,
 		.restore = restore,
 		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
+		.prealloc = restore ? NULL : ctx->prealloc,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -1193,8 +1390,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {
@@ -1211,7 +1408,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (r == TTM_POOL_ITER_RETRY)
 			continue;
 
-		/* Acquire a page (pool / system) for this order. */
+		/* Acquire a page (prealloc / pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
@@ -1243,6 +1440,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			alloc->defrag_bytes_remaining -=
 				(s64)(1UL << it.order) << PAGE_SHIFT;
 
+		if (it.order < it.beneficial_order)
+			alloc->nr_suboptimal_pages += 0x1 << it.order;
 		it.alloc_count++;
 	}
 
@@ -1306,6 +1505,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	/* Report the unused defrag budget back to the caller. */
 	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
 	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
@@ -1358,6 +1560,15 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 			if (ret)
 				return ret;
+
+			/*
+			 * __ttm_pool_alloc() counts each freshly (re)allocated
+			 * chunk against nr_suboptimal_pages, but a chunk whose
+			 * backup copy was interrupted and is finished here
+			 * resumes past that loop, so account for it directly.
+			 */
+			if (restore->order < ttm_pool_beneficial_order(pool))
+				alloc.nr_suboptimal_pages += 1U << restore->order;
 		}
 		if (!alloc.remaining_pages) {
 			ret = ttm_pool_apply_caching(&alloc);
@@ -1367,11 +1578,16 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			kfree(tt->restore);
 			tt->restore = NULL;
 
+			tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
 			return 0;
 		}
 	}
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
+	return ret;
 }
 
 /**
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index 6124d2627b47..a80304f179ba 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -237,6 +237,17 @@ struct ttm_operation_ctx {
 	 * @defrag_old_tt is set.
 	 */
 	s64 defrag_bytes_remaining;
+	/**
+	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
+	 * move. The pool allocator consumes these instead of allocating fresh
+	 * beneficial-order pages under the lock, moving the (potentially
+	 * reclaim/compaction stalling) high-order allocations out of the
+	 * critical section. NULL means allocate in-line as usual. The pool
+	 * silently falls back to in-line allocation for any shortfall. The
+	 * allocator consumes it for beneficial-order chunks of any populate;
+	 * drivers only set it for a defrag move.
+	 */
+	struct ttm_pool_prealloc *prealloc;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
index 753203980e2c..71670350eb15 100644
--- a/include/drm/ttm/ttm_pool.h
+++ b/include/drm/ttm/ttm_pool.h
@@ -81,6 +81,35 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx);
 void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
+/**
+ * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
+ * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
+ *         short); each entry is the head page of a 1 << @order chunk.
+ * @order: Page order of every preallocated chunk.
+ * @caching: CPU caching applied to the pages, so leftovers can be restored
+ *           to write-back before being freed.
+ * @count: Number of valid entries in @pages.
+ * @used: Number of entries already consumed by the pool allocator.
+ *
+ * Defrag pages are interchangeable, so only a count of beneficial-order chunks
+ * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
+ * __ttm_pool_alloc() drains it; any unused tail is released by
+ * ttm_pool_prealloc_fini().
+ */
+struct ttm_pool_prealloc {
+	struct page **pages;
+	unsigned int order;
+	enum ttm_caching caching;
+	unsigned int count;
+	unsigned int used;
+};
+
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count);
+void ttm_pool_prealloc_fini(struct ttm_pool *pool,
+			    struct ttm_pool_prealloc *pp);
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
+
 void ttm_pool_init(struct ttm_pool *pool, struct device *dev,
 		   int nid, unsigned int alloc_flags);
 void ttm_pool_fini(struct ttm_pool *pool);
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 55c35bcf134d..81a5446ca693 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -135,6 +135,12 @@ struct ttm_tt {
 	enum ttm_caching caching;
 	/** @restore: Partial restoration from backup state. TTM private */
 	struct ttm_pool_tt_restore *restore;
+	/**
+	 * @nr_suboptimal_pages: Number of pages backed below the pool's
+	 * beneficial order. Recorded by the pool allocator after populate; a
+	 * defrag pass reads it to size its out-of-lock preallocation exactly.
+	 */
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -201,6 +207,17 @@ static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 }
 
+/**
+ * ttm_tt_suboptimal_pages() - Pages backed below the pool's beneficial order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: Number of pages a defrag move would (re)allocate at beneficial order.
+ */
+static inline u32 ttm_tt_suboptimal_pages(const struct ttm_tt *tt)
+{
+	return tt->nr_suboptimal_pages;
+}
+
 /**
  * ttm_tt_create
  *
-- 
2.34.1

[PATCH v3 10/33] drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()

[Matthew Brost]

ttm_pool_prealloc_fill() lets a driver preallocate beneficial-order pages
outside the dma-resv lock for a defragmentation move, so the high-order
allocations that may stall in reclaim/compaction happen off the critical
section.  That bag is deliberately partial: it holds only interchangeable
beneficial-order chunks and the allocator consumes it opportunistically,
falling back to in-line allocation for everything else.

An ordinary populate of a system-memory buffer object has the same
problem - the whole backing is allocated under the dma-resv lock - but no
way to move it out of the critical section, because the existing bag does
not cover sub-beneficial orders and does not describe a full tt.

Add a "full" flavour of the same preallocation mechanism:

  - struct ttm_pool_prealloc gains a @full flag.  In full mode the bag is
    an ordered tiling of an entire tt at mixed orders (pool pages
    included), rather than a set of interchangeable beneficial-order
    chunks.  The per-chunk order is recorded on the page itself and read
    back at consume time, so no parallel order array is needed.

  - ttm_pool_prealloc_fill_full() allocates the whole backing for a tt of
    @num_pages pages up front, mirroring the in-line allocator's order
    walk (highest order first, capped by the remaining page count).  It
    reuses the existing unlocked helpers (ttm_pool_prealloc_gfp(),
    ttm_pool_set_caching()) via a new ttm_pool_take_page() helper that
    tries a same-order pool page before a fresh system allocation.

  - ttm_pool_prealloc_fini() frees any unconsumed tail at each chunk's own
    order when @full.

  - ttm_pool_prealloc_fill_full() takes a @beneficial_reclaim_backoff
    argument, plumbed to ttm_pool_alloc_page(), that lets the caller
    allocate higher-order chunks without aggressive reclaim/compaction.
    This suits a driver that upgrades suboptimal pages to beneficial order
    in place later (e.g. via a background defragmenter) and so has no need
    to stall the user context reclaiming for contiguity up front.

  - __ttm_pool_alloc() consumes a full bag with a single early branch in
    ttm_pool_iter_acquire_page(): install the next chunk as-is (order
    taken from the page) and return; once the best-effort bag is drained
    the loop falls through to normal in-line allocation.  The defrag
    harvest and reuse-old phases are inert here as there is no
    defrag_old_tt.

Like the defrag bag, this is best-effort: a short fill is fine (the pool
allocates the shortfall in-line) and DMA-alloc pools are unsupported
(caching is applied on the kernel mapping).  No functional change for
existing users; the new fill routine has no in-tree caller yet.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/ttm/ttm_pool.c | 232 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  19 +--
 include/drm/ttm/ttm_pool.h     |  27 +++-
 3 files changed, 258 insertions(+), 20 deletions(-)

diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
index da7d132c8297..4746f5d8cd7e 100644
--- a/drivers/gpu/drm/ttm/ttm_pool.c
+++ b/drivers/gpu/drm/ttm/ttm_pool.c
@@ -1046,6 +1046,7 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 	pp->caching = tt_caching;
 	pp->count = 0;
 	pp->used = 0;
+	pp->full = false;
 
 	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
 	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
@@ -1102,6 +1103,187 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 }
 EXPORT_SYMBOL(ttm_pool_prealloc_fill);
 
+/*
+ * Acquire a single *@order chunk, trying a same-order pool page first (already
+ * carries @caching) then a fresh system allocation (write-back). Runs unlocked.
+ * When @beneficial_reclaim_backoff is set, higher-order system allocations skip
+ * aggressive reclaim/compaction (see ttm_pool_alloc_page()).
+ *
+ * A fresh system page is write-back and needs the tt's caching applied later in
+ * bulk; *@need_caching is set true for it and false for a (pre-cached) pool
+ * page.
+ *
+ * On a genuine failure *@order is left untouched (the caller lowers it by one
+ * and retries). Fault injection instead pretends allocation at (or above) the
+ * beneficial order failed and lowers *@order to the beneficial order, so the
+ * caller's next attempt lands below it - mirroring the in-line acquire path's
+ * fail_beneficial jump and exercising the beneficial_order_failed tracking
+ * without driving the system into real fragmentation.
+ *
+ * Returns NULL if no page of *@order could be obtained.
+ */
+static struct page *ttm_pool_take_page(struct ttm_pool *pool,
+				       enum ttm_caching caching,
+				       unsigned int *order, gfp_t gfp,
+				       bool *need_caching,
+				       bool beneficial_reclaim_backoff)
+{
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	struct ttm_pool_type *pt;
+	struct page *p;
+
+	*need_caching = false;
+
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && beneficial &&
+	    *order >= beneficial &&
+	    should_fail(&beneficial_order_fault_inject, 1)) {
+		*order = beneficial;
+		return NULL;
+	}
+
+	pt = ttm_pool_select_type(pool, caching, *order);
+	if (pt) {
+		p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
+		if (p)
+			return p;	/* pool page already carries @caching */
+	}
+
+	p = ttm_pool_alloc_page(pool, gfp, *order, beneficial_reclaim_backoff);
+	if (!p)
+		return NULL;
+
+	/* Fresh system page is write-back; caller applies @caching in bulk. */
+	*need_caching = true;
+	return p;
+}
+
+/**
+ * ttm_pool_prealloc_fill_full() - Preallocate a whole tt's backing outside any
+ * lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; marked @full and tiled with mixed-order chunks.
+ * @num_pages: Number of 4K pages the tt needs backed.
+ *
+ * Unlike ttm_pool_prealloc_fill() (which parks a bag of interchangeable
+ * beneficial-order defrag chunks), this allocates the ENTIRE backing for a tt
+ * of @num_pages pages up front, at whatever mix of orders is available (pool
+ * pages included), so that the subsequent __ttm_pool_alloc() under the dma-resv
+ * lock merely installs the pages without stalling in reclaim/compaction. The
+ * chunk orders mirror the in-line allocator's order walk; each chunk's order is
+ * recorded on the page itself and read back at consume time.
+ *
+ * May sleep/reclaim freely as it runs unlocked. When
+ * @beneficial_reclaim_backoff is set, higher-order chunks are taken without
+ * aggressive reclaim/compaction: the caller relies on a background
+ * defragmenter to upgrade to beneficial-order pages in place later, so there
+ * is no need to stall the user context reclaiming for contiguity here. A short
+ * fill is fine: the pool falls back to in-line allocation for the shortfall.
+ * DMA-alloc pools are not supported (the bag stays empty). Returns 0 (release
+ * with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff)
+{
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool) | __GFP_ZERO |
+		__GFP_RETRY_MAYFAIL | __GFP_NOWARN;
+	unsigned int highest = MAX_PAGE_ORDER;
+	unsigned int remaining = num_pages;
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = 0;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+	pp->full = true;
+
+	/* Caching is applied on the kernel mapping, so DMA-alloc is unsupported. */
+	if (!num_pages || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	/* Worst case is one order-0 chunk per page. */
+	pp->pages = kvzalloc_objs(*pp->pages, num_pages);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Fresh write-back chunks get their caching changed in a single
+	 * set_pages_array_*() after the fill; collect their constituent pages
+	 * into an unpacked scratch array (one entry per 4K page) since the
+	 * packed bag holds mixed-order chunks. Cached tts and pool pages need
+	 * no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, num_pages);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (remaining) {
+		unsigned int order = min_t(unsigned int, highest,
+					   __fls(remaining));
+		bool need_caching;
+		struct page *p;
+
+		for (;;) {
+			p = ttm_pool_take_page(pool, tt_caching, &order, gfp,
+					       &need_caching,
+					       beneficial_reclaim_backoff);
+			if (p)
+				break;
+			if (!order)
+				goto apply_caching;	/* short fill */
+			/*
+			 * Genuine failure: retry one order down. A fault
+			 * injection lowered @order to the beneficial order
+			 * above, so this drops it below - landing on a
+			 * sub-optimal (defrag-worthy) chunk.
+			 */
+			order--;
+		}
+
+		pp->pages[pp->count++] = p;
+		if (cpages && need_caching)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+		remaining -= 1u << order;
+		highest = order;
+	}
+
+apply_caching:
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++) {
+			struct page *p = pp->pages[i];
+
+			ttm_pool_free_page(pool, tt_caching,
+					   ttm_pool_page_order_nodma(p), p,
+					   false);
+		}
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill_full);
+
 /**
  * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
  * @pool: The pool the pages came from.
@@ -1111,9 +1293,13 @@ void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
 {
 	unsigned int i;
 
-	for (i = pp->used; i < pp->count; ++i)
-		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
-				   false);
+	for (i = pp->used; i < pp->count; ++i) {
+		struct page *p = pp->pages[i];
+		unsigned int order = pp->full ?
+			ttm_pool_page_order_nodma(p) : pp->order;
+
+		ttm_pool_free_page(pool, pp->caching, order, p, false);
+	}
 	kvfree(pp->pages);
 	pp->pages = NULL;
 	pp->count = 0;
@@ -1251,9 +1437,10 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries, in order: a beneficial-order page preallocated outside the
- * dma-resv lock (defrag), a same-order pool page, then a fresh system
- * allocation. Fault injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a chunk from a full out-of-lock prealloc (populate
+ * of the whole tt), a beneficial-order page preallocated outside the dma-resv
+ * lock (defrag), a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
@@ -1262,6 +1449,39 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 
 	it->p = NULL;
 
+	/*
+	 * Full prealloc: the entire backing was allocated (pool + system, at
+	 * mixed orders) outside the dma-resv lock. Install the next chunk as-is,
+	 * taking its order from the page. Falls through to in-line allocation
+	 * once the best-effort bag is drained.
+	 */
+	if (pp && pp->full && pp->used < pp->count) {
+		it->fail_beneficial = false;
+		it->p = pp->pages[pp->used++];
+		it->order = ttm_pool_page_order_nodma(it->p);
+		it->page_caching = it->tt->caching;
+
+		/*
+		 * The full prealloc tiles at whatever orders were available
+		 * without stalling in reclaim. A sub-beneficial chunk only
+		 * means a sub-optimal backing (defrag-worthy) when it sits in a
+		 * beneficial-order aligned region that fully fits in the tt -
+		 * that region could have been a beneficial-order page. A low
+		 * order in the unaligned trailing tail is expected and must not
+		 * flag the tt (mirrors ttm_pool_harvest_remaining()).
+		 */
+		if (it->beneficial_order && it->order < it->beneficial_order) {
+			pgoff_t bnr = 1UL << it->beneficial_order;
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+
+			if (round_down(off, bnr) + bnr <= it->tt->num_pages)
+				it->tt->page_flags |=
+					TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		}
+		return;
+	}
+
 	/*
 	 * Fault injection: pretend allocation at (or above) the device's
 	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
diff --git a/include/drm/ttm/ttm_bo.h b/include/drm/ttm/ttm_bo.h
index a80304f179ba..52ec7e5c0d2d 100644
--- a/include/drm/ttm/ttm_bo.h
+++ b/include/drm/ttm/ttm_bo.h
@@ -238,14 +238,17 @@ struct ttm_operation_ctx {
 	 */
 	s64 defrag_bytes_remaining;
 	/**
-	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
-	 * move. The pool allocator consumes these instead of allocating fresh
-	 * beneficial-order pages under the lock, moving the (potentially
-	 * reclaim/compaction stalling) high-order allocations out of the
-	 * critical section. NULL means allocate in-line as usual. The pool
-	 * silently falls back to in-line allocation for any shortfall. The
-	 * allocator consumes it for beneficial-order chunks of any populate;
-	 * drivers only set it for a defrag move.
+	 * @prealloc: Pages preallocated outside the dma-resv lock. The pool
+	 * allocator consumes these instead of allocating under the lock, moving
+	 * the (potentially reclaim/compaction stalling) allocations out of the
+	 * critical section. NULL means allocate in-line as usual. Two flavours,
+	 * distinguished by ttm_pool_prealloc::full:
+	 *   - defrag bag (full=false): a bag of interchangeable beneficial-order
+	 *     chunks consumed only for beneficial-order chunks of a defrag move.
+	 *   - full tiling (full=true): the entire backing of the tt, at mixed
+	 *     orders, consumed for every chunk of an ordinary populate.
+	 * The allocator silently falls back to in-line allocation for any
+	 * shortfall.
 	 */
 	struct ttm_pool_prealloc *prealloc;
 	/**
diff --git a/include/drm/ttm/ttm_pool.h b/include/drm/ttm/ttm_pool.h
index 71670350eb15..777d5aa08f6a 100644
--- a/include/drm/ttm/ttm_pool.h
+++ b/include/drm/ttm/ttm_pool.h
@@ -83,18 +83,27 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
 /**
  * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
- * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
- *         short); each entry is the head page of a 1 << @order chunk.
- * @order: Page order of every preallocated chunk.
+ * @pages: Array of @count preallocated chunks; each entry is the head page of a
+ *         chunk. In defrag mode every chunk is @order pages; in @full mode the
+ *         chunks tile a whole tt at mixed orders and the per-chunk order is read
+ *         back from the page itself.
+ * @order: Page order of every preallocated chunk (defrag mode only; 0 in @full
+ *         mode where each chunk carries its own order).
  * @caching: CPU caching applied to the pages, so leftovers can be restored
  *           to write-back before being freed.
  * @count: Number of valid entries in @pages.
  * @used: Number of entries already consumed by the pool allocator.
+ * @full: When set the bag holds a full, mixed-order tiling of an entire tt
+ *        (produced by ttm_pool_prealloc_fill_full()) rather than a bag of
+ *        interchangeable beneficial-order defrag chunks. The pool allocator
+ *        then drains the bag in order for every populate, taking each chunk's
+ *        order from the page, and falls back to in-line allocation only for any
+ *        shortfall.
  *
  * Defrag pages are interchangeable, so only a count of beneficial-order chunks
- * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
- * __ttm_pool_alloc() drains it; any unused tail is released by
- * ttm_pool_prealloc_fini().
+ * is needed. ttm_pool_prealloc_fill() / ttm_pool_prealloc_fill_full() populate
+ * this outside the lock and __ttm_pool_alloc() drains it; any unused tail is
+ * released by ttm_pool_prealloc_fini().
  */
 struct ttm_pool_prealloc {
 	struct page **pages;
@@ -102,10 +111,16 @@ struct ttm_pool_prealloc {
 	enum ttm_caching caching;
 	unsigned int count;
 	unsigned int used;
+	bool full;
 };
 
 int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 			   struct ttm_pool_prealloc *pp, unsigned int count);
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff);
 void ttm_pool_prealloc_fini(struct ttm_pool *pool,
 			    struct ttm_pool_prealloc *pp);
 unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
-- 
2.34.1

[PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

[Matthew Brost]

From: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Drivers that use the GPU SVM core to DMA-map system memory for GPU
access may want to keep track of how many pages they have mapped, for
example to expose statistics or to detect leaks. Doing this accounting
in the driver around drm_gpusvm_get_pages() / drm_gpusvm_unmap_pages()
is error prone: the number, order and kind of the dma_addr[] entries can
change between map and unmap (migration, partial unmaps, the iova vs
non-iova paths), which makes symmetric accounting hard to get right.

Add an optional @dma_map_account callback to struct drm_gpusvm_ops and
invoke it once per &drm_pagemap_addr entry at the exact points where the
entry is DMA-mapped (sign == +1) in drm_gpusvm_get_pages() and unmapped
(sign == -1) in __drm_gpusvm_unmap_pages(). Since the map error path and
every unmap/free path funnel through __drm_gpusvm_unmap_pages() for the
entries that were actually mapped, the accounting is symmetric by
construction. The callback receives the full &drm_pagemap_addr so the
driver can use the order and the proto field to distinguish system
memory mappings from device interconnect mappings.

The callback must also be usable by the core drm_gpusvm_pages-only API
(mm == NULL), as used e.g. for userptr-style mappings. Relax
drm_gpusvm_init() to allow such users to pass a restricted @ops that
only implements the page-level hooks; the full-SVM hooks (@invalidate)
and the chunk configuration must still be unset in that mode.

Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
 drivers/gpu/drm/drm_gpusvm.c | 17 +++++++++++++++--
 include/drm/drm_gpusvm.h     | 19 +++++++++++++++++++
 2 files changed, 34 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
index fcfe635bc195..512f52471419 100644
--- a/drivers/gpu/drm/drm_gpusvm.c
+++ b/drivers/gpu/drm/drm_gpusvm.c
@@ -474,8 +474,15 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
 			return -EINVAL;
 		mmgrab(mm);
 	} else {
-		/* No full SVM mode, only core drm_gpusvm_pages API. */
-		if (ops || num_chunks || mm_range || notifier_size)
+		/*
+		 * No full SVM mode, only core drm_gpusvm_pages API. A
+		 * restricted @ops that only implements the page-level hooks
+		 * (e.g. @dma_map_account) is allowed; the full-SVM hooks must
+		 * not be set.
+		 */
+		if (num_chunks || mm_range || notifier_size)
+			return -EINVAL;
+		if (ops && ops->invalidate)
 			return -EINVAL;
 	}
 
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
 			i += 1 << addr->order;
 		}
 
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
diff --git a/include/drm/drm_gpusvm.h b/include/drm/drm_gpusvm.h
index b7d987bf76aa..36f35816a569 100644
--- a/include/drm/drm_gpusvm.h
+++ b/include/drm/drm_gpusvm.h
@@ -75,6 +75,25 @@ struct drm_gpusvm_ops {
 	void (*invalidate)(struct drm_gpusvm *gpusvm,
 			   struct drm_gpusvm_notifier *notifier,
 			   const struct mmu_notifier_range *mmu_range);
+
+	/**
+	 * @dma_map_account: Account a DMA-mapping change (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @addr: The address descriptor of the chunk being (un)mapped
+	 * @sign: +1 when @addr has just been DMA-mapped, -1 when it is being
+	 *        unmapped
+	 *
+	 * Called once per &drm_pagemap_addr entry, when the entry is
+	 * DMA-mapped by drm_gpusvm_get_pages() (@sign == +1) and when it is
+	 * unmapped (@sign == -1), so the driver can keep symmetric accounting
+	 * of the pages it has mapped for GPU access. The @addr->proto field
+	 * can be used to distinguish system-memory mappings from device
+	 * interconnect mappings. May be provided in both full SVM mode and the
+	 * core drm_gpusvm_pages-only mode.
+	 */
+	void (*dma_map_account)(struct drm_gpusvm *gpusvm,
+				const struct drm_pagemap_addr *addr,
+				int sign);
 };
 
 /**
-- 
2.34.1

[PATCH v3 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

[Matthew Brost]

From: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Expose per-page-order DMA mapping counts for the system memory that the
xe driver maps for GPU access, split into two categories: TTM buffer
objects, and the GPU SVM / userptr ranges (which share a single
drm_gpusvm instance per VM).

The stats are visible at:

  <debugfs>/dri/<N>/dma_mapped_pages

and are broken out into two rows (bo, svm/userptr), one column per page
order, mirroring the layout of the TTM pool page_pool stat.

For TTM BOs the per-order counts are derived from the DMA scatter-gather
table: each mapped segment is physically contiguous, so its page count is
split into the largest power-of-two runs (a segment of N pages contributes
a 2^k chunk for the largest k with 2^k <= N, repeated for the remainder).
Driving the accounting off the sg table rather than the pages[] array keeps
it correct even when pages[] entries are cleared while still DMA-mapped,
as happens on the defrag page-borrowing path.

For the SVM and userptr ranges the accounting is driven by the
drm_gpusvm @dma_map_account callback, which fires once per dma_addr[]
entry at the exact point the entry is DMA-mapped and unmapped. This
makes the accounting symmetric by construction and avoids driver-side
walks that could drift across migration, partial unmaps and the iova vs
non-iova paths. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps
and counted; device interconnect (VRAM, P2P) entries are skipped, so the
counter reflects only system pages actually mapped through the DMA layer.

The callback is shared between the full SVM instance (fault mode) and the
core drm_gpusvm_pages-only instance used for userptr, so userptr
mappings are accounted in both the CONFIG_DRM_XE_GPUSVM=y and =n (but
CONFIG_DRM_GPUSVM=y) configurations, into the same svm/userptr counter.

Since the counts are only ever consumed through debugfs, gate the counter
storage and all accounting on CONFIG_DEBUG_FS so that builds without
debugfs carry no extra atomic traffic or device state; the drm_gpusvm
callback is simply left unregistered in that case.

On device teardown assert that every counter has returned to zero, to
catch any unbalanced accounting (a leaked DMA mapping) during testing.

Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

---

v3:
 - Fix drm_gpusvm_init() argument order, clamp DMA-page order to
   NR_PAGE_ORDERS - 1, and drain DMA-page accounting after
   xe_bo_dev_fini() (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c           | 46 ++++++++++++++++++++
 drivers/gpu/drm/xe/xe_debugfs.c      | 26 ++++++++++++
 drivers/gpu/drm/xe/xe_device.c       | 30 +++++++++++++
 drivers/gpu/drm/xe/xe_device_types.h | 17 ++++++++
 drivers/gpu/drm/xe/xe_svm.c          | 44 ++++++++++++++++++-
 drivers/gpu/drm/xe/xe_svm.h          |  4 +-
 drivers/gpu/drm/xe/xe_userptr.c      | 63 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_userptr.h      |  1 +
 8 files changed, 228 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 85e6d9a0f575..60e0e568aa31 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -367,6 +367,50 @@ struct xe_ttm_tt {
 	bool purgeable;
 };
 
+/*
+ * xe_tt_account_dma_pages - account DMA-mapped tt pages per order
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt whose sg table to walk
+ * @sign: +1 to add, -1 to subtract
+ *
+ * Walk the tt's DMA scatter-gather table and split each (physically
+ * contiguous) segment into the largest power-of-two page runs: a segment of N
+ * pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for
+ * the remainder. Accumulate the count of 2^k pages into
+ * xe->mem.dma_mapped_pages[k]. Driving this off the sg table rather than
+ * tt->pages keeps the accounting correct even when pages[] entries are cleared
+ * while still DMA-mapped (e.g. the defrag page-borrowing path).
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_sg(xe_tt->sg, sg, i) {
+		unsigned long nr_pages = sg->length >> PAGE_SHIFT;
+
+		while (nr_pages) {
+			unsigned int order = min_t(unsigned int,
+						   __fls(nr_pages),
+						   MAX_PAGE_ORDER);
+			unsigned long chunk = 1UL << order;
+
+			atomic_long_add((long)chunk * sign,
+					&xe->mem.dma_mapped_pages[order]);
+
+			nr_pages -= chunk;
+		}
+	}
+}
+#else
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -396,6 +440,7 @@ static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 		return ret;
 	}
 
+	xe_tt_account_dma_pages(xe, xe_tt, 1);
 	return 0;
 }
 
@@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
+		xe_tt_account_dma_pages(xe, xe_tt, -1);
 		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);
diff --git a/drivers/gpu/drm/xe/xe_debugfs.c b/drivers/gpu/drm/xe/xe_debugfs.c
index 8c391c7b017a..97b47c5c4213 100644
--- a/drivers/gpu/drm/xe/xe_debugfs.c
+++ b/drivers/gpu/drm/xe/xe_debugfs.c
@@ -6,6 +6,7 @@
 #include "xe_debugfs.h"
 
 #include <linux/debugfs.h>
+#include <linux/mmzone.h>
 #include <linux/fault-inject.h>
 #include <linux/string_helpers.h>
 
@@ -231,6 +232,28 @@ static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
 	return 0;
 }
 
+static int dma_mapped_pages_show(struct seq_file *m, void *data)
+{
+	struct xe_device *xe = m->private;
+	unsigned int i;
+
+	seq_printf(m, "%-11s ", "");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " ---%2u---", i);
+	seq_printf(m, "\n%-11s:", "bo");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+	seq_printf(m, "\n%-11s:", "svm/userptr");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	seq_puts(m, "\n");
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(dma_mapped_pages);
+
 static const struct drm_info_list debugfs_list[] = {
 	{"info", info, 0},
 	{ .name = "sriov_info", .show = sriov_info, },
@@ -632,6 +655,9 @@ void xe_debugfs_register(struct xe_device *xe)
 	if (man)
 		ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
 
+	debugfs_create_file("dma_mapped_pages", 0444, root, xe,
+			    &dma_mapped_pages_fops);
+
 	for_each_tile(tile, xe, tile_id)
 		xe_tile_debugfs_register(tile);
 
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index ad7f3e61d457..97cc2d185e45 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -455,12 +455,42 @@ bool xe_device_is_admin_only(const struct xe_device *xe)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+	unsigned int i;
+
+	/*
+	 * All BOs, userptr VMAs and SVM ranges must have been torn down by the
+	 * time the device is destroyed, so every DMA-mapped-pages counter must
+	 * have returned to zero. A non-zero value indicates unbalanced
+	 * accounting, i.e. a missing unmap-side decrement.
+	 */
+	for (i = 0; i < NR_PAGE_ORDERS; i++) {
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	}
+}
+#else
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+}
+#endif
+
 static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
 	xe_bo_dev_fini(&xe->bo_device);
 
+	/*
+	 * Assert the DMA-mapped-pages accounting only after xe_bo_dev_fini()
+	 * has flushed the async BO-free worker: BOs pending async free still
+	 * hold their DMA mappings (and counts) until the worker runs, so
+	 * asserting before the flush could trip a false positive.
+	 */
+	xe_device_assert_dma_pages_zero(xe);
+
 	if (xe->preempt_fence_wq)
 		destroy_workqueue(xe->preempt_fence_wq);
 
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 022e08205897..95e7cea91cbf 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -6,6 +6,7 @@
 #ifndef _XE_DEVICE_TYPES_H_
 #define _XE_DEVICE_TYPES_H_
 
+#include <linux/mmzone.h>
 #include <linux/pci.h>
 
 #include <drm/drm_device.h>
@@ -279,6 +280,22 @@ struct xe_device {
 		struct xe_shrinker *shrinker;
 		/** @mem.stolen_mgr: stolen memory manager. */
 		struct xe_ttm_stolen_mgr *stolen_mgr;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+		/**
+		 * @mem.dma_mapped_pages: number of DMA-mapped pages per page
+		 * order currently live for this device, for TTM BOs. Only
+		 * accounted when CONFIG_DEBUG_FS is enabled, since it is solely
+		 * exposed through debugfs.
+		 */
+		atomic_long_t dma_mapped_pages[NR_PAGE_ORDERS];
+		/**
+		 * @mem.dma_mapped_pages_svm: number of DMA-mapped pages per
+		 * page order currently live for this device, for SVM ranges
+		 * and userptr VMAs (both share a single drm_gpusvm instance).
+		 * Only accounted when CONFIG_DEBUG_FS is enabled.
+		 */
+		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
+#endif
 	} mem;
 
 	/** @sriov: device level virtualization data */
diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c
index b228a737cfd6..ebe48cb76ad9 100644
--- a/drivers/gpu/drm/xe/xe_svm.c
+++ b/drivers/gpu/drm/xe/xe_svm.c
@@ -827,10 +827,50 @@ static int xe_svm_get_pagemaps(struct xe_vm *vm)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_svm_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Maintain per-order counts of the system-memory pages that the full SVM
+ * instance (SVM ranges, and userptr VMAs in fault mode) has DMA-mapped for GPU
+ * access. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_svm_dma_map_account(struct drm_gpusvm *gpusvm,
+				   const struct drm_pagemap_addr *addr,
+				   int sign)
+{
+	struct xe_device *xe = gpusvm_to_vm(gpusvm)->xe;
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	/*
+	 * The CPU-PTE map order reported by hmm can exceed MAX_PAGE_ORDER
+	 * (e.g. an order-18 1GB mapping), so clamp the array index to avoid an
+	 * out-of-bounds access. Clamping is symmetric across the +1/-1 calls,
+	 * so the per-order counters stay balanced.
+	 */
+	order = min(order, (unsigned int)NR_PAGE_ORDERS - 1);
+
+	atomic_long_add((long)(1UL << addr->order) * sign,
+			&xe->mem.dma_mapped_pages_svm[order]);
+}
+#endif
+
 static const struct drm_gpusvm_ops gpusvm_ops = {
 	.range_alloc = xe_svm_range_alloc,
 	.range_free = xe_svm_range_free,
 	.invalidate = xe_svm_invalidate,
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	.dma_map_account = xe_svm_dma_map_account,
+#endif
 };
 
 static const unsigned long fault_chunk_sizes[] = {
@@ -925,8 +965,8 @@ int xe_svm_init(struct xe_vm *vm)
 		}
 	} else {
 		err = drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-				      NULL, 0, 0, 0, NULL,
-				      NULL, 0);
+				      NULL, 0, 0, 0,
+				      xe_userptr_gpusvm_ops_get(), NULL, 0);
 	}
 
 	return err;
diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h
index a921556d3466..8e347137a7a6 100644
--- a/drivers/gpu/drm/xe/xe_svm.h
+++ b/drivers/gpu/drm/xe/xe_svm.h
@@ -194,6 +194,7 @@ struct drm_pagemap *xe_drm_pagemap_from_fd(int fd, u32 region_instance);
 
 #else
 #include <linux/interval_tree.h>
+#include "xe_userptr.h"
 #include "xe_vm.h"
 
 struct drm_pagemap_addr;
@@ -236,7 +237,8 @@ int xe_svm_init(struct xe_vm *vm)
 {
 #if IS_ENABLED(CONFIG_DRM_GPUSVM)
 	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-			       NULL, 0, 0, 0, NULL, NULL, 0);
+			       NULL, 0, 0, 0,
+			       xe_userptr_gpusvm_ops_get(), NULL, 0);
 #else
 	return 0;
 #endif
diff --git a/drivers/gpu/drm/xe/xe_userptr.c b/drivers/gpu/drm/xe/xe_userptr.c
index 8b2d461ea0b2..6b9e1530ead4 100644
--- a/drivers/gpu/drm/xe/xe_userptr.c
+++ b/drivers/gpu/drm/xe/xe_userptr.c
@@ -8,9 +8,72 @@
 
 #include <linux/mm.h>
 
+#include <drm/drm_pagemap.h>
+
 #include "xe_tlb_inval.h"
 #include "xe_trace_bo.h"
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_userptr_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Account, per page order, the system-memory pages DMA-mapped for GPU access
+ * through the drm_gpusvm_pages instance shared by userptr (and, in fault mode,
+ * SVM). Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_userptr_dma_map_account(struct drm_gpusvm *gpusvm,
+				       const struct drm_pagemap_addr *addr,
+				       int sign)
+{
+	struct xe_vm *vm = container_of(gpusvm, struct xe_vm, svm.gpusvm);
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	/*
+	 * The CPU-PTE map order reported by hmm can exceed MAX_PAGE_ORDER
+	 * (e.g. an order-18 1GB mapping), so clamp the array index to avoid an
+	 * out-of-bounds access. Clamping is symmetric across the +1/-1 calls,
+	 * so the per-order counters stay balanced.
+	 */
+	order = min(order, (unsigned int)NR_PAGE_ORDERS - 1);
+
+	atomic_long_add((long)(1UL << addr->order) * sign,
+			&vm->xe->mem.dma_mapped_pages_svm[order]);
+}
+
+static const struct drm_gpusvm_ops xe_userptr_gpusvm_ops = {
+	.dma_map_account = xe_userptr_dma_map_account,
+};
+#endif
+
+/**
+ * xe_userptr_gpusvm_ops_get() - Accounting ops for the simple gpusvm instance
+ *
+ * The core drm_gpusvm_pages-only instance used for userptr (and, in fault
+ * mode, shared with SVM) is initialised with these ops so that DMA-mapped
+ * system pages are accounted and exposed through debugfs. Returns NULL when
+ * CONFIG_DEBUG_FS is disabled, in which case no accounting is kept and the
+ * instance is initialised without ops.
+ *
+ * Return: Pointer to the restricted &drm_gpusvm_ops, or NULL.
+ */
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void)
+{
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	return &xe_userptr_gpusvm_ops;
+#else
+	return NULL;
+#endif
+}
+
 static void xe_userptr_assert_in_notifier(struct xe_vm *vm)
 {
 	lockdep_assert(lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 0) ||
diff --git a/drivers/gpu/drm/xe/xe_userptr.h b/drivers/gpu/drm/xe/xe_userptr.h
index 2a3cd1b5efbb..1e9601be5713 100644
--- a/drivers/gpu/drm/xe/xe_userptr.h
+++ b/drivers/gpu/drm/xe/xe_userptr.h
@@ -108,6 +108,7 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm);
 int xe_vm_userptr_check_repin(struct xe_vm *vm);
 int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma);
 int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma);
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void);
 #else
 static inline void xe_userptr_remove(struct xe_userptr_vma *uvma) {}
 
-- 
2.34.1

[PATCH v3 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

[Matthew Brost]

On L2-flush-optimized HW with a dma-resv (non-fault) VM, evicting a BO
only needs to flush the L2 cache before the migration copy runs; the
mappings themselves are torn down and rebuilt lazily via
drm_gpuvm_bo_evict() and a subsequent rebind. Today this flush is done
by waiting for the BO to go idle and then issuing a synchronous TLB
invalidation per mapping VMA from inside xe_bo_trigger_rebind(). Both
the idle wait and the synchronous invalidation stall the calling thread
while holding the BO dma-resv lock, serializing the move behind all
in-flight GPU work on the BO.

Replace this with an asynchronous flush. Add xe_vm_flush_vm_bo_tlb_async()
which, for each VMA mapping the BO on each present tile, queues a TLB
invalidation job on the tile migrate (kernel) exec queue. The jobs depend
on the BO's in-flight GPU work, captured once as a singleton over
DMA_RESV_USAGE_BOOKKEEP, so the flush only fires once the GPU is done with
the current mapping. Each job's completion fence is installed into the
BO's dma-resv as a DMA_RESV_USAGE_KERNEL fence, so the migration copy -
which waits on the resv - waits on the flush without stalling this thread.

No PTEs are zapped and vma->tile_invalidated is left untouched: the
mapping stays valid until the lazy rebind, and the only work performed
here is the L2 flush. On any failure the caller falls back to the
existing blocking wait-idle plus xe_vm_invalidate_vma() path.

Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Tejas Upadhyay <tejas.upadhyay@intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Gate the asynchronous L2 flush on !ctx->no_wait_gpu (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c |  13 ++++
 drivers/gpu/drm/xe/xe_vm.c | 125 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h |   5 ++
 3 files changed, 143 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 60e0e568aa31..4ee48cdc02ac 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -732,6 +732,19 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_device_is_l2_flush_optimized(xe))
 				continue;
+
+			/*
+			 * Attempt to flush L2 async, fallback to sync flush on
+			 * failure. Skip the async attempt under no_wait_gpu
+			 * (e.g. the shrinker's reclaim pass): job creation
+			 * allocates with GFP_KERNEL, which must not run while
+			 * holding the dma-resv lock under fs_reclaim. The sync
+			 * fallback below honors no_wait_gpu (returns -EBUSY if
+			 * the BO is not idle).
+			 */
+			if (!ctx->no_wait_gpu &&
+			    !xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
+				continue;
 		}
 
 		if (!idle) {
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 73ac031ffb04..4557a8a4d270 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -39,6 +39,7 @@
 #include "xe_sync.h"
 #include "xe_tile.h"
 #include "xe_tlb_inval.h"
+#include "xe_tlb_inval_job.h"
 #include "xe_trace_bo.h"
 #include "xe_vm_madvise.h"
 #include "xe_wa.h"
@@ -4401,6 +4402,130 @@ int xe_vm_invalidate_vma(struct xe_vma *vma)
 	return ret;
 }
 
+/*
+ * xe_vma_tlb_flush_client - Queue an async TLB flush for one VMA on one client
+ *
+ * Create and push a TLB invalidation job on the tile migrate (kernel) exec
+ * queue covering @vma's range, depending on @dep (the BO's in-flight GPU work)
+ * so the flush only fires once the GPU is done with the current mapping. The
+ * job's completion fence is installed into @resv as a KERNEL fence so the
+ * subsequent migration waits on the flush. No PTEs are zapped; this only
+ * flushes L2 via the TLB invalidation.
+ */
+static int xe_vma_tlb_flush_client(struct xe_vm *vm, struct xe_vma *vma,
+				   struct xe_tile *tile, struct xe_gt *gt,
+				   struct dma_resv *resv, struct dma_fence *dep,
+				   int type)
+{
+	struct xe_exec_queue *q = xe_migrate_exec_queue(tile->migrate);
+	struct xe_tlb_inval_job *job;
+	struct dma_fence *fence;
+	int err;
+
+	job = xe_tlb_inval_job_create(q, &gt->tlb_inval,
+				      q->tlb_inval[type].dep_scheduler, vm,
+				      xe_vma_start(vma), xe_vma_end(vma), type);
+	if (IS_ERR(job))
+		return PTR_ERR(job);
+
+	err = xe_tlb_inval_job_alloc_dep(job);
+	if (err)
+		goto out_put;
+
+	err = dma_resv_reserve_fences(resv, 1);
+	if (err)
+		goto out_put;
+
+	/* Cannot fail; consumes a ref on @dep and returns a referenced fence. */
+	fence = xe_tlb_inval_job_push(job, tile->migrate, dep);
+	dma_resv_add_fence(resv, fence, DMA_RESV_USAGE_KERNEL);
+	dma_fence_put(fence);
+
+out_put:
+	/* Drop the creation reference (destroys the job if it was not pushed). */
+	xe_tlb_inval_job_put(job);
+	return err;
+}
+
+/**
+ * xe_vm_flush_vm_bo_tlb_async - Asynchronously flush TLBs for a vm_bo's mappings
+ * @vm: The VM @vm_bo belongs to
+ * @bo: The buffer object being moved
+ * @vm_bo: The gpuvm_bo linking @bo into @vm
+ *
+ * On L2-flush-optimized HW a BO move only needs to flush L2 (via a TLB
+ * invalidation) for the BO's live mappings; the mappings themselves are torn
+ * down and rebuilt lazily via the eviction/rebind path, so no PTEs need to be
+ * zapped here. Rather than blocking the caller on a synchronous invalidation,
+ * issue a TLB invalidation job per VMA per TLB-invalidation client (per present
+ * tile, primary and media GT). Each job waits on the BO's in-flight GPU work
+ * (all dma-resv usages) and its completion fence is installed into the BO's
+ * dma-resv KERNEL slots, so the following migration waits on the flush without
+ * stalling this thread.
+ *
+ * The caller must hold the BO's dma-resv lock and @vm must not be in fault
+ * mode.
+ *
+ * Return: 0 on success, negative error code on failure. On failure the caller
+ * should fall back to the blocking xe_vm_invalidate_vma() path; any jobs
+ * already queued install harmless extra flush fences.
+ */
+int xe_vm_flush_vm_bo_tlb_async(struct xe_vm *vm, struct xe_bo *bo,
+				struct drm_gpuvm_bo *vm_bo)
+{
+	struct xe_device *xe = vm->xe;
+	struct dma_resv *resv = bo->ttm.base.resv;
+	struct dma_fence *dep = NULL;
+	struct drm_gpuva *gpuva;
+	int err;
+
+	dma_resv_assert_held(resv);
+	xe_assert(xe, !xe_vm_in_fault_mode(vm));
+
+	/*
+	 * Single fence capturing all in-flight GPU work on the BO; the TLB
+	 * invalidation jobs depend on it so the flush fires only once the GPU
+	 * is done with the current mapping.
+	 */
+	err = dma_resv_get_singleton(resv, DMA_RESV_USAGE_BOOKKEEP, &dep);
+	if (err)
+		return err;
+	if (!dep)
+		dep = dma_fence_get_stub();
+
+	drm_gpuvm_bo_for_each_va(gpuva, vm_bo) {
+		struct xe_vma *vma = gpuva_to_vma(gpuva);
+		struct xe_tile *tile;
+		u8 id;
+
+		if (xe_vma_is_null(vma) || xe_vma_is_cpu_addr_mirror(vma))
+			continue;
+
+		for_each_tile(tile, xe, id) {
+			if (!(vma->tile_present & BIT(id)))
+				continue;
+
+			err = xe_vma_tlb_flush_client(vm, vma, tile,
+						      tile->primary_gt, resv, dep,
+						      XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT);
+			if (err)
+				goto out;
+
+			if (tile->media_gt) {
+				err = xe_vma_tlb_flush_client(vm, vma, tile,
+							      tile->media_gt, resv, dep,
+							      XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT);
+				if (err)
+					goto out;
+			}
+		}
+	}
+
+out:
+	dma_fence_put(dep);
+	return err;
+}
+
 int xe_vm_validate_protected(struct xe_vm *vm)
 {
 	struct drm_gpuva *gpuva;
diff --git a/drivers/gpu/drm/xe/xe_vm.h b/drivers/gpu/drm/xe/xe_vm.h
index c5b900f38ded..dd5b070eaede 100644
--- a/drivers/gpu/drm/xe/xe_vm.h
+++ b/drivers/gpu/drm/xe/xe_vm.h
@@ -26,6 +26,8 @@ struct ttm_buffer_object;
 
 struct dma_fence;
 
+struct xe_bo;
+struct drm_gpuvm_bo;
 struct xe_exec_queue;
 struct xe_file;
 struct xe_pagefault;
@@ -254,6 +256,9 @@ int xe_vm_invalidate_vma(struct xe_vma *vma);
 
 int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_batch *batch);
 
+int xe_vm_flush_vm_bo_tlb_async(struct xe_vm *vm, struct xe_bo *bo,
+				struct drm_gpuvm_bo *vm_bo);
+
 int xe_vm_validate_protected(struct xe_vm *vm);
 
 static inline void xe_vm_queue_rebind_worker(struct xe_vm *vm)
-- 
2.34.1

[PATCH v3 14/33] drm/xe: Destroy page tables after unlinking all VMAs on VM close

[Matthew Brost]

xe_vm_close_and_put() freed the VM's page-table roots via
xe_vm_pt_destroy() before destroying the "contested" VMAs - those backed
by external (shared) BOs, which are deferred to xe_vma_destroy_unlocked()
because unlinking them requires taking the external BO's dma-resv lock
(via drm_exec), not vm->lock.

Until such a VMA is unlinked it remains reachable from its BO through
drm_gem_for_each_gpuvm_bo(). A concurrent BO move that only holds the
BO's dma-resv lock - TTM eviction, the shrinker, or the defrag worker -
calls xe_bo_move_notify() -> xe_bo_trigger_rebind() ->
xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which dereferences
vm->pt_root[tile->id]. With the page tables already freed this is a
use-after-free / NULL dereference, e.g.:

  Oops: general protection fault [...] kernel NULL pointer dereference
  RIP: xe_pt_zap_ptes+0x7c [xe]
  Call Trace:
   xe_vm_invalidate_vma_submit+0x64/0x130 [xe]
   xe_vm_invalidate_vma+0x27/0x60 [xe]
   xe_bo_move_notify+0x1e7/0x3d0 [xe]
   xe_bo_move+0x62d/0x1620 [xe]
   ttm_bo_handle_move_mem+0x243/0x330 [ttm]
   ttm_bo_validate+0xb4/0x1a0 [ttm]

The race is not closable with a flag or a NULL pt_root check in the
invalidate path: the freeing side holds only vm->lock while the walker
holds only the BO's resv, so there is no common lock to serialise on.

Fix the ordering instead: destroy the page tables only after every VMA -
including the contested external-BO VMAs - has been unlinked. Once the
contested loop completes no BO can reach a VMA of this VM, so freeing the
page tables (under vm->lock, to satisfy xe_vm_assert_held()) is safe.
xe_vm_free_scratch() still runs before the free as it reads
pt_root[]->level.

Fixes: dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_vm.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index 4557a8a4d270..9f55d22cb985 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -1934,15 +1934,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		vma->gpuva.flags |= XE_VMA_DESTROYED;
 	}
 
-	/*
-	 * All vm operations will add shared fences to resv.
-	 * The only exception is eviction for a shared object,
-	 * but even so, the unbind when evicted would still
-	 * install a fence to resv. Hence it's safe to
-	 * destroy the pagetables immediately.
-	 */
 	xe_vm_free_scratch(vm);
-	xe_vm_pt_destroy(vm);
 	xe_vm_unlock(vm);
 
 	/*
@@ -1956,6 +1948,22 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		xe_vma_destroy_unlocked(vma);
 	}
 
+	/*
+	 * Destroy the page tables only after every VMA has been unlinked. The
+	 * contested (external-BO) VMAs above are unlinked from their BO under
+	 * the BO's dma-resv lock, not vm->lock, so they can still be reached
+	 * via drm_gem_for_each_gpuvm_bo() by a concurrent BO move (eviction,
+	 * shrinker or the defrag worker) that only holds the BO's resv. Such a
+	 * move calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which
+	 * dereferences vm->pt_root[]; freeing the page tables while those VMAs
+	 * are still linked is a use-after-free. All vm operations add shared
+	 * fences to the resv (eviction of a shared object still installs an
+	 * unbind fence), so no GPU work outlives this point.
+	 */
+	xe_vm_lock(vm, false);
+	xe_vm_pt_destroy(vm);
+	xe_vm_unlock(vm);
+
 	xe_svm_fini(vm);
 
 	up_write(&vm->lock);
-- 
2.34.1

[PATCH v3 15/33] drm/xe: Track BOs backed at a sub-optimal page order

[Matthew Brost]

Add per-device tracking of BOs whose backing TT pages were allocated
below the pool's beneficial order, as reported by the
TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (read via
ttm_tt_is_beneficial_order_failed()). These BOs are future candidates for
a defragmentation pass that reallocates their pages at the beneficial
order.

Add a mem.defrag sub-struct (lock, list, atomic count) to struct
xe_device and a defrag_link to struct xe_bo, initialised at BO create
and device init. Membership is updated from xe_bo_move() via
xe_bo_defrag_update(): a BO is added when it is a populated, needed,
non-pinned ttm_bo_type_device object flagged
TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED, and removed otherwise. BOs are also
removed on pin, on purge, moved to dontneed, and at destroy.

Only ttm_bo_type_device BOs are tracked.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c           | 115 +++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.h           |   1 +
 drivers/gpu/drm/xe/xe_bo_types.h     |   6 ++
 drivers/gpu/drm/xe/xe_device.c       |   2 +
 drivers/gpu/drm/xe/xe_device_types.h |  19 +++++
 5 files changed, 143 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 4ee48cdc02ac..ff324caab6da 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -926,6 +926,9 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
 	}
 }
 
+static void xe_bo_defrag_update(struct xe_bo *bo);
+static void xe_bo_defrag_remove(struct xe_bo *bo);
+
 /**
  * xe_bo_set_purgeable_state() - Set BO purgeable state with validation
  * @bo: Buffer object
@@ -954,6 +957,7 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
 
 	bo->purgeable.state = new_state;
 	xe_bo_set_purgeable_shrinker(bo, new_state);
+	xe_bo_defrag_update(bo);
 }
 
 /**
@@ -1003,9 +1007,100 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	/* Commit the state transition only once invalidation was queued */
 	xe_bo_set_purgeable_state(bo, XE_MADV_PURGEABLE_PURGED);
 
+	/*
+	 * The backing store is gone, so the BO no longer needs defrag. Drop it
+	 * from the defrag list here so the count stays accurate even for purge
+	 * paths that don't pass through xe_bo_move() (e.g. swap_notify).
+	 */
+	xe_bo_defrag_remove(bo);
+
 	return 0;
 }
 
+/**
+ * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
+ * @xe: The xe device
+ *
+ * Initialize the list, lock and count used to track BOs whose backing TT
+ * pages were allocated at a sub-optimal order.
+ */
+void xe_bo_defrag_init_early(struct xe_device *xe)
+{
+	spin_lock_init(&xe->mem.defrag.lock);
+	INIT_LIST_HEAD(&xe->mem.defrag.list);
+	atomic_set(&xe->mem.defrag.count, 0);
+}
+
+static bool xe_bo_needs_defrag(struct xe_bo *bo)
+{
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_tt *tt = ttm_bo->ttm;
+
+	xe_bo_assert_held(bo);
+
+	return ttm_bo->type == ttm_bo_type_device && tt &&
+		!xe_bo_madv_is_dontneed(bo) &&
+		ttm_tt_is_populated(tt) &&
+		ttm_tt_is_beneficial_order_failed(tt) &&
+		ttm_bo->resource && ttm_bo->resource->mem_type == XE_PL_TT &&
+		!xe_bo_is_pinned(bo);
+}
+
+static void xe_bo_defrag_add(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	xe_bo_assert_held(bo);
+	xe_assert(xe, xe_bo_needs_defrag(bo));
+
+	scoped_guard(spinlock, &xe->mem.defrag.lock) {
+		if (list_empty(&bo->defrag_link)) {
+			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
+			atomic_inc(&xe->mem.defrag.count);
+		}
+	}
+}
+
+static void __xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	guard(spinlock)(&xe->mem.defrag.lock);
+	if (!list_empty(&bo->defrag_link)) {
+		list_del_init(&bo->defrag_link);
+		atomic_dec(&xe->mem.defrag.count);
+	}
+}
+
+static void xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (list_empty(&bo->defrag_link))
+		return;
+
+	__xe_bo_defrag_remove(bo);
+}
+
+/**
+ * xe_bo_defrag_update() - Update defrag list membership for a BO
+ * @bo: The buffer object
+ *
+ * Add @bo to the device defrag list when it is a ttm_bo_type_device BO resident
+ * in XE_PL_TT with a populated TT whose pages were allocated at a sub-optimal
+ * order (ttm_tt_is_beneficial_order_failed()) and it isn't pinned. Otherwise
+ * ensure it is removed from the list.
+ */
+static void xe_bo_defrag_update(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (xe_bo_needs_defrag(bo))
+		xe_bo_defrag_add(bo);
+	else
+		xe_bo_defrag_remove(bo);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
@@ -1231,6 +1326,8 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		xe_tt_unmap_sg(xe, ttm_bo->ttm);
 	}
 
+	xe_bo_defrag_update(bo);
+
 	return ret;
 }
 
@@ -1267,6 +1364,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 	if (lret > 0) {
 		xe_ttm_tt_account_subtract(xe, bo->ttm);
 		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
+
+		/*
+		 * The pages are gone, so a tracked BO no longer needs defrag.
+		 * This shrinker purge path frees pages without going through
+		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
+		 * bos are never tracked, so guard for real xe bos.
+		 */
+		if (xe_bo_is_xe_bo(bo))
+			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
 	}
 
 	return lret;
@@ -1909,6 +2015,8 @@ static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo)
 		list_del(&bo->vram_userfault_link);
 	mutex_unlock(&xe->mem_access.vram_userfault.lock);
 
+	__xe_bo_defrag_remove(bo);
+
 	kfree(bo);
 }
 
@@ -2414,6 +2522,7 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 	bo->ttm.base.funcs = &xe_gem_object_funcs;
 	bo->ttm.priority = XE_BO_PRIORITY_NORMAL;
 	INIT_LIST_HEAD(&bo->pinned_link);
+	INIT_LIST_HEAD(&bo->defrag_link);
 #ifdef CONFIG_PROC_FS
 	INIT_LIST_HEAD(&bo->client_link);
 #endif
@@ -3020,6 +3129,8 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3078,6 +3189,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3113,6 +3226,7 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -3145,6 +3259,7 @@ void xe_bo_unpin(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 }
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 6340317f7d2e..4186cb51af87 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -224,6 +224,7 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
 int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
+void xe_bo_defrag_init_early(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h
index fcc63ae3f455..9eea9ab380e5 100644
--- a/drivers/gpu/drm/xe/xe_bo_types.h
+++ b/drivers/gpu/drm/xe/xe_bo_types.h
@@ -54,6 +54,12 @@ struct xe_bo {
 	struct ttm_bo_kmap_obj kmap;
 	/** @pinned_link: link to present / evicted list of pinned BO */
 	struct list_head pinned_link;
+	/**
+	 * @defrag_link: link into @xe_device.mem.defrag.list for BOs whose
+	 * backing TT pages were allocated at a sub-optimal order. Protected by
+	 * @xe_device.mem.defrag.lock.
+	 */
+	struct list_head defrag_link;
 #ifdef CONFIG_PROC_FS
 	/**
 	 * @client: @xe_drm_client which created the bo
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index 97cc2d185e45..bb93425a5c70 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -607,6 +607,8 @@ int xe_device_init_early(struct xe_device *xe)
 	if (err)
 		return err;
 
+	xe_bo_defrag_init_early(xe);
+
 	xe->preempt_fence_wq = alloc_ordered_workqueue("xe-preempt-fence-wq",
 						       WQ_MEM_RECLAIM);
 	xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 95e7cea91cbf..9e6244d8f41b 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -296,6 +296,25 @@ struct xe_device {
 		 */
 		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
 #endif
+		/**
+		 * @mem.defrag: Tracking of BOs whose backing TT pages were
+		 * allocated at a sub-optimal (smaller than beneficial) order.
+		 *
+		 * Such BOs are candidates for a future defragmentation pass that
+		 * tries to reallocate their pages at the device's beneficial
+		 * order. Only ttm_bo_type_device BOs are tracked.
+		 */
+		struct {
+			/** @mem.defrag.lock: Protects @mem.defrag.list. */
+			spinlock_t lock;
+			/** @mem.defrag.list: List of struct xe_bo defrag_link. */
+			struct list_head list;
+			/**
+			 * @mem.defrag.count: Number of BOs currently on
+			 * @mem.defrag.list.
+			 */
+			atomic_t count;
+		} defrag;
 	} mem;
 
 	/** @sriov: device level virtualization data */
-- 
2.34.1

[PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure

[Matthew Brost]

Once a meaningful number of BOs are already tracked as being backed at a
sub-optimal order, continuing to push hard for beneficial-order pages
during populate mostly wastes time in reclaim/compaction.

In xe_ttm_tt_populate(), set ctx->beneficial_reclaim_backoff when the
device defrag count reaches XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD (2)
so the TTM pool backs off and allocations make forward progress. The
deferred defrag pass is responsible for upgrading these objects to
beneficial order later.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index ff324caab6da..02395968d613 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -41,6 +41,14 @@
 #include "xe_vm.h"
 #include "xe_vram_types.h"
 
+/*
+ * Once this many BOs are tracked on the device defrag list (i.e. were backed
+ * with a sub-optimal page order), request that the TTM pool backs off from
+ * aggressive reclaim at the beneficial order during populate, so that
+ * allocations make forward progress instead of stalling.
+ */
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",
@@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
 		err = ttm_tt_restore(ttm_dev, tt, ctx);
 	} else {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		if (atomic_read(&xe->mem.defrag.count) >=
+		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+			ctx->beneficial_reclaim_backoff = true;
+
 		ttm_tt_clear_backed_up(tt);
 		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	}
-- 
2.34.1

[PATCH v3 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

[Matthew Brost]

Add xe_migrate_copy_defrag(), a copy helper for the upcoming page
defragmentation path where a single BO has its backing reallocated at a
more beneficial page order and its contents must be relocated from the
old pages to the new ones entirely on the GPU.

The contents are relocated in up to two passes:

 - Data pass (always): copy the data pages verbatim. The pages are
   mapped without the compression PAT, so the (possibly compressed)
   bytes are copied as-is rather than being decompressed and
   recompressed, and no CCS aux copy is appended.

 - CCS pass (need_ccs): copy the CCS aux (compression) state. Both the
   old and new data pages are mapped with the compression PAT so the
   copy engine accesses the aux state indirectly, issuing one
   indirect -> indirect XY_CTRL_SURF_COPY_BLT; the data blit is skipped.

The two passes are issued separately because they require opposite
compression PAT settings on the data page mappings (cleared for the
verbatim data copy, set for the indirect CCS access), which a single
mapping cannot satisfy. Both passes run in order on the same migrate
queue and are captured by the returned fence.

The source pages are passed explicitly via an sg_table argument (the
old, still populated backing) since they no longer match the BO's
current backing. The boolean arguments to __xe_migrate_copy() are folded
into struct xe_migrate_copy_flags to keep the growing set of copy
variants readable.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_migrate.h |   8 ++
 2 files changed, 128 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 92d5e81ceac2..305c6e1942fb 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -858,13 +858,39 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
 	return flush_flags;
 }
 
+/**
+ * struct xe_migrate_copy_flags - Flags for __xe_migrate_copy()
+ * @copy_only_ccs: Only copy the CCS aux state, aliasing the destination data
+ * mapping to the source (used when the data pages are shared between src/dst).
+ * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
+ * compression PAT index for the data copy.
+ * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
+ * two distinct page sets (old @src_sg pages -> new pages). The data blit is
+ * skipped (the data was copied by the @defrag_copy pass) and both the source
+ * and destination data pages are mapped with the compression PAT so the aux
+ * state is copied indirect -> indirect.
+ * @defrag_copy: Defrag move pass that copies the data pages between two
+ * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
+ * aux state. The data pages are mapped without the compression PAT so the
+ * (possibly compressed) bytes are copied verbatim rather than being
+ * decompressed and recompressed, and no CCS aux copy is appended; the aux
+ * state is migrated separately by the @defrag_ccs pass. The two passes need
+ * opposite compression PAT settings, which is why they cannot be combined.
+ */
+struct xe_migrate_copy_flags {
+	u32 copy_only_ccs : 1;
+	u32 is_vram_resolve : 1;
+	u32 defrag_ccs : 1;
+	u32 defrag_copy : 1;
+};
+
 static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct xe_bo *src_bo,
 					   struct xe_bo *dst_bo,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
-					   bool copy_only_ccs,
-					   bool is_vram_resolve)
+					   struct sg_table *src_sg,
+					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
 	struct xe_device *xe = gt_to_xe(gt);
@@ -881,20 +907,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	bool src_is_vram = mem_type_is_vram(src->mem_type);
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 	bool type_device = src_bo->ttm.type == ttm_bo_type_device;
-	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
+	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
+		!flags.defrag_copy;
 	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
-		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
-	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
+		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
+		!flags.defrag_copy;
+	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
+		!flags.defrag_ccs && !flags.defrag_copy;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
 	 * Otherwise, only use the compression PAT index for device memory
 	 * when copying from VRAM to system memory.
 	 */
-	bool use_comp_pat = is_vram_resolve || (type_device &&
+	bool use_comp_pat = flags.is_vram_resolve || (type_device &&
 			    xe_device_has_flat_ccs(xe) &&
 			    GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram);
 
+	/*
+	 * The defrag CCS pass migrates the aux state from the old pages
+	 * (@src_sg) to the new pages with both sides accessed indirectly, so
+	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
+	 */
+
 	/* Copying CCS between two different BOs is not supported yet. */
 	if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
 		return ERR_PTR(-EINVAL);
@@ -903,7 +938,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
@@ -912,7 +947,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
+				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
 
@@ -942,7 +978,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
 					      &src_L0_ofs, &src_L0_pt, 0, 0,
 					      avail_pts);
-		if (copy_only_ccs) {
+		if (flags.copy_only_ccs) {
 			dst_L0_ofs = src_L0_ofs;
 		} else {
 			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
@@ -963,7 +999,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		}
 
 		/* Add copy commands size here */
-		batch_size += ((copy_only_ccs) ? 0 : emit_copy_cmd_len(xe)) +
+		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
+			       emit_copy_cmd_len(xe)) +
 			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
@@ -975,13 +1012,15 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		if (src_is_vram && xe_migrate_allow_identity(src_L0, &src_it))
 			xe_res_next(&src_it, src_L0);
 		else
-			emit_pte(m, bb, src_L0_pt, src_is_vram, copy_system_ccs || use_comp_pat,
+			emit_pte(m, bb, src_L0_pt, src_is_vram,
+				 copy_system_ccs || use_comp_pat || flags.defrag_ccs,
 				 &src_it, src_L0, src);
 
 		if (dst_is_vram && xe_migrate_allow_identity(src_L0, &dst_it))
 			xe_res_next(&dst_it, src_L0);
-		else if (!copy_only_ccs)
-			emit_pte(m, bb, dst_L0_pt, dst_is_vram, copy_system_ccs,
+		else if (!flags.copy_only_ccs)
+			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
+				 copy_system_ccs || flags.defrag_ccs,
 				 &dst_it, src_L0, dst);
 
 		if (copy_system_ccs)
@@ -990,7 +1029,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
 		update_idx = bb->len;
 
-		if (!copy_only_ccs)
+		if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
 		if (needs_ccs_emit)
@@ -1078,7 +1117,69 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs)
 {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, copy_only_ccs, false);
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .copy_only_ccs = copy_only_ccs,
+				 });
+}
+
+/**
+ * xe_migrate_copy_defrag() - Migrate a BO's backing to a reallocated page set.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
+ * @src: The source TTM resource (the old backing).
+ * @dst: The dst TTM resource (the freshly reallocated backing).
+ * @src_sg: Scatter-gather table of the source (old) backing pages.
+ * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
+ *
+ * Used by the defrag path where a single BO has its backing reallocated at a
+ * more beneficial page order. The contents are relocated from the old pages
+ * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
+ * in up to two passes:
+ *
+ *  - Data pass (always): copy the data pages verbatim. The pages are mapped
+ *    without the compression PAT, so the (possibly compressed) bytes are copied
+ *    as-is rather than being decompressed and recompressed, and no CCS aux copy
+ *    is appended.
+ *
+ *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
+ *    and new data pages are mapped with the compression PAT so the copy engine
+ *    accesses the aux state indirectly, issuing one indirect -> indirect
+ *    XY_CTRL_SURF_COPY_BLT; the data blit is skipped.
+ *
+ * The two passes are issued separately because they require opposite
+ * compression PAT settings on the data page mappings (cleared for the verbatim
+ * data copy, set for the indirect CCS access), which a single mapping cannot
+ * satisfy. Both passes run in order on the same migrate queue.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+					 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				   (struct xe_migrate_copy_flags) {
+				   .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
 }
 
 /**
@@ -1096,7 +1197,10 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res)
 {
-	return __xe_migrate_copy(m, bo, bo, res, res, false, true);
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .is_vram_resolve = true,
+				 });
 }
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
index 965c45889c72..e4f996942a6d 100644
--- a/drivers/gpu/drm/xe/xe_migrate.h
+++ b/drivers/gpu/drm/xe/xe_migrate.h
@@ -11,6 +11,7 @@
 struct dma_fence;
 struct drm_pagemap_addr;
 struct iosys_map;
+struct sg_table;
 struct ttm_resource;
 
 struct xe_bo;
@@ -132,6 +133,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs);
+
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res);
-- 
2.34.1

[PATCH v3 18/33] drm/xe: Handle defrag moves in xe_bo_move()

[Matthew Brost]

Teach xe_bo_move() to service a defrag move, where TTM reallocates a
BO's backing at the device's beneficial order and hands the old, still
populated tt to the move via the BO's defrag_old_tt.

A defrag move always has existing contents to copy, so force
move_lacks_source false and fall through the XE_PL_TT -> XE_PL_TT no-op
shortcut so the copy path runs. The relocation is performed entirely on
the GPU via xe_migrate_copy_defrag(), reading the old pages from the
stashed tt's sg table and writing the freshly reallocated backing.

Both copy passes run in order on the same migrate queue and are captured
by the returned fence, so the teardown is pipelined via
ttm_bo_move_accel_cleanup(): the old tt is handed to a ghost object and
only unpopulated and freed once the fence signals, so the move need not
block. The VF CCS attach/detach handling is extended to treat a defrag
move (ctx->defrag) like the XE_PL_SYSTEM <-> XE_PL_TT transitions it
mirrors.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 02395968d613..bdcc9bed5d36 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1170,6 +1170,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	move_lacks_source = !old_mem || (handle_system_ccs ? (!bo->ccs_cleared) :
 					 (!mem_type_is_vram(old_mem_type) && !tt_has_data));
 
+	/*
+	 * A defrag move always copies the existing contents from the stashed
+	 * old tt into the freshly (re)allocated backing.
+	 */
+	if (ttm_bo->defrag_old_tt)
+		move_lacks_source = false;
+
 	needs_clear = (ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) ||
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
@@ -1206,9 +1213,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	/*
 	 * Failed multi-hop where the old_mem is still marked as
 	 * TTM_PL_FLAG_TEMPORARY, should just be a dummy move.
+	 *
+	 * For a defrag move the old and new tt differ, so fallthrough to the
+	 * copy path instead of treating it as a no-op.
 	 */
 	if (old_mem_type == XE_PL_TT &&
-	    new_mem->mem_type == XE_PL_TT) {
+	    new_mem->mem_type == XE_PL_TT && !ttm_bo->defrag_old_tt) {
 		ttm_bo_move_null(ttm_bo, new_mem);
 		goto out;
 	}
@@ -1279,6 +1289,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 			flags |= XE_MIGRATE_CLEAR_FLAG_CCS_DATA;
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
+	} else if (ttm_bo->defrag_old_tt) {
+		struct xe_ttm_tt *old_xe_tt =
+			container_of(ttm_bo->defrag_old_tt, struct xe_ttm_tt, ttm);
+
+		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
+					       old_xe_tt->sg, handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
@@ -1315,13 +1331,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	 * BBs from BO as it is no longer needed.
 	 */
 	if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT &&
-	    new_mem->mem_type == XE_PL_SYSTEM)
+	    (new_mem->mem_type == XE_PL_SYSTEM || ctx->defrag))
 		xe_sriov_vf_ccs_detach_bo(bo);
 
 	if (IS_VF_CCS_READY(xe) &&
 	    ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
-	     (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
-	    handle_system_ccs)
+	     ((old_mem_type == XE_PL_SYSTEM || ctx->defrag) &&
+	      new_mem->mem_type == XE_PL_TT)) && handle_system_ccs)
 		ret = xe_sriov_vf_ccs_attach_bo(bo);
 
 out:
-- 
2.34.1

[PATCH v3 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves

[Matthew Brost]

A defrag move may harvest already beneficial-order chunks from the old
tt, so the freshly reallocated backing borrows (references) some of the
same physical pages. For those pages the source and destination GPU
mappings alias the same memory, making the data blit a pointless
self-copy.

Pass the source (old) tt into __xe_migrate_copy() instead of just its sg
table, so the data pass can compare the src and dst tt page arrays at the
matching offset and only blit runs of pages that actually differ. A new
emit_copy_chunks() helper walks each L0 chunk, skips borrowed-page runs,
and coalesces contiguous to-copy pages into a single emit_copy(). It runs
twice per chunk: once with a NULL batch buffer to count the blits for
batch sizing, then again to emit them.

When an entire L0 chunk turns out to be borrowed, skip it outright: no
batch buffer is allocated and no GPU commands are issued, the cursors are
just advanced past the chunk. If every chunk of the move is borrowed the
copy is a complete no-op, so hand back a signaled stub fence.

The comparison is a cheap per-page pointer test (src page != dst page)
that coalesces contiguous to-copy pages into a single blit. PTEs are
still emitted for partially borrowed chunks, since that setup is cheap and
only the data copies are worth skipping. Only the defrag data pass is
affected; the separate CCS pass still migrates all aux state.

Since the Xe tt sg lives in the xe_ttm_tt subclass rather than the base
ttm_tt, add an xe_tt_sg() helper to fetch it from a struct ttm_tt.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

----

v3:
 - Fix start_page page-shift and blit pitch for src-natural to
   dst-packed copies (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c      |  22 +++++-
 drivers/gpu/drm/xe/xe_bo.h      |   2 +
 drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_migrate.h |   3 +-
 4 files changed, 142 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index bdcc9bed5d36..7f6ee9b8d12e 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -473,6 +473,22 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
 	return xe_tt->sg;
 }
 
+/**
+ * xe_tt_sg() - Return the DMA scatter-gather table backing a ttm_tt.
+ * @tt: The ttm_tt to query.
+ *
+ * Like xe_bo_sg(), but takes the ttm_tt directly. Used by the defrag copy
+ * path which needs the old (source) tt's sg table without a backing xe_bo.
+ *
+ * Return: The sg table, or NULL if the tt is not currently mapped.
+ */
+struct sg_table *xe_tt_sg(struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	return xe_tt->sg;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
@@ -1290,11 +1306,9 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
 	} else if (ttm_bo->defrag_old_tt) {
-		struct xe_ttm_tt *old_xe_tt =
-			container_of(ttm_bo->defrag_old_tt, struct xe_ttm_tt, ttm);
-
 		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
-					       old_xe_tt->sg, handle_system_ccs);
+					       ttm_bo->defrag_old_tt,
+					       handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 4186cb51af87..32211016a722 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -110,6 +110,7 @@ enum xe_madv_purgeable_state {
 };
 
 struct sg_table;
+struct ttm_tt;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -547,6 +548,7 @@ void xe_bo_dev_init(struct xe_bo_dev *bo_device);
 void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
 
 struct sg_table *xe_bo_sg(struct xe_bo *bo);
+struct sg_table *xe_tt_sg(struct ttm_tt *tt);
 
 /*
  * xe_sg_segment_size() - Provides upper limit for sg segment size.
diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 305c6e1942fb..9990a7144a9f 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -811,6 +811,63 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
 		emit_xy_fast_copy(gt, bb, src_ofs, dst_ofs, size, pitch);
 }
 
+/*
+ * emit_copy_chunks() - Emit the data blits for one defrag L0 chunk, skipping
+ * borrowed pages.
+ * @gt: The GT.
+ * @bb: The batch buffer to emit into, or NULL to only count the blits needed.
+ * @src_L0_ofs: GPU offset of the source pages mapped for this L0 chunk.
+ * @dst_L0_ofs: GPU offset of the destination pages mapped for this L0 chunk.
+ * @src_L0: Byte size of the L0 chunk (page aligned).
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ *
+ * A defragmentation move may harvest already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) some of the same physical pages.
+ * For those pages the source and destination GPU mappings alias the same
+ * memory and the data blit is a pointless self-copy. Walk the L0 chunk and
+ * only blit runs of pages that actually differ (src page != dst page),
+ * coalescing contiguous to-copy pages into a single blit.
+ *
+ * Return: The number of blits emitted (or that would be emitted when @bb is
+ * NULL).
+ */
+static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
+			    u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
+			    struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
+			    pgoff_t start_page)
+{
+	u32 npages = src_L0 >> PAGE_SHIFT;
+	u32 i = 0, nblits = 0;
+
+	while (i < npages) {
+		u32 run = 0;
+
+		/* Skip a run of borrowed pages (src and dst alias). */
+		if (src_tt->pages[start_page + i] ==
+		    dst_tt->pages[start_page + i]) {
+			i++;
+			continue;
+		}
+
+		/* Coalesce a run of pages that actually need copying. */
+		while (i + run < npages &&
+		       src_tt->pages[start_page + i + run] !=
+		       dst_tt->pages[start_page + i + run])
+			run++;
+
+		if (bb)
+			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
+				  dst_L0_ofs + (u64)i * PAGE_SIZE,
+				  run * PAGE_SIZE, XE_PAGE_SIZE);
+		nblits++;
+		i += run;
+	}
+
+	return nblits;
+}
+
 static u64 xe_migrate_batch_base(struct xe_migrate *m, bool usm)
 {
 	return usm ? m->usm_batch_base_ofs : m->batch_base_ofs;
@@ -865,13 +922,15 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
  * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
  * compression PAT index for the data copy.
  * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
- * two distinct page sets (old @src_sg pages -> new pages). The data blit is
+ * two distinct page sets (old @src_tt pages -> new pages). The data blit is
  * skipped (the data was copied by the @defrag_copy pass) and both the source
  * and destination data pages are mapped with the compression PAT so the aux
  * state is copied indirect -> indirect.
  * @defrag_copy: Defrag move pass that copies the data pages between two
- * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
- * aux state. The data pages are mapped without the compression PAT so the
+ * distinct page sets (old @src_tt pages -> new pages) without touching the CCS
+ * aux state. Pages the new tt borrowed from @src_tt (harvested already
+ * beneficial-order chunks) alias the same memory on both sides and are skipped.
+ * The data pages are mapped without the compression PAT so the
  * (possibly compressed) bytes are copied verbatim rather than being
  * decompressed and recompressed, and no CCS aux copy is appended; the aux
  * state is migrated separately by the @defrag_ccs pass. The two passes need
@@ -889,7 +948,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct xe_bo *dst_bo,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
-					   struct sg_table *src_sg,
+					   struct ttm_tt *src_tt,
 					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
@@ -914,6 +973,14 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		!flags.defrag_copy;
 	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
 		!flags.defrag_ccs && !flags.defrag_copy;
+	/*
+	 * On the defrag data pass the new (destination) tt may have borrowed
+	 * already beneficial-order pages from the old (source) tt, so skip the
+	 * self-copy for any page they share. Compare the tt page arrays at the
+	 * matching offset.
+	 */
+	struct ttm_tt *dst_tt = dst_bo->ttm.ttm;
+	bool skip_borrowed = flags.defrag_copy && src_tt && dst_tt;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
@@ -926,7 +993,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 
 	/*
 	 * The defrag CCS pass migrates the aux state from the old pages
-	 * (@src_sg) to the new pages with both sides accessed indirectly, so
+	 * (@src_tt) to the new pages with both sides accessed indirectly, so
 	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
 	 */
 
@@ -938,7 +1005,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo), 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
@@ -947,7 +1014,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
+		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo),
 				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
@@ -961,6 +1028,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		u64 ccs_ofs, ccs_size;
 		u32 ccs_pt;
 		u32 pte_flags;
+		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> PAGE_SHIFT;
+		u32 ndata_blits = 1;
 
 		bool usm = xe->info.has_usm;
 		u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
@@ -998,9 +1067,30 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 			xe_assert(xe, IS_ALIGNED(ccs_it.start, PAGE_SIZE));
 		}
 
-		/* Add copy commands size here */
+		/*
+		 * Add copy commands size here. On the defrag data pass the blit
+		 * is split into one command per run of non-borrowed pages, so
+		 * size for the number of runs actually needed.
+		 */
+		if (skip_borrowed)
+			ndata_blits = emit_copy_chunks(gt, NULL, 0, 0, src_L0,
+						       src_tt, dst_tt, start_page);
+
+		/*
+		 * If every page in this chunk was borrowed there is nothing to
+		 * copy: don't build a batch or issue any GPU commands, just
+		 * advance the cursors past the chunk. The non-vram tt cursors
+		 * would otherwise be advanced by emit_pte().
+		 */
+		if (skip_borrowed && !ndata_blits) {
+			xe_res_next(&src_it, src_L0);
+			xe_res_next(&dst_it, src_L0);
+			size -= src_L0;
+			continue;
+		}
+
 		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
-			       emit_copy_cmd_len(xe)) +
+			       ndata_blits * emit_copy_cmd_len(xe)) +
 			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
@@ -1029,7 +1119,10 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
 		update_idx = bb->len;
 
-		if (!flags.copy_only_ccs && !flags.defrag_ccs)
+		if (skip_borrowed)
+			emit_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0,
+					 src_tt, dst_tt, start_page);
+		else if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
 		if (needs_ccs_emit)
@@ -1088,6 +1181,14 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(err);
 	}
 
+	/*
+	 * Every chunk of a defrag data pass may have been borrowed (nothing
+	 * copied), leaving no job and a NULL fence. Hand back a signaled stub
+	 * so callers always get a valid fence to wait on.
+	 */
+	if (!fence)
+		return dma_fence_get_stub();
+
 	return fence;
 }
 
@@ -1129,18 +1230,21 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
  * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
  * @src: The source TTM resource (the old backing).
  * @dst: The dst TTM resource (the freshly reallocated backing).
- * @src_sg: Scatter-gather table of the source (old) backing pages.
+ * @src_tt: The source (old) backing tt, providing both the scatter-gather
+ * table and page pointers of the old pages.
  * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
  *
  * Used by the defrag path where a single BO has its backing reallocated at a
  * more beneficial page order. The contents are relocated from the old pages
- * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
+ * (@src_tt) to the new pages (the BO's current backing) entirely on the GPU,
  * in up to two passes:
  *
  *  - Data pass (always): copy the data pages verbatim. The pages are mapped
  *    without the compression PAT, so the (possibly compressed) bytes are copied
  *    as-is rather than being decompressed and recompressed, and no CCS aux copy
- *    is appended.
+ *    is appended. Pages the new backing borrowed from @src_tt (harvested
+ *    already beneficial-order chunks) alias the same memory on both sides and
+ *    are skipped.
  *
  *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
  *    and new data pages are mapped with the compression PAT so the copy engine
@@ -1159,11 +1263,11 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct xe_bo *bo,
 					 struct ttm_resource *src,
 					 struct ttm_resource *dst,
-					 struct sg_table *src_sg,
+					 struct ttm_tt *src_tt,
 					 bool need_ccs)
 {
 	struct dma_fence *fence2, *fence =
-		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt,
 				  (struct xe_migrate_copy_flags) {
 					 .defrag_copy = true,
 					 });
@@ -1171,7 +1275,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	if (IS_ERR(fence) || !need_ccs)
 		return fence;
 
-	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
 				   (struct xe_migrate_copy_flags) {
 				   .defrag_ccs = true,
 				   });
diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
index e4f996942a6d..4094ea1c19c1 100644
--- a/drivers/gpu/drm/xe/xe_migrate.h
+++ b/drivers/gpu/drm/xe/xe_migrate.h
@@ -13,6 +13,7 @@ struct drm_pagemap_addr;
 struct iosys_map;
 struct sg_table;
 struct ttm_resource;
+struct ttm_tt;
 
 struct xe_bo;
 struct xe_gt;
@@ -137,7 +138,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct xe_bo *bo,
 					 struct ttm_resource *src,
 					 struct ttm_resource *dst,
-					 struct sg_table *src_sg,
+					 struct ttm_tt *src_tt,
 					 bool need_ccs);
 
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
-- 
2.34.1

[PATCH v3 20/33] drm/xe: Add a page defragmentation worker

[Matthew Brost]

BOs whose backing TT pages were allocated below the device's beneficial
order (tracked via the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag)
are added to a per-device defrag list. Add a delayed worker that
periodically walks this list and attempts to re-back each BO with
beneficial-order pages, improving TLB efficiency for long-lived objects
that were initially allocated under memory pressure.

The worker is kicked when the list first transitions to non-empty and
reschedules itself while work remains. On a failed pass it backs off
exponentially (XE_BO_DEFRAG_INTERVAL_MS up to
XE_BO_DEFRAG_INTERVAL_MAX_MS) and retries the whole list later; the
interval is reset to the default on the next fresh enqueue. The worker
stops scheduling once the list drains.

A defrag move synchronously reallocates and re-copies a BO's backing
store, which is not free. To avoid stalling concurrent active work when
many BOs (or one very large BO) become eligible at once (e.g. after a
burst of memory pressure, which would otherwise manifest as a large FPS
drop while the worker churns through the whole list), cap the number of
newly (re)allocated bytes processed per run to XE_BO_DEFRAG_SIZE_LIMIT
and reschedule. Only the pages a move actually reallocates are charged -
pages harvested (borrowed) from the old backing are free and do not count
- so the budget tracks real defrag work and an object larger than the
budget is upgraded in budget-sized slices across runs. This spreads the
defrag effort over time, keeping it in the background and yielding to
userspace progress.

Defragmenting a BO forces a TTM move (ctx.defrag) that reallocates the
backing at the beneficial order and relocates the contents on the GPU
(see xe_bo_move()). The per-BO move is bounded by the run's remaining
byte budget via ctx.defrag_bytes_remaining; the pool allocator reports
how much it spent so the worker can charge it accurately. When
beneficial-order pages cannot be obtained the new allocation is
discharged during populate and the BO keeps its original backing, staying
on the list for a later retry.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Compute the needs-more state under the BO lock via an out-param
   instead of re-reading bo->ttm.ttm after unlock (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c           | 296 ++++++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h           |   1 +
 drivers/gpu/drm/xe/xe_device.c       |   9 +
 drivers/gpu/drm/xe/xe_device_types.h |  11 +
 4 files changed, 313 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 7f6ee9b8d12e..907cfa57481f 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -31,6 +31,7 @@
 #include "xe_pat.h"
 #include "xe_pm.h"
 #include "xe_preempt_fence.h"
+#include "xe_printk.h"
 #include "xe_pxp.h"
 #include "xe_res_cursor.h"
 #include "xe_shrinker.h"
@@ -49,6 +50,35 @@
  */
 #define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
 
+/*
+ * Maximum number of bytes of newly (re)allocated backing the defrag worker will
+ * produce in a single run before yielding and rescheduling itself.
+ *
+ * A defrag move synchronously reallocates and re-copies a BO's backing store,
+ * which is not free. If a large number of BOs (or one very large BO) become
+ * eligible at once (e.g. after a burst of memory pressure), processing them all
+ * in one worker run would hold things up for a long, unbounded stretch and can
+ * visibly starve concurrent active work (e.g. a large FPS drop). Instead, cap
+ * the bytes actually upgraded per run and requeue, spreading the defrag effort
+ * out over time so it stays in the background and yields to userspace progress.
+ * Only pages a move truly reallocates are charged; pages harvested from the old
+ * backing are free, so an object larger than the budget is upgraded in
+ * budget-sized slices across runs.
+ */
+#define XE_BO_DEFRAG_SIZE_LIMIT			SZ_32M
+
+/* Default delay before (re)running the defrag worker, in milliseconds. */
+#define XE_BO_DEFRAG_INTERVAL_MS		25
+
+/*
+ * Upper bound for the (exponentially backed off) defrag worker interval, in
+ * milliseconds, so repeated failures don't push the retry arbitrarily far out.
+ */
+#define XE_BO_DEFRAG_INTERVAL_MAX_MS		15000	/* 15 seconds */
+
+static void xe_bo_defrag_worker(struct work_struct *w);
+static void xe_place_from_ttm_type(u32 mem_type, struct ttm_place *place);
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",
@@ -624,8 +654,10 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	} else {
 		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
 
-		if (atomic_read(&xe->mem.defrag.count) >=
-		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+		if (ctx->defrag)
+			ctx->beneficial_reclaim_backoff = false;
+		else if (atomic_read(&xe->mem.defrag.count) >=
+			 XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
 			ctx->beneficial_reclaim_backoff = true;
 
 		ttm_tt_clear_backed_up(tt);
@@ -1047,18 +1079,54 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	return 0;
 }
 
+static void xe_bo_defrag_fini(void *arg)
+{
+	struct xe_device *xe = arg;
+
+	disable_delayed_work_sync(&xe->mem.defrag.worker);
+}
+
 /**
  * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
  * @xe: The xe device
  *
- * Initialize the list, lock and count used to track BOs whose backing TT
- * pages were allocated at a sub-optimal order.
+ * Initialize the list, lock, count and delayed worker used to track and
+ * defragment BOs whose backing TT pages were allocated at a sub-optimal order.
+ *
+ * The devm action that cancels the worker on teardown is registered separately
+ * by xe_bo_defrag_init(), which must run after the migrate contexts the
+ * worker depends on have been initialized.
  */
 void xe_bo_defrag_init_early(struct xe_device *xe)
 {
 	spin_lock_init(&xe->mem.defrag.lock);
 	INIT_LIST_HEAD(&xe->mem.defrag.list);
 	atomic_set(&xe->mem.defrag.count, 0);
+	xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+	INIT_DELAYED_WORK(&xe->mem.defrag.worker, xe_bo_defrag_worker);
+}
+
+/**
+ * xe_bo_defrag_init() - Register defrag worker teardown
+ * @xe: The xe device
+ *
+ * Register the devm action that disables and drains the defrag worker on
+ * device teardown. This must be called after xe_migrate_init() so that, with
+ * devm's reverse-order cleanup, the worker is stopped before the migrate
+ * contexts it relies on (via xe_bo_defrag_one() -> ttm_bo_validate() ->
+ * xe_bo_move() -> xe_migrate_copy_defrag()) are torn down.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int xe_bo_defrag_init(struct xe_device *xe)
+{
+	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
+}
+
+static void xe_bo_defrag_schedule(struct xe_device *xe)
+{
+	schedule_delayed_work(&xe->mem.defrag.worker,
+			      msecs_to_jiffies(xe->mem.defrag.interval_ms));
 }
 
 static bool xe_bo_needs_defrag(struct xe_bo *bo)
@@ -1079,16 +1147,24 @@ static bool xe_bo_needs_defrag(struct xe_bo *bo)
 static void xe_bo_defrag_add(struct xe_bo *bo)
 {
 	struct xe_device *xe = xe_bo_device(bo);
+	bool kick = false;
 
 	xe_bo_assert_held(bo);
 	xe_assert(xe, xe_bo_needs_defrag(bo));
 
 	scoped_guard(spinlock, &xe->mem.defrag.lock) {
 		if (list_empty(&bo->defrag_link)) {
+			/* Kick the worker when the list transitions to non-empty. */
+			kick = list_empty(&xe->mem.defrag.list);
+			if (kick)
+				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
 			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
 			atomic_inc(&xe->mem.defrag.count);
 		}
 	}
+
+	if (kick)
+		xe_bo_defrag_schedule(xe);
 }
 
 static void __xe_bo_defrag_remove(struct xe_bo *bo)
@@ -1131,6 +1207,218 @@ static void xe_bo_defrag_update(struct xe_bo *bo)
 		xe_bo_defrag_remove(bo);
 }
 
+/*
+ * Attempt to defragment a single BO by forcing a move that reallocates its
+ * backing at the device's beneficial order. @budget bounds how many bytes of
+ * new pages the move may (re)allocate; the remainder of the object is completed
+ * by harvesting the old backing, and the BO stays flagged for a later pass. On
+ * return *@consumed holds the bytes actually (re)allocated (0 on failure or
+ * when nothing needed reallocating). *@needs_more is set to true when the move
+ * succeeded but the BO still has sub-beneficial backing (partial upgrade) and
+ * should be revisited; it is computed under the BO lock so the caller does not
+ * need to re-inspect bo->ttm.ttm after the lock is dropped. Returns 0 if the BO
+ * no longer needs to be tracked (either defragmented or no longer eligible), or
+ * a negative error code if the attempt should be retried later.
+ */
+static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
+			    u64 budget, u64 *consumed, bool *needs_more)
+{
+	struct ttm_operation_ctx ctx = {
+		.gfp_retry_mayfail = true,
+		.defrag = true,
+		.defrag_bytes_remaining = budget,
+	};
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_placement placement;
+	struct ttm_place place;
+	int ret = 0;
+
+	*consumed = 0;
+	*needs_more = false;
+
+	xe_bo_lock(bo, false);
+
+	/* Re-check eligibility under the BO lock. */
+	if (!xe_bo_needs_defrag(bo)) {
+		xe_bo_defrag_remove(bo);
+		goto unlock;
+	}
+
+	xe_place_from_ttm_type(ttm_bo->resource->mem_type, &place);
+	place.flags |= TTM_PL_FLAG_TEMPORARY;
+	placement.num_placement = 1;
+	placement.placement = &place;
+
+	/*
+	 * On success the move reallocates the backing at beneficial order and
+	 * drops the BO from the defrag list. On failure the BO keeps its
+	 * original backing and stays on the list for a later retry.
+	 */
+	ret = ttm_bo_validate(ttm_bo, &placement, &ctx);
+
+	/*
+	 * The pool allocator decremented defrag_bytes_remaining as it
+	 * (re)allocated pages, so the unused budget tells us how much was spent.
+	 * Only meaningful when the move succeeded. A successful move may spend
+	 * nothing - e.g. when the object was already fully upgraded and every
+	 * chunk is harvested (borrowed) from the old backing for free - so a
+	 * zero spend is legitimate and must be charged as zero, not the whole
+	 * budget.
+	 */
+	if (!ret) {
+		s64 spent = (s64)budget - ctx.defrag_bytes_remaining;
+
+		*consumed = spent > 0 ? spent : 0;
+	}
+
+	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
+	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
+
+	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm))
+		*needs_more = true;
+
+unlock:
+	xe_bo_unlock(bo);
+	return ret;
+}
+
+static void xe_bo_defrag_worker(struct work_struct *w)
+{
+	struct delayed_work *dwork = to_delayed_work(w);
+	struct xe_device *xe =
+		container_of(dwork, struct xe_device, mem.defrag.worker);
+	u64 defrag_bytes = 0;
+	bool requeue = false;
+	int idx;
+
+	if (!drm_dev_enter(&xe->drm, &idx))
+		return;
+
+	if (!xe_pm_runtime_get_if_in_use(xe)) {
+		/*
+		 * The device is (runtime) suspending/suspended. Reschedule so
+		 * the pass retries once it is active again: the enqueue-time
+		 * kick in xe_bo_defrag_add() only fires on an empty->non-empty
+		 * transition, so it will not re-arm the worker while the list
+		 * is still non-empty. Requeue here to avoid stranding tracked
+		 * BOs. The get attempt does not resume the device, so this is
+		 * just a lightweight CPU-side timer while suspended.
+		 */
+		if (atomic_read(&xe->mem.defrag.count))
+			xe_bo_defrag_schedule(xe);
+		drm_dev_exit(idx);
+		return;
+	}
+
+	/*
+	 * Process at most XE_BO_DEFRAG_SIZE_LIMIT bytes of newly (re)allocated
+	 * backing per run rather than draining the whole list in one go. Only
+	 * the pages a move actually reallocates are charged - pages harvested
+	 * (borrowed) from the old backing are free and do not count - so the
+	 * budget tracks real defrag work. A single object larger than the
+	 * budget is upgraded in slices across successive runs. Each defrag is
+	 * synchronous and relatively expensive, so bounding the work per run
+	 * keeps the worker from monopolising resources and lets concurrent
+	 * active work make progress; any remaining work is handled by a
+	 * follow-up run scheduled below.
+	 */
+	while (defrag_bytes < XE_BO_DEFRAG_SIZE_LIMIT) {
+		struct xe_bo *first, *bo;
+		u64 consumed = 0;
+		bool needs_more = false;
+		int ret;
+
+		scoped_guard(spinlock, &xe->mem.defrag.lock) {
+			bool again;
+
+			/* Progress, reset interval */
+			if (defrag_bytes)
+				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+
+			do {
+				first = list_first_entry_or_null(&xe->mem.defrag.list,
+								 struct xe_bo,
+								 defrag_link);
+				bo = first ? xe_bo_get_unless_zero(first) : NULL;
+				again = first && !bo;
+				if (again) {
+					list_del_init(&first->defrag_link);
+					atomic_dec(&xe->mem.defrag.count);
+				}
+			} while (again);
+		}
+
+		if (!bo)
+			break;
+
+		/*
+		 * Bound the per-BO move to the run's remaining byte budget. A
+		 * large object is upgraded in budget-sized slices across runs;
+		 * the slices already upgraded come back as free already-
+		 * beneficial harvests, so each run only spends budget on new
+		 * forward progress.
+		 */
+		ret = xe_bo_defrag_one(xe, bo, XE_BO_DEFRAG_SIZE_LIMIT - defrag_bytes,
+				       &consumed, &needs_more);
+		defrag_bytes += consumed;
+
+		if (ret || needs_more) {
+			scoped_guard(spinlock, &xe->mem.defrag.lock) {
+				if (ret)
+					/*
+					 * Abort the pass and retry the whole
+					 * list later, backing off exponentially
+					 * on every failure.
+					 */
+					xe->mem.defrag.interval_ms =
+						min(xe->mem.defrag.interval_ms * 2,
+						    (unsigned int)XE_BO_DEFRAG_INTERVAL_MAX_MS);
+				else
+					/* Progress, reset interval */
+					xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+			}
+
+			xe_bo_put(bo);
+			requeue = true;
+			break;
+		}
+
+		xe_bo_put(bo);
+	}
+
+	/*
+	 * Decide whether to reschedule:
+	 *
+	 *  - The loop hit its per-run limit (defrag_bytes >=
+	 *    XE_BO_DEFRAG_SIZE_LIMIT): we stopped early to spread the work out,
+	 *    so requeue at the default interval (reset here since this run made
+	 *    progress) whenever the list still has entries.
+	 *
+	 *  - The loop aborted on a failed defrag (requeue): retry later at the
+	 *    backed-off interval computed above.
+	 *
+	 *  - The loop drained the list (broke on the empty list): nothing left to
+	 *    do, so stop scheduling. A fresh enqueue will kick the worker again.
+	 */
+	if (defrag_bytes >= XE_BO_DEFRAG_SIZE_LIMIT) {
+		struct xe_bo *bo;
+
+		scoped_guard(spinlock, &xe->mem.defrag.lock) {
+			bo = list_first_entry_or_null(&xe->mem.defrag.list,
+						      struct xe_bo, defrag_link);
+			xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+		}
+
+		if (bo)
+			xe_bo_defrag_schedule(xe);
+	} else if (requeue) {
+		xe_bo_defrag_schedule(xe);
+	}
+
+	xe_pm_runtime_put(xe);
+	drm_dev_exit(idx);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 32211016a722..38d50f8122cb 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -226,6 +226,7 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
 void xe_bo_defrag_init_early(struct xe_device *xe);
+int xe_bo_defrag_init(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index bb93425a5c70..19d38e14c910 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -1069,6 +1069,15 @@ int xe_device_probe(struct xe_device *xe)
 			return err;
 	}
 
+	/*
+	 * Register the defrag worker teardown now that the migrate contexts it
+	 * depends on are initialized, so devm's reverse-order cleanup stops the
+	 * worker before those contexts are torn down.
+	 */
+	err = xe_bo_defrag_init(xe);
+	if (err)
+		return err;
+
 	err = xe_pagefault_init(xe);
 	if (err)
 		return err;
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 9e6244d8f41b..6e68add4d652 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -314,6 +314,17 @@ struct xe_device {
 			 * @mem.defrag.list.
 			 */
 			atomic_t count;
+			/**
+			 * @mem.defrag.worker: Delayed worker that walks
+			 * @mem.defrag.list trying to reallocate BO backing
+			 * store at the device's beneficial order.
+			 */
+			struct delayed_work worker;
+			/**
+			 * @mem.defrag.interval_ms: Reschedule interval for
+			 * @mem.defrag.worker, in milliseconds.
+			 */
+			unsigned int interval_ms;
 		} defrag;
 	} mem;
 
-- 
2.34.1

[PATCH v3 21/33] drm/xe: Add defrag GT stats

[Matthew Brost]

Add observability for the page defragmentation worker.

Introduce four primary-GT statistics counters: defrag_added_count
(BOs queued onto the defrag list after being backed at a sub-optimal
page order), defrag_success_count (BOs fully re-backed at the beneficial
order), defrag_partial_success_count (BOs only partially re-backed at the
beneficial order) defrag_failed_count (attempts that failed and left the
BO on its original backing) and defrag_mb_moved. The
ineligible/already-removed fast path is intentionally not counted as
either success or failure.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c             | 18 +++++++++++++++++-
 drivers/gpu/drm/xe/xe_gt_stats.c       |  5 +++++
 drivers/gpu/drm/xe/xe_gt_stats_types.h | 17 +++++++++++++++++
 3 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 907cfa57481f..073d926ef9ac 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -26,6 +26,7 @@
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
 #include "xe_ggtt.h"
+#include "xe_gt_stats.h"
 #include "xe_map.h"
 #include "xe_migrate.h"
 #include "xe_pat.h"
@@ -1160,6 +1161,8 @@ static void xe_bo_defrag_add(struct xe_bo *bo)
 				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
 			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
 			atomic_inc(&xe->mem.defrag.count);
+			xe_gt_stats_incr(xe_root_mmio_gt(xe),
+					 XE_GT_STATS_ID_DEFRAG_ADDED_COUNT, 1);
 		}
 	}
 
@@ -1274,8 +1277,21 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
 	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
 
-	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm))
+	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {
 		*needs_more = true;
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
+				 1);
+	} else {
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 ret ? XE_GT_STATS_ID_DEFRAG_FAILED_COUNT :
+				       XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT, 1);
+	}
+
+	if (!ret)
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_MB_MOVED,
+				 *consumed >> 20);
 
 unlock:
 	xe_bo_unlock(bo);
diff --git a/drivers/gpu/drm/xe/xe_gt_stats.c b/drivers/gpu/drm/xe/xe_gt_stats.c
index 789397514f3e..68e60d9500c8 100644
--- a/drivers/gpu/drm/xe/xe_gt_stats.c
+++ b/drivers/gpu/drm/xe/xe_gt_stats.c
@@ -154,6 +154,11 @@ static const char *const stat_description[__XE_GT_STATS_NUM_IDS] = {
 	DEF_STAT_STR(PRL_2M_ENTRY_COUNT, "prl_2m_entry_count"),
 	DEF_STAT_STR(PRL_ISSUED_COUNT, "prl_issued_count"),
 	DEF_STAT_STR(PRL_ABORTED_COUNT, "prl_aborted_count"),
+	DEF_STAT_STR(DEFRAG_ADDED_COUNT, "defrag_added_count"),
+	DEF_STAT_STR(DEFRAG_SUCCESS_COUNT, "defrag_success_count"),
+	DEF_STAT_STR(DEFRAG_PARTIAL_SUCCESS_COUNT, "defrag_partial_success_count"),
+	DEF_STAT_STR(DEFRAG_FAILED_COUNT, "defrag_failed_count"),
+	DEF_STAT_STR(DEFRAG_MB_MOVED, "defrag_mb_moved"),
 };
 
 /**
diff --git a/drivers/gpu/drm/xe/xe_gt_stats_types.h b/drivers/gpu/drm/xe/xe_gt_stats_types.h
index 425491bed6c4..c42b494390f8 100644
--- a/drivers/gpu/drm/xe/xe_gt_stats_types.h
+++ b/drivers/gpu/drm/xe/xe_gt_stats_types.h
@@ -122,6 +122,18 @@
  * @XE_GT_STATS_ID_PRL_ABORTED_COUNT: Times the page reclaim process was
  *   aborted.
  *
+ * @XE_GT_STATS_ID_DEFRAG_ADDED_COUNT: Times a buffer object was added to the
+ *   page defragmentation list after being backed at a sub-optimal page order.
+ * @XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT: Times the defrag worker successfully
+ *   re-backed a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT: Times the defrag worker
+ *   partially re-backed a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_FAILED_COUNT: Times the defrag worker failed to
+ *   re-back a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_MB_MOVED: Data (MiB) actually (re)allocated while
+ *   re-backing buffer objects at the beneficial page order. Chunks borrowed
+ *   from the old backing for free are not counted.
+ *
  * @__XE_GT_STATS_NUM_IDS: Number of valid IDs; not a real counter.
  *
  * See Documentation/gpu/xe/xe_gt_stats.rst.
@@ -181,6 +193,11 @@ enum xe_gt_stats_id {
 	XE_GT_STATS_ID_PRL_2M_ENTRY_COUNT,
 	XE_GT_STATS_ID_PRL_ISSUED_COUNT,
 	XE_GT_STATS_ID_PRL_ABORTED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_ADDED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT,
+	XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
+	XE_GT_STATS_ID_DEFRAG_FAILED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_MB_MOVED,
 	/* must be the last entry */
 	__XE_GT_STATS_NUM_IDS,
 };
-- 
2.34.1

[PATCH v3 22/33] drm/xe: Add Kconfig.profile options for BO defrag configuration

[Matthew Brost]

Add Kconfig.profile options to make XE_BO_DEFRAG_* defines configurable:
- DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD: Threshold for TTM reclaim
  backoff
- DRM_XE_BO_DEFRAG_SIZE_LIMIT: Maximum number of bytes to defrag per
  work run
- DRM_XE_BO_DEFRAG_INTERVAL_MS: Default delay before defrag worker runs
- DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS: Upper bound for defrag worker
  interval

Update xe_bo.c to use these Kconfig options as defaults via #ifdef
guards, maintaining backward compatibility with hardcoded values when
not configured.

Additionally, disable defrag completely when XE_BO_DEFRAG_SIZE_LIMIT is
set less than 2M by returning early from xe_bo_defrag_update() (and
xe_bo_defrag_remove()) so BOs are never added to the defrag list or the
worker kicked; xe_bo_defrag_add() and xe_bo_defrag_schedule() assert the
limit is at least 2M.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-haiku-4.5
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/Kconfig.profile | 40 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.c         | 26 +++++++++++++++++++
 2 files changed, 66 insertions(+)

diff --git a/drivers/gpu/drm/xe/Kconfig.profile b/drivers/gpu/drm/xe/Kconfig.profile
index e07517d120e0..e0aade41f53e 100644
--- a/drivers/gpu/drm/xe/Kconfig.profile
+++ b/drivers/gpu/drm/xe/Kconfig.profile
@@ -74,3 +74,43 @@ config DRM_XE_ENABLE_SCHEDTIMEOUT_LIMIT
 	  to apply to applicable user. For elevated user, all above MIN
 	  and MAX values will apply when this configuration is enable to
 	  apply limitation. By default limitation is applied.
+
+config DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+	int "BO defrag reclaim backoff threshold"
+	default 2
+	range 1 1000
+	help
+	  Once this many BOs are tracked on the device defrag list (i.e. were
+	  backed with a sub-optimal page order), request that the TTM pool backs
+	  off from aggressive reclaim at the beneficial order during populate,
+	  so that allocations make forward progress instead of stalling.
+
+config DRM_XE_BO_DEFRAG_SIZE_LIMIT
+	int "Maximum bytes to defrag per work run"
+	default 33554432
+	range 0 1073741824
+	help
+	  Maximum number of bytes the defrag worker will process in a single run
+	  before yielding and rescheduling itself. Set to less than large page
+	  size (2M) to disable defrag. A defrag move synchronously reallocates
+	  and re-copies a BO's backing store, which is not free. If a large
+	  number of BOs become eligible at once, processing them all in one
+	  worker run would hold things up for a long, unbounded stretch.
+	  Instead, cap the work done per run and requeue, spreading the defrag
+	  effort out over time.
+
+config DRM_XE_BO_DEFRAG_INTERVAL_MS
+	int "Default delay before defrag worker run (ms)"
+	default 25
+	range 1 1000
+	help
+	  Default delay before (re)running the defrag worker, in milliseconds.
+
+config DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+	int "Upper bound for defrag worker interval (ms)"
+	default 15000
+	range 100 1000000
+	help
+	  Upper bound for the (exponentially backed off) defrag worker interval,
+	  in milliseconds, so repeated failures don't push the retry arbitrarily
+	  far out. 15000 ms = 15 seconds.
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 073d926ef9ac..0bdd1c920404 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -49,7 +49,11 @@
  * aggressive reclaim at the beneficial order during populate, so that
  * allocations make forward progress instead of stalling.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	CONFIG_DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+#else
 #define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+#endif
 
 /*
  * Maximum number of bytes of newly (re)allocated backing the defrag worker will
@@ -65,17 +69,31 @@
  * Only pages a move truly reallocates are charged; pages harvested from the old
  * backing are free, so an object larger than the budget is upgraded in
  * budget-sized slices across runs.
+ *
+ * Set to 0 to disable defrag completely.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_SIZE_LIMIT
+#define XE_BO_DEFRAG_SIZE_LIMIT			CONFIG_DRM_XE_BO_DEFRAG_SIZE_LIMIT
+#else
 #define XE_BO_DEFRAG_SIZE_LIMIT			SZ_32M
+#endif
 
 /* Default delay before (re)running the defrag worker, in milliseconds. */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MS
+#define XE_BO_DEFRAG_INTERVAL_MS		CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MS
+#else
 #define XE_BO_DEFRAG_INTERVAL_MS		25
+#endif
 
 /*
  * Upper bound for the (exponentially backed off) defrag worker interval, in
  * milliseconds, so repeated failures don't push the retry arbitrarily far out.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+#define XE_BO_DEFRAG_INTERVAL_MAX_MS		CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+#else
 #define XE_BO_DEFRAG_INTERVAL_MAX_MS		15000	/* 15 seconds */
+#endif
 
 static void xe_bo_defrag_worker(struct work_struct *w);
 static void xe_place_from_ttm_type(u32 mem_type, struct ttm_place *place);
@@ -1126,6 +1144,7 @@ int xe_bo_defrag_init(struct xe_device *xe)
 
 static void xe_bo_defrag_schedule(struct xe_device *xe)
 {
+	xe_assert(xe, XE_BO_DEFRAG_SIZE_LIMIT >= SZ_2M);
 	schedule_delayed_work(&xe->mem.defrag.worker,
 			      msecs_to_jiffies(xe->mem.defrag.interval_ms));
 }
@@ -1152,6 +1171,7 @@ static void xe_bo_defrag_add(struct xe_bo *bo)
 
 	xe_bo_assert_held(bo);
 	xe_assert(xe, xe_bo_needs_defrag(bo));
+	xe_assert(xe, XE_BO_DEFRAG_SIZE_LIMIT >= SZ_2M);
 
 	scoped_guard(spinlock, &xe->mem.defrag.lock) {
 		if (list_empty(&bo->defrag_link)) {
@@ -1185,6 +1205,9 @@ static void xe_bo_defrag_remove(struct xe_bo *bo)
 {
 	xe_bo_assert_held(bo);
 
+	if (XE_BO_DEFRAG_SIZE_LIMIT < SZ_2M)
+		return;
+
 	if (list_empty(&bo->defrag_link))
 		return;
 
@@ -1204,6 +1227,9 @@ static void xe_bo_defrag_update(struct xe_bo *bo)
 {
 	xe_bo_assert_held(bo);
 
+	if (XE_BO_DEFRAG_SIZE_LIMIT < SZ_2M)
+		return;
+
 	if (xe_bo_needs_defrag(bo))
 		xe_bo_defrag_add(bo);
 	else
-- 
2.34.1

[PATCH v3 23/33] drm/xe: Defrag using out-of-lock page preallocation

[Matthew Brost]

Convert xe_bo_defrag_one() to a three-phase scheme that moves the
expensive beneficial-order allocations out of the BO dma-resv lock:

 1. Under the lock, re-check eligibility, capture the tt cpu-caching and
    read the exact sub-optimal page count (ttm_tt_suboptimal_pages()),
    bounded by the run budget.
 2. Drop the lock and preallocate that many beneficial-order chunks via
    ttm_pool_prealloc_fill(); reclaim/compaction stalls and the cpu
    cache mode change both happen here, outside the lock.
 3. Re-take the lock, re-check, and validate consuming the prealloc bag.
    Any shortfall harvests the old backing instead of reclaiming, so the
    move never stalls under the lock; the BO re-queues for a later pass.

With both allocation and caching hoisted out, the BO-lock held time
drops to copy-bound (median ~0.27ms, p95 ~2ms); the remaining tail is
lock-wait behind active rendering, as intended.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Use DIV_ROUND_UP_ULL() for the 64-bit preallocation chunk count
   (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c | 46 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 45 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 0bdd1c920404..8b89d34f37d4 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1258,16 +1258,56 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 		.defrag_bytes_remaining = budget,
 	};
 	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_pool *pool = &ttm_bo->bdev->pool;
+	struct ttm_pool_prealloc pp = {};
 	struct ttm_placement placement;
 	struct ttm_place place;
+	enum ttm_caching tt_caching;
+	unsigned int order, want = 0;
 	int ret = 0;
 
 	*consumed = 0;
 	*needs_more = false;
 
+	/*
+	 * Phase 1: under the BO lock, re-check eligibility and estimate how
+	 * many beneficial-order chunks the move will (re)allocate, bounded by
+	 * the run's byte budget. Then drop the lock so the high-order, possibly
+	 * reclaim/compaction stalling allocations happen unlocked.
+	 */
+	xe_bo_lock(bo, false);
+	if (!xe_bo_needs_defrag(bo)) {
+		xe_bo_defrag_remove(bo);
+		xe_bo_unlock(bo);
+		goto out;
+	}
+	tt_caching = bo->ttm.ttm->caching;
+	order = ttm_pool_prealloc_order(pool);
+	if (order && ttm_bo->ttm) {
+		u32 suboptimal = ttm_tt_suboptimal_pages(ttm_bo->ttm);
+		u64 cap = min_t(u64, budget, xe_bo_size(bo)) >> PAGE_SHIFT;
+
+		/* Prealloc only the beneficial-order chunks the move replaces. */
+		want = DIV_ROUND_UP_ULL(min_t(u64, suboptimal, cap), 1UL << order);
+	}
+	xe_bo_unlock(bo);
+
+	/* Phase 2: preallocate outside the lock; */
+	if (want) {
+		ret = ttm_pool_prealloc_fill(pool, tt_caching, &pp, want);
+		if (ret || !pp.count) {
+			ret = ret ?: -ENOMEM;
+			xe_gt_stats_incr(xe_root_mmio_gt(xe),
+					 XE_GT_STATS_ID_DEFRAG_FAILED_COUNT, 1);
+			goto out_err;
+		}
+	}
+
+	/*
+	 * Phase 3: re-take the lock, re-check, and validate using the prealloc.
+	 */
 	xe_bo_lock(bo, false);
 
-	/* Re-check eligibility under the BO lock. */
 	if (!xe_bo_needs_defrag(bo)) {
 		xe_bo_defrag_remove(bo);
 		goto unlock;
@@ -1277,6 +1317,7 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	place.flags |= TTM_PL_FLAG_TEMPORARY;
 	placement.num_placement = 1;
 	placement.placement = &place;
+	ctx.prealloc = &pp;
 
 	/*
 	 * On success the move reallocates the backing at beneficial order and
@@ -1321,6 +1362,9 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 unlock:
 	xe_bo_unlock(bo);
+out_err:
+	ttm_pool_prealloc_fini(pool, &pp);
+out:
 	return ret;
 }
 
-- 
2.34.1

[PATCH v3 24/33] drm/xe: Add defrag profiling tracepoints

[Matthew Brost]

Add two tracepoints to profile the page defragmentation worker:

 - xe_bo_defrag_worker: emitted on worker entry and exit with the
   current defrag list count and the bytes processed in the run.
 - xe_bo_defrag_one: emitted per BO with the run budget, the prealloc
   fill ratio (got/want) and the lock-held, lock-wait and prealloc
   durations, plus the bytes consumed and the result.

These break down the cost of a defrag attempt into its locked vs
unlocked phases, making it possible to confirm that the expensive
beneficial-order allocation and cpu cache mode change happen outside
the BO dma-resv lock and that the residual lock-held time is copy-bound
(with the tail dominated by lock-wait behind active rendering).

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Move the ns-to-us division into TP_fast_assign() to avoid a 64-bit
   divide in TP_printk() (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c       | 18 +++++++++-
 drivers/gpu/drm/xe/xe_trace_bo.h | 59 ++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 8b89d34f37d4..25e5b72f870c 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1263,7 +1263,9 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	struct ttm_placement placement;
 	struct ttm_place place;
 	enum ttm_caching tt_caching;
-	unsigned int order, want = 0;
+	unsigned int order, want = 0, got = 0;
+	u64 lock_ns = 0, prealloc_ns = 0, lockwait_ns = 0;
+	ktime_t t0;
 	int ret = 0;
 
 	*consumed = 0;
@@ -1294,6 +1296,7 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 	/* Phase 2: preallocate outside the lock; */
 	if (want) {
+		t0 = ktime_get();
 		ret = ttm_pool_prealloc_fill(pool, tt_caching, &pp, want);
 		if (ret || !pp.count) {
 			ret = ret ?: -ENOMEM;
@@ -1301,12 +1304,16 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 					 XE_GT_STATS_ID_DEFRAG_FAILED_COUNT, 1);
 			goto out_err;
 		}
+		prealloc_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
 	}
 
 	/*
 	 * Phase 3: re-take the lock, re-check, and validate using the prealloc.
 	 */
+	t0 = ktime_get();
 	xe_bo_lock(bo, false);
+	lockwait_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
+	t0 = ktime_get();
 
 	if (!xe_bo_needs_defrag(bo)) {
 		xe_bo_defrag_remove(bo);
@@ -1362,9 +1369,13 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 unlock:
 	xe_bo_unlock(bo);
+	lock_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
+	got = pp.used;
 out_err:
 	ttm_pool_prealloc_fini(pool, &pp);
 out:
+	trace_xe_bo_defrag_one(bo, budget, want, got, lock_ns, prealloc_ns,
+			       lockwait_ns, *consumed, ret);
 	return ret;
 }
 
@@ -1396,6 +1407,9 @@ static void xe_bo_defrag_worker(struct work_struct *w)
 		return;
 	}
 
+	trace_xe_bo_defrag_worker(xe, 0, atomic_read(&xe->mem.defrag.count),
+				  true);
+
 	/*
 	 * Process at most XE_BO_DEFRAG_SIZE_LIMIT bytes of newly (re)allocated
 	 * backing per run rather than draining the whole list in one go. Only
@@ -1503,6 +1517,8 @@ static void xe_bo_defrag_worker(struct work_struct *w)
 
 	xe_pm_runtime_put(xe);
 	drm_dev_exit(idx);
+	trace_xe_bo_defrag_worker(xe, defrag_bytes,
+				  atomic_read(&xe->mem.defrag.count), false);
 }
 
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
diff --git a/drivers/gpu/drm/xe/xe_trace_bo.h b/drivers/gpu/drm/xe/xe_trace_bo.h
index 86323cf3be2c..b159d37d39a1 100644
--- a/drivers/gpu/drm/xe/xe_trace_bo.h
+++ b/drivers/gpu/drm/xe/xe_trace_bo.h
@@ -85,6 +85,65 @@ TRACE_EVENT(xe_bo_move,
 		      __get_str(new_placement_name), __get_str(device_id))
 );
 
+TRACE_EVENT(xe_bo_defrag_worker,
+	    TP_PROTO(struct xe_device *xe, u64 defrag_bytes, int count,
+		     bool enter),
+	    TP_ARGS(xe, defrag_bytes, count, enter),
+	    TP_STRUCT__entry(
+		     __string(dev, dev_name(xe->drm.dev))
+		     __field(u64, defrag_bytes)
+		     __field(int, count)
+		     __field(bool, enter)
+		     ),
+	    TP_fast_assign(
+		   __assign_str(dev);
+		   __entry->defrag_bytes = defrag_bytes;
+		   __entry->count = count;
+		   __entry->enter = enter;
+		   ),
+	    TP_printk("dev=%s %s defrag_bytes=%llu count=%d",
+		      __get_str(dev), __entry->enter ? "enter" : "exit",
+		      __entry->defrag_bytes, __entry->count)
+);
+
+TRACE_EVENT(xe_bo_defrag_one,
+	    TP_PROTO(struct xe_bo *bo, u64 budget, unsigned int prealloc_want,
+		     unsigned int prealloc_got, u64 lock_ns, u64 prealloc_ns,
+		     u64 lockwait_ns, u64 consumed, int ret),
+	    TP_ARGS(bo, budget, prealloc_want, prealloc_got, lock_ns,
+		    prealloc_ns, lockwait_ns, consumed, ret),
+	    TP_STRUCT__entry(
+		     __string(dev, __dev_name_bo(bo))
+		     __field(size_t, size)
+		     __field(u64, budget)
+		     __field(unsigned int, prealloc_want)
+		     __field(unsigned int, prealloc_got)
+		     __field(u64, lock_us)
+		     __field(u64, prealloc_us)
+		     __field(u64, lockwait_us)
+		     __field(u64, consumed)
+		     __field(int, ret)
+		     ),
+	    TP_fast_assign(
+		   __assign_str(dev);
+		   __entry->size = xe_bo_size(bo);
+		   __entry->budget = budget;
+		   __entry->prealloc_want = prealloc_want;
+		   __entry->prealloc_got = prealloc_got;
+		   __entry->lock_us = div_u64(lock_ns, 1000);
+		   __entry->prealloc_us = div_u64(prealloc_ns, 1000);
+		   __entry->lockwait_us = div_u64(lockwait_ns, 1000);
+		   __entry->consumed = consumed;
+		   __entry->ret = ret;
+		   ),
+	    TP_printk("dev=%s size=%zu budget=%llu prealloc=%u/%u lock_us=%llu lockwait_us=%llu prealloc_us=%llu consumed=%llu ret=%d",
+		      __get_str(dev), __entry->size, __entry->budget,
+		      __entry->prealloc_got, __entry->prealloc_want,
+		      __entry->lock_us, __entry->lockwait_us,
+		      __entry->prealloc_us, __entry->consumed,
+		      __entry->ret)
+);
+
 DECLARE_EVENT_CLASS(xe_vma,
 		    TP_PROTO(struct xe_vma *vma),
 		    TP_ARGS(vma),
-- 
2.34.1

[PATCH v3 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock

[Matthew Brost]

xe_gem_create_ioctl() populates a buffer object's backing store inside
xe_validation_guard(), i.e. under the dma-resv lock.  For a system-memory
buffer that means the whole page allocation - potentially stalling in
reclaim or compaction - happens in the critical section, holding the
reservation and blocking anything that contends on it.

Use the TTM full-preallocation mechanism to move that work out of the
lock.  For a system-only, non-deferred user BO, allocate the entire tt
backing up front, before entering the validation guard, and hand the bag
down to the populate so it merely installs the pages:

  - Thread an optional struct ttm_pool_prealloc through xe_bo_create_user()
    -> __xe_bo_create_locked() -> xe_bo_init_locked(), which stashes it in
    ttm_operation_ctx::prealloc.  All other callers pass NULL.

  - In xe_gem_create_ioctl(), for XE_BO_FLAG_SYSTEM BOs that are not VRAM,
    stolen or deferred-backing, call ttm_pool_prealloc_fill_full() before
    the guard and ttm_pool_prealloc_fini() after it.  The bag size covers
    both the data pages and the appended CCS backup pages, matching the tt
    layout in xe_ttm_tt_create(), and the caching mirrors the same path.

The fill is best-effort: on a short fill (or if it is skipped) the pool
allocates the remainder in-line as before, so this is purely an
optimisation with no behavioural change to the created BO.  fini() only
releases the unconsumed tail, so it is safe on every success, error and
OOM-retry path (consumed pages are owned by the tt).

The fill requests reclaim backoff for higher-order chunks
(@beneficial_reclaim_backoff): Xe runs a background defragmenter that
upgrades suboptimal pages to beneficial order in place, so there is no
need to stall this user context reclaiming/compacting for contiguity.

To size the preallocation before the xe_bo exists, factor the CCS-pages
decision in xe_bo_needs_ccs_pages() into a parameterised helper,
__xe_bo_needs_ccs_pages(), and query it directly from the ioctl.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Preallocate only when the backing fits the available TT space
   (Sashiko)
---
 drivers/gpu/drm/xe/tests/xe_bo.c      |   8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c |   2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c |   6 +-
 drivers/gpu/drm/xe/xe_bo.c            | 139 +++++++++++++++++++++-----
 drivers/gpu/drm/xe/xe_bo.h            |   8 +-
 drivers/gpu/drm/xe/xe_dma_buf.c       |   2 +-
 6 files changed, 128 insertions(+), 37 deletions(-)

diff --git a/drivers/gpu/drm/xe/tests/xe_bo.c b/drivers/gpu/drm/xe/tests/xe_bo.c
index 49c95ed67d7e..78781317b420 100644
--- a/drivers/gpu/drm/xe/tests/xe_bo.c
+++ b/drivers/gpu/drm/xe/tests/xe_bo.c
@@ -141,7 +141,7 @@ static void ccs_test_run_tile(struct xe_device *xe, struct xe_tile *tile,
 		kunit_info(test, "Testing system memory\n");
 
 	bo = xe_bo_create_user(xe, NULL, SZ_1M, DRM_XE_GEM_CPU_CACHING_WC,
-			       bo_flags, exec);
+			       bo_flags, NULL, exec);
 	if (IS_ERR(bo)) {
 		KUNIT_FAIL(test, "Failed to create bo.\n");
 		return;
@@ -220,7 +220,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
 		xe_vm_lock(vm, false);
 		bo = xe_bo_create_user(xe, vm, 0x10000,
 				       DRM_XE_GEM_CPU_CACHING_WC,
-				       bo_flags, exec);
+				       bo_flags, NULL, exec);
 		xe_vm_unlock(vm);
 		if (IS_ERR(bo)) {
 			KUNIT_FAIL(test, "bo create err=%pe\n", bo);
@@ -229,7 +229,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
 
 		external = xe_bo_create_user(xe, NULL, 0x10000,
 					     DRM_XE_GEM_CPU_CACHING_WC,
-					     bo_flags, NULL);
+					     bo_flags, NULL, NULL);
 		if (IS_ERR(external)) {
 			KUNIT_FAIL(test, "external bo create err=%pe\n", external);
 			goto cleanup_bo;
@@ -494,7 +494,7 @@ static int shrink_test_run_device(struct xe_device *xe)
 		/* We can create bos using WC caching here. But it is slower. */
 		bo = xe_bo_create_user(xe, NULL, XE_BO_SHRINK_SIZE,
 				       DRM_XE_GEM_CPU_CACHING_WB,
-				       XE_BO_FLAG_SYSTEM, NULL);
+				       XE_BO_FLAG_SYSTEM, NULL, NULL);
 		if (IS_ERR(bo)) {
 			if (bo != ERR_PTR(-ENOMEM) && bo != ERR_PTR(-ENOSPC) &&
 			    bo != ERR_PTR(-EINTR) && bo != ERR_PTR(-ERESTARTSYS))
diff --git a/drivers/gpu/drm/xe/tests/xe_dma_buf.c b/drivers/gpu/drm/xe/tests/xe_dma_buf.c
index 0be8440b3976..e4fdc247c0c7 100644
--- a/drivers/gpu/drm/xe/tests/xe_dma_buf.c
+++ b/drivers/gpu/drm/xe/tests/xe_dma_buf.c
@@ -127,7 +127,7 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
 
 	kunit_info(test, "running %s\n", __func__);
 	bo = xe_bo_create_user(xe, NULL, size, DRM_XE_GEM_CPU_CACHING_WC,
-			       params->mem_mask, NULL);
+			       params->mem_mask, NULL, NULL);
 	if (IS_ERR(bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
 			   PTR_ERR(bo));
diff --git a/drivers/gpu/drm/xe/tests/xe_migrate.c b/drivers/gpu/drm/xe/tests/xe_migrate.c
index 3c1be809be82..41de648b2c86 100644
--- a/drivers/gpu/drm/xe/tests/xe_migrate.c
+++ b/drivers/gpu/drm/xe/tests/xe_migrate.c
@@ -647,7 +647,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 				   DRM_XE_GEM_CPU_CACHING_WC,
 				   XE_BO_FLAG_SYSTEM |
 				   XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				   XE_BO_FLAG_PINNED, NULL);
+				   XE_BO_FLAG_PINNED, NULL, NULL);
 
 	if (IS_ERR(sys_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@@ -673,7 +673,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 	ccs_bo = xe_bo_create_user(xe, NULL, SZ_4M,
 				   DRM_XE_GEM_CPU_CACHING_WC,
 				   bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				   XE_BO_FLAG_PINNED, NULL);
+				   XE_BO_FLAG_PINNED, NULL, NULL);
 
 	if (IS_ERR(ccs_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@@ -698,7 +698,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 	vram_bo = xe_bo_create_user(xe, NULL, SZ_4M,
 				    DRM_XE_GEM_CPU_CACHING_WC,
 				    bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				    XE_BO_FLAG_PINNED, NULL);
+				    XE_BO_FLAG_PINNED, NULL, NULL);
 	if (IS_ERR(vram_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
 			   PTR_ERR(vram_bo));
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 25e5b72f870c..a646a72122d4 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -1948,7 +1948,7 @@ int xe_bo_notifier_prepare_pinned(struct xe_bo *bo)
 		backup = xe_bo_init_locked(xe, NULL, NULL, bo->ttm.base.resv, NULL, xe_bo_size(bo),
 					   DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
 					   XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-					   XE_BO_FLAG_PINNED, &exec);
+					   XE_BO_FLAG_PINNED, NULL, &exec);
 		if (IS_ERR(backup)) {
 			drm_exec_retry_on_contention(&exec);
 			ret = PTR_ERR(backup);
@@ -2089,7 +2089,7 @@ int xe_bo_evict_pinned(struct xe_bo *bo)
 						   xe_bo_size(bo),
 						   DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel,
 						   XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-						   XE_BO_FLAG_PINNED, &exec);
+						   XE_BO_FLAG_PINNED, NULL, &exec);
 			if (IS_ERR(backup)) {
 				drm_exec_retry_on_contention(&exec);
 				ret = PTR_ERR(backup);
@@ -2884,6 +2884,8 @@ void xe_bo_free(struct xe_bo *bo)
  * @cpu_caching: The cpu caching used for system memory backing store.
  * @type: The TTM buffer object type.
  * @flags: XE_BO_FLAG_ flags.
+ * @prealloc: Optional pages preallocated outside the dma-resv lock to back the
+ * bo (see ttm_operation_ctx::prealloc), or NULL.
  * @exec: The drm_exec transaction to use for exhaustive eviction.
  *
  * Initialize or create an xe buffer object. On failure, any allocated buffer
@@ -2895,12 +2897,14 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 				struct xe_tile *tile, struct dma_resv *resv,
 				struct ttm_lru_bulk_move *bulk, size_t size,
 				u16 cpu_caching, enum ttm_bo_type type,
-				u32 flags, struct drm_exec *exec)
+				u32 flags, struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec)
 {
 	struct ttm_operation_ctx ctx = {
 		.interruptible = true,
 		.no_wait_gpu = false,
 		.gfp_retry_mayfail = true,
+		.prealloc = prealloc,
 	};
 	struct ttm_placement *placement;
 	uint32_t alignment;
@@ -3077,7 +3081,8 @@ __xe_bo_create_locked(struct xe_device *xe,
 		      struct xe_tile *tile, struct xe_vm *vm,
 		      size_t size, u64 start, u64 end,
 		      u16 cpu_caching, enum ttm_bo_type type, u32 flags,
-		      u64 alignment, struct drm_exec *exec)
+		      u64 alignment, struct ttm_pool_prealloc *prealloc,
+		      struct drm_exec *exec)
 {
 	struct xe_bo *bo = NULL;
 	int err;
@@ -3102,7 +3107,7 @@ __xe_bo_create_locked(struct xe_device *xe,
 			       vm && !xe_vm_in_fault_mode(vm) &&
 			       flags & XE_BO_FLAG_USER ?
 			       &vm->lru_bulk_move : NULL, size,
-			       cpu_caching, type, flags, exec);
+			       cpu_caching, type, flags, prealloc, exec);
 	if (IS_ERR(bo))
 		return bo;
 
@@ -3177,7 +3182,7 @@ struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile,
 				  struct drm_exec *exec)
 {
 	return __xe_bo_create_locked(xe, tile, vm, size, 0, ~0ULL, 0, type,
-				     flags, 0, exec);
+				     flags, 0, NULL, exec);
 }
 
 static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *tile,
@@ -3193,7 +3198,7 @@ static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *til
 	xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {.interruptible = intr},
 			    ret) {
 		bo = __xe_bo_create_locked(xe, tile, NULL, size, 0, ~0ULL,
-					   cpu_caching, type, flags, alignment, &exec);
+					   cpu_caching, type, flags, alignment, NULL, &exec);
 		drm_exec_retry_on_contention(&exec);
 		if (IS_ERR(bo)) {
 			ret = PTR_ERR(bo);
@@ -3213,6 +3218,9 @@ static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *til
  * @size: The storage size to use for the bo.
  * @cpu_caching: The caching mode to be used for system backing store.
  * @flags: XE_BO_FLAG_ flags.
+ * @prealloc: Optional pages preallocated outside the dma-resv lock to back the
+ * bo (see ttm_operation_ctx::prealloc), or NULL. Only consumed on the locked
+ * (@vm or @exec) path.
  * @exec: The drm_exec transaction to use for exhaustive eviction, or NULL
  * if such a transaction should be initiated by the call.
  *
@@ -3223,7 +3231,8 @@ static struct xe_bo *xe_bo_create_novm(struct xe_device *xe, struct xe_tile *til
 struct xe_bo *xe_bo_create_user(struct xe_device *xe,
 				struct xe_vm *vm, size_t size,
 				u16 cpu_caching,
-				u32 flags, struct drm_exec *exec)
+				u32 flags, struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec)
 {
 	struct xe_bo *bo;
 
@@ -3233,7 +3242,7 @@ struct xe_bo *xe_bo_create_user(struct xe_device *xe,
 		xe_assert(xe, exec);
 		bo = __xe_bo_create_locked(xe, NULL, vm, size, 0, ~0ULL,
 					   cpu_caching, ttm_bo_type_device,
-					   flags, 0, exec);
+					   flags, 0, prealloc, exec);
 		if (!IS_ERR(bo))
 			xe_bo_unlock_vm_held(bo);
 	} else {
@@ -3272,7 +3281,7 @@ struct xe_bo *xe_bo_create_pin_range_novm(struct xe_device *xe, struct xe_tile *
 
 	xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, err) {
 		bo = __xe_bo_create_locked(xe, tile, NULL, size, start, end,
-					   0, type, flags, 0, &exec);
+					   0, type, flags, 0, NULL, &exec);
 		if (IS_ERR(bo)) {
 			drm_exec_retry_on_contention(&exec);
 			err = PTR_ERR(bo);
@@ -3311,7 +3320,7 @@ static struct xe_bo *xe_bo_create_pin_map_at_aligned(struct xe_device *xe,
 
 	bo = __xe_bo_create_locked(xe, tile, vm, size, start, end, 0, type,
 				   flags | XE_BO_FLAG_NEEDS_CPU_ACCESS | XE_BO_FLAG_PINNED,
-				   alignment, exec);
+				   alignment, NULL, exec);
 	if (IS_ERR(bo))
 		return bo;
 
@@ -3926,6 +3935,10 @@ static int gem_create_user_extensions(struct xe_device *xe, struct xe_bo *bo,
 	return 0;
 }
 
+static bool __xe_bo_needs_ccs_pages(struct xe_device *xe,
+				    enum ttm_bo_type type, u32 flags,
+				    u16 cpu_caching);
+
 int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			struct drm_file *file)
 {
@@ -3936,6 +3949,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	struct drm_exec exec;
 	struct xe_vm *vm = NULL;
 	struct xe_bo *bo;
+	struct ttm_pool_prealloc prealloc = {};
+	bool have_prealloc = false;
 	unsigned int bo_flags;
 	u32 handle;
 	int err;
@@ -4018,6 +4033,64 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			return -ENOENT;
 	}
 
+	/*
+	 * For system-only BOs, preallocate the whole backing outside the
+	 * dma-resv lock so the populate under the validation guard does not
+	 * stall in reclaim/compaction. The caching mirrors the system-BO path
+	 * in xe_ttm_tt_create(); the pool falls back to in-line allocation for
+	 * any shortfall, so a failed fill is harmless.
+	 */
+	if ((bo_flags & XE_BO_FLAG_SYSTEM) &&
+	    !(bo_flags & (XE_BO_FLAG_DEFER_BACKING | XE_BO_FLAG_VRAM_MASK |
+			  XE_BO_FLAG_STOLEN))) {
+		struct ttm_resource_manager *tt_man =
+			ttm_manager_type(&xe->ttm, XE_PL_TT);
+		enum ttm_caching caching = ttm_cached;
+		unsigned long num_pages = args->size >> PAGE_SHIFT;
+
+		if (!IS_DGFX(xe) &&
+		    args->cpu_caching == DRM_XE_GEM_CPU_CACHING_WC)
+			caching = ttm_write_combined;
+
+		/*
+		 * Cover the CCS backup pages appended to the tt (see
+		 * xe_ttm_tt_create()) so the whole tt is preallocated.
+		 */
+		if (__xe_bo_needs_ccs_pages(xe, ttm_bo_type_device, bo_flags,
+					    args->cpu_caching))
+			num_pages += DIV_ROUND_UP(xe_device_ccs_bytes(xe,
+								      args->size),
+						  PAGE_SIZE);
+
+		/*
+		 * Only prealloc when the backing fits the currently-available
+		 * TT space (manager size minus current usage). This skips the
+		 * up-front reclaim/compaction both for over-large,
+		 * user-controlled sizes and when TT is already near full; the
+		 * populate under the validation guard handles those cases.
+		 */
+		if (tt_man) {
+			u64 used = ttm_resource_manager_usage(tt_man);
+			u64 shift_size = tt_man->size << PAGE_SHIFT;
+			u64 avail = shift_size > used ? shift_size - used : 0;
+
+			if ((u64)num_pages << PAGE_SHIFT <= avail) {
+				/*
+				 * Back off aggressive reclaim/compaction for
+				 * higher-order chunks here: Xe runs a background
+				 * defragmenter that upgrades suboptimal pages to
+				 * beneficial order in place later, so there is no
+				 * need to stall this user context reclaiming for
+				 * contiguity.
+				 */
+				ttm_pool_prealloc_fill_full(&xe->ttm.pool,
+							    caching, &prealloc,
+							    num_pages, true);
+				have_prealloc = true;
+			}
+		}
+	}
+
 	err = 0;
 	xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {.interruptible = true},
 			    err) {
@@ -4028,7 +4101,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 				break;
 		}
 		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
-				       bo_flags, &exec);
+				       bo_flags, have_prealloc ? &prealloc : NULL,
+				       &exec);
 		drm_exec_retry_on_contention(&exec);
 		if (IS_ERR(bo)) {
 			err = PTR_ERR(bo);
@@ -4036,6 +4110,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			break;
 		}
 	}
+	if (have_prealloc)
+		ttm_pool_prealloc_fini(&xe->ttm.pool, &prealloc);
 	if (err)
 		goto out_vm;
 
@@ -4331,21 +4407,19 @@ int xe_bo_evict(struct xe_bo *bo, struct drm_exec *exec)
 	return 0;
 }
 
-/**
- * xe_bo_needs_ccs_pages - Whether a bo needs to back up CCS pages when
- * placed in system memory.
- * @bo: The xe_bo
- *
- * Return: true if extra pages need to be allocated, false otherwise.
+/*
+ * Whether a bo with the given attributes needs to back up CCS pages when placed
+ * in system memory. Split out so callers can query before the xe_bo exists (see
+ * xe_gem_create_ioctl() preallocation).
  */
-bool xe_bo_needs_ccs_pages(struct xe_bo *bo)
+static bool __xe_bo_needs_ccs_pages(struct xe_device *xe,
+				    enum ttm_bo_type type, u32 flags,
+				    u16 cpu_caching)
 {
-	struct xe_device *xe = xe_bo_device(bo);
-
 	if (GRAPHICS_VER(xe) >= 20 && IS_DGFX(xe))
 		return false;
 
-	if (!xe_device_has_flat_ccs(xe) || bo->ttm.type != ttm_bo_type_device)
+	if (!xe_device_has_flat_ccs(xe) || type != ttm_bo_type_device)
 		return false;
 
 	/* On discrete GPUs, if the GPU can access this buffer from
@@ -4353,24 +4427,37 @@ bool xe_bo_needs_ccs_pages(struct xe_bo *bo)
 	 * can't be used since there's no CCS storage associated with
 	 * non-VRAM addresses.
 	 */
-	if (IS_DGFX(xe) && (bo->flags & XE_BO_FLAG_SYSTEM))
+	if (IS_DGFX(xe) && (flags & XE_BO_FLAG_SYSTEM))
 		return false;
 
 	/* Check if userspace explicitly requested no compression */
-	if (bo->flags & XE_BO_FLAG_NO_COMPRESSION)
+	if (flags & XE_BO_FLAG_NO_COMPRESSION)
 		return false;
 
 	/*
 	 * For WB (Write-Back) CPU caching mode, check if the device
 	 * supports WB compression with coherency.
 	 */
-	if (bo->cpu_caching == DRM_XE_GEM_CPU_CACHING_WB &&
+	if (cpu_caching == DRM_XE_GEM_CPU_CACHING_WB &&
 	    xe->pat.idx[XE_CACHE_WB_COMPRESSION] == XE_PAT_INVALID_IDX)
 		return false;
 
 	return true;
 }
 
+/**
+ * xe_bo_needs_ccs_pages - Whether a bo needs to back up CCS pages when
+ * placed in system memory.
+ * @bo: The xe_bo
+ *
+ * Return: true if extra pages need to be allocated, false otherwise.
+ */
+bool xe_bo_needs_ccs_pages(struct xe_bo *bo)
+{
+	return __xe_bo_needs_ccs_pages(xe_bo_device(bo), bo->ttm.type,
+				       bo->flags, bo->cpu_caching);
+}
+
 /**
  * __xe_bo_release_dummy() - Dummy kref release function
  * @kref: The embedded struct kref.
@@ -4477,7 +4564,7 @@ int xe_bo_dumb_create(struct drm_file *file_priv,
 			       DRM_XE_GEM_CPU_CACHING_WC,
 			       XE_BO_FLAG_VRAM_IF_DGFX(xe_device_get_root_tile(xe)) |
 			       XE_BO_FLAG_FORCE_WC |
-			       XE_BO_FLAG_NEEDS_CPU_ACCESS, NULL);
+			       XE_BO_FLAG_NEEDS_CPU_ACCESS, NULL, NULL);
 	if (IS_ERR(bo))
 		return PTR_ERR(bo);
 
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 38d50f8122cb..18fcccd082b7 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -111,6 +111,7 @@ enum xe_madv_purgeable_state {
 
 struct sg_table;
 struct ttm_tt;
+struct ttm_pool_prealloc;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -119,13 +120,16 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 				struct xe_tile *tile, struct dma_resv *resv,
 				struct ttm_lru_bulk_move *bulk, size_t size,
 				u16 cpu_caching, enum ttm_bo_type type,
-				u32 flags, struct drm_exec *exec);
+				u32 flags, struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec);
 struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile,
 				  struct xe_vm *vm, size_t size,
 				  enum ttm_bo_type type, u32 flags,
 				  struct drm_exec *exec);
 struct xe_bo *xe_bo_create_user(struct xe_device *xe, struct xe_vm *vm, size_t size,
-				u16 cpu_caching, u32 flags, struct drm_exec *exec);
+				u16 cpu_caching, u32 flags,
+				struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec);
 struct xe_bo *xe_bo_create_pin_map(struct xe_device *xe, struct xe_tile *tile,
 				   struct xe_vm *vm, size_t size,
 				   enum ttm_bo_type type, u32 flags,
diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c b/drivers/gpu/drm/xe/xe_dma_buf.c
index 8a920e58245c..ec947d86a1ce 100644
--- a/drivers/gpu/drm/xe/xe_dma_buf.c
+++ b/drivers/gpu/drm/xe/xe_dma_buf.c
@@ -302,7 +302,7 @@ xe_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
 
 		bo = xe_bo_init_locked(xe, NULL, NULL, resv, NULL, dma_buf->size,
 				       0, /* Will require 1way or 2way for vm_bind */
-				       ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, &exec);
+				       ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, NULL, &exec);
 		drm_exec_retry_on_contention(&exec);
 		if (IS_ERR(bo)) {
 			ret = PTR_ERR(bo);
-- 
2.34.1

[PATCH v3 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl

[Matthew Brost]

Add a xe_gem_create_ioctl tracepoint to help profile GEM creation, in
particular the cost of populating a BO while holding the dma-resv lock
versus the total IOCTL cost. This is useful for evaluating the
out-of-lock preallocation path, which aims to shrink the time spent
under the lock.

The tracepoint records the BO size, the requested placement mask, the
CPU caching mode, the VM asid (0 when no VM is bound), the total time
spent in the IOCTL, and the time spent under the dma-resv lock (the
xe_validation_guard region where the lock is held).

The asid is captured at VM lookup time rather than at trace emission,
since the VM reference is dropped before the tracepoint fires.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Guard lock_ns against an unset lock_start and use div_u64() in the
   tracepoint (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c       | 17 +++++++++++++++++
 drivers/gpu/drm/xe/xe_trace_bo.h | 31 +++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index a646a72122d4..caf5aa0bab53 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -3953,6 +3953,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	bool have_prealloc = false;
 	unsigned int bo_flags;
 	u32 handle;
+	ktime_t ioctl_start = ktime_get();
+	ktime_t lock_start = ktime_set(0, 0);
+	u64 lock_ns = 0;
+	u32 asid = 0;
 	int err;
 
 	if (XE_IOCTL_DBG(xe, args->pad[0] || args->pad[1] || args->pad[2]) ||
@@ -4031,6 +4035,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 		vm = xe_vm_lookup(xef, args->vm_id);
 		if (XE_IOCTL_DBG(xe, !vm))
 			return -ENOENT;
+		asid = vm->usm.asid;
 	}
 
 	/*
@@ -4100,6 +4105,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			if (err)
 				break;
 		}
+
+		if (lock_start == ktime_set(0, 0))
+			lock_start = ktime_get();
+
 		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
 				       bo_flags, have_prealloc ? &prealloc : NULL,
 				       &exec);
@@ -4110,6 +4119,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			break;
 		}
 	}
+	if (lock_start != ktime_set(0, 0))
+		lock_ns = ktime_to_ns(ktime_sub(ktime_get(), lock_start));
 	if (have_prealloc)
 		ttm_pool_prealloc_fini(&xe->ttm.pool, &prealloc);
 	if (err)
@@ -4140,6 +4151,12 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	if (vm)
 		xe_vm_put(vm);
 
+	trace_xe_gem_create_ioctl(xe, args->size, args->placement,
+				  args->cpu_caching, asid,
+				  div_u64(ktime_to_ns(ktime_sub(ktime_get(),
+							ioctl_start)), 1000),
+				  div_u64(lock_ns, 1000));
+
 	return err;
 }
 
diff --git a/drivers/gpu/drm/xe/xe_trace_bo.h b/drivers/gpu/drm/xe/xe_trace_bo.h
index b159d37d39a1..63db37edadc2 100644
--- a/drivers/gpu/drm/xe/xe_trace_bo.h
+++ b/drivers/gpu/drm/xe/xe_trace_bo.h
@@ -144,6 +144,37 @@ TRACE_EVENT(xe_bo_defrag_one,
 		      __entry->ret)
 );
 
+TRACE_EVENT(xe_gem_create_ioctl,
+	    TP_PROTO(struct xe_device *xe, u64 size, u32 placement,
+		     u16 caching, u32 asid, u64 ioctl_us, u64 lock_us),
+	    TP_ARGS(xe, size, placement, caching, asid, ioctl_us, lock_us),
+
+	    TP_STRUCT__entry(
+		     __string(dev, dev_name(xe->drm.dev))
+		     __field(u64, size)
+		     __field(u32, placement)
+		     __field(u16, caching)
+		     __field(u32, asid)
+		     __field(u64, ioctl_us)
+		     __field(u64, lock_us)
+		     ),
+
+	    TP_fast_assign(
+		   __assign_str(dev);
+		   __entry->size = size;
+		   __entry->placement = placement;
+		   __entry->caching = caching;
+		   __entry->asid = asid;
+		   __entry->ioctl_us = ioctl_us;
+		   __entry->lock_us = lock_us;
+		   ),
+
+	    TP_printk("dev=%s, size=%llu, placement=0x%x, caching=%u, asid=%u, ioctl_us=%llu, lock_us=%llu",
+		      __get_str(dev), __entry->size, __entry->placement,
+		      __entry->caching, __entry->asid, __entry->ioctl_us,
+		      __entry->lock_us)
+);
+
 DECLARE_EVENT_CLASS(xe_vma,
 		    TP_PROTO(struct xe_vma *vma),
 		    TP_ARGS(vma),
-- 
2.34.1

[PATCH v3 27/33] drm/xe: Add IOVA-based xe_res_cursor variant

[Matthew Brost]

Introduce xe_res_first_iova() to walk a buffer that was DMA mapped using
the IOVA-based DMA API (dma_iova_try_alloc() and friends). Such a mapping
is described by a struct dma_iova_state and is a single contiguous IOVA
range, so the cursor walks it as one segment.

Add an is_iova flag to struct xe_res_cursor to distinguish this case.
When set, xe_res_dma() returns the IOVA backing the current position
(state->addr + cursor offset), xe_res_next() advances within the single
contiguous segment, and xe_res_is_vram() returns false since IOVA
mappings never point to same-device VRAM.

The flag is explicitly cleared in the existing xe_res_first(),
xe_res_first_sg() and xe_res_first_dma() initializers so stale stack
values cannot leak into the new path.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_res_cursor.h | 56 +++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_res_cursor.h b/drivers/gpu/drm/xe/xe_res_cursor.h
index 0522caafd89d..f0f1d699e16e 100644
--- a/drivers/gpu/drm/xe/xe_res_cursor.h
+++ b/drivers/gpu/drm/xe/xe_res_cursor.h
@@ -24,6 +24,7 @@
 #ifndef _XE_RES_CURSOR_H_
 #define _XE_RES_CURSOR_H_
 
+#include <linux/dma-mapping.h>
 #include <linux/scatterlist.h>
 
 #include <drm/drm_pagemap.h>
@@ -67,6 +68,12 @@ struct xe_res_cursor {
 	u64 dma_start;
 	/** @dma_seg_size: Size of the current DMA segment. */
 	u64 dma_seg_size;
+	/**
+	 * @is_iova: Whether the cursor walks a single contiguous IOVA
+	 * mapping described by a struct dma_iova_state. When true, the
+	 * DMA address is @dma_start + @start and never points to VRAM.
+	 */
+	bool is_iova;
 };
 
 static struct gpu_buddy *xe_res_get_buddy(struct ttm_resource *res)
@@ -93,6 +100,7 @@ static inline void xe_res_first(struct ttm_resource *res,
 {
 	cur->sgl = NULL;
 	cur->dma_addr = NULL;
+	cur->is_iova = false;
 	if (!res)
 		goto fallback;
 
@@ -224,6 +232,7 @@ static inline void xe_res_first_sg(const struct sg_table *sg,
 	cur->dma_addr = NULL;
 	cur->sgl = sg->sgl;
 	cur->mem_type = XE_PL_TT;
+	cur->is_iova = false;
 	__xe_res_sg_next(cur);
 }
 
@@ -255,6 +264,41 @@ static inline void xe_res_first_dma(const struct drm_pagemap_addr *dma_addr,
 	__xe_res_dma_next(cur);
 	cur->sgl = NULL;
 	cur->mem_type = XE_PL_TT;
+	cur->is_iova = false;
+}
+
+/**
+ * xe_res_first_iova() - initialize a xe_res_cursor with a dma_iova_state
+ *
+ * @state: struct dma_iova_state describing a single contiguous IOVA mapping
+ * @start: Start of the range
+ * @size: Size of the range
+ * @cur: cursor object to initialize
+ *
+ * Start walking over the range of an allocation that was DMA mapped using the
+ * IOVA-based DMA API (see dma_iova_try_alloc()). Such a mapping is a single
+ * contiguous IOVA range, so the cursor walks it as one segment. Subsequent
+ * calls to xe_res_next() advance within that range and xe_res_dma() returns
+ * the IOVA backing the current position. xe_res_is_vram() always returns false
+ * for such a cursor.
+ */
+static inline void xe_res_first_iova(struct dma_iova_state *state,
+				     u64 start, u64 size,
+				     struct xe_res_cursor *cur)
+{
+	XE_WARN_ON(!state);
+	XE_WARN_ON(start + size > dma_iova_size(state));
+
+	cur->node = NULL;
+	cur->start = start;
+	cur->remaining = size;
+	cur->size = size;
+	cur->dma_addr = NULL;
+	cur->sgl = NULL;
+	cur->dma_start = state->addr;
+	cur->dma_seg_size = dma_iova_size(state);
+	cur->mem_type = XE_PL_TT;
+	cur->is_iova = true;
 }
 
 /**
@@ -283,6 +327,13 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
 		return;
 	}
 
+	if (cur->is_iova) {
+		/* Single contiguous IOVA segment. */
+		cur->start += size;
+		cur->size = cur->remaining;
+		return;
+	}
+
 	if (cur->dma_addr) {
 		cur->start += size;
 		__xe_res_dma_next(cur);
@@ -334,7 +385,7 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
  */
 static inline u64 xe_res_dma(const struct xe_res_cursor *cur)
 {
-	if (cur->dma_addr)
+	if (cur->is_iova || cur->dma_addr)
 		return cur->dma_start + cur->start;
 	else if (cur->sgl)
 		return sg_dma_address(cur->sgl) + cur->start;
@@ -351,6 +402,9 @@ static inline u64 xe_res_dma(const struct xe_res_cursor *cur)
  */
 static inline bool xe_res_is_vram(const struct xe_res_cursor *cur)
 {
+	if (cur->is_iova)
+		return false;
+
 	if (cur->dma_addr)
 		return cur->dma_addr->proto == XE_INTERCONNECT_VRAM;
 
-- 
2.34.1

[PATCH v3 28/33] drm/xe: Use IOVA-based DMA mapping for eligible tt BOs

[Matthew Brost]

Add support for mapping a ttm_tt's pages using the IOVA-based DMA API
(dma_iova_try_alloc()/link()/sync()/unlink()/free()) as an alternative
to the scatter-gather list. An IOVA mapping is a single contiguous DMA
range, which is cheaper to walk and program than an sg list.

Whether the IOVA API can be used depends on the IOMMU mode, so the
decision is made once per tt at xe_ttm_tt_create() time and recorded in
struct xe_ttm_tt::use_iova. To keep the scope limited, only device BOs
that are not GGTT-mapped are eligible: dma-buf imports manage their own
mapping and GGTT bindings are programmed directly from the sg list.
dma_iova_try_alloc() may still fail, in which case the tt transparently
falls back to the sg list, which remains the default.

The IOVA reservation is allocated at tt creation and freed at tt
destroy. xe_tt_map_iova() links the tt pages (coalescing physically
contiguous runs) and syncs the IOTLB; xe_tt_unmap_iova() unlinks them
but keeps the reservation so the tt can be remapped without
re-allocating IOVA space. xe_tt_map()/xe_tt_unmap() dispatch between the
IOVA and sg backends, and xe_bo_dma_unmap_pinned() learns about the IOVA
case.

Consumers that walked the sg list via xe_res_first_sg(xe_bo_sg()) now go
through the new xe_res_first_tt() helper, which initializes either an
IOVA cursor (xe_res_first_iova()) or an sg cursor depending on the tt's
mapping mode.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/tests/xe_migrate.c |   6 +-
 drivers/gpu/drm/xe/xe_bo.c            | 239 ++++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_bo.h            |   5 +-
 drivers/gpu/drm/xe/xe_ggtt.c          |   2 +-
 drivers/gpu/drm/xe/xe_migrate.c       |  14 +-
 drivers/gpu/drm/xe/xe_pt.c            |   2 +-
 6 files changed, 237 insertions(+), 31 deletions(-)

diff --git a/drivers/gpu/drm/xe/tests/xe_migrate.c b/drivers/gpu/drm/xe/tests/xe_migrate.c
index 41de648b2c86..119134e67c89 100644
--- a/drivers/gpu/drm/xe/tests/xe_migrate.c
+++ b/drivers/gpu/drm/xe/tests/xe_migrate.c
@@ -253,7 +253,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test,
 	if (xe_bo_is_vram(pt))
 		xe_res_first(pt->ttm.resource, 0, xe_bo_size(pt), &src_it);
 	else
-		xe_res_first_sg(xe_bo_sg(pt), 0, xe_bo_size(pt), &src_it);
+		xe_res_first_tt(pt->ttm.ttm, 0, xe_bo_size(pt), &src_it);
 
 	emit_pte(m, bb, NUM_KERNEL_PDE - 1, xe_bo_is_vram(pt), false,
 		 &src_it, XE_PAGE_SIZE, pt->ttm.resource);
@@ -385,12 +385,12 @@ static struct dma_fence *blt_copy(struct xe_tile *tile,
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 
 	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_tt(src_bo->ttm.ttm, 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 
 	if (!dst_is_vram)
-		xe_res_first_sg(xe_bo_sg(dst_bo), 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index caf5aa0bab53..d54e69aefeba 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -6,6 +6,7 @@
 #include "xe_bo.h"
 
 #include <linux/dma-buf.h>
+#include <linux/dma-mapping.h>
 #include <linux/nospec.h>
 
 #include <drm/drm_drv.h>
@@ -15,6 +16,7 @@
 #include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_device.h>
 #include <drm/ttm/ttm_placement.h>
+#include <drm/ttm/ttm_pool.h>
 #include <drm/ttm/ttm_tt.h>
 #include <uapi/drm/xe_drm.h>
 
@@ -420,6 +422,32 @@ struct xe_ttm_tt {
 	struct ttm_tt ttm;
 	struct sg_table sgt;
 	struct sg_table *sg;
+	/**
+	 * @iova: IOVA state describing the single contiguous IOVA mapping of
+	 * the tt pages. Only valid when @use_iova is true.
+	 */
+	struct dma_iova_state iova;
+	/**
+	 * @use_iova: Whether this tt is DMA mapped using the IOVA-based DMA
+	 * API instead of a scatter-gather list. Decided once at tt creation
+	 * time and stable for the lifetime of the tt.
+	 */
+	bool use_iova;
+	/**
+	 * @iova_linked: Whether the tt pages are currently linked into the
+	 * IOVA mapping. Only meaningful when @use_iova is true.
+	 */
+	bool iova_linked;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	/**
+	 * @iova_dma_pages: Per-order count of DMA-mapped pages this tt
+	 * accounted into xe->mem.dma_mapped_pages[] for its IOVA mapping,
+	 * recorded at link time so the unlink can apply an exactly matching
+	 * decrement without re-walking tt->pages (which TTM may mutate, e.g.
+	 * during defrag). Counter ownership stays isolated in xe_bo.c.
+	 */
+	unsigned long iova_dma_pages[NR_PAGE_ORDERS];
+#endif
 	/** @purgeable: Whether the content of the pages of @ttm is purgeable. */
 	bool purgeable;
 };
@@ -468,6 +496,52 @@ static void xe_tt_account_dma_pages(struct xe_device *xe,
 }
 #endif
 
+/*
+ * xe_tt_account_iova_add - account one freshly linked IOVA folio
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt being mapped
+ * @order: page order of the linked folio
+ *
+ * Add the folio's 2^order pages to xe->mem.dma_mapped_pages[order] and record
+ * the same amount in @xe_tt so xe_tt_unaccount_iova() can later subtract an
+ * exactly matching count without re-walking tt->pages.
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_iova_add(struct xe_device *xe,
+				   struct xe_ttm_tt *xe_tt, unsigned int order)
+{
+	unsigned long chunk = 1UL << order;
+
+	xe_tt->iova_dma_pages[order] += chunk;
+	atomic_long_add(chunk, &xe->mem.dma_mapped_pages[order]);
+}
+
+static void xe_tt_unaccount_iova(struct xe_device *xe,
+				 struct xe_ttm_tt *xe_tt)
+{
+	unsigned int order;
+
+	for (order = 0; order < NR_PAGE_ORDERS; order++) {
+		unsigned long pages = xe_tt->iova_dma_pages[order];
+
+		if (pages) {
+			atomic_long_sub(pages, &xe->mem.dma_mapped_pages[order]);
+			xe_tt->iova_dma_pages[order] = 0;
+		}
+	}
+}
+#else
+static void xe_tt_account_iova_add(struct xe_device *xe,
+				   struct xe_ttm_tt *xe_tt, unsigned int order)
+{
+}
+
+static void xe_tt_unaccount_iova(struct xe_device *xe,
+				 struct xe_ttm_tt *xe_tt)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -514,28 +588,129 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	}
 }
 
-struct sg_table *xe_bo_sg(struct xe_bo *bo)
+/*
+ * xe_tt_map_iova - DMA map a tt using the IOVA-based DMA API.
+ *
+ * Link all of the tt's pages into the IOVA range reserved at tt creation
+ * time, one folio at a time, and sync the IOTLB once at the end. The IOVA
+ * reservation itself is kept across map/unmap cycles and only released when
+ * the tt is destroyed.
+ */
+static int xe_tt_map_iova(struct xe_device *xe, struct ttm_tt *tt)
 {
-	struct ttm_tt *tt = bo->ttm.ttm;
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+	struct device *dev = xe->drm.dev;
+	unsigned long num_pages = tt->num_pages;
+	unsigned long i = 0;
+	size_t mapped = 0;
+	int ret;
+
+	if (xe_tt->iova_linked)
+		return 0;
+
+	while (i < num_pages) {
+		struct page *page = tt->pages[i];
+		unsigned int order = ttm_pool_page_order_nodma(page);
+		size_t len = PAGE_SIZE << order;
+
+		ret = dma_iova_link(dev, &xe_tt->iova, page_to_phys(page),
+				    mapped, len, DMA_BIDIRECTIONAL,
+				    DMA_ATTR_SKIP_CPU_SYNC);
+		if (ret)
+			goto err_unlink;
 
-	return xe_tt->sg;
+		xe_tt_account_iova_add(xe, xe_tt, order);
+		mapped += len;
+		i += 1UL << order;
+	}
+
+	ret = dma_iova_sync(dev, &xe_tt->iova, 0, mapped);
+	if (ret)
+		goto err_unlink;
+
+	xe_tt->iova_linked = true;
+	return 0;
+
+err_unlink:
+	if (mapped) {
+		dma_iova_unlink(dev, &xe_tt->iova, 0, mapped,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+		xe_tt_unaccount_iova(xe, xe_tt);
+	}
+	return ret;
 }
 
-/**
- * xe_tt_sg() - Return the DMA scatter-gather table backing a ttm_tt.
- * @tt: The ttm_tt to query.
+/*
+ * xe_tt_unmap_iova - Unlink a tt's pages from its IOVA mapping.
  *
- * Like xe_bo_sg(), but takes the ttm_tt directly. Used by the defrag copy
- * path which needs the old (source) tt's sg table without a backing xe_bo.
+ * Unlink the pages but keep the IOVA reservation so the tt can be remapped
+ * later without re-allocating IOVA space.
+ */
+static void xe_tt_unmap_iova(struct xe_device *xe, struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->iova_linked) {
+		/*
+		 * Unmap with attrs 0 (matching the sg unmap path) so that the
+		 * CPU copy-back sync is performed for non-coherent or bounced
+		 * (SWIOTLB) mappings, even though the link was done with
+		 * DMA_ATTR_SKIP_CPU_SYNC.
+		 */
+		dma_iova_unlink(xe->drm.dev, &xe_tt->iova, 0,
+				dma_iova_size(&xe_tt->iova),
+				DMA_BIDIRECTIONAL, 0);
+		xe_tt_unaccount_iova(xe, xe_tt);
+		xe_tt->iova_linked = false;
+	}
+}
+
+/*
+ * xe_tt_map - DMA map a tt, using IOVA when available and sg otherwise.
+ */
+static int xe_tt_map(struct xe_device *xe, struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->use_iova)
+		return xe_tt_map_iova(xe, tt);
+
+	return xe_tt_map_sg(xe, tt);
+}
+
+/*
+ * xe_tt_unmap - Undo xe_tt_map().
+ */
+static void xe_tt_unmap(struct xe_device *xe, struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	if (xe_tt->use_iova)
+		xe_tt_unmap_iova(xe, tt);
+	else
+		xe_tt_unmap_sg(xe, tt);
+}
+
+/**
+ * xe_res_first_tt() - Initialize a resource cursor over a ttm_tt's DMA mapping.
+ * @tt: The ttm_tt to walk.
+ * @start: Start of the range, in bytes.
+ * @size: Size of the range, in bytes.
+ * @cur: The cursor to initialize.
  *
- * Return: The sg table, or NULL if the tt is not currently mapped.
+ * Initialize @cur to walk the DMA addresses backing @tt, transparently
+ * handling both the IOVA-based mapping (single contiguous range) and the
+ * scatter-gather list fallback.
  */
-struct sg_table *xe_tt_sg(struct ttm_tt *tt)
+void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
+		     struct xe_res_cursor *cur)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
-	return xe_tt->sg;
+	if (xe_tt->use_iova)
+		xe_res_first_iova(&xe_tt->iova, start, size, cur);
+	else
+		xe_res_first_sg(xe_tt->sg, start, size, cur);
 }
 
 /*
@@ -651,6 +826,21 @@ static struct ttm_tt *xe_ttm_tt_create(struct ttm_buffer_object *ttm_bo,
 		}
 	}
 
+	/*
+	 * Eligible BOs are DMA mapped using the IOVA-based DMA API, which
+	 * yields a single contiguous IOVA range instead of a scatter-gather
+	 * list. Only device BOs that are not GGTT-mapped qualify: dma-buf
+	 * imports manage their own mapping and GGTT bindings are programmed
+	 * directly from the sg list. dma_iova_try_alloc() may still fail
+	 * depending on the IOMMU mode, in which case we transparently fall
+	 * back to the sg list. The reservation lives for the tt's lifetime.
+	 */
+	if (ttm_bo->type != ttm_bo_type_sg &&
+	    !(bo->flags & (XE_BO_FLAG_GGTT | XE_BO_FLAG_GGTT_ALL)) &&
+	    dma_iova_try_alloc(xe->drm.dev, &xe_tt->iova, 0,
+			       (size_t)tt->num_pages << PAGE_SHIFT))
+		xe_tt->use_iova = true;
+
 	return tt;
 }
 
@@ -700,7 +890,7 @@ static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 	    !(tt->page_flags & TTM_TT_FLAG_EXTERNAL_MAPPABLE))
 		return;
 
-	xe_tt_unmap_sg(xe, tt);
+	xe_tt_unmap(xe, tt);
 
 	ttm_pool_free(&ttm_dev->pool, tt);
 	xe_ttm_tt_account_subtract(xe, tt);
@@ -709,6 +899,19 @@ static void xe_ttm_tt_unpopulate(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 
 static void xe_ttm_tt_destroy(struct ttm_device *ttm_dev, struct ttm_tt *tt)
 {
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	/*
+	 * Release the IOVA reservation made in xe_ttm_tt_create(). The pages
+	 * must already have been unlinked by xe_tt_unmap().
+	 */
+	if (xe_tt->use_iova) {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		xe_assert(xe, !xe_tt->iova_linked);
+		dma_iova_free(xe->drm.dev, &xe_tt->iova);
+	}
+
 	ttm_tt_fini(tt);
 	kfree(tt);
 }
@@ -1557,7 +1760,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	/* Bo creation path, moving to system or TT. */
 	if ((!old_mem && ttm) && !handle_system_ccs) {
 		if (new_mem->mem_type == XE_PL_TT)
-			ret = xe_tt_map_sg(xe, ttm);
+			ret = xe_tt_map(xe, ttm);
 		if (!ret)
 			ttm_bo_move_null(ttm_bo, new_mem);
 		goto out;
@@ -1587,7 +1790,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
 	if (new_mem->mem_type == XE_PL_TT) {
-		ret = xe_tt_map_sg(xe, ttm);
+		ret = xe_tt_map(xe, ttm);
 		if (ret)
 			goto out;
 	}
@@ -1757,7 +1960,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		if (IS_VF_CCS_READY(xe))
 			xe_sriov_vf_ccs_detach_bo(bo);
 
-		xe_tt_unmap_sg(xe, ttm_bo->ttm);
+		xe_tt_unmap(xe, ttm_bo->ttm);
 	}
 
 	xe_bo_defrag_update(bo);
@@ -1785,7 +1988,7 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 		if (lret)
 			return lret;
 
-		xe_tt_unmap_sg(xe, bo->ttm);
+		xe_tt_unmap(xe, bo->ttm);
 		ttm_bo_move_null(bo, new_resource);
 	}
 
@@ -2212,6 +2415,8 @@ int xe_bo_dma_unmap_pinned(struct xe_bo *bo)
 						 DMA_BIDIRECTIONAL);
 			ttm_bo->sg = NULL;
 			xe_tt->sg = NULL;
+		} else if (xe_tt->use_iova) {
+			xe_tt_unmap_iova(ttm_to_xe_device(ttm_bo->bdev), tt);
 		} else if (xe_tt->sg) {
 			dma_unmap_sgtable(ttm_to_xe_device(ttm_bo->bdev)->drm.dev,
 					  xe_tt->sg,
@@ -3779,7 +3984,7 @@ dma_addr_t __xe_bo_addr(struct xe_bo *bo, u64 offset, size_t page_size)
 	if (!xe_bo_is_vram(bo) && !xe_bo_is_stolen(bo)) {
 		xe_assert(xe, bo->ttm.ttm);
 
-		xe_res_first_sg(xe_bo_sg(bo), page << PAGE_SHIFT,
+		xe_res_first_tt(bo->ttm.ttm, page << PAGE_SHIFT,
 				page_size, &cur);
 		return xe_res_dma(&cur) + offset;
 	} else {
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 18fcccd082b7..3e2eaefecb8f 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -552,8 +552,9 @@ void xe_bo_dev_init(struct xe_bo_dev *bo_device);
 
 void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
 
-struct sg_table *xe_bo_sg(struct xe_bo *bo);
-struct sg_table *xe_tt_sg(struct ttm_tt *tt);
+struct xe_res_cursor;
+void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
+		     struct xe_res_cursor *cur);
 
 /*
  * xe_sg_segment_size() - Provides upper limit for sg segment size.
diff --git a/drivers/gpu/drm/xe/xe_ggtt.c b/drivers/gpu/drm/xe/xe_ggtt.c
index 8ec23862477f..4084bc183e06 100644
--- a/drivers/gpu/drm/xe/xe_ggtt.c
+++ b/drivers/gpu/drm/xe/xe_ggtt.c
@@ -699,7 +699,7 @@ static void xe_ggtt_map_bo(struct xe_ggtt *ggtt, struct xe_ggtt_node *node,
 	if (!xe_bo_is_vram(bo) && !xe_bo_is_stolen(bo)) {
 		xe_assert(xe_bo_device(bo), bo->ttm.ttm);
 
-		for (xe_res_first_sg(xe_bo_sg(bo), 0, xe_bo_size(bo), &cur);
+		for (xe_res_first_tt(bo->ttm.ttm, 0, xe_bo_size(bo), &cur);
 		     cur.remaining; xe_res_next(&cur, XE_PAGE_SIZE))
 			ggtt->pt_ops->ggtt_set_pte(ggtt, end - cur.remaining,
 						   pte | xe_res_dma(&cur));
diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 9990a7144a9f..666c715a9067 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -1005,16 +1005,16 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_tt(src_tt ? src_tt : src_bo->ttm.ttm, 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
-		xe_res_first_sg(xe_bo_sg(dst_bo), 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo),
+		xe_res_first_tt(src_tt ? src_tt : src_bo->ttm.ttm,
 				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
@@ -1383,9 +1383,9 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
 
 	ctx = &xe->sriov.vf.ccs.contexts[read_write];
 
-	xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+	xe_res_first_tt(src_bo->ttm.ttm, 0, size, &src_it);
 
-	xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+	xe_res_first_tt(src_bo->ttm.ttm, xe_bo_ccs_pages_start(src_bo),
 			PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 			&ccs_it);
 
@@ -1588,7 +1588,7 @@ struct dma_fence *xe_migrate_vram_copy_chunk(struct xe_bo *vram_bo, u64 vram_off
 	xe_assert(xe, !range_overflows(sysmem_offset, size, (u64)sysmem_bo->ttm.base.size));
 
 	xe_res_first(vram, vram_offset, size, &vram_it);
-	xe_res_first_sg(xe_bo_sg(sysmem_bo), sysmem_offset, size, &sysmem_it);
+	xe_res_first_tt(sysmem_bo->ttm.ttm, sysmem_offset, size, &sysmem_it);
 
 	while (size) {
 		u32 pte_flags = PTE_UPDATE_FLAG_IS_VRAM;
@@ -1800,7 +1800,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,
 		clear_only_system_ccs = true;
 
 	if (!clear_vram)
-		xe_res_first_sg(xe_bo_sg(bo), 0, xe_bo_size(bo), &src_it);
+		xe_res_first_tt(bo->ttm.ttm, 0, xe_bo_size(bo), &src_it);
 	else
 		xe_res_first(src, 0, xe_bo_size(bo), &src_it);
 
diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c
index 7111b5d0f6aa..185d15fb0a97 100644
--- a/drivers/gpu/drm/xe/xe_pt.c
+++ b/drivers/gpu/drm/xe/xe_pt.c
@@ -797,7 +797,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,
 			xe_res_first(bo->ttm.resource, xe_vma_bo_offset(vma),
 				     xe_vma_size(vma), &curs);
 		else
-			xe_res_first_sg(xe_bo_sg(bo), xe_vma_bo_offset(vma),
+			xe_res_first_tt(bo->ttm.ttm, xe_vma_bo_offset(vma),
 					xe_vma_size(vma), &curs);
 	} else if (!range) {
 		curs.size = xe_vma_size(vma);
-- 
2.34.1

[PATCH v3 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

[Matthew Brost]

A future patch performs the destination DMA mapping of a defrag move in
two stages when the BO is IOVA mapped: a temporary contiguous mapping of
just tht changed pages used for the GPU copy, followed by a post-copy
finalize stet that tears that mapping down and links the new backing
into the (transferred) IOVA reservation. The finalize step must run only
after the GPU copy it depends on has completed, so it is dispatched as
an xe_dep_job,

Add a per-device xe_dep_scheduler dedicated to these finalize jobs,
created in xe_bo_defrag_init() (after the migrate contexts are up) and
torn down in xe_bo_defrag_fini(). No users yet; the job itself is added
in a later patch.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c           | 15 +++++++++++++++
 drivers/gpu/drm/xe/xe_device_types.h | 10 ++++++++++
 2 files changed, 25 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index d54e69aefeba..c5022c16411f 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -25,6 +25,7 @@
 #include <trace/events/gpu_mem.h>
 
 #include "xe_device.h"
+#include "xe_dep_scheduler.h"
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
 #include "xe_ggtt.h"
@@ -1306,6 +1307,10 @@ static void xe_bo_defrag_fini(void *arg)
 	struct xe_device *xe = arg;
 
 	disable_delayed_work_sync(&xe->mem.defrag.worker);
+	if (xe->mem.defrag.iova_sched) {
+		xe_dep_scheduler_fini(xe->mem.defrag.iova_sched);
+		xe->mem.defrag.iova_sched = NULL;
+	}
 }
 
 /**
@@ -1342,6 +1347,16 @@ void xe_bo_defrag_init_early(struct xe_device *xe)
  */
 int xe_bo_defrag_init(struct xe_device *xe)
 {
+#define XE_BO_MAX_IOVA_DEFRAG_JOBS	16	/* Picking a reasonable value */
+	struct xe_dep_scheduler *iova_sched;
+
+	iova_sched = xe_dep_scheduler_create(xe, NULL, "xe_iova_defrag",
+					     XE_BO_MAX_IOVA_DEFRAG_JOBS);
+	if (IS_ERR(iova_sched))
+		return PTR_ERR(iova_sched);
+	xe->mem.defrag.iova_sched = iova_sched;
+#undef XE_BO_MAX_IOVA_DEFRAG_JOBS
+
 	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
 }
 
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 6e68add4d652..b55d549d5e30 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -38,6 +38,7 @@
 
 struct drm_pagemap_shrinker;
 struct intel_display;
+struct xe_dep_scheduler;
 struct intel_dg_nvm_dev;
 struct xe_ggtt;
 struct xe_i2c;
@@ -325,6 +326,15 @@ struct xe_device {
 			 * @mem.defrag.worker, in milliseconds.
 			 */
 			unsigned int interval_ms;
+			/**
+			 * @mem.defrag.iova_sched: Dependency scheduler used to
+			 * run the post-copy IOVA finalize job for defrag moves
+			 * of IOVA-mapped BOs. The job tears down the temporary
+			 * copy-step IOVA mapping and links the new backing into
+			 * the (transferred) reservation once the GPU copy that
+			 * its dependencies gate has completed.
+			 */
+			struct xe_dep_scheduler *iova_sched;
 		} defrag;
 	} mem;
 
-- 
2.34.1

[PATCH v3 30/33] drm/xe: Add packed copy-step IOVA mapping for defrag

[Matthew Brost]

Add machinery to map only changed pages during defrag when the
destination TT is IOVA-mapped. Only the non-borrowed (freshly
allocated) pages need copying; borrowed pages are aliases from
the old TT.

Add xe_tt_iova_copy_init() to diff new vs old TT folios and
record changed-page runs. Add xe_tt_map_iova_copy() to link
those changed pages packed contiguously at the front of the
new TT's IOVA reservation, synced and ready for GPU blits.

No users yet; wired up in later patches.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 364 +++++++++++++++++++++++++++++++++++++
 1 file changed, 364 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index c5022c16411f..c087f17ad6db 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -531,6 +531,31 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 		}
 	}
 }
+
+/*
+ * xe_tt_account_iova_pages - account a tt's pages from a full folio walk
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt to account (its @iova_dma_pages must start zeroed)
+ *
+ * Walk @xe_tt's pages one folio at a time (the same head-page order walk used
+ * by xe_tt_map_iova()) and account each into xe->mem.dma_mapped_pages[] and
+ * @xe_tt->iova_dma_pages[]. Used to (re)derive accounting that exactly matches
+ * the pages a tt is actually mapped with.
+ */
+__maybe_unused
+static void xe_tt_account_iova_pages(struct xe_device *xe,
+				     struct xe_ttm_tt *xe_tt)
+{
+	struct ttm_tt *tt = &xe_tt->ttm;
+	unsigned long i = 0;
+
+	while (i < tt->num_pages) {
+		unsigned int order = ttm_pool_page_order_nodma(tt->pages[i]);
+
+		xe_tt_account_iova_add(xe, xe_tt, order);
+		i += 1UL << order;
+	}
+}
 #else
 static void xe_tt_account_iova_add(struct xe_device *xe,
 				   struct xe_ttm_tt *xe_tt, unsigned int order)
@@ -541,6 +566,12 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 				 struct xe_ttm_tt *xe_tt)
 {
 }
+
+__maybe_unused
+static void xe_tt_account_iova_pages(struct xe_device *xe,
+				     struct xe_ttm_tt *xe_tt)
+{
+}
 #endif
 
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
@@ -714,6 +745,339 @@ void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
 		xe_res_first_sg(xe_tt->sg, start, size, cur);
 }
 
+/**
+ * struct xe_tt_iova_run - A contiguous run of changed pages in a defrag move
+ * @start: First page offset (natural ttm_tt index) of the run.
+ * @count: Number of pages in the run.
+ *
+ * Runs are folio aligned: defrag borrows whole beneficial-order chunks, so a
+ * folio is either entirely borrowed (unchanged) or entirely freshly allocated
+ * (changed), never split across the boundary.
+ */
+struct xe_tt_iova_run {
+	pgoff_t start;
+	pgoff_t count;
+};
+
+/**
+ * struct xe_tt_iova_link - A snapshot folio link descriptor for defrag relink
+ * @phys: Physical address of the folio's head page.
+ * @offset: Natural byte offset into the reservation to link the folio at.
+ * @len: Byte length of the folio (PAGE_SIZE << order).
+ *
+ * Captured at copy-init time while the destination tt's page array is valid and
+ * stable under the BO lock. The post-copy finalize job runs asynchronously,
+ * after TTM may have mutated (or torn down) the destination tt page array, so
+ * it relinks purely from these snapshots and never dereferences that array.
+ */
+struct xe_tt_iova_link {
+	phys_addr_t phys;
+	size_t offset;
+	size_t len;
+};
+
+/**
+ * struct xe_tt_iova_copy - Changed-page subset of an IOVA defrag move
+ * @runs: Array of changed-page runs, in ascending natural tt offset.
+ * @nr_runs: Number of entries in @runs.
+ * @links: Per-folio link snapshots covering the changed runs, in ascending
+ *         natural offset. Drives the asynchronous finalize relink so it never
+ *         touches the destination tt's (possibly mutated) page array.
+ * @nr_links: Number of entries in @links.
+ * @nr_changed: Total number of changed pages (sum of the run counts). This is
+ *              also the size, in pages, of the packed copy-step IOVA mapping
+ *              produced by xe_tt_map_iova_copy().
+ */
+struct xe_tt_iova_copy {
+	struct xe_tt_iova_run *runs;
+	unsigned int nr_runs;
+	struct xe_tt_iova_link *links;
+	unsigned int nr_links;
+	pgoff_t nr_changed;
+};
+
+/*
+ * xe_tt_iova_walk_changed - Walk a defrag move's changed-page runs.
+ *
+ * Iterate @new_tt one folio at a time (folio order read from TTM via
+ * ttm_pool_page_order_nodma()) and compare each folio's head page against
+ * @old_tt at the same natural offset. Pages the new backing borrowed from the
+ * old tt alias the same struct page and are unchanged; freshly allocated pages
+ * differ. Coalesce contiguous changed folios into runs and, for each run,
+ * invoke @fn. Returns the total number of changed pages.
+ */
+__maybe_unused
+static pgoff_t xe_tt_iova_walk_changed(struct ttm_tt *new_tt,
+				       struct ttm_tt *old_tt,
+				       void (*fn)(pgoff_t start, pgoff_t count,
+						  void *arg),
+				       void *arg)
+{
+	pgoff_t i = 0, n = new_tt->num_pages;
+	pgoff_t run_start = 0, nr_changed = 0;
+	bool in_run = false;
+
+	while (i < n) {
+		unsigned int order = ttm_pool_page_order_nodma(new_tt->pages[i]);
+		pgoff_t nr = 1UL << order;
+		bool changed = new_tt->pages[i] != old_tt->pages[i];
+
+		if (changed) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+			nr_changed += nr;
+		} else if (in_run) {
+			if (fn)
+				fn(run_start, i - run_start, arg);
+			in_run = false;
+		}
+
+		i += nr;
+	}
+
+	if (in_run && fn)
+		fn(run_start, n - run_start, arg);
+
+	return nr_changed;
+}
+
+struct xe_tt_iova_run_collect {
+	struct ttm_tt *new_tt;
+	struct xe_tt_iova_run *runs;
+	unsigned int nr_runs;
+	unsigned int runs_cap;
+	struct xe_tt_iova_link *links;
+	unsigned int nr_links;
+	unsigned int links_cap;
+	int err;
+};
+
+/* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
+__maybe_unused
+static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
+			   size_t elem)
+{
+	unsigned int new_cap;
+	void *p;
+
+	if (need <= *cap)
+		return 0;
+
+	new_cap = *cap ? *cap * 2 : 64;
+	if (new_cap < need)
+		new_cap = need;
+
+	p = krealloc_array(*arr, new_cap, elem, GFP_KERNEL);
+	if (!p)
+		return -ENOMEM;
+
+	*arr = p;
+	*cap = new_cap;
+	return 0;
+}
+
+__maybe_unused
+static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
+{
+	struct xe_tt_iova_run_collect *c = arg;
+	pgoff_t i = start, end = start + count;
+
+	if (c->err)
+		return;
+
+	if (xe_tt_iova_grow((void **)&c->runs, &c->runs_cap, c->nr_runs + 1,
+			    sizeof(*c->runs))) {
+		c->err = -ENOMEM;
+		return;
+	}
+	c->runs[c->nr_runs].start = start;
+	c->runs[c->nr_runs].count = count;
+	c->nr_runs++;
+
+	while (i < end) {
+		struct page *page = c->new_tt->pages[i];
+		unsigned int order = ttm_pool_page_order_nodma(page);
+
+		if (xe_tt_iova_grow((void **)&c->links, &c->links_cap,
+				    c->nr_links + 1, sizeof(*c->links))) {
+			c->err = -ENOMEM;
+			return;
+		}
+		c->links[c->nr_links].phys = page_to_phys(page);
+		c->links[c->nr_links].offset = (size_t)i << PAGE_SHIFT;
+		c->links[c->nr_links].len = PAGE_SIZE << order;
+		c->nr_links++;
+
+		i += 1UL << order;
+	}
+}
+
+/**
+ * xe_tt_iova_copy_init() - Compute the changed-page subset of a defrag move
+ * @new_tt: The freshly (re)allocated destination tt.
+ * @old_tt: The stashed source tt (bo->defrag_old_tt).
+ * @copy: Output description of the changed pages.
+ *
+ * Diff @new_tt against @old_tt and record the runs of changed (non-borrowed)
+ * pages into @copy, along with a per-folio snapshot of their physical
+ * addresses, natural offsets and lengths. The result drives both the packed
+ * copy-step IOVA mapping (xe_tt_map_iova_copy()) and the post-copy finalize
+ * that relinks the new backing into the IOVA reservation. The finalize runs
+ * asynchronously, after TTM may have mutated @new_tt->pages, so it consumes the
+ * snapshot rather than re-reading the page array. @new_tt and @old_tt must
+ * describe the same BO (equal num_pages) and both be populated.
+ *
+ * The caller owns @copy and must release it with xe_tt_iova_copy_fini().
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+__maybe_unused
+static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
+				struct xe_tt_iova_copy *copy)
+{
+	struct xe_tt_iova_run_collect c = { .new_tt = new_tt };
+
+	memset(copy, 0, sizeof(*copy));
+
+	/*
+	 * Single pass: collect the changed runs and per-folio link snapshots
+	 * directly, growing the arrays on demand. Changed runs are few (the
+	 * borrowed majority is skipped), so the geometric reallocs are rare and
+	 * this avoids a second full O(num_pages) walk just to size the arrays.
+	 */
+	copy->nr_changed = xe_tt_iova_walk_changed(new_tt, old_tt,
+						   xe_tt_iova_store_run, &c);
+	if (c.err) {
+		kfree(c.runs);
+		kfree(c.links);
+		return c.err;
+	}
+	if (!c.nr_runs)
+		return 0;
+
+	copy->runs = c.runs;
+	copy->nr_runs = c.nr_runs;
+	copy->links = c.links;
+	copy->nr_links = c.nr_links;
+
+	return 0;
+}
+
+/**
+ * xe_tt_iova_copy_fini() - Release a struct xe_tt_iova_copy
+ * @copy: The copy description to release.
+ */
+__maybe_unused
+static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
+{
+	kfree(copy->runs);
+	kfree(copy->links);
+	copy->runs = NULL;
+	copy->links = NULL;
+	copy->nr_runs = 0;
+	copy->nr_links = 0;
+	copy->nr_changed = 0;
+}
+
+/**
+ * xe_tt_map_iova_copy() - DMA map the changed-page subset for a defrag copy
+ * @xe: The xe device.
+ * @new_tt: The destination tt (must be IOVA eligible, use_iova == true).
+ * @copy: The changed-page description from xe_tt_iova_copy_init().
+ *
+ * Link only the changed pages of @new_tt into its IOVA reservation, packed
+ * contiguously at the front of the range ([0, @copy->nr_changed) pages). This
+ * provides a compact, contiguous destination for the GPU defrag copy without
+ * mapping the (majority) borrowed pages, and lets the whole temporary mapping
+ * later be torn down with a single dma_iova_destroy().
+ *
+ * The reservation's remaining tail is left unlinked. This does not establish
+ * the tt's final mapping (@iova_linked stays false); the post-copy finalize is
+ * responsible for that.
+ *
+ * Return: 0 on success, negative error code on failure (any partial linkage is
+ * unwound).
+ */
+__maybe_unused
+static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
+			       const struct xe_tt_iova_copy *copy)
+{
+	struct xe_ttm_tt *xe_tt = container_of(new_tt, struct xe_ttm_tt, ttm);
+	struct device *dev = xe->drm.dev;
+	size_t mapped = 0;
+	unsigned int r;
+	int ret;
+
+	xe_assert(xe, xe_tt->use_iova);
+	xe_assert(xe, !xe_tt->iova_linked);
+
+	for (r = 0; r < copy->nr_runs; r++) {
+		pgoff_t i = copy->runs[r].start;
+		pgoff_t end = i + copy->runs[r].count;
+
+		while (i < end) {
+			struct page *page = new_tt->pages[i];
+			unsigned int order = ttm_pool_page_order_nodma(page);
+			size_t len = PAGE_SIZE << order;
+
+			ret = dma_iova_link(dev, &xe_tt->iova,
+					    page_to_phys(page), mapped, len,
+					    DMA_BIDIRECTIONAL,
+					    DMA_ATTR_SKIP_CPU_SYNC);
+			if (ret)
+				goto err_unlink;
+
+			mapped += len;
+			i += 1UL << order;
+		}
+	}
+
+	ret = dma_iova_sync(dev, &xe_tt->iova, 0, mapped);
+	if (ret)
+		goto err_unlink;
+
+	return 0;
+
+err_unlink:
+	if (mapped)
+		dma_iova_unlink(dev, &xe_tt->iova, 0, mapped,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+	return ret;
+}
+
+/*
+ * xe_tt_iova_transfer - Transfer a full IOVA reservation old_tt -> new_tt.
+ *
+ * Move the (fully linked) IOVA reservation from the old tt to the new tt,
+ * leaving the old tt without a reservation so its destroy path does not double
+ * free. The new tt keeps its existing reservation snapshotted in the finalize
+ * job for teardown. Called at the defrag finalize job's point of no failure.
+ *
+ * Also rebalance the per-order DMA-page accounting: drop the old tt's
+ * accounting and account the pages the new tt actually ends up mapped with.
+ * The finalize relink leaves the new tt's mapping equal to new_tt->pages
+ * (borrowed pages alias the old tt; changed runs use the new pages, which may
+ * have a different order distribution), so deriving the accounting from a full
+ * folio walk of the new tt keeps xe->mem.dma_mapped_pages[] exact while the new
+ * mapping lives, rather than carrying the old tt's stale distribution.
+ */
+__maybe_unused
+static void xe_tt_iova_transfer(struct xe_device *xe,
+				struct xe_ttm_tt *new_tt,
+				struct xe_ttm_tt *old_tt)
+{
+	new_tt->iova = old_tt->iova;
+	new_tt->iova_linked = true;
+
+	xe_tt_unaccount_iova(xe, old_tt);
+	xe_tt_account_iova_pages(xe, new_tt);
+
+	old_tt->iova_linked = false;
+	old_tt->use_iova = false;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
-- 
2.34.1

[PATCH v3 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

[Matthew Brost]

A defrag move with IOVA enabled only DMA-maps the changed (non-borrowed)
pages of the new tt, packed contiguously into the front of its IOVA
reservation (xe_tt_map_iova_copy()). The GPU copy therefore has an
asymmetric geometry: the source is read from the old tt at its natural
page offsets (via the old tt's full IOVA), while the destination only
exposes the changed pages, packed.

Teach __xe_migrate_copy() about this packed-destination mode:

- Add a @dst_packed_pages argument. When non-zero the destination tt
  cursor walks just that many packed pages and the source (not the
  destination) drives the chunk size, so the smaller packed prefix never
  clamps the source walk.

- Size the destination PTE update for each chunk from the changed-page
  count (xe_migrate_count_changed_pages()) rather than the full chunk,
  and skip the destination PTEs entirely for an all-borrowed chunk.

- Extend emit_copy_chunks() with a @pack_dst flag so each emitted run is
  blitted from its natural source offset into a running packed
  destination offset that only advances for changed runs. The
  all-borrowed skip leaves the packed destination cursor untouched.

Expose this via xe_migrate_copy_defrag_iova(). It is only valid when the
BO needs no CCS pass, since the separate CCS pass migrates aux state for
all pages and would require the full destination mapping. The existing
xe_migrate_copy_defrag() (full destination mapping) remains the path for
CCS BOs and the non-IOVA fallback.

No users yet; xe_bo_move() is wired to this path in a later patch.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>
---
 drivers/gpu/drm/xe/xe_migrate.c | 307 ++++++++++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_migrate.h |   8 +
 2 files changed, 296 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 666c715a9067..e1dc2fb5c5cb 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -811,6 +811,64 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
 		emit_xy_fast_copy(gt, bb, src_ofs, dst_ofs, size, pitch);
 }
 
+/*
+ * xe_migrate_count_changed_pages() - Count non-borrowed pages in a defrag chunk.
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ * @npages: Number of pages in the L0 chunk.
+ *
+ * A defragmentation move may harvest already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) some of the same physical pages.
+ * Count the pages that actually differ (src page != dst page) within the chunk.
+ *
+ * Return: The number of non-borrowed (changed) pages.
+ */
+static u32 xe_migrate_count_changed_pages(struct ttm_tt *src_tt,
+					  struct ttm_tt *dst_tt,
+					  pgoff_t start_page, u32 npages)
+{
+	u32 i, nchg = 0;
+
+	for (i = 0; i < npages; i++)
+		if (src_tt->pages[start_page + i] !=
+		    dst_tt->pages[start_page + i])
+			nchg++;
+
+	return nchg;
+}
+
+/*
+ * xe_migrate_leading_borrowed_pages() - Count the run of borrowed pages at the
+ * head of the remaining copy.
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of the next page to copy within the tt page arrays.
+ * @max_pages: Pages left to copy.
+ *
+ * A defragmentation move borrows the already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) the same physical pages for the
+ * unchanged majority of a large object. Those pages alias on both the source
+ * and destination GPU mappings and need no copy. Count how many consecutive
+ * pages starting at @start_page are borrowed so the caller can advance past the
+ * whole run at once instead of rediscovering it one L0 chunk at a time.
+ *
+ * Return: The number of leading borrowed (src page == dst page) pages, capped
+ * at @max_pages.
+ */
+static u32 xe_migrate_leading_borrowed_pages(struct ttm_tt *src_tt,
+					     struct ttm_tt *dst_tt,
+					     pgoff_t start_page, u32 max_pages)
+{
+	u32 n = 0;
+
+	while (n < max_pages &&
+	       src_tt->pages[start_page + n] == dst_tt->pages[start_page + n])
+		n++;
+
+	return n;
+}
+
 /*
  * emit_copy_chunks() - Emit the data blits for one defrag L0 chunk, skipping
  * borrowed pages.
@@ -822,6 +880,9 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
  * @src_tt: Source (old) tt providing the page pointers.
  * @dst_tt: Destination (new) tt providing the page pointers.
  * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ * @pack_dst: If true the destination only maps the changed pages packed
+ *	      contiguously, so the destination offset advances by changed-page
+ *	      runs instead of tracking the source page index.
  *
  * A defragmentation move may harvest already beneficial-order chunks from the
  * old tt, so the new tt references (borrows) some of the same physical pages.
@@ -830,16 +891,21 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
  * only blit runs of pages that actually differ (src page != dst page),
  * coalescing contiguous to-copy pages into a single blit.
  *
+ * When @pack_dst is set the source pages are read from their natural offsets
+ * (@src_L0_ofs + page index) while the destination pages were packed into a
+ * contiguous prefix, so the destination offset is a running counter that only
+ * advances for emitted (changed) runs.
+ *
  * Return: The number of blits emitted (or that would be emitted when @bb is
  * NULL).
  */
 static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
 			    u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
 			    struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
-			    pgoff_t start_page)
+			    pgoff_t start_page, bool pack_dst)
 {
 	u32 npages = src_L0 >> PAGE_SHIFT;
-	u32 i = 0, nblits = 0;
+	u32 i = 0, pack = 0, nblits = 0;
 
 	while (i < npages) {
 		u32 run = 0;
@@ -859,9 +925,71 @@ static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
 
 		if (bb)
 			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
-				  dst_L0_ofs + (u64)i * PAGE_SIZE,
+				  dst_L0_ofs +
+				  (u64)(pack_dst ? pack : i) * PAGE_SIZE,
 				  run * PAGE_SIZE, XE_PAGE_SIZE);
 		nblits++;
+		pack += run;
+		i += run;
+	}
+
+	return nblits;
+}
+
+/*
+ * emit_ccs_copy_chunks() - Emit the indirect CCS aux copies for one packed
+ * defrag L0 chunk, skipping borrowed pages.
+ * @gt: The GT.
+ * @bb: The batch buffer to emit into, or NULL to only count the copies needed.
+ * @src_L0_ofs: GPU offset of the source pages mapped (comp PAT) for this chunk.
+ * @dst_L0_ofs: GPU offset of the destination pages mapped (comp PAT) for this
+ *	        chunk (the packed changed-page prefix).
+ * @src_L0: Byte size of the L0 chunk (page aligned).
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ *
+ * Companion to emit_copy_chunks() for the packed defrag CCS pass. The flat-CCS
+ * aux state is migrated indirectly through the data-page mappings, so it
+ * follows the physical page regardless of the (packed) GPU VA layout. Borrowed
+ * pages alias the same physical page on both sides, so their aux is already
+ * identical and is skipped; only the changed runs are copied, from the source
+ * at its natural offset to the destination at its packed offset.
+ *
+ * Return: The number of CCS copies emitted (or that would be emitted when @bb
+ * is NULL).
+ */
+static u32 emit_ccs_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
+				u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
+				struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
+				pgoff_t start_page)
+{
+	u32 npages = src_L0 >> PAGE_SHIFT;
+	u32 i = 0, pack = 0, nblits = 0;
+
+	while (i < npages) {
+		u32 run = 0;
+
+		/* Skip a run of borrowed pages (src and dst alias). */
+		if (src_tt->pages[start_page + i] ==
+		    dst_tt->pages[start_page + i]) {
+			i++;
+			continue;
+		}
+
+		/* Coalesce a run of pages whose aux must be migrated. */
+		while (i + run < npages &&
+		       src_tt->pages[start_page + i + run] !=
+		       dst_tt->pages[start_page + i + run])
+			run++;
+
+		if (bb)
+			emit_copy_ccs(gt, bb,
+				      dst_L0_ofs + (u64)pack * PAGE_SIZE, true,
+				      src_L0_ofs + (u64)i * PAGE_SIZE, true,
+				      run * PAGE_SIZE);
+		nblits++;
+		pack += run;
 		i += run;
 	}
 
@@ -949,6 +1077,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
 					   struct ttm_tt *src_tt,
+					   pgoff_t dst_packed_pages,
 					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
@@ -981,6 +1110,23 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	 */
 	struct ttm_tt *dst_tt = dst_bo->ttm.ttm;
 	bool skip_borrowed = flags.defrag_copy && src_tt && dst_tt;
+	/*
+	 * Defrag-IOVA copy: the destination tt only maps the changed
+	 * (non-borrowed) pages, packed contiguously into the first
+	 * @dst_packed_pages pages of its IOVA. The source is read from the old
+	 * tt at natural offsets and each changed run is blitted into the packed
+	 * destination prefix.
+	 */
+	bool pack_dst = dst_packed_pages != 0;
+	/*
+	 * Packed defrag CCS pass: migrate the flat-CCS aux state of the changed
+	 * pages only, from the source at its natural offset to the destination
+	 * packed prefix. Borrowed pages share the physical page (and thus its
+	 * aux) on both sides, so they are skipped exactly like the data pass.
+	 */
+	bool ccs_pack = flags.defrag_ccs && pack_dst && src_tt && dst_tt;
+	/* Per-run, borrowed-skipping defrag emission (data or packed CCS). */
+	bool defrag_runs = skip_borrowed || ccs_pack;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
@@ -1009,7 +1155,9 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
-		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0,
+				pack_dst ? (u64)dst_packed_pages << PAGE_SHIFT :
+				size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
@@ -1030,29 +1178,86 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		u32 pte_flags;
 		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> PAGE_SHIFT;
 		u32 ndata_blits = 1;
+		u32 nchg = 0;
+		u64 dst_size;
 
 		bool usm = xe->info.has_usm;
 		u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
 
+		/*
+		 * Defrag fast-forward: the new (destination) tt borrows the
+		 * unchanged beneficial-order chunks from the old tt, so for a
+		 * large object the bulk of the copy is borrowed pages that alias
+		 * the same memory on both sides. Skip the entire leading
+		 * borrowed run in a single cursor stride rather than discovering
+		 * it one (8 MB) L0 chunk at a time - each such pass would
+		 * otherwise pay xe_migrate_res_sizes() + pte_update_size() + a
+		 * full chunk walk only to copy nothing. The non-vram tt cursors
+		 * are normally advanced by emit_pte(); replicate that with
+		 * xe_res_next(). In pack_dst mode the destination only maps the
+		 * changed pages, so its cursor must not advance for borrowed
+		 * pages.
+		 */
+		if (defrag_runs) {
+			u32 brun = xe_migrate_leading_borrowed_pages(src_tt,
+								     dst_tt,
+								     start_page,
+								     size >> PAGE_SHIFT);
+
+			if (brun) {
+				u64 brun_bytes = (u64)brun << PAGE_SHIFT;
+
+				xe_res_next(&src_it, brun_bytes);
+				if (!pack_dst)
+					xe_res_next(&dst_it, brun_bytes);
+				size -= brun_bytes;
+				continue;
+			}
+		}
+
 		src_L0 = xe_migrate_res_sizes(m, &src_it);
 		dst_L0 = xe_migrate_res_sizes(m, &dst_it);
 
 		drm_dbg(&xe->drm, "Pass %u, sizes: %llu & %llu\n",
 			pass++, src_L0, dst_L0);
 
-		src_L0 = min(src_L0, dst_L0);
+		/*
+		 * In pack_dst mode the destination is the (smaller) packed
+		 * prefix, so it must not clamp the source chunk size; the
+		 * source drives the walk and the destination is sized
+		 * independently from the changed-page count below.
+		 */
+		if (!pack_dst)
+			src_L0 = min(src_L0, dst_L0);
 
 		pte_flags = src_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
 		pte_flags |= use_comp_pat ? PTE_UPDATE_FLAG_IS_COMP_PTE : 0;
 		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
 					      &src_L0_ofs, &src_L0_pt, 0, 0,
 					      avail_pts);
+		/*
+		 * The source chunk size may have been clamped above, so derive
+		 * the packed destination size from the changed-page count of
+		 * the final chunk.
+		 */
+		if (pack_dst) {
+			nchg = xe_migrate_count_changed_pages(src_tt, dst_tt,
+							      start_page,
+							      src_L0 >> PAGE_SHIFT);
+			dst_size = (u64)nchg << PAGE_SHIFT;
+		} else {
+			dst_size = src_L0;
+		}
 		if (flags.copy_only_ccs) {
 			dst_L0_ofs = src_L0_ofs;
+		} else if (pack_dst && !nchg) {
+			/* All borrowed: no destination PTEs for this chunk. */
+			dst_L0_ofs = 0;
+			dst_L0_pt = 0;
 		} else {
 			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
 			batch_size += pte_update_size(m, pte_flags, dst,
-						      &dst_it, &src_L0,
+						      &dst_it, &dst_size,
 						      &dst_L0_ofs, &dst_L0_pt,
 						      0, avail_pts, avail_pts);
 		}
@@ -1072,26 +1277,31 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		 * is split into one command per run of non-borrowed pages, so
 		 * size for the number of runs actually needed.
 		 */
-		if (skip_borrowed)
+		if (defrag_runs)
 			ndata_blits = emit_copy_chunks(gt, NULL, 0, 0, src_L0,
-						       src_tt, dst_tt, start_page);
+						       src_tt, dst_tt, start_page,
+						       pack_dst);
 
 		/*
 		 * If every page in this chunk was borrowed there is nothing to
 		 * copy: don't build a batch or issue any GPU commands, just
 		 * advance the cursors past the chunk. The non-vram tt cursors
-		 * would otherwise be advanced by emit_pte().
+		 * would otherwise be advanced by emit_pte(). In pack_dst mode
+		 * the destination consumed no packed pages, so leave its cursor
+		 * untouched.
 		 */
-		if (skip_borrowed && !ndata_blits) {
+		if (defrag_runs && !ndata_blits) {
 			xe_res_next(&src_it, src_L0);
-			xe_res_next(&dst_it, src_L0);
+			if (!pack_dst)
+				xe_res_next(&dst_it, src_L0);
 			size -= src_L0;
 			continue;
 		}
 
 		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
 			       ndata_blits * emit_copy_cmd_len(xe)) +
-			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
+			(ccs_pack ? ndata_blits * EMIT_COPY_CCS_DW :
+			 (needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
 		if (IS_ERR(bb)) {
@@ -1111,7 +1321,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		else if (!flags.copy_only_ccs)
 			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
 				 copy_system_ccs || flags.defrag_ccs,
-				 &dst_it, src_L0, dst);
+				 &dst_it, dst_size, dst);
 
 		if (copy_system_ccs)
 			emit_pte(m, bb, ccs_pt, false, false, &ccs_it, ccs_size, src);
@@ -1121,16 +1331,22 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 
 		if (skip_borrowed)
 			emit_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0,
-					 src_tt, dst_tt, start_page);
+					 src_tt, dst_tt, start_page, pack_dst);
 		else if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
-		if (needs_ccs_emit)
+		if (ccs_pack) {
+			if (emit_ccs_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs,
+						 src_L0, src_tt, dst_tt,
+						 start_page))
+				flush_flags = MI_FLUSH_DW_CCS;
+		} else if (needs_ccs_emit) {
 			flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
 							  IS_DGFX(xe) ? src_is_vram : src_is_pltt,
 							  dst_L0_ofs,
 							  IS_DGFX(xe) ? dst_is_vram : dst_is_pltt,
 							  src_L0, ccs_ofs, copy_ccs);
+		}
 
 		job = xe_bb_create_migration_job(m->q, bb,
 						 xe_migrate_batch_base(m, usm),
@@ -1218,7 +1434,7 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs)
 {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL, 0,
 				 (struct xe_migrate_copy_flags) {
 					 .copy_only_ccs = copy_only_ccs,
 				 });
@@ -1267,7 +1483,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 bool need_ccs)
 {
 	struct dma_fence *fence2, *fence =
-		__xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt, 0,
 				  (struct xe_migrate_copy_flags) {
 					 .defrag_copy = true,
 					 });
@@ -1275,7 +1491,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	if (IS_ERR(fence) || !need_ccs)
 		return fence;
 
-	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt, 0,
 				   (struct xe_migrate_copy_flags) {
 				   .defrag_ccs = true,
 				   });
@@ -1286,6 +1502,59 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	return fence2;
 }
 
+/**
+ * xe_migrate_copy_defrag_iova() - Copy a defrag move into a packed IOVA dst.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented.
+ * @src: The source (old) TTM resource.
+ * @dst: The destination (new) TTM resource.
+ * @src_tt: The stashed source (old) tt providing the page pointers, fully
+ *	    IOVA-mapped at natural offsets.
+ * @dst_packed_pages: Number of changed pages packed into the destination tt's
+ *	    IOVA prefix by xe_tt_map_iova_copy().
+ * @need_ccs: Whether the BO has flat-CCS aux state that must be migrated too.
+ *
+ * Defrag copy variant for IOVA-mapped BOs where the destination only maps the
+ * changed (non-borrowed) pages, packed contiguously into the first
+ * @dst_packed_pages of its IOVA. The source is read from the old tt at its
+ * natural offsets and each changed run is blitted into the packed destination
+ * prefix. When @need_ccs is set a second pass migrates the flat-CCS aux state
+ * of the same changed pages indirectly through their (packed) data mappings;
+ * borrowed pages share the physical page and its aux, so only the changed runs
+ * are touched.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag_iova(struct xe_migrate *m,
+					      struct xe_bo *bo,
+					      struct ttm_resource *src,
+					      struct ttm_resource *dst,
+					      struct ttm_tt *src_tt,
+					      pgoff_t dst_packed_pages,
+					      bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt, dst_packed_pages,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+				 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+				   dst_packed_pages,
+				   (struct xe_migrate_copy_flags) {
+					 .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
+}
+
 /**
  * xe_migrate_resolve() - Resolve and decompress a buffer object if required.
  * @m: The migrate context
@@ -1301,7 +1570,7 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res)
 {
-	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL, 0,
 				 (struct xe_migrate_copy_flags) {
 					 .is_vram_resolve = true,
 				 });
diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
index 4094ea1c19c1..da066cdf01bf 100644
--- a/drivers/gpu/drm/xe/xe_migrate.h
+++ b/drivers/gpu/drm/xe/xe_migrate.h
@@ -141,6 +141,14 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct ttm_tt *src_tt,
 					 bool need_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag_iova(struct xe_migrate *m,
+					      struct xe_bo *bo,
+					      struct ttm_resource *src,
+					      struct ttm_resource *dst,
+					      struct ttm_tt *src_tt,
+					      pgoff_t dst_packed_pages,
+					      bool need_ccs);
+
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res);
-- 
2.34.1

[PATCH v3 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job

[Matthew Brost]

Wire up end-to-end defrag for IOVA-mapped BOs. A defrag move
is eligible whenever the destination TT is IOVA mapped. The
packed-copy step moves only changed pages; the IOVA reservation
is transferred to the new backing so device-visible addresses
are preserved, avoiding GPU rebind overhead.

Add xe_bo_defrag_use_iova() to select the IOVA defrag path.
Add xe_bo_defrag_iova_copy() to drive the move: diff new vs
stashed old TT, link changed pages packed, blit them, then
queue a post-copy finalize job.

The finalize job tears down the temporary packed mapping,
transfers the full old TT reservation to the new one, and
relinks the copied pages at natural offsets.

Borrowed pages skip the copy and simply adopt the old
reservation.

Since reservation transfer preserves device-visible addresses,
xe_bo_trigger_rebind() skips drm_gpuvm_bo_evict() marking;
GPU PTEs stay valid.

Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Check dma_iova_sync(), wait the copy fence before unlink on error,
   and gate the fast path on old_tt->use_iova (Sashiko)
---
 drivers/gpu/drm/xe/xe_bo.c | 325 +++++++++++++++++++++++++++++++++++--
 1 file changed, 311 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index c087f17ad6db..36290f58ccc1 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -13,6 +13,7 @@
 #include <drm/drm_dumb_buffers.h>
 #include <drm/drm_gem_ttm_helper.h>
 #include <drm/drm_managed.h>
+#include <drm/gpu_scheduler.h>
 #include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_device.h>
 #include <drm/ttm/ttm_placement.h>
@@ -25,6 +26,7 @@
 #include <trace/events/gpu_mem.h>
 
 #include "xe_device.h"
+#include "xe_dep_job_types.h"
 #include "xe_dep_scheduler.h"
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
@@ -542,7 +544,6 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
  * @xe_tt->iova_dma_pages[]. Used to (re)derive accounting that exactly matches
  * the pages a tt is actually mapped with.
  */
-__maybe_unused
 static void xe_tt_account_iova_pages(struct xe_device *xe,
 				     struct xe_ttm_tt *xe_tt)
 {
@@ -567,7 +568,6 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 {
 }
 
-__maybe_unused
 static void xe_tt_account_iova_pages(struct xe_device *xe,
 				     struct xe_ttm_tt *xe_tt)
 {
@@ -806,7 +806,6 @@ struct xe_tt_iova_copy {
  * differ. Coalesce contiguous changed folios into runs and, for each run,
  * invoke @fn. Returns the total number of changed pages.
  */
-__maybe_unused
 static pgoff_t xe_tt_iova_walk_changed(struct ttm_tt *new_tt,
 				       struct ttm_tt *old_tt,
 				       void (*fn)(pgoff_t start, pgoff_t count,
@@ -855,7 +854,6 @@ struct xe_tt_iova_run_collect {
 };
 
 /* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
-__maybe_unused
 static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 			   size_t elem)
 {
@@ -878,7 +876,6 @@ static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 	return 0;
 }
 
-__maybe_unused
 static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
 {
 	struct xe_tt_iova_run_collect *c = arg;
@@ -933,7 +930,6 @@ static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
  *
  * Return: 0 on success, -ENOMEM on allocation failure.
  */
-__maybe_unused
 static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
 				struct xe_tt_iova_copy *copy)
 {
@@ -969,7 +965,6 @@ static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
  * xe_tt_iova_copy_fini() - Release a struct xe_tt_iova_copy
  * @copy: The copy description to release.
  */
-__maybe_unused
 static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
 {
 	kfree(copy->runs);
@@ -1000,7 +995,6 @@ static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
  * Return: 0 on success, negative error code on failure (any partial linkage is
  * unwound).
  */
-__maybe_unused
 static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
 			       const struct xe_tt_iova_copy *copy)
 {
@@ -1047,6 +1041,133 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
 	return ret;
 }
 
+/**
+ * struct xe_iova_defrag_job - Post-copy finalize job for an IOVA defrag move.
+ *
+ * After the GPU defrag copy into the packed copy-step mapping completes, this
+ * dependency job runs the CPU-side IOMMU fixups at the point of no failure:
+ * tear down the temporary packed mapping, then per changed run unlink the old
+ * pages and link the new pages into the (transferred) reservation at their
+ * natural offset.
+ */
+struct xe_iova_defrag_job {
+	/** @dep: base generic dependency Xe job. */
+	struct xe_dep_job dep;
+	/** @xe: The xe device. */
+	struct xe_device *xe;
+	/** @new_tt: The destination tt being finalized (owns the reservation). */
+	struct ttm_tt *new_tt;
+	/**
+	 * @copy_iova: Snapshot of the new tt's original IOVA reservation that
+	 * held the temporary packed copy-step mapping. Destroyed by the job
+	 * (the new tt itself adopts the old tt's reservation at job creation).
+	 */
+	struct dma_iova_state copy_iova;
+	/** @copy_bytes: Byte length of the packed copy-step mapping. */
+	size_t copy_bytes;
+	/** @copy: Changed-page description; the job owns and frees @copy.runs. */
+	struct xe_tt_iova_copy copy;
+};
+
+static struct dma_fence *xe_iova_defrag_job_run(struct xe_dep_job *dep_job)
+{
+	struct xe_iova_defrag_job *job =
+		container_of(dep_job, typeof(*job), dep);
+	struct xe_device *xe = job->xe;
+	struct device *dev = xe->drm.dev;
+	struct xe_ttm_tt *xe_tt =
+		container_of(job->new_tt, struct xe_ttm_tt, ttm);
+	unsigned int r, l;
+
+	/*
+	 * Tear down the temporary packed copy-step mapping and free its
+	 * reservation. The new tt has already adopted the old tt's full
+	 * reservation (transferred at job creation), so this only touches the
+	 * snapshot taken before the transfer.
+	 *
+	 * This mapping was the GPU blit's write destination, so tear it down
+	 * with attrs 0 (not DMA_ATTR_SKIP_CPU_SYNC): for non-coherent or
+	 * bounced (SWIOTLB) mappings the device->CPU copy-back / cache sync of
+	 * the freshly copied data happens during unlink, exactly as
+	 * xe_tt_unmap_iova() does. Skipping it would silently drop the
+	 * defragmented data on those configs.
+	 */
+	if (job->copy_bytes)
+		dma_iova_destroy(dev, &job->copy_iova, job->copy_bytes,
+				 DMA_BIDIRECTIONAL, 0);
+	else
+		dma_iova_free(dev, &job->copy_iova);
+
+	/*
+	 * For each changed run, unlink the old pages (still linked from the
+	 * adopted reservation).
+	 */
+	for (r = 0; r < job->copy.nr_runs; r++) {
+		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
+		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
+
+		dma_iova_unlink(dev, &xe_tt->iova, off, len,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+	}
+
+	/*
+	 * Relink the freshly copied new pages from the snapshot captured at job
+	 * creation. The destination tt's page array may have been mutated (or
+	 * torn down) by TTM since then, so it is never dereferenced here.
+	 * Borrowed pages are shared with the old tt and stay correctly linked
+	 * for free, so the snapshot only covers the changed runs.
+	 */
+	for (l = 0; l < job->copy.nr_links; l++) {
+		int ret = dma_iova_link(dev, &xe_tt->iova,
+					job->copy.links[l].phys,
+					job->copy.links[l].offset,
+					job->copy.links[l].len,
+					DMA_BIDIRECTIONAL,
+					DMA_ATTR_SKIP_CPU_SYNC);
+		/*
+		 * Point of no failure: the data is already copied and the
+		 * reservation ownership has been transferred. A relink failure
+		 * leaves a hole in the mapping; there is no safe recovery, so
+		 * just warn.
+		 */
+		drm_WARN_ON(&xe->drm, ret);
+	}
+
+	/* Sync the relinked changed runs into the device's view. */
+	for (r = 0; r < job->copy.nr_runs; r++) {
+		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
+		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
+
+		/*
+		 * Point of no failure (as with the relink above): warn on a
+		 * sync error so a stale IOMMU cache does not fail silently.
+		 */
+		drm_WARN_ON(&xe->drm, dma_iova_sync(dev, &xe_tt->iova, off, len));
+	}
+
+	/*
+	 * All work is synchronous CPU-side IOMMU fixup, so there is no hardware
+	 * fence to wait on; returning NULL makes the scheduler signal the job's
+	 * finished fence immediately.
+	 */
+	return NULL;
+}
+
+static void xe_iova_defrag_job_free(struct xe_dep_job *dep_job)
+{
+	struct xe_iova_defrag_job *job =
+		container_of(dep_job, typeof(*job), dep);
+
+	xe_tt_iova_copy_fini(&job->copy);
+	drm_sched_job_cleanup(&job->dep.drm);
+	kfree(job);
+}
+
+static const struct xe_dep_job_ops xe_iova_defrag_job_ops = {
+	.run_job = xe_iova_defrag_job_run,
+	.free_job = xe_iova_defrag_job_free,
+};
+
 /*
  * xe_tt_iova_transfer - Transfer a full IOVA reservation old_tt -> new_tt.
  *
@@ -1063,7 +1184,6 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
  * folio walk of the new tt keeps xe->mem.dma_mapped_pages[] exact while the new
  * mapping lives, rather than carrying the old tt's stale distribution.
  */
-__maybe_unused
 static void xe_tt_iova_transfer(struct xe_device *xe,
 				struct xe_ttm_tt *new_tt,
 				struct xe_ttm_tt *old_tt)
@@ -1078,6 +1198,154 @@ static void xe_tt_iova_transfer(struct xe_device *xe,
 	old_tt->use_iova = false;
 }
 
+/*
+ * xe_bo_defrag_iova_copy - Run an IOVA defrag move with a packed copy mapping.
+ *
+ * Only the changed (non-borrowed) pages of @new_tt are DMA mapped, packed
+ * contiguously into the front of its reservation, the GPU copy blits them from
+ * the old tt at natural offsets, and a post-copy finalize job adopts the old
+ * tt's full reservation and relinks the changed runs. Returns the finalize
+ * job's finished fence (the fence the move waits on), or an ERR_PTR on failure
+ * with all partial state unwound.
+ */
+static struct dma_fence *
+xe_bo_defrag_iova_copy(struct xe_device *xe, struct xe_migrate *migrate,
+		       struct xe_bo *bo, struct ttm_resource *old_mem,
+		       struct ttm_resource *new_mem, struct ttm_tt *old_tt,
+		       bool handle_system_ccs)
+{
+	struct ttm_tt *new_tt = bo->ttm.ttm;
+	struct xe_ttm_tt *new_xe_tt = container_of(new_tt, struct xe_ttm_tt, ttm);
+	struct xe_ttm_tt *old_xe_tt = container_of(old_tt, struct xe_ttm_tt, ttm);
+	struct drm_sched_entity *entity =
+		xe_dep_scheduler_entity(xe->mem.defrag.iova_sched);
+	struct xe_iova_defrag_job *job;
+	struct dma_fence *copy_fence;
+	struct dma_fence *fence;
+	struct xe_tt_iova_copy copy;
+	bool packed = false;
+	int err;
+
+	xe_assert(xe, new_xe_tt->use_iova && !new_xe_tt->iova_linked);
+	xe_assert(xe, old_xe_tt->use_iova && old_xe_tt->iova_linked);
+
+	err = xe_tt_iova_copy_init(new_tt, old_tt, &copy);
+	if (err)
+		return ERR_PTR(err);
+
+	if (copy.nr_changed) {
+		err = xe_tt_map_iova_copy(xe, new_tt, &copy);
+		if (err)
+			goto err_copy;
+		packed = true;
+
+		copy_fence = xe_migrate_copy_defrag_iova(migrate, bo, old_mem,
+							 new_mem, old_tt,
+							 copy.nr_changed,
+							 handle_system_ccs);
+	} else {
+		copy_fence = dma_fence_get_stub();
+	}
+	if (IS_ERR(copy_fence)) {
+		err = PTR_ERR(copy_fence);
+		goto err_unmap;
+	}
+
+	job = kzalloc(sizeof(*job), GFP_KERNEL);
+	if (!job) {
+		err = -ENOMEM;
+		goto err_fence;
+	}
+
+	job->dep.ops = &xe_iova_defrag_job_ops;
+	job->xe = xe;
+	job->new_tt = new_tt;
+	job->copy_iova = new_xe_tt->iova;
+	job->copy_bytes = (size_t)copy.nr_changed << PAGE_SHIFT;
+	job->copy = copy;
+
+	err = drm_sched_job_init(&job->dep.drm, entity, 1, NULL, 0);
+	if (err)
+		goto err_job;
+
+	err = drm_sched_job_add_dependency(&job->dep.drm,
+					   dma_fence_get(copy_fence));
+	if (err)
+		goto err_job_init;
+
+	/*
+	 * Point of no failure: transfer the old tt's full reservation to the
+	 * new tt. The finalize job (already armed below) destroys the packed
+	 * copy mapping snapshot and relinks the changed runs.
+	 */
+	xe_tt_iova_transfer(xe, new_xe_tt, old_xe_tt);
+	memset(&copy, 0, sizeof(copy)); /* ownership moved to the job */
+
+	drm_sched_job_arm(&job->dep.drm);
+	fence = dma_fence_get(&job->dep.drm.s_fence->finished);
+	drm_sched_entity_push_job(&job->dep.drm);
+
+	dma_fence_put(copy_fence);
+
+	return fence;
+
+err_job_init:
+	drm_sched_job_cleanup(&job->dep.drm);
+err_job:
+	kfree(job);
+err_fence:
+	/*
+	 * The GPU copy into the packed destination mapping was already
+	 * submitted; it must finish before we tear that mapping down below,
+	 * otherwise the blit would fault or scribble on unmapped IOVA. (Reached
+	 * only with a valid, submitted copy_fence; the IS_ERR(copy_fence) case
+	 * jumps straight to err_unmap with no GPU work in flight.)
+	 */
+	if (packed)
+		dma_fence_wait(copy_fence, false);
+	dma_fence_put(copy_fence);
+err_unmap:
+	if (packed)
+		dma_iova_unlink(xe->drm.dev, &new_xe_tt->iova, 0,
+				(size_t)copy.nr_changed << PAGE_SHIFT,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+err_copy:
+	xe_tt_iova_copy_fini(&copy);
+	return ERR_PTR(err);
+}
+
+/*
+ * xe_bo_defrag_use_iova - Whether a defrag move should use the packed IOVA path.
+ *
+ * Eligible when the move is a defrag (stashed old tt) and the destination tt is
+ * IOVA mapped. The IOVA reservation is transferred to the new backing, so the
+ * device-visible addresses are preserved and no GPU rebind is needed (see
+ * xe_bo_trigger_rebind()); this makes the path worthwhile regardless of BO
+ * size. A flat-CCS aux pass, if needed, is migrated packed alongside the data
+ * (changed pages only), so it does not disqualify the path. Otherwise the
+ * full-mapping defrag path is used.
+ */
+static bool xe_bo_defrag_use_iova(struct ttm_buffer_object *ttm_bo)
+{
+	struct xe_ttm_tt *xe_tt, *old_xe_tt;
+
+	if (!ttm_bo->defrag_old_tt || !ttm_bo->ttm)
+		return false;
+
+	xe_tt = container_of(ttm_bo->ttm, struct xe_ttm_tt, ttm);
+	old_xe_tt = container_of(ttm_bo->defrag_old_tt, struct xe_ttm_tt, ttm);
+
+	/*
+	 * Both tts must be IOVA mapped: the packed copy reads the old tt's IOVA
+	 * and the finalize job transfers the old tt's reservation to the new
+	 * one. use_iova is a deterministic per-device decision taken at tt
+	 * creation, so for a given BO the old and new tt always agree; the
+	 * explicit old-tt check guards against ever routing a sg-fallback old
+	 * tt into xe_tt_iova_transfer() (which would copy an invalid iova).
+	 */
+	return xe_tt->use_iova && old_xe_tt->use_iova;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
@@ -1357,6 +1625,7 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 	struct drm_gem_object *obj = &bo->ttm.base;
 	struct drm_gpuvm_bo *vm_bo;
 	bool idle = false;
+	bool defrag_iova;
 	int ret = 0;
 
 	dma_resv_assert_held(bo->ttm.base.resv);
@@ -1369,12 +1638,25 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 		dma_resv_iter_end(&cursor);
 	}
 
+	/*
+	 * A defrag move that reallocates IOVA-mapped backing preserves the
+	 * device-visible addresses: the IOVA reservation is transferred from
+	 * the old to the new tt, so the existing GPU PTEs stay valid and the
+	 * BO must not be marked evicted (the rebind paths in xe_exec_ioctl()
+	 * and preempt_rebind_work_func() would otherwise needlessly rebind).
+	 * Preempt fences still need to be triggered, and fault-mode / L2
+	 * flush-optimized configs still need a TLB invalidation below, so only
+	 * the eviction marking is skipped.
+	 */
+	defrag_iova = xe_bo_defrag_use_iova(&bo->ttm);
+
 	drm_gem_for_each_gpuvm_bo(vm_bo, obj) {
 		struct xe_vm *vm = gpuvm_to_vm(vm_bo->vm);
 		struct drm_gpuva *gpuva;
 
 		if (!xe_vm_in_fault_mode(vm)) {
-			drm_gpuvm_bo_evict(vm_bo, true);
+			if (!defrag_iova)
+				drm_gpuvm_bo_evict(vm_bo, true);
 			/*
 			 * L2 cache may not be flushed, so ensure that is done in
 			 * xe_vm_invalidate_vma() below
@@ -1394,6 +1676,8 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			if (!ctx->no_wait_gpu &&
 			    !xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
 				continue;
+			else if (defrag_iova)
+				drm_gpuvm_bo_evict(vm_bo, true);
 		}
 
 		if (!idle) {
@@ -2120,6 +2404,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	bool needs_clear;
 	bool handle_system_ccs = (!IS_DGFX(xe) && xe_bo_needs_ccs_pages(bo) &&
 				  ttm && ttm_tt_is_populated(ttm)) ? true : false;
+	bool defrag_iova = xe_bo_defrag_use_iova(ttm_bo);
 	int ret = 0;
 
 	/*
@@ -2168,7 +2453,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	needs_clear = (ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) ||
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
-	if (new_mem->mem_type == XE_PL_TT) {
+	/*
+	 * For a defrag move using the packed IOVA path the destination tt is
+	 * not fully mapped here; only the changed pages are mapped (packed) for
+	 * the copy, and the finalize job establishes the full mapping.
+	 */
+	if (new_mem->mem_type == XE_PL_TT && !defrag_iova) {
 		ret = xe_tt_map(xe, ttm);
 		if (ret)
 			goto out;
@@ -2278,9 +2568,16 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
 	} else if (ttm_bo->defrag_old_tt) {
-		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
-					       ttm_bo->defrag_old_tt,
-					       handle_system_ccs);
+		if (defrag_iova)
+			fence = xe_bo_defrag_iova_copy(xe, migrate, bo, old_mem,
+						       new_mem,
+						       ttm_bo->defrag_old_tt,
+						       handle_system_ccs);
+		else
+			fence = xe_migrate_copy_defrag(migrate, bo, old_mem,
+						       new_mem,
+						       ttm_bo->defrag_old_tt,
+						       handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
-- 
2.34.1

[PATCH v3 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock

[Matthew Brost]

Populating a GTT (system) buffer object under the reservation lock can
stall in reclaim and compaction while trying to satisfy beneficial-order
allocations, holding the lock for the duration.

Mirror the Xe out-of-lock preallocation for pure system BOs: when a
gem_create request targets AMDGPU_GEM_DOMAIN_GTT only, fill the full page
backing up front via ttm_pool_prealloc_fill_full() before taking the
reservation lock. The populate under the lock then simply installs these
pages instead of reclaiming and compacting in the critical section. The
fill is best-effort - a short fill falls back to the normal in-lock
allocation for the remaining pages, and any leftover pages are released
by ttm_pool_prealloc_fini() on all exit paths.

Unlike Xe, amdgpu has no background defragmenter, so the higher-order
reclaim backoff is left disabled (normal reclaim). If the pool uses
dma-alloc (swiotlb), the fill bails and the feature is a silent no-op.

Factor the tt pool selection into amdgpu_ttm_tt_pool_id() and
amdgpu_ttm_pool() so the preallocation targets the exact pool the
populate will consume, and thread an optional prealloc bag through
amdgpu_gem_object_create() and amdgpu_bo_param into the populate ctx.

Cc: amd-gfx@lists.freedesktop.org
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Carlos Santa <carlos.santa@intel.com>
Cc: Ryan Neph <ryanneph@google.com>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: David Airlie <airlied@gmail.com>
Cc: Simona Vetter <simona@ffwll.ch>
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Matthew Brost <matthew.brost@intel.com>

---

v3:
 - Keep WC caching only when USWC is supported and bound the
   preallocation to available GTT space (Sashiko)
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 73 +++++++++++++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |  5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |  3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 39 +++++++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |  3 +
 8 files changed, 108 insertions(+), 25 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 20831dbebc31..940f58848a97 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -345,7 +345,7 @@ create_dmamap_sg_bo(struct amdgpu_device *adev,
 
 	ret = amdgpu_gem_object_create(adev, mem->bo->tbo.base.size, 1,
 			AMDGPU_GEM_DOMAIN_CPU, AMDGPU_GEM_CREATE_PREEMPTIBLE | flags,
-			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0);
+			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0, NULL);
 
 	amdgpu_bo_unreserve(mem->bo);
 
@@ -1811,7 +1811,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
 		 domain_string(alloc_domain), xcp_id);
 
 	ret = amdgpu_gem_object_create(adev, aligned_size, 1, alloc_domain, alloc_flags,
-				       bo_type, NULL, &gobj, xcp_id + 1);
+				       bo_type, NULL, &gobj, xcp_id + 1, NULL);
 	if (ret) {
 		pr_debug("Failed to create BO on domain %s. ret %d\n",
 			 domain_string(alloc_domain), ret);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index b33c300e26e2..51510e831129 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -435,7 +435,7 @@ amdgpu_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
 
 	ret = amdgpu_gem_object_create(adev, dma_buf->size, PAGE_SIZE,
 				       AMDGPU_GEM_DOMAIN_CPU, flags,
-				       ttm_bo_type_sg, resv, &gobj, 0);
+				       ttm_bo_type_sg, resv, &gobj, 0, NULL);
 	if (ret)
 		goto error;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index 6a0699746fbc..e8b732218e7d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -39,6 +39,7 @@
 #include <drm/drm_exec.h>
 #include <drm/drm_gem_ttm_helper.h>
 #include <drm/ttm/ttm_tt.h>
+#include <drm/ttm/ttm_pool.h>
 #include <drm/drm_syncobj.h>
 
 #include "amdgpu.h"
@@ -168,7 +169,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
 			     int alignment, u32 initial_domain,
 			     u64 flags, enum ttm_bo_type type,
 			     struct dma_resv *resv,
-			     struct drm_gem_object **obj, int8_t xcp_id_plus1)
+			     struct drm_gem_object **obj, int8_t xcp_id_plus1,
+			     struct ttm_pool_prealloc *prealloc)
 {
 	struct amdgpu_bo *bo;
 	struct amdgpu_bo_user *ubo;
@@ -188,6 +190,7 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
 	bp.domain = initial_domain;
 	bp.bo_ptr_size = sizeof(struct amdgpu_bo);
 	bp.xcp_id_plus1 = xcp_id_plus1;
+	bp.prealloc = prealloc;
 
 	r = amdgpu_bo_create_user(adev, &bp, &ubo);
 	if (r)
@@ -412,6 +415,8 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 	struct dma_resv *resv = NULL;
 	struct drm_gem_object *gobj;
 	uint32_t handle, initial_domain;
+	struct ttm_pool_prealloc prealloc = {};
+	struct ttm_pool *prealloc_pool = NULL;
 	int r;
 
 	/* reject invalid gem flags */
@@ -443,10 +448,57 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 		flags |= AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
 	}
 
+	/*
+	 * For system-only (pure GTT) BOs, preallocate the whole page backing
+	 * up front, outside the reservation lock. Populate under the lock then
+	 * just installs these pages instead of reclaiming/compacting in the
+	 * critical section. Best-effort: a short fill falls back to the normal
+	 * in-lock allocation for the missing pages.
+	 */
+	if (args->in.domains == AMDGPU_GEM_DOMAIN_GTT) {
+		struct ttm_resource_manager *gtt_man =
+			ttm_manager_type(&adev->mman.bdev, TTM_PL_TT);
+		int32_t xcp_id = adev->gmc.mem_partitions ? fpriv->xcp_id : 0;
+		int32_t pool_id = amdgpu_ttm_tt_pool_id(adev, xcp_id);
+		/*
+		 * Mirror the USWC handling in amdgpu_bo_create(): the flag is
+		 * stripped when the platform can't do write-combining, in which
+		 * case the tt is created ttm_cached. The prealloc caching must
+		 * match the final tt caching exactly, otherwise ttm_pool_free()
+		 * would restore the wrong PAT state on these pages (PAT aliasing
+		 * / leak of WC pages into the cached allocator).
+		 */
+		bool uswc = (flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC) &&
+			amdgpu_bo_support_uswc(flags);
+		enum ttm_caching caching =
+			uswc ? ttm_write_combined : ttm_cached;
+
+		/*
+		 * Only prealloc when the request fits the currently-available
+		 * GTT (total manager size minus current usage). This mirrors the
+		 * amdgpu_bo_validate_size() bound applied later in
+		 * amdgpu_bo_create() while also skipping the up-front reclaim /
+		 * compaction when GTT is already near full or the user-controlled
+		 * size is over-large (the creation path handles those anyway).
+		 */
+		if (gtt_man) {
+			u64 used = ttm_resource_manager_usage(gtt_man);
+			u64 avail = gtt_man->size > used ?
+				gtt_man->size - used : 0;
+
+			if (size <= avail) {
+				prealloc_pool = amdgpu_ttm_pool(adev, pool_id);
+				ttm_pool_prealloc_fill_full(prealloc_pool,
+							    caching, &prealloc,
+							    PFN_UP(size), false);
+			}
+		}
+	}
+
 	if (flags & AMDGPU_GEM_CREATE_VM_ALWAYS_VALID) {
 		r = amdgpu_bo_reserve(vm->root.bo, false);
 		if (r)
-			return r;
+			goto out_prealloc;
 
 		resv = vm->root.bo->tbo.base.resv;
 	}
@@ -455,7 +507,8 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 retry:
 	r = amdgpu_gem_object_create(adev, size, args->in.alignment,
 				     initial_domain,
-				     flags, ttm_bo_type_device, resv, &gobj, fpriv->xcp_id + 1);
+				     flags, ttm_bo_type_device, resv, &gobj,
+				     fpriv->xcp_id + 1, prealloc_pool ? &prealloc : NULL);
 	if (r && r != -ERESTARTSYS) {
 		if (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) {
 			flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
@@ -479,17 +532,21 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 		amdgpu_bo_unreserve(vm->root.bo);
 	}
 	if (r)
-		return r;
+		goto out_prealloc;
 
 	r = drm_gem_handle_create(filp, gobj, &handle);
 	/* drop reference from allocate - handle holds it now */
 	drm_gem_object_put(gobj);
 	if (r)
-		return r;
+		goto out_prealloc;
 
 	memset(args, 0, sizeof(*args));
 	args->out.handle = handle;
-	return 0;
+
+out_prealloc:
+	if (prealloc_pool)
+		ttm_pool_prealloc_fini(prealloc_pool, &prealloc);
+	return r;
 }
 
 int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,
@@ -528,7 +585,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,
 
 	/* create a gem object to contain this object in */
 	r = amdgpu_gem_object_create(adev, args->size, 0, AMDGPU_GEM_DOMAIN_CPU,
-				     0, ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1);
+				     0, ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1, NULL);
 	if (r)
 		return r;
 
@@ -1298,7 +1355,7 @@ int amdgpu_mode_dumb_create(struct drm_file *file_priv,
 	domain = amdgpu_bo_get_preferred_domain(adev,
 				amdgpu_display_supported_domains(adev, flags));
 	r = amdgpu_gem_object_create(adev, args->size, 0, domain, flags,
-				     ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1);
+				     ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1, NULL);
 	if (r)
 		return -ENOMEM;
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
index b558336bc4c6..706aeca011f1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
@@ -35,6 +35,8 @@
 
 extern const struct drm_gem_object_funcs amdgpu_gem_object_funcs;
 
+struct ttm_pool_prealloc;
+
 unsigned long amdgpu_gem_timeout(uint64_t timeout_ns);
 
 /*
@@ -45,7 +47,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
 			     int alignment, u32 initial_domain,
 			     u64 flags, enum ttm_bo_type type,
 			     struct dma_resv *resv,
-			     struct drm_gem_object **obj, int8_t xcp_id_plus1);
+			     struct drm_gem_object **obj, int8_t xcp_id_plus1,
+			     struct ttm_pool_prealloc *prealloc);
 int amdgpu_mode_dumb_create(struct drm_file *file_priv,
 			    struct drm_device *dev,
 			    struct drm_mode_create_dumb *args);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index f98bfba59a2c..18c4cf3f35a5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -632,7 +632,8 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 		/* We opt to avoid OOM on system pages allocations */
 		.gfp_retry_mayfail = true,
 		.allow_res_evict = bp->type != ttm_bo_type_kernel,
-		.resv = bp->resv
+		.resv = bp->resv,
+		.prealloc = bp->prealloc,
 	};
 	struct amdgpu_bo *bo;
 	unsigned long page_align, size = bp->size;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index ff11a0903499..11f1d403f152 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -45,6 +45,8 @@
 #define to_amdgpu_bo_user(abo) container_of((abo), struct amdgpu_bo_user, bo)
 #define to_amdgpu_bo_vm(abo) container_of((abo), struct amdgpu_bo_vm, bo)
 
+struct ttm_pool_prealloc;
+
 struct amdgpu_bo_param {
 	unsigned long			size;
 	int				byte_align;
@@ -58,6 +60,8 @@ struct amdgpu_bo_param {
 	void				(*destroy)(struct ttm_buffer_object *bo);
 	/* xcp partition number plus 1, 0 means any partition */
 	int8_t				xcp_id_plus1;
+	/* optional out-of-lock preallocated backing (system/GTT only) */
+	struct ttm_pool_prealloc	*prealloc;
 };
 
 /* bo virtual addresses in a vm */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index b10b0878df37..c9cd4714f515 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1173,6 +1173,30 @@ void amdgpu_ttm_mmio_remap_free_sgt(struct device *dev,
 	kfree(sgt);
 }
 
+/*
+ * amdgpu_ttm_tt_pool_id - compute the ttm pool id backing a given xcp
+ *
+ * Mirrors the mapping used when creating the gtt ttm_tt, so callers that only
+ * have an xcp id (e.g. an out-of-lock preallocation before the bo exists) pick
+ * the same pool the populate will use.
+ */
+int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id)
+{
+	if (adev->gmc.mem_partitions && xcp_id >= 0)
+		return KFD_XCP_MEM_ID(adev, xcp_id);
+
+	return xcp_id;
+}
+
+/* amdgpu_ttm_pool - select the ttm pool for a given pool id */
+struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id)
+{
+	if (adev->mman.ttm_pools && pool_id >= 0)
+		return &adev->mman.ttm_pools[pool_id];
+
+	return &adev->mman.bdev.pool;
+}
+
 /**
  * amdgpu_ttm_tt_create - Create a ttm_tt object for a given BO
  *
@@ -1194,10 +1218,7 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo,
 		return NULL;
 
 	gtt->gobj = &bo->base;
-	if (adev->gmc.mem_partitions && abo->xcp_id >= 0)
-		gtt->pool_id = KFD_XCP_MEM_ID(adev, abo->xcp_id);
-	else
-		gtt->pool_id = abo->xcp_id;
+	gtt->pool_id = amdgpu_ttm_tt_pool_id(adev, abo->xcp_id);
 
 	if (abo->flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC)
 		caching = ttm_write_combined;
@@ -1239,10 +1260,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev,
 	if (ttm->page_flags & TTM_TT_FLAG_EXTERNAL)
 		return 0;
 
-	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
-		pool = &adev->mman.ttm_pools[gtt->pool_id];
-	else
-		pool = &adev->mman.bdev.pool;
+	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
 	ret = ttm_pool_alloc(pool, ttm, ctx);
 	if (ret)
 		return ret;
@@ -1284,10 +1302,7 @@ static void amdgpu_ttm_tt_unpopulate(struct ttm_device *bdev,
 
 	adev = amdgpu_ttm_adev(bdev);
 
-	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
-		pool = &adev->mman.ttm_pools[gtt->pool_id];
-	else
-		pool = &adev->mman.bdev.pool;
+	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
 
 	return ttm_pool_free(pool, ttm);
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
index ff9e2e346609..f90844937b73 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
@@ -207,6 +207,9 @@ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo);
 void amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo);
 uint64_t amdgpu_ttm_domain_start(struct amdgpu_device *adev, uint32_t type);
 
+int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id);
+struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id);
+
 #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
 int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo,
 				 struct amdgpu_hmm_range *range);
-- 
2.34.1

Re: [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] An empty preallocation bag causes a defrag move to perform all allocations in-line under the dma-resv lock, causing severe reclaim/compaction stalls and defeating the design goal.
--

--- Patch [9]: [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock ---
commit a99093b0428c713eac3edfaabcb993852aeabdbd
Author: Matthew Brost <matthew.brost@intel.com>

drm/ttm: Preallocate beneficial-order defrag pages outside the lock

This commit introduces a preallocation bag for TTM pool allocations to
hoist expensive high-order allocations out of the dma-resv critical
section. It records suboptimal pages during populate so a defragmentation
pass can appropriately size its out-of-lock preallocation, and harvests the
remainder of the old tt if the preallocation runs out.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c

[ ... ]

> @@ -990,16 +1168,20 @@ struct ttm_pool_alloc_iter {
>  };
>  
>  /*
> - * Defrag move budget exhausted: the upgrade can make no further progress this
> - * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
> - * entry.
> + * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
> + * upgrade can make no further progress this pass. Snapshot @defrag_capped is
> + * set only when a byte budget was in force at entry.
>   */
>  static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
>  {
> +	const struct ttm_pool_prealloc *pp = it->prealloc;
> +
>  	if (!it->defrag_old_tt)
>  		return false;
> +	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
> +		return true;
>  
> -	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
> +	return pp && pp->count && pp->used >= pp->count;

[Severity: High]
If the out-of-lock preallocation fails to allocate any pages due to memory
pressure, pp->count will be 0. Does the && pp->count condition cause this
function to return false in that scenario?

If this returns false for an empty preallocation bag, __ttm_pool_alloc()
will bypass the exhaustion check. The allocator would then proceed with
in-line system page allocations while holding the dma-resv lock.

Because an empty bag is most likely on a fragmented system, these in-line
allocations could stall aggressively in direct reclaim and compaction,
which seems to bypass the goal of hoisting these allocations out of the
critical section.

Could this check be updated to return pp && pp->used >= pp->count; so
that an empty bag correctly aborts the defrag move?

>  }
>  
>  /*

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=9

Re: [PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [High] Uninitialized memory read of `svm_pages->dma_addr[0].dir` passed to `dma_iova_destroy()` on the error path.
- [High] System memory pages are erroneously unmapped using the device-specific unmap callback in mixed SVM ranges.
- [High] Device memory DMA mappings are leaked on the error path because `svm_pages->dpagemap` is assigned too late.
--

--- Patch [11]: [PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback ---
commit c12d7531ab5aab8f1172b95a5e7473303dec85e1
Author: Thomas Hellström <thomas.hellstrom@linux.intel.com>

drm/gpusvm: Add a DMA-mapping accounting callback

The patch adds a dma_map_account callback to drm_gpusvm_ops to allow drivers
to symmetrically account for DMA-mapped pages during map and unmap operations.

> diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
> index fcfe635bc1956..512f52471419e 100644
> --- a/drivers/gpu/drm/drm_gpusvm.c
> +++ b/drivers/gpu/drm/drm_gpusvm.c

[ ... ]

> @@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
>  			else if (dpagemap && dpagemap->ops->device_unmap)
>  				dpagemap->ops->device_unmap(dpagemap,
>  							    dev, addr);

[Severity: High]
This isn't a bug introduced by this patch, but does the error path in
drm_gpusvm_get_pages() trigger an uninitialized memory read here?

If the first page's mapping fails (e.g., dma_iova_link() fails), the code
jumps to the err_unmap label with num_dma_mapped = 0. Because
dma_iova_try_alloc() succeeded, use_iova evaluates to true in
__drm_gpusvm_unmap_pages(). This unconditionally executes dma_iova_destroy()
just before this loop, reading svm_pages->dma_addr[0].dir by value:

    if (use_iova)
        dma_iova_destroy(dev, &svm_pages->state, svm_pages->state_offset,
                         svm_pages->dma_addr[0].dir, 0);

Since svm_pages->dma_addr was allocated using kvmalloc_objs without
zero-initialization, this passes an uninitialized enum dma_data_direction into
a kernel DMA API, which can trigger KMSAN panics or BUG_ON checks in DMA
debugging routines.

[Severity: High]
This is a pre-existing issue, but does this logic misroute system memory pages
to the device unmap callback?

When ctx->allow_mixed is true, an SVM range can contain both system and device
pages. In this scenario, use_iova evaluates to true, and svm_pages->dpagemap
is initialized to the device's pagemap. During __drm_gpusvm_unmap_pages(), the
loop iterates over all pages. For a system page, addr->proto ==
DRM_INTERCONNECT_SYSTEM is true, but !use_iova evaluates to false.

This causes the execution to fall through to the else if condition without
validating the page type:

    else if (dpagemap && dpagemap->ops->device_unmap)
        dpagemap->ops->device_unmap(dpagemap, dev, addr);

This incorrectly executes the device's unmap callback on a system memory page
(containing an IOVA address), which seems likely to cause memory corruption or
a crash.

[Severity: High]
This is also a pre-existing issue, but are device memory DMA mappings leaked
on the error path because svm_pages->dpagemap is assigned too late?

If an error occurs during drm_gpusvm_get_pages() after successfully mapping
one or more device private pages, the code jumps to the err_unmap cleanup
path. However, svm_pages->dpagemap is only updated at the very end of the
success path:

    if (pagemap) {
        flags.has_devmem_pages = true;
        drm_pagemap_get(dpagemap);
        drm_pagemap_put(svm_pages->dpagemap);
        svm_pages->dpagemap = dpagemap;
    }

Consequently, inside __drm_gpusvm_unmap_pages(), the local dpagemap variable
is NULL, causing the condition dpagemap && dpagemap->ops->device_unmap to
evaluate to false. The device unmap callback is skipped, leaking the device
memory mappings. The newly added dma_map_account callback will un-account
these leaked pages, causing internal accounting to irreversibly drift from the
hardware state.

> +			if (gpusvm->ops && gpusvm->ops->dma_map_account)
> +				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
>  			i += 1 << addr->order;
>  		}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=11

Re: [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The beneficial-order reclaim backoff optimization is inadvertently skipped when restoring backed-up buffer objects (swap-ins).
--

--- Patch [16]: [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure ---
commit 2099c6d8da1f6677d9db38b2ffe92ba3eb96f8f0
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Back off beneficial-order reclaim under defrag pressure

Once a meaningful number of BOs are tracked as being backed at a
sub-optimal order, this commit configures the TTM pool to back off from
aggressive reclaim during populate. This allows allocations to make
forward progress, leaving upgrades to beneficial orders for a deferred
defrag pass.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
>  	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
>  		err = ttm_tt_restore(ttm_dev, tt, ctx);
>  	} else {
> +		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
> +
> +		if (atomic_read(&xe->mem.defrag.count) >=
> +		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
> +			ctx->beneficial_reclaim_backoff = true;
> +

[Severity: Medium]
Does this unintentionally skip the backoff optimization for swap-ins?

Because the beneficial_reclaim_backoff flag is only set inside the else
branch, any swapped-out buffer object being restored via ttm_tt_restore()
under fragmentation pressure will miss this optimization.

Without the flag, ttm_tt_restore() will continue pushing hard for
beneficial-order pages, potentially causing the exact reclaim and
compaction stalls this commit intended to avoid.

Would it make sense to move the device defrag count check and flag
evaluation above the ttm_tt_is_backed_up() check so both paths benefit
from the optimization?

>  		ttm_tt_clear_backed_up(tt);
>  		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
>  	}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=16

Re: [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Negative defrag budget values are incorrectly treated as unlimited instead of exhausted.
--

--- Patch [8]: [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move ---
commit 933e390472c0331c941d5caa6b241841ec7bb236
Author: Matthew Brost <matthew.brost@intel.com>

drm/ttm: Bound page (re)allocation per defragmentation move

This commit introduces a per-move byte budget to bound the reallocation
size during TTM object defragmentation. It updates the pool allocator to
track upgraded pages against this budget, stopping the upgrade process and
harvesting the remaining old pages once the budget is exhausted.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index 52f5e64f05ddc..1673e81b8df81 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c

[ ... ]

> @@ -1121,10 +1187,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
>  		gfp_flags |= GFP_HIGHUSER;
>  
>  	it.gfp_flags = gfp_flags;
> +	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;

[Severity: Medium]
Does this initialization correctly handle negative budget values?

If a caller uses the same ttm_operation_ctx for multiple defragmentation
moves, the first move that overshoots the budget can leave
defrag_bytes_remaining as a negative value.

When this negative value is passed to the next __ttm_pool_alloc() call,
the alloc->defrag_bytes_remaining > 0 check evaluates to false.

Will this disable the cap entirely (defrag_capped = false) and cause the
allocator to perform an unlimited upgrade, potentially reintroducing the
unbounded stall this budget was designed to prevent? Should this condition
be != 0 instead to properly enforce negative or exhausted budgets?

>  	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
>  	     alloc->remaining_pages;
>  	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=8

Re: [PATCH v3 21/33] drm/xe: Add defrag GT stats

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The `defrag_mb_moved` counter loses precision and will significantly undercount the amount of data moved.
--

--- Patch [21]: [PATCH v3 21/33] drm/xe: Add defrag GT stats ---
commit 84ae24372375a8f7d60dca398232ca340eba4dd8
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Add defrag GT stats

This commit introduces observability for the page defragmentation worker by
adding GT statistics counters. It tracks counts for buffer objects added to
the defrag list, successful defragmentations, partial successes, failures,
and the amount of data moved in megabytes.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 907cfa57481f4..073d926ef9ace 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -1274,8 +1277,21 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
>  	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
>  	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
>  
> -	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm))
> +	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {
>  		*needs_more = true;
> +		xe_gt_stats_incr(xe_root_mmio_gt(xe),
> +				 XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
> +				 1);
> +	} else {
> +		xe_gt_stats_incr(xe_root_mmio_gt(xe),
> +				 ret ? XE_GT_STATS_ID_DEFRAG_FAILED_COUNT :
> +				       XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT, 1);
> +	}
> +
> +	if (!ret)
> +		xe_gt_stats_incr(xe_root_mmio_gt(xe),
> +				 XE_GT_STATS_ID_DEFRAG_MB_MOVED,
> +				 *consumed >> 20);

[Severity: Medium]
Does this right-shift truncate the accumulation of data moved? If *consumed
is less than 1 MiB (for example, a 64 KiB chunk), *consumed >> 20 evaluates
to 0, meaning it will not be counted at all.

Could this lead to significantly undercounting the actual data moved? Would
it be better to accumulate raw bytes in the counter and only convert to
megabytes when formatting the output for userspace?

>  
>  unlock:
>  	xe_bo_unlock(bo);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=21

Re: [PATCH v3 05/33] drm/ttm: Support defragmentation moves

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Data loss due to `ttm_bo_move_memcpy` and driver hardware copy routines ignoring `bo->defrag_old_tt`.
- [Critical] Asynchronous GPU copy Use-After-Free due to synchronous teardown on defrag move failure.
- [High] Victim BOs during multihop bounce moves are misclassified as defrag targets.
--

--- Patch [5]: [PATCH v3 05/33] drm/ttm: Support defragmentation moves ---
commit fba7dcbd46a60b9e14c15b0602b46ebb23a7bc37
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/ttm: Support defragmentation moves

This commit introduces infrastructure to re-back an already-resident buffer
object with fresh pages at a more beneficial order. It stashes the old
backing in a new defrag_old_tt pointer and creates a fresh one during the
move.

> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -125,49 +125,133 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
>  {
>  	struct ttm_device *bdev = bo->bdev;
>  	bool old_use_tt, new_use_tt;
> +	bool defrag;
>  	int ret;
>  
>  	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
>  	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
>  
> -	ttm_bo_unmap_virtual(bo);
> +	/*
> +	 * Only the BO that is the actual defrag target (moved in place via
> +	 * ttm_bo_validate(), evict == false) is being defragmented. The same
> +	 * ctx->defrag is threaded into evictions triggered while allocating
> +	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
> +	 * an unrelated victim BO (evict == true) being misclassified as a
> +	 * defrag move and needlessly reallocating its backing.
> +	 */
> +	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
> +		 ttm_tt_is_populated(bo->ttm);

[Severity: High]
Are victim buffer objects misclassified as defragmentation targets during
multihop bounce moves?

When a victim BO's eviction requires a multihop bounce, ttm_bo_evict()
calls ttm_bo_bounce_temp_buffer(), which subsequently calls
ttm_bo_handle_move_mem() with evict = false.

If ctx->defrag is true from the surrounding defragmentation pass, this
condition will evaluate to true for the bounce move.

Could this unnecessarily reallocate the victim's backing, and subsequently
trigger the WARN_ON_ONCE below if a bounce is requested?

[ ... ]

> +	/*
> +	 * A defrag move deferred its virtual unmap until the new backing was
> +	 * successfully populated; tear the userspace mappings down now, before
> +	 * the move swaps in the new pages, so the next fault re-establishes
> +	 * them against the new backing.
> +	 */
> +	if (defrag)
> +		ttm_bo_unmap_virtual(bo);
> +
>  	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);

[Severity: Critical]
Does this result in data loss if the driver falls back to the generic copy
routines?

If a driver opts into defragmentation via ctx->defrag = true,
ttm_bo_handle_move_mem() allocates a new TT backing (bo->ttm) and stashes
the old data in bo->defrag_old_tt before calling the driver's move callback.

If the driver falls back to the generic ttm_bo_move_memcpy() or uses a
hardware copy that directly relies on bo->ttm:

drivers/gpu/drm/ttm/ttm_bo_util.c:ttm_bo_move_memcpy() {
    ...
    dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
    ...
    src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
    ...
}

Won't both the source and destination initialize using the newly allocated
uninitialized bo->ttm, copying garbage onto itself, while the actual data
in bo->defrag_old_tt is ignored and eventually freed?

[ ... ]

>  out_err:
> -	if (!old_use_tt)
> +	if (bo->defrag_old_tt) {
> +		/* Failed defrag move: restore the original backing. */
> +		ttm_bo_tt_destroy(bo);

[Severity: Critical]
Can this synchronously free the backing memory while an asynchronous GPU copy
is still executing?

During a defrag move, the driver's move callback may launch an asynchronous
hardware copy to populate the new TT backing.

If a subsequent cleanup step fails (for example, ttm_bo_move_accel_cleanup()
returning -ENOMEM), the failure falls through to this out_err block.

If the memory is synchronously freed here using ttm_bo_tt_destroy(), won't
the GPU continue executing the copy into the now-freed system pages,
causing memory corruption?

> +		bo->ttm = bo->defrag_old_tt;
> +		bo->defrag_old_tt = NULL;
> +	} else if (!old_use_tt) {
>  		ttm_bo_tt_destroy(bo);
> +	}
>  
>  	return ret;
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=5

Re: [PATCH v3 27/33] drm/xe: Add IOVA-based xe_res_cursor variant

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The newly added `is_iova` block in `xe_res_next()` is mathematically unreachable and completely redundant.
--

--- Patch [27]: [PATCH v3 27/33] drm/xe: Add IOVA-based xe_res_cursor variant ---
commit 41ab8ab5c8919ddc065db2f6f0c78d6b5b1b51d0
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add IOVA-based xe_res_cursor variant

This patch introduces a variant of xe_res_cursor for walking buffer objects
DMA mapped using the IOVA-based DMA API. It adds an is_iova flag to
distinguish single contiguous IOVA mappings and updates the cursor functions
to handle this contiguous segment appropriately.

> diff --git a/drivers/gpu/drm/xe/xe_res_cursor.h b/drivers/gpu/drm/xe/xe_res_cursor.h
> index 0522caafd89d7..f0f1d699e16e5 100644
> --- a/drivers/gpu/drm/xe/xe_res_cursor.h
> +++ b/drivers/gpu/drm/xe/xe_res_cursor.h
[ ... ]
> @@ -283,6 +327,13 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
>  		return;
>  	}
>  
> +	if (cur->is_iova) {
> +		/* Single contiguous IOVA segment. */
> +		cur->start += size;
> +		cur->size = cur->remaining;
> +		return;
> +	}
> +

[Severity: Low]
Is this block reachable?

In xe_res_first_iova(), cur->size is initialized to the full size, matching
cur->remaining.

When xe_res_next(cur, size) is called, it first decrements cur->remaining:

	cur->remaining -= size;
	if (!cur->remaining)
		return;

If the cursor isn't exhausted, the original cur->remaining must have been
greater than size. Since cur->size equals the original cur->remaining for
single contiguous IOVA mappings, cur->size will also be greater than size. 

This means execution will always enter the generic segment advancement block
immediately preceding this one:

	if (cur->size > size) {
		cur->size -= size;
		cur->start += size;
		return;
	}

Does this make the newly added is_iova block mathematically unreachable dead
code?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=27

Re: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Defragmentation of DMA-alloc or external buffer objects causes out-of-bounds memory corruption.
- [Low] Comment falsely documents non-existent capping and prealloc logic.
--

--- Patch [7]: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves ---
commit 6003b76a02bde0abed96778f39a94035b5bc6acd
Author: Matthew Brost <matthew.brost@intel.com>

drm/ttm: Harvest beneficial-order pages on defragmentation moves
    
A defragmentation move relocates a populated tt into a freshly allocated
one so that sub-beneficial-order chunks can be upgraded to the device's
beneficial order.

> diff --git a/drivers/gpu/drm/ttm/ttm_pool.c b/drivers/gpu/drm/ttm/ttm_pool.c
> index 2598d2f33767e..52f5e64f05ddc 100644
> --- a/drivers/gpu/drm/ttm/ttm_pool.c
> +++ b/drivers/gpu/drm/ttm/ttm_pool.c

[ ... ]

> +static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
> +				      struct ttm_tt *old_tt, pgoff_t off,
> +				      struct ttm_pool_alloc_state *alloc)
> +{
> +	/*
> +	 * @off is always an old-tt chunk head here, never a tail page, so
> +	 * reading the per-chunk order below is safe. This holds because:
> +	 *  - Old chunks are self-aligned: greedy largest-power-of-two fill
> +	 *    places an order-k chunk at a 2^k-aligned offset, so any
> +	 *    beneficial-order-aligned offset is a chunk boundary.
> +	 *  - Fresh defrag allocations are capped at the beneficial order (see
> +	 *    ttm_pool_iter_acquire_page()), so they advance @off in beneficial
> +	 *    steps and never overshoot into the middle of a larger old chunk;
> +	 *    super-beneficial old chunks are always harvested whole by
> +	 *    ttm_pool_iter_reuse_old().
> +	 *  - We are only reached once the defrag budget/prealloc is exhausted,
> +	 *    which in the capped/prealloc config leaves @off beneficial-aligned
> +	 *    (or, in the sub-beneficial tail, already walking old boundaries).
> +	 */

[Severity: Low]
Are the invariants described in this comment block referring to code that is 
missing from this series?

The documentation states that allocations are capped and references 
ttm_pool_iter_acquire_page(), but that function does not appear to contain any
capping logic.

The comment also refers to a defrag budget/prealloc exhaustion, but those
mechanisms do not seem to be present in the code.

> +	while (alloc->remaining_pages) {
> +		struct page *p = old_tt->pages[off];

[ ... ]

> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index b645a18181843..2388ff794a594 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c

[ ... ]

> +void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
> +{
> +	pgoff_t i = 0, run_start = 0;
> +	bool in_run = false;

[ ... ]

> +	while (i < tt->num_pages) {
> +		struct page *sp = src->pages[i];
> +		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
> +		pgoff_t nr = 1UL << order;

[Severity: Critical]
Can this lead to a massive out-of-bounds memset and memcpy when
defragmentation targets a dma-alloc pool or externally allocated buffer object?

If a defragmentation move is initiated in ttm_bo_handle_move_mem() on a 
buffer object using a dma-alloc pool or with TTM_TT_FLAG_EXTERNAL set, page 
private data often contains a kernel pointer or arbitrary foreign data 
instead of an allocation order.

Since ttm_pool_page_order_nodma() treats this data as an integer, the 
resulting order calculation would yield an astronomically large chunk size 
for nr. This size is then used in the memset operation immediately below, 
and in the memcpy operation within ttm_pool_harvest_page():

drivers/gpu/drm/ttm/ttm_pool.c:ttm_pool_harvest_page() {
    ...
    memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
    ...
}

Does ttm_bo_handle_move_mem() need validation to block defragmentation 
on these configurations?

> +
> +		if (sp && tt->pages[i] == sp) {
> +			if (!in_run) {
> +				run_start = i;
> +				in_run = true;
> +			}
> +		} else if (in_run) {
> +			memset(&tt->pages[run_start], 0,
> +			       (i - run_start) * sizeof(*tt->pages));

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=7

Re: [PATCH v3 20/33] drm/xe: Add a page defragmentation worker

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] The defrag worker can be scheduled during probe teardown but is never canceled if probe fails early, leading to a UAF of the `xe_device`.
- [High] The defrag worker continuously polls at a high frequency (40Hz / 25ms) while the device is runtime-suspended or idle, causing significant power regression.
- [High] The defrag worker enters an infinite 40Hz retry loop on the same BO when memory is fragmented, bypassing exponential backoff.
--

--- Patch [20]: [PATCH v3 20/33] drm/xe: Add a page defragmentation worker ---
commit 1f23cf5472cb7a98f4987467477a919155eb0dc0
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Add a page defragmentation worker

This patch adds a delayed worker to the DRM XE driver to periodically
defragment buffer objects whose backing TT pages were allocated below the
optimal order. It bounds the work done per run and applies exponential
backoff on failures.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> +static void xe_bo_defrag_worker(struct work_struct *w)
> +{
[ ... ]
> +	if (!drm_dev_enter(&xe->drm, &idx))
> +		return;
> +
> +	if (!xe_pm_runtime_get_if_in_use(xe)) {
> +		/*
> +		 * The device is (runtime) suspending/suspended. Reschedule so
> +		 * the pass retries once it is active again: the enqueue-time
> +		 * kick in xe_bo_defrag_add() only fires on an empty->non-empty
> +		 * transition, so it will not re-arm the worker while the list
> +		 * is still non-empty. Requeue here to avoid stranding tracked
> +		 * BOs. The get attempt does not resume the device, so this is
> +		 * just a lightweight CPU-side timer while suspended.
> +		 */
> +		if (atomic_read(&xe->mem.defrag.count))
> +			xe_bo_defrag_schedule(xe);
> +		drm_dev_exit(idx);
> +		return;
> +	}

[Severity: High]
Will this cause the worker to endlessly poll at a high frequency while the
device is suspended?

Looking at xe_bo_defrag_worker() in xe_bo.c, when the device is runtime
suspended, xe_pm_runtime_get_if_in_use(xe) returns false. The worker then
hits this early return path and schedules itself via
xe_bo_defrag_schedule(xe) if the defrag count is non-zero.

Since this rescheduling uses the current interval_ms (which is typically
25ms) and applies no backoff, the worker will repeatedly wake up the CPU
every 25ms (40Hz) endlessly while the device is suspended, preventing deep
sleep C-states.

> +
> +	/*
> +	 * Process at most XE_BO_DEFRAG_SIZE_LIMIT bytes of newly (re)allocated
[ ... ]
> +		if (ret || needs_more) {
> +			scoped_guard(spinlock, &xe->mem.defrag.lock) {
> +				if (ret)
> +					/*
> +					 * Abort the pass and retry the whole
> +					 * list later, backing off exponentially
> +					 * on every failure.
> +					 */
> +					xe->mem.defrag.interval_ms =
> +						min(xe->mem.defrag.interval_ms * 2,
> +						    (unsigned int)XE_BO_DEFRAG_INTERVAL_MAX_MS);
> +				else
> +					/* Progress, reset interval */
> +					xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
> +			}
> +
> +			xe_bo_put(bo);
> +			requeue = true;
> +			break;
> +		}

[Severity: High]
Does this create an infinite 40Hz retry loop on the same BO when memory is
fragmented, bypassing the exponential backoff?

When xe_bo_defrag_one() successfully satisfies an allocation using
lower-order pages, ret is 0 but needs_more is true. In this block, because
ret is 0, the else branch resets the interval_ms to 25ms, and the loop
breaks with requeue set to true.

Since the BO is never moved to the back of the list or removed, it remains
at the head. On the next run in 25ms, the worker attempts to defrag the
exact same BO again, likely hitting the exact same fragmentation fallback,
and reschedules at 25ms again.

> +
> +		xe_bo_put(bo);
> +	}
[ ... ]
> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
[ ... ]
> @@ -1069,6 +1069,15 @@ int xe_device_probe(struct xe_device *xe)
>  			return err;
>  	}
>  
> +	/*
> +	 * Register the defrag worker teardown now that the migrate contexts it
> +	 * depends on are initialized, so devm's reverse-order cleanup stops the
> +	 * worker before those contexts are torn down.
> +	 */
> +	err = xe_bo_defrag_init(xe);
> +	if (err)
> +		return err;

[Severity: Critical]
Is there a use-after-free risk if the device probe fails before this point?

The devm action to cleanly cancel the worker (xe_bo_defrag_init) is
registered late in xe_device_probe(). If the probe fails before this
registration (e.g., during xe_display_init_early() or xe_gt_init()), the
devm teardown will unpin BOs.

Unpinning a BO calls xe_bo_unpin() -> xe_bo_defrag_update(), which can
schedule the worker. Because the devm cleanup action was never registered,
the worker won't be cancelled during teardown, leaving it active after the
xe struct is freed.

> +
>  	err = xe_pagefault_init(xe);
>  	if (err)
>  		return err;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=20

Re: [PATCH v3 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [Critical] `xe_device_ccs_bytes()` truncates the calculated CCS size to `u32`, causing massive under-allocation of the `ttm_tt` pages array for buffers >= 1TB.
--

--- Patch [25]: [PATCH v3 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock ---
commit 4ef9a2744ced48ab96c8a05eeb4fe0fa0b3bafac
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Preallocate system BO backing outside the dma-resv lock
    
This commit uses the TTM full-preallocation mechanism to allocate the entire
tt backing up front before entering the validation guard for system-only,
non-deferred user BOs. It sizes the preallocation by querying the CCS-pages
decision directly from the ioctl.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 25e5b72f870c0..a646a72122d40 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -4018,6 +4033,64 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
>  			return -ENOENT;
>  	}
>  
> +	/*
> +	 * For system-only BOs, preallocate the whole backing outside the
> +	 * dma-resv lock so the populate under the validation guard does not
> +	 * stall in reclaim/compaction. The caching mirrors the system-BO path
> +	 * in xe_ttm_tt_create(); the pool falls back to in-line allocation for
> +	 * any shortfall, so a failed fill is harmless.
> +	 */
> +	if ((bo_flags & XE_BO_FLAG_SYSTEM) &&
> +	    !(bo_flags & (XE_BO_FLAG_DEFER_BACKING | XE_BO_FLAG_VRAM_MASK |
> +			  XE_BO_FLAG_STOLEN))) {
> +		struct ttm_resource_manager *tt_man =
> +			ttm_manager_type(&xe->ttm, XE_PL_TT);
> +		enum ttm_caching caching = ttm_cached;
> +		unsigned long num_pages = args->size >> PAGE_SHIFT;
> +
> +		if (!IS_DGFX(xe) &&
> +		    args->cpu_caching == DRM_XE_GEM_CPU_CACHING_WC)
> +			caching = ttm_write_combined;
> +
> +		/*
> +		 * Cover the CCS backup pages appended to the tt (see
> +		 * xe_ttm_tt_create()) so the whole tt is preallocated.
> +		 */
> +		if (__xe_bo_needs_ccs_pages(xe, ttm_bo_type_device, bo_flags,
> +					    args->cpu_caching))
> +			num_pages += DIV_ROUND_UP(xe_device_ccs_bytes(xe,
> +								      args->size),
> +						  PAGE_SIZE);

[Severity: Critical]
This is a pre-existing issue, but does xe_device_ccs_bytes() truncate the
calculated size for very large buffers?

Looking at drivers/gpu/drm/xe/xe_device.c:

u32 xe_device_ccs_bytes(struct xe_device *xe, u64 size)
{
	return xe_device_has_flat_ccs(xe) ?
		DIV_ROUND_UP_ULL(size, NUM_BYTES_PER_CCS_BYTE(xe)) : 0;
}

Because the return type is u32, if a user requests a buffer of 1TB or larger,
the calculated CCS size truncates. For a 1TB buffer with a 256:1 compression
ratio, size / 256 exactly equals 0x100000000, which truncates to 0.

This means num_pages will not include the necessary CCS backup pages (and
similarly extra_pages in xe_ttm_tt_create() will be 0), which can lead to a
massive heap buffer overflow when the driver or GPU writes past the end of
the under-sized ttm_tt pages array.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=25

Re: [PATCH v3 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Critical] Missing decrement of DMA mapped pages counter in `xe_bo_dma_unmap_pinned()` leads to leaked accounting and kernel assertion failure during device teardown.

Pre-existing issues:
- [High] `xe_bo_dev_fini()` is called before `xe->destroy_wq` is flushed, leading to Use-After-Free of the TTM device and false positive assertion failures on device teardown.
--

--- Patch [12]: [PATCH v3 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order ---
commit 670e42692f03a24b49683eec7ff3afb39e28a858
Author: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Subject: drm/xe: Add debugfs stats for DMA-mapped pages per order

Expose per-page-order DMA mapping counts for the system memory that the
xe driver maps for GPU access, split into two categories: TTM buffer
objects, and the GPU SVM / userptr ranges.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 85e6d9a0f575b..60e0e568aa31f 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c
[ ... ]
> @@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
>  	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
>  
>  	if (xe_tt->sg) {
> +		xe_tt_account_dma_pages(xe, xe_tt, -1);
>  		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
>  				  DMA_BIDIRECTIONAL, 0);
>  		sg_free_table(xe_tt->sg);

[Severity: Critical]
Will this accounting leak if the scatter-gather table is unmapped elsewhere?

Looking at xe_bo_dma_unmap_pinned() in drivers/gpu/drm/xe/xe_bo.c, it unmaps
and frees the sg table without updating the accounting:

	} else if (xe_tt->sg) {
		dma_unmap_sgtable(ttm_to_xe_device(ttm_bo->bdev)->drm.dev,
				  xe_tt->sg,
				  DMA_BIDIRECTIONAL, 0);
		sg_free_table(xe_tt->sg);
		xe_tt->sg = NULL;
	}

Because xe_bo_dma_unmap_pinned() clears xe_tt->sg, a later call to
xe_tt_unmap_sg() will see xe_tt->sg as NULL and skip the decrement, leading to
a permanent accounting leak and a guaranteed assertion failure on device
teardown.

> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> index ad7f3e61d4579..97cc2d185e45d 100644
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
[ ... ]
>  static void xe_device_destroy(struct drm_device *dev, void *dummy)
>  {
>  	struct xe_device *xe = to_xe_device(dev);
>  
>  	xe_bo_dev_fini(&xe->bo_device);
>  
> +	/*
> +	 * Assert the DMA-mapped-pages accounting only after xe_bo_dev_fini()
> +	 * has flushed the async BO-free worker: BOs pending async free still
> +	 * hold their DMA mappings (and counts) until the worker runs, so
> +	 * asserting before the flush could trip a false positive.
> +	 */
> +	xe_device_assert_dma_pages_zero(xe);
> +
>  	if (xe->preempt_fence_wq)
>  		destroy_workqueue(xe->preempt_fence_wq);

[Severity: High]
This is a pre-existing issue, but will this new assertion falsely trip due
to the teardown order?

Because xe->destroy_wq is destroyed after this assertion, and work items in
xe->destroy_wq might still hold final references to BOs, those BOs won't be
unmapped until the workqueue is drained.

Since the assertion now runs before the workqueues drop their BO references,
it will incorrectly trigger because those BOs still have active DMA mappings.

Additionally, since xe_bo_dev_fini() flushes the async BO-free worker before
the workqueues drop BOs, the worker can run concurrently with or after
ttm_device_fini(), which is a pre-existing use-after-free risk exposed by
this change.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=12

Re: [PATCH v3 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Early return on xe_dep_scheduler_create() failure skips registering the devm teardown action for the defrag worker, leading to a use-after-free.
--

--- Patch [29]: [PATCH v3 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize ---
commit 4091ca99808dbd61930846733ec6c77fcca46fe9
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

This commit adds a per-device dependency scheduler for IOVA defrag finalize
jobs, which tears down temporary copy mappings and links new backing memory.
The scheduler is created in xe_bo_defrag_init() and destroyed in
xe_bo_defrag_fini().

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index d54e69aefeba5..c5022c16411f6 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -1342,6 +1347,16 @@ void xe_bo_defrag_init_early(struct xe_device *xe)
>   */
>  int xe_bo_defrag_init(struct xe_device *xe)
>  {
> +#define XE_BO_MAX_IOVA_DEFRAG_JOBS	16	/* Picking a reasonable value */
> +	struct xe_dep_scheduler *iova_sched;
> +
> +	iova_sched = xe_dep_scheduler_create(xe, NULL, "xe_iova_defrag",
> +					     XE_BO_MAX_IOVA_DEFRAG_JOBS);
> +	if (IS_ERR(iova_sched))
> +		return PTR_ERR(iova_sched);

[Severity: High]
If xe_dep_scheduler_create() fails and we return early, does this skip
registering the teardown action xe_bo_defrag_fini?

Since xe_bo_defrag_init_early() initializes the defrag delayed worker
earlier in device creation, could the worker already be armed via
schedule_delayed_work() if BOs hit sub-optimal page allocations during
other initializations?

If probe fails and devres unwinds, the xe_device struct is freed. Since
the early return bypasses devm_add_action_or_reset(), the worker is never
canceled. Could this lead to a use-after-free when the armed worker
eventually fires and dereferences the freed xe_device?

> +	xe->mem.defrag.iova_sched = iova_sched;
> +#undef XE_BO_MAX_IOVA_DEFRAG_JOBS
> +
>  	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=29

Re: [PATCH v3 18/33] drm/xe: Handle defrag moves in xe_bo_move()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Pipelined defrag move detaches VF CCS metadata before the GPU copy completes, risking CCS metadata corruption on VF migration.
- [Medium] Using `ctx->defrag` to identify a defrag move incorrectly applies defrag-specific VF CCS attach/detach logic to unrelated victim BOs evicted during a defrag operation.

Pre-existing issues:
- [Critical] Returning an error from `xe_bo_move()` after successfully committing the move causes TTM to free the new resource, leaving `bo->resource` as a dangling pointer.
--

--- Patch [18]: [PATCH v3 18/33] drm/xe: Handle defrag moves in xe_bo_move() ---
commit 22e925edd18b0faf0ff581a79813091d539553b8
Author: Matthew Brost <matthew.brost@intel.com>
Subject: drm/xe: Handle defrag moves in xe_bo_move()

This commit modifies xe_bo_move() to handle defragmentation moves, allowing
TTM to reallocate a BO's backing at an optimal order while copying existing
contents. It introduces a fast path for defrag moves via GPU relocation,
pipelines the teardown asynchronously, and extends VF CCS attach/detach
logic to cover defrag transitions.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index 02395968d613b..bdcc9bed5d36b 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -1315,13 +1331,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
>  	 * BBs from BO as it is no longer needed.
>  	 */
>  	if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT &&
> -	    new_mem->mem_type == XE_PL_SYSTEM)
> +	    (new_mem->mem_type == XE_PL_SYSTEM || ctx->defrag))
>  		xe_sriov_vf_ccs_detach_bo(bo);

[Severity: High]
Does this code synchronously detach VF CCS metadata before the GPU copy
completes?

Since a defrag move is pipelined asynchronously (evict is false, so
ttm_bo_move_accel_cleanup() delegates to a ghost object without waiting for
the fence), xe_bo_move() immediately calls xe_sriov_vf_ccs_detach_bo(bo).

This synchronously clears the GuC CCS save/restore batch buffers for the
old TT. If a VF migration triggers while the async GPU copy job is still in
flight, the GuC won't save the old TT's CCS state because its batch buffers
were detached.

When the copy job resumes, can it copy uninitialized or garbage data from
the old hardware CCS region into the new TT, leading to CCS metadata
corruption?

>  	if (IS_VF_CCS_READY(xe) &&
>  	    ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
> -	     (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
> -	    handle_system_ccs)
> +	     ((old_mem_type == XE_PL_SYSTEM || ctx->defrag) &&
> +	      new_mem->mem_type == XE_PL_TT)) && handle_system_ccs)
>  		ret = xe_sriov_vf_ccs_attach_bo(bo);

[Severity: Medium]
Does checking ctx->defrag instead of ttm_bo->defrag_old_tt misclassify
victim BOs?

TTM passes the original ttm_operation_ctx down to all evictions triggered
by an allocation, which means ctx->defrag is true for any victim BO evicted
during a defrag pass.

If unrelated BOs (such as those evicted from STOLEN to TT) have VF CCS
logic applied to them because ctx->defrag is true, will this modify their
hardware CCS tracking unexpectedly and break standard eviction flows?

[Severity: Critical]
This is a pre-existing issue, but returning an error from xe_bo_move()
after successfully committing the move causes TTM to free the new resource,
leaving bo->resource as a dangling pointer.

In xe_bo_move(), the move is successfully committed via
ttm_bo_move_accel_cleanup() which assigns the new resource to bo->resource.
If a subsequent operation like xe_sriov_vf_ccs_attach_bo(bo) fails (e.g.
returns -ENOMEM), xe_bo_move() propagates this error back to TTM.

TTM's error recovery in ttm_bo_validate() then unconditionally calls
ttm_resource_free(bo, &res) for the new allocation. Since bo->resource
already points to this new allocation, does this free the memory while
leaving bo->resource as a dangling pointer, creating a use-after-free risk?

>  
>  out:

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=18

Re: [PATCH v3 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Tracepoint metric lock_ns incorrectly includes lock-dropped backoff sleep time due to WW mutex contention retries.
--

--- Patch [26]: [PATCH v3 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl ---
commit 08fb65ed7bb6e79b15efc1ef9667995aaebd39fe
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Add tracepoint for xe_gem_create_ioctl

This commit adds a tracepoint to profile GEM creation in the xe driver.
It records the BO size, placement, CPU caching, ASID, total IOCTL
duration, and the time spent under the dma-resv lock to help evaluate
out-of-lock preallocation performance.

> diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
> index a646a72122d40..caf5aa0bab533 100644
> --- a/drivers/gpu/drm/xe/xe_bo.c
> +++ b/drivers/gpu/drm/xe/xe_bo.c

[ ... ]

> @@ -4100,6 +4105,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
>  			if (err)
>  				break;
>  		}
> +
> +		if (lock_start == ktime_set(0, 0))
> +			lock_start = ktime_get();
> +

[Severity: Medium]
Will this lock_start timestamp incorrectly include lock-dropped backoff
sleep time when a WW mutex contention retry occurs?

When contention happens, drm_exec_cleanup() drops all locks and the next
iteration might sleep in dma_resv_lock_slow_interruptible() without holding
any dma-resv locks. Because of the zero-check guard here, lock_start is
retained from the first iteration.

Could this cause the lock_ns metric to encapsulate the unbounded lock-dropped
sleep time and corrupt the intended profiling metric? If vm != NULL, removing
this zero-check would exclude the sleep (which happens early in
xe_vm_drm_exec_lock). However, if vm == NULL, the sleep occurs inside
xe_bo_create_user(), so does this require capturing the start time inside
the locking functions or accumulating the actual duration per iteration?

>  		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
>  				       bo_flags, have_prealloc ? &prealloc : NULL,
>  				       &exec);

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=26

Re: [PATCH v3 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Critical] Overloading `dst_packed_pages == 0` to mean 'not a packed destination' causes out-of-bounds cursor access and memory corruption when a packed defrag move has exactly zero changed pages.

Pre-existing issues:
- [Critical] During a TT-to-TT CCS defrag pass on DGFX systems, an uninitialized stack variable (`ccs_ofs`) is passed to the GPU blitter, causing memory corruption.
--

--- Patch [31]: [PATCH v3 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies ---
commit f45aa97c2f11618672e201ef92ad9ef7898d5e9c
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

This commit introduces a packed-destination mode in __xe_migrate_copy() for
defrag moves with IOVA enabled. It adds a dst_packed_pages argument to handle
asymmetric geometries where the destination only exposes changed pages. The
destination PTE update size is calculated based on the changed-page count.

> diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
> index e9f050eab8871..8eb5e331d5156 100644
> --- a/drivers/gpu/drm/xe/xe_migrate.c
> +++ b/drivers/gpu/drm/xe/xe_migrate.c

[ ... ]

> @@ -1033,7 +1179,9 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
>  	else
>  		xe_res_first(src, 0, size, &src_it);
>  	if (!dst_is_vram)
> -		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
> +		xe_res_first_tt(dst_bo->ttm.ttm, 0,
> +				pack_dst ? (u64)dst_packed_pages << PAGE_SHIFT :
> +				size, &dst_it);
>  	else
>  		xe_res_first(dst, 0, size, &dst_it);

[Severity: Critical]
Does overloading dst_packed_pages == 0 to mean 'not a packed destination'
cause issues when a packed defrag move has exactly zero changed pages?

If a defragmentation move executes under memory pressure and fails to allocate
any new pages, it results in 100% borrowed pages. The IOVA mapping function
legitimately maps 0 pages, so xe_migrate_copy_defrag_iova() is called with
dst_packed_pages = 0.

Because dst_packed_pages == 0, pack_dst becomes false (set earlier in
__xe_migrate_copy). This forces the fallback to a full-size non-packed copy,
and initializes dst_it requesting the full BO size against a destination IOVA
that was packed to size 0.

Can this walk the broken cursor and fetch garbage DMA addresses for the GPU
blitter, corrupting memory?

[ ... ]

> @@ -1145,16 +1355,22 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
>  
>  		if (skip_borrowed)
>  			emit_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0,
> -					 src_tt, dst_tt, start_page);
> +					 src_tt, dst_tt, start_page, pack_dst);
>  		else if (!flags.copy_only_ccs && !flags.defrag_ccs)
>  			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
>  
> -		if (needs_ccs_emit)
> +		if (ccs_pack) {
> +			if (emit_ccs_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs,
> +						 src_L0, src_tt, dst_tt,
> +						 start_page))
> +				flush_flags = MI_FLUSH_DW_CCS;
> +		} else if (needs_ccs_emit) {
>  			flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
>  							  IS_DGFX(xe) ? src_is_vram : src_is_pltt,
>  							  dst_L0_ofs,
>  							  IS_DGFX(xe) ? dst_is_vram : dst_is_pltt,
>  							  src_L0, ccs_ofs, copy_ccs);
> +		}

[Severity: Critical]
This is a pre-existing issue, but can this pass an uninitialized ccs_ofs stack
variable to the GPU blitter during a TT-to-TT CCS defrag pass on DGFX systems?

When __xe_migrate_copy() is called with flags.defrag_ccs = true,
copy_system_ccs evaluates to false, leaving ccs_ofs uninitialized.

Because !flags.defrag_copy is true, needs_ccs_emit evaluates to true and calls
xe_migrate_ccs_copy(). For a TT source on DGFX, IS_DGFX(xe) ? src_is_vram :
src_is_pltt evaluates to false (meaning src_is_indirect is false).

xe_migrate_ccs_copy() then consumes the uninitialized ccs_ofs:

drivers/gpu/drm/xe/xe_migrate.c:xe_migrate_ccs_copy() {
    ...
    } else if (copy_ccs) {
        if (!src_is_indirect)
            src_ofs = ccs_ofs;
    ...
}

Does this cause the uninitialized src_ofs to be emitted into the GPU batch
buffer, resulting in memory corruption?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=31

Re: [PATCH v3 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] TOCTOU race on `fpriv->xcp_id` causes pool allocation mismatches and potential NUMA pool pollution.
--

--- Patch [33]: [PATCH v3 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock ---
commit 4e5e075ea8b2b453d727a0f32343eda951ff2819
Author: Matthew Brost <matthew.brost@intel.com>
drm/amdgpu: Preallocate system BO pages outside the reservation lock

This commit introduces out-of-lock preallocation for pure system buffer
objects to avoid stalling under the reservation lock. By mirroring the Xe
driver's approach, it fills the full page backing up front before taking the
lock.

> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
[ ... ]
> @@ -443,10 +448,57 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
>  		flags |= AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
>  	}
>  
> +	/*
> +	 * For system-only (pure GTT) BOs, preallocate the whole page backing
> +	 * up front, outside the reservation lock. Populate under the lock then
> +	 * just installs these pages instead of reclaiming/compacting in the
> +	 * critical section. Best-effort: a short fill falls back to the normal
> +	 * in-lock allocation for the missing pages.
> +	 */
> +	if (args->in.domains == AMDGPU_GEM_DOMAIN_GTT) {
> +		struct ttm_resource_manager *gtt_man =
> +			ttm_manager_type(&adev->mman.bdev, TTM_PL_TT);
> +		int32_t xcp_id = adev->gmc.mem_partitions ? fpriv->xcp_id : 0;
> +		int32_t pool_id = amdgpu_ttm_tt_pool_id(adev, xcp_id);

[ ... ]

> @@ -455,7 +507,8 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
>  retry:
>  	r = amdgpu_gem_object_create(adev, size, args->in.alignment,
>  				     initial_domain,
> -				     flags, ttm_bo_type_device, resv, &gobj, fpriv->xcp_id + 1);
> +				     flags, ttm_bo_type_device, resv, &gobj,
> +				     fpriv->xcp_id + 1, prealloc_pool ? &prealloc : NULL);

[Severity: Medium]
Is it possible for fpriv->xcp_id to change between these two reads?

If a concurrent thread updates fpriv->xcp_id (for example, via context
creation in amdgpu_xcp_select_scheds changing it from AMDGPU_XCP_NO_PARTITION
to a specific partition), the first read could select the generic TTM pool
for preallocation.

The second read could then evaluate to a specific partition, initializing
the BO with the partition-specific TTM pool. Could this mismatch result
in pages allocated from the generic pool being consumed by the partition
pool, potentially degrading NUMA locality or causing accounting mismatches?

Would it be safer to read fpriv->xcp_id once into a local variable and pass
that same value to both amdgpu_ttm_tt_pool_id() and
amdgpu_gem_object_create()?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260711025619.2540575-1-matthew.brost@intel.com?part=33

Re: [PATCH v3 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock

[Christian König]

On 7/11/26 04:56, Matthew Brost wrote:
> Populating a GTT (system) buffer object under the reservation lock can
> stall in reclaim and compaction while trying to satisfy beneficial-order
> allocations, holding the lock for the duration.

Yeah and that is perfectly intentional behavior.

I really don't see any reason for any driver to change that.

The purpose of the dma_resv lock is to stall other allocation for the same object it protects, making sure that we don't have multiple threads allocating memory for the same buffer object.

Why in the world should that be a problem?

Regards,
Christian.

> 
> Mirror the Xe out-of-lock preallocation for pure system BOs: when a
> gem_create request targets AMDGPU_GEM_DOMAIN_GTT only, fill the full page
> backing up front via ttm_pool_prealloc_fill_full() before taking the
> reservation lock. The populate under the lock then simply installs these
> pages instead of reclaiming and compacting in the critical section. The
> fill is best-effort - a short fill falls back to the normal in-lock
> allocation for the remaining pages, and any leftover pages are released
> by ttm_pool_prealloc_fini() on all exit paths.
> 
> Unlike Xe, amdgpu has no background defragmenter, so the higher-order
> reclaim backoff is left disabled (normal reclaim). If the pool uses
> dma-alloc (swiotlb), the fill bails and the feature is a silent no-op.
> 
> Factor the tt pool selection into amdgpu_ttm_tt_pool_id() and
> amdgpu_ttm_pool() so the preallocation targets the exact pool the
> populate will consume, and thread an optional prealloc bag through
> amdgpu_gem_object_create() and amdgpu_bo_param into the populate ctx.
> 
> Cc: amd-gfx@lists.freedesktop.org
> Cc: Alex Deucher <alexander.deucher@amd.com>
> Cc: Carlos Santa <carlos.santa@intel.com>
> Cc: Ryan Neph <ryanneph@google.com>
> Cc: Christian Koenig <christian.koenig@amd.com>
> Cc: Huang Rui <ray.huang@amd.com>
> Cc: Matthew Auld <matthew.auld@intel.com>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Maxime Ripard <mripard@kernel.org>
> Cc: Thomas Zimmermann <tzimmermann@suse.de>
> Cc: David Airlie <airlied@gmail.com>
> Cc: Simona Vetter <simona@ffwll.ch>
> Cc: dri-devel@lists.freedesktop.org
> Cc: linux-kernel@vger.kernel.org
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Assisted-by: GitHub_Copilot:claude-opus-4.8
> Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> 
> ---
> 
> v3:
>  - Keep WC caching only when USWC is supported and bound the
>    preallocation to available GTT space (Sashiko)
> ---
>  .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  4 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |  2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 73 +++++++++++++++++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |  5 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |  3 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  4 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 39 +++++++---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |  3 +
>  8 files changed, 108 insertions(+), 25 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index 20831dbebc31..940f58848a97 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -345,7 +345,7 @@ create_dmamap_sg_bo(struct amdgpu_device *adev,
>  
>  	ret = amdgpu_gem_object_create(adev, mem->bo->tbo.base.size, 1,
>  			AMDGPU_GEM_DOMAIN_CPU, AMDGPU_GEM_CREATE_PREEMPTIBLE | flags,
> -			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0);
> +			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0, NULL);
>  
>  	amdgpu_bo_unreserve(mem->bo);
>  
> @@ -1811,7 +1811,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
>  		 domain_string(alloc_domain), xcp_id);
>  
>  	ret = amdgpu_gem_object_create(adev, aligned_size, 1, alloc_domain, alloc_flags,
> -				       bo_type, NULL, &gobj, xcp_id + 1);
> +				       bo_type, NULL, &gobj, xcp_id + 1, NULL);
>  	if (ret) {
>  		pr_debug("Failed to create BO on domain %s. ret %d\n",
>  			 domain_string(alloc_domain), ret);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> index b33c300e26e2..51510e831129 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> @@ -435,7 +435,7 @@ amdgpu_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
>  
>  	ret = amdgpu_gem_object_create(adev, dma_buf->size, PAGE_SIZE,
>  				       AMDGPU_GEM_DOMAIN_CPU, flags,
> -				       ttm_bo_type_sg, resv, &gobj, 0);
> +				       ttm_bo_type_sg, resv, &gobj, 0, NULL);
>  	if (ret)
>  		goto error;
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> index 6a0699746fbc..e8b732218e7d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> @@ -39,6 +39,7 @@
>  #include <drm/drm_exec.h>
>  #include <drm/drm_gem_ttm_helper.h>
>  #include <drm/ttm/ttm_tt.h>
> +#include <drm/ttm/ttm_pool.h>
>  #include <drm/drm_syncobj.h>
>  
>  #include "amdgpu.h"
> @@ -168,7 +169,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
>  			     int alignment, u32 initial_domain,
>  			     u64 flags, enum ttm_bo_type type,
>  			     struct dma_resv *resv,
> -			     struct drm_gem_object **obj, int8_t xcp_id_plus1)
> +			     struct drm_gem_object **obj, int8_t xcp_id_plus1,
> +			     struct ttm_pool_prealloc *prealloc)
>  {
>  	struct amdgpu_bo *bo;
>  	struct amdgpu_bo_user *ubo;
> @@ -188,6 +190,7 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
>  	bp.domain = initial_domain;
>  	bp.bo_ptr_size = sizeof(struct amdgpu_bo);
>  	bp.xcp_id_plus1 = xcp_id_plus1;
> +	bp.prealloc = prealloc;
>  
>  	r = amdgpu_bo_create_user(adev, &bp, &ubo);
>  	if (r)
> @@ -412,6 +415,8 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
>  	struct dma_resv *resv = NULL;
>  	struct drm_gem_object *gobj;
>  	uint32_t handle, initial_domain;
> +	struct ttm_pool_prealloc prealloc = {};
> +	struct ttm_pool *prealloc_pool = NULL;
>  	int r;
>  
>  	/* reject invalid gem flags */
> @@ -443,10 +448,57 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
>  		flags |= AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
>  	}
>  
> +	/*
> +	 * For system-only (pure GTT) BOs, preallocate the whole page backing
> +	 * up front, outside the reservation lock. Populate under the lock then
> +	 * just installs these pages instead of reclaiming/compacting in the
> +	 * critical section. Best-effort: a short fill falls back to the normal
> +	 * in-lock allocation for the missing pages.
> +	 */
> +	if (args->in.domains == AMDGPU_GEM_DOMAIN_GTT) {
> +		struct ttm_resource_manager *gtt_man =
> +			ttm_manager_type(&adev->mman.bdev, TTM_PL_TT);
> +		int32_t xcp_id = adev->gmc.mem_partitions ? fpriv->xcp_id : 0;
> +		int32_t pool_id = amdgpu_ttm_tt_pool_id(adev, xcp_id);
> +		/*
> +		 * Mirror the USWC handling in amdgpu_bo_create(): the flag is
> +		 * stripped when the platform can't do write-combining, in which
> +		 * case the tt is created ttm_cached. The prealloc caching must
> +		 * match the final tt caching exactly, otherwise ttm_pool_free()
> +		 * would restore the wrong PAT state on these pages (PAT aliasing
> +		 * / leak of WC pages into the cached allocator).
> +		 */
> +		bool uswc = (flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC) &&
> +			amdgpu_bo_support_uswc(flags);
> +		enum ttm_caching caching =
> +			uswc ? ttm_write_combined : ttm_cached;
> +
> +		/*
> +		 * Only prealloc when the request fits the currently-available
> +		 * GTT (total manager size minus current usage). This mirrors the
> +		 * amdgpu_bo_validate_size() bound applied later in
> +		 * amdgpu_bo_create() while also skipping the up-front reclaim /
> +		 * compaction when GTT is already near full or the user-controlled
> +		 * size is over-large (the creation path handles those anyway).
> +		 */
> +		if (gtt_man) {
> +			u64 used = ttm_resource_manager_usage(gtt_man);
> +			u64 avail = gtt_man->size > used ?
> +				gtt_man->size - used : 0;
> +
> +			if (size <= avail) {
> +				prealloc_pool = amdgpu_ttm_pool(adev, pool_id);
> +				ttm_pool_prealloc_fill_full(prealloc_pool,
> +							    caching, &prealloc,
> +							    PFN_UP(size), false);
> +			}
> +		}
> +	}
> +
>  	if (flags & AMDGPU_GEM_CREATE_VM_ALWAYS_VALID) {
>  		r = amdgpu_bo_reserve(vm->root.bo, false);
>  		if (r)
> -			return r;
> +			goto out_prealloc;
>  
>  		resv = vm->root.bo->tbo.base.resv;
>  	}
> @@ -455,7 +507,8 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
>  retry:
>  	r = amdgpu_gem_object_create(adev, size, args->in.alignment,
>  				     initial_domain,
> -				     flags, ttm_bo_type_device, resv, &gobj, fpriv->xcp_id + 1);
> +				     flags, ttm_bo_type_device, resv, &gobj,
> +				     fpriv->xcp_id + 1, prealloc_pool ? &prealloc : NULL);
>  	if (r && r != -ERESTARTSYS) {
>  		if (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) {
>  			flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
> @@ -479,17 +532,21 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
>  		amdgpu_bo_unreserve(vm->root.bo);
>  	}
>  	if (r)
> -		return r;
> +		goto out_prealloc;
>  
>  	r = drm_gem_handle_create(filp, gobj, &handle);
>  	/* drop reference from allocate - handle holds it now */
>  	drm_gem_object_put(gobj);
>  	if (r)
> -		return r;
> +		goto out_prealloc;
>  
>  	memset(args, 0, sizeof(*args));
>  	args->out.handle = handle;
> -	return 0;
> +
> +out_prealloc:
> +	if (prealloc_pool)
> +		ttm_pool_prealloc_fini(prealloc_pool, &prealloc);
> +	return r;
>  }
>  
>  int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,
> @@ -528,7 +585,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data,
>  
>  	/* create a gem object to contain this object in */
>  	r = amdgpu_gem_object_create(adev, args->size, 0, AMDGPU_GEM_DOMAIN_CPU,
> -				     0, ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1);
> +				     0, ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1, NULL);
>  	if (r)
>  		return r;
>  
> @@ -1298,7 +1355,7 @@ int amdgpu_mode_dumb_create(struct drm_file *file_priv,
>  	domain = amdgpu_bo_get_preferred_domain(adev,
>  				amdgpu_display_supported_domains(adev, flags));
>  	r = amdgpu_gem_object_create(adev, args->size, 0, domain, flags,
> -				     ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1);
> +				     ttm_bo_type_device, NULL, &gobj, fpriv->xcp_id + 1, NULL);
>  	if (r)
>  		return -ENOMEM;
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
> index b558336bc4c6..706aeca011f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h
> @@ -35,6 +35,8 @@
>  
>  extern const struct drm_gem_object_funcs amdgpu_gem_object_funcs;
>  
> +struct ttm_pool_prealloc;
> +
>  unsigned long amdgpu_gem_timeout(uint64_t timeout_ns);
>  
>  /*
> @@ -45,7 +47,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
>  			     int alignment, u32 initial_domain,
>  			     u64 flags, enum ttm_bo_type type,
>  			     struct dma_resv *resv,
> -			     struct drm_gem_object **obj, int8_t xcp_id_plus1);
> +			     struct drm_gem_object **obj, int8_t xcp_id_plus1,
> +			     struct ttm_pool_prealloc *prealloc);
>  int amdgpu_mode_dumb_create(struct drm_file *file_priv,
>  			    struct drm_device *dev,
>  			    struct drm_mode_create_dumb *args);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> index f98bfba59a2c..18c4cf3f35a5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> @@ -632,7 +632,8 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
>  		/* We opt to avoid OOM on system pages allocations */
>  		.gfp_retry_mayfail = true,
>  		.allow_res_evict = bp->type != ttm_bo_type_kernel,
> -		.resv = bp->resv
> +		.resv = bp->resv,
> +		.prealloc = bp->prealloc,
>  	};
>  	struct amdgpu_bo *bo;
>  	unsigned long page_align, size = bp->size;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> index ff11a0903499..11f1d403f152 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
> @@ -45,6 +45,8 @@
>  #define to_amdgpu_bo_user(abo) container_of((abo), struct amdgpu_bo_user, bo)
>  #define to_amdgpu_bo_vm(abo) container_of((abo), struct amdgpu_bo_vm, bo)
>  
> +struct ttm_pool_prealloc;
> +
>  struct amdgpu_bo_param {
>  	unsigned long			size;
>  	int				byte_align;
> @@ -58,6 +60,8 @@ struct amdgpu_bo_param {
>  	void				(*destroy)(struct ttm_buffer_object *bo);
>  	/* xcp partition number plus 1, 0 means any partition */
>  	int8_t				xcp_id_plus1;
> +	/* optional out-of-lock preallocated backing (system/GTT only) */
> +	struct ttm_pool_prealloc	*prealloc;
>  };
>  
>  /* bo virtual addresses in a vm */
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> index b10b0878df37..c9cd4714f515 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> @@ -1173,6 +1173,30 @@ void amdgpu_ttm_mmio_remap_free_sgt(struct device *dev,
>  	kfree(sgt);
>  }
>  
> +/*
> + * amdgpu_ttm_tt_pool_id - compute the ttm pool id backing a given xcp
> + *
> + * Mirrors the mapping used when creating the gtt ttm_tt, so callers that only
> + * have an xcp id (e.g. an out-of-lock preallocation before the bo exists) pick
> + * the same pool the populate will use.
> + */
> +int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id)
> +{
> +	if (adev->gmc.mem_partitions && xcp_id >= 0)
> +		return KFD_XCP_MEM_ID(adev, xcp_id);
> +
> +	return xcp_id;
> +}
> +
> +/* amdgpu_ttm_pool - select the ttm pool for a given pool id */
> +struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id)
> +{
> +	if (adev->mman.ttm_pools && pool_id >= 0)
> +		return &adev->mman.ttm_pools[pool_id];
> +
> +	return &adev->mman.bdev.pool;
> +}
> +
>  /**
>   * amdgpu_ttm_tt_create - Create a ttm_tt object for a given BO
>   *
> @@ -1194,10 +1218,7 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo,
>  		return NULL;
>  
>  	gtt->gobj = &bo->base;
> -	if (adev->gmc.mem_partitions && abo->xcp_id >= 0)
> -		gtt->pool_id = KFD_XCP_MEM_ID(adev, abo->xcp_id);
> -	else
> -		gtt->pool_id = abo->xcp_id;
> +	gtt->pool_id = amdgpu_ttm_tt_pool_id(adev, abo->xcp_id);
>  
>  	if (abo->flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC)
>  		caching = ttm_write_combined;
> @@ -1239,10 +1260,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev,
>  	if (ttm->page_flags & TTM_TT_FLAG_EXTERNAL)
>  		return 0;
>  
> -	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
> -		pool = &adev->mman.ttm_pools[gtt->pool_id];
> -	else
> -		pool = &adev->mman.bdev.pool;
> +	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
>  	ret = ttm_pool_alloc(pool, ttm, ctx);
>  	if (ret)
>  		return ret;
> @@ -1284,10 +1302,7 @@ static void amdgpu_ttm_tt_unpopulate(struct ttm_device *bdev,
>  
>  	adev = amdgpu_ttm_adev(bdev);
>  
> -	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
> -		pool = &adev->mman.ttm_pools[gtt->pool_id];
> -	else
> -		pool = &adev->mman.bdev.pool;
> +	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
>  
>  	return ttm_pool_free(pool, ttm);
>  }
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> index ff9e2e346609..f90844937b73 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> @@ -207,6 +207,9 @@ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo);
>  void amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo);
>  uint64_t amdgpu_ttm_domain_start(struct amdgpu_device *adev, uint32_t type);
>  
> +int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id);
> +struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id);
> +
>  #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
>  int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo,
>  				 struct amdgpu_hmm_range *range);