[PATCH V16 00/12] Add memory page offlining support

[PATCH V16 00/12] Add memory page offlining support

[Tejas Upadhyay]

This functionality represents a significant step in making
the xe driver gracefully handle hardware memory degradation.
By integrating with the DRM Buddy allocator, the driver
can permanently "carve out" faulty memory so it isn't reused
by subsequent allocations.

IGT tests for testing this via injecting simple single address and
duplicate address fault to unit test functionality:
https://patchwork.freedesktop.org/patch/740601/

v16:
- Correct sysfs patches with moving code with rcu lock
- In case purge fail let next alloc decide final failure
- Remove addr_to_block API, its being pulled from drm-tip
- Remove some unused code and replace where existing API can be used
v15:
- Split few big patches into small
- Avoid vram_mgr lock in sysfs
- fix missing queue_pages counter increment
v14:
- Solve sashiko reviews
- Remove SOFT->HARD offline patch, decision is taken based on -EEXIST
- Dump gpu buddy allocated patch dropped
v13:
- Add fault inject and remove standlone debugfs
v12:
- Fix Sashiko review comments
v11:
- Add BAN reason for UMD to know about offlining
- Add support for soft offline mode
- Rebase and remove dummy lockdep annotation patch, as it merged from upstream
v10:
- Remove RFC
v7:
- Improve debugfs warning messages
- Use scope_guard for locking(MattB)
- Adapt addition of queue member of LRC BO(MattB)
- Extend and use xe_ttm_bo_purge API for vram pages(MattB)
- Handle dma_buf_map requests for native and remote(MattB)
- Address if in never initialized block, set block to NULL
- Add lockdep in gpu buddy (MattB)
- Correct allocated_addr_to_block logic (MattA)
V6:
- Add more specific tests to noncritical bo sections
- Handle smooth exit of user created exec queues
- Break code and make purge specific static API
V5:
- Sysfs "max_pages" addition
- Reset block->private NULL post purge
- Remove wedge, return -EIO to system controller will initiate reset
- Add debugfs tests to trigger different test scenarios manually and via igt
- Rename addr_to_tbo to addr_to_block and move under gpu/buddy.c
V4: API reworks, add configfs for policy reservation and apply config everywhere
V3: use res_to_mem_region to avoid use of block->private (MattA)
V2:
- some fixes and clean up on errors
- Added xe_vram_addr_to_region helper to avoid other use of block->private(MattB)

Tejas Upadhyay (12):
  drm/xe: Link VRAM object with gpu buddy
  drm/xe: Link LRC BO and its execution Queue
  drm/xe: Extend BO purge to handle vram pages as well
  drm/xe/bo: Make xe_bo_is_user() public
  drm/xe: Guard teardown paths against purged BOs
  drm/xe/vram: Extract buddy alloc and free helpers
  drm/xe/vram: Add page offline data structures and lifecycle
  drm/xe/vram: Add VRAM page offline fault handler
  drm/xe/configfs: Add vram bad page reservation policy
  drm/xe: Add sysfs interface for bad gpu vram pages
  drm/xe/uapi: Expose ban reason in EXEC_QUEUE_GET_PROPERTY_BAN
  drm/xe: Add fault-inject based VRAM page offline injection

 drivers/gpu/drm/xe/xe_bo.c                 |  13 +-
 drivers/gpu/drm/xe/xe_bo.h                 |   5 +-
 drivers/gpu/drm/xe/xe_bo_types.h           |   3 +
 drivers/gpu/drm/xe/xe_configfs.c           |  67 ++-
 drivers/gpu/drm/xe/xe_configfs.h           |   2 +
 drivers/gpu/drm/xe/xe_debugfs.c            |  50 ++
 drivers/gpu/drm/xe/xe_debugfs.h            |   2 +
 drivers/gpu/drm/xe/xe_device_sysfs.c       |   7 +
 drivers/gpu/drm/xe/xe_dma_buf.c            |   3 +
 drivers/gpu/drm/xe/xe_exec_queue.c         |  14 +-
 drivers/gpu/drm/xe/xe_exec_queue_types.h   |   7 +-
 drivers/gpu/drm/xe/xe_execlist.c           |   4 +-
 drivers/gpu/drm/xe/xe_guc_submit.c         |  32 +-
 drivers/gpu/drm/xe/xe_lrc.c                |   1 +
 drivers/gpu/drm/xe/xe_pt.c                 |   5 +-
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c       | 603 ++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.h       |   3 +
 drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h |  40 ++
 include/uapi/drm/xe_drm.h                  |  18 +-
 19 files changed, 834 insertions(+), 45 deletions(-)

-- 
2.52.0

[PATCH V16 01/12] drm/xe: Link VRAM object with gpu buddy

[Tejas Upadhyay]

Setup to link TTM buffer object inside gpu buddy. This functionality
is critical for supporting the memory page offline feature on CRI,
where identified faulty pages must be traced back to their
originating buffer for safe removal.

V2(MattB): Clear block->private in xe_ttm_vram_mgr_del as well

Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Reviewed-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index b518f7dec680..5ab5dfdb183c 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -54,6 +54,7 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
 	struct xe_ttm_vram_mgr *mgr = to_xe_ttm_vram_mgr(man);
 	struct xe_ttm_vram_mgr_resource *vres;
 	struct gpu_buddy *mm = &mgr->mm;
+	struct gpu_buddy_block *block;
 	u64 size, min_page_size;
 	unsigned long lpfn;
 	int err;
@@ -138,6 +139,8 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
 	}
 
 	mgr->visible_avail -= vres->used_visible_size;
+	list_for_each_entry(block, &vres->blocks, link)
+		block->private = tbo;
 	mutex_unlock(&mgr->lock);
 
 	if (!(vres->base.placement & TTM_PL_FLAG_CONTIGUOUS) &&
@@ -176,8 +179,11 @@ static void xe_ttm_vram_mgr_del(struct ttm_resource_manager *man,
 		to_xe_ttm_vram_mgr_resource(res);
 	struct xe_ttm_vram_mgr *mgr = to_xe_ttm_vram_mgr(man);
 	struct gpu_buddy *mm = &mgr->mm;
+	struct gpu_buddy_block *block;
 
 	mutex_lock(&mgr->lock);
+	list_for_each_entry(block, &vres->blocks, link)
+		block->private = NULL;
 	gpu_buddy_free_list(mm, &vres->blocks, 0);
 	mgr->visible_avail += vres->used_visible_size;
 	mutex_unlock(&mgr->lock);
-- 
2.52.0

[PATCH V16 02/12] drm/xe: Link LRC BO and its execution Queue

[Tejas Upadhyay]

To establish a link between an LRC BO (Logical Ring Context
Buffer Object) and its corresponding execution Queue in the
drm/xe driver, you need to store a back-pointer to the queue
within the BO's private data structure. This allows the
driver to identify and take corrective action on the specific
queue if the LRC BO encounters an error (e.g., memory
corruption or eviction issues).

V3(Sashiko):
- Placeholder of 8 byte for non-lrc bo is acceptable
- Assign bo-q is safe just use READ_ONCE/WRITE_ONCE
V2(MattB):
- Handle multiqueue

Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_bo_types.h   | 3 +++
 drivers/gpu/drm/xe/xe_exec_queue.c | 6 ++++++
 drivers/gpu/drm/xe/xe_lrc.c        | 1 +
 3 files changed, 10 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_bo_types.h b/drivers/gpu/drm/xe/xe_bo_types.h
index e45f24301050..a9c48e440669 100644
--- a/drivers/gpu/drm/xe/xe_bo_types.h
+++ b/drivers/gpu/drm/xe/xe_bo_types.h
@@ -20,6 +20,7 @@
 struct xe_device;
 struct xe_mem_pool_node;
 struct xe_vm;
+struct xe_exec_queue;
 
 #define XE_BO_MAX_PLACEMENTS	3
 
@@ -42,6 +43,8 @@ struct xe_bo {
 	u32 flags;
 	/** @vm: VM this BO is attached to, for extobj this will be NULL */
 	struct xe_vm *vm;
+	/** @q: Queue this BO is attached to, mostly for LRC BO, NULL otherwise */
+	struct xe_exec_queue *q;
 	/** @tile: Tile this BO is attached to (kernel BO only) */
 	struct xe_tile *tile;
 	/** @placements: valid placements for this BO */
diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
index 9f3d022a1463..d6c7f346f49b 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue.c
+++ b/drivers/gpu/drm/xe/xe_exec_queue.c
@@ -390,6 +390,12 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)
 				goto err_lrc;
 			}
 
+			/*
+			 * The queue ref counts the LRCs, thus it safe for the LRC BO to hold a
+			 * pointer to queue without reference.  The reader holds dma_resv (
+			 * xe_bo_lock) which serializes with xe_lrc_finish().
+			 */
+			WRITE_ONCE(lrc->bo->q, xe_exec_queue_multi_queue_primary(q));
 			xe_exec_queue_set_lrc(q, lrc, i);
 
 			if (__lrc)
diff --git a/drivers/gpu/drm/xe/xe_lrc.c b/drivers/gpu/drm/xe/xe_lrc.c
index 35b4e8289b5f..675902753735 100644
--- a/drivers/gpu/drm/xe/xe_lrc.c
+++ b/drivers/gpu/drm/xe/xe_lrc.c
@@ -1066,6 +1066,7 @@ static void xe_lrc_set_ppgtt(struct xe_lrc *lrc, struct xe_vm *vm)
 static void xe_lrc_finish(struct xe_lrc *lrc)
 {
 	xe_hw_fence_ctx_finish(&lrc->fence_ctx);
+	WRITE_ONCE(lrc->bo->q, NULL);
 	xe_bo_unpin_map_no_vm(lrc->bo);
 	xe_bo_unpin_map_no_vm(lrc->seqno_bo);
 }
-- 
2.52.0

[PATCH V16 03/12] drm/xe: Extend BO purge to handle vram pages as well

[Tejas Upadhyay]

Recent driver update introduce support for purgeable buffer
objects (BOs), extending the API to include VRAM pages to
better manage memory pressure and enable memory offlining.

Reviewed-by: Arvind Yadav <arvind.yadav@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 5 +----
 drivers/gpu/drm/xe/xe_bo.h | 1 +
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index dde309821237..52f81e972ada 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -921,7 +921,7 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
  *
  * Return: 0 on success, negative error code on failure
  */
-static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operation_ctx *ctx)
+int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operation_ctx *ctx)
 {
 	struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
 	struct ttm_placement place = {};
@@ -929,9 +929,6 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 
 	xe_bo_assert_held(bo);
 
-	if (!ttm_bo->ttm)
-		return 0;
-
 	if (!xe_bo_madv_is_dontneed(bo))
 		return 0;
 
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index e8081af5bfc1..90b15fff36c7 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -600,6 +600,7 @@ struct xe_bo_shrink_flags {
 long xe_bo_shrink(struct ttm_operation_ctx *ctx, struct ttm_buffer_object *bo,
 		  const struct xe_bo_shrink_flags flags,
 		  unsigned long *scanned);
+int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operation_ctx *ctx);
 
 /**
  * xe_bo_is_mem_type - Whether the bo currently resides in the given
-- 
2.52.0

[PATCH V16 04/12] drm/xe/bo: Make xe_bo_is_user() public

[Tejas Upadhyay]

Export xe_bo_is_user() so it can be used by the VRAM page offline
code to distinguish user-created BOs from kernel BOs when deciding
whether a faulty page can be safely purged or requires a full reset.

Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.c | 8 +++++++-
 drivers/gpu/drm/xe/xe_bo.h | 1 +
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 52f81e972ada..b077d137da57 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -158,7 +158,13 @@ bool xe_bo_is_vm_bound(struct xe_bo *bo)
 	return !list_empty(&bo->ttm.base.gpuva.list);
 }
 
-static bool xe_bo_is_user(struct xe_bo *bo)
+/**
+ * xe_bo_is_user - Check if BO is user-created
+ * @bo: The BO
+ *
+ * Returns: true if @bo was created by userspace
+ */
+bool xe_bo_is_user(struct xe_bo *bo)
 {
 	return bo->flags & XE_BO_FLAG_USER;
 }
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 90b15fff36c7..eede678ad303 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -601,6 +601,7 @@ long xe_bo_shrink(struct ttm_operation_ctx *ctx, struct ttm_buffer_object *bo,
 		  const struct xe_bo_shrink_flags flags,
 		  unsigned long *scanned);
 int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operation_ctx *ctx);
+bool xe_bo_is_user(struct xe_bo *bo);
 
 /**
  * xe_bo_is_mem_type - Whether the bo currently resides in the given
-- 
2.52.0

[PATCH V16 05/12] drm/xe: Guard teardown paths against purged BOs

[Tejas Upadhyay]

VRAM page offlining can purge BOs that are still referenced by page
tables, exec queues, and DMA-buf exports. Add xe_bo_is_purged()
guards in the teardown paths to prevent unpinning or mapping an
already-purged BO:

- xe_bo_unpin_map_no_vm(): skip unpin if purged
- xe_dma_buf_map(): return -ENOENT early if purged
- xe_exec_queue_update_run_ticks(): skip LRC timestamp read if purged
- xe_pt_destroy(): skip unpin if purged

v2(Himal):
- take dma_resv lock before calling xe_bo_is_purged()

Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_bo.h         | 3 ++-
 drivers/gpu/drm/xe/xe_dma_buf.c    | 3 +++
 drivers/gpu/drm/xe/xe_exec_queue.c | 8 ++++++--
 drivers/gpu/drm/xe/xe_pt.c         | 5 ++++-
 4 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index eede678ad303..dfcd0e57073b 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -364,7 +364,8 @@ static inline void xe_bo_unpin_map_no_vm(struct xe_bo *bo)
 {
 	if (likely(bo)) {
 		xe_bo_lock(bo, false);
-		xe_bo_unpin(bo);
+		if (!xe_bo_is_purged(bo))
+			xe_bo_unpin(bo);
 		xe_bo_unlock(bo);
 
 		xe_bo_put(bo);
diff --git a/drivers/gpu/drm/xe/xe_dma_buf.c b/drivers/gpu/drm/xe/xe_dma_buf.c
index bf0728838ead..5d9f1cd24b7f 100644
--- a/drivers/gpu/drm/xe/xe_dma_buf.c
+++ b/drivers/gpu/drm/xe/xe_dma_buf.c
@@ -104,6 +104,9 @@ static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,
 	struct sg_table *sgt;
 	int r = 0;
 
+	if (xe_bo_is_purged(bo))
+		return ERR_PTR(-ENOENT);
+
 	if (!attach->peer2peer && !xe_bo_can_migrate(bo, XE_PL_TT))
 		return ERR_PTR(-EOPNOTSUPP);
 
diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
index d6c7f346f49b..5432710c1cfb 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue.c
+++ b/drivers/gpu/drm/xe/xe_exec_queue.c
@@ -1575,8 +1575,12 @@ void xe_exec_queue_update_run_ticks(struct xe_exec_queue *q)
 	 * errors.
 	 */
 	lrc = q->lrc[0];
-	new_ts = xe_lrc_update_timestamp(lrc, &old_ts);
-	q->xef->run_ticks[q->class] += (new_ts - old_ts) * q->width;
+	xe_bo_lock(lrc->bo, false);
+	if (!xe_bo_is_purged(lrc->bo)) {
+		new_ts = xe_lrc_update_timestamp(lrc, &old_ts);
+		q->xef->run_ticks[q->class] += (new_ts - old_ts) * q->width;
+	}
+	xe_bo_unlock(lrc->bo);
 
 	drm_dev_exit(idx);
 }
diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c
index 5d990c1c3740..b6575a6bada4 100644
--- a/drivers/gpu/drm/xe/xe_pt.c
+++ b/drivers/gpu/drm/xe/xe_pt.c
@@ -212,7 +212,10 @@ void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred)
 		return;
 
 	XE_WARN_ON(!list_empty(&pt->bo->ttm.base.gpuva.list));
-	xe_bo_unpin(pt->bo);
+	xe_bo_lock(pt->bo, false);
+	if (!xe_bo_is_purged(pt->bo))
+		xe_bo_unpin(pt->bo);
+	xe_bo_unlock(pt->bo);
 	xe_bo_put_deferred(pt->bo, deferred);
 
 	if (pt->level > 0 && pt->num_live) {
-- 
2.52.0

[PATCH V16 06/12] drm/xe/vram: Extract buddy alloc and free helpers

[Tejas Upadhyay]

Factor out xe_ttm_vram_buddy_alloc() and xe_ttm_vram_buddy_free()
from xe_ttm_vram_mgr_new() and xe_ttm_vram_mgr_del(). These helpers
consolidate block allocation with visible-size tracking and
block->private tagging, making them reusable by the upcoming VRAM
page offline reservation path.

No functional change.

Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 82 +++++++++++++++++-----------
 1 file changed, 51 insertions(+), 31 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index 5ab5dfdb183c..49eeec90a470 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -46,6 +46,40 @@ static inline bool xe_is_vram_mgr_blocks_contiguous(struct gpu_buddy *mm,
 	return true;
 }
 
+static int xe_ttm_vram_buddy_alloc(struct xe_ttm_vram_mgr *mgr, u64 start,
+				   u64 end, u64 size, u64 min_page_size,
+				   struct list_head *blocks, unsigned long flags,
+				   void *priv, u64 *used_visible)
+{
+	struct gpu_buddy *mm = &mgr->mm;
+	struct gpu_buddy_block *block;
+	int err;
+
+	err = gpu_buddy_alloc_blocks(mm, start, end, size, min_page_size, blocks, flags);
+	if (err)
+		return err;
+
+	list_for_each_entry(block, blocks, link)
+		block->private = priv;
+
+	if (end <= mgr->visible_size) {
+		*used_visible = size;
+	} else {
+		list_for_each_entry(block, blocks, link) {
+			u64 blk_start = gpu_buddy_block_offset(block);
+
+			if (blk_start < mgr->visible_size) {
+				u64 blk_end = blk_start + gpu_buddy_block_size(mm, block);
+
+				*used_visible += min(blk_end, mgr->visible_size) - blk_start;
+			}
+		}
+	}
+
+	mgr->visible_avail -= *used_visible;
+	return 0;
+}
+
 static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
 			       struct ttm_buffer_object *tbo,
 			       const struct ttm_place *place,
@@ -54,7 +88,6 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
 	struct xe_ttm_vram_mgr *mgr = to_xe_ttm_vram_mgr(man);
 	struct xe_ttm_vram_mgr_resource *vres;
 	struct gpu_buddy *mm = &mgr->mm;
-	struct gpu_buddy_block *block;
 	u64 size, min_page_size;
 	unsigned long lpfn;
 	int err;
@@ -115,32 +148,12 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
 		goto error_unlock;
 	}
 
-	err = gpu_buddy_alloc_blocks(mm, (u64)place->fpfn << PAGE_SHIFT,
-				     (u64)lpfn << PAGE_SHIFT, size,
-				     min_page_size, &vres->blocks, vres->flags);
+	err = xe_ttm_vram_buddy_alloc(mgr, (u64)place->fpfn << PAGE_SHIFT,
+				      (u64)lpfn << PAGE_SHIFT, size,
+				      min_page_size, &vres->blocks, vres->flags,
+				      tbo, &vres->used_visible_size);
 	if (err)
 		goto error_unlock;
-
-	if (lpfn <= mgr->visible_size >> PAGE_SHIFT) {
-		vres->used_visible_size = size;
-	} else {
-		struct gpu_buddy_block *block;
-
-		list_for_each_entry(block, &vres->blocks, link) {
-			u64 start = gpu_buddy_block_offset(block);
-
-			if (start < mgr->visible_size) {
-				u64 end = start + gpu_buddy_block_size(mm, block);
-
-				vres->used_visible_size +=
-					min(end, mgr->visible_size) - start;
-			}
-		}
-	}
-
-	mgr->visible_avail -= vres->used_visible_size;
-	list_for_each_entry(block, &vres->blocks, link)
-		block->private = tbo;
 	mutex_unlock(&mgr->lock);
 
 	if (!(vres->base.placement & TTM_PL_FLAG_CONTIGUOUS) &&
@@ -172,20 +185,27 @@ static int xe_ttm_vram_mgr_new(struct ttm_resource_manager *man,
 	return err;
 }
 
+static void xe_ttm_vram_buddy_free(struct xe_ttm_vram_mgr *mgr,
+				   struct list_head *blocks,
+				   u64 used_visible)
+{
+	struct gpu_buddy_block *block;
+
+	list_for_each_entry(block, blocks, link)
+		block->private = NULL;
+	gpu_buddy_free_list(&mgr->mm, blocks, 0);
+	mgr->visible_avail += used_visible;
+}
+
 static void xe_ttm_vram_mgr_del(struct ttm_resource_manager *man,
 				struct ttm_resource *res)
 {
 	struct xe_ttm_vram_mgr_resource *vres =
 		to_xe_ttm_vram_mgr_resource(res);
 	struct xe_ttm_vram_mgr *mgr = to_xe_ttm_vram_mgr(man);
-	struct gpu_buddy *mm = &mgr->mm;
-	struct gpu_buddy_block *block;
 
 	mutex_lock(&mgr->lock);
-	list_for_each_entry(block, &vres->blocks, link)
-		block->private = NULL;
-	gpu_buddy_free_list(mm, &vres->blocks, 0);
-	mgr->visible_avail += vres->used_visible_size;
+	xe_ttm_vram_buddy_free(mgr, &vres->blocks, vres->used_visible_size);
 	mutex_unlock(&mgr->lock);
 
 	ttm_resource_fini(man, res);
-- 
2.52.0

[PATCH V16 07/12] drm/xe/vram: Add page offline data structures and lifecycle

[Tejas Upadhyay]

Add xe_ttm_vram_offline_resource to track individual offlined VRAM
pages, and extend xe_ttm_vram_mgr with offlined_pages/queued_pages
lists and their counters.

Initialize the lists in __xe_ttm_vram_mgr_init() and add
xe_ttm_vram_free_bad_pages() to release all tracked pages during
xe_ttm_vram_mgr_fini() teardown.

v2(Himal):
- Address possible leak in xe_ttm_vram_mgr_fini()
- Remove unused dev and add comment for used_visible_size 0

Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c       | 25 +++++++++++++++++++
 drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h | 28 ++++++++++++++++++++++
 2 files changed, 53 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index 49eeec90a470..1885f2aa64df 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -300,6 +300,25 @@ static const struct ttm_resource_manager_func xe_ttm_vram_mgr_func = {
 	.debug	= xe_ttm_vram_mgr_debug
 };
 
+static void xe_ttm_vram_free_bad_pages(struct xe_ttm_vram_mgr *mgr)
+{
+	struct xe_ttm_vram_offline_resource *pos, *n;
+
+	list_for_each_entry_safe(pos, n, &mgr->offlined_pages, offlined_link) {
+		xe_ttm_vram_buddy_free(mgr, &pos->blocks, pos->used_visible_size);
+		list_del_rcu(&pos->offlined_link);
+		--mgr->n_offlined_pages;
+		kfree_rcu(pos, rcu);
+	}
+	list_for_each_entry_safe(pos, n, &mgr->queued_pages, queued_link) {
+		/* queued entries have no buddy reservation yet */
+		xe_ttm_vram_buddy_free(mgr, &pos->blocks, 0);
+		list_del_rcu(&pos->queued_link);
+		--mgr->n_queued_pages;
+		kfree_rcu(pos, rcu);
+	}
+}
+
 static void xe_ttm_vram_mgr_fini(struct drm_device *dev, void *arg)
 {
 	struct xe_device *xe = to_xe_device(dev);
@@ -308,6 +327,10 @@ static void xe_ttm_vram_mgr_fini(struct drm_device *dev, void *arg)
 
 	ttm_resource_manager_set_used(man, false);
 
+	mutex_lock(&mgr->lock);
+	xe_ttm_vram_free_bad_pages(mgr);
+	mutex_unlock(&mgr->lock);
+
 	if (ttm_resource_manager_evict_all(&xe->ttm, man))
 		return;
 
@@ -338,6 +361,8 @@ int __xe_ttm_vram_mgr_init(struct xe_device *xe, struct xe_ttm_vram_mgr *mgr,
 	err = drmm_mutex_init(&xe->drm, &mgr->lock);
 	if (err)
 		return err;
+	INIT_LIST_HEAD(&mgr->offlined_pages);
+	INIT_LIST_HEAD(&mgr->queued_pages);
 	mgr->default_page_size = default_page_size;
 	mgr->visible_size = io_size;
 	mgr->visible_avail = io_size;
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
index 9106da056b49..9dc6fc5a3c38 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
@@ -19,6 +19,14 @@ struct xe_ttm_vram_mgr {
 	struct ttm_resource_manager manager;
 	/** @mm: DRM buddy allocator which manages the VRAM */
 	struct gpu_buddy mm;
+	/** @offlined_pages: List of offlined pages */
+	struct list_head offlined_pages;
+	/** @n_offlined_pages: Number of offlined pages */
+	u16 n_offlined_pages;
+	/** @queued_pages: List of queued pages */
+	struct list_head queued_pages;
+	/** @n_queued_pages: Number of queued pages */
+	u16 n_queued_pages;
 	/** @visible_size: Proped size of the CPU visible portion */
 	u64 visible_size;
 	/** @visible_avail: CPU visible portion still unallocated */
@@ -45,4 +53,24 @@ struct xe_ttm_vram_mgr_resource {
 	unsigned long flags;
 };
 
+/**
+ * struct xe_ttm_vram_offline_resource - Tracks a single offlined VRAM page
+ */
+struct xe_ttm_vram_offline_resource {
+	/** @offlined_link: Link into mgr->offlined_pages */
+	struct list_head offlined_link;
+	/** @queued_link: Link into mgr->queued_pages */
+	struct list_head queued_link;
+	/** @blocks: Buddy blocks reserved for this page */
+	struct list_head blocks;
+	/** @used_visible_size: CPU-visible bytes consumed */
+	u64 used_visible_size;
+	/** @addr: Faulty DPA reported by HW */
+	u64 addr;
+	/** @status: Reservation status (0=pending, 1=fail) */
+	bool status;
+	/** @rcu: RCU head for deferred freeing */
+	struct rcu_head rcu;
+};
+
 #endif
-- 
2.52.0

[PATCH V16 08/12] drm/xe/vram: Add VRAM page offline fault handler

[Tejas Upadhyay]

Add the core VRAM page offlining logic to handle HW-reported faulty
physical addresses:

- xe_ttm_vram_purge_page(): Purges the BO containing the faulty
  address. Bans the associated VM (if page table BO) and exec queue
  (if LRC BO). Moves xe_exec_queue_kill() outside xe_bo_lock() to
  avoid AB-BA deadlock with vm->lock. Uses READ_ONCE(bo->q) to
  safely access the exec queue pointer.

- xe_ttm_vram_page_already_processed(): Checks if an address is
  already tracked in offlined_pages or queued_pages lists to avoid
  double-processing.

- xe_ttm_vram_reserve_page_at_addr(): Two-phase reservation that
  first queues the page, purges the BO outside the lock, then
  reserves the buddy block. Handles both allocated (BO present)
  and free page cases. Returns -EIO for critical kernel BOs to
  trigger system reset.

- xe_ttm_vram_addr_to_region(): Maps a DPA to its VRAM region.
  Uses GSMBASE MMIO register to detect GSM addresses (returns NULL
  for reset path). Returns ERR_PTR(-EOPNOTSUPP) for addresses
  outside any known region.

- xe_ttm_vram_handle_addr_fault(): Entry point called by RAS.
  Returns -EEXIST if already processed, -EIO for GSM/critical BO,
  -EOPNOTSUPP if out of bounds.

v11(Himal):
- match everywhere with enum vs bool for status member
- Fix comment and remove unused var
- if purge fail let next alloc confirm failure
- pass absolute address, useful for multi tile

Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c       | 291 +++++++++++++++++++++
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.h       |   1 +
 drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h |  14 +-
 3 files changed, 304 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index 1885f2aa64df..c3b2d5370a08 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -11,9 +11,14 @@
 #include <drm/ttm/ttm_placement.h>
 #include <drm/ttm/ttm_range_manager.h>
 
+#include "regs/xe_regs.h"
 #include "xe_bo.h"
 #include "xe_device.h"
+#include "xe_exec_queue.h"
+#include "xe_lrc.h"
+#include "xe_mmio.h"
 #include "xe_res_cursor.h"
+#include "xe_ttm_stolen_mgr.h"
 #include "xe_ttm_vram_mgr.h"
 #include "xe_vram_types.h"
 
@@ -519,3 +524,289 @@ u64 xe_ttm_vram_get_avail(struct ttm_resource_manager *man)
 
 	return avail;
 }
+
+static int xe_ttm_vram_purge_page(struct xe_device *xe, struct xe_bo *bo)
+{
+	struct ttm_operation_ctx ctx = {};
+	struct xe_exec_queue *q_to_put = NULL;
+	struct xe_exec_queue *q = NULL;
+	struct xe_vm *vm = NULL;
+	u32	flags;
+	int ret = 0;
+
+	xe_bo_lock(bo, false);
+	if (bo->vm)
+		vm = xe_vm_get(bo->vm);
+	flags = bo->flags;
+	xe_bo_unlock(bo);
+	/*  Ban VM if BO is PPGTT */
+	if (vm && (flags & XE_BO_FLAG_PAGETABLE)) {
+		down_write(&vm->lock);
+		xe_vm_kill(vm, true);
+		up_write(&vm->lock);
+	}
+	if (vm)
+		xe_vm_put(vm);
+
+	xe_bo_lock(bo, false);
+	q = READ_ONCE(bo->q);
+	/*  Ban exec queue if BO is lrc */
+	if (q && xe_exec_queue_get_unless_zero(q)) {
+		/* ban queue */
+		q_to_put = q;
+	}
+
+	if (bo->purgeable.state == XE_MADV_PURGEABLE_PURGED) {
+		/* Already purged by shrinker during unlocked window — nothing to do */
+		xe_bo_unlock(bo);
+		goto out;
+	}
+
+	xe_bo_set_purgeable_state(bo, XE_MADV_PURGEABLE_DONTNEED);
+	ttm_bo_unmap_virtual(&bo->ttm);   /* nuke CPU mmap + VRAM IO mappings */
+	if (xe_bo_is_pinned(bo))
+		xe_bo_unpin(bo);
+	ret = xe_ttm_bo_purge(&bo->ttm, &ctx);
+	xe_bo_unlock(bo);
+
+out:
+	if (q_to_put) {
+		xe_exec_queue_kill(q_to_put);
+		xe_exec_queue_put(q_to_put);
+	}
+
+	return ret;
+}
+
+static bool xe_ttm_vram_page_already_processed(struct xe_ttm_vram_mgr *mgr,
+					       u64 addr)
+{
+	struct xe_ttm_vram_offline_resource *pos;
+
+	lockdep_assert_held(&mgr->lock);
+
+	list_for_each_entry(pos, &mgr->offlined_pages, offlined_link) {
+		if (pos->addr == addr)
+			return true;
+	}
+
+	list_for_each_entry(pos, &mgr->queued_pages, queued_link) {
+		if (pos->addr == addr)
+			return true;
+	}
+
+	return false;
+}
+
+static int xe_ttm_vram_reserve_page_at_addr(struct xe_device *xe, u64 addr,
+					    struct xe_ttm_vram_mgr *vram_mgr, struct gpu_buddy *mm)
+{
+	struct xe_ttm_vram_offline_resource *nentry;
+	struct ttm_buffer_object *tbo = NULL;
+	struct xe_bo *pbo_to_put = NULL;
+	struct gpu_buddy_block *block;
+	u64 size = SZ_4K;
+	int ret = 0;
+
+	scoped_guard(mutex, &vram_mgr->lock) {
+		if (xe_ttm_vram_page_already_processed(vram_mgr, addr))
+			return -EEXIST;
+		block = gpu_buddy_allocated_addr_to_block(mm, addr);
+		if (WARN_ON(IS_ERR(block)))
+			return PTR_ERR(block);
+
+		nentry = kzalloc_obj(*nentry);
+		if (!nentry)
+			return -ENOMEM;
+		INIT_LIST_HEAD(&nentry->blocks);
+		nentry->status = XE_PAGE_RESERVE_PENDING;
+		nentry->addr = addr;
+
+		if (block) {
+			struct xe_bo *pbo;
+
+			if (!block->private) {
+				/* Race: another thread just reserved this block */
+				kfree(nentry);
+				return -EEXIST;
+			}
+			tbo = block->private;
+			pbo = ttm_to_xe_bo(tbo);
+
+			/* Get reference safely - BO may have zero refcount */
+			if (!xe_bo_get_unless_zero(pbo)) {
+				kfree(nentry);
+				return -ENOENT;
+			}
+			/*
+			 * Critical kernel BO? Best-effort check without resv lock;
+			 * worst case a concurrent pin causes reset path unnecessarily.
+			 */
+			if ((pbo->ttm.type == ttm_bo_type_kernel &&
+			     !(pbo->flags & XE_BO_FLAG_PINNED_LATE_RESTORE)) ||
+			    (xe_bo_is_user(pbo) && xe_bo_is_pinned(pbo))) {
+				kfree(nentry);
+				pbo_to_put = pbo;
+				drm_err(&xe->drm,
+					"%s: addr: 0x%llx is critical kernel bo, requesting SBR\n",
+					__func__, addr);
+				break;
+			}
+			++vram_mgr->n_queued_pages;
+			list_add_rcu(&nentry->queued_link, &vram_mgr->queued_pages);
+		}
+	}
+
+	/* Deferred put outside lock to avoid recursive deadlock */
+	if (pbo_to_put) {
+		xe_bo_put(pbo_to_put);
+		/* Hint System controller driver for reset with -EIO  */
+		return -EIO;
+	}
+
+	if (block) {
+		struct xe_ttm_vram_offline_resource *pos, *n;
+		struct xe_bo *pbo = ttm_to_xe_bo(tbo);
+
+		/*
+		 * Purge BO containing address - reference held from above.
+		 * Note: brief window between purge (freeing blocks) and re-reserve
+		 * below. If another allocation claims the block, buddy_alloc fails
+		 * and the status will be shown as failed reservation.
+		 */
+		ret = xe_ttm_vram_purge_page(xe, pbo);
+		xe_bo_put(pbo);
+		if (ret)
+			drm_warn(&xe->drm, "Purge failed at addr:0x%llx, ret:%d\n", addr, ret);
+
+		/* Reserve page at address addr*/
+		scoped_guard(mutex, &vram_mgr->lock) {
+			ret = xe_ttm_vram_buddy_alloc(vram_mgr, addr, addr + size,
+						      size, size, &nentry->blocks,
+						      GPU_BUDDY_RANGE_ALLOCATION,
+						      NULL, &nentry->used_visible_size);
+			if (ret) {
+				drm_warn(&xe->drm,
+					 "Could not reserve page at addr:0x%llx, ret:%d\n",
+					 addr, ret);
+				nentry->status = XE_PAGE_RESERVE_FAIL;
+				return ret;
+			}
+
+			list_for_each_entry_safe(pos, n, &vram_mgr->queued_pages, queued_link) {
+				if (pos->addr == nentry->addr) {
+					--vram_mgr->n_queued_pages;
+					list_del_rcu(&pos->queued_link);
+					break;
+				}
+			}
+			list_add_rcu(&nentry->offlined_link, &vram_mgr->offlined_pages);
+			/* RAS will send command to FW for offlining page based on ret value */
+			++vram_mgr->n_offlined_pages;
+			return ret;
+		}
+	} else {
+		struct xe_ttm_vram_offline_resource *pos, *n;
+
+		scoped_guard(mutex, &vram_mgr->lock) {
+			++vram_mgr->n_queued_pages;
+			list_add_rcu(&nentry->queued_link, &vram_mgr->queued_pages);
+			ret = xe_ttm_vram_buddy_alloc(vram_mgr, addr, addr + size,
+						      size, size, &nentry->blocks,
+						      GPU_BUDDY_RANGE_ALLOCATION,
+						      NULL, &nentry->used_visible_size);
+			if (ret) {
+				drm_warn(&xe->drm,
+					 "Could not reserve page at addr:0x%llx, ret:%d\n",
+					 addr, ret);
+				nentry->status = XE_PAGE_RESERVE_FAIL;
+				return ret;
+			}
+
+			list_for_each_entry_safe(pos, n, &vram_mgr->queued_pages, queued_link) {
+				if (pos->addr == nentry->addr) {
+					--vram_mgr->n_queued_pages;
+					list_del_rcu(&pos->queued_link);
+					break;
+				}
+			}
+			++vram_mgr->n_offlined_pages;
+			list_add_rcu(&nentry->offlined_link, &vram_mgr->offlined_pages);
+			/* RAS will send command to FW for offlining page based on ret value */
+		}
+	}
+	/* Success */
+	return ret;
+}
+
+static struct xe_vram_region *xe_ttm_vram_addr_to_region(struct xe_device *xe, u64 addr)
+{
+	u64 raw_offset = xe_mmio_read64_2x32(&xe_device_get_root_tile(xe)->mmio, GSMBASE);
+	/* force a 4K (4096 bytes) page alignment */
+	u64 gsmbase_dpa = raw_offset & ~(u64)(PAGE_SIZE - 1);
+	struct xe_vram_region *vr;
+	struct xe_tile *tile;
+	int id;
+
+	/* Addr from GSM? */
+	if (addr >= gsmbase_dpa)
+		/* Return NULL so the caller can request reset (SBR) */
+		return NULL;
+
+	for_each_tile(tile, xe, id) {
+		vr = tile->mem.vram;
+		if (addr >= vr->dpa_base &&
+		    addr < vr->dpa_base + vr->usable_size)
+			return vr;
+	}
+
+	/*
+	 * Return an explicit error pointer so the caller knows the addr
+	 * is invalid and should be ignored, NOT SBR.
+	 */
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+/**
+ * xe_ttm_vram_handle_addr_fault - Handle vram physical address error flaged
+ * @xe: pointer to parent device
+ * @addr: physical faulty address
+ *
+ * Handle the physcial faulty address error on specific tile.
+ *
+ * Returns 0 for success, negative error code otherwise as follow:
+ * * %-EIO - critical BO or address outside any VRAM region; next action is reset.
+ * * %-EOPNOTSUPP - log-only policy or unknown address; no further action.
+ * * %-ENOMEM - allocation failure; next action is reset.
+ * * %-ENXIO - address not found in buddy; next action is reset.
+ * * %-EEXIST - address already processed; no further action.
+ * * % Any other negative error - next action is reset.
+ */
+int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr)
+{
+	struct xe_ttm_vram_mgr *vram_mgr;
+	struct xe_vram_region *vr;
+	struct gpu_buddy *mm;
+
+	vr = xe_ttm_vram_addr_to_region(xe, addr);
+	if (IS_ERR(vr)) {
+		/*
+		 * The addr is outside VRAM and GSM.
+		 * Log a debug message if needed, and safely exit/ignore.
+		 */
+		drm_dbg(&xe->drm, "Address %llx is out of bounds, ignoring fault.\n", addr);
+		return PTR_ERR(vr);
+	}
+	if (!vr) {
+		drm_err(&xe->drm, "%s:%d GSM addr:%llx error requesting SBR\n",
+			__func__, __LINE__, addr);
+		/* Hint System controller driver for reset with -EIO  */
+		return -EIO;
+	}
+	vram_mgr = &vr->ttm;
+	mm = &vram_mgr->mm;
+
+	/* Reserve page at address */
+	return xe_ttm_vram_reserve_page_at_addr(xe, addr - vr->dpa_base, vram_mgr, mm);
+}
+EXPORT_SYMBOL(xe_ttm_vram_handle_addr_fault);
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
index 87b7fae5edba..d5392beff30c 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
@@ -31,6 +31,7 @@ u64 xe_ttm_vram_get_cpu_visible_size(struct ttm_resource_manager *man);
 void xe_ttm_vram_get_used(struct ttm_resource_manager *man,
 			  u64 *used, u64 *used_visible);
 
+int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr);
 static inline struct xe_ttm_vram_mgr_resource *
 to_xe_ttm_vram_mgr_resource(struct ttm_resource *res)
 {
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
index 9dc6fc5a3c38..dc97b0ad0e51 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
@@ -53,6 +53,16 @@ struct xe_ttm_vram_mgr_resource {
 	unsigned long flags;
 };
 
+/**
+ * enum xe_page_reserve_status - Buddy reservation status
+ * @XE_PAGE_RESERVE_PENDING: reservation in progress
+ * @XE_PAGE_RESERVE_FAIL: reservation failed
+ */
+enum xe_page_reserve_status {
+	XE_PAGE_RESERVE_PENDING = 0,
+	XE_PAGE_RESERVE_FAIL,
+};
+
 /**
  * struct xe_ttm_vram_offline_resource - Tracks a single offlined VRAM page
  */
@@ -67,8 +77,8 @@ struct xe_ttm_vram_offline_resource {
 	u64 used_visible_size;
 	/** @addr: Faulty DPA reported by HW */
 	u64 addr;
-	/** @status: Reservation status (0=pending, 1=fail) */
-	bool status;
+	/** @status: buddy reservation status */
+	enum xe_page_reserve_status status;
 	/** @rcu: RCU head for deferred freeing */
 	struct rcu_head rcu;
 };
-- 
2.52.0

[PATCH V16 09/12] drm/xe/configfs: Add vram bad page reservation policy

[Tejas Upadhyay]

The interface enables setting the policy for how bad pages are
handled in VRAM. This is crucial for maintaining system
stability in scenarios where VRAM degradation occurs.

By default policy will be "reserve", which can be changed to
"logging" only. val 0 means Logging only, 1 means logging and
offlining.

v4:
- document val 0/1 meaning (Himal)
v3:
- All FW communication moved under RAS
v2:
- Add CRI check and rebase

Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
---
 drivers/gpu/drm/xe/xe_configfs.c     | 67 +++++++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_configfs.h     |  2 +
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 10 +++++
 3 files changed, 78 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_configfs.c b/drivers/gpu/drm/xe/xe_configfs.c
index 052cce962161..ead59208858f 100644
--- a/drivers/gpu/drm/xe/xe_configfs.c
+++ b/drivers/gpu/drm/xe/xe_configfs.c
@@ -61,7 +61,8 @@
  *	    ├── survivability_mode
  *	    ├── gt_types_allowed
  *	    ├── engines_allowed
- *	    └── enable_psmi
+ *          ├── enable_psmi
+ *          └── bad_page_reservation
  *
  * After configuring the attributes as per next section, the device can be
  * probed with::
@@ -159,6 +160,19 @@
  *
  * This attribute can only be set before binding to the device.
  *
+ * Bad pages reservation:
+ * ---------------------
+ *
+ * Controls how bad VRAM pages are handled:
+ *  0 - Logging only (report in dmesg, no offlining)
+ *  1 - Logging and offlining (default)
+ *
+ *  Example to disable offlining::
+ *
+ *      # echo 0 > /sys/kernel/config/xe/0000:03:00.0/bad_page_reservation
+ *
+ * This attribute can only be set before binding to the device.
+ *
  * Context restore BB
  * ------------------
  *
@@ -275,6 +289,7 @@ struct xe_config_group_device {
 		bool survivability_mode;
 		bool enable_psmi;
 		bool enable_multi_queue;
+		bool bad_page_reservation;
 		struct {
 			unsigned int max_vfs;
 			bool admin_only_pf;
@@ -295,6 +310,7 @@ static const struct xe_config_device device_defaults = {
 	.survivability_mode = false,
 	.enable_psmi = false,
 	.enable_multi_queue = true,
+	.bad_page_reservation = true,
 	.sriov = {
 		.max_vfs = XE_DEFAULT_MAX_VFS,
 		.admin_only_pf = XE_DEFAULT_ADMIN_ONLY_PF,
@@ -616,6 +632,32 @@ static ssize_t enable_multi_queue_store(struct config_item *item, const char *pa
 	return len;
 }
 
+static ssize_t bad_page_reservation_show(struct config_item *item, char *page)
+{
+	struct xe_config_device *dev = to_xe_config_device(item);
+
+	return sprintf(page, "%d\n", dev->bad_page_reservation);
+}
+
+static ssize_t bad_page_reservation_store(struct config_item *item, const char *page, size_t len)
+{
+	struct xe_config_group_device *dev = to_xe_config_group_device(item);
+	bool val;
+	int ret;
+
+	ret = kstrtobool(page, &val);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&dev->lock);
+	if (is_bound(dev))
+		return -EBUSY;
+
+	dev->config.bad_page_reservation = val;
+
+	return len;
+}
+
 static bool wa_bb_read_advance(bool dereference, char **p,
 			       const char *append, size_t len,
 			       size_t *max_size)
@@ -855,6 +897,7 @@ CONFIGFS_ATTR(, ctx_restore_mid_bb);
 CONFIGFS_ATTR(, ctx_restore_post_bb);
 CONFIGFS_ATTR(, enable_multi_queue);
 CONFIGFS_ATTR(, enable_psmi);
+CONFIGFS_ATTR(, bad_page_reservation);
 CONFIGFS_ATTR(, engines_allowed);
 CONFIGFS_ATTR(, gt_types_allowed);
 CONFIGFS_ATTR(, survivability_mode);
@@ -864,6 +907,7 @@ static struct configfs_attribute *xe_config_device_attrs[] = {
 	&attr_ctx_restore_post_bb,
 	&attr_enable_multi_queue,
 	&attr_enable_psmi,
+	&attr_bad_page_reservation,
 	&attr_engines_allowed,
 	&attr_gt_types_allowed,
 	&attr_survivability_mode,
@@ -1142,6 +1186,7 @@ static void dump_custom_dev_config(struct pci_dev *pdev,
 	PRI_CUSTOM_ATTR("%llx", engines_allowed);
 	PRI_CUSTOM_ATTR("%d", enable_multi_queue);
 	PRI_CUSTOM_ATTR("%d", enable_psmi);
+	PRI_CUSTOM_ATTR("%d", bad_page_reservation);
 	PRI_CUSTOM_ATTR("%d", survivability_mode);
 	PRI_CUSTOM_ATTR("%u", sriov.admin_only_pf);
 
@@ -1290,6 +1335,26 @@ bool xe_configfs_get_enable_multi_queue(struct pci_dev *pdev)
 	return ret;
 }
 
+/**
+ * xe_configfs_get_bad_page_reservation - get configfs bad_page_reservation setting
+ * @pdev: pci device
+ *
+ * Return: bad_page_reservation setting in configfs
+ */
+bool xe_configfs_get_bad_page_reservation(struct pci_dev *pdev)
+{
+	struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
+	bool ret;
+
+	if (!dev)
+		return device_defaults.bad_page_reservation;
+
+	ret = dev->config.bad_page_reservation;
+	config_group_put(&dev->group);
+
+	return ret;
+}
+
 /**
  * xe_configfs_get_ctx_restore_mid_bb - get configfs ctx_restore_mid_bb setting
  * @pdev: pci device
diff --git a/drivers/gpu/drm/xe/xe_configfs.h b/drivers/gpu/drm/xe/xe_configfs.h
index 4fbbeafba473..7405cc5f3207 100644
--- a/drivers/gpu/drm/xe/xe_configfs.h
+++ b/drivers/gpu/drm/xe/xe_configfs.h
@@ -24,6 +24,7 @@ bool xe_configfs_media_gt_allowed(struct pci_dev *pdev);
 u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev);
 bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev);
 bool xe_configfs_get_enable_multi_queue(struct pci_dev *pdev);
+bool xe_configfs_get_bad_page_reservation(struct pci_dev *pdev);
 u32 xe_configfs_get_ctx_restore_mid_bb(struct pci_dev *pdev,
 				       enum xe_engine_class class,
 				       const u32 **cs);
@@ -44,6 +45,7 @@ static inline bool xe_configfs_media_gt_allowed(struct pci_dev *pdev) { return t
 static inline u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev) { return U64_MAX; }
 static inline bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev) { return false; }
 static inline bool xe_configfs_get_enable_multi_queue(struct pci_dev *pdev) { return true; }
+static inline bool xe_configfs_get_bad_page_reservation(struct pci_dev *pdev) { return true; }
 static inline u32 xe_configfs_get_ctx_restore_mid_bb(struct pci_dev *pdev,
 						     enum xe_engine_class class,
 						     const u32 **cs) { return 0; }
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index c3b2d5370a08..a48196ce303f 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -13,6 +13,7 @@
 
 #include "regs/xe_regs.h"
 #include "xe_bo.h"
+#include "xe_configfs.h"
 #include "xe_device.h"
 #include "xe_exec_queue.h"
 #include "xe_lrc.h"
@@ -787,6 +788,7 @@ int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr)
 	struct xe_ttm_vram_mgr *vram_mgr;
 	struct xe_vram_region *vr;
 	struct gpu_buddy *mm;
+	bool policy;
 
 	vr = xe_ttm_vram_addr_to_region(xe, addr);
 	if (IS_ERR(vr)) {
@@ -806,6 +808,14 @@ int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr)
 	vram_mgr = &vr->ttm;
 	mm = &vram_mgr->mm;
 
+	policy = xe_configfs_get_bad_page_reservation(to_pci_dev(xe->drm.dev));
+	if (!policy) {
+		drm_err(&xe->drm, "0x%llx is reported as corrupted address by HW\n",
+			addr);
+		/* Let RAS report to FW to drop addr from SRAM queue */
+		return -EOPNOTSUPP;
+	}
+
 	/* Reserve page at address */
 	return xe_ttm_vram_reserve_page_at_addr(xe, addr - vr->dpa_base, vram_mgr, mm);
 }
-- 
2.52.0

[PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram pages

[Tejas Upadhyay]

Include a sysfs interface designed to expose information about bad
VRAM pages — those identified as having hardware faults (e.g., ECC
errors). This interface allows userspace tools and administrators to
monitor the health of the GPU's local memory and track the status of
page retirement. Details on bad gpu vram pages can be found under
/sys/bus/pci/devices/<bdf>/vram_bad_pages.

The format is: pfn : gpu_page_size : flags

flags:
  R: reserved, this gpu page is reserved.
  P: pending for reserve, this gpu page is marked as bad, will be
     reserved in next window of page_reserve.
  F: unable to reserve, this gpu page can't be reserved due to some
     reasons.

For example, cat /sys/bus/pci/devices/<bdf>/vram_bad_pages:
  max_pages : 10000
  0x0000000000000000 : 0x0000000000001000 : R
  0x0000000000001234 : 0x0000000000001000 : P

The sysfs binary attribute is created under the PCI device kobject
when the platform supports it and the configfs bad_page_reservation
policy is enabled. Uses RCU-protected list traversal so reads never
block normal VRAM allocation operations.

Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_device_sysfs.c       |   7 +
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c       | 141 +++++++++++++++++++++
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.h       |   1 +
 drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h |   2 +
 4 files changed, 151 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_device_sysfs.c b/drivers/gpu/drm/xe/xe_device_sysfs.c
index a73e0e957cb0..47c5be4180fe 100644
--- a/drivers/gpu/drm/xe/xe_device_sysfs.c
+++ b/drivers/gpu/drm/xe/xe_device_sysfs.c
@@ -8,12 +8,14 @@
 #include <linux/pci.h>
 #include <linux/sysfs.h>
 
+#include "xe_configfs.h"
 #include "xe_device.h"
 #include "xe_device_sysfs.h"
 #include "xe_mmio.h"
 #include "xe_pcode_api.h"
 #include "xe_pcode.h"
 #include "xe_pm.h"
+#include "xe_ttm_vram_mgr.h"
 
 /**
  * DOC: Xe device sysfs
@@ -267,6 +269,7 @@ static const struct attribute_group auto_link_downgrade_attr_group = {
 int xe_device_sysfs_init(struct xe_device *xe)
 {
 	struct device *dev = xe->drm.dev;
+	bool policy;
 	int ret;
 
 	if (xe->d3cold.capable) {
@@ -285,5 +288,9 @@ int xe_device_sysfs_init(struct xe_device *xe)
 			return ret;
 	}
 
+	policy = xe_configfs_get_bad_page_reservation(to_pci_dev(dev));
+	if (xe->info.platform == XE_CRESCENTISLAND && policy)
+		xe_ttm_vram_sysfs_init(xe);
+
 	return 0;
 }
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index a48196ce303f..73cd06c22d37 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -820,3 +820,144 @@ int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr)
 	return xe_ttm_vram_reserve_page_at_addr(xe, addr - vr->dpa_base, vram_mgr, mm);
 }
 EXPORT_SYMBOL(xe_ttm_vram_handle_addr_fault);
+
+static size_t serialize_bad_pages(struct xe_ttm_vram_mgr *mgr, char *buf, size_t max_len)
+{
+	struct xe_ttm_vram_offline_resource *pos;
+	struct gpu_buddy_block *block;
+	size_t s = 0;
+	int printed;
+	int count = 0;
+
+	rcu_read_lock();
+
+	printed = scnprintf(buf + s, max_len - s, "max_pages: %d\n", mgr->max_pages);
+	s += printed;
+
+	list_for_each_entry_rcu(pos, &mgr->offlined_pages, offlined_link) {
+		if (count >= 10000 || s >= max_len)
+			break;
+
+		block = list_first_entry_or_null(&pos->blocks, struct gpu_buddy_block, link);
+		if (!block)
+			continue;
+
+		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx : %c\n",
+				    gpu_buddy_block_offset(block) >> PAGE_SHIFT,
+				    gpu_buddy_block_size(&mgr->mm, block), 'R');
+		s += printed;
+		count++;
+	}
+	list_for_each_entry_rcu(pos, &mgr->queued_pages, queued_link) {
+		u64 pfn, blk_size;
+
+		if (count >= 10000 || s >= max_len)
+			break;
+
+		block = list_first_entry_or_null(&pos->blocks, struct gpu_buddy_block, link);
+		if (block) {
+			pfn = gpu_buddy_block_offset(block) >> PAGE_SHIFT;
+			blk_size = gpu_buddy_block_size(&mgr->mm, block);
+		} else {
+			pfn = pos->addr >> PAGE_SHIFT;
+			blk_size = PAGE_SIZE;
+		}
+
+		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx : %c\n",
+				    pfn, blk_size, pos->status ? 'F' : 'P');
+		s += printed;
+		count++;
+	}
+
+	rcu_read_unlock();
+	return s;
+}
+
+static ssize_t vram_bad_pages_bin_read(struct file *filp, struct kobject *kobj,
+				       const struct bin_attribute *attr, char *buf,
+				       loff_t off, size_t count)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct pci_dev *pdev = to_pci_dev(dev);
+	struct ttm_resource_manager *man;
+	struct xe_ttm_vram_mgr *mgr;
+	size_t allocation_size;
+	struct xe_device *xe;
+	size_t full_data_len;
+	int active_entries;
+	char *temp_buf;
+
+	xe = pdev_to_xe_device(pdev);
+	man = ttm_manager_type(&xe->ttm, XE_PL_VRAM0);
+	if (!man)
+		return -ENODEV;
+	mgr = to_xe_ttm_vram_mgr(man);
+
+	active_entries = READ_ONCE(mgr->n_offlined_pages) + READ_ONCE(mgr->n_queued_pages);
+
+	if (active_entries > 10000)
+		active_entries = 10000;
+
+	allocation_size = 64 + (active_entries * 48);
+
+	temp_buf = kvmalloc(allocation_size, GFP_KERNEL);
+	if (!temp_buf)
+		return -ENOMEM;
+
+	/* serialize_bad_pages uses rcu_read_lock internally */
+	full_data_len = serialize_bad_pages(mgr, temp_buf, allocation_size);
+
+	if (off >= full_data_len) {
+		kvfree(temp_buf);
+		return 0;
+	}
+
+	if (off + count > full_data_len)
+		count = full_data_len - off;
+
+	memcpy(buf, temp_buf + off, count);
+
+	kvfree(temp_buf);
+	return count;
+}
+
+static const struct bin_attribute bin_attr_vram_bad_pages = {
+	.attr = { .name = "vram_bad_pages", .mode = 0444 },
+	.read = vram_bad_pages_bin_read,
+	.size = 0,
+};
+
+static void xe_ttm_vram_sysfs_fini(void *arg)
+{
+	struct xe_device *xe = arg;
+	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
+
+	sysfs_remove_bin_file(&pdev->dev.kobj, &bin_attr_vram_bad_pages);
+}
+
+/**
+ * xe_ttm_vram_sysfs_init - Initialize vram bad pages sysfs binary file
+ * @xe: Xe Device object
+ *
+ * Creates a binary sysfs file under the PCI device for reading
+ * offlined and queued VRAM pages. Supports large entry counts
+ * via offset/count pagination.
+ *
+ * Returns: 0 on success, negative error code on error.
+ */
+int xe_ttm_vram_sysfs_init(struct xe_device *xe)
+{
+	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
+	int err;
+
+	err = sysfs_create_bin_file(&pdev->dev.kobj, &bin_attr_vram_bad_pages);
+	if (err) {
+		dev_err(&pdev->dev,
+			"Failed to create vram_bad_pages sysfs: %d\n",
+			err);
+		return err;
+	}
+
+	return devm_add_action_or_reset(&pdev->dev, xe_ttm_vram_sysfs_fini, xe);
+}
+EXPORT_SYMBOL(xe_ttm_vram_sysfs_init);
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
index d5392beff30c..eb55b0f74ef3 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
@@ -32,6 +32,7 @@ void xe_ttm_vram_get_used(struct ttm_resource_manager *man,
 			  u64 *used, u64 *used_visible);
 
 int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr);
+int xe_ttm_vram_sysfs_init(struct xe_device *xe);
 static inline struct xe_ttm_vram_mgr_resource *
 to_xe_ttm_vram_mgr_resource(struct ttm_resource *res)
 {
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
index dc97b0ad0e51..efcf3e1d4e80 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
@@ -37,6 +37,8 @@ struct xe_ttm_vram_mgr {
 	struct mutex lock;
 	/** @mem_type: The TTM memory type */
 	u32 mem_type;
+	/** @max_pages: max pages that can be in offline queue retrieved from FW */
+	u16 max_pages;
 };
 
 /**
-- 
2.52.0

Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram pages

[Michal Wajdeczko]

On 8/17/2026 8:51 AM, Tejas Upadhyay wrote:
> Include a sysfs interface designed to expose information about bad
> VRAM pages — those identified as having hardware faults (e.g., ECC
> errors). This interface allows userspace tools and administrators to
> monitor the health of the GPU's local memory and track the status of
> page retirement. Details on bad gpu vram pages can be found under
> /sys/bus/pci/devices/<bdf>/vram_bad_pages.

since those new files are xe driver specific, shouldn't we refer to
them using

	/sys/bus/pci/drivers/xe/<bdf>/vram...

> 
> The format is: pfn : gpu_page_size : flags

kernel documentation [1] says

	"Mixing types, expressing multiple lines of data, and doing
	fancy formatting of data is heavily frowned upon"

[1] https://docs.kernel.org/filesystems/sysfs.html#attributes

so to follow the guidelines maybe we expose the separate files:

/sys/bus/pci/drivers/xe/<bdf>/vram_page_size		u64
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count	u64
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved	u64[]
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending	u64[]
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_failed	u64[]

or

/sys/bus/pci/drivers/xe/<bdf>
|
+-- vram/
    +-- page_size	u64
    +-- bad_pages/
        +-- count	u64
        +-- reserved	u64[]
        +-- pending	u64[]
        +-- failed	u64[]

then 

/sys/bus/pci/drivers/xe/<bdf>/vram_page_size:0x1000
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count:5
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved:0x0000000000000000
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001234000
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001235000
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001236000
/sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001237000

> 
> flags:
>   R: reserved, this gpu page is reserved.
>   P: pending for reserve, this gpu page is marked as bad, will be
>      reserved in next window of page_reserve.
>   F: unable to reserve, this gpu page can't be reserved due to some
>      reasons.
> 
> For example, cat /sys/bus/pci/devices/<bdf>/vram_bad_pages:
>   max_pages : 10000
>   0x0000000000000000 : 0x0000000000001000 : R
>   0x0000000000001234 : 0x0000000000001000 : P
> 
> The sysfs binary attribute is created under the PCI device kobject
> when the platform supports it and the configfs bad_page_reservation
> policy is enabled. Uses RCU-protected list traversal so reads never
> block normal VRAM allocation operations.
> 
> Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
> ---
>  drivers/gpu/drm/xe/xe_device_sysfs.c       |   7 +
>  drivers/gpu/drm/xe/xe_ttm_vram_mgr.c       | 141 +++++++++++++++++++++
>  drivers/gpu/drm/xe/xe_ttm_vram_mgr.h       |   1 +
>  drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h |   2 +
>  4 files changed, 151 insertions(+)
> 
> diff --git a/drivers/gpu/drm/xe/xe_device_sysfs.c b/drivers/gpu/drm/xe/xe_device_sysfs.c
> index a73e0e957cb0..47c5be4180fe 100644
> --- a/drivers/gpu/drm/xe/xe_device_sysfs.c
> +++ b/drivers/gpu/drm/xe/xe_device_sysfs.c
> @@ -8,12 +8,14 @@
>  #include <linux/pci.h>
>  #include <linux/sysfs.h>
>  
> +#include "xe_configfs.h"
>  #include "xe_device.h"
>  #include "xe_device_sysfs.h"
>  #include "xe_mmio.h"
>  #include "xe_pcode_api.h"
>  #include "xe_pcode.h"
>  #include "xe_pm.h"
> +#include "xe_ttm_vram_mgr.h"
>  
>  /**
>   * DOC: Xe device sysfs
> @@ -267,6 +269,7 @@ static const struct attribute_group auto_link_downgrade_attr_group = {
>  int xe_device_sysfs_init(struct xe_device *xe)
>  {
>  	struct device *dev = xe->drm.dev;
> +	bool policy;
>  	int ret;
>  
>  	if (xe->d3cold.capable) {
> @@ -285,5 +288,9 @@ int xe_device_sysfs_init(struct xe_device *xe)
>  			return ret;
>  	}
>  
> +	policy = xe_configfs_get_bad_page_reservation(to_pci_dev(dev));
> +	if (xe->info.platform == XE_CRESCENTISLAND && policy)
> +		xe_ttm_vram_sysfs_init(xe);
> +
>  	return 0;
>  }
> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> index a48196ce303f..73cd06c22d37 100644
> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> @@ -820,3 +820,144 @@ int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr)
>  	return xe_ttm_vram_reserve_page_at_addr(xe, addr - vr->dpa_base, vram_mgr, mm);
>  }
>  EXPORT_SYMBOL(xe_ttm_vram_handle_addr_fault);
> +
> +static size_t serialize_bad_pages(struct xe_ttm_vram_mgr *mgr, char *buf, size_t max_len)
> +{
> +	struct xe_ttm_vram_offline_resource *pos;
> +	struct gpu_buddy_block *block;
> +	size_t s = 0;
> +	int printed;
> +	int count = 0;
> +
> +	rcu_read_lock();
> +
> +	printed = scnprintf(buf + s, max_len - s, "max_pages: %d\n", mgr->max_pages);
> +	s += printed;
> +
> +	list_for_each_entry_rcu(pos, &mgr->offlined_pages, offlined_link) {
> +		if (count >= 10000 || s >= max_len)
> +			break;
> +
> +		block = list_first_entry_or_null(&pos->blocks, struct gpu_buddy_block, link);
> +		if (!block)
> +			continue;
> +
> +		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx : %c\n",
> +				    gpu_buddy_block_offset(block) >> PAGE_SHIFT,
> +				    gpu_buddy_block_size(&mgr->mm, block), 'R');
> +		s += printed;
> +		count++;
> +	}
> +	list_for_each_entry_rcu(pos, &mgr->queued_pages, queued_link) {
> +		u64 pfn, blk_size;
> +
> +		if (count >= 10000 || s >= max_len)
> +			break;
> +
> +		block = list_first_entry_or_null(&pos->blocks, struct gpu_buddy_block, link);
> +		if (block) {
> +			pfn = gpu_buddy_block_offset(block) >> PAGE_SHIFT;
> +			blk_size = gpu_buddy_block_size(&mgr->mm, block);
> +		} else {
> +			pfn = pos->addr >> PAGE_SHIFT;
> +			blk_size = PAGE_SIZE;
> +		}
> +
> +		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx : %c\n",
> +				    pfn, blk_size, pos->status ? 'F' : 'P');
> +		s += printed;
> +		count++;
> +	}
> +
> +	rcu_read_unlock();
> +	return s;
> +}
> +
> +static ssize_t vram_bad_pages_bin_read(struct file *filp, struct kobject *kobj,
> +				       const struct bin_attribute *attr, char *buf,
> +				       loff_t off, size_t count)
> +{
> +	struct device *dev = kobj_to_dev(kobj);
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +	struct ttm_resource_manager *man;
> +	struct xe_ttm_vram_mgr *mgr;
> +	size_t allocation_size;
> +	struct xe_device *xe;
> +	size_t full_data_len;
> +	int active_entries;
> +	char *temp_buf;
> +
> +	xe = pdev_to_xe_device(pdev);
> +	man = ttm_manager_type(&xe->ttm, XE_PL_VRAM0);
> +	if (!man)
> +		return -ENODEV;
> +	mgr = to_xe_ttm_vram_mgr(man);
> +
> +	active_entries = READ_ONCE(mgr->n_offlined_pages) + READ_ONCE(mgr->n_queued_pages);
> +
> +	if (active_entries > 10000)
> +		active_entries = 10000;
> +
> +	allocation_size = 64 + (active_entries * 48);
> +
> +	temp_buf = kvmalloc(allocation_size, GFP_KERNEL);
> +	if (!temp_buf)
> +		return -ENOMEM;
> +
> +	/* serialize_bad_pages uses rcu_read_lock internally */
> +	full_data_len = serialize_bad_pages(mgr, temp_buf, allocation_size);
> +
> +	if (off >= full_data_len) {
> +		kvfree(temp_buf);
> +		return 0;
> +	}
> +
> +	if (off + count > full_data_len)
> +		count = full_data_len - off;
> +
> +	memcpy(buf, temp_buf + off, count);
> +
> +	kvfree(temp_buf);
> +	return count;
> +}
> +
> +static const struct bin_attribute bin_attr_vram_bad_pages = {
> +	.attr = { .name = "vram_bad_pages", .mode = 0444 },
> +	.read = vram_bad_pages_bin_read,
> +	.size = 0,
> +};
> +
> +static void xe_ttm_vram_sysfs_fini(void *arg)
> +{
> +	struct xe_device *xe = arg;
> +	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
> +
> +	sysfs_remove_bin_file(&pdev->dev.kobj, &bin_attr_vram_bad_pages);
> +}
> +
> +/**
> + * xe_ttm_vram_sysfs_init - Initialize vram bad pages sysfs binary file
> + * @xe: Xe Device object
> + *
> + * Creates a binary sysfs file under the PCI device for reading
> + * offlined and queued VRAM pages. Supports large entry counts
> + * via offset/count pagination.
> + *
> + * Returns: 0 on success, negative error code on error.
> + */
> +int xe_ttm_vram_sysfs_init(struct xe_device *xe)
> +{
> +	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
> +	int err;
> +
> +	err = sysfs_create_bin_file(&pdev->dev.kobj, &bin_attr_vram_bad_pages);
> +	if (err) {
> +		dev_err(&pdev->dev,
> +			"Failed to create vram_bad_pages sysfs: %d\n",
> +			err);
> +		return err;
> +	}
> +
> +	return devm_add_action_or_reset(&pdev->dev, xe_ttm_vram_sysfs_fini, xe);
> +}
> +EXPORT_SYMBOL(xe_ttm_vram_sysfs_init);
> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> index d5392beff30c..eb55b0f74ef3 100644
> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> @@ -32,6 +32,7 @@ void xe_ttm_vram_get_used(struct ttm_resource_manager *man,
>  			  u64 *used, u64 *used_visible);
>  
>  int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr);
> +int xe_ttm_vram_sysfs_init(struct xe_device *xe);
>  static inline struct xe_ttm_vram_mgr_resource *
>  to_xe_ttm_vram_mgr_resource(struct ttm_resource *res)
>  {
> diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> index dc97b0ad0e51..efcf3e1d4e80 100644
> --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> @@ -37,6 +37,8 @@ struct xe_ttm_vram_mgr {
>  	struct mutex lock;
>  	/** @mem_type: The TTM memory type */
>  	u32 mem_type;
> +	/** @max_pages: max pages that can be in offline queue retrieved from FW */
> +	u16 max_pages;
>  };
>  
>  /**

RE: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram pages

[Upadhyay, Tejas]

> -----Original Message-----
> From: Wajdeczko, Michal <michal.wajdeczko@intel.com>
> Sent: 17 August 2026 16:57
> To: Upadhyay, Tejas <tejas.upadhyay@intel.com>; intel-
> xe@lists.freedesktop.org; Vivi, Rodrigo <rodrigo.vivi@intel.com>; Thomas
> Hellström <thomas.hellstrom@linux.intel.com>
> Cc: Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>
> Subject: Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram
> pages
> 
> 
> 
> On 8/17/2026 8:51 AM, Tejas Upadhyay wrote:
> > Include a sysfs interface designed to expose information about bad
> > VRAM pages — those identified as having hardware faults (e.g., ECC
> > errors). This interface allows userspace tools and administrators to
> > monitor the health of the GPU's local memory and track the status of
> > page retirement. Details on bad gpu vram pages can be found under
> > /sys/bus/pci/devices/<bdf>/vram_bad_pages.
> 
> since those new files are xe driver specific, shouldn't we refer to them using
> 
> 	/sys/bus/pci/drivers/xe/<bdf>/vram...
> 
> >
> > The format is: pfn : gpu_page_size : flags
> 
> kernel documentation [1] says
> 
> 	"Mixing types, expressing multiple lines of data, and doing
> 	fancy formatting of data is heavily frowned upon"
> 
> [1] https://docs.kernel.org/filesystems/sysfs.html#attributes
> 
> so to follow the guidelines maybe we expose the separate files:
> 
> /sys/bus/pci/drivers/xe/<bdf>/vram_page_size		u64
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count	u64
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved	u64[]
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending	u64[]
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_failed	u64[]
> 
> or
> 
> /sys/bus/pci/drivers/xe/<bdf>
> |
> +-- vram/
>     +-- page_size	u64
>     +-- bad_pages/
>         +-- count	u64
>         +-- reserved	u64[]
>         +-- pending	u64[]
>         +-- failed	u64[]
> 
> then
> 
> /sys/bus/pci/drivers/xe/<bdf>/vram_page_size:0x1000
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count:5
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved:0x000000000000
> 0000
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001234
> 000
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001235
> 000
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001236
> 000
> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001237
> 000

Thanks for comment, this is documented format by design doc. Sysman also depending on this format. So I don’t see this can be done without design being changed for everyone.

Tejas
> 
> >
> > flags:
> >   R: reserved, this gpu page is reserved.
> >   P: pending for reserve, this gpu page is marked as bad, will be
> >      reserved in next window of page_reserve.
> >   F: unable to reserve, this gpu page can't be reserved due to some
> >      reasons.
> >
> > For example, cat /sys/bus/pci/devices/<bdf>/vram_bad_pages:
> >   max_pages : 10000
> >   0x0000000000000000 : 0x0000000000001000 : R
> >   0x0000000000001234 : 0x0000000000001000 : P
> >
> > The sysfs binary attribute is created under the PCI device kobject
> > when the platform supports it and the configfs bad_page_reservation
> > policy is enabled. Uses RCU-protected list traversal so reads never
> > block normal VRAM allocation operations.
> >
> > Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
> > ---
> >  drivers/gpu/drm/xe/xe_device_sysfs.c       |   7 +
> >  drivers/gpu/drm/xe/xe_ttm_vram_mgr.c       | 141
> +++++++++++++++++++++
> >  drivers/gpu/drm/xe/xe_ttm_vram_mgr.h       |   1 +
> >  drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h |   2 +
> >  4 files changed, 151 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/xe/xe_device_sysfs.c
> > b/drivers/gpu/drm/xe/xe_device_sysfs.c
> > index a73e0e957cb0..47c5be4180fe 100644
> > --- a/drivers/gpu/drm/xe/xe_device_sysfs.c
> > +++ b/drivers/gpu/drm/xe/xe_device_sysfs.c
> > @@ -8,12 +8,14 @@
> >  #include <linux/pci.h>
> >  #include <linux/sysfs.h>
> >
> > +#include "xe_configfs.h"
> >  #include "xe_device.h"
> >  #include "xe_device_sysfs.h"
> >  #include "xe_mmio.h"
> >  #include "xe_pcode_api.h"
> >  #include "xe_pcode.h"
> >  #include "xe_pm.h"
> > +#include "xe_ttm_vram_mgr.h"
> >
> >  /**
> >   * DOC: Xe device sysfs
> > @@ -267,6 +269,7 @@ static const struct attribute_group
> > auto_link_downgrade_attr_group = {  int xe_device_sysfs_init(struct
> > xe_device *xe)  {
> >  	struct device *dev = xe->drm.dev;
> > +	bool policy;
> >  	int ret;
> >
> >  	if (xe->d3cold.capable) {
> > @@ -285,5 +288,9 @@ int xe_device_sysfs_init(struct xe_device *xe)
> >  			return ret;
> >  	}
> >
> > +	policy = xe_configfs_get_bad_page_reservation(to_pci_dev(dev));
> > +	if (xe->info.platform == XE_CRESCENTISLAND && policy)
> > +		xe_ttm_vram_sysfs_init(xe);
> > +
> >  	return 0;
> >  }
> > diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > index a48196ce303f..73cd06c22d37 100644
> > --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > @@ -820,3 +820,144 @@ int xe_ttm_vram_handle_addr_fault(struct
> xe_device *xe, u64 addr)
> >  	return xe_ttm_vram_reserve_page_at_addr(xe, addr - vr->dpa_base,
> > vram_mgr, mm);  }  EXPORT_SYMBOL(xe_ttm_vram_handle_addr_fault);
> > +
> > +static size_t serialize_bad_pages(struct xe_ttm_vram_mgr *mgr, char
> > +*buf, size_t max_len) {
> > +	struct xe_ttm_vram_offline_resource *pos;
> > +	struct gpu_buddy_block *block;
> > +	size_t s = 0;
> > +	int printed;
> > +	int count = 0;
> > +
> > +	rcu_read_lock();
> > +
> > +	printed = scnprintf(buf + s, max_len - s, "max_pages: %d\n", mgr-
> >max_pages);
> > +	s += printed;
> > +
> > +	list_for_each_entry_rcu(pos, &mgr->offlined_pages, offlined_link) {
> > +		if (count >= 10000 || s >= max_len)
> > +			break;
> > +
> > +		block = list_first_entry_or_null(&pos->blocks, struct
> gpu_buddy_block, link);
> > +		if (!block)
> > +			continue;
> > +
> > +		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx
> : %c\n",
> > +				    gpu_buddy_block_offset(block) >>
> PAGE_SHIFT,
> > +				    gpu_buddy_block_size(&mgr->mm, block),
> 'R');
> > +		s += printed;
> > +		count++;
> > +	}
> > +	list_for_each_entry_rcu(pos, &mgr->queued_pages, queued_link) {
> > +		u64 pfn, blk_size;
> > +
> > +		if (count >= 10000 || s >= max_len)
> > +			break;
> > +
> > +		block = list_first_entry_or_null(&pos->blocks, struct
> gpu_buddy_block, link);
> > +		if (block) {
> > +			pfn = gpu_buddy_block_offset(block) >> PAGE_SHIFT;
> > +			blk_size = gpu_buddy_block_size(&mgr->mm, block);
> > +		} else {
> > +			pfn = pos->addr >> PAGE_SHIFT;
> > +			blk_size = PAGE_SIZE;
> > +		}
> > +
> > +		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx
> : %c\n",
> > +				    pfn, blk_size, pos->status ? 'F' : 'P');
> > +		s += printed;
> > +		count++;
> > +	}
> > +
> > +	rcu_read_unlock();
> > +	return s;
> > +}
> > +
> > +static ssize_t vram_bad_pages_bin_read(struct file *filp, struct kobject
> *kobj,
> > +				       const struct bin_attribute *attr, char *buf,
> > +				       loff_t off, size_t count)
> > +{
> > +	struct device *dev = kobj_to_dev(kobj);
> > +	struct pci_dev *pdev = to_pci_dev(dev);
> > +	struct ttm_resource_manager *man;
> > +	struct xe_ttm_vram_mgr *mgr;
> > +	size_t allocation_size;
> > +	struct xe_device *xe;
> > +	size_t full_data_len;
> > +	int active_entries;
> > +	char *temp_buf;
> > +
> > +	xe = pdev_to_xe_device(pdev);
> > +	man = ttm_manager_type(&xe->ttm, XE_PL_VRAM0);
> > +	if (!man)
> > +		return -ENODEV;
> > +	mgr = to_xe_ttm_vram_mgr(man);
> > +
> > +	active_entries = READ_ONCE(mgr->n_offlined_pages) +
> > +READ_ONCE(mgr->n_queued_pages);
> > +
> > +	if (active_entries > 10000)
> > +		active_entries = 10000;
> > +
> > +	allocation_size = 64 + (active_entries * 48);
> > +
> > +	temp_buf = kvmalloc(allocation_size, GFP_KERNEL);
> > +	if (!temp_buf)
> > +		return -ENOMEM;
> > +
> > +	/* serialize_bad_pages uses rcu_read_lock internally */
> > +	full_data_len = serialize_bad_pages(mgr, temp_buf, allocation_size);
> > +
> > +	if (off >= full_data_len) {
> > +		kvfree(temp_buf);
> > +		return 0;
> > +	}
> > +
> > +	if (off + count > full_data_len)
> > +		count = full_data_len - off;
> > +
> > +	memcpy(buf, temp_buf + off, count);
> > +
> > +	kvfree(temp_buf);
> > +	return count;
> > +}
> > +
> > +static const struct bin_attribute bin_attr_vram_bad_pages = {
> > +	.attr = { .name = "vram_bad_pages", .mode = 0444 },
> > +	.read = vram_bad_pages_bin_read,
> > +	.size = 0,
> > +};
> > +
> > +static void xe_ttm_vram_sysfs_fini(void *arg) {
> > +	struct xe_device *xe = arg;
> > +	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
> > +
> > +	sysfs_remove_bin_file(&pdev->dev.kobj,
> &bin_attr_vram_bad_pages); }
> > +
> > +/**
> > + * xe_ttm_vram_sysfs_init - Initialize vram bad pages sysfs binary
> > +file
> > + * @xe: Xe Device object
> > + *
> > + * Creates a binary sysfs file under the PCI device for reading
> > + * offlined and queued VRAM pages. Supports large entry counts
> > + * via offset/count pagination.
> > + *
> > + * Returns: 0 on success, negative error code on error.
> > + */
> > +int xe_ttm_vram_sysfs_init(struct xe_device *xe) {
> > +	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
> > +	int err;
> > +
> > +	err = sysfs_create_bin_file(&pdev->dev.kobj,
> &bin_attr_vram_bad_pages);
> > +	if (err) {
> > +		dev_err(&pdev->dev,
> > +			"Failed to create vram_bad_pages sysfs: %d\n",
> > +			err);
> > +		return err;
> > +	}
> > +
> > +	return devm_add_action_or_reset(&pdev->dev,
> xe_ttm_vram_sysfs_fini,
> > +xe); } EXPORT_SYMBOL(xe_ttm_vram_sysfs_init);
> > diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > index d5392beff30c..eb55b0f74ef3 100644
> > --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > @@ -32,6 +32,7 @@ void xe_ttm_vram_get_used(struct
> ttm_resource_manager *man,
> >  			  u64 *used, u64 *used_visible);
> >
> >  int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr);
> > +int xe_ttm_vram_sysfs_init(struct xe_device *xe);
> >  static inline struct xe_ttm_vram_mgr_resource *
> > to_xe_ttm_vram_mgr_resource(struct ttm_resource *res)  { diff --git
> > a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > index dc97b0ad0e51..efcf3e1d4e80 100644
> > --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > @@ -37,6 +37,8 @@ struct xe_ttm_vram_mgr {
> >  	struct mutex lock;
> >  	/** @mem_type: The TTM memory type */
> >  	u32 mem_type;
> > +	/** @max_pages: max pages that can be in offline queue retrieved
> from FW */
> > +	u16 max_pages;
> >  };
> >
> >  /**

Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram pages

[Rodrigo Vivi]

On Mon, Aug 17, 2026 at 02:58:31PM +0000, Upadhyay, Tejas wrote:
> 
> 
> > -----Original Message-----
> > From: Wajdeczko, Michal <michal.wajdeczko@intel.com>
> > Sent: 17 August 2026 16:57
> > To: Upadhyay, Tejas <tejas.upadhyay@intel.com>; intel-
> > xe@lists.freedesktop.org; Vivi, Rodrigo <rodrigo.vivi@intel.com>; Thomas
> > Hellström <thomas.hellstrom@linux.intel.com>
> > Cc: Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>
> > Subject: Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram
> > pages
> > 
> > 
> > 
> > On 8/17/2026 8:51 AM, Tejas Upadhyay wrote:
> > > Include a sysfs interface designed to expose information about bad
> > > VRAM pages — those identified as having hardware faults (e.g., ECC
> > > errors). This interface allows userspace tools and administrators to
> > > monitor the health of the GPU's local memory and track the status of
> > > page retirement. Details on bad gpu vram pages can be found under
> > > /sys/bus/pci/devices/<bdf>/vram_bad_pages.
> > 
> > since those new files are xe driver specific, shouldn't we refer to them using
> > 
> > 	/sys/bus/pci/drivers/xe/<bdf>/vram...
> > 
> > >
> > > The format is: pfn : gpu_page_size : flags
> > 
> > kernel documentation [1] says
> > 
> > 	"Mixing types, expressing multiple lines of data, and doing
> > 	fancy formatting of data is heavily frowned upon"
> > 
> > [1] https://docs.kernel.org/filesystems/sysfs.html#attributes
> > 
> > so to follow the guidelines maybe we expose the separate files:
> > 
> > /sys/bus/pci/drivers/xe/<bdf>/vram_page_size		u64
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count	u64
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved	u64[]
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending	u64[]
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_failed	u64[]
> > 
> > or
> > 
> > /sys/bus/pci/drivers/xe/<bdf>
> > |
> > +-- vram/
> >     +-- page_size	u64
> >     +-- bad_pages/
> >         +-- count	u64
> >         +-- reserved	u64[]
> >         +-- pending	u64[]
> >         +-- failed	u64[]
> > 
> > then
> > 
> > /sys/bus/pci/drivers/xe/<bdf>/vram_page_size:0x1000
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count:5
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved:0x000000000000
> > 0000
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001234
> > 000
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001235
> > 000
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001236
> > 000
> > /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001237
> > 000
> 
> Thanks for comment, this is documented format by design doc. Sysman also depending on this format. So I don’t see this can be done without design being changed for everyone.

Internal design docs don't superseed upstream documentation.
It is the other way around.

But also, the files will be there one way or another. Both paths
are valid, so I don't believe that change in here force changes
in the userspace. Although, yes consistency is good...

That said, I don't have a strong feeling for one way or the other.

Since we are adding to the device level anyway, I believe it should
be okay. But Michal, do you know any doc or any precedence that kind
of force us to go the other way?

> 
> Tejas
> > 
> > >
> > > flags:
> > >   R: reserved, this gpu page is reserved.
> > >   P: pending for reserve, this gpu page is marked as bad, will be
> > >      reserved in next window of page_reserve.
> > >   F: unable to reserve, this gpu page can't be reserved due to some
> > >      reasons.
> > >
> > > For example, cat /sys/bus/pci/devices/<bdf>/vram_bad_pages:
> > >   max_pages : 10000
> > >   0x0000000000000000 : 0x0000000000001000 : R
> > >   0x0000000000001234 : 0x0000000000001000 : P
> > >
> > > The sysfs binary attribute is created under the PCI device kobject
> > > when the platform supports it and the configfs bad_page_reservation
> > > policy is enabled. Uses RCU-protected list traversal so reads never
> > > block normal VRAM allocation operations.
> > >
> > > Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
> > > ---
> > >  drivers/gpu/drm/xe/xe_device_sysfs.c       |   7 +
> > >  drivers/gpu/drm/xe/xe_ttm_vram_mgr.c       | 141
> > +++++++++++++++++++++
> > >  drivers/gpu/drm/xe/xe_ttm_vram_mgr.h       |   1 +
> > >  drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h |   2 +
> > >  4 files changed, 151 insertions(+)
> > >
> > > diff --git a/drivers/gpu/drm/xe/xe_device_sysfs.c
> > > b/drivers/gpu/drm/xe/xe_device_sysfs.c
> > > index a73e0e957cb0..47c5be4180fe 100644
> > > --- a/drivers/gpu/drm/xe/xe_device_sysfs.c
> > > +++ b/drivers/gpu/drm/xe/xe_device_sysfs.c
> > > @@ -8,12 +8,14 @@
> > >  #include <linux/pci.h>
> > >  #include <linux/sysfs.h>
> > >
> > > +#include "xe_configfs.h"
> > >  #include "xe_device.h"
> > >  #include "xe_device_sysfs.h"
> > >  #include "xe_mmio.h"
> > >  #include "xe_pcode_api.h"
> > >  #include "xe_pcode.h"
> > >  #include "xe_pm.h"
> > > +#include "xe_ttm_vram_mgr.h"
> > >
> > >  /**
> > >   * DOC: Xe device sysfs
> > > @@ -267,6 +269,7 @@ static const struct attribute_group
> > > auto_link_downgrade_attr_group = {  int xe_device_sysfs_init(struct
> > > xe_device *xe)  {
> > >  	struct device *dev = xe->drm.dev;
> > > +	bool policy;
> > >  	int ret;
> > >
> > >  	if (xe->d3cold.capable) {
> > > @@ -285,5 +288,9 @@ int xe_device_sysfs_init(struct xe_device *xe)
> > >  			return ret;
> > >  	}
> > >
> > > +	policy = xe_configfs_get_bad_page_reservation(to_pci_dev(dev));
> > > +	if (xe->info.platform == XE_CRESCENTISLAND && policy)
> > > +		xe_ttm_vram_sysfs_init(xe);
> > > +
> > >  	return 0;
> > >  }
> > > diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > > b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > > index a48196ce303f..73cd06c22d37 100644
> > > --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > > +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
> > > @@ -820,3 +820,144 @@ int xe_ttm_vram_handle_addr_fault(struct
> > xe_device *xe, u64 addr)
> > >  	return xe_ttm_vram_reserve_page_at_addr(xe, addr - vr->dpa_base,
> > > vram_mgr, mm);  }  EXPORT_SYMBOL(xe_ttm_vram_handle_addr_fault);
> > > +
> > > +static size_t serialize_bad_pages(struct xe_ttm_vram_mgr *mgr, char
> > > +*buf, size_t max_len) {
> > > +	struct xe_ttm_vram_offline_resource *pos;
> > > +	struct gpu_buddy_block *block;
> > > +	size_t s = 0;
> > > +	int printed;
> > > +	int count = 0;
> > > +
> > > +	rcu_read_lock();
> > > +
> > > +	printed = scnprintf(buf + s, max_len - s, "max_pages: %d\n", mgr-
> > >max_pages);
> > > +	s += printed;
> > > +
> > > +	list_for_each_entry_rcu(pos, &mgr->offlined_pages, offlined_link) {
> > > +		if (count >= 10000 || s >= max_len)
> > > +			break;
> > > +
> > > +		block = list_first_entry_or_null(&pos->blocks, struct
> > gpu_buddy_block, link);
> > > +		if (!block)
> > > +			continue;
> > > +
> > > +		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx
> > : %c\n",
> > > +				    gpu_buddy_block_offset(block) >>
> > PAGE_SHIFT,
> > > +				    gpu_buddy_block_size(&mgr->mm, block),
> > 'R');
> > > +		s += printed;
> > > +		count++;
> > > +	}
> > > +	list_for_each_entry_rcu(pos, &mgr->queued_pages, queued_link) {
> > > +		u64 pfn, blk_size;
> > > +
> > > +		if (count >= 10000 || s >= max_len)
> > > +			break;
> > > +
> > > +		block = list_first_entry_or_null(&pos->blocks, struct
> > gpu_buddy_block, link);
> > > +		if (block) {
> > > +			pfn = gpu_buddy_block_offset(block) >> PAGE_SHIFT;
> > > +			blk_size = gpu_buddy_block_size(&mgr->mm, block);
> > > +		} else {
> > > +			pfn = pos->addr >> PAGE_SHIFT;
> > > +			blk_size = PAGE_SIZE;
> > > +		}
> > > +
> > > +		printed = scnprintf(buf + s, max_len - s, "0x%016llx : 0x%016llx
> > : %c\n",
> > > +				    pfn, blk_size, pos->status ? 'F' : 'P');
> > > +		s += printed;
> > > +		count++;
> > > +	}
> > > +
> > > +	rcu_read_unlock();
> > > +	return s;
> > > +}
> > > +
> > > +static ssize_t vram_bad_pages_bin_read(struct file *filp, struct kobject
> > *kobj,
> > > +				       const struct bin_attribute *attr, char *buf,
> > > +				       loff_t off, size_t count)
> > > +{
> > > +	struct device *dev = kobj_to_dev(kobj);
> > > +	struct pci_dev *pdev = to_pci_dev(dev);
> > > +	struct ttm_resource_manager *man;
> > > +	struct xe_ttm_vram_mgr *mgr;
> > > +	size_t allocation_size;
> > > +	struct xe_device *xe;
> > > +	size_t full_data_len;
> > > +	int active_entries;
> > > +	char *temp_buf;
> > > +
> > > +	xe = pdev_to_xe_device(pdev);
> > > +	man = ttm_manager_type(&xe->ttm, XE_PL_VRAM0);
> > > +	if (!man)
> > > +		return -ENODEV;
> > > +	mgr = to_xe_ttm_vram_mgr(man);
> > > +
> > > +	active_entries = READ_ONCE(mgr->n_offlined_pages) +
> > > +READ_ONCE(mgr->n_queued_pages);
> > > +
> > > +	if (active_entries > 10000)
> > > +		active_entries = 10000;
> > > +
> > > +	allocation_size = 64 + (active_entries * 48);
> > > +
> > > +	temp_buf = kvmalloc(allocation_size, GFP_KERNEL);
> > > +	if (!temp_buf)
> > > +		return -ENOMEM;
> > > +
> > > +	/* serialize_bad_pages uses rcu_read_lock internally */
> > > +	full_data_len = serialize_bad_pages(mgr, temp_buf, allocation_size);
> > > +
> > > +	if (off >= full_data_len) {
> > > +		kvfree(temp_buf);
> > > +		return 0;
> > > +	}
> > > +
> > > +	if (off + count > full_data_len)
> > > +		count = full_data_len - off;
> > > +
> > > +	memcpy(buf, temp_buf + off, count);
> > > +
> > > +	kvfree(temp_buf);
> > > +	return count;
> > > +}
> > > +
> > > +static const struct bin_attribute bin_attr_vram_bad_pages = {
> > > +	.attr = { .name = "vram_bad_pages", .mode = 0444 },
> > > +	.read = vram_bad_pages_bin_read,
> > > +	.size = 0,
> > > +};
> > > +
> > > +static void xe_ttm_vram_sysfs_fini(void *arg) {
> > > +	struct xe_device *xe = arg;
> > > +	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
> > > +
> > > +	sysfs_remove_bin_file(&pdev->dev.kobj,
> > &bin_attr_vram_bad_pages); }
> > > +
> > > +/**
> > > + * xe_ttm_vram_sysfs_init - Initialize vram bad pages sysfs binary
> > > +file
> > > + * @xe: Xe Device object
> > > + *
> > > + * Creates a binary sysfs file under the PCI device for reading
> > > + * offlined and queued VRAM pages. Supports large entry counts
> > > + * via offset/count pagination.
> > > + *
> > > + * Returns: 0 on success, negative error code on error.
> > > + */
> > > +int xe_ttm_vram_sysfs_init(struct xe_device *xe) {
> > > +	struct pci_dev *pdev = to_pci_dev(xe->drm.dev);
> > > +	int err;
> > > +
> > > +	err = sysfs_create_bin_file(&pdev->dev.kobj,
> > &bin_attr_vram_bad_pages);
> > > +	if (err) {
> > > +		dev_err(&pdev->dev,
> > > +			"Failed to create vram_bad_pages sysfs: %d\n",
> > > +			err);
> > > +		return err;
> > > +	}
> > > +
> > > +	return devm_add_action_or_reset(&pdev->dev,
> > xe_ttm_vram_sysfs_fini,
> > > +xe); } EXPORT_SYMBOL(xe_ttm_vram_sysfs_init);
> > > diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > > b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > > index d5392beff30c..eb55b0f74ef3 100644
> > > --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > > +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
> > > @@ -32,6 +32,7 @@ void xe_ttm_vram_get_used(struct
> > ttm_resource_manager *man,
> > >  			  u64 *used, u64 *used_visible);
> > >
> > >  int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr);
> > > +int xe_ttm_vram_sysfs_init(struct xe_device *xe);
> > >  static inline struct xe_ttm_vram_mgr_resource *
> > > to_xe_ttm_vram_mgr_resource(struct ttm_resource *res)  { diff --git
> > > a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > > b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > > index dc97b0ad0e51..efcf3e1d4e80 100644
> > > --- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > > +++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr_types.h
> > > @@ -37,6 +37,8 @@ struct xe_ttm_vram_mgr {
> > >  	struct mutex lock;
> > >  	/** @mem_type: The TTM memory type */
> > >  	u32 mem_type;
> > > +	/** @max_pages: max pages that can be in offline queue retrieved
> > from FW */
> > > +	u16 max_pages;
> > >  };
> > >
> > >  /**
> 

Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram pages

[Michal Wajdeczko]

On 8/17/2026 6:06 PM, Rodrigo Vivi wrote:
> On Mon, Aug 17, 2026 at 02:58:31PM +0000, Upadhyay, Tejas wrote:
>>
>>
>>> -----Original Message-----
>>> From: Wajdeczko, Michal <michal.wajdeczko@intel.com>
>>> Sent: 17 August 2026 16:57
>>> To: Upadhyay, Tejas <tejas.upadhyay@intel.com>; intel-
>>> xe@lists.freedesktop.org; Vivi, Rodrigo <rodrigo.vivi@intel.com>; Thomas
>>> Hellström <thomas.hellstrom@linux.intel.com>
>>> Cc: Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>
>>> Subject: Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram
>>> pages
>>>
>>>
>>>
>>> On 8/17/2026 8:51 AM, Tejas Upadhyay wrote:
>>>> Include a sysfs interface designed to expose information about bad
>>>> VRAM pages — those identified as having hardware faults (e.g., ECC
>>>> errors). This interface allows userspace tools and administrators to
>>>> monitor the health of the GPU's local memory and track the status of
>>>> page retirement. Details on bad gpu vram pages can be found under
>>>> /sys/bus/pci/devices/<bdf>/vram_bad_pages.
>>>
>>> since those new files are xe driver specific, shouldn't we refer to them using
>>>
>>> 	/sys/bus/pci/drivers/xe/<bdf>/vram...
>>>
>>>>
>>>> The format is: pfn : gpu_page_size : flags
>>>
>>> kernel documentation [1] says
>>>
>>> 	"Mixing types, expressing multiple lines of data, and doing
>>> 	fancy formatting of data is heavily frowned upon"
>>>
>>> [1] https://docs.kernel.org/filesystems/sysfs.html#attributes
>>>
>>> so to follow the guidelines maybe we expose the separate files:
>>>
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_page_size		u64
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count	u64
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved	u64[]
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending	u64[]
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_failed	u64[]
>>>
>>> or
>>>
>>> /sys/bus/pci/drivers/xe/<bdf>
>>> |
>>> +-- vram/
>>>     +-- page_size	u64
>>>     +-- bad_pages/
>>>         +-- count	u64
>>>         +-- reserved	u64[]
>>>         +-- pending	u64[]
>>>         +-- failed	u64[]
>>>
>>> then
>>>
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_page_size:0x1000
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count:5
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved:0x000000000000
>>> 0000
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001234
>>> 000
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001235
>>> 000
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001236
>>> 000
>>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001237
>>> 000
>>
>> Thanks for comment, this is documented format by design doc. Sysman also depending on this format. So I don’t see this can be done without design being changed for everyone.
> 
> Internal design docs don't superseed upstream documentation.
> It is the other way around.
> 
> But also, the files will be there one way or another. Both paths
> are valid, so I don't believe that change in here force changes
> in the userspace. Although, yes consistency is good...
> 
> That said, I don't have a strong feeling for one way or the other.
> 
> Since we are adding to the device level anyway, I believe it should
> be okay. But Michal, do you know any doc or any precedence that kind
> of force us to go the other way?

hmm, are we talking here about the attribute format or folder layout?

if about the latter, no strong feeling either ("files will be there
one way or another")

but if about the former, then the same documentation [1] earlier says:

	"Attributes should be ASCII text files, preferably with only
	"one value per file. It is noted that it may not be efficient
	"to contain only one value per file, so it is socially acceptable
	"to express an array of values of the same type.

and my proposal with separate files meets that expectations (there will
be either single value in the file or array of values of the same type),
opposed to original idea of array of offset:page_size:flag tuples

> 
>>
>> Tejas
>>>
>>>>
>>>> flags:
>>>>   R: reserved, this gpu page is reserved.
>>>>   P: pending for reserve, this gpu page is marked as bad, will be
>>>>      reserved in next window of page_reserve.
>>>>   F: unable to reserve, this gpu page can't be reserved due to some
>>>>      reasons.
>>>>
>>>> For example, cat /sys/bus/pci/devices/<bdf>/vram_bad_pages:
>>>>   max_pages : 10000
>>>>   0x0000000000000000 : 0x0000000000001000 : R
>>>>   0x0000000000001234 : 0x0000000000001000 : P
>>>>
>>>> The sysfs binary attribute is created under the PCI device kobject
>>>> when the platform supports it and the configfs bad_page_reservation
>>>> policy is enabled. Uses RCU-protected list traversal so reads never
>>>> block normal VRAM allocation operations.
>>>>

Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram pages

[Rodrigo Vivi]

On Mon, Aug 17, 2026 at 07:09:46PM +0200, Michal Wajdeczko wrote:
> 
> 
> On 8/17/2026 6:06 PM, Rodrigo Vivi wrote:
> > On Mon, Aug 17, 2026 at 02:58:31PM +0000, Upadhyay, Tejas wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: Wajdeczko, Michal <michal.wajdeczko@intel.com>
> >>> Sent: 17 August 2026 16:57
> >>> To: Upadhyay, Tejas <tejas.upadhyay@intel.com>; intel-
> >>> xe@lists.freedesktop.org; Vivi, Rodrigo <rodrigo.vivi@intel.com>; Thomas
> >>> Hellström <thomas.hellstrom@linux.intel.com>
> >>> Cc: Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>
> >>> Subject: Re: [PATCH V16 10/12] drm/xe: Add sysfs interface for bad gpu vram
> >>> pages
> >>>
> >>>
> >>>
> >>> On 8/17/2026 8:51 AM, Tejas Upadhyay wrote:
> >>>> Include a sysfs interface designed to expose information about bad
> >>>> VRAM pages — those identified as having hardware faults (e.g., ECC
> >>>> errors). This interface allows userspace tools and administrators to
> >>>> monitor the health of the GPU's local memory and track the status of
> >>>> page retirement. Details on bad gpu vram pages can be found under
> >>>> /sys/bus/pci/devices/<bdf>/vram_bad_pages.
> >>>
> >>> since those new files are xe driver specific, shouldn't we refer to them using
> >>>
> >>> 	/sys/bus/pci/drivers/xe/<bdf>/vram...
> >>>
> >>>>
> >>>> The format is: pfn : gpu_page_size : flags
> >>>
> >>> kernel documentation [1] says
> >>>
> >>> 	"Mixing types, expressing multiple lines of data, and doing
> >>> 	fancy formatting of data is heavily frowned upon"
> >>>
> >>> [1] https://docs.kernel.org/filesystems/sysfs.html#attributes
> >>>
> >>> so to follow the guidelines maybe we expose the separate files:
> >>>
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_page_size		u64
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count	u64
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved	u64[]
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending	u64[]
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_failed	u64[]
> >>>
> >>> or
> >>>
> >>> /sys/bus/pci/drivers/xe/<bdf>
> >>> |
> >>> +-- vram/
> >>>     +-- page_size	u64
> >>>     +-- bad_pages/
> >>>         +-- count	u64
> >>>         +-- reserved	u64[]
> >>>         +-- pending	u64[]
> >>>         +-- failed	u64[]
> >>>
> >>> then
> >>>
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_page_size:0x1000
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_count:5
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_reserved:0x000000000000
> >>> 0000
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001234
> >>> 000
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001235
> >>> 000
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001236
> >>> 000
> >>> /sys/bus/pci/drivers/xe/<bdf>/vram_bad_pages_pending:0x0000000001237
> >>> 000
> >>
> >> Thanks for comment, this is documented format by design doc. Sysman also depending on this format. So I don’t see this can be done without design being changed for everyone.
> > 
> > Internal design docs don't superseed upstream documentation.
> > It is the other way around.
> > 
> > But also, the files will be there one way or another. Both paths
> > are valid, so I don't believe that change in here force changes
> > in the userspace. Although, yes consistency is good...
> > 
> > That said, I don't have a strong feeling for one way or the other.
> > 
> > Since we are adding to the device level anyway, I believe it should
> > be okay. But Michal, do you know any doc or any precedence that kind
> > of force us to go the other way?
> 
> hmm, are we talking here about the attribute format or folder layout?
> 
> if about the latter, no strong feeling either ("files will be there
> one way or another")
> 
> but if about the former, then the same documentation [1] earlier says:
> 
> 	"Attributes should be ASCII text files, preferably with only
> 	"one value per file. It is noted that it may not be efficient
> 	"to contain only one value per file, so it is socially acceptable
> 	"to express an array of values of the same type.
> 
> and my proposal with separate files meets that expectations (there will
> be either single value in the file or array of values of the same type),
> opposed to original idea of array of offset:page_size:flag tuples

doh! I'm sorry... my comment was purely driven by the other sentence above:
"since those new files are xe driver specific, shouldn't we refer to them using"

But now I looked at the content o the patch itself. This patch as is is
a BIG NO! It is against the sysfs rules. Period. Internal spec and other
components need to adjust.

Also please do not repeat the same PVC mistakes with tenths of lingering
sysfs entries. Organize this per directory as Michal told.

Another thing, make a design that is future ready, use 'vram0/' as the name
of the directory with vram0 stuff. Like we have freq0/ for instance.

Perhaps even

+-- vram0/
  +-- pages/
     +-- size u64
     +-- bad_pages/
         +-- count u64
         +-- reserved      u64[]
         +-- pending       u64[]
         +-- failed        u64[]

Thanks,
Rodrigo.

> 
> > 
> >>
> >> Tejas
> >>>
> >>>>
> >>>> flags:
> >>>>   R: reserved, this gpu page is reserved.
> >>>>   P: pending for reserve, this gpu page is marked as bad, will be
> >>>>      reserved in next window of page_reserve.
> >>>>   F: unable to reserve, this gpu page can't be reserved due to some
> >>>>      reasons.
> >>>>
> >>>> For example, cat /sys/bus/pci/devices/<bdf>/vram_bad_pages:
> >>>>   max_pages : 10000
> >>>>   0x0000000000000000 : 0x0000000000001000 : R
> >>>>   0x0000000000001234 : 0x0000000000001000 : P
> >>>>
> >>>> The sysfs binary attribute is created under the PCI device kobject
> >>>> when the platform supports it and the configfs bad_page_reservation
> >>>> policy is enabled. Uses RCU-protected list traversal so reads never
> >>>> block normal VRAM allocation operations.
> >>>>

[PATCH V16 11/12] drm/xe/uapi: Expose ban reason in EXEC_QUEUE_GET_PROPERTY_BAN

[Tejas Upadhyay]

Extend DRM_XE_EXEC_QUEUE_GET_PROPERTY_BAN to return a bitmask indicating
the reason for the ban, rather than a simple boolean. This allows
userspace to distinguish between different ban causes:

- DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG (bit 0): exec queue was banned
  due to a GPU hang or job timeout detected by the TDR.
- DRM_XE_EXEC_QUEUE_BAN_REASON_PAGE_OFFLINE (bit 1): exec queue was
  banned because a VRAM page backing its resources was taken offline.

The ban_reason field is added to struct xe_exec_queue and set at the
point where the ban is triggered:
- In guc_exec_queue_timedout_job() for GPU hang.
- In xe_ttm_vram_purge_page() for memory page offline, before calling
  xe_exec_queue_kill() or xe_vm_kill().

The reset_status op is updated to return u64 with the reason bitmask.
When a queue is banned but no explicit reason was recorded (e.g., from a
generic CAT error), it defaults to GPU_HANG for backward compatibility.
A value of 0 means the exec queue is not banned.

v3(Rodrigo):
- Add doc in xe_drm.h
v2(Sashiko):
- Use atomic_t for ban_reason to fix concurrent updates from TDR and
  page-offline
- Guard GPU_HANG bit with !exec_queue_killed to avoid masking
  page-offline reason
- Clear ban_reason on queue recovery (clear_exec_queue_banned path)
- Use atomic_read in guc_exec_queue_reset_status for lockless read

Assisted-by: Copilot:claude-opus-4.6
Acked-by: José Roberto de Souza <jose.souza@intel.com>
Acked-by: Michal Mrozek <michal.mrozek@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_exec_queue_types.h |  7 ++++--
 drivers/gpu/drm/xe/xe_execlist.c         |  4 +--
 drivers/gpu/drm/xe/xe_guc_submit.c       | 32 ++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c     |  8 ++++++
 include/uapi/drm/xe_drm.h                | 18 ++++++++++++-
 5 files changed, 59 insertions(+), 10 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_exec_queue_types.h b/drivers/gpu/drm/xe/xe_exec_queue_types.h
index b2276559c2f6..a21916359e2f 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue_types.h
+++ b/drivers/gpu/drm/xe/xe_exec_queue_types.h
@@ -156,6 +156,9 @@ struct xe_exec_queue {
 	 */
 	unsigned long flags;
 
+	/** @ban_reason: Bitmask of ban reasons (DRM_XE_EXEC_QUEUE_BAN_REASON_*) */
+	atomic_t ban_reason;
+
 	union {
 		/** @multi_gt_list: list head for VM bind engines if multi-GT */
 		struct list_head multi_gt_list;
@@ -350,8 +353,8 @@ struct xe_exec_queue_ops {
 	 * signalled when this function is called.
 	 */
 	void (*resume)(struct xe_exec_queue *q);
-	/** @reset_status: check exec queue reset status */
-	bool (*reset_status)(struct xe_exec_queue *q);
+	/** @reset_status: check exec queue ban status, returns ban reason bitmask */
+	u64 (*reset_status)(struct xe_exec_queue *q);
 };
 
 #endif
diff --git a/drivers/gpu/drm/xe/xe_execlist.c b/drivers/gpu/drm/xe/xe_execlist.c
index cc33ae80e8cf..534e7c4e0099 100644
--- a/drivers/gpu/drm/xe/xe_execlist.c
+++ b/drivers/gpu/drm/xe/xe_execlist.c
@@ -452,10 +452,10 @@ static void execlist_exec_queue_resume(struct xe_exec_queue *q)
 	/* NIY */
 }
 
-static bool execlist_exec_queue_reset_status(struct xe_exec_queue *q)
+static u64 execlist_exec_queue_reset_status(struct xe_exec_queue *q)
 {
 	/* NIY */
-	return false;
+	return 0;
 }
 
 static const struct xe_exec_queue_ops execlist_exec_queue_ops = {
diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
index 9036f89dff7d..f52577de1ac2 100644
--- a/drivers/gpu/drm/xe/xe_guc_submit.c
+++ b/drivers/gpu/drm/xe/xe_guc_submit.c
@@ -6,6 +6,7 @@
 #include "xe_guc_submit.h"
 
 #include <linux/bitfield.h>
+#include <uapi/drm/xe_drm.h>
 #include <linux/bitmap.h>
 #include <linux/circ_buf.h>
 #include <linux/dma-fence-array.h>
@@ -1593,6 +1594,12 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
 	if (!exec_queue_killed(q))
 		wedged = guc_submit_hint_wedged(exec_queue_to_guc(q));
 
+	/*
+	 * Only tag as GPU hang if this is the original timeout, not a
+	 * consequence of a prior kill (e.g., page-offline).
+	 */
+	if (!exec_queue_killed(q))
+		atomic_or(DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG, &q->ban_reason);
 	set_exec_queue_banned(q);
 
 	/* Kick job / queue off hardware */
@@ -1676,6 +1683,9 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
 		if (timeout_needs_gt_reset(q, job, skip_timeout_check)) {
 			if (!xe_sched_invalidate_job(job, 2)) {
 				clear_exec_queue_banned(q);
+				/* protect concurrent page offline reasons */
+				atomic_andnot(DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG,
+					      &q->ban_reason);
 				xe_gt_reset_async(q->gt);
 				goto rearm;
 			}
@@ -2570,13 +2580,25 @@ static void guc_exec_queue_multi_queue_drop_suspend(struct xe_exec_queue *q)
 	}
 }
 
-static bool guc_exec_queue_reset_status(struct xe_exec_queue *q)
+static u64 guc_exec_queue_reset_status(struct xe_exec_queue *q)
 {
-	if (xe_exec_queue_is_multi_queue_secondary(q) &&
-	    guc_exec_queue_reset_status(xe_exec_queue_multi_queue_primary(q)))
-		return true;
+	if (xe_exec_queue_is_multi_queue_secondary(q)) {
+		u64 status = guc_exec_queue_reset_status(xe_exec_queue_multi_queue_primary(q));
 
-	return exec_queue_reset(q) || exec_queue_killed_or_banned_or_wedged(q);
+		if (status)
+			return status;
+	}
+
+	if (exec_queue_reset(q) || exec_queue_killed_or_banned_or_wedged(q)) {
+		u64 reason = atomic_read_acquire(&q->ban_reason);
+
+		/* If no specific reason was recorded, default to GPU hang */
+		if (!reason)
+			reason = DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG;
+		return reason;
+	}
+
+	return 0;
 }
 
 /*
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index 73cd06c22d37..2731bbc50864 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -7,6 +7,7 @@
 #include <drm/drm_managed.h>
 #include <drm/drm_drv.h>
 #include <drm/drm_buddy.h>
+#include <uapi/drm/xe_drm.h>
 
 #include <drm/ttm/ttm_placement.h>
 #include <drm/ttm/ttm_range_manager.h>
@@ -542,7 +543,12 @@ static int xe_ttm_vram_purge_page(struct xe_device *xe, struct xe_bo *bo)
 	xe_bo_unlock(bo);
 	/*  Ban VM if BO is PPGTT */
 	if (vm && (flags & XE_BO_FLAG_PAGETABLE)) {
+		struct xe_exec_queue *eq;
+
 		down_write(&vm->lock);
+		list_for_each_entry(eq, &vm->preempt.exec_queues, lr.link)
+			atomic_or(DRM_XE_EXEC_QUEUE_BAN_REASON_PAGE_OFFLINE, &eq->ban_reason);
+		smp_wmb(); /* Force all queue bits to be visible before killing the VM */
 		xe_vm_kill(vm, true);
 		up_write(&vm->lock);
 	}
@@ -554,6 +560,8 @@ static int xe_ttm_vram_purge_page(struct xe_device *xe, struct xe_bo *bo)
 	/*  Ban exec queue if BO is lrc */
 	if (q && xe_exec_queue_get_unless_zero(q)) {
 		/* ban queue */
+		atomic_or(DRM_XE_EXEC_QUEUE_BAN_REASON_PAGE_OFFLINE, &q->ban_reason);
+		smp_wmb(); /* Force bit change to finish before state change triggers */
 		q_to_put = q;
 	}
 
diff --git a/include/uapi/drm/xe_drm.h b/include/uapi/drm/xe_drm.h
index 509202a7b13e..ee4a921b2e6e 100644
--- a/include/uapi/drm/xe_drm.h
+++ b/include/uapi/drm/xe_drm.h
@@ -1491,6 +1491,12 @@ struct drm_xe_exec_queue_destroy {
  *
  * The @property can be:
  *  - %DRM_XE_EXEC_QUEUE_GET_PROPERTY_BAN
+ *
+ * For %DRM_XE_EXEC_QUEUE_GET_PROPERTY_BAN, @value is a bitmask of ban reasons:
+ *  - %DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG - banned due to GPU hang/timeout
+ *  - %DRM_XE_EXEC_QUEUE_BAN_REASON_PAGE_OFFLINE - banned due to memory page offline
+ *
+ * A @value of 0 means the exec queue is not banned.
  */
 struct drm_xe_exec_queue_get_property {
 	/** @extensions: Pointer to the first extension struct, if any */
@@ -1503,7 +1509,17 @@ struct drm_xe_exec_queue_get_property {
 	/** @property: property to get */
 	__u32 property;
 
-	/** @value: property value */
+	/**
+	 * @value: property value
+	 *
+	 * For %DRM_XE_EXEC_QUEUE_GET_PROPERTY_BAN, this is a bitmask of:
+	 *  - %DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG - banned due to GPU hang/timeout
+	 *  - %DRM_XE_EXEC_QUEUE_BAN_REASON_PAGE_OFFLINE - banned due to memory page offline
+	 *
+	 * Value of 0 means the exec queue is not banned.
+	 */
+#define DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG		(1 << 0)
+#define DRM_XE_EXEC_QUEUE_BAN_REASON_PAGE_OFFLINE	(1 << 1)
 	__u64 value;
 
 	/** @reserved: Reserved */
-- 
2.52.0

[PATCH V16 12/12] drm/xe: Add fault-inject based VRAM page offline injection

[Tejas Upadhyay]

Add a fault-inject based debugfs interface for testing VRAM page
offlining. This replaces the previous standalone debugfs approach
with the standard kernel fault-inject infrastructure.

Two debugfs entries are created under the xe debugfs root for
CRI platforms:
- inject_mempage_offline/: Standard fault-inject knobs (probability,
  times, interval, etc.) - only available with
  CONFIG_FAULT_INJECTION_DEBUG_FS
- inject_mempage_offline_trigger: Write a PFN value to inject a
  specific page, or write "0" to auto-pick the last unallocated
  VRAM page

The trigger accepts:
- "0"      : auto-pick last unallocated page
- "0xPFN"  : inject fault at a specific PFN address

On kernels with CONFIG_FAULT_INJECTION_DEBUG_FS, injection is gated
by should_fail() (probability/times must be configured first).
On kernels without it, the trigger always injects directly.

The injection reports the page as faulted via
xe_ttm_vram_handle_addr_fault(), exercising the full page offlining
path.

Usage (with CONFIG_FAULT_INJECTION_DEBUG_FS):
  echo 100 > inject_mempage_offline/probability
  echo 1 > inject_mempage_offline/times
  echo 0 > inject_mempage_offline_trigger

Usage (without CONFIG_FAULT_INJECTION_DEBUG_FS):
  echo 0 > inject_mempage_offline_trigger

v3(Himal):
- Use FAULT_ACTION
v2(sashiko):
- use cond_resched()
- validate input first and fix addr < 0 case
- validate vr, move block, found var as local to scope_guard

Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
---
 drivers/gpu/drm/xe/xe_debugfs.c      | 50 ++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_debugfs.h      |  2 ++
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.c | 52 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_ttm_vram_mgr.h |  1 +
 4 files changed, 105 insertions(+)

diff --git a/drivers/gpu/drm/xe/xe_debugfs.c b/drivers/gpu/drm/xe/xe_debugfs.c
index 28135f84e286..72258f3ce7b0 100644
--- a/drivers/gpu/drm/xe/xe_debugfs.c
+++ b/drivers/gpu/drm/xe/xe_debugfs.c
@@ -32,6 +32,7 @@
 #include "xe_sriov_vf.h"
 #include "xe_step.h"
 #include "xe_tile_debugfs.h"
+#include "xe_ttm_vram_mgr.h"
 #include "xe_vsec.h"
 #include "xe_wa.h"
 
@@ -44,12 +45,18 @@
 DECLARE_FAULT_ATTR(gt_reset_failure);
 DECLARE_FAULT_ATTR(inject_csc_hw_error);
 DECLARE_FAULT_ATTR(wedge_cold_reset);
+DECLARE_FAULT_ATTR(inject_mempage_offline);
 
 static bool csc_hw_error_available(struct xe_device *xe)
 {
 	return !IS_SRIOV_VF(xe) && xe->info.platform == XE_BATTLEMAGE;
 }
 
+static bool is_crescent_island(struct xe_device *xe)
+{
+	return xe->info.platform == XE_CRESCENTISLAND;
+}
+
 /*
  * Fault injection table.  Each entry registers a debugfs attribute; add a
  * matching FAULT_ACTION() below for every entry added here.
@@ -66,6 +73,9 @@ static struct {
 	  .is_visible = csc_hw_error_available },
 	{ .name = "wedge_cold_reset",
 	  .attr = &wedge_cold_reset },
+	{ .name = "inject_mempage_offline",
+	  .attr = &inject_mempage_offline,
+	  .is_visible = is_crescent_island },
 };
 
 /*
@@ -81,6 +91,41 @@ bool xe_fault_##name(void)				\
 FAULT_ACTION(gt_reset, gt_reset_failure)
 FAULT_ACTION(csc_hw_error, inject_csc_hw_error)
 FAULT_ACTION(wedge_cold_reset, wedge_cold_reset)
+FAULT_ACTION(mempage_offline, inject_mempage_offline)
+
+static ssize_t inject_mempage_offline_trigger(struct file *f,
+					      const char __user *ubuf,
+					      size_t size, loff_t *pos)
+{
+	struct xe_device *xe = file_inode(f)->i_private;
+	struct xe_tile *tile = xe_device_get_root_tile(xe);
+	struct xe_vram_region *vr = tile->mem.vram;
+	u64 pfn;
+	int ret;
+
+	if (!vr)
+		return -ENODEV;
+
+	ret = kstrtou64_from_user(ubuf, size, 0, &pfn);
+	if (ret)
+		return ret;
+
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION_DEBUG_FS) &&
+	    !xe_fault_mempage_offline())
+		return size;
+
+	if (pfn == 0)
+		return xe_ttm_vram_inject_fault(xe) ?: size;
+
+	/* User provided PFN - convert to DPA and inject */
+	return xe_ttm_vram_handle_addr_fault(xe,
+					     (pfn << PAGE_SHIFT) + vr->dpa_base) ?: size;
+}
+
+static const struct file_operations inject_mempage_offline_fops = {
+	.owner = THIS_MODULE,
+	.write = inject_mempage_offline_trigger,
+};
 
 static void xe_fault_inject_debugfs_register(struct xe_device *xe,
 					     struct dentry *root)
@@ -95,6 +140,11 @@ static void xe_fault_inject_debugfs_register(struct xe_device *xe,
 		fault_create_debugfs_attr(xe_fault_inject_entry[i].name, root,
 					  xe_fault_inject_entry[i].attr);
 	}
+
+	if (is_crescent_island(xe)) {
+		debugfs_create_file("inject_mempage_offline_trigger", 0200,
+				    root, xe, &inject_mempage_offline_fops);
+	}
 }
 
 static void read_residency_counter(struct xe_device *xe, struct xe_mmio *mmio,
diff --git a/drivers/gpu/drm/xe/xe_debugfs.h b/drivers/gpu/drm/xe/xe_debugfs.h
index 0dcd28fd7dc0..88d91c78036b 100644
--- a/drivers/gpu/drm/xe/xe_debugfs.h
+++ b/drivers/gpu/drm/xe/xe_debugfs.h
@@ -14,11 +14,13 @@ struct xe_device;
 bool xe_fault_gt_reset(void);
 bool xe_fault_csc_hw_error(void);
 bool xe_fault_wedge_cold_reset(void);
+bool xe_fault_mempage_offline(void);
 void xe_debugfs_register(struct xe_device *xe);
 #else
 static inline bool xe_fault_gt_reset(void) { return false; }
 static inline bool xe_fault_csc_hw_error(void) { return false; }
 static inline bool xe_fault_wedge_cold_reset(void) { return false; }
+static inline bool xe_fault_mempage_offline(void) { return false; }
 static inline void xe_debugfs_register(struct xe_device *xe) { }
 #endif
 
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
index 2731bbc50864..3243600717be 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.c
@@ -829,6 +829,58 @@ int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr)
 }
 EXPORT_SYMBOL(xe_ttm_vram_handle_addr_fault);
 
+/**
+ * xe_ttm_vram_inject_fault - Inject a VRAM page fault for testing
+ * @xe: xe device instance
+ *
+ * Picks the last unallocated VRAM page and reports it as faulted
+ * via xe_ttm_vram_handle_addr_fault(). Used by the fault-inject
+ * debugfs interface for testing page offlining.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int xe_ttm_vram_inject_fault(struct xe_device *xe)
+{
+	struct xe_tile *tile = xe_device_get_root_tile(xe);
+	struct xe_vram_region *vr = tile->mem.vram;
+	struct xe_ttm_vram_mgr *vram_mgr = &vr->ttm;
+	struct gpu_buddy *mm = &vram_mgr->mm;
+	u64 addr;
+
+	if (vr->actual_physical_size < SZ_4K)
+		return -ENOSPC;
+
+	addr = vr->actual_physical_size - SZ_4K;
+	while (addr < vr->actual_physical_size) {
+		struct gpu_buddy_block *block;
+		bool found = false;
+
+		scoped_guard(mutex, &vram_mgr->lock) {
+			block = gpu_buddy_allocated_addr_to_block(mm, addr);
+			if (!block)
+				found = true;
+		}
+
+		/*
+		 * Intentional race window: xe_ttm_vram_handle_addr_fault()
+		 * re-acquires vram_mgr->lock internally, so we cannot hold
+		 * it here. A concurrent allocation claiming this page between
+		 * the two calls is an acceptable false negative for this
+		 * test-only path.
+		 */
+		if (found)
+			return xe_ttm_vram_handle_addr_fault(xe, addr + vr->dpa_base);
+
+		cond_resched();
+		if (addr == 0)
+			break;
+		addr -= SZ_4K;
+	}
+
+	return -ENOSPC;
+}
+EXPORT_SYMBOL(xe_ttm_vram_inject_fault);
+
 static size_t serialize_bad_pages(struct xe_ttm_vram_mgr *mgr, char *buf, size_t max_len)
 {
 	struct xe_ttm_vram_offline_resource *pos;
diff --git a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
index eb55b0f74ef3..9feb999a1f8d 100644
--- a/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
+++ b/drivers/gpu/drm/xe/xe_ttm_vram_mgr.h
@@ -32,6 +32,7 @@ void xe_ttm_vram_get_used(struct ttm_resource_manager *man,
 			  u64 *used, u64 *used_visible);
 
 int xe_ttm_vram_handle_addr_fault(struct xe_device *xe, u64 addr);
+int xe_ttm_vram_inject_fault(struct xe_device *xe);
 int xe_ttm_vram_sysfs_init(struct xe_device *xe);
 static inline struct xe_ttm_vram_mgr_resource *
 to_xe_ttm_vram_mgr_resource(struct ttm_resource *res)
-- 
2.52.0