Re: [PATCH v6 00/12] drm/xe/madvise: Add support for purgeable buffer objects

[Matthew Brost <matthew.brost@intel.com>]

On Tue, Mar 03, 2026 at 03:05:59PM -0700, Souza, Jose wrote:
> On Tue, 2026-03-03 at 20:49 +0530, Arvind Yadav wrote:
> > This patch series introduces comprehensive support for purgeable
> > buffer objects
> > in the Xe driver, enabling userspace to provide memory usage hints
> > for better
> > memory management under system pressure.
> > 
> > Overview:
> > 
> > Purgeable memory allows applications to mark buffer objects as "not
> > currently
> > needed" (DONTNEED), making them eligible for kernel reclamation
> > during memory
> > pressure. This helps prevent OOM conditions and enables more
> > efficient GPU
> > memory utilization for workloads with temporary or regeneratable data
> > (caches,
> > intermediate results, decoded frames, etc.).
> > 
> > Purgeable BO Lifecycle:
> > 1. WILLNEED (default): BO actively needed, kernel preserves backing
> > store
> > 2. DONTNEED (user hint): BO contents discardable, eligible for
> > purging
> > 3. PURGED (kernel action): Backing store reclaimed during memory
> > pressure
> > 
> > Key Design Principles:
> >   - i915 compatibility: "Once purged, always purged" semantics -
> > purged BOs
> >     remain permanently invalid and must be destroyed/recreated
> >   - Per-VMA state tracking: Each VMA tracks its own purgeable state,
> > BO is
> >     only marked DONTNEED when ALL VMAs across ALL VMs agree (Thomas
> > Hellström)
> >   - Safety first: Imported/exported dma-bufs blocked from purgeable
> > state -
> >     no visibility into external device usage (Matt Roper)
> >   - Multiple protection layers: Validation in madvise, VM bind, mmap,
> > CPU
> >     and GPU fault handlers. GPU page faults on DONTNEED BOs are
> > rejected in
> >     xe_pagefault_begin() to preserve the GPU PTE invalidation done at
> > madvise
> >     time; without this the rebind path would re-map real pages and
> > undo the
> >     PTE zap, preventing the shrinker from ever reclaiming the BO.
> >   - Correct GPU PTE zapping: madvise_purgeable() explicitly sets
> >     skip_invalidation per VMA (false for DONTNEED, true for WILLNEED,
> > purged
> >     and dmabuf-shared BOs) so DONTNEED always triggers a GPU PTE zap
> >     regardless of prior madvise state.
> >   - Scratch PTE support: Fault-mode VMs use scratch pages for safe
> > zero reads
> >     on purged BO access.
> >   - TTM shrinker integration: Encapsulated helpers manage xe_ttm_tt-
> > >purgeable
> >     flag and shrinker page accounting (shrinkable vs purgeable
> > buckets)
> 
> 
> I get Engine memory CAT errors when using this feature:
> 
> [  240.301213] xe 0000:00:02.0: [drm] Tile0: GT0: Fault response:
> Unsuccessful -EINVAL
> [  240.301301] xe 0000:00:02.0: [drm] Tile0: GT0: Engine memory CAT
> error [18]: class=rcs, logical_mask: 0x1, guc_id=17
> [  240.302871] xe 0000:00:02.0: [drm] Tile0: GT0: Engine reset:
> engine_class=rcs, logical_mask: 0x1, guc_id=17, state=0x249
> [  240.302885] xe 0000:00:02.0: [drm] Tile0: GT0: Timedout job:
> seqno=4294967169, lrc_seqno=4294967169, guc_id=17, flags=0x0 in
> arb_map_buffer_ [3374]
> [  240.302892] xe 0000:00:02.0: [drm:xe_devcoredump [xe]] Multiple
> hangs are occurring, but only the first snapshot was taken
> 
> Mesa creates VM with DRM_XE_VM_CREATE_FLAG_SCRATCH_PAGE, probably you
> don't have a IGT test with this scenario.
> 
> @cc Rodrigo
> 
> Other issue is not related to your patches but drm_xe_madvise only
> works with non-canonical addresses and some time ago was agreed that
> all the user-visible addresses would be in canonical format.
> Not sure if we can do anything at this point but letting you know.
> 

We actually might be able to fix it to accept canonical addresses, what we
can't blindly do is make non-canonical addresses stop working... 

It might create weird scenario for UMDs though if canonical addresses
work on some kernel but not others but perhaps since this is Mesa's
first use of madvise we get this in as part of purgable and only NEO
would have to deal with this scenario.

Matt

> 
> > 
> > v2 Changes:
> >   - Reordered patches: Moved shared BO helper before main
> > implementation for
> >     proper dependency order
> >   - Fixed reference counting in mmap offset validation (use
> > drm_gem_object_put)
> >   - Removed incorrect claims about madvise(WILLNEED) restoring purged
> > BOs
> >   - Fixed error code documentation inconsistencies
> >   - Initialize purge_state_val fields to prevent kernel memory leaks
> >   - Use xe_bo_trigger_rebind() for async TLB invalidation (Thomas
> > Hellström)
> >   - Add NULL rebind with scratch PTEs for fault mode (Thomas
> > Hellström)
> >   - Implement i915-compatible retained field logic (Thomas Hellström)
> >   - Skip BO validation for purged BOs in page fault handler (crash
> > fix)
> >   - Add scratch VM check in page fault path (non-scratch VMs fail
> > fault)
> > 
> > v3 Changes (addressing Matt and Thomas Hellström feedback):
> >   - Per-VMA purgeable state tracking: Added xe_vma->purgeable_state
> > field
> >   - Complete VMA check: xe_bo_all_vmas_dontneed() walks all VMAs
> > across all
> >     VMs to ensure unanimous DONTNEED before marking BO purgeable
> >   - VMA unbind recheck: Added xe_bo_recheck_purgeable_on_vma_unbind()
> > to
> >     re-evaluate BO state when VMAs are destroyed
> >   - Block external dma-bufs: Added xe_bo_is_external_dmabuf() check
> > using
> >     drm_gem_is_imported() and obj->dma_buf to prevent purging
> > imported/exported BOs
> >   - Consistent lockdep enforcement: Added xe_bo_assert_held() to all
> > helpers
> >     that access madv_purgeable state
> >   - Simplified page table logic: Renamed is_null to is_null_or_purged
> > in
> >     xe_pt_stage_bind_entry() - purged BOs treated identically to null
> > VMAs
> >   - Removed unnecessary checks: Dropped redundant "&& bo" check in
> > xe_ttm_bo_purge()
> >   - Xe-specific warnings: Changed drm_warn() to XE_WARN_ON() in purge
> > path
> >   - Moved purge checks under locks: Purge state validation now done
> > after
> >     acquiring dma-resv lock in vma_lock_and_validate() and
> > xe_pagefault_begin()
> >   - Race-free fault handling: Removed unlocked purge check from
> >     xe_pagefault_handle_vma(), moved to locked xe_pagefault_begin()
> >   - Shrinker helper functions: Added xe_bo_set_purgeable_shrinker()
> > and
> >     xe_bo_clear_purgeable_shrinker() to encapsulate TTM purgeable
> > flag updates
> >     and shrinker page accounting, improving code clarity and
> > maintainability
> > 
> > v4 Changes (addressing Matt and Thomas Hellström feedback):
> >   - UAPI: Removed '__u64 reserved' field from purge_state_val union
> > to fit
> >     16-byte size constraint (Matt)
> >   - Changed madv_purgeable from atomic_t to u32 across all patches
> > (Matt)
> >   - CPU fault handling: Added purged check to fastpath
> > (xe_bo_cpu_fault_fastpath)
> >     to prevent hang when accessing existing mmap of purged BO
> > 
> > v5 Changes (addressing Matt and Thomas Hellström feedback):
> >   - Add locking documentation to madv_purgeable field comment (Matt)
> >   - Introduce xe_bo_set_purgeable_state() helper (void return) to
> > centralize
> >     madv_purgeable updates with xe_bo_assert_held() and state
> > transition
> >     validation using explicit enum checks (no transition out of
> > PURGED) (Matt)
> >   - Make xe_ttm_bo_purge() return int and propagate failures from
> >     xe_bo_move(); handle xe_bo_trigger_rebind() failures (e.g.
> > no_wait_gpu
> >     paths) rather than silently ignoring (Matt)
> >   - Replace drm_WARN_ON with xe_assert for better Xe-specific
> > assertions (Matt)
> >   - Hook purgeable handling into
> > madvise_funcs[DRM_XE_VMA_ATTR_PURGEABLE_STATE]
> >     instead of special-case path in xe_vm_madvise_ioctl() (Matt)
> >   - Track purgeable retained return via xe_madvise_details and
> > perform
> >     copy_to_user() from xe_madvise_details_fini() after locks are
> > dropped (Matt)
> >   - Set madvise_funcs[DRM_XE_VMA_ATTR_PURGEABLE_STATE] to NULL with
> >     __maybe_unused on madvise_purgeable() to maintain bisectability
> > until
> >     shrinker integration is complete in final patch (Matt)
> >   - Call xe_bo_recheck_purgeable_on_vma_unbind() from
> > xe_vma_destroy()
> >     right after drm_gpuva_unlink() where we already hold the BO lock,
> >     drop the trylock-based late destroy path (Matt)
> >   - Move purgeable_state into xe_vma_mem_attr with the other madvise
> >     attributes (Matt)
> >   - Drop READ_ONCE since the BO lock already protects us (Matt)
> >   - Keep returning false when there are no VMAs - otherwise we'd mark
> >     BOs purgeable without any user hint (Matt)
> >   -  Use struct xe_vma_lock_and_validate_flags instead of multiple
> > bool
> >     parameters to improve readability and prevent argument
> > transposition (Matt)
> >   - Fix LRU crash while running shrink test
> >   - Skip xe_bo_validate() for purged BOs in xe_gpuvm_validate()
> >   - Split ghost BO and zero-refcount handling in xe_bo_shrink()
> > (Thomas)
> > 
> > v6 Changes (addressing Jose Souza, Thomas Hellström and Matt Brost
> > feedback):
> >   - Document DONTNEED blocking behavior in uAPI: Clearly describe
> > which
> >     operations are blocked and with what error codes. (Thomas, Matt)
> >   - Block VM_BIND to DONTNEED BOs: Return -EBUSY to prevent creating
> > new
> >     VMAs to purgeable BOs (undefined behavior). (Thomas, Matt)
> >   - Block CPU faults to DONTNEED BOs: Return VM_FAULT_SIGBUS in both
> > fastpath
> >     and slowpath to prevent undefined behavior. (Thomas, Matt)
> >   - Block new mmap() to DONTNEED/purged BOs: Return -EBUSY for
> > DONTNEED,
> >     -EINVAL for PURGED. (Thomas, Matt)
> >   - Block dma-buf export of DONTNEED/purged BOs: Return -EBUSY for
> > DONTNEED,
> >     -EINVAL for PURGED. (Thomas, Matt)
> >   - Fix state transition bug: xe_bo_all_vmas_dontneed() now returns
> > enum to
> >     distinguish NO_VMAS (preserve state) from WILLNEED (has active
> > VMAs),
> >     preventing incorrect DONTNEED → WILLNEED flip on last VMA unmap
> > (Matt)
> >   - Set skip_invalidation explicitly in madvise_purgeable() to ensure
> >     DONTNEED always zaps GPU PTEs regardless of prior madvise state.
> >   - Add DRM_XE_QUERY_CONFIG_FLAG_HAS_PURGING_SUPPORT for userspace
> >     feature detection. (Jose)
> > 
> > Arvind Yadav (11):
> >   drm/xe/bo: Add purgeable bo state tracking and field madv to xe_bo
> >   drm/xe/madvise: Implement purgeable buffer object support
> >   drm/xe/bo: Block CPU faults to purgeable buffer objects
> >   drm/xe/vm: Prevent binding of purged buffer objects
> >   drm/xe/madvise: Implement per-VMA purgeable state tracking
> >   drm/xe/madvise: Block imported and exported dma-bufs
> >   drm/xe/bo: Block mmap of DONTNEED/purged BOs
> >   drm/xe/dma_buf: Block export of DONTNEED/purged BOs
> >   drm/xe/bo: Add purgeable shrinker state helpers
> >   drm/xe/madvise: Enable purgeable buffer object IOCTL support
> >   drm/xe/bo: Skip zero-refcount BOs in shrinker
> > 
> > Himal Prasad Ghimiray (1):
> >   drm/xe/uapi: Add UAPI support for purgeable buffer objects
> > 
> >  drivers/gpu/drm/xe/xe_bo.c         | 223 +++++++++++++++++++++--
> >  drivers/gpu/drm/xe/xe_bo.h         |  60 ++++++
> >  drivers/gpu/drm/xe/xe_bo_types.h   |   6 +
> >  drivers/gpu/drm/xe/xe_dma_buf.c    |  21 +++
> >  drivers/gpu/drm/xe/xe_pagefault.c  |  19 ++
> >  drivers/gpu/drm/xe/xe_pt.c         |  40 +++-
> >  drivers/gpu/drm/xe/xe_query.c      |   2 +
> >  drivers/gpu/drm/xe/xe_svm.c        |   1 +
> >  drivers/gpu/drm/xe/xe_vm.c         | 100 ++++++++--
> >  drivers/gpu/drm/xe/xe_vm_madvise.c | 283
> > +++++++++++++++++++++++++++++
> >  drivers/gpu/drm/xe/xe_vm_madvise.h |   3 +
> >  drivers/gpu/drm/xe/xe_vm_types.h   |  11 ++
> >  include/uapi/drm/xe_drm.h          |  60 ++++++
> >  13 files changed, 793 insertions(+), 36 deletions(-)