Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v3 0/1] drm/xe/guc_ads: allocate UM queues in a separate UC BO
@ 2026-07-22  5:34 Jia Yao
  2026-07-22  5:34 ` [PATCH v3 1/1] " Jia Yao
                   ` (5 more replies)
  0 siblings, 6 replies; 17+ messages in thread
From: Jia Yao @ 2026-07-22  5:34 UTC (permalink / raw)
  To: intel-xe; +Cc: Jia Yao

== Problem ==

On platforms with Unified Shared Memory (USM), the xe driver maintains
three hardware page-fault queues (PAGE_FAULT, PAGE_FAULT_RESPONSE,
ACCESS_COUNTER) inside the GuC Additional Data Structures (ADS) blob.
The ADS blob is mapped WB on the CPU side.

xe_guc_ads_populate() is called on every GT reset to reprogram the ADS
blob; it begins with a full memset() of the blob to zero.  The zeroed
cache lines for the UM queue region reside in the CPU cache (LLC, WB)
but are not immediately written back to DRAM.

GAM hardware writes fault descriptors to the queue via DPA - a
non-coherent path that bypasses the CPU cache and writes directly to
memory.  GuC reads the queue via GGTT (WB), which hits the CPU cache.
As a result GuC sees the stale zeroed data from the CPU cache instead
of the descriptor written to DRAM by GAM.  The driver then receives an
all-zero fault descriptor and returns -EINVAL, causing an unnecessary
engine reset.

This is documented in HSD as:
  "When Fault queue is updated by the SW (Buffer zeroed), it can reside
   in the CPU cache (L4:WB). A non coherent write from GAM HW would not
   consult the CPU cache and it will eventually get written into the
   memory (depends upon eviction/flushes) and CPU cache has stale data.
   When GUC HW reads the fault queue it gets it from the CPU cache,
   which is stale data."

The window is narrow but reliably reproducible: run xe_exec_reset
--run-subtest gt-reset followed immediately by
xe_exec_system_allocator --run-subtest
threads-many-execqueues-mmap-new-race.

== Reproduction ==

  # Requires a platform with has_usm (e.g. Xe3 iGFX)
  sudo xe_exec_reset --run-subtest gt-reset
  sudo xe_exec_system_allocator --run-subtest threads-many-execqueues-mmap-new-race

  Symptom in dmesg:
    [xe] ASID: 0, Faulted Address: 0x0000000000000000, FaultType: 0

  The all-zero fault descriptor is the tell-tale sign of the stale
  cache line race.

== Fix ==

Allocate the UM queues in a dedicated UC BO (ads->um_queue_bo).  UC
mapping on the CPU side ensures the memset bypasses the cache and the
zeroed data lands in DRAM immediately.  GuC's non-coherent read then
goes to DRAM and sees the correct descriptor written by GAM.

A separate BO also eliminates the secondary race where a descriptor
written by the GPU between GDRST and GuC restart could be zeroed by the
next call to xe_guc_ads_populate().

On dGFX, VRAM placement is used so the DPA is derived from
vram_region_gpu_offset, avoiding the IOVA aliasing issue seen on
platforms with limited DRAM (e.g. BMG 4 GB) where a SYSTEM BO's DMA
address can alias a PCI MMIO reserved window.

Why not just flush the cache after memset?

An earlier approach called drm_clflush_virt_range() on the UM queue
region immediately after the memset to evict the stale zero lines from
LLC.  This reduces the failure rate significantly but does not fully
eliminate the race for two reasons.

First, the Linux kernel maintains a WB direct mapping (linear map) that
covers all physical memory.  Even after flushing the vmap alias, the
CPU can speculatively prefetch the same physical page through the WB
direct-map alias, silently repopulating the cache with zeros.  LNL is
known to be particularly susceptible to this speculative prefetch
behaviour.  XE_BO_FLAG_NEEDS_UC calls set_memory_uc() which updates
both the vmap PTE and the direct-map PTE to UC, closing this aliasing
window entirely.

Second, the flush approach leaves the memory mapped WB on the CPU side
permanently.  Any future SW access - a debug read, a tracing hook, an
inadvertent touch - can re-dirty the cache line and reintroduce the
race.  UC mapping makes the correct behaviour unconditional and
self-documenting.

Jia Yao (1):
  drm/xe/guc_ads: allocate UM queues in a separate UC BO

 drivers/gpu/drm/xe/xe_guc_ads.c       | 64 ++++++++++++++++-----------
 drivers/gpu/drm/xe/xe_guc_ads_types.h |  8 +++-
 2 files changed, 46 insertions(+), 26 deletions(-)

-- 
2.43.0


^ permalink raw reply	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2026-07-30 13:12 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-22  5:34 [PATCH v3 0/1] drm/xe/guc_ads: allocate UM queues in a separate UC BO Jia Yao
2026-07-22  5:34 ` [PATCH v3 1/1] " Jia Yao
2026-07-22  9:18   ` Matthew Auld
2026-07-22 23:00   ` Gwan-gyeong Mun
2026-07-22  5:41 ` ✗ CI.checkpatch: warning for " Patchwork
2026-07-22  5:42 ` ✓ CI.KUnit: success " Patchwork
2026-07-22  6:17 ` ✓ Xe.CI.BAT: " Patchwork
2026-07-22 20:20 ` ✓ Xe.CI.FULL: " Patchwork
2026-07-30  0:47 ` [PATCH v4 0/1] " Jia Yao
2026-07-30  0:47   ` [PATCH v4 1/3] drm/xe/guc_ads: allocate UM queues in a separate BO Jia Yao
2026-07-30 10:28     ` Matthew Auld
2026-07-30  0:47   ` [PATCH v4 2/3] drm/xe/guc_ads: allocate UM queues in VRAM on dGFX Jia Yao
2026-07-30 10:51     ` Matthew Auld
2026-07-30 13:12       ` Matthew Auld
2026-07-30  0:47   ` [PATCH v4 3/3] drm/xe/guc_ads: use uncached mapping for UM queue BO Jia Yao
2026-07-30  9:49     ` Michal Wajdeczko
2026-07-30 11:23     ` Matthew Auld

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox