Re: [PATCH 00/13] drm/xe: Isolate wedged devices from hardware access

["Thomas Hellström" <thomas.hellstrom@linux.intel.com>]

On Thu, 2026-08-27 at 15:47 +0530, Arvind Yadav wrote:
> A permanently wedged device can still access hardware through pending
> work,
> existing VRAM mappings and background driver paths. This can cause
> unwanted
> DMA or MMIO access while userspace is preparing device recovery.
> 
> This series separates temporary PCI error recovery from permanent
> wedging and
> introduces a common device-I/O-blocked state.
> 
> After a permanent wedge, the driver:
> - stops queued VM, SVM and GuC work.
> - shuts down the display.
> - invalidates existing CPU mappings to VRAM.
> - maps later CPU faults to a per-BO dummy page.
> - stops interrupts and clears PCI bus mastering.
> - rejects new VRAM allocations.
> - notifies userspace only after isolation completes.
> - drops queued page faults and stops VM, SVM and GuC work.
> 
> System suspend waits for wedge isolation to complete before skipping
> the
> normal suspend path. The worker cannot reach synchronize_srcu() until
> all
> GTs have been declared wedged and their pending fences have been
> signalled.
> 
> The device remains isolated until userspace performs the advertised
> recovery,
> such as driver rebind or bus reset.
> 
> The series also fixes IRQ uninstall so requested handlers are freed
> after
> interrupts have already been suspended.
> 
> Arvind Yadav (13):
>   drm/xe/irq: Always free requested IRQs on uninstall
>   drm/xe: Separate AER reset state from device wedging
>   drm/xe: Drop queued page faults when device I/O is blocked
>   drm/xe: Stop VM work when device I/O is blocked
>   drm/xe: Send wedged notification from a worker
>   drm/xe: Reuse one dummy page per BO after wedge
>   drm/xe: Invalidate existing VRAM mappings on wedge
>   drm/xe/irq: Serialize IRQ suspend and resume
>   drm/xe: Isolate a wedged device before notifying userspace
>   drm/xe/ttm: Reject VRAM allocations on wedged devices
>   drm/xe/guc: Skip timeout recovery on a wedged device
>   drm/xe: Skip PM notifier preparation for wedged devices
>   drm/xe: Block BO VM access when device I/O is unavailable
> 
>  drivers/gpu/drm/xe/display/xe_display.c |  18 +++-
>  drivers/gpu/drm/xe/xe_bo.c              | 101 +++++++++++++++++++-
>  drivers/gpu/drm/xe/xe_bo.h              |   1 +
>  drivers/gpu/drm/xe/xe_bo_types.h        |   4 +
>  drivers/gpu/drm/xe/xe_device.c          | 118 ++++++++++++++++++++--
> --
>  drivers/gpu/drm/xe/xe_device.h          |  12 +++
>  drivers/gpu/drm/xe/xe_device_types.h    |  26 ++++++
>  drivers/gpu/drm/xe/xe_guc_ct.c          |   4 +-
>  drivers/gpu/drm/xe/xe_guc_pc.c          |  10 +-
>  drivers/gpu/drm/xe/xe_guc_rc.c          |   4 +-
>  drivers/gpu/drm/xe/xe_guc_submit.c      |  13 ++-
>  drivers/gpu/drm/xe/xe_guc_tlb_inval.c   |   8 +-
>  drivers/gpu/drm/xe/xe_irq.c             |  32 +++++--
>  drivers/gpu/drm/xe/xe_pagefault.c       |  40 ++++++++
>  drivers/gpu/drm/xe/xe_pci.c             |  23 ++++-
>  drivers/gpu/drm/xe/xe_pci_error.c       |  22 ++---
>  drivers/gpu/drm/xe/xe_pm.c              |  27 ++++++
>  drivers/gpu/drm/xe/xe_sriov_pf.c        |   2 +-
>  drivers/gpu/drm/xe/xe_svm.c             |  21 ++++-
>  drivers/gpu/drm/xe/xe_ttm_vram_mgr.c    |   4 +
>  drivers/gpu/drm/xe/xe_vm.c              |  16 ++++
>  21 files changed, 446 insertions(+), 60 deletions(-)

Hi, Arvind.

I think this series lacks context to some extent. Much of the code is
also inheritly racy [1]. because current hardware accesses aren't
drained before reset or wedging. What I think is needed here is a
design detailing

1) What type of user-space accesses and workqueues are completely
blocked (for example sharing the newly introduced vram_userfault.srcu)
This can perhaps also be combined with the drm SRCU since I figure
there might be multiple places where we ensure hardware is present
using drm_dev_enter / drm_dev_exit and also want to check whether it's
wedged or not.

See an example in
https://gitlab.freedesktop.org/thomash/kernel/-/commits/wedge?ref_type=heads
(Note that this is just an example, not fully adapted to the intentions
in this series).

2) How is user-space supposed to respond to a recoverable PCIe reset?
Currently? If a change is planned, In the future? Is it the case that
currently we flag "In reset" and then force the user to close its drm
connection and re-discover a recovered device? Is this about to change
and hence the distinction between PCIe reset and permanent wedging?

It looks like AMD might have an rwsem that just blocks callers over a
reset and perhaps notifies them that, for example VRAM is lost. In any
case, in addition to locking out new callers we need to drain old
callers:

[1]
Thread A                   Thread B

is_io_allowed()
			    disable_io()

access_hardware()	    reset()

Bang!

Thanks,
Thomas