Re: [PATCH] drm/xe: Skip schedule disable for an already reset exec queue

[Matthew Brost <matthew.brost@intel.com>]

On Fri, Sep 18, 2026 at 03:38:39AM +0800, Jagmeet Randhawa wrote:
> A memory CAT error causes GuC to reset the offending context and notify
> the driver. xe_guc_exec_queue_memory_cat_error_handler() treats this the
> same as an engine reset and calls xe_guc_exec_queue_reset_trigger_cleanup(),
> which sets EXEC_QUEUE_STATE_RESET.
> 
> Roughly 200us later the TDR runs for the same queue. It reads
> exec_queue_reset(), uses the result only to set err = -EIO, and then sends
> a schedule disable (H2G SCHED_CONTEXT_MODE_SET) anyway. GuC has already
> destroyed the context, so no SCHED_DONE is ever returned. The TDR then
> waits out its HZ*5 timeout, reports "Schedule disable failed to respond"
> and calls xe_gt_reset_async(), escalating a per-context failure into a
> full GT reset that kills every context on the tile and reloads GuC.
> 

There is a mode where a CAT error or GuC queue reset disables scheduling
or tears down the context, but we do not enable that mode.

I see there is a multi-queue related issue mentioned below, and it is
certainly possible that we have a bug there. However, I'm fairly
confident that this patch will break non-multi-queue flows.

On my BMG system running drm-tip:

On my BMG + drm-tip:

cd /sys/kernel/debug/tracing/events/xe
for f in xe_exec_queue*/enable; do
    echo 1 > "$f"
done
cd -
cd /sys/kernel/debug/tracing/events/xe
for f in xe_sched_job_*/enable; do
    echo 1 > "$f"
done
cd -

root@DUT6235BMGFRD:mbrost# xe_exec_reset --r cat-error
IGT-Version: 2.5-gaccf33cc6 (x86_64) (Linux: 7.3.0-rc3-xe+ x86_64)
Using IGT_SRANDOM=1789675008 for randomisation
Opened device: /dev/dri/card0
Starting subtest: cat-error
Subtest cat-error: SUCCESS (0.027s)

The in ftrace:

 38    kworker/u64:2-130     [012] .....   265.216703: xe_exec_queue_memory_cat_error: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x3, flags=0x0
 39    kworker/u64:2-130     [012] .....   265.217326: xe_exec_queue_scheduling_disable: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x249, flags=0x0
 40    kworker/u64:6-154     [011] .....   265.218858: xe_sched_job_free: dev=0000:03:00.0, fence=00000000ba06a299, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=3, batch_addr=0x0    000001a04a0, guc_state=0x3, flags=0x0, error=0
 41    kworker/u64:4-134     [000] .....   265.219506: xe_exec_queue_reset: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x249, flags=0x0
 42    kworker/u64:4-134     [000] .....   265.219507: xe_exec_queue_scheduling_done: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x249, flags=0x0
 43    kworker/u64:2-130     [012] .....   265.219518: xe_sched_job_timedout: dev=0000:03:00.0, fence=00000000f8118031, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=2, batch_addr    =0x0000002a0000, guc_state=0x241, flags=0x0, error=0
 44    kworker/u64:2-130     [012] .....   265.242149: xe_sched_job_set_error: dev=0000:03:00.0, fence=00000000f8118031, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=2, batch_add    r=0x0000002a0000, guc_state=0x241, flags=0x0, error=-5
 45    kworker/u64:2-130     [012] .....   265.242154: xe_sched_job_free: dev=0000:03:00.0, fence=00000000f8118031, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=2, batch_addr=0x0    000002a0000, guc_state=0x241, flags=0x0, error=-5

Lines 39, 43, 44, 45 are the TDR.
Lines 38, 41, 42 are the G2H handlers.

It isn't safe to signal a job's fences until scheduling has been
disabled, as that is the point at which we know the queue is no longer
accessing memory. A job's fence is what prevents the associated memory
from being moved while the job may still be touching it.

After you change, the above test takes ~4 to complete too.

Please rethink this patch.

Matt

> The failure is intermittent because the enclosing guard is
> (exec_queue_enabled || exec_queue_pending_disable): it is a race between
> the CAT error cleanup and the TDR. When cleanup wins the block is skipped
> and recovery is clean; when the TDR wins the disable is sent and the
> timeout fires.
> 
> Skip the handshake entirely when the context has already been reset. The
> send, the HZ*5 wait and the GT reset escalation all live in the same block,
> so a single condition removes all three. err = -EIO is preserved so
> userspace still receives the correct error, and execution falls through to
> the normal cleanup path, which performs the deregistration.
> 
> This matches existing behaviour elsewhere in the driver:
> xe_guc_exec_queue_reset_handler() calls the same
> xe_guc_exec_queue_reset_trigger_cleanup() and performs no disable
> handshake at all. The TDR path was simply inconsistent with it.
> 
> Evidence, captured with existing ftrace events only (no instrumentation):
> 
>   Healthy acknowledgements take ~350us:
>     4806.244755 xe_exec_queue_scheduling_enable  guc_id=2 guc_state=0x7
>     4806.245126 xe_exec_queue_scheduling_done    guc_id=2 guc_state=0x7
> 
>   The failure:
>     4806.245395 xe_exec_queue_memory_cat_error   guc_id=2 guc_state=0x3
>     4806.245592 xe_exec_queue_scheduling_disable guc_id=2 guc_state=0x249
>                 <5.18s, no scheduling_done for any guc_id>
>     4811.425872 xe_sched_job_timedout            guc_id=2 guc_state=0x200
> 
>   guc_state=0x249 is REGISTERED|PENDING_DISABLE|RESET|BANNED. The RESET bit
>   confirms the exec_queue_reset() branch was taken and the disable was sent
>   regardless. scheduling_done never fires, so handle_sched_done() never runs
>   and the acknowledgement genuinely never arrives.
> 
>   With the fix, scheduling_disable is traced at guc_state=0x2d9
>   (adds DESTROYED|KILLED), i.e. it now originates from
>   disable_scheduling_deregister() on the normal cleanup path, and GuC
>   acknowledges it every time.
> 
> Testing:
>   igt@xe_exec_reset@multi-queue-cancel               0/10 failures
>                                                      (baseline 10/10 failures)
>   igt@xe_exec_reset@multi-queue-cancel-on-secondary  passing
>   Subtest runtime drops from ~5.2s to ~85ms.
> 
> Signed-off-by: Jagmeet Randhawa <jagmeet.randhawa@intel.com>
> ---
>  drivers/gpu/drm/xe/xe_guc_submit.c | 14 ++++++++++----
>  1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> index f3ba8abfc228..03450590042a 100644
> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> @@ -1609,14 +1609,20 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
>  		atomic_or(DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG, &q->ban_reason);
>  	set_exec_queue_banned(q);
>  
> +	/*
> +	 * GuC already reset this context (the CAT error handler treats a CAT
> +	 * error as an engine reset), so there is nothing left to disable and
> +	 * SCHED_DONE will never arrive. Skip the handshake rather than waiting
> +	 * HZ*5 and escalating to a full GT reset.
> +	 */
> +	if (exec_queue_reset(primary))
> +		err = -EIO;
> +
>  	/* Kick job / queue off hardware */
> -	if (!xe_device_is_in_reset(xe) && !wedged &&
> +	if (!xe_device_is_in_reset(xe) && !wedged && !exec_queue_reset(primary) &&
>  	    (exec_queue_enabled(primary) || exec_queue_pending_disable(primary))) {
>  		int ret;
>  
> -		if (exec_queue_reset(primary))
> -			err = -EIO;
> -
>  		if (xe_uc_fw_is_running(&guc->fw)) {
>  			/*
>  			 * Wait for any pending G2H to flush out before
> -- 
> 2.53.0
>