Re: [PATCH] drm/xe: Skip schedule disable for an already reset exec queue

["Randhawa, Jagmeet" <jagmeet.randhawa@intel.com>]

On 9/17/2026 1:10 PM, Matthew Brost wrote:
> On Fri, Sep 18, 2026 at 03:38:39AM +0800, Jagmeet Randhawa wrote:
>> A memory CAT error causes GuC to reset the offending context and notify
>> the driver. xe_guc_exec_queue_memory_cat_error_handler() treats this the
>> same as an engine reset and calls xe_guc_exec_queue_reset_trigger_cleanup(),
>> which sets EXEC_QUEUE_STATE_RESET.
>>
>> Roughly 200us later the TDR runs for the same queue. It reads
>> exec_queue_reset(), uses the result only to set err = -EIO, and then sends
>> a schedule disable (H2G SCHED_CONTEXT_MODE_SET) anyway. GuC has already
>> destroyed the context, so no SCHED_DONE is ever returned. The TDR then
>> waits out its HZ*5 timeout, reports "Schedule disable failed to respond"
>> and calls xe_gt_reset_async(), escalating a per-context failure into a
>> full GT reset that kills every context on the tile and reloads GuC.
>>
> There is a mode where a CAT error or GuC queue reset disables scheduling
> or tears down the context, but we do not enable that mode.
>
> I see there is a multi-queue related issue mentioned below, and it is
> certainly possible that we have a bug there. However, I'm fairly
> confident that this patch will break non-multi-queue flows.
>
> On my BMG system running drm-tip:
>
> On my BMG + drm-tip:
>
> cd /sys/kernel/debug/tracing/events/xe
> for f in xe_exec_queue*/enable; do
>      echo 1 > "$f"
> done
> cd -
> cd /sys/kernel/debug/tracing/events/xe
> for f in xe_sched_job_*/enable; do
>      echo 1 > "$f"
> done
> cd -
>
> root@DUT6235BMGFRD:mbrost# xe_exec_reset --r cat-error
> IGT-Version: 2.5-gaccf33cc6 (x86_64) (Linux: 7.3.0-rc3-xe+ x86_64)
> Using IGT_SRANDOM=1789675008 for randomisation
> Opened device: /dev/dri/card0
> Starting subtest: cat-error
> Subtest cat-error: SUCCESS (0.027s)
>
> The in ftrace:
>
>   38    kworker/u64:2-130     [012] .....   265.216703: xe_exec_queue_memory_cat_error: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x3, flags=0x0
>   39    kworker/u64:2-130     [012] .....   265.217326: xe_exec_queue_scheduling_disable: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x249, flags=0x0
>   40    kworker/u64:6-154     [011] .....   265.218858: xe_sched_job_free: dev=0000:03:00.0, fence=00000000ba06a299, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=3, batch_addr=0x0    000001a04a0, guc_state=0x3, flags=0x0, error=0
>   41    kworker/u64:4-134     [000] .....   265.219506: xe_exec_queue_reset: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x249, flags=0x0
>   42    kworker/u64:4-134     [000] .....   265.219507: xe_exec_queue_scheduling_done: dev=0000:03:00.0, 0:0x1, gt=0, width=1, guc_id=2, guc_state=0x249, flags=0x0
>   43    kworker/u64:2-130     [012] .....   265.219518: xe_sched_job_timedout: dev=0000:03:00.0, fence=00000000f8118031, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=2, batch_addr    =0x0000002a0000, guc_state=0x241, flags=0x0, error=0
>   44    kworker/u64:2-130     [012] .....   265.242149: xe_sched_job_set_error: dev=0000:03:00.0, fence=00000000f8118031, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=2, batch_add    r=0x0000002a0000, guc_state=0x241, flags=0x0, error=-5
>   45    kworker/u64:2-130     [012] .....   265.242154: xe_sched_job_free: dev=0000:03:00.0, fence=00000000f8118031, seqno=4294967169, lrc_seqno=4294967169, gt=0, guc_id=2, batch_addr=0x0    000002a0000, guc_state=0x241, flags=0x0, error=-5
>
> Lines 39, 43, 44, 45 are the TDR.
> Lines 38, 41, 42 are the G2H handlers.
>
> It isn't safe to signal a job's fences until scheduling has been
> disabled, as that is the point at which we know the queue is no longer
> accessing memory. A job's fence is what prevents the associated memory
> from being moved while the job may still be touching it.
>
> After you change, the above test takes ~4 to complete too.
>
> Please rethink this patch.
>
> Matt
  Thanks for the detailed review and for the BMG trace, let me revisit 
the logic. Please ignore this patch for now.

Thanks,
  Jagmeet
>
>> The failure is intermittent because the enclosing guard is
>> (exec_queue_enabled || exec_queue_pending_disable): it is a race between
>> the CAT error cleanup and the TDR. When cleanup wins the block is skipped
>> and recovery is clean; when the TDR wins the disable is sent and the
>> timeout fires.
>>
>> Skip the handshake entirely when the context has already been reset. The
>> send, the HZ*5 wait and the GT reset escalation all live in the same block,
>> so a single condition removes all three. err = -EIO is preserved so
>> userspace still receives the correct error, and execution falls through to
>> the normal cleanup path, which performs the deregistration.
>>
>> This matches existing behaviour elsewhere in the driver:
>> xe_guc_exec_queue_reset_handler() calls the same
>> xe_guc_exec_queue_reset_trigger_cleanup() and performs no disable
>> handshake at all. The TDR path was simply inconsistent with it.
>>
>> Evidence, captured with existing ftrace events only (no instrumentation):
>>
>>    Healthy acknowledgements take ~350us:
>>      4806.244755 xe_exec_queue_scheduling_enable  guc_id=2 guc_state=0x7
>>      4806.245126 xe_exec_queue_scheduling_done    guc_id=2 guc_state=0x7
>>
>>    The failure:
>>      4806.245395 xe_exec_queue_memory_cat_error   guc_id=2 guc_state=0x3
>>      4806.245592 xe_exec_queue_scheduling_disable guc_id=2 guc_state=0x249
>>                  <5.18s, no scheduling_done for any guc_id>
>>      4811.425872 xe_sched_job_timedout            guc_id=2 guc_state=0x200
>>
>>    guc_state=0x249 is REGISTERED|PENDING_DISABLE|RESET|BANNED. The RESET bit
>>    confirms the exec_queue_reset() branch was taken and the disable was sent
>>    regardless. scheduling_done never fires, so handle_sched_done() never runs
>>    and the acknowledgement genuinely never arrives.
>>
>>    With the fix, scheduling_disable is traced at guc_state=0x2d9
>>    (adds DESTROYED|KILLED), i.e. it now originates from
>>    disable_scheduling_deregister() on the normal cleanup path, and GuC
>>    acknowledges it every time.
>>
>> Testing:
>>    igt@xe_exec_reset@multi-queue-cancel               0/10 failures
>>                                                       (baseline 10/10 failures)
>>    igt@xe_exec_reset@multi-queue-cancel-on-secondary  passing
>>    Subtest runtime drops from ~5.2s to ~85ms.
>>
>> Signed-off-by: Jagmeet Randhawa <jagmeet.randhawa@intel.com>
>> ---
>>   drivers/gpu/drm/xe/xe_guc_submit.c | 14 ++++++++++----
>>   1 file changed, 10 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
>> index f3ba8abfc228..03450590042a 100644
>> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
>> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
>> @@ -1609,14 +1609,20 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
>>   		atomic_or(DRM_XE_EXEC_QUEUE_BAN_REASON_GPU_HANG, &q->ban_reason);
>>   	set_exec_queue_banned(q);
>>   
>> +	/*
>> +	 * GuC already reset this context (the CAT error handler treats a CAT
>> +	 * error as an engine reset), so there is nothing left to disable and
>> +	 * SCHED_DONE will never arrive. Skip the handshake rather than waiting
>> +	 * HZ*5 and escalating to a full GT reset.
>> +	 */
>> +	if (exec_queue_reset(primary))
>> +		err = -EIO;
>> +
>>   	/* Kick job / queue off hardware */
>> -	if (!xe_device_is_in_reset(xe) && !wedged &&
>> +	if (!xe_device_is_in_reset(xe) && !wedged && !exec_queue_reset(primary) &&
>>   	    (exec_queue_enabled(primary) || exec_queue_pending_disable(primary))) {
>>   		int ret;
>>   
>> -		if (exec_queue_reset(primary))
>> -			err = -EIO;
>> -
>>   		if (xe_uc_fw_is_running(&guc->fw)) {
>>   			/*
>>   			 * Wait for any pending G2H to flush out before
>> -- 
>> 2.53.0
>>