Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Arvind Yadav" <arvind.yadav@intel.com>, intel-xe@lists.freedesktop.org
Subject: Re: [PATCH] CI_ONLY: Defer TDR execution while submission is suspended
Date: Wed, 18 Nov 2026 10:46:22 +0000	[thread overview]
Message-ID: <sashiko-outbox-220743@kernel.org> (raw)
In-Reply-To: <20261118101543.2210570-1-arvind.yadav@intel.com>

Thank you for your contribution! Sashiko AI review found 4 potential
issues to consider:
- [Medium] drm/xe: broken TDR and submission recovery on xe_uc_suspend()
  failure
- [Medium] drm/xe: work_tdr left disabled on resume and wedge failure
  paths
- [Medium] drm/xe: infinite busy loop in uc_reset_wait() when suspending
  wedged device
- [Low] drm/xe: CI_ONLY commit markers and undocumented
  xe_device_resume() change
It also noticed 1 related pre-existing issue this patch does not
introduce:
- [High] drm/xe: infinite loop in uc_reset_wait() [1]
--

commit dc721c0cb578b8e8efd82320e9f07c85073891f4
Author: Arvind Yadav <arvind.yadav@intel.com>

> CI_ONLY: Defer TDR execution while submission is suspended
> 
> CI ONLY...
> guc_exec_queue_stop() can ban a queue with an incomplete job and request
> immediate TDR cleanup. During suspend this runs from the PM thread,
> while the timeout worker runs on gt->ordered_wq. The timeout worker can
> therefore capture engine registers and restart scheduler submission
> while GuC submission is being stopped and the device is powering down.
> 
> Disable each queue's timeout work before stopping submission. Do this
> from gt->ordered_wq so any running drm_sched_job_timedout() has finished,
> including the pending-list updates outside the Xe timeout callback.
> Keep the GuC stop and suspend operations on the PM caller's thread.
> 
> Keep timeout work disabled until all queues have been restarted. Also
> disable timeout work on queues created while submission is suspended.
> Re-enable it on suspend failure and teardown abort paths so the new
> disable count cannot prevent cleanup indefinitely.
> 
> Leave the scheduler timeout value unchanged. An immediate cleanup
> request made while timeout work is disabled still sets it to zero,
> so restarting submission re-arms that cleanup without an extra delay.
> 
> Fixes: dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
> Link: https://sashiko.dev/#/bug/linux-449968a5-0b4a-4c5d-acd3-7ced29f0129b
> Signed-off-by: Arvind Yadav <arvind.yadav@intel.com>

[Severity: Low]
This isn't a bug, but the commit subject and body still contain temporary
"CI_ONLY" markers, and the changelog does not mention or explain the
xe_wedge_declare() change in xe_device_resume().

> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
> index 6911c2ac39f82..2de67e4119539 100644
> --- a/drivers/gpu/drm/xe/xe_device.c
> +++ b/drivers/gpu/drm/xe/xe_device.c
> @@ -1509,8 +1509,13 @@ int xe_device_resume(struct xe_device *xe, bool flr)
>  
>  	for_each_gt(gt, xe, id) {
>  		err = xe_gt_resume(gt);
> -		if (err)
> +		if (err) {
> +			/* GuC boot failed before submission could restart. */
> +			if (!flr && err == -ENOEXEC)
> +				xe_wedge_declare(xe, DRM_WEDGE_RECOVERY_REBIND |
> +						 DRM_WEDGE_RECOVERY_BUS_RESET);

[Severity: Medium]
Can declaring the device wedged here while guc->submission_state.stopped
remains 1 cause a later xe_uc_suspend() call to spin forever in
uc_reset_wait()?

When xe_device_resume() fails after xe_uc_suspend() has set
guc->submission_state.stopped to 1, xe_wedge_declare() marks the device
wedged without clearing stopped.  On a subsequent system suspend,
xe_uc_suspend() calls uc_reset_wait(), where xe_guc_submit_reset_wait()
returns immediately because xe_wedged() is true while xe_uc_reset_prepare()
keeps returning 1 because stopped is still set, looping indefinitely
without sleeping.

>  			break;
> +		}
>  	}

[Severity: Medium]
Does work_tdr remain permanently disabled if resume fails along any other
path before xe_guc_submit_start()?

Only !flr && err == -ENOEXEC in xe_device_resume() calls xe_wedge_declare().
If xe_pcode_ready() or xe_bo_restore_early() fails earlier, if
xe_gt_resume() fails with another error such as -ETIMEDOUT or -EIO (or skips
remaining GTs after break), if D3Cold runtime resume fails in
xe_pm_runtime_resume(), or if xe->wedged.mode is
XE_WEDGED_MODE_UPON_ANY_HANG_NO_RESET or XE_WEDGED_MODE_NEVER,
guc_submit_resume_tdr() is never reached.  Pending jobs on banned or killed
queues then never time out or signal their fences, hanging xe_vm_close() and
show_run_ticks().

[ ... ]

> diff --git a/drivers/gpu/drm/xe/xe_uc.c b/drivers/gpu/drm/xe/xe_uc.c
> index b32ba504041c5..aa154800a8815 100644
> --- a/drivers/gpu/drm/xe/xe_uc.c
> +++ b/drivers/gpu/drm/xe/xe_uc.c
> @@ -362,17 +362,41 @@ void xe_uc_suspend_prepare(struct xe_uc *uc)
>  	xe_guc_stop_prepare(&uc->guc);
>  }
>  
> +struct xe_uc_suspend_work {
> +	struct work_struct work;
> +	struct xe_uc *uc;
> +};
> +
> +static void uc_suspend_work(struct work_struct *w)
> +{
> +	struct xe_uc_suspend_work *suspend =
> +		container_of(w, struct xe_uc_suspend_work, work);
> +	xe_guc_submit_suspend_tdr(&suspend->uc->guc);
> +}
> +
>  int xe_uc_suspend(struct xe_uc *uc)
>  {
> +	struct xe_uc_suspend_work suspend = { .uc = uc };
> +	int err;
> +
>  	/* GuC submission not enabled, nothing to do */
>  	if (!xe_device_uc_enabled(uc_to_xe(uc)))
>  		return 0;
>  
>  	uc_reset_wait(uc);

[Severity: High]
This problem wasn't introduced by this patch, but can uc_reset_wait() spin
in an infinite loop [1] when suspending a wedged device?

drivers/gpu/drm/xe/xe_uc.c:uc_reset_wait() {
	int ret;

again:
	xe_guc_reset_wait(&uc->guc);

	ret = xe_uc_reset_prepare(uc);
	if (ret)
		goto again;
}

If the device is already wedged with guc->submission_state.stopped set to 1,
xe_guc_submit_reset_wait() returns immediately because xe_wedged() is true,
while xe_uc_reset_prepare() atomically sets stopped and returns its previous
value of 1, jumping back to again in a tight non-sleeping loop.

> +
> +	/* Drain the complete timeout worker before examining queue state. */
> +	INIT_WORK_ONSTACK(&suspend.work, uc_suspend_work);
> +	queue_work(uc_to_gt(uc)->ordered_wq, &suspend.work);
> +	flush_work(&suspend.work);
> +	destroy_work_on_stack(&suspend.work);
>  
>  	xe_uc_stop(uc);
> +	err = xe_guc_suspend(&uc->guc);
> +	if (err)
> +		xe_guc_submit_resume_tdr(&uc->guc);
>  
> -	return xe_guc_suspend(&uc->guc);
> +	return err;
>  }

[Severity: Medium]
If xe_guc_suspend() fails here, is calling xe_guc_submit_resume_tdr() safe
after uc_reset_wait() and xe_uc_stop() have already stopped submission?

By the time xe_guc_suspend() runs, guc->submission_state.stopped is 1, CT is
stopped, and guc_exec_queue_stop() has stopped each queue's scheduler,
cleared EXEC_QUEUE_STATE_REGISTERED and EXEC_QUEUE_STATE_ENABLED, and banned
user queues with started jobs (setting base.timeout to 0).  If
xe_guc_softreset() fails, GuC firmware is still running, so resuming TDR
immediately runs guc_exec_queue_timedout_job() on banned queues: because
REGISTERED and ENABLED were cleared, it skips disabling scheduling in GuC,
signals fences early, restarts the scheduler against a stopped CT, and
frees the queue's guc_id and LRC locally while GuC is still active.

In addition, idle queues remain stopped so later guc_exec_queue_destroy()
calls never process STATIC_MSG_CLEANUP, guc->submission_state.stopped stays
1 so subsequent suspends or GT resets stall, and on multi-GT systems a
suspend failure on a later GT in xe_device_suspend() or
xe_pm_runtime_suspend() leaves work_tdr disabled on earlier GTs that already
completed xe_gt_suspend().

[1] https://sashiko.dev/#/bug/linux-62eea921-894e-4f73-bbf0-ac02cad95b25

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261118101543.2210570-1-arvind.yadav@intel.com?part=1

      reply	other threads:[~2026-11-18 10:46 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 10:15 [PATCH] CI_ONLY: Defer TDR execution while submission is suspended Arvind Yadav
2026-11-18 10:46 ` sashiko-bot [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-220743@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=arvind.yadav@intel.com \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox