From: Rodrigo Vivi <rodrigo.vivi@intel.com>
To: "Yadav, Sanjay Kumar" <sanjay.kumar.yadav@intel.com>
Cc: <intel-xe@lists.freedesktop.org>,
Matthew Auld <matthew.auld@intel.com>,
Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Subject: Re: [PATCH 1/3] drm/xe: fix job timeout recovery for unstarted jobs and kernel queues
Date: Tue, 9 Jun 2026 11:22:46 -0400 [thread overview]
Message-ID: <aigvxsRgK5hjWkJa@intel.com> (raw)
In-Reply-To: <8dfe4463-476d-4606-850c-c9c34bbf4408@intel.com>
On Thu, Jun 04, 2026 at 12:16:38PM +0530, Yadav, Sanjay Kumar wrote:
>
> On 03-06-2026 20:31, Rodrigo Vivi wrote:
> > Jobs that GuC never scheduled were silently errored out instead of
> > triggering a GT reset. Kernel jobs that exhaust all recovery attempts
> > should wedge the device rather than silently fail, and userspace VM bind
> > queues should stay permanently banned rather than being reset and retried.
> >
> > The queue is banned early in the timeout handler to signal the G2H
> > scheduling-done handler so it wakes the disable-scheduling waiter; without
> > it the waiter sleeps the full 5s timeout. For kernel queues the ban is
> > cleared before rearming so that guc_exec_queue_start() can resubmit jobs
> > after the GT reset — a banned queue would block resubmission and cause an
> > infinite TDR loop.
> >
> > Cc: Matthew Auld <matthew.auld@intel.com>
> > Cc: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
> > Cc: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
> > Assisted-by: GitHub-Copilot:claude-sonnet-4.6
> > Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Tested-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
> LGTM
> Tested with Reproducer IGT and xe_evict - No IGT stuck recovery works
> correctly and subsequent migration tests pass.
Thank you. Since I changed the code a bit on the v2, I didn't keep your
tested by.
Could you please check the v2 as well?
https://lore.kernel.org/intel-xe/20260609144412.244678-3-rodrigo.vivi@intel.com/T/#t
> > ---
> > drivers/gpu/drm/xe/xe_guc_submit.c | 31 +++++++++++++++++++++---------
> > 1 file changed, 22 insertions(+), 9 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> > index ab501513d806..bbccba367626 100644
> > --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> > +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> > @@ -158,6 +158,11 @@ static void set_exec_queue_banned(struct xe_exec_queue *q)
> > atomic_or(EXEC_QUEUE_STATE_BANNED, &q->guc->state);
> > }
> > +static void clear_exec_queue_banned(struct xe_exec_queue *q)
> > +{
> > + atomic_andnot(EXEC_QUEUE_STATE_BANNED, &q->guc->state);
> > +}
> > +
> > static bool exec_queue_suspended(struct xe_exec_queue *q)
> > {
> > return atomic_read(&q->guc->state) & EXEC_QUEUE_STATE_SUSPENDED;
> > @@ -1376,7 +1381,8 @@ static bool check_timeout(struct xe_exec_queue *q, struct xe_sched_job *job)
> > xe_sched_job_seqno(job), xe_sched_job_lrc_seqno(job),
> > q->guc->id);
> > - return xe_sched_invalidate_job(job, 2);
> > + /* GuC never scheduled this job - let the caller trigger a GT reset. */
> > + return true;
> > }
> > ctx_timestamp = lower_32_bits(xe_lrc_timestamp(q->lrc[0]));
> > @@ -1622,19 +1628,26 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
> > q->guc->id, q->flags);
> > /*
> > - * Kernel jobs should never fail, nor should VM jobs if they do
> > - * somethings has gone wrong and the GT needs a reset
> > + * Kernel jobs should never fail permanently. Attempt GT reset and
> > + * resubmit; if karma is exhausted the hardware is unrecoverable so
> > + * wedge the device.
> > + *
> > + * Userspace VM bind queues are banned permanently on timeout
> > +. * No reset is attempted, the ban already
> > + * signals the G2H handler, and the queue stays banned so the job
> > + * errors out cleanly.
> > */
> > - xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_KERNEL,
> > - "Kernel-submitted job timed out\n");
> > - xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_VM && !exec_queue_killed(q),
> > - "VM job timed out on non-killed execqueue\n");
> > - if (!wedged && (q->flags & EXEC_QUEUE_FLAG_KERNEL ||
> > - (q->flags & EXEC_QUEUE_FLAG_VM && !exec_queue_killed(q)))) {
> > + if (!wedged && q->flags & EXEC_QUEUE_FLAG_KERNEL) {
> > if (!xe_sched_invalidate_job(job, 2)) {
> > + clear_exec_queue_banned(q);
> > xe_gt_reset_async(q->gt);
> > goto rearm;
> > }
> > + xe_gt_WARN(q->gt, true, "Kernel-submitted job timed out\n");
> > + xe_device_declare_wedged(gt_to_xe(q->gt));
> > + } else if (!wedged && q->flags & EXEC_QUEUE_FLAG_VM &&
> > + !exec_queue_killed(q)) {
> > + xe_gt_WARN(q->gt, true, "VM job timed out on non-killed execqueue\n");
> > }
> > /* Mark all outstanding jobs as bad, thus completing them */
prev parent reply other threads:[~2026-06-09 15:22 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-03 15:01 [PATCH 1/3] drm/xe: fix job timeout recovery for unstarted jobs and kernel queues Rodrigo Vivi
2026-06-03 15:01 ` [PATCH 2/3] drm/xe/lrc: fix spurious warning when reading context timestamp Rodrigo Vivi
2026-06-03 15:01 ` [PATCH 3/3] drm/xe/lrc: remove engine_id PPHWSP stash Rodrigo Vivi
2026-06-03 15:06 ` ✗ CI.checkpatch: warning for series starting with [1/3] drm/xe: fix job timeout recovery for unstarted jobs and kernel queues Patchwork
2026-06-03 15:08 ` ✓ CI.KUnit: success " Patchwork
2026-06-03 15:52 ` ✓ Xe.CI.BAT: " Patchwork
2026-06-03 16:31 ` [PATCH 1/3] " Ghimiray, Himal Prasad
2026-06-03 19:00 ` Rodrigo Vivi
2026-06-04 6:43 ` Ghimiray, Himal Prasad
2026-06-04 2:32 ` ✓ Xe.CI.FULL: success for series starting with [1/3] " Patchwork
2026-06-04 6:46 ` [PATCH 1/3] " Yadav, Sanjay Kumar
2026-06-09 15:22 ` Rodrigo Vivi [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aigvxsRgK5hjWkJa@intel.com \
--to=rodrigo.vivi@intel.com \
--cc=himal.prasad.ghimiray@intel.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=matthew.auld@intel.com \
--cc=sanjay.kumar.yadav@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox