[RFC PATCH 1/3] drm/xe: skip banning kernel migration queue on TDR timeout

[RFC PATCH 1/3] drm/xe: skip banning kernel migration queue on TDR timeout

[Sanjay Yadav]

guc_exec_queue_timedout_job() unconditionally bans the queue once a
job times out. For the kernel migration queue this is fatal — once
banned, no page table migrations can complete and the GPU is
effectively dead until driver reload.

The submission is already stopped and the timed-out job is erred out,
so banning is not needed for correctness. GT reset handles the actual
hardware recovery. Skip banning for kernel queues so they remain
available after reset.

Fixes: bb63e7257e63 ("drm/xe: Avoid toggling schedule state to check LRC timestamp in TDR")
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Assisted-by: Claude:claude-opus-4.6
Suggested-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
---
 drivers/gpu/drm/xe/xe_guc_submit.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
index ab501513d806..e6ad57cbbf0e 100644
--- a/drivers/gpu/drm/xe/xe_guc_submit.c
+++ b/drivers/gpu/drm/xe/xe_guc_submit.c
@@ -1543,7 +1543,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
 	if (!exec_queue_killed(q))
 		wedged = guc_submit_hint_wedged(exec_queue_to_guc(q));
 
-	set_exec_queue_banned(q);
+	if (!(q->flags & EXEC_QUEUE_FLAG_KERNEL))
+		set_exec_queue_banned(q);
 
 	/* Kick job / queue off hardware */
 	if (!wedged && (exec_queue_enabled(primary) ||
-- 
2.52.0

[RFC PATCH 2/3] drm/sched: fix drm_sched_tdr_queue_imm to not corrupt timeout value

[Sanjay Yadav]

drm_sched_tdr_queue_imm() sets sched->timeout to 0 and never restores
it. This breaks all future TDR timers — jobs get timed out instantly
before they even start running on hardware.

Use mod_delayed_work() directly to fire the TDR worker immediately
without modifying the timeout field. This preserves the original
timeout value for subsequent job submissions.

Fixes: 8ec5a4e5ce97 ("drm/xe: Resume TDR after GT reset")
Cc: <stable@vger.kernel.org> # v6.13+
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Assisted-by: Claude:claude-opus-4.6
Suggested-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
---
 drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
index 818d3d4434b5..be144e244745 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -212,8 +212,8 @@ static void drm_sched_start_timeout_unlocked(struct drm_gpu_scheduler *sched)
 void drm_sched_tdr_queue_imm(struct drm_gpu_scheduler *sched)
 {
 	spin_lock(&sched->job_list_lock);
-	sched->timeout = 0;
-	drm_sched_start_timeout(sched);
+	if (!list_empty(&sched->pending_list))
+		mod_delayed_work(sched->timeout_wq, &sched->work_tdr, 0);
 	spin_unlock(&sched->job_list_lock);
 }
 EXPORT_SYMBOL(drm_sched_tdr_queue_imm);
-- 
2.52.0

Re: [RFC PATCH 2/3] drm/sched: fix drm_sched_tdr_queue_imm to not corrupt timeout value

[Rodrigo Vivi]

On Wed, Jun 03, 2026 at 05:36:41PM +0530, Sanjay Yadav wrote:
> drm_sched_tdr_queue_imm() sets sched->timeout to 0 and never restores
> it. This breaks all future TDR timers — jobs get timed out instantly
> before they even start running on hardware.
> 
> Use mod_delayed_work() directly to fire the TDR worker immediately
> without modifying the timeout field. This preserves the original
> timeout value for subsequent job submissions.
> 
> Fixes: 8ec5a4e5ce97 ("drm/xe: Resume TDR after GT reset")
> Cc: <stable@vger.kernel.org> # v6.13+
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Assisted-by: Claude:claude-opus-4.6
> Suggested-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
> Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
> ---
>  drivers/gpu/drm/scheduler/sched_main.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
> index 818d3d4434b5..be144e244745 100644
> --- a/drivers/gpu/drm/scheduler/sched_main.c
> +++ b/drivers/gpu/drm/scheduler/sched_main.c
> @@ -212,8 +212,8 @@ static void drm_sched_start_timeout_unlocked(struct drm_gpu_scheduler *sched)
>  void drm_sched_tdr_queue_imm(struct drm_gpu_scheduler *sched)
>  {
>  	spin_lock(&sched->job_list_lock);
> -	sched->timeout = 0;
> -	drm_sched_start_timeout(sched);
> +	if (!list_empty(&sched->pending_list))
> +		mod_delayed_work(sched->timeout_wq, &sched->work_tdr, 0);

No, please. If there's something wrong with the timeout clear we need
to get that fixed at the drm layer instead of doing our own.

>  	spin_unlock(&sched->job_list_lock);
>  }
>  EXPORT_SYMBOL(drm_sched_tdr_queue_imm);
> -- 
> 2.52.0
> 

[RFC PATCH 3/3] drm/xe: don't cancel other pending jobs on kernel migration queue timeout

[Sanjay Yadav]

The kernel migration queue scheduler is shared across all VMs on a tile.
When a job times out and GT recovery is exhausted (karma > threshold),
guc_exec_queue_timedout_job() falls through to error out the job and
cancel all remaining pending jobs via drm_sched_for_each_pending_job().

After a GT reset, the hardware is recovered and those pending jobs can
run fine. Skip cancellation for kernel queues so that other VMs' migration
operations can complete after GT reset recovers the engine.

Fixes: 95f27831ee3c ("drm/xe: Stop abusing DRM scheduler internals")
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Assisted-by: Claude:claude-opus-4.6
Suggested-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
---
 drivers/gpu/drm/xe/xe_guc_submit.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
index e6ad57cbbf0e..dd3ae58019b1 100644
--- a/drivers/gpu/drm/xe/xe_guc_submit.c
+++ b/drivers/gpu/drm/xe/xe_guc_submit.c
@@ -1640,8 +1640,15 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
 
 	/* Mark all outstanding jobs as bad, thus completing them */
 	xe_sched_job_set_error(job, err);
-	drm_sched_for_each_pending_job(tmp_job, &sched->base, NULL)
-		xe_sched_job_set_error(to_xe_sched_job(tmp_job), -ECANCELED);
+
+	/*
+	 * For kernel queues (migration), don't cancel other pending jobs.
+	 * They belong to different VMs sharing the same scheduler and will
+	 * be resubmitted after GT reset recovers the hardware.
+	 */
+	if (!(q->flags & EXEC_QUEUE_FLAG_KERNEL))
+		drm_sched_for_each_pending_job(tmp_job, &sched->base, NULL)
+			xe_sched_job_set_error(to_xe_sched_job(tmp_job), -ECANCELED);
 
 	if (xe_exec_queue_is_multi_queue(q)) {
 		xe_guc_exec_queue_group_start(q);
-- 
2.52.0

Re: [RFC PATCH 1/3] drm/xe: skip banning kernel migration queue on TDR timeout

[Matthew Auld]

On 03/06/2026 13:06, Sanjay Yadav wrote:
> guc_exec_queue_timedout_job() unconditionally bans the queue once a
> job times out. For the kernel migration queue this is fatal — once
> banned, no page table migrations can complete and the GPU is
> effectively dead until driver reload.
> 
> The submission is already stopped and the timed-out job is erred out,
> so banning is not needed for correctness. GT reset handles the actual
> hardware recovery. Skip banning for kernel queues so they remain
> available after reset.

Is wedging/reload not the more correct thing here? Kernel job is usually 
performing critical and potentially security sensitive work, like memory 
clearing, migrations, binding etc. If something goes wrong in one of 
those jobs, how should we go about recovering from that? Is driver 
reload/wedge not the more appropriate thing here, or least would need a 
more elaborate recovery?

For example, memclear get nuked, what stops the user from accessing 
uncleared memory later? Or a migration/copy/save/restore/ job gets 
nuked, from correctness pov how do we recover from that?

> 
> Fixes: bb63e7257e63 ("drm/xe: Avoid toggling schedule state to check LRC timestamp in TDR")
> Cc: Matthew Brost <matthew.brost@intel.com>
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Assisted-by: Claude:claude-opus-4.6
> Suggested-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
> Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
> ---
>   drivers/gpu/drm/xe/xe_guc_submit.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> index ab501513d806..e6ad57cbbf0e 100644
> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> @@ -1543,7 +1543,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
>   	if (!exec_queue_killed(q))
>   		wedged = guc_submit_hint_wedged(exec_queue_to_guc(q));
>   
> -	set_exec_queue_banned(q);
> +	if (!(q->flags & EXEC_QUEUE_FLAG_KERNEL))
> +		set_exec_queue_banned(q);
>   
>   	/* Kick job / queue off hardware */
>   	if (!wedged && (exec_queue_enabled(primary) ||

Re: [RFC PATCH 1/3] drm/xe: skip banning kernel migration queue on TDR timeout

[Rodrigo Vivi]

On Wed, Jun 03, 2026 at 01:42:25PM +0100, Matthew Auld wrote:
> On 03/06/2026 13:06, Sanjay Yadav wrote:
> > guc_exec_queue_timedout_job() unconditionally bans the queue once a
> > job times out. For the kernel migration queue this is fatal — once
> > banned, no page table migrations can complete and the GPU is
> > effectively dead until driver reload.
> > 
> > The submission is already stopped and the timed-out job is erred out,
> > so banning is not needed for correctness. GT reset handles the actual
> > hardware recovery. Skip banning for kernel queues so they remain
> > available after reset.
> 
> Is wedging/reload not the more correct thing here? Kernel job is usually
> performing critical and potentially security sensitive work, like memory
> clearing, migrations, binding etc. If something goes wrong in one of those
> jobs, how should we go about recovering from that? Is driver reload/wedge
> not the more appropriate thing here, or least would need a more elaborate
> recovery?
> 
> For example, memclear get nuked, what stops the user from accessing
> uncleared memory later? Or a migration/copy/save/restore/ job gets nuked,
> from correctness pov how do we recover from that?

I agree with Matt here something is off. we cannot blindly skip these
kernel submission cases... (This and the other patch in this series)

> 
> > 
> > Fixes: bb63e7257e63 ("drm/xe: Avoid toggling schedule state to check LRC timestamp in TDR")
> > Cc: Matthew Brost <matthew.brost@intel.com>
> > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > Assisted-by: Claude:claude-opus-4.6
> > Suggested-by: Himal Prasad Ghimiray <himal.prasad.ghimiray@intel.com>
> > Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
> > ---
> >   drivers/gpu/drm/xe/xe_guc_submit.c | 3 ++-
> >   1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> > index ab501513d806..e6ad57cbbf0e 100644
> > --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> > +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> > @@ -1543,7 +1543,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
> >   	if (!exec_queue_killed(q))
> >   		wedged = guc_submit_hint_wedged(exec_queue_to_guc(q));
> > -	set_exec_queue_banned(q);
> > +	if (!(q->flags & EXEC_QUEUE_FLAG_KERNEL))
> > +		set_exec_queue_banned(q);
> >   	/* Kick job / queue off hardware */
> >   	if (!wedged && (exec_queue_enabled(primary) ||
> 

Re: [RFC PATCH 1/3] drm/xe: skip banning kernel migration queue on TDR timeout

[Hellstrom, Thomas]

On Wed, 2026-06-03 at 09:52 -0400, Rodrigo Vivi wrote:
> On Wed, Jun 03, 2026 at 01:42:25PM +0100, Matthew Auld wrote:
> > On 03/06/2026 13:06, Sanjay Yadav wrote:
> > > guc_exec_queue_timedout_job() unconditionally bans the queue once
> > > a
> > > job times out. For the kernel migration queue this is fatal —
> > > once
> > > banned, no page table migrations can complete and the GPU is
> > > effectively dead until driver reload.
> > > 
> > > The submission is already stopped and the timed-out job is erred
> > > out,
> > > so banning is not needed for correctness. GT reset handles the
> > > actual
> > > hardware recovery. Skip banning for kernel queues so they remain
> > > available after reset.
> > 
> > Is wedging/reload not the more correct thing here? Kernel job is
> > usually
> > performing critical and potentially security sensitive work, like
> > memory
> > clearing, migrations, binding etc. If something goes wrong in one
> > of those
> > jobs, how should we go about recovering from that? Is driver
> > reload/wedge
> > not the more appropriate thing here, or least would need a more
> > elaborate
> > recovery?
> > 
> > For example, memclear get nuked, what stops the user from accessing
> > uncleared memory later? Or a migration/copy/save/restore/ job gets
> > nuked,
> > from correctness pov how do we recover from that?
> 
> I agree with Matt here something is off. we cannot blindly skip these
> kernel submission cases... (This and the other patch in this series)

+1

Since I'm not 100% up-to-date with the scheduling I asked Claude to
look for any software-propagated errors or apparent SW bugs that could
cause the migration jobs to fail (assuming that our batches are OK
ofc), and it couldn't find one. It seems like the drm scheduler handles
the dependency timeout case correctly.

If there are recoverable resets that might occasionally spill over to
the migration queue we should figure that out and in that case consider
implementing async SW clearing / migration for the failed jobs (like
i915) to avoid wedging. But I'd rather see us not doing that if it's
not strictly necessary with a proven reproducer.

Thanks,
Thomas


> 
> > 
> > > 
> > > Fixes: bb63e7257e63 ("drm/xe: Avoid toggling schedule state to
> > > check LRC timestamp in TDR")
> > > Cc: Matthew Brost <matthew.brost@intel.com>
> > > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > > Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > > Assisted-by: Claude:claude-opus-4.6
> > > Suggested-by: Himal Prasad Ghimiray
> > > <himal.prasad.ghimiray@intel.com>
> > > Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
> > > ---
> > >   drivers/gpu/drm/xe/xe_guc_submit.c | 3 ++-
> > >   1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c
> > > b/drivers/gpu/drm/xe/xe_guc_submit.c
> > > index ab501513d806..e6ad57cbbf0e 100644
> > > --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> > > +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> > > @@ -1543,7 +1543,8 @@ guc_exec_queue_timedout_job(struct
> > > drm_sched_job *drm_job)
> > >   	if (!exec_queue_killed(q))
> > >   		wedged =
> > > guc_submit_hint_wedged(exec_queue_to_guc(q));
> > > -	set_exec_queue_banned(q);
> > > +	if (!(q->flags & EXEC_QUEUE_FLAG_KERNEL))
> > > +		set_exec_queue_banned(q);
> > >   	/* Kick job / queue off hardware */
> > >   	if (!wedged && (exec_queue_enabled(primary) ||
> >