Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Daniel Charles" <daniel.charles@intel.com>,
	intel-xe@lists.freedesktop.org
Subject: Re: [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade
Date: Thu, 19 Nov 2026 18:18:50 +0000	[thread overview]
Message-ID: <sashiko-outbox-223764@kernel.org> (raw)
In-Reply-To: <20261119174621.1311154-4-daniel.charles@intel.com>

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] drm/xe: guc_exec_queue_kill() on zero-ref queue in pause_abort
- [High] drm/xe: async kill disable_scheduling() races with
  suspend_pending
- [High] drm/xe: primary kill disables multi-queue group without group
  cleanup
--

commit 99b38ad21631b92773b295aba24f48100578bda0
Author: Matthew Brost <matthew.brost@intel.com>

drm/xe: Disable scheduling early on FD close to avoid CAT error cascade

This commit adds a static KILL scheduler message queued from
guc_exec_queue_kill() to disable GuC context scheduling immediately
outside of the serialized TDR path.

> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> index 4087a38ad3d66..671852b562105 100644
> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c

[ ... ]

> @@ -2087,12 +2093,44 @@ static void __guc_exec_queue_process_msg_cgp_sync(struct xe_sched_msg *msg,
>  		guc_exec_queue_send_cgp_sync(q, 0);
>  }
>  
> +static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg,
> +					      bool bound)
> +{
> +	struct xe_exec_queue *q = msg->private_data;
> +	struct xe_guc *guc = exec_queue_to_guc(q);
> +	int ret;
> +
> +	/* At zero references, the queued CLEANUP message handles disabling. */
> +	if (!kref_read(&q->refcount) || !bound || exec_queue_wedged(q))
> +		return;
> +
> +	ret = wait_event_timeout(q->guc->state_wq,
> +				 (!exec_queue_pending_enable(q) &&
> +				  !exec_queue_pending_disable(q)) ||
> +				 xe_guc_read_stopped(guc) ||
> +				 vf_recovery(guc), HZ * 5);
> +	if (!ret && !vf_recovery(guc)) {
> +		xe_gt_warn(q->gt, "Pending enable/disable failed to respond on kill, guc_id=%d\n",
> +			   q->guc->id);
> +		xe_gt_reset_async(q->gt);
> +		xe_sched_tdr_queue_imm(&q->guc->sched);
> +		return;
> +	}
> +
> +	if (xe_guc_read_stopped(guc) || vf_recovery(guc))
> +		return;
> +
> +	if (exec_queue_enabled(q))
> +		disable_scheduling(q, true);
> +}

[Severity: High]
Can returning immediately after disable_scheduling() here race with a
queued SUSPEND message and the incoming SCHED_DONE G2H?

guc_exec_queue_suspend() checks exec_queue_killed_or_banned_or_wedged()
before __guc_exec_queue_suspend() takes sched->msg_lock, and
guc_exec_queue_kill() sets EXEC_QUEUE_STATE_KILLED before taking
sched->msg_lock, so STATIC_MSG_SUSPEND can be queued right behind
STATIC_MSG_KILL:

CPU0                              work_process_msg                  ct->g2h_wq
guc_exec_queue_suspend()
  !killed_or_banned_or_wedged()
  (passes)
CPU1: guc_exec_queue_kill()
  set_exec_queue_killed()
  try_add_msg(KILL)
__guc_exec_queue_suspend()
  suspend_pending = true
  try_add_msg(SUSPEND)
                                  __guc_exec_queue_process_msg_kill()
                                    disable_scheduling(q, true)
                                      clear_exec_queue_enabled()
                                      set_exec_queue_pending_disable()
                                  __guc_exec_queue_process_msg_suspend()
                                    else if (q->guc->suspend_pending)
                                      set_exec_queue_suspended(q)
                                                                    handle_sched_done()
                                                                      suspend_pending &&
                                                                      exec_queue_suspended()
                                      suspend_fence_signal(q)         suspend_fence_signal(q)
                                                                        <-- trips assert

Because __guc_exec_queue_process_msg_kill() does not wait for its
PENDING_DISABLE to finish, __guc_exec_queue_process_msg_suspend() runs
while KILL's disable is still in flight, enters the unlocked
"else if (q->guc->suspend_pending)" branch, signals the suspend fence
before GuC has actually preempted the context, and races with
handle_sched_done() on q->guc->suspend_pending (tripping
xe_gt_assert(guc_to_gt(guc), q->guc->suspend_pending) in
suspend_fence_signal()).  Also, setting EXEC_QUEUE_STATE_SUSPENDED while
KILL's PENDING_DISABLE is in flight causes
guc_exec_queue_revert_pending_state_change() during VF migration recovery
to misattribute the pending disable to SUSPEND and set
q->guc->needs_suspend = true after suspend_pending was already cleared.

[ ... ]

> @@ -2262,8 +2303,21 @@ static void guc_exec_queue_reinit_kernel(struct xe_exec_queue *q)
>  	xe_sched_reinit(&q->guc->sched);
>  }
>  
> +static bool guc_exec_queue_try_add_msg(struct xe_exec_queue *q,
> +				       struct xe_sched_msg *msg,
> +				       u32 opcode);
> +
> +#define STATIC_MSG_CLEANUP	0
> +#define STATIC_MSG_SUSPEND	1
> +#define STATIC_MSG_RESUME	2
> +#define STATIC_MSG_CGP_SYNC	3
> +#define STATIC_MSG_KILL		4
>  static void guc_exec_queue_kill(struct xe_exec_queue *q)
>  {
> +	struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_KILL;
> +
> +	xe_gt_assert(q->gt, kref_read(&q->refcount));

[Severity: High]
Can xe_guc_submit_pause_abort() hit this assertion and leak a runtime PM
reference when a queue in exec_queue_lookup already has a zero refcount?

When the last reference to an exec queue is dropped,
guc_exec_queue_destroy() only queues STATIC_MSG_CLEANUP; the queue remains
in guc->submission_state.exec_queue_lookup until the async destroy worker
runs guc_exec_queue_fini() -> release_guc_id().  Unlike
xe_guc_submit_wedge(), xe_guc_submit_pause_abort() iterates
exec_queue_lookup and calls guc_exec_queue_kill(q) without checking
kref_read(&q->refcount) (for example in guc_submit_fini() right after
dropping the wedge reference, or during vf_post_migration_abort() and
xe_guc_flr_prepare()):

drivers/gpu/drm/xe/xe_guc_submit.c:xe_guc_submit_pause_abort() {
	...
	xa_for_each(&guc->submission_state.exec_queue_lookup, index, q) {
		...
		xe_sched_submission_start(sched);
		guc_exec_queue_kill(q);
	}
}

Besides tripping the new assertion, queuing STATIC_MSG_KILL on a non-wedged
zero-refcount queue takes a runtime PM reference via MSG_PM_REF in
guc_exec_queue_add_msg().  If the already-in-flight CLEANUP / destroy_async
path reaches guc_exec_queue_fini() -> xe_sched_fini() and cancels
work_process_msg before STATIC_MSG_KILL is processed, that runtime PM
reference is never released.

>  	trace_xe_exec_queue_kill(q);
>  	set_exec_queue_killed(q);
>  	/*
> @@ -2273,6 +2327,27 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q)
>  	 * stays set until the queue is torn down.
>  	 */
>  	suspend_wait_wake(q);
> +
> +	/*
> +	 * We eagerly send a message to ourselves to disable scheduling, as the
> +	 * TDR is serialized (i.e., only one exec queue is processed at a time).
> +	 * If an FD is closed with many exec queues, the TDR can be slower than
> +	 * the GuC scheduling contexts with invalid page tables, creating a
> +	 * cascade of CAT errors and engine resets, which is quite slow. Avoid
> +	 * this by immediately disabling scheduling outside of the TDR.
> +	 *
> +	 * Only the primary's own KILL may do this for a multi-queue group.
> +	 * Group jobs share its ordered submit workqueue and the killed primary
> +	 * prevents later member jobs from submitting to GuC.
> +	 */
> +	if (!xe_exec_queue_is_multi_queue_secondary(q) && !exec_queue_wedged(q)) {
> +		struct xe_gpu_scheduler *sched = &q->guc->sched;
> +
> +		xe_sched_msg_lock(sched);
> +		guc_exec_queue_try_add_msg(q, msg, KILL);
> +		xe_sched_msg_unlock(sched);
> +	}
> +
>  	xe_guc_exec_queue_trigger_cleanup(q);
>  }

[Severity: High]
If a multi-queue primary queue is killed while secondary queues still have
in-flight jobs, does disabling scheduling on the primary without triggering
group cleanup leave the secondary jobs stranded?

Every secondary queue holds a kref on the primary queue until the
secondary is finalized, so when the primary is killed (e.g. userspace
destroys the primary queue first, or VRAM page-offline kills bo->q),
kref_read(&primary->refcount) is non-zero and
__guc_exec_queue_process_msg_kill() disables the shared GuC context for the
whole group.  However, guc_exec_queue_kill() only calls
xe_guc_exec_queue_trigger_cleanup(primary) rather than
xe_guc_exec_queue_group_trigger_cleanup(primary).  When the primary has no
pending jobs of its own, primary's TDR never runs to set group->banned or
schedule TDR on the secondaries: LR secondary jobs stall forever, and
non-LR secondary jobs eventually hit guc_exec_queue_timedout_job() with
skip_timeout_check == false, logging a false GPU hang, capturing a
devcoredump, and triggering a full GT reset via xe_gt_reset_async() for
any secondary job that had not started yet.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261119174621.1311154-1-daniel.charles@intel.com?part=3

  reply	other threads:[~2026-11-19 18:19 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 17:46 [PATCH v6 0/4] FD close cleanup for XE Daniel Charles
2026-11-19 17:46 ` [PATCH v6 1/4] drm/xe: Read scheduler message opcode under lock Daniel Charles
2026-11-19 17:46 ` [PATCH v6 2/4] drm/xe: Notify scheduling-state waiters per exec queue Daniel Charles
2026-11-19 18:06   ` sashiko-bot
2026-11-19 19:16     ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade Daniel Charles
2026-11-19 18:18   ` sashiko-bot [this message]
2026-11-19 19:26     ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 4/4] drm/xe: Synchronously wait for queue kill on FD close Daniel Charles
2026-11-19 18:21   ` sashiko-bot
2026-11-19 19:44   ` Cavitt, Jonathan
2026-11-20  0:16 ` ✗ CI.checkpatch: warning for FD close cleanup for XE (rev6) Patchwork
2026-11-20  0:18 ` ✓ CI.KUnit: success " Patchwork
2026-11-20  0:58 ` ✓ Xe.CI.BAT: " Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-223764@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=daniel.charles@intel.com \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox