From: sashiko-bot@kernel.org
To: "Daniel Charles" <daniel.charles@intel.com>,
intel-xe@lists.freedesktop.org
Subject: Re: [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade
Date: Thu, 19 Nov 2026 18:18:50 +0000 [thread overview]
Message-ID: <sashiko-outbox-223764@kernel.org> (raw)
In-Reply-To: <20261119174621.1311154-4-daniel.charles@intel.com>
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] drm/xe: guc_exec_queue_kill() on zero-ref queue in pause_abort
- [High] drm/xe: async kill disable_scheduling() races with
suspend_pending
- [High] drm/xe: primary kill disables multi-queue group without group
cleanup
--
commit 99b38ad21631b92773b295aba24f48100578bda0
Author: Matthew Brost <matthew.brost@intel.com>
drm/xe: Disable scheduling early on FD close to avoid CAT error cascade
This commit adds a static KILL scheduler message queued from
guc_exec_queue_kill() to disable GuC context scheduling immediately
outside of the serialized TDR path.
> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> index 4087a38ad3d66..671852b562105 100644
> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
[ ... ]
> @@ -2087,12 +2093,44 @@ static void __guc_exec_queue_process_msg_cgp_sync(struct xe_sched_msg *msg,
> guc_exec_queue_send_cgp_sync(q, 0);
> }
>
> +static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg,
> + bool bound)
> +{
> + struct xe_exec_queue *q = msg->private_data;
> + struct xe_guc *guc = exec_queue_to_guc(q);
> + int ret;
> +
> + /* At zero references, the queued CLEANUP message handles disabling. */
> + if (!kref_read(&q->refcount) || !bound || exec_queue_wedged(q))
> + return;
> +
> + ret = wait_event_timeout(q->guc->state_wq,
> + (!exec_queue_pending_enable(q) &&
> + !exec_queue_pending_disable(q)) ||
> + xe_guc_read_stopped(guc) ||
> + vf_recovery(guc), HZ * 5);
> + if (!ret && !vf_recovery(guc)) {
> + xe_gt_warn(q->gt, "Pending enable/disable failed to respond on kill, guc_id=%d\n",
> + q->guc->id);
> + xe_gt_reset_async(q->gt);
> + xe_sched_tdr_queue_imm(&q->guc->sched);
> + return;
> + }
> +
> + if (xe_guc_read_stopped(guc) || vf_recovery(guc))
> + return;
> +
> + if (exec_queue_enabled(q))
> + disable_scheduling(q, true);
> +}
[Severity: High]
Can returning immediately after disable_scheduling() here race with a
queued SUSPEND message and the incoming SCHED_DONE G2H?
guc_exec_queue_suspend() checks exec_queue_killed_or_banned_or_wedged()
before __guc_exec_queue_suspend() takes sched->msg_lock, and
guc_exec_queue_kill() sets EXEC_QUEUE_STATE_KILLED before taking
sched->msg_lock, so STATIC_MSG_SUSPEND can be queued right behind
STATIC_MSG_KILL:
CPU0 work_process_msg ct->g2h_wq
guc_exec_queue_suspend()
!killed_or_banned_or_wedged()
(passes)
CPU1: guc_exec_queue_kill()
set_exec_queue_killed()
try_add_msg(KILL)
__guc_exec_queue_suspend()
suspend_pending = true
try_add_msg(SUSPEND)
__guc_exec_queue_process_msg_kill()
disable_scheduling(q, true)
clear_exec_queue_enabled()
set_exec_queue_pending_disable()
__guc_exec_queue_process_msg_suspend()
else if (q->guc->suspend_pending)
set_exec_queue_suspended(q)
handle_sched_done()
suspend_pending &&
exec_queue_suspended()
suspend_fence_signal(q) suspend_fence_signal(q)
<-- trips assert
Because __guc_exec_queue_process_msg_kill() does not wait for its
PENDING_DISABLE to finish, __guc_exec_queue_process_msg_suspend() runs
while KILL's disable is still in flight, enters the unlocked
"else if (q->guc->suspend_pending)" branch, signals the suspend fence
before GuC has actually preempted the context, and races with
handle_sched_done() on q->guc->suspend_pending (tripping
xe_gt_assert(guc_to_gt(guc), q->guc->suspend_pending) in
suspend_fence_signal()). Also, setting EXEC_QUEUE_STATE_SUSPENDED while
KILL's PENDING_DISABLE is in flight causes
guc_exec_queue_revert_pending_state_change() during VF migration recovery
to misattribute the pending disable to SUSPEND and set
q->guc->needs_suspend = true after suspend_pending was already cleared.
[ ... ]
> @@ -2262,8 +2303,21 @@ static void guc_exec_queue_reinit_kernel(struct xe_exec_queue *q)
> xe_sched_reinit(&q->guc->sched);
> }
>
> +static bool guc_exec_queue_try_add_msg(struct xe_exec_queue *q,
> + struct xe_sched_msg *msg,
> + u32 opcode);
> +
> +#define STATIC_MSG_CLEANUP 0
> +#define STATIC_MSG_SUSPEND 1
> +#define STATIC_MSG_RESUME 2
> +#define STATIC_MSG_CGP_SYNC 3
> +#define STATIC_MSG_KILL 4
> static void guc_exec_queue_kill(struct xe_exec_queue *q)
> {
> + struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_KILL;
> +
> + xe_gt_assert(q->gt, kref_read(&q->refcount));
[Severity: High]
Can xe_guc_submit_pause_abort() hit this assertion and leak a runtime PM
reference when a queue in exec_queue_lookup already has a zero refcount?
When the last reference to an exec queue is dropped,
guc_exec_queue_destroy() only queues STATIC_MSG_CLEANUP; the queue remains
in guc->submission_state.exec_queue_lookup until the async destroy worker
runs guc_exec_queue_fini() -> release_guc_id(). Unlike
xe_guc_submit_wedge(), xe_guc_submit_pause_abort() iterates
exec_queue_lookup and calls guc_exec_queue_kill(q) without checking
kref_read(&q->refcount) (for example in guc_submit_fini() right after
dropping the wedge reference, or during vf_post_migration_abort() and
xe_guc_flr_prepare()):
drivers/gpu/drm/xe/xe_guc_submit.c:xe_guc_submit_pause_abort() {
...
xa_for_each(&guc->submission_state.exec_queue_lookup, index, q) {
...
xe_sched_submission_start(sched);
guc_exec_queue_kill(q);
}
}
Besides tripping the new assertion, queuing STATIC_MSG_KILL on a non-wedged
zero-refcount queue takes a runtime PM reference via MSG_PM_REF in
guc_exec_queue_add_msg(). If the already-in-flight CLEANUP / destroy_async
path reaches guc_exec_queue_fini() -> xe_sched_fini() and cancels
work_process_msg before STATIC_MSG_KILL is processed, that runtime PM
reference is never released.
> trace_xe_exec_queue_kill(q);
> set_exec_queue_killed(q);
> /*
> @@ -2273,6 +2327,27 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q)
> * stays set until the queue is torn down.
> */
> suspend_wait_wake(q);
> +
> + /*
> + * We eagerly send a message to ourselves to disable scheduling, as the
> + * TDR is serialized (i.e., only one exec queue is processed at a time).
> + * If an FD is closed with many exec queues, the TDR can be slower than
> + * the GuC scheduling contexts with invalid page tables, creating a
> + * cascade of CAT errors and engine resets, which is quite slow. Avoid
> + * this by immediately disabling scheduling outside of the TDR.
> + *
> + * Only the primary's own KILL may do this for a multi-queue group.
> + * Group jobs share its ordered submit workqueue and the killed primary
> + * prevents later member jobs from submitting to GuC.
> + */
> + if (!xe_exec_queue_is_multi_queue_secondary(q) && !exec_queue_wedged(q)) {
> + struct xe_gpu_scheduler *sched = &q->guc->sched;
> +
> + xe_sched_msg_lock(sched);
> + guc_exec_queue_try_add_msg(q, msg, KILL);
> + xe_sched_msg_unlock(sched);
> + }
> +
> xe_guc_exec_queue_trigger_cleanup(q);
> }
[Severity: High]
If a multi-queue primary queue is killed while secondary queues still have
in-flight jobs, does disabling scheduling on the primary without triggering
group cleanup leave the secondary jobs stranded?
Every secondary queue holds a kref on the primary queue until the
secondary is finalized, so when the primary is killed (e.g. userspace
destroys the primary queue first, or VRAM page-offline kills bo->q),
kref_read(&primary->refcount) is non-zero and
__guc_exec_queue_process_msg_kill() disables the shared GuC context for the
whole group. However, guc_exec_queue_kill() only calls
xe_guc_exec_queue_trigger_cleanup(primary) rather than
xe_guc_exec_queue_group_trigger_cleanup(primary). When the primary has no
pending jobs of its own, primary's TDR never runs to set group->banned or
schedule TDR on the secondaries: LR secondary jobs stall forever, and
non-LR secondary jobs eventually hit guc_exec_queue_timedout_job() with
skip_timeout_check == false, logging a false GPU hang, capturing a
devcoredump, and triggering a full GT reset via xe_gt_reset_async() for
any secondary job that had not started yet.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119174621.1311154-1-daniel.charles@intel.com?part=3
next prev parent reply other threads:[~2026-11-19 18:19 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 17:46 [PATCH v6 0/4] FD close cleanup for XE Daniel Charles
2026-11-19 17:46 ` [PATCH v6 1/4] drm/xe: Read scheduler message opcode under lock Daniel Charles
2026-11-19 17:46 ` [PATCH v6 2/4] drm/xe: Notify scheduling-state waiters per exec queue Daniel Charles
2026-11-19 18:06 ` sashiko-bot
2026-11-19 19:16 ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade Daniel Charles
2026-11-19 18:18 ` sashiko-bot [this message]
2026-11-19 19:26 ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 4/4] drm/xe: Synchronously wait for queue kill on FD close Daniel Charles
2026-11-19 18:21 ` sashiko-bot
2026-11-19 19:44 ` Cavitt, Jonathan
2026-11-20 0:16 ` ✗ CI.checkpatch: warning for FD close cleanup for XE (rev6) Patchwork
2026-11-20 0:18 ` ✓ CI.KUnit: success " Patchwork
2026-11-20 0:58 ` ✓ Xe.CI.BAT: " Patchwork
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=sashiko-outbox-223764@kernel.org \
--to=sashiko-bot@kernel.org \
--cc=daniel.charles@intel.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=sashiko-reviews@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox