From: sashiko-bot@kernel.org
To: "Daniel Charles" <daniel.charles@intel.com>,
intel-xe@lists.freedesktop.org
Subject: Re: [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade
Date: Thu, 19 Nov 2026 18:18:50 +0000 [thread overview]
Message-ID: <sashiko-outbox-223764@kernel.org> (raw)
In-Reply-To: <20261119174621.1311154-4-daniel.charles@intel.com>
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] drm/xe: guc_exec_queue_kill() on zero-ref queue in pause_abort
- [High] drm/xe: async kill disable_scheduling() races with
suspend_pending
- [High] drm/xe: primary kill disables multi-queue group without group
cleanup
--
commit 99b38ad21631b92773b295aba24f48100578bda0
Author: Matthew Brost <matthew.brost@intel.com>
drm/xe: Disable scheduling early on FD close to avoid CAT error cascade
This commit adds a static KILL scheduler message queued from
guc_exec_queue_kill() to disable GuC context scheduling immediately
outside of the serialized TDR path.
> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> index 4087a38ad3d66..671852b562105 100644
> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
[ ... ]
> @@ -2087,12 +2093,44 @@ static void __guc_exec_queue_process_msg_cgp_sync(struct xe_sched_msg *msg,
> guc_exec_queue_send_cgp_sync(q, 0);
> }
>
> +static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg,
> + bool bound)
> +{
> + struct xe_exec_queue *q = msg->private_data;
> + struct xe_guc *guc = exec_queue_to_guc(q);
> + int ret;
> +
> + /* At zero references, the queued CLEANUP message handles disabling. */
> + if (!kref_read(&q->refcount) || !bound || exec_queue_wedged(q))
> + return;
> +
> + ret = wait_event_timeout(q->guc->state_wq,
> + (!exec_queue_pending_enable(q) &&
> + !exec_queue_pending_disable(q)) ||
> + xe_guc_read_stopped(guc) ||
> + vf_recovery(guc), HZ * 5);
> + if (!ret && !vf_recovery(guc)) {
> + xe_gt_warn(q->gt, "Pending enable/disable failed to respond on kill, guc_id=%d\n",
> + q->guc->id);
> + xe_gt_reset_async(q->gt);
> + xe_sched_tdr_queue_imm(&q->guc->sched);
> + return;
> + }
> +
> + if (xe_guc_read_stopped(guc) || vf_recovery(guc))
> + return;
> +
> + if (exec_queue_enabled(q))
> + disable_scheduling(q, true);
> +}
[Severity: High]
Can returning immediately after disable_scheduling() here race with a
queued SUSPEND message and the incoming SCHED_DONE G2H?
guc_exec_queue_suspend() checks exec_queue_killed_or_banned_or_wedged()
before __guc_exec_queue_suspend() takes sched->msg_lock, and
guc_exec_queue_kill() sets EXEC_QUEUE_STATE_KILLED before taking
sched->msg_lock, so STATIC_MSG_SUSPEND can be queued right behind
STATIC_MSG_KILL:
CPU0 work_process_msg ct->g2h_wq
guc_exec_queue_suspend()
!killed_or_banned_or_wedged()
(passes)
CPU1: guc_exec_queue_kill()
set_exec_queue_killed()
try_add_msg(KILL)
__guc_exec_queue_suspend()
suspend_pending = true
try_add_msg(SUSPEND)
__guc_exec_queue_process_msg_kill()
disable_scheduling(q, true)
clear_exec_queue_enabled()
set_exec_queue_pending_disable()
__guc_exec_queue_process_msg_suspend()
else if (q->guc->suspend_pending)
set_exec_queue_suspended(q)
handle_sched_done()
suspend_pending &&
exec_queue_suspended()
suspend_fence_signal(q) suspend_fence_signal(q)
<-- trips assert
Because __guc_exec_queue_process_msg_kill() does not wait for its
PENDING_DISABLE to finish, __guc_exec_queue_process_msg_suspend() runs
while KILL's disable is still in flight, enters the unlocked
"else if (q->guc->suspend_pending)" branch, signals the suspend fence
before GuC has actually preempted the context, and races with
handle_sched_done() on q->guc->suspend_pending (tripping
xe_gt_assert(guc_to_gt(guc), q->guc->suspend_pending) in
suspend_fence_signal()). Also, setting EXEC_QUEUE_STATE_SUSPENDED while
KILL's PENDING_DISABLE is in flight causes
guc_exec_queue_revert_pending_state_change() during VF migration recovery
to misattribute the pending disable to SUSPEND and set
q->guc->needs_suspend = true after suspend_pending was already cleared.
[ ... ]
> @@ -2262,8 +2303,21 @@ static void guc_exec_queue_reinit_kernel(struct xe_exec_queue *q)
> xe_sched_reinit(&q->guc->sched);
> }
>
> +static bool guc_exec_queue_try_add_msg(struct xe_exec_queue *q,
> + struct xe_sched_msg *msg,
> + u32 opcode);
> +
> +#define STATIC_MSG_CLEANUP 0
> +#define STATIC_MSG_SUSPEND 1
> +#define STATIC_MSG_RESUME 2
> +#define STATIC_MSG_CGP_SYNC 3
> +#define STATIC_MSG_KILL 4
> static void guc_exec_queue_kill(struct xe_exec_queue *q)
> {
> + struct xe_sched_msg *msg = q->guc->static_msgs + STATIC_MSG_KILL;
> +
> + xe_gt_assert(q->gt, kref_read(&q->refcount));
[Severity: High]
Can xe_guc_submit_pause_abort() hit this assertion and leak a runtime PM
reference when a queue in exec_queue_lookup already has a zero refcount?
When the last reference to an exec queue is dropped,
guc_exec_queue_destroy() only queues STATIC_MSG_CLEANUP; the queue remains
in guc->submission_state.exec_queue_lookup until the async destroy worker
runs guc_exec_queue_fini() -> release_guc_id(). Unlike
xe_guc_submit_wedge(), xe_guc_submit_pause_abort() iterates
exec_queue_lookup and calls guc_exec_queue_kill(q) without checking
kref_read(&q->refcount) (for example in guc_submit_fini() right after
dropping the wedge reference, or during vf_post_migration_abort() and
xe_guc_flr_prepare()):
drivers/gpu/drm/xe/xe_guc_submit.c:xe_guc_submit_pause_abort() {
...
xa_for_each(&guc->submission_state.exec_queue_lookup, index, q) {
...
xe_sched_submission_start(sched);
guc_exec_queue_kill(q);
}
}
Besides tripping the new assertion, queuing STATIC_MSG_KILL on a non-wedged
zero-refcount queue takes a runtime PM reference via MSG_PM_REF in
guc_exec_queue_add_msg(). If the already-in-flight CLEANUP / destroy_async
path reaches guc_exec_queue_fini() -> xe_sched_fini() and cancels
work_process_msg before STATIC_MSG_KILL is processed, that runtime PM
reference is never released.
> trace_xe_exec_queue_kill(q);
> set_exec_queue_killed(q);
> /*
> @@ -2273,6 +2327,27 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q)
> * stays set until the queue is torn down.
> */
> suspend_wait_wake(q);
> +
> + /*
> + * We eagerly send a message to ourselves to disable scheduling, as the
> + * TDR is serialized (i.e., only one exec queue is processed at a time).
> + * If an FD is closed with many exec queues, the TDR can be slower than
> + * the GuC scheduling contexts with invalid page tables, creating a
> + * cascade of CAT errors and engine resets, which is quite slow. Avoid
> + * this by immediately disabling scheduling outside of the TDR.
> + *
> + * Only the primary's own KILL may do this for a multi-queue group.
> + * Group jobs share its ordered submit workqueue and the killed primary
> + * prevents later member jobs from submitting to GuC.
> + */
> + if (!xe_exec_queue_is_multi_queue_secondary(q) && !exec_queue_wedged(q)) {
> + struct xe_gpu_scheduler *sched = &q->guc->sched;
> +
> + xe_sched_msg_lock(sched);
> + guc_exec_queue_try_add_msg(q, msg, KILL);
> + xe_sched_msg_unlock(sched);
> + }
> +
> xe_guc_exec_queue_trigger_cleanup(q);
> }
[Severity: High]
If a multi-queue primary queue is killed while secondary queues still have
in-flight jobs, does disabling scheduling on the primary without triggering
group cleanup leave the secondary jobs stranded?
Every secondary queue holds a kref on the primary queue until the
secondary is finalized, so when the primary is killed (e.g. userspace
destroys the primary queue first, or VRAM page-offline kills bo->q),
kref_read(&primary->refcount) is non-zero and
__guc_exec_queue_process_msg_kill() disables the shared GuC context for the
whole group. However, guc_exec_queue_kill() only calls
xe_guc_exec_queue_trigger_cleanup(primary) rather than
xe_guc_exec_queue_group_trigger_cleanup(primary). When the primary has no
pending jobs of its own, primary's TDR never runs to set group->banned or
schedule TDR on the secondaries: LR secondary jobs stall forever, and
non-LR secondary jobs eventually hit guc_exec_queue_timedout_job() with
skip_timeout_check == false, logging a false GPU hang, capturing a
devcoredump, and triggering a full GT reset via xe_gt_reset_async() for
any secondary job that had not started yet.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119174621.1311154-1-daniel.charles@intel.com?part=3
next prev parent reply other threads:[~2026-11-19 18:19 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 17:46 [PATCH v6 0/4] FD close cleanup for XE Daniel Charles
2026-11-19 17:46 ` [PATCH v6 1/4] drm/xe: Read scheduler message opcode under lock Daniel Charles
2026-11-19 17:46 ` [PATCH v6 2/4] drm/xe: Notify scheduling-state waiters per exec queue Daniel Charles
2026-11-19 18:06 ` sashiko-bot
2026-11-19 19:16 ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade Daniel Charles
2026-11-19 18:18 ` sashiko-bot [this message]
2026-11-19 19:26 ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 4/4] drm/xe: Synchronously wait for queue kill on FD close Daniel Charles
2026-11-19 18:21 ` sashiko-bot
2026-11-19 19:44 ` Cavitt, Jonathan
2026-11-20 0:16 ` ✗ CI.checkpatch: warning for FD close cleanup for XE (rev6) Patchwork
2026-11-20 0:18 ` ✓ CI.KUnit: success " Patchwork
2026-11-20 0:58 ` ✓ Xe.CI.BAT: " Patchwork
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=sashiko-outbox-223764@kernel.org \
--to=sashiko-bot@kernel.org \
--cc=daniel.charles@intel.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=sashiko-reviews@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.