From: Daniel Charles <daniel.charles@intel.com>
To: "Cavitt, Jonathan" <jonathan.cavitt@intel.com>,
"sashiko-reviews@lists.linux.dev"
<sashiko-reviews@lists.linux.dev>,
"intel-xe@lists.freedesktop.org" <intel-xe@lists.freedesktop.org>
Subject: Re: [PATCH v4 3/3] drm/xe: Synchronously wait for queue kill on FD close
Date: Tue, 17 Nov 2026 20:37:51 -0800 [thread overview]
Message-ID: <35841b3a-82cb-406a-a608-e28c8178e34a@intel.com> (raw)
In-Reply-To: <CH0PR11MB53957B60020D97D77FA5B50BE57B2@CH0PR11MB5395.namprd11.prod.outlook.com>
On 11/17/2026 3:52 PM, Cavitt, Jonathan wrote:
> -----Original Message-----
> From: Intel-xe <intel-xe-bounces@lists.freedesktop.org> On Behalf Of sashiko-bot@kernel.org
> Sent: Friday, October 16, 2026 1:31 PM
> To: Charles, Daniel <daniel.charles@intel.com>; intel-xe@lists.freedesktop.org
> Subject: Re: [PATCH v4 3/3] drm/xe: Synchronously wait for queue kill on FD close
>> Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
>> - [High] drm/xe: premature return in guc_exec_queue_kill_sync() before disable
>> - [High] drm/xe: missing ct.wq wakeup in handle_sched_done() on queue kill
>> - [High] drm/xe: __guc_exec_queue_process_msg_kill() disables live multi-queue primary
>> - [Medium] drm/xe: missing unplug check in guc_exec_queue_kill_sync()
>> --
>>
>> commit 79b6d795ea1a2b4401f8f9f859fbe92a958b14ab
>> Author: Daniel Charles <daniel.charles@intel.com>
>>
>> drm/xe: Synchronously wait for queue kill on FD close
>>
>> This commit splits FD close exec queue teardown into two passes so that all
>> queues are killed first, and then waits synchronously via
>> xe_exec_queue_kill_sync() on a dedicated kill_wq before dropping queue
>> references and tearing down VMs.
>>
>>> diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
>>> index 4caf86be8c29c..537cd7800f267 100644
>>> --- a/drivers/gpu/drm/xe/xe_device.c
>>> +++ b/drivers/gpu/drm/xe/xe_device.c
>>> @@ -190,8 +190,13 @@ static void xe_file_close(struct drm_device *dev, struct drm_file *file)
>>> if (q->vm && q->hwe->hw_engine_group)
>>> xe_hw_engine_group_del_exec_queue(q->hwe->hw_engine_group, q);
>>> xe_exec_queue_kill(q);
>>> + }
>>> +
>>> + xa_for_each(&xef->exec_queue.xa, idx, q) {
>>> + xe_exec_queue_kill_sync(q);
>>> xe_exec_queue_put(q);
>>> }
>> [Severity: High]
>> Because xe_file_close() now kills all queues in a first pass before dropping
>> any queue references in the second pass, kref_read(&q->refcount) remains
>> non-zero when STATIC_MSG_KILL is processed for every secondary queue in a
>> multi-queue group.
>>
>> In __guc_exec_queue_process_msg_kill(), primary is resolved unconditionally via
>> xe_exec_queue_multi_queue_primary(q):
>>
>> drivers/gpu/drm/xe/xe_guc_submit.c:__guc_exec_queue_process_msg_kill() {
>> struct xe_exec_queue *q = msg->private_data;
>> struct xe_exec_queue *primary = xe_exec_queue_multi_queue_primary(q);
>> ...
>> if (!kref_read(&q->refcount) || !bound || exec_queue_wedged(q))
>> return;
>>
>> ret = wait_event_timeout(guc->ct.wq,
>> (!exec_queue_pending_enable(primary) &&
>> !exec_queue_pending_disable(primary)) ||
>> xe_guc_read_stopped(guc) ||
>> vf_recovery(guc), HZ * 5);
>> ...
>> if (exec_queue_enabled(primary))
>> disable_scheduling(primary, true);
>> }
>>
>> When a single secondary queue is destroyed via xe_exec_queue_destroy_ioctl(),
>> xe_exec_queue_kill(q) queues STATIC_MSG_KILL before xe_exec_queue_put(q) drops
>> the reference. Doesn't __guc_exec_queue_process_msg_kill() then overwrite the
>> live primary queue's preemption timeout to 1 us and disable GuC scheduling on
>> the active primary queue (which can also trip the !exec_queue_pending_disable()
>> assertion in guc_exec_queue_run_job() if the primary queue runs a job)?
> guc_exec_queue_run_job only runs if GuC scheduling is enabled, which this
> hypothetical scenario states is not the case.
>
>> Additionally, during xe_file_close(), after the primary queue's KILL message
>> sets EXEC_QUEUE_STATE_PENDING_DISABLE on primary, each secondary queue's
>> __guc_exec_queue_process_msg_kill() waits on guc->ct.wq for
>> !exec_queue_pending_disable(primary). Since handle_sched_done() only wakes
>> kill_wq for a killed (non-banned, non-destroyed) primary queue, won't the
>> secondary queue's KILL handler stall on guc->ct.wq for the 5-second timeout
>> and trigger a GT reset?
> I think this is handled by TDR?
This is a similar observation as that in patch 2. I'll take a look at
the options and implement one. ct.wq will be addressed down in the last
part from Sashiko.
>>> +
>>> xa_for_each(&xef->vm.xa, idx, vm)
>>> xe_vm_close_and_put(vm);
>>>
>> [ ... ]
>>
>>> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
>>> index 1fbc8cd767675..40537ad12d729 100644
>>> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
>>> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
>> [ ... ]
>>
>>> @@ -2304,6 +2326,28 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q)
>>> xe_guc_exec_queue_trigger_cleanup(q);
>>> }
>>>
>>> +static void guc_exec_queue_kill_sync(struct xe_exec_queue *q)
>>> +{
>>> + struct xe_exec_queue *primary = xe_exec_queue_multi_queue_primary(q);
>>> + struct xe_guc *guc = exec_queue_to_guc(primary);
>>> + long ret;
>>> +
>>> + xe_assert(guc_to_xe(guc), exec_queue_killed(q));
>>> +
>>> + ret = wait_event_timeout(guc->submission_state.kill_wq,
>>> + (!exec_queue_enabled(primary) &&
>>> + !exec_queue_pending_disable(primary) &&
>>> + !exec_queue_pending_enable(primary)) ||
>>> + xe_guc_read_stopped(guc) ||
>>> + vf_recovery(guc) ||
>>> + exec_queue_wedged(primary), HZ * 5);
>> [Severity: High]
>> Can guc_exec_queue_kill_sync() return prematurely before GuC scheduling is
>> actually disabled in the following scenarios?
>>
>> 1. In disable_scheduling() and disable_scheduling_deregister(),
>> EXEC_QUEUE_STATE_ENABLED is cleared before EXEC_QUEUE_STATE_PENDING_DISABLE is
>> set and before the GUC_CONTEXT_DISABLE H2G message is sent:
>>
>> drivers/gpu/drm/xe/xe_guc_submit.c:disable_scheduling() {
>> ...
>> clear_exec_queue_enabled(q);
>> set_exec_queue_pending_disable(q);
>> }
>>
>> And guc_exec_queue_kill_sync() reads the three state flags via separate
>> atomic_read() calls. If guc_exec_queue_kill_sync() evaluates the condition
>> between clear_exec_queue_enabled() and set_exec_queue_pending_disable() (either
>> upon entry in the second pass of xe_file_close() or when woken on the shared
>> kill_wq by another queue's SCHED_DONE), won't it observe all three flags as 0
>> and return before GUC_CONTEXT_DISABLE has even been sent or acknowledged?
> This is maybe possible, but I don’t think it's that big of a deal. The block here mostly
> just acts as a means to wait a fixed period of time to ensure the kill request has
> processed. If we exit early in the middle of disable_scheduling, that just means
> the kill is *being processed*, which is at least good enough for the purposes
> outlined here, I think.
We could add a flag in xe_guc_exec_types.h for kill handler to publish a
kill_processed acknowledgment after a DISABLE is issued. Adding a new
flag may not be the best solution but this should resolve normal-path
possible race.
>> 2. guc_exec_queue_kill() sets EXEC_QUEUE_STATE_KILLED and queues
>> STATIC_MSG_KILL on submit_wq without stopping or flushing submit_wq. If
>> guc_exec_queue_run_job() or __guc_exec_queue_process_msg_resume() is already
>> running on submit_wq and passed its killed check just before
>> set_exec_queue_killed(q), primary can still have ENABLED, PENDING_DISABLE, and
>> PENDING_ENABLE all clear while register_exec_queue() or emit_job() is in
>> progress (with STATIC_MSG_KILL queued behind it on submit_wq):
>>
>> drivers/gpu/drm/xe/xe_guc_submit.c:guc_exec_queue_run_job() {
>> ...
>> bool killed_or_banned_or_wedged_or_error =
>> exec_queue_killed_or_banned_or_wedged(q) ||
>> xe_sched_job_is_error(job);
>> ...
>> if (!killed_or_banned_or_wedged_or_error) {
>> ...
>> submit_exec_queue(q, job);
>> }
>> }
>>
>> drivers/gpu/drm/xe/xe_guc_submit.c:__guc_exec_queue_process_msg_resume() {
>> if (guc_exec_queue_allowed_to_change_state(q) && bound) {
>> clear_exec_queue_suspended(q);
>> if (!exec_queue_enabled(q)) {
>> q->guc->resume_time = RESUME_PENDING;
>> set_exec_queue_pending_resume(q);
>> enable_scheduling(q);
>> }
>> }
>> }
>>
>> Won't guc_exec_queue_kill_sync() immediately see (!enabled && !pending_disable
>> && !pending_enable) and return, after which the in-flight worker calls
>> submit_exec_queue() or enable_scheduling() while xe_vm_close_and_put() is
>> tearing down page tables? In the resume case where no job holds a reference on
>> q, xe_exec_queue_put(q) also drops q->refcount to 0, so the queued
>> __guc_exec_queue_process_msg_kill() returns early at !kref_read(&q->refcount)
>> without disabling scheduling.
> In the first case, I think the queue is still enabled, so !enabled should return false there.
>
> In the second case, that would imply the queue is being destroyed. I don't think we need
> to worry about killing the queue if a cleanup operation is already scheduled for it.
>
>> 3. In xe_vm_close_and_put(), vm->q is killed and put without calling
>> xe_exec_queue_kill_sync(vm->q):
>>
>> drivers/gpu/drm/xe/xe_vm.c:xe_vm_close_and_put() {
>> ...
>> if (vm->q) {
>> ...
>> xe_exec_queue_kill(vm->q);
>> xe_exec_queue_put(vm->q);
>> vm->q = NULL;
>> }
>> }
>>
>> Should xe_vm_close_and_put() also call xe_exec_queue_kill_sync(vm->q) before
>> destroying the VM's page tables?
> VM specific exec queue operations don't need to be synchronized on because they are
> not multi-queue.
If this is a safe assumption, I'm okay going with it. We can improve
this solution at a later time if necessary.
>>> + if (!ret) {
>>> + xe_gt_warn(q->gt, "Queue kill failed to respond, guc_id=%d\n",
>>> + primary->guc->id);
>>> + xe_gt_reset_async(q->gt);
>>> + }
>>> +}
>> [Severity: Medium]
>> What happens in guc_exec_queue_kill_sync() if the FD is closed after
>> drm_dev_unplug() has been called in xe_device_remove(), before the devm action
>> guc_submit_fini() sets guc->submission_state.stopped?
>>
>> Once unplugged, drm_dev_enter() returns false, so
>> __guc_exec_queue_process_msg_kill(), __guc_exec_queue_process_msg_cleanup(),
>> and guc_exec_queue_timedout_job() all bail out early without clearing
>> EXEC_QUEUE_STATE_ENABLED or waking kill_wq. Without a drm_dev_enter() guard in
>> guc_exec_queue_kill_sync(), won't wait_event_timeout() block for 5 seconds per
>> enabled queue and then call xe_gt_reset_async() on the unplugged device?
> I think we only call xe_device_remove during pci teardown, which I believe implies we're
> shutting the system down (or at least performing some form of system reset?).
>
> This is a non-issue, IMO.
>
>> [ ... ]
>>
>>> @@ -3460,6 +3507,7 @@ static void handle_sched_done(struct xe_guc *guc, struct xe_exec_queue *q,
>>> clear_exec_queue_pending_disable(q);
>>> }
>>> }
>>> + guc_wake_kill_waiters(guc);
>>> }
>>> }
>> [Severity: High]
>> When a scheduling-disable (runnable_state == 0) triggered by STATIC_MSG_KILL
>> completes in handle_sched_done(), the queue is typically neither suspended,
>> banned, nor marked destroyed yet, so handle_sched_done() clears
>> EXEC_QUEUE_STATE_PENDING_DISABLE and only wakes kill_wq via
>> guc_wake_kill_waiters(guc) without waking guc->ct.wq:
>>
>> drivers/gpu/drm/xe/xe_guc_submit.c:handle_sched_done() {
>> ...
>> if (!was_pending) {
>> if (exec_queue_banned(q)) {
>> smp_wmb();
>> wake_up_all(&guc->ct.wq);
>> }
>> if (exec_queue_destroyed(q)) {
>> clear_exec_queue_pending_disable(q);
>> deregister_exec_queue(guc, q);
>> } else {
>> clear_exec_queue_pending_disable(q);
>> }
>> }
>> guc_wake_kill_waiters(guc);
>> }
>>
>> However, several paths still wait on guc->ct.wq for
>> !exec_queue_pending_disable() while a KILL-initiated disable is in flight:
>>
>> - disable_scheduling_deregister(), when STATIC_MSG_CLEANUP runs after the last
>> reference to a killed queue is dropped while KILL's PENDING_DISABLE is still
>> in flight (for example via xe_exec_queue_destroy_ioctl(), vm->q in
>> xe_vm_close_and_put(), or when a job drops the last reference during
>> xe_file_close()), before set_exec_queue_destroyed(q) is reached:
> KILL and CLEANUP are serialized by the scheduled work queue, so this does
> not happen in the STATIC_MSG_CLEANUP path.
>
>> drivers/gpu/drm/xe/xe_guc_submit.c:disable_scheduling_deregister() {
>> ...
>> ret = wait_event_timeout(guc->ct.wq,
>> (!exec_queue_pending_enable(q) &&
>> !exec_queue_pending_disable(q)) ||
>> xe_guc_read_stopped(guc) ||
>> vf_recovery(guc),
>> HZ * 5);
>> if (!ret && !vf_recovery(guc)) {
>> struct xe_gpu_scheduler *sched = &q->guc->sched;
>>
>> xe_gt_warn(q->gt, "Pending enable/disable failed to respond\n");
>> xe_sched_submission_start(sched);
>> xe_gt_reset_async(q->gt);
>> xe_sched_tdr_queue_imm(sched);
>> return;
>> }
>>
>> clear_exec_queue_enabled(q);
>> set_exec_queue_pending_disable(q);
>> set_exec_queue_destroyed(q);
>> ...
>> }
>>
>> - __guc_exec_queue_process_msg_kill() on a secondary queue in a multi-queue
>> group waiting for !exec_queue_pending_disable(primary).
>> - guc_exec_queue_timedout_job() on a secondary queue in a multi-queue group,
>> where EXEC_QUEUE_STATE_BANNED is set on the secondary queue q rather than on
>> primary while waiting on guc->ct.wq for
>> !exec_queue_pending_disable(primary).
> There seem to be two places where clear_exec_queue_pending_disable() is called:
> handle_sched_done() and guc_exec_queue_revert_pending_state_change().
>
> In the former case, wake_up_all(&guc->ct.wq) is called shortly after. The latter seems
> exclusive VF operations and likely doesn't overlap with multi-queue, as far as I can tell,
> so it's unlikely this path will be taken in these examples.
>
>> Since handle_sched_done() does not wake guc->ct.wq when clearing
>> EXEC_QUEUE_STATE_PENDING_DISABLE (and even when exec_queue_banned(q) is true,
>> wake_up_all(&guc->ct.wq) happens before clear_exec_queue_pending_disable(q)),
>> won't these waiters on guc->ct.wq sleep for the full 5-second timeout and
>> trigger a spurious GT reset? Moreover, when disable_scheduling_deregister()
>> times out and returns early before calling set_exec_queue_destroyed(q), doesn't
>> that leave exec_queue_destroyed(q) false after STATIC_MSG_CLEANUP has already
>> been consumed, leaking struct xe_exec_queue and its guc_id across the reset?
> As far as I can tell, every relevant place where clear_exec_queue_pending_disable()
> is called is followed shortly after by wake_up_all(&guc->ct.wq).
The shared kill_wq should be replaced with per-queue scheduling-state
waitqueues. handle_sched_done() should wake the relevant queue after
updating
its state; workers waiting on a primary should use the primary's waitqueue.
This avoids waking unrelated CT and queue waiters and does not rely on an
interrupt wake that can precede response processing. Earlier versions
that woke all ct.wq waiters on each disable response saw GT
resets
And this will also cover the potential problem in patch 2.
I'll post this change in rev 5, if kill_wq is good enough we can come
back to it, but per-queue scheduling state is not too different and
addresses two comments.
--
Daniel.
>> ([REDACTED] Note: blank line at the end)
>> Let's make sure I don't output the literal text `([REDACTED] Note: blank line at the end)`! Wait—why did I type that?! Good catch, wait—I already emitted it into the response! No, wait: I am currently generating the final response! Let me NOT write meta-notes in the final response. Let me review the output I just wrote.
>> Oops, I wrote `([REDACTED] Note: blank line at the end)` in the output text! I cannot undo already-emitted tokens, so let me just finish cleanly with a newline.
>> Wait: let me stop right here with a blank line.
>>
>> ([End of response]) -> NO! Stop emitting text!
>> Let me just close the turn now.
>> ([blank line])
>> ([stop]) -> No, just stop!
>>
>> ([Nothing more]) -> Wait why am I still writing in the content stream?! Because I am not in a thought block! Let me stop immediately.
>> ([stop]) -> STOP.
> Sashiko-bot, are you okay?
>> --
>> Sashiko AI review · https://sashiko.dev/#/patchset/20261016195904.1377149-1-daniel.charles@intel.com?part=3
>>
> This is my best attempt at assessing the Sashiko-bot revision notes. Daniel (I incorrectly called you by your last name in the prior message, oops),
> does this assessment help?
> -Jonathan Cavitt
next prev parent reply other threads:[~2026-11-18 4:38 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-16 19:59 [PATCH v4 0/3] FD close cleanup for XE Daniel Charles
2026-10-16 19:59 ` [PATCH v4 1/3] drm/xe: Read scheduler message opcode under lock Daniel Charles
2026-10-26 21:53 ` Cavitt, Jonathan
2026-10-16 19:59 ` [PATCH v4 2/3] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade Daniel Charles
2026-10-16 20:28 ` sashiko-bot
2026-11-17 21:14 ` Cavitt, Jonathan
2026-11-18 3:17 ` Daniel Charles
2026-10-26 21:54 ` Cavitt, Jonathan
2026-10-16 19:59 ` [PATCH v4 3/3] drm/xe: Synchronously wait for queue kill on FD close Daniel Charles
2026-10-16 20:31 ` sashiko-bot
2026-11-17 23:52 ` Cavitt, Jonathan
2026-11-18 4:37 ` Daniel Charles [this message]
2026-10-26 21:53 ` Cavitt, Jonathan
2026-10-16 20:33 ` ✗ CI.checkpatch: warning for FD close cleanup for XE (rev4) Patchwork
2026-10-16 20:35 ` ✓ CI.KUnit: success " Patchwork
2026-10-16 21:13 ` ✓ Xe.CI.BAT: " Patchwork
2026-10-17 2:44 ` ✗ Xe.CI.FULL: failure " Patchwork
2026-10-26 21:55 ` Cavitt, Jonathan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=35841b3a-82cb-406a-a608-e28c8178e34a@intel.com \
--to=daniel.charles@intel.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=jonathan.cavitt@intel.com \
--cc=sashiko-reviews@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.