From: Daniel Charles <daniel.charles@intel.com>
To: intel-xe@lists.freedesktop.org
Cc: x.wang@intel.com, jia.yao@intel.com, matthew.brost@intel.com,
jonathan.cavitt@intel.com,
Daniel Charles <daniel.charles@intel.com>
Subject: [PATCH v6 4/4] drm/xe: Synchronously wait for queue kill on FD close
Date: Thu, 19 Nov 2026 09:46:21 -0800 [thread overview]
Message-ID: <20261119174621.1311154-5-daniel.charles@intel.com> (raw)
In-Reply-To: <20261119174621.1311154-1-daniel.charles@intel.com>
GuC may still schedule contexts while FD close tears down their page
tables, producing CAT errors and engine resets. Split FD close into
two passes: kill every queue, then wait for each queue before dropping
its reference.
Require the KILL handler to acknowledge processing before accepting a
single scheduling-state snapshot. For multi-queue groups, all members
wait for the primary KILL acknowledgement and scheduling state on the
primary state waitqueue.
The wait also exits on GuC submission stop, VF recovery, or wedge. On
timeout, request a GT reset asynchronously; the wait does not establish
hardware quiescence on those paths.
v2:
- Bound the wait with a timeout, GT reset on timeout (Sashiko, Matt B)
- Drop the flush_work() loop (Matt B)
- Use xe_assert() for the killed-queue check (Matt B)
- Merge the put loop into the kill_sync loop (Matt B)
v3:
- Wait on a dedicated kill_wq instead of waking ct.wq on every
scheduling-disable SCHED_DONE (CI)
v5: Rewrote the description for the primary KILL acknowledgement and
per-queue state waitqueue.
Assisted-by: LLM
Signed-off-by: Daniel Charles <daniel.charles@intel.com>
---
drivers/gpu/drm/xe/xe_device.c | 5 +++
drivers/gpu/drm/xe/xe_exec_queue.c | 13 ++++++
drivers/gpu/drm/xe/xe_exec_queue.h | 1 +
drivers/gpu/drm/xe/xe_exec_queue_types.h | 2 +
drivers/gpu/drm/xe/xe_guc_exec_queue_types.h | 6 +++
drivers/gpu/drm/xe/xe_guc_submit.c | 47 ++++++++++++++++++++
6 files changed, 74 insertions(+)
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index f5a581d39c96..cc27a1604d59 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -194,8 +194,13 @@ static void xe_file_close(struct drm_device *dev, struct drm_file *file)
if (q->vm && q->hwe->hw_engine_group)
xe_hw_engine_group_del_exec_queue(q->hwe->hw_engine_group, q);
xe_exec_queue_kill(q);
+ }
+
+ xa_for_each(&xef->exec_queue.xa, idx, q) {
+ xe_exec_queue_kill_sync(q);
xe_exec_queue_put(q);
}
+
xa_for_each(&xef->vm.xa, idx, vm)
xe_vm_close_and_put(vm);
diff --git a/drivers/gpu/drm/xe/xe_exec_queue.c b/drivers/gpu/drm/xe/xe_exec_queue.c
index ddd9e38aa80b..00941a9853b5 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue.c
+++ b/drivers/gpu/drm/xe/xe_exec_queue.c
@@ -1662,6 +1662,19 @@ void xe_exec_queue_kill(struct xe_exec_queue *q)
xe_vm_remove_compute_exec_queue(q->vm, q);
}
+/**
+ * xe_exec_queue_kill_sync() - Wait for queue scheduling to stop
+ * @q: The exec queue
+ *
+ * This function waits for the backend to finish stopping the queue after
+ * xe_exec_queue_kill(). It is safe to call multiple times.
+ */
+void xe_exec_queue_kill_sync(struct xe_exec_queue *q)
+{
+ if (q->ops->kill_sync)
+ q->ops->kill_sync(q);
+}
+
/**
* xe_exec_queue_destroy_ioctl() - Destroy an exec queue via IOCTL
* @dev: DRM device
diff --git a/drivers/gpu/drm/xe/xe_exec_queue.h b/drivers/gpu/drm/xe/xe_exec_queue.h
index d48775ef6baf..b7079c35e822 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue.h
+++ b/drivers/gpu/drm/xe/xe_exec_queue.h
@@ -120,6 +120,7 @@ bool xe_exec_queue_is_lr(struct xe_exec_queue *q);
bool xe_exec_queue_is_idle(struct xe_exec_queue *q, int extra_jobs);
void xe_exec_queue_kill(struct xe_exec_queue *q);
+void xe_exec_queue_kill_sync(struct xe_exec_queue *q);
int xe_exec_queue_create_ioctl(struct drm_device *dev, void *data,
struct drm_file *file);
diff --git a/drivers/gpu/drm/xe/xe_exec_queue_types.h b/drivers/gpu/drm/xe/xe_exec_queue_types.h
index 44a41ddc7a05..611d348145a6 100644
--- a/drivers/gpu/drm/xe/xe_exec_queue_types.h
+++ b/drivers/gpu/drm/xe/xe_exec_queue_types.h
@@ -296,6 +296,8 @@ struct xe_exec_queue_ops {
void (*reinit_kernel)(struct xe_exec_queue *q);
/** @kill: Kill inflight submissions for backend */
void (*kill)(struct xe_exec_queue *q);
+ /** @kill_sync: Wait for the backend to stop queue scheduling */
+ void (*kill_sync)(struct xe_exec_queue *q);
/** @fini: Undoes the init() for submission backend */
void (*fini)(struct xe_exec_queue *q);
/**
diff --git a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
index 3f5b1950a248..cb0e460f4762 100644
--- a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
+++ b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
@@ -54,6 +54,12 @@ struct xe_guc_exec_queue {
wait_queue_head_t suspend_wait;
/** @state_wq: wait queue for scheduling enable/disable state changes */
wait_queue_head_t state_wq;
+ /**
+ * @kill_processed: KILL handler has drained earlier submission work and
+ * issued any required disable. Published with release semantics and
+ * never cleared by repeated kill requests.
+ */
+ bool kill_processed;
/**
* @suspend_pending: a suspend of the exec_queue is pending.
* Protected by @sched.msg_lock.
diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
index 671852b56210..ec308b06c1be 100644
--- a/drivers/gpu/drm/xe/xe_guc_submit.c
+++ b/drivers/gpu/drm/xe/xe_guc_submit.c
@@ -2104,6 +2104,9 @@ static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg,
if (!kref_read(&q->refcount) || !bound || exec_queue_wedged(q))
return;
+ if (READ_ONCE(q->guc->kill_processed))
+ return;
+
ret = wait_event_timeout(q->guc->state_wq,
(!exec_queue_pending_enable(q) &&
!exec_queue_pending_disable(q)) ||
@@ -2122,6 +2125,10 @@ static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg,
if (exec_queue_enabled(q))
disable_scheduling(q, true);
+
+ /* Publish only after disable_scheduling() has established pending state. */
+ smp_store_release(&q->guc->kill_processed, true);
+ wake_up_all(&q->guc->state_wq);
}
#define CLEANUP 1 /* Non-zero values to catch uninitialized msg */
@@ -2351,6 +2358,45 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q)
xe_guc_exec_queue_trigger_cleanup(q);
}
+static bool guc_exec_queue_kill_complete(struct xe_exec_queue *primary)
+{
+ unsigned int state;
+
+ /*
+ * Pair with the primary KILL handler's release so earlier submission
+ * work has finished and any required DISABLE has been issued.
+ */
+ if (!smp_load_acquire(&primary->guc->kill_processed))
+ return false;
+
+ state = atomic_read(&primary->guc->state);
+
+ return !(state & (EXEC_QUEUE_STATE_ENABLED |
+ EXEC_QUEUE_STATE_PENDING_DISABLE |
+ EXEC_QUEUE_STATE_PENDING_ENABLE));
+}
+
+static void guc_exec_queue_kill_sync(struct xe_exec_queue *q)
+{
+ struct xe_exec_queue *primary = xe_exec_queue_multi_queue_primary(q);
+ struct xe_guc *guc = exec_queue_to_guc(primary);
+ long ret;
+
+ xe_assert(guc_to_xe(guc), exec_queue_killed(q));
+
+ /* FD close has already killed every primary before waiting on any member. */
+ ret = wait_event_timeout(primary->guc->state_wq,
+ guc_exec_queue_kill_complete(primary) ||
+ xe_guc_read_stopped(guc) ||
+ vf_recovery(guc) ||
+ exec_queue_wedged(primary), HZ * 5);
+ if (!ret) {
+ xe_gt_warn(q->gt, "Queue kill failed to respond, guc_id=%d\n",
+ primary->guc->id);
+ xe_gt_reset_async(q->gt);
+ }
+}
+
static void guc_exec_queue_add_msg(struct xe_exec_queue *q, struct xe_sched_msg *msg,
u32 opcode)
{
@@ -2829,6 +2875,7 @@ static const struct xe_exec_queue_ops guc_exec_queue_ops = {
.init = guc_exec_queue_init,
.reinit_kernel = guc_exec_queue_reinit_kernel,
.kill = guc_exec_queue_kill,
+ .kill_sync = guc_exec_queue_kill_sync,
.fini = guc_exec_queue_fini,
.destroy = guc_exec_queue_destroy,
.set_priority = guc_exec_queue_set_priority,
--
2.43.0
next prev parent reply other threads:[~2026-11-19 17:51 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 17:46 [PATCH v6 0/4] FD close cleanup for XE Daniel Charles
2026-11-19 17:46 ` [PATCH v6 1/4] drm/xe: Read scheduler message opcode under lock Daniel Charles
2026-11-19 17:46 ` [PATCH v6 2/4] drm/xe: Notify scheduling-state waiters per exec queue Daniel Charles
2026-11-19 18:06 ` sashiko-bot
2026-11-19 19:16 ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade Daniel Charles
2026-11-19 18:18 ` sashiko-bot
2026-11-19 19:26 ` Cavitt, Jonathan
2026-11-19 17:46 ` Daniel Charles [this message]
2026-11-19 18:21 ` [PATCH v6 4/4] drm/xe: Synchronously wait for queue kill on FD close sashiko-bot
2026-11-19 19:44 ` Cavitt, Jonathan
2026-11-20 0:16 ` ✗ CI.checkpatch: warning for FD close cleanup for XE (rev6) Patchwork
2026-11-20 0:18 ` ✓ CI.KUnit: success " Patchwork
2026-11-20 0:58 ` ✓ Xe.CI.BAT: " Patchwork
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119174621.1311154-5-daniel.charles@intel.com \
--to=daniel.charles@intel.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=jia.yao@intel.com \
--cc=jonathan.cavitt@intel.com \
--cc=matthew.brost@intel.com \
--cc=x.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox