Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Daniel Charles" <daniel.charles@intel.com>,
	intel-xe@lists.freedesktop.org
Subject: Re: [PATCH v6 4/4] drm/xe: Synchronously wait for queue kill on FD close
Date: Thu, 19 Nov 2026 18:21:42 +0000	[thread overview]
Message-ID: <sashiko-outbox-223765@kernel.org> (raw)
In-Reply-To: <20261119174621.1311154-5-daniel.charles@intel.com>

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] drm/xe: timeout in guc_exec_queue_kill_sync() after reset or VF
  recovery
- [Medium] drm/xe: missing kill_processed reset in
  guc_exec_queue_reinit_kernel()
- [Medium] drm/xe: guc_exec_queue_kill_sync() stalls and resets on
  unplugged device
--

commit 4827880406140970103161afc0f5446cedf1ee73
Author: Daniel Charles <daniel.charles@intel.com>

drm/xe: Synchronously wait for queue kill on FD close

This commit splits FD close into two passes that first kill all queues on
the file and then wait for each queue's scheduling to stop before dropping
its reference and tearing down VMs.

> diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> index 671852b562105..ec308b06c1bee 100644
> --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> @@ -2104,6 +2104,9 @@ static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg,
>  	if (!kref_read(&q->refcount) || !bound || exec_queue_wedged(q))
>  		return;
>  
> +	if (READ_ONCE(q->guc->kill_processed))
> +		return;

[Severity: Medium]
Does guc_exec_queue_reinit_kernel() need to reset q->guc->kill_processed to
false?

During PCI FLR preparation, xe_guc_flr_prepare() ->
xe_guc_submit_pause_abort() kills all queues, including kernel queues.
When FLR completes, guc_exec_queue_reinit_kernel() clears q->guc->state and
reinitializes the scheduler, but leaves q->guc->kill_processed set:

drivers/gpu/drm/xe/xe_guc_submit.c:guc_exec_queue_reinit_kernel() {
	xe_gt_assert(q->gt, q->flags & EXEC_QUEUE_FLAG_KERNEL);

	atomic_set(&q->guc->state, 0);
	xe_sched_reinit(&q->guc->sched);
}

If STATIC_MSG_KILL remained queued in sched->msgs when submission stopped,
xe_guc_submit_start() will also run __guc_exec_queue_process_msg_kill() on
the reinitialized kernel queue because it does not check
exec_queue_killed(q), disabling its scheduling and setting kill_processed.
Once kill_processed is true, any subsequent kill on that kernel queue
returns early without disabling scheduling.

> +
>  	ret = wait_event_timeout(q->guc->state_wq,
>  				 (!exec_queue_pending_enable(q) &&
>  				  !exec_queue_pending_disable(q)) ||
> @@ -2122,6 +2125,10 @@ static void __guc_exec_queue_process_msg_kill(struct xe_sched_msg *msg,
>  
>  	if (exec_queue_enabled(q))
>  		disable_scheduling(q, true);
> +
> +	/* Publish only after disable_scheduling() has established pending state. */
> +	smp_store_release(&q->guc->kill_processed, true);
> +	wake_up_all(&q->guc->state_wq);
>  }
>  
>  #define CLEANUP				1	/* Non-zero values to catch uninitialized msg */
> @@ -2351,6 +2358,45 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q)
>  	xe_guc_exec_queue_trigger_cleanup(q);
>  }
>  
> +static bool guc_exec_queue_kill_complete(struct xe_exec_queue *primary)
> +{
> +	unsigned int state;
> +
> +	/*
> +	 * Pair with the primary KILL handler's release so earlier submission
> +	 * work has finished and any required DISABLE has been issued.
> +	 */
> +	if (!smp_load_acquire(&primary->guc->kill_processed))
> +		return false;
> +
> +	state = atomic_read(&primary->guc->state);
> +
> +	return !(state & (EXEC_QUEUE_STATE_ENABLED |
> +			  EXEC_QUEUE_STATE_PENDING_DISABLE |
> +			  EXEC_QUEUE_STATE_PENDING_ENABLE));
> +}

[Severity: High]
Can guc_exec_queue_kill_complete() remain permanently false after a GT
reset or SR-IOV VF migration recovery, causing guc_exec_queue_kill_sync()
to time out and trigger a spurious GT reset?

When xe_sched_process_msg_work() dequeues STATIC_MSG_KILL and runs
__guc_exec_queue_process_msg_kill() while a GT reset or VF recovery is in
progress (or when its wait_event_timeout() times out or is woken by
xe_guc_read_stopped(guc) or vf_recovery(guc)), it returns early without
setting q->guc->kill_processed.  Once the GT reset or VF recovery finishes,
xe_guc_read_stopped(guc) and vf_recovery(guc) return to false, but
STATIC_MSG_KILL has already been consumed and kill_processed is still
false.

In addition, if __guc_exec_queue_process_msg_kill() already called
disable_scheduling() and set kill_processed before a VF migration pause,
guc_exec_queue_revert_pending_state_change() reverts the pending disable
back to EXEC_QUEUE_STATE_ENABLED without queuing a kill disable replay:

drivers/gpu/drm/xe/xe_guc_submit.c:guc_exec_queue_revert_pending_state_change() {
...
	if (pending_disable) {
		if (!pending_enable)
			set_exec_queue_enabled(q);
		clear_exec_queue_pending_disable(q);
		xe_gt_dbg(guc_to_gt(guc), "Replay DISABLE - guc_id=%d",
			  q->guc->id);
	}
...
}

In both cases, guc_exec_queue_kill_complete(primary) stays false after
reset or recovery completes, so guc_exec_queue_kill_sync() stalls for 5
seconds per affected queue on FD close and calls xe_gt_reset_async(q->gt).

> +
> +static void guc_exec_queue_kill_sync(struct xe_exec_queue *q)
> +{
> +	struct xe_exec_queue *primary = xe_exec_queue_multi_queue_primary(q);
> +	struct xe_guc *guc = exec_queue_to_guc(primary);
> +	long ret;
> +
> +	xe_assert(guc_to_xe(guc), exec_queue_killed(q));
> +
> +	/* FD close has already killed every primary before waiting on any member. */
> +	ret = wait_event_timeout(primary->guc->state_wq,
> +				 guc_exec_queue_kill_complete(primary) ||
> +				 xe_guc_read_stopped(guc) ||
> +				 vf_recovery(guc) ||
> +				 exec_queue_wedged(primary), HZ * 5);
> +	if (!ret) {
> +		xe_gt_warn(q->gt, "Queue kill failed to respond, guc_id=%d\n",
> +			   primary->guc->id);
> +		xe_gt_reset_async(q->gt);
> +	}
> +}

[Severity: Medium]
Can guc_exec_queue_kill_sync() stall and queue a GT reset when an FD is
closed after drm_dev_unplug()?

In xe_device_remove(), drm_dev_unplug(&xe->drm) runs before
xe_bo_pci_dev_remove_all(xe), while guc->submission_state.stopped is only
set later when devres_release_all() invokes guc_submit_fini().  If
xe_file_close() runs in that window, drm_dev_enter() in
guc_exec_queue_process_msg() sets bound = false, so
__guc_exec_queue_process_msg_kill() returns immediately without setting
kill_processed or waking state_wq.  Because guc_exec_queue_kill_sync() does
not check drm_dev_enter() or drm_dev_is_unplugged(), it waits on state_wq
until guc_submit_fini() runs, or times out after 5 seconds per queue and
calls xe_gt_reset_async() on the unplugged device.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261119174621.1311154-1-daniel.charles@intel.com?part=4

  reply	other threads:[~2026-11-19 18:21 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 17:46 [PATCH v6 0/4] FD close cleanup for XE Daniel Charles
2026-11-19 17:46 ` [PATCH v6 1/4] drm/xe: Read scheduler message opcode under lock Daniel Charles
2026-11-19 17:46 ` [PATCH v6 2/4] drm/xe: Notify scheduling-state waiters per exec queue Daniel Charles
2026-11-19 18:06   ` sashiko-bot
2026-11-19 19:16     ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 3/4] drm/xe: Disable scheduling early on FD close to avoid CAT error cascade Daniel Charles
2026-11-19 18:18   ` sashiko-bot
2026-11-19 19:26     ` Cavitt, Jonathan
2026-11-19 17:46 ` [PATCH v6 4/4] drm/xe: Synchronously wait for queue kill on FD close Daniel Charles
2026-11-19 18:21   ` sashiko-bot [this message]
2026-11-19 19:44   ` Cavitt, Jonathan
2026-11-20  0:16 ` ✗ CI.checkpatch: warning for FD close cleanup for XE (rev6) Patchwork
2026-11-20  0:18 ` ✓ CI.KUnit: success " Patchwork
2026-11-20  0:58 ` ✓ Xe.CI.BAT: " Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-223765@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=daniel.charles@intel.com \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox