Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Umesh Nerlige Ramappa" <umesh.nerlige.ramappa@intel.com>,
	intel-xe@lists.freedesktop.org
Subject: Re: [PATCH v7 7/7] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset
Date: Wed, 18 Nov 2026 03:51:02 +0000	[thread overview]
Message-ID: <sashiko-outbox-220052@kernel.org> (raw)
In-Reply-To: <20261118025140.3782678-16-umesh.nerlige.ramappa@intel.com>

Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [High] drm/xe/oa: out-of-bounds vmap read in
  xe_oa_buffer_check_unlocked() on reset
- [Medium] drm/xe/oa: HW access and missing drain in xe_oa_stream_init()
  error path
- [Medium] drm/xe/oa: drm_dev_is_unplugged() race in xe_oa_release()
  skips ttm.wq flush
- [Medium] drm/xe/oa: OA buffer BO destroy can outlive flush_workqueue()
  on unplug
- [Medium] drm/xe/oa: unnecessary fence wait in xe_oa_quiesce_oa_unit()
  when wedged
--

commit 3beacae376d78af14b851e582701fe6ee70c6d4c
Author: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>

drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset

This commit quiesces active OA streams on driver unbind and PCI reset by
marking streams dead, releasing device-bound resources, and rejecting
further operations on the stream fd until it is closed.

> diff --git a/drivers/gpu/drm/xe/xe_oa.c b/drivers/gpu/drm/xe/xe_oa.c
> index c7ccf273ced75..07ec4255cf089 100644
> --- a/drivers/gpu/drm/xe/xe_oa.c
> +++ b/drivers/gpu/drm/xe/xe_oa.c
> @@ -261,13 +261,18 @@ static bool mert_wa_14026633728(struct xe_oa_stream *s)
>  
>  static bool xe_oa_buffer_check_unlocked(struct xe_oa_stream *stream)
>  {
> -	u32 gtt_offset = xe_bo_ggtt_addr(stream->oa_buffer.bo);
> -	u32 tail, hw_tail, partial_report_size, available;
> +	u32 gtt_offset, tail, hw_tail, partial_report_size, available;
>  	int report_size = stream->oa_buffer.format->size;
>  	unsigned long flags;
>  
>  	spin_lock_irqsave(&stream->oa_buffer.ptr_lock, flags);
>  
> +	if (stream->dead) {
> +		spin_unlock_irqrestore(&stream->oa_buffer.ptr_lock, flags);
> +		return false;
> +	}
> +
> +	gtt_offset = xe_bo_ggtt_addr(stream->oa_buffer.bo);
>  	hw_tail = xe_oa_hw_tail_read(stream);
>  	hw_tail -= gtt_offset;

[Severity: High]
Can xe_oa_buffer_check_unlocked() read stream->oa_buffer.bo->vmap out of
bounds when the device is wedged or resetting before xe_oa_fini() runs?

stream->dead is only set in xe_oa_quiesce_oa_unit() during xe_oa_fini().
During a PCI slot reset (between prepare_device_for_reset() and
devres_release_group() in xe_pci_error_slot_reset()), surprise hot-unplug,
or runtime wedge, stream->dead is still false while the 200 Hz
poll_check_timer and xe_oa_read() continue calling
xe_oa_buffer_check_unlocked().

When xe_oa_hw_tail_read() returns 0xffffffff on a disabled PCI device or 0
after a secondary bus reset, subtracting gtt_offset wraps the u32 hw_tail
to a value >= stream->oa_buffer.circ_size. Because xe_oa_circ_diff() does
not clamp or wrap values >= circ_size, the backward walk passes that tail
offset into oa_report_id(stream, tail) and indexes stream->oa_buffer.bo's
vmap past the end of the buffer.

[ ... ]

> @@ -1129,12 +1152,28 @@ static int xe_oa_emit_oa_config(struct xe_oa_stream *stream, struct xe_oa_config
>  	if (err == -ENOENT)
>  		xe_oa_config_cb(fence, &ofence->cb);
>  
> -	/* If nothing needs to be signaled we wait synchronously */
> +	/*
> +	 * If nothing needs to be signaled we wait synchronously unless the device
> +	 * is unplugged/wedged.
> +	 */
>  	if (!num_signal) {
> -		dma_fence_wait(&ofence->base, false);
> +		while (!dma_fence_wait_timeout(&ofence->base, false, HZ / 10)) {
> +			if (drm_dev_is_unplugged(&gt_to_xe(stream->gt)->drm)) {
> +				ret = -ENODEV;
> +				break;
> +			}
> +			if (xe_wedged(gt_to_xe(stream->gt))) {
> +				ret = -ECANCELED;
> +				break;
> +			}
> +		}
>  		dma_fence_put(&ofence->base);
>  	}
>  
> +	/* The callers clean up the syncs on error */
> +	if (ret)
> +		return ret;

[Severity: Medium]
When xe_oa_emit_oa_config() returns early here during xe_oa_stream_init()
-> xe_oa_enable_metric_set(), does the error unwind path in
xe_oa_stream_init() mishandle the in-flight batch and wedged state?

drivers/gpu/drm/xe/xe_oa.c:xe_oa_stream_init() {
    ...
err_put_k_exec_q:
	xe_oa_disable_metric_set(stream);
	xe_exec_queue_put(stream->k_exec_q);
err_free_oa_buf:
	xe_oa_free_oa_buffer(stream);
err_fw_put:
	xe_force_wake_put(gt_to_fw(gt), stream->fw_ref);
    ...
}

Because stream->oa_unit->exclusive_stream has not been set yet,
stream->dead is false. If the wait aborted because the device wedged,
err_put_k_exec_q still performs MMIO in xe_oa_disable_metric_set() and
xe_force_wake_put(), and when stream->exec_q is set it submits a context
restore batch and waits in xe_oa_stream_drain() on the wedged device.

If the wait aborted because the device was unplugged, when stream->exec_q
is set xe_oa_disable_metric_set() sees !stream->dead and blocks in
xe_oa_stream_drain() inside the drm_dev_enter() section of xe_oa_ioctl();
when stream->exec_q is NULL, nothing drains stream->last_fence before
xe_oa_disable_metric_set(), xe_force_wake_put(), and xe_pm_runtime_put()
run.

[ ... ]

> @@ -1753,17 +1811,37 @@ static void xe_oa_destroy_locked(struct xe_oa_stream *stream)
>  	kfree(stream);
>  }
>  
> +static void xe_oa_quiesce_oa_unit(struct xe_oa_unit *u, bool hw_accessible);
> +
>  static int xe_oa_release(struct inode *inode, struct file *file)
>  {
>  	struct xe_oa_stream *stream = file->private_data;
>  	struct xe_gt *gt = stream->gt;
> +	struct xe_device *xe = gt_to_xe(gt);
> +	int idx;
>  
>  	mutex_lock(&gt->oa.gt_lock);
> -	xe_oa_destroy_locked(stream);
> +	if (drm_dev_enter(&xe->drm, &idx)) {
> +		xe_oa_destroy_locked(stream);
> +		drm_dev_exit(idx);
> +	} else {
> +		if (!stream->dead)
> +			xe_oa_quiesce_oa_unit(stream->oa_unit, !xe_wedged(xe));
> +
> +		xe_oa_destroy_locked(stream);
> +	}
>  	mutex_unlock(&gt->oa.gt_lock);
>  
> +	/*
> +	 * Freeing the OA buffer may defer the BO destroy to the TTM workqueue. Once the device
> +	 * is unplugged, this may be the last reference on the drm device, so let the destroy
> +	 * finish before drm_dev_put() releases the GGTT.
> +	 */
> +	if (drm_dev_is_unplugged(&xe->drm))
> +		flush_workqueue(xe->ttm.wq);
> +
>  	/* Release the reference the OA stream kept on the driver */
> -	drm_dev_put(&gt_to_xe(gt)->drm);
> +	drm_dev_put(&xe->drm);
>  
>  	return 0;
>  }

[Severity: Medium]
Can xe_oa_release() race with drm_dev_unplug() after drm_dev_exit() and
skip flush_workqueue(xe->ttm.wq) before dropping the last drm_device
reference?

CPU0                                  CPU1
xe_oa_release()
  drm_dev_enter(&xe->drm, &idx)
  xe_oa_destroy_locked(stream)
    ...
      queue_work(xe->ttm.wq, ...)
  drm_dev_exit(idx)
  if (drm_dev_is_unplugged(&xe->drm))
    --> false
                                      xe_pci_remove()
                                        xe_device_remove()
                                          drm_dev_unplug(&xe->drm)
                                        ...
                                        devm_drm_dev_init_release()
                                          drm_dev_put(&xe->drm)
  drm_dev_put(&xe->drm)
    drm_dev_release()
      ggtt_fini_early()               <-- runs before ttm.wq destroys BO

Because ggtt_fini_early() is registered via drmm after xe_device_destroy()
(which calls ttm_device_fini(&xe->ttm)), dropping the last drm_device
reference without flushing xe->ttm.wq tears down ggtt->mm while
ttm_bo_delayed_delete() -> xe_ggtt_remove_bo() is still queued.

[ ... ]

> @@ -2633,6 +2734,97 @@ static void xe_oa_unregister(void *arg)
>  	oa->metrics_kobj = NULL;
>  }
>  
> +static void xe_oa_quiesce_oa_unit(struct xe_oa_unit *u, bool hw_accessible)
> +{
> +	struct xe_oa_stream *stream = u->exclusive_stream;
> +	unsigned long flags;
> +
> +	if (!stream)
> +		return;
> +
> +	mutex_lock(&stream->stream_lock);
> +
> +	spin_lock_irqsave(&stream->oa_buffer.ptr_lock, flags);
> +	WRITE_ONCE(stream->dead, true);
> +	spin_unlock_irqrestore(&stream->oa_buffer.ptr_lock, flags);
> +
> +	if (stream->enabled && hw_accessible)
> +		xe_oa_disable_locked(stream);
> +
> +	stream->enabled = false;
> +
> +	if (stream->sample)
> +		hrtimer_cancel(&stream->poll_check_timer);
> +
> +	wake_up_all(&stream->poll_wq);
> +
> +	xe_oa_stream_drain(stream);
> +
> +	/*
> +	 * Restore the HW state while the stream still holds its runtime PM ref and exec_q.
> +	 * The device is not reset on unbind, so this can't be left to a later reset.
> +	 */
> +	if (hw_accessible)
> +		xe_oa_stream_hw_teardown(stream);

[Severity: Medium]
Should xe_oa_stream_drain(stream) be skipped when !hw_accessible?

When the device is wedged, xe_oa_stream_hw_teardown() is skipped and
free_oa_config_bo() hands stream->last_fence to xe_bb_free(), which does
not require the fence to be signaled first. However, stream->last_fence
can remain unsignaled on a wedged device (for example when
xe_oa_emit_oa_config() broke out of its wait with -ECANCELED, or when an
OA batch is waiting on an unsignaled input dependency), causing
xe_oa_stream_drain() to stall uninterruptibly for job_timeout_ms + HZ and
log a timeout error.

> +
> +	/*
> +	 * Keep the OA buffer pinned until xe_oa_release(). An mmap holds a reference on the
> +	 * BO, so unpinning it here leaves it on the TTM LRU, where the shrinker may take the
> +	 * last reference and free the BO after the GGTT and the shrinker itself are gone.
> +	 * drm_dev_unplug() has zapped the userspace mappings and refaults get a dummy page.
> +	 * Drop the DMA mapping now, while pinned: the fd may be released, and the BO
> +	 * unpinned, while xe_bo_pinned_fini() DMA-unmaps it without the BO lock.
> +	 */
> +	xe_bo_dma_unmap_pinned(stream->oa_buffer.bo);

[Severity: Medium]
Can the final reference on stream->oa_buffer.bo be dropped after
xe_oa_release() flushes xe->ttm.wq?

xe_bo_dma_unmap_pinned() leaves bo->pinned_link on
xe->pinned.early.kernel_bo_present, where xe_bo_pinned_fini() ->
xe_bo_apply_to_pinned() still visits it and takes a temporary reference on
the BO outside xe->pinned.lock:

CPU0 (devres_release_all)             CPU1 (close OA fd)
xe_bo_pinned_fini()
  xe_bo_apply_to_pinned()
    xe_bo_get(bo)
    spin_unlock(&xe->pinned.lock)
                                      xe_oa_release()
                                        xe_oa_destroy_locked()
                                          xe_bo_unpin_map_no_vm(bo)
                                            xe_bo_unpin(bo)
                                            xe_bo_put(bo)  <-- ref > 0
                                        flush_workqueue(xe->ttm.wq)
    xe_bo_dma_unmap_pinned(bo)
    xe_bo_put(bo)
      ttm_bo_release()
        queue_work(xe->ttm.wq, ...)
  ...
  devm_drm_dev_init_release()
    drm_dev_put(&xe->drm)             drm_dev_put(&xe->drm)
      ggtt_fini_early()               <-- runs before ttm.wq destroys BO

Because CPU0 holds a reference on bo while CPU1 runs xe_bo_put(bo) and
flush_workqueue(xe->ttm.wq), CPU0's xe_bo_put(bo) queues delayed BO
destruction onto xe->ttm.wq after the flush, allowing ggtt_fini_early()
and xe_shrinker_fini() to tear down the GGTT and shrinker before the BO is
destroyed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261118025140.3782678-9-umesh.nerlige.ramappa@intel.com?part=7

  reply	other threads:[~2026-11-18  3:51 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  2:51 [PATCH v7 0/7] Cleanup OA stream on PCI unbind/remove Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 1/7] drm/xe/oa: Use the drm device address_space for the OA stream fd Umesh Nerlige Ramappa
2026-11-18  3:14   ` sashiko-bot
2026-11-18  2:51 ` [PATCH v7 2/7] drm/xe/oa: Split OA stream destroy into HW teardown and resource release Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 3/7] drm/xe/oa: Take the runtime PM reference in xe_oa_destroy_locked() Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 4/7] drm/xe/oa: Skip OA HW teardown on a wedged device Umesh Nerlige Ramappa
2026-11-18  3:25   ` sashiko-bot
2026-11-18  2:51 ` [PATCH v7 5/7] drm/xe/oa: Wait for in-flight OA batches before stream HW teardown Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 6/7] drm/xe: Reject vm_ops->access on BO mappings of an unplugged device Umesh Nerlige Ramappa
2026-11-18  3:05   ` sashiko-bot
2026-11-18  2:51 ` [PATCH v7 7/7] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset Umesh Nerlige Ramappa
2026-11-18  3:51   ` sashiko-bot [this message]
2026-11-18  3:01 ` ✓ CI.KUnit: success for Cleanup OA stream on PCI unbind/remove (rev7) Patchwork
2026-11-18  3:37 ` ✓ Xe.CI.BAT: " Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-220052@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=umesh.nerlige.ramappa@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox