All of lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Umesh Nerlige Ramappa" <umesh.nerlige.ramappa@intel.com>,
	intel-xe@lists.freedesktop.org
Subject: Re: [PATCH v7 7/7] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset
Date: Wed, 18 Nov 2026 03:51:02 +0000	[thread overview]
Message-ID: <sashiko-outbox-220052@kernel.org> (raw)
In-Reply-To: <20261118025140.3782678-16-umesh.nerlige.ramappa@intel.com>

Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [High] drm/xe/oa: out-of-bounds vmap read in
  xe_oa_buffer_check_unlocked() on reset
- [Medium] drm/xe/oa: HW access and missing drain in xe_oa_stream_init()
  error path
- [Medium] drm/xe/oa: drm_dev_is_unplugged() race in xe_oa_release()
  skips ttm.wq flush
- [Medium] drm/xe/oa: OA buffer BO destroy can outlive flush_workqueue()
  on unplug
- [Medium] drm/xe/oa: unnecessary fence wait in xe_oa_quiesce_oa_unit()
  when wedged
--

commit 3beacae376d78af14b851e582701fe6ee70c6d4c
Author: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>

drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset

This commit quiesces active OA streams on driver unbind and PCI reset by
marking streams dead, releasing device-bound resources, and rejecting
further operations on the stream fd until it is closed.

> diff --git a/drivers/gpu/drm/xe/xe_oa.c b/drivers/gpu/drm/xe/xe_oa.c
> index c7ccf273ced75..07ec4255cf089 100644
> --- a/drivers/gpu/drm/xe/xe_oa.c
> +++ b/drivers/gpu/drm/xe/xe_oa.c
> @@ -261,13 +261,18 @@ static bool mert_wa_14026633728(struct xe_oa_stream *s)
>  
>  static bool xe_oa_buffer_check_unlocked(struct xe_oa_stream *stream)
>  {
> -	u32 gtt_offset = xe_bo_ggtt_addr(stream->oa_buffer.bo);
> -	u32 tail, hw_tail, partial_report_size, available;
> +	u32 gtt_offset, tail, hw_tail, partial_report_size, available;
>  	int report_size = stream->oa_buffer.format->size;
>  	unsigned long flags;
>  
>  	spin_lock_irqsave(&stream->oa_buffer.ptr_lock, flags);
>  
> +	if (stream->dead) {
> +		spin_unlock_irqrestore(&stream->oa_buffer.ptr_lock, flags);
> +		return false;
> +	}
> +
> +	gtt_offset = xe_bo_ggtt_addr(stream->oa_buffer.bo);
>  	hw_tail = xe_oa_hw_tail_read(stream);
>  	hw_tail -= gtt_offset;

[Severity: High]
Can xe_oa_buffer_check_unlocked() read stream->oa_buffer.bo->vmap out of
bounds when the device is wedged or resetting before xe_oa_fini() runs?

stream->dead is only set in xe_oa_quiesce_oa_unit() during xe_oa_fini().
During a PCI slot reset (between prepare_device_for_reset() and
devres_release_group() in xe_pci_error_slot_reset()), surprise hot-unplug,
or runtime wedge, stream->dead is still false while the 200 Hz
poll_check_timer and xe_oa_read() continue calling
xe_oa_buffer_check_unlocked().

When xe_oa_hw_tail_read() returns 0xffffffff on a disabled PCI device or 0
after a secondary bus reset, subtracting gtt_offset wraps the u32 hw_tail
to a value >= stream->oa_buffer.circ_size. Because xe_oa_circ_diff() does
not clamp or wrap values >= circ_size, the backward walk passes that tail
offset into oa_report_id(stream, tail) and indexes stream->oa_buffer.bo's
vmap past the end of the buffer.

[ ... ]

> @@ -1129,12 +1152,28 @@ static int xe_oa_emit_oa_config(struct xe_oa_stream *stream, struct xe_oa_config
>  	if (err == -ENOENT)
>  		xe_oa_config_cb(fence, &ofence->cb);
>  
> -	/* If nothing needs to be signaled we wait synchronously */
> +	/*
> +	 * If nothing needs to be signaled we wait synchronously unless the device
> +	 * is unplugged/wedged.
> +	 */
>  	if (!num_signal) {
> -		dma_fence_wait(&ofence->base, false);
> +		while (!dma_fence_wait_timeout(&ofence->base, false, HZ / 10)) {
> +			if (drm_dev_is_unplugged(&gt_to_xe(stream->gt)->drm)) {
> +				ret = -ENODEV;
> +				break;
> +			}
> +			if (xe_wedged(gt_to_xe(stream->gt))) {
> +				ret = -ECANCELED;
> +				break;
> +			}
> +		}
>  		dma_fence_put(&ofence->base);
>  	}
>  
> +	/* The callers clean up the syncs on error */
> +	if (ret)
> +		return ret;

[Severity: Medium]
When xe_oa_emit_oa_config() returns early here during xe_oa_stream_init()
-> xe_oa_enable_metric_set(), does the error unwind path in
xe_oa_stream_init() mishandle the in-flight batch and wedged state?

drivers/gpu/drm/xe/xe_oa.c:xe_oa_stream_init() {
    ...
err_put_k_exec_q:
	xe_oa_disable_metric_set(stream);
	xe_exec_queue_put(stream->k_exec_q);
err_free_oa_buf:
	xe_oa_free_oa_buffer(stream);
err_fw_put:
	xe_force_wake_put(gt_to_fw(gt), stream->fw_ref);
    ...
}

Because stream->oa_unit->exclusive_stream has not been set yet,
stream->dead is false. If the wait aborted because the device wedged,
err_put_k_exec_q still performs MMIO in xe_oa_disable_metric_set() and
xe_force_wake_put(), and when stream->exec_q is set it submits a context
restore batch and waits in xe_oa_stream_drain() on the wedged device.

If the wait aborted because the device was unplugged, when stream->exec_q
is set xe_oa_disable_metric_set() sees !stream->dead and blocks in
xe_oa_stream_drain() inside the drm_dev_enter() section of xe_oa_ioctl();
when stream->exec_q is NULL, nothing drains stream->last_fence before
xe_oa_disable_metric_set(), xe_force_wake_put(), and xe_pm_runtime_put()
run.

[ ... ]

> @@ -1753,17 +1811,37 @@ static void xe_oa_destroy_locked(struct xe_oa_stream *stream)
>  	kfree(stream);
>  }
>  
> +static void xe_oa_quiesce_oa_unit(struct xe_oa_unit *u, bool hw_accessible);
> +
>  static int xe_oa_release(struct inode *inode, struct file *file)
>  {
>  	struct xe_oa_stream *stream = file->private_data;
>  	struct xe_gt *gt = stream->gt;
> +	struct xe_device *xe = gt_to_xe(gt);
> +	int idx;
>  
>  	mutex_lock(&gt->oa.gt_lock);
> -	xe_oa_destroy_locked(stream);
> +	if (drm_dev_enter(&xe->drm, &idx)) {
> +		xe_oa_destroy_locked(stream);
> +		drm_dev_exit(idx);
> +	} else {
> +		if (!stream->dead)
> +			xe_oa_quiesce_oa_unit(stream->oa_unit, !xe_wedged(xe));
> +
> +		xe_oa_destroy_locked(stream);
> +	}
>  	mutex_unlock(&gt->oa.gt_lock);
>  
> +	/*
> +	 * Freeing the OA buffer may defer the BO destroy to the TTM workqueue. Once the device
> +	 * is unplugged, this may be the last reference on the drm device, so let the destroy
> +	 * finish before drm_dev_put() releases the GGTT.
> +	 */
> +	if (drm_dev_is_unplugged(&xe->drm))
> +		flush_workqueue(xe->ttm.wq);
> +
>  	/* Release the reference the OA stream kept on the driver */
> -	drm_dev_put(&gt_to_xe(gt)->drm);
> +	drm_dev_put(&xe->drm);
>  
>  	return 0;
>  }

[Severity: Medium]
Can xe_oa_release() race with drm_dev_unplug() after drm_dev_exit() and
skip flush_workqueue(xe->ttm.wq) before dropping the last drm_device
reference?

CPU0                                  CPU1
xe_oa_release()
  drm_dev_enter(&xe->drm, &idx)
  xe_oa_destroy_locked(stream)
    ...
      queue_work(xe->ttm.wq, ...)
  drm_dev_exit(idx)
  if (drm_dev_is_unplugged(&xe->drm))
    --> false
                                      xe_pci_remove()
                                        xe_device_remove()
                                          drm_dev_unplug(&xe->drm)
                                        ...
                                        devm_drm_dev_init_release()
                                          drm_dev_put(&xe->drm)
  drm_dev_put(&xe->drm)
    drm_dev_release()
      ggtt_fini_early()               <-- runs before ttm.wq destroys BO

Because ggtt_fini_early() is registered via drmm after xe_device_destroy()
(which calls ttm_device_fini(&xe->ttm)), dropping the last drm_device
reference without flushing xe->ttm.wq tears down ggtt->mm while
ttm_bo_delayed_delete() -> xe_ggtt_remove_bo() is still queued.

[ ... ]

> @@ -2633,6 +2734,97 @@ static void xe_oa_unregister(void *arg)
>  	oa->metrics_kobj = NULL;
>  }
>  
> +static void xe_oa_quiesce_oa_unit(struct xe_oa_unit *u, bool hw_accessible)
> +{
> +	struct xe_oa_stream *stream = u->exclusive_stream;
> +	unsigned long flags;
> +
> +	if (!stream)
> +		return;
> +
> +	mutex_lock(&stream->stream_lock);
> +
> +	spin_lock_irqsave(&stream->oa_buffer.ptr_lock, flags);
> +	WRITE_ONCE(stream->dead, true);
> +	spin_unlock_irqrestore(&stream->oa_buffer.ptr_lock, flags);
> +
> +	if (stream->enabled && hw_accessible)
> +		xe_oa_disable_locked(stream);
> +
> +	stream->enabled = false;
> +
> +	if (stream->sample)
> +		hrtimer_cancel(&stream->poll_check_timer);
> +
> +	wake_up_all(&stream->poll_wq);
> +
> +	xe_oa_stream_drain(stream);
> +
> +	/*
> +	 * Restore the HW state while the stream still holds its runtime PM ref and exec_q.
> +	 * The device is not reset on unbind, so this can't be left to a later reset.
> +	 */
> +	if (hw_accessible)
> +		xe_oa_stream_hw_teardown(stream);

[Severity: Medium]
Should xe_oa_stream_drain(stream) be skipped when !hw_accessible?

When the device is wedged, xe_oa_stream_hw_teardown() is skipped and
free_oa_config_bo() hands stream->last_fence to xe_bb_free(), which does
not require the fence to be signaled first. However, stream->last_fence
can remain unsignaled on a wedged device (for example when
xe_oa_emit_oa_config() broke out of its wait with -ECANCELED, or when an
OA batch is waiting on an unsignaled input dependency), causing
xe_oa_stream_drain() to stall uninterruptibly for job_timeout_ms + HZ and
log a timeout error.

> +
> +	/*
> +	 * Keep the OA buffer pinned until xe_oa_release(). An mmap holds a reference on the
> +	 * BO, so unpinning it here leaves it on the TTM LRU, where the shrinker may take the
> +	 * last reference and free the BO after the GGTT and the shrinker itself are gone.
> +	 * drm_dev_unplug() has zapped the userspace mappings and refaults get a dummy page.
> +	 * Drop the DMA mapping now, while pinned: the fd may be released, and the BO
> +	 * unpinned, while xe_bo_pinned_fini() DMA-unmaps it without the BO lock.
> +	 */
> +	xe_bo_dma_unmap_pinned(stream->oa_buffer.bo);

[Severity: Medium]
Can the final reference on stream->oa_buffer.bo be dropped after
xe_oa_release() flushes xe->ttm.wq?

xe_bo_dma_unmap_pinned() leaves bo->pinned_link on
xe->pinned.early.kernel_bo_present, where xe_bo_pinned_fini() ->
xe_bo_apply_to_pinned() still visits it and takes a temporary reference on
the BO outside xe->pinned.lock:

CPU0 (devres_release_all)             CPU1 (close OA fd)
xe_bo_pinned_fini()
  xe_bo_apply_to_pinned()
    xe_bo_get(bo)
    spin_unlock(&xe->pinned.lock)
                                      xe_oa_release()
                                        xe_oa_destroy_locked()
                                          xe_bo_unpin_map_no_vm(bo)
                                            xe_bo_unpin(bo)
                                            xe_bo_put(bo)  <-- ref > 0
                                        flush_workqueue(xe->ttm.wq)
    xe_bo_dma_unmap_pinned(bo)
    xe_bo_put(bo)
      ttm_bo_release()
        queue_work(xe->ttm.wq, ...)
  ...
  devm_drm_dev_init_release()
    drm_dev_put(&xe->drm)             drm_dev_put(&xe->drm)
      ggtt_fini_early()               <-- runs before ttm.wq destroys BO

Because CPU0 holds a reference on bo while CPU1 runs xe_bo_put(bo) and
flush_workqueue(xe->ttm.wq), CPU0's xe_bo_put(bo) queues delayed BO
destruction onto xe->ttm.wq after the flush, allowing ggtt_fini_early()
and xe_shrinker_fini() to tear down the GGTT and shrinker before the BO is
destroyed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261118025140.3782678-9-umesh.nerlige.ramappa@intel.com?part=7

  reply	other threads:[~2026-11-18  3:51 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  2:51 [PATCH v7 0/7] Cleanup OA stream on PCI unbind/remove Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 1/7] drm/xe/oa: Use the drm device address_space for the OA stream fd Umesh Nerlige Ramappa
2026-11-18  3:14   ` sashiko-bot
2026-11-18  2:51 ` [PATCH v7 2/7] drm/xe/oa: Split OA stream destroy into HW teardown and resource release Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 3/7] drm/xe/oa: Take the runtime PM reference in xe_oa_destroy_locked() Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 4/7] drm/xe/oa: Skip OA HW teardown on a wedged device Umesh Nerlige Ramappa
2026-11-18  3:25   ` sashiko-bot
2026-11-18  2:51 ` [PATCH v7 5/7] drm/xe/oa: Wait for in-flight OA batches before stream HW teardown Umesh Nerlige Ramappa
2026-11-18  2:51 ` [PATCH v7 6/7] drm/xe: Reject vm_ops->access on BO mappings of an unplugged device Umesh Nerlige Ramappa
2026-11-18  3:05   ` sashiko-bot
2026-11-18  2:51 ` [PATCH v7 7/7] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset Umesh Nerlige Ramappa
2026-11-18  3:51   ` sashiko-bot [this message]
2026-11-18  3:01 ` ✓ CI.KUnit: success for Cleanup OA stream on PCI unbind/remove (rev7) Patchwork
2026-11-18  3:37 ` ✓ Xe.CI.BAT: " Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-220052@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=umesh.nerlige.ramappa@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.