Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Umesh Nerlige Ramappa" <umesh.nerlige.ramappa@intel.com>,
	intel-xe@lists.freedesktop.org
Subject: Re: [PATCH v8 8/8] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset
Date: Thu, 19 Nov 2026 01:11:41 +0000	[thread overview]
Message-ID: <sashiko-outbox-222279@kernel.org> (raw)
In-Reply-To: <20261119002737.3899877-18-umesh.nerlige.ramappa@intel.com>

Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [Medium] drm/xe/oa: drm_dev_is_unplugged() race in xe_oa_release()
  skips ttm.wq flush
- [Medium] drm/xe/oa: OA buffer BO references can outlive
  flush_workqueue() on unplug
- [Medium] drm/xe/oa: missing or blocking drain in xe_oa_stream_init()
  error path
- [Medium] drm/xe/oa: NULL sg_table deref on resume after
  xe_bo_dma_unmap_pinned()
- [Medium] drm/xe/oa: unconditional drain and batch submit on wedged
  device in destroy
--

commit 2b379416db1422398b8cce664304d4acea228c64
Author: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>

drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset

This commit quiesces active OA streams on driver unbind or PCI reset by
marking them dead and tearing down their hardware and device resources
while keeping the stream structure and pinned OA buffer allocated until
the stream file descriptor is closed.

> diff --git a/drivers/gpu/drm/xe/xe_oa.c b/drivers/gpu/drm/xe/xe_oa.c
> index 223419347eb7b..0b51bc818ef47 100644
> --- a/drivers/gpu/drm/xe/xe_oa.c
> +++ b/drivers/gpu/drm/xe/xe_oa.c

[ ... ]

> @@ -937,6 +959,13 @@ static void xe_oa_stream_destroy(struct xe_oa_stream *stream)
>  	xe_oa_disable_metric_set(stream);
>  	xe_exec_queue_put(stream->k_exec_q);
>  
> +	/*
> +	 * The OA buffer is freed on release of the stream fd, an mmap may still hold a
> +	 * reference on it. Drop its DMA mapping now, while pinned: the fd may be released, and
> +	 * the BO unpinned, while xe_bo_pinned_fini() DMA-unmaps it without the BO lock.
> +	 */
> +	xe_bo_dma_unmap_pinned(stream->oa_buffer.bo);
> +
>  	xe_force_wake_put(gt_to_fw(gt), stream->fw_ref);
>  	xe_pm_runtime_put(stream->oa->xe);

[Severity: Medium]
Can calling xe_bo_dma_unmap_pinned() here while stream->oa_buffer.bo stays
pinned on xe->pinned.early.kernel_bo_present cause a NULL pointer
dereference if the device suspends and resumes before
xe_oa_free_oa_buffer() unpins the BO?

CPU0                                  CPU1
xe_oa_release()
  xe_oa_destroy_locked()
    xe_oa_stream_destroy()
      xe_bo_dma_unmap_pinned(bo)
        xe_tt->sg = NULL
      xe_pm_runtime_put()
    xe_pm_runtime_put()
                                      xe_pm_suspend()
                                        xe_bo_evict_all()
                                      xe_pm_resume()
                                        xe_bo_restore_early()
                                          xe_bo_restore_and_map_ggtt()
                                            xe_ggtt_map_bo()
                                              xe_res_first_sg(NULL, ...)
                                                <-- NULL pointer dereference
  xe_oa_free_oa_buffer()
    xe_bo_unpin_map_no_vm(bo)

Non-MERT OA buffers are created with XE_BO_FLAG_SYSTEM | XE_BO_FLAG_GGTT
and remain linked on xe->pinned.early.kernel_bo_present until
xe_bo_unpin() removes bo->pinned_link.  Once xe_oa_destroy_locked() drops
the last runtime PM reference, a system or D3cold runtime suspend/resume
cycle will evict and restore the pinned BO via xe_ggtt_map_bo(), passing
xe_bo_sg(bo) == NULL to xe_res_first_sg().

[ ... ]

> @@ -1754,19 +1786,35 @@ static long xe_oa_ioctl(struct file *file,
>  	return ret;
>  }
>  
> +/*
> + * Disable the stream, restore the HW state and release everything but the OA buffer and the
> + * stream struct, the same whether the stream fd is closed or the device goes away, and
> + * whether or not the HW still responds: all HW waits are bounded. Afterwards the stream is
> + * dead and any further read, poll, mmap or ioctl on the stream fd fails with -ENODEV.
> + */
>  static void xe_oa_destroy_locked(struct xe_oa_stream *stream)
>  {
> +	unsigned long flags;
> +
>  	xe_pm_runtime_get(gt_to_xe(stream->gt));
> +	mutex_lock(&stream->stream_lock);
> +
> +	spin_lock_irqsave(&stream->oa_buffer.ptr_lock, flags);
> +	WRITE_ONCE(stream->dead, true);
> +	spin_unlock_irqrestore(&stream->oa_buffer.ptr_lock, flags);
>  
>  	if (stream->enabled)
>  		xe_oa_disable_locked(stream);
>  
> +	wake_up_all(&stream->poll_wq);
> +
>  	xe_oa_stream_destroy(stream);

[Severity: Medium]
When the device is wedged, does xe_oa_stream_destroy() stall for multiple
seconds and log a timeout error?

xe_oa_stream_destroy() unconditionally calls xe_oa_stream_drain() and
xe_oa_disable_metric_set() without checking xe_wedged().  If
stream->last_fence is unsignaled when the device wedges (for example when
a config batch is waiting on an input fence or after xe_oa_emit_oa_config()
breaks out of its wait loop on wedge), xe_oa_stream_drain() sleeps
uninterruptibly for job_timeout_ms + HZ and prints "Timed out waiting for
OA batch".  Then, when stream->exec_q is set, xe_oa_disable_metric_set()
calls xe_oa_configure_oa_context(stream, false) to submit a new batch to
the already-wedged queue (which guc_exec_queue_run_job() does not submit
to hardware) and waits again in xe_oa_stream_drain() until TDR or timeout.

[Severity: Medium]
Similarly, in xe_oa_stream_init(), what happens when
xe_oa_enable_metric_set() -> xe_oa_emit_oa_config() cuts its synchronous
wait short on device unplug or wedge (introduced in commit "drm/xe/oa:
Fence OA ioctls and mmap against device unplug and wedging")?

drivers/gpu/drm/xe/xe_oa.c:xe_oa_stream_init() {
	...
err_put_k_exec_q:
	xe_oa_disable_metric_set(stream);
	xe_exec_queue_put(stream->k_exec_q);
err_free_oa_buf:
	xe_oa_free_oa_buffer(stream);
err_fw_put:
	xe_force_wake_put(gt_to_fw(gt), stream->fw_ref);
	xe_pm_runtime_put(stream->oa->xe);
	xe_oa_free_configs(stream);
	...
}

At that point the config batch has already been submitted to
stream->k_exec_q and recorded in stream->last_fence.  Unlike
xe_oa_stream_destroy(), err_put_k_exec_q does not call
xe_oa_stream_drain() before xe_oa_disable_metric_set().  When
stream->exec_q is NULL, the in-flight config batch on stream->k_exec_q is
never drained before OA registers, forcewake, and runtime PM are released.
When stream->exec_q is non-NULL, xe_oa_disable_metric_set() queues a
context-restore batch behind the unsignaled config batch and blocks in
xe_oa_stream_drain() for job_timeout_ms + HZ while still inside the
drm_dev_enter() section of xe_observation_ioctl(), blocking
drm_dev_unplug().

>  
>  	if (stream->exec_q)
>  		xe_exec_queue_put(stream->exec_q);
>  
>  	drm_syncobj_put(stream->ufence_syncobj);
> +	mutex_unlock(&stream->stream_lock);
>  	xe_pm_runtime_put(gt_to_xe(stream->gt));
>  }
>  
> @@ -1775,14 +1823,24 @@ static int xe_oa_release(struct inode *inode, struct file *file)
>  	struct xe_oa_stream *stream = file->private_data;
>  	struct xe_gt *gt = stream->gt;
>  
> +	/* gt_lock serializes against xe_oa_fini(), which destroys the stream on device removal */
>  	mutex_lock(&gt->oa.gt_lock);
> -	xe_oa_destroy_locked(stream);
> +	if (!stream->dead)
> +		xe_oa_destroy_locked(stream);
>  	mutex_unlock(&gt->oa.gt_lock);
>  
>  	xe_oa_free_oa_buffer(stream);
>  	mutex_destroy(&stream->stream_lock);
>  	kfree(stream);
>  
> +	/*
> +	 * Freeing the OA buffer may defer the BO destroy to the TTM workqueue. Once the device
> +	 * is unplugged, this may be the last reference on the drm device, so let the destroy
> +	 * finish before drm_dev_put() releases the GGTT.
> +	 */
> +	if (drm_dev_is_unplugged(&gt_to_xe(gt)->drm))
> +		flush_workqueue(gt_to_xe(gt)->ttm.wq);
> +
>  	/* Release the reference the OA stream kept on the driver */
>  	drm_dev_put(&gt_to_xe(gt)->drm);

[Severity: Medium]
Can xe_oa_release() race with drm_dev_unplug() and skip flushing
xe->ttm.wq before dropping the final drm_device reference?

CPU0                                  CPU1
xe_oa_release()
  xe_oa_free_oa_buffer(stream)
    ...
      queue_work(xe->ttm.wq,
                 &bo->delayed_delete)
  drm_dev_is_unplugged(&xe->drm)
    --> false
                                      xe_device_remove()
                                        drm_dev_unplug(&xe->drm)
                                        ...
                                        drm_dev_put(&xe->drm)
  drm_dev_put(&xe->drm)
    drm_managed_release()
      ggtt_fini_early()
        drm_mm_takedown(&ggtt->mm)
                                      ttm_bo_delayed_delete()
                                        xe_ttm_bo_destroy()
                                          xe_ggtt_remove_bo(ggtt, bo)
                                            <-- use-after-free on ggtt

drm_dev_is_unplugged() drops its SRCU read lock immediately, and nothing
during device removal flushes xe->ttm.wq for an already-deleted BO.  Because
ggtt_fini_early() is registered with drmm after xe_device_destroy()
(which calls ttm_device_fini() to drain xe->ttm.wq), the final
drm_dev_put() tears down and frees tile->mem.ggtt while
bo->delayed_delete is still pending on xe->ttm.wq.

[Severity: Medium]
Even after the device is already unplugged, can a temporary reference on
stream->oa_buffer.bo outlive xe_oa_free_oa_buffer() and cause
flush_workqueue(xe->ttm.wq) to run before bo->delayed_delete is queued?

CPU0                                  CPU1
xe_device_remove()
  ...
  xe_bo_pinned_fini()
    xe_bo_apply_to_pinned()
      xe_bo_get(bo)
      spin_unlock(&xe->pinned.lock)
                                      xe_oa_release()
                                        xe_oa_free_oa_buffer()
                                          xe_bo_unpin_map_no_vm(bo)
                                            xe_bo_put(bo)
                                        flush_workqueue(xe->ttm.wq)
      xe_bo_dma_unmap_pinned(bo)
      xe_bo_put(bo)
        queue_work(xe->ttm.wq,
                   &bo->delayed_delete)
  drm_dev_put(&xe->drm)
                                        drm_dev_put(&xe->drm)
                                          drm_managed_release()
                                            ggtt_fini_early()
                                              <-- ggtt freed before
                                                  bo->delayed_delete runs

Because xe_oa_stream_destroy() leaves stream->oa_buffer.bo on
xe->pinned.early.kernel_bo_present, xe_bo_pinned_fini() still visits it
and holds a reference outside xe->pinned.lock.  Similarly, in
xe_bo_unpin_map_no_vm(), xe_bo_unpin() puts the BO back on the TTM LRU
and unlocks dma-resv before xe_bo_put(bo), allowing xe_shrinker_scan() to
grab a temporary BO reference.  In either case, xe_oa_release() flushes
xe->ttm.wq before the last BO reference is dropped.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261119002737.3899877-10-umesh.nerlige.ramappa@intel.com?part=8

  reply	other threads:[~2026-11-19  1:11 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19  0:27 [PATCH v8 0/8] Cleanup OA stream on PCI unbind/remove Umesh Nerlige Ramappa
2026-11-19  0:27 ` [PATCH v8 1/8] drm/xe/oa: Ignore a HW tail pointer outside the OA buffer Umesh Nerlige Ramappa
2026-11-19  0:39   ` sashiko-bot
2026-11-19  0:27 ` [PATCH v8 2/8] drm/xe/oa: Use the drm device address_space for the OA stream fd Umesh Nerlige Ramappa
2026-11-19  0:39   ` sashiko-bot
2026-11-19  0:27 ` [PATCH v8 3/8] drm/xe/oa: Wait for in-flight OA batches before stream HW teardown Umesh Nerlige Ramappa
2026-11-19  0:27 ` [PATCH v8 4/8] drm/xe: Reject vm_ops->access on BO mappings of an unplugged device Umesh Nerlige Ramappa
2026-11-19  0:39   ` sashiko-bot
2026-11-19  0:27 ` [PATCH v8 5/8] drm/xe/oa: Fence OA ioctls and mmap against device unplug and wedging Umesh Nerlige Ramappa
2026-11-19  0:27 ` [PATCH v8 6/8] drm/xe/oa: Take the runtime PM reference in xe_oa_destroy_locked() Umesh Nerlige Ramappa
2026-11-19  0:27 ` [PATCH v8 7/8] drm/xe/oa: Free the OA buffer and the stream after the stream destroy Umesh Nerlige Ramappa
2026-11-19  0:48   ` sashiko-bot
2026-11-19  0:27 ` [PATCH v8 8/8] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset Umesh Nerlige Ramappa
2026-11-19  1:11   ` sashiko-bot [this message]
2026-11-19  6:05 ` ✓ CI.KUnit: success for Cleanup OA stream on PCI unbind/remove (rev8) Patchwork
2026-11-19  6:46 ` ✓ Xe.CI.BAT: " Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-222279@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    --cc=umesh.nerlige.ramappa@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox