Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Shekhar Chauhan <shekhar.chauhan@intel.com>
To: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>,
	<intel-xe@lists.freedesktop.org>
Subject: Re: [PATCH v8 8/8] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset
Date: Fri, 20 Nov 2026 13:01:04 +0530	[thread overview]
Message-ID: <93e5b6be-6087-43a2-b8c8-8607cbcc7d71@intel.com> (raw)
In-Reply-To: <sashiko-outbox-222279@kernel.org>


On 11/19/2026 6:41, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 5 potential
> issues to consider:
> - [Medium] drm/xe/oa: drm_dev_is_unplugged() race in xe_oa_release()
>    skips ttm.wq flush
> - [Medium] drm/xe/oa: OA buffer BO references can outlive
>    flush_workqueue() on unplug
> - [Medium] drm/xe/oa: missing or blocking drain in xe_oa_stream_init()
>    error path
> - [Medium] drm/xe/oa: NULL sg_table deref on resume after
>    xe_bo_dma_unmap_pinned()
> - [Medium] drm/xe/oa: unconditional drain and batch submit on wedged
>    device in destroy
> --
>
> commit 2b379416db1422398b8cce664304d4acea228c64
> Author: Umesh Nerlige Ramappa <umesh.nerlige.ramappa@intel.com>
>
> drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset
>
> This commit quiesces active OA streams on driver unbind or PCI reset by
> marking them dead and tearing down their hardware and device resources
> while keeping the stream structure and pinned OA buffer allocated until
> the stream file descriptor is closed.

I'm unsure on the case reported by Sashiko.  Please have a look.

-shekhar

>
>> diff --git a/drivers/gpu/drm/xe/xe_oa.c b/drivers/gpu/drm/xe/xe_oa.c
>> index 223419347eb7b..0b51bc818ef47 100644
>> --- a/drivers/gpu/drm/xe/xe_oa.c
>> +++ b/drivers/gpu/drm/xe/xe_oa.c
> [ ... ]
>
>> @@ -937,6 +959,13 @@ static void xe_oa_stream_destroy(struct xe_oa_stream *stream)
>>   	xe_oa_disable_metric_set(stream);
>>   	xe_exec_queue_put(stream->k_exec_q);
>>   
>> +	/*
>> +	 * The OA buffer is freed on release of the stream fd, an mmap may still hold a
>> +	 * reference on it. Drop its DMA mapping now, while pinned: the fd may be released, and
>> +	 * the BO unpinned, while xe_bo_pinned_fini() DMA-unmaps it without the BO lock.
>> +	 */
>> +	xe_bo_dma_unmap_pinned(stream->oa_buffer.bo);
>> +
>>   	xe_force_wake_put(gt_to_fw(gt), stream->fw_ref);
>>   	xe_pm_runtime_put(stream->oa->xe);
> [Severity: Medium]
> Can calling xe_bo_dma_unmap_pinned() here while stream->oa_buffer.bo stays
> pinned on xe->pinned.early.kernel_bo_present cause a NULL pointer
> dereference if the device suspends and resumes before
> xe_oa_free_oa_buffer() unpins the BO?
>
> CPU0                                  CPU1
> xe_oa_release()
>    xe_oa_destroy_locked()
>      xe_oa_stream_destroy()
>        xe_bo_dma_unmap_pinned(bo)
>          xe_tt->sg = NULL
>        xe_pm_runtime_put()
>      xe_pm_runtime_put()
>                                        xe_pm_suspend()
>                                          xe_bo_evict_all()
>                                        xe_pm_resume()
>                                          xe_bo_restore_early()
>                                            xe_bo_restore_and_map_ggtt()
>                                              xe_ggtt_map_bo()
>                                                xe_res_first_sg(NULL, ...)
>                                                  <-- NULL pointer dereference
>    xe_oa_free_oa_buffer()
>      xe_bo_unpin_map_no_vm(bo)
>
> Non-MERT OA buffers are created with XE_BO_FLAG_SYSTEM | XE_BO_FLAG_GGTT
> and remain linked on xe->pinned.early.kernel_bo_present until
> xe_bo_unpin() removes bo->pinned_link.  Once xe_oa_destroy_locked() drops
> the last runtime PM reference, a system or D3cold runtime suspend/resume
> cycle will evict and restore the pinned BO via xe_ggtt_map_bo(), passing
> xe_bo_sg(bo) == NULL to xe_res_first_sg().
>
> [ ... ]
>
>> @@ -1754,19 +1786,35 @@ static long xe_oa_ioctl(struct file *file,
>>   	return ret;
>>   }
>>   
>> +/*
>> + * Disable the stream, restore the HW state and release everything but the OA buffer and the
>> + * stream struct, the same whether the stream fd is closed or the device goes away, and
>> + * whether or not the HW still responds: all HW waits are bounded. Afterwards the stream is
>> + * dead and any further read, poll, mmap or ioctl on the stream fd fails with -ENODEV.
>> + */
>>   static void xe_oa_destroy_locked(struct xe_oa_stream *stream)
>>   {
>> +	unsigned long flags;
>> +
>>   	xe_pm_runtime_get(gt_to_xe(stream->gt));
>> +	mutex_lock(&stream->stream_lock);
>> +
>> +	spin_lock_irqsave(&stream->oa_buffer.ptr_lock, flags);
>> +	WRITE_ONCE(stream->dead, true);
>> +	spin_unlock_irqrestore(&stream->oa_buffer.ptr_lock, flags);
>>   
>>   	if (stream->enabled)
>>   		xe_oa_disable_locked(stream);
>>   
>> +	wake_up_all(&stream->poll_wq);
>> +
>>   	xe_oa_stream_destroy(stream);
> [Severity: Medium]
> When the device is wedged, does xe_oa_stream_destroy() stall for multiple
> seconds and log a timeout error?
>
> xe_oa_stream_destroy() unconditionally calls xe_oa_stream_drain() and
> xe_oa_disable_metric_set() without checking xe_wedged().  If
> stream->last_fence is unsignaled when the device wedges (for example when
> a config batch is waiting on an input fence or after xe_oa_emit_oa_config()
> breaks out of its wait loop on wedge), xe_oa_stream_drain() sleeps
> uninterruptibly for job_timeout_ms + HZ and prints "Timed out waiting for
> OA batch".  Then, when stream->exec_q is set, xe_oa_disable_metric_set()
> calls xe_oa_configure_oa_context(stream, false) to submit a new batch to
> the already-wedged queue (which guc_exec_queue_run_job() does not submit
> to hardware) and waits again in xe_oa_stream_drain() until TDR or timeout.
>
> [Severity: Medium]
> Similarly, in xe_oa_stream_init(), what happens when
> xe_oa_enable_metric_set() -> xe_oa_emit_oa_config() cuts its synchronous
> wait short on device unplug or wedge (introduced in commit "drm/xe/oa:
> Fence OA ioctls and mmap against device unplug and wedging")?
>
> drivers/gpu/drm/xe/xe_oa.c:xe_oa_stream_init() {
> 	...
> err_put_k_exec_q:
> 	xe_oa_disable_metric_set(stream);
> 	xe_exec_queue_put(stream->k_exec_q);
> err_free_oa_buf:
> 	xe_oa_free_oa_buffer(stream);
> err_fw_put:
> 	xe_force_wake_put(gt_to_fw(gt), stream->fw_ref);
> 	xe_pm_runtime_put(stream->oa->xe);
> 	xe_oa_free_configs(stream);
> 	...
> }
>
> At that point the config batch has already been submitted to
> stream->k_exec_q and recorded in stream->last_fence.  Unlike
> xe_oa_stream_destroy(), err_put_k_exec_q does not call
> xe_oa_stream_drain() before xe_oa_disable_metric_set().  When
> stream->exec_q is NULL, the in-flight config batch on stream->k_exec_q is
> never drained before OA registers, forcewake, and runtime PM are released.
> When stream->exec_q is non-NULL, xe_oa_disable_metric_set() queues a
> context-restore batch behind the unsignaled config batch and blocks in
> xe_oa_stream_drain() for job_timeout_ms + HZ while still inside the
> drm_dev_enter() section of xe_observation_ioctl(), blocking
> drm_dev_unplug().
>
>>   
>>   	if (stream->exec_q)
>>   		xe_exec_queue_put(stream->exec_q);
>>   
>>   	drm_syncobj_put(stream->ufence_syncobj);
>> +	mutex_unlock(&stream->stream_lock);
>>   	xe_pm_runtime_put(gt_to_xe(stream->gt));
>>   }
>>   
>> @@ -1775,14 +1823,24 @@ static int xe_oa_release(struct inode *inode, struct file *file)
>>   	struct xe_oa_stream *stream = file->private_data;
>>   	struct xe_gt *gt = stream->gt;
>>   
>> +	/* gt_lock serializes against xe_oa_fini(), which destroys the stream on device removal */
>>   	mutex_lock(&gt->oa.gt_lock);
>> -	xe_oa_destroy_locked(stream);
>> +	if (!stream->dead)
>> +		xe_oa_destroy_locked(stream);
>>   	mutex_unlock(&gt->oa.gt_lock);
>>   
>>   	xe_oa_free_oa_buffer(stream);
>>   	mutex_destroy(&stream->stream_lock);
>>   	kfree(stream);
>>   
>> +	/*
>> +	 * Freeing the OA buffer may defer the BO destroy to the TTM workqueue. Once the device
>> +	 * is unplugged, this may be the last reference on the drm device, so let the destroy
>> +	 * finish before drm_dev_put() releases the GGTT.
>> +	 */
>> +	if (drm_dev_is_unplugged(&gt_to_xe(gt)->drm))
>> +		flush_workqueue(gt_to_xe(gt)->ttm.wq);
>> +
>>   	/* Release the reference the OA stream kept on the driver */
>>   	drm_dev_put(&gt_to_xe(gt)->drm);
> [Severity: Medium]
> Can xe_oa_release() race with drm_dev_unplug() and skip flushing
> xe->ttm.wq before dropping the final drm_device reference?
>
> CPU0                                  CPU1
> xe_oa_release()
>    xe_oa_free_oa_buffer(stream)
>      ...
>        queue_work(xe->ttm.wq,
>                   &bo->delayed_delete)
>    drm_dev_is_unplugged(&xe->drm)
>      --> false
>                                        xe_device_remove()
>                                          drm_dev_unplug(&xe->drm)
>                                          ...
>                                          drm_dev_put(&xe->drm)
>    drm_dev_put(&xe->drm)
>      drm_managed_release()
>        ggtt_fini_early()
>          drm_mm_takedown(&ggtt->mm)
>                                        ttm_bo_delayed_delete()
>                                          xe_ttm_bo_destroy()
>                                            xe_ggtt_remove_bo(ggtt, bo)
>                                              <-- use-after-free on ggtt
>
> drm_dev_is_unplugged() drops its SRCU read lock immediately, and nothing
> during device removal flushes xe->ttm.wq for an already-deleted BO.  Because
> ggtt_fini_early() is registered with drmm after xe_device_destroy()
> (which calls ttm_device_fini() to drain xe->ttm.wq), the final
> drm_dev_put() tears down and frees tile->mem.ggtt while
> bo->delayed_delete is still pending on xe->ttm.wq.
>
> [Severity: Medium]
> Even after the device is already unplugged, can a temporary reference on
> stream->oa_buffer.bo outlive xe_oa_free_oa_buffer() and cause
> flush_workqueue(xe->ttm.wq) to run before bo->delayed_delete is queued?
>
> CPU0                                  CPU1
> xe_device_remove()
>    ...
>    xe_bo_pinned_fini()
>      xe_bo_apply_to_pinned()
>        xe_bo_get(bo)
>        spin_unlock(&xe->pinned.lock)
>                                        xe_oa_release()
>                                          xe_oa_free_oa_buffer()
>                                            xe_bo_unpin_map_no_vm(bo)
>                                              xe_bo_put(bo)
>                                          flush_workqueue(xe->ttm.wq)
>        xe_bo_dma_unmap_pinned(bo)
>        xe_bo_put(bo)
>          queue_work(xe->ttm.wq,
>                     &bo->delayed_delete)
>    drm_dev_put(&xe->drm)
>                                          drm_dev_put(&xe->drm)
>                                            drm_managed_release()
>                                              ggtt_fini_early()
>                                                <-- ggtt freed before
>                                                    bo->delayed_delete runs
>
> Because xe_oa_stream_destroy() leaves stream->oa_buffer.bo on
> xe->pinned.early.kernel_bo_present, xe_bo_pinned_fini() still visits it
> and holds a reference outside xe->pinned.lock.  Similarly, in
> xe_bo_unpin_map_no_vm(), xe_bo_unpin() puts the BO back on the TTM LRU
> and unlocks dma-resv before xe_bo_put(bo), allowing xe_shrinker_scan() to
> grab a temporary BO reference.  In either case, xe_oa_release() flushes
> xe->ttm.wq before the last BO reference is dropped.
>
-- 
Shekhar Chauhan
Linux Graphics Software Engineer
Linux Platform Enabling
Intel Corporation


  reply	other threads:[~2026-11-20  7:31 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19  0:27 [PATCH v8 0/8] Cleanup OA stream on PCI unbind/remove Umesh Nerlige Ramappa
2026-11-19  0:27 ` [PATCH v8 1/8] drm/xe/oa: Ignore a HW tail pointer outside the OA buffer Umesh Nerlige Ramappa
2026-11-19  0:39   ` sashiko-bot
2026-11-20  7:21   ` Shekhar Chauhan
2026-11-19  0:27 ` [PATCH v8 2/8] drm/xe/oa: Use the drm device address_space for the OA stream fd Umesh Nerlige Ramappa
2026-11-19  0:39   ` sashiko-bot
2026-11-20  7:23   ` Shekhar Chauhan
2026-11-19  0:27 ` [PATCH v8 3/8] drm/xe/oa: Wait for in-flight OA batches before stream HW teardown Umesh Nerlige Ramappa
2026-11-20  7:24   ` Shekhar Chauhan
2026-11-19  0:27 ` [PATCH v8 4/8] drm/xe: Reject vm_ops->access on BO mappings of an unplugged device Umesh Nerlige Ramappa
2026-11-19  0:39   ` sashiko-bot
2026-11-19 22:36   ` Matthew Brost
2026-11-19  0:27 ` [PATCH v8 5/8] drm/xe/oa: Fence OA ioctls and mmap against device unplug and wedging Umesh Nerlige Ramappa
2026-11-20  7:25   ` Shekhar Chauhan
2026-11-19  0:27 ` [PATCH v8 6/8] drm/xe/oa: Take the runtime PM reference in xe_oa_destroy_locked() Umesh Nerlige Ramappa
2026-11-20  7:27   ` Shekhar Chauhan
2026-11-19  0:27 ` [PATCH v8 7/8] drm/xe/oa: Free the OA buffer and the stream after the stream destroy Umesh Nerlige Ramappa
2026-11-19  0:48   ` sashiko-bot
2026-11-20  7:29   ` Shekhar Chauhan
2026-11-19  0:27 ` [PATCH v8 8/8] drm/xe/oa: Quiesce OA streams on driver unbind and PCI reset Umesh Nerlige Ramappa
2026-11-19  1:11   ` sashiko-bot
2026-11-20  7:31     ` Shekhar Chauhan [this message]
2026-11-19  6:05 ` ✓ CI.KUnit: success for Cleanup OA stream on PCI unbind/remove (rev8) Patchwork
2026-11-19  6:46 ` ✓ Xe.CI.BAT: " Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=93e5b6be-6087-43a2-b8c8-8607cbcc7d71@intel.com \
    --to=shekhar.chauhan@intel.com \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=umesh.nerlige.ramappa@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox