Linux kernel -stable discussions
 help / color / mirror / Atom feed
* [PATCH v3] accel/ivpu: Defer recovery until probe completes
@ 2026-11-10 12:15 Karol Wachowski
  2026-11-12 17:35 ` Jeff Hugo
  2026-11-15 15:24 ` Toast
  0 siblings, 2 replies; 4+ messages in thread
From: Karol Wachowski @ 2026-11-10 12:15 UTC (permalink / raw)
  To: Daksh Sahni, dri-devel
  Cc: Andrzej Kacprowski, Oded Gabbay, Jeff Hugo, dawid.osuchowski,
	Karol Wachowski, stable

An error IRQ that fires after ivpu_boot() enables interrupts queues
recovery_work during probe. If probe then fails, the device is torn
down while the work runs, and the work faults on the unmapped BARs:

  BUG: unable to handle page fault for address: ffffcfc6680200f4
  Workqueue: events_unbound ivpu_pm_recovery_work [intel_vpu]
  RIP: 0010:ivpu_hw_ip_ipc_rx_addr_get+0xc/0xb0 [intel_vpu]
  Call Trace:
   ivpu_wait_for_ready+0x155/0x220 [intel_vpu]
   ivpu_boot+0x9e/0x270 [intel_vpu]
   ivpu_resume+0xa4/0xd0 [intel_vpu]
   ivpu_pm_reset_complete+0x2c/0x90 [intel_vpu]
   ivpu_pm_recovery_work+0xf6/0x140 [intel_vpu]

If probe succeeds, the recovery runs concurrently with the rest of
probe, and its pm_runtime_disable() discards the initial autosuspend
request, leaving the NPU active until first use. Any recovery that
starts while the NPU is idle loses the pending autosuspend the same way.

Keep recovery_work disabled from ivpu_pm_init() until ivpu_pm_enable().
A trigger during probe still runs diagnostics and masks interrupts, and
ivpu_pm_enable() runs the deferred recovery before the initial
autosuspend request. A trigger racing with ivpu_pm_enable() can cause
a second, harmless recovery. Request autosuspend at the end of
ivpu_pm_reset_complete() so the NPU suspends again after every reset.

Fixes: 852be13f3bd3 ("accel/ivpu: Add PM support")
Cc: stable@vger.kernel.org
Reported-by: Daksh Sahni <swizzguy76@gmail.com>
Closes: https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
Signed-off-by: Karol Wachowski <karol.wachowski@linux.intel.com>
---

Notes:
    v3:
    - Request autosuspend at the end of ivpu_pm_reset_complete().
      pm_runtime_disable() in ivpu_pm_reset_begin() cancels a pending
      autosuspend request and pm_runtime_enable() does not re-issue it, so
      a recovery racing with the initial autosuspend request in
      ivpu_pm_enable() left the NPU active until first use (Sashiko).
      A trigger racing with ivpu_pm_enable() can still queue recovery
      twice. The second run is a full reset that leaves the device in a
      consistent state, and autosuspend is now re-requested after each
      reset.
    - Replace pm_runtime_mark_last_busy() with pm_request_autosuspend() in
      ivpu_pm_reset_complete(); it marks the device busy itself.
    - Shorten the commit message.
    - v2: https://lore.kernel.org/r/20261110085003.3527014-1-karol.wachowski@linux.intel.com

    v2:
    - Rework of Daksh's "accel/ivpu: Disable recovery work on probe failure":
      https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
    - Keep recovery_work disabled from ivpu_pm_init() until ivpu_pm_enable()
      instead of disabling it on the probe error path, which is too late as
      recovery may already run concurrently with ivpu_boot() (Sashiko)
    - Run a recovery requested during probe from ivpu_pm_enable() and wait
      for it before the initial autosuspend request, so the NPU is reset
      before use and the autosuspend is not lost when probe succeeds

    v1:
    - Link: https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com

 drivers/accel/ivpu/ivpu_pm.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/accel/ivpu/ivpu_pm.c b/drivers/accel/ivpu/ivpu_pm.c
index aa813262253e..e4b79c946ca0 100644
--- a/drivers/accel/ivpu/ivpu_pm.c
+++ b/drivers/accel/ivpu/ivpu_pm.c
@@ -150,8 +150,8 @@ static void ivpu_pm_reset_complete(struct ivpu_device *vdev)
 	up_write(&vdev->pm->reset_lock);
 	atomic_set(&vdev->pm->reset_pending, 0);
 
-	pm_runtime_mark_last_busy(vdev->drm.dev);
 	pm_runtime_enable(vdev->drm.dev);
+	pm_request_autosuspend(vdev->drm.dev);
 }
 
 static void ivpu_pm_recovery_work(struct work_struct *work)
@@ -415,6 +415,7 @@ void ivpu_pm_init(struct ivpu_device *vdev)
 	atomic_set(&pm->engine_reset_counter, 0);
 
 	INIT_WORK(&pm->recovery_work, ivpu_pm_recovery_work);
+	disable_work(&pm->recovery_work);
 	INIT_DELAYED_WORK(&pm->job_timeout_work, ivpu_job_timeout_work);
 
 	if (ivpu_disable_recovery)
@@ -439,6 +440,12 @@ void ivpu_pm_enable(struct ivpu_device *vdev)
 {
 	struct device *dev = vdev->drm.dev;
 
+	enable_work(&vdev->pm->recovery_work);
+	if (atomic_read(&vdev->pm->reset_pending)) {
+		queue_work(system_dfl_wq, &vdev->pm->recovery_work);
+		flush_work(&vdev->pm->recovery_work);
+	}
+
 	pm_runtime_allow(dev);
 	pm_runtime_put_autosuspend(dev);
 }

base-commit: 54d3ff1b5ac08d0e28029b93a1719ddd9ff670a4
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH v3] accel/ivpu: Defer recovery until probe completes
  2026-11-10 12:15 [PATCH v3] accel/ivpu: Defer recovery until probe completes Karol Wachowski
@ 2026-11-12 17:35 ` Jeff Hugo
  2026-11-15 15:24 ` Toast
  1 sibling, 0 replies; 4+ messages in thread
From: Jeff Hugo @ 2026-11-12 17:35 UTC (permalink / raw)
  To: Karol Wachowski, Daksh Sahni, dri-devel
  Cc: Andrzej Kacprowski, Oded Gabbay, dawid.osuchowski, stable

On 11/10/2026 5:15 AM, Karol Wachowski wrote:
> An error IRQ that fires after ivpu_boot() enables interrupts queues
> recovery_work during probe. If probe then fails, the device is torn
> down while the work runs, and the work faults on the unmapped BARs:
> 
>    BUG: unable to handle page fault for address: ffffcfc6680200f4
>    Workqueue: events_unbound ivpu_pm_recovery_work [intel_vpu]
>    RIP: 0010:ivpu_hw_ip_ipc_rx_addr_get+0xc/0xb0 [intel_vpu]
>    Call Trace:
>     ivpu_wait_for_ready+0x155/0x220 [intel_vpu]
>     ivpu_boot+0x9e/0x270 [intel_vpu]
>     ivpu_resume+0xa4/0xd0 [intel_vpu]
>     ivpu_pm_reset_complete+0x2c/0x90 [intel_vpu]
>     ivpu_pm_recovery_work+0xf6/0x140 [intel_vpu]
> 
> If probe succeeds, the recovery runs concurrently with the rest of
> probe, and its pm_runtime_disable() discards the initial autosuspend
> request, leaving the NPU active until first use. Any recovery that
> starts while the NPU is idle loses the pending autosuspend the same way.
> 
> Keep recovery_work disabled from ivpu_pm_init() until ivpu_pm_enable().
> A trigger during probe still runs diagnostics and masks interrupts, and
> ivpu_pm_enable() runs the deferred recovery before the initial
> autosuspend request. A trigger racing with ivpu_pm_enable() can cause
> a second, harmless recovery. Request autosuspend at the end of
> ivpu_pm_reset_complete() so the NPU suspends again after every reset.
> 
> Fixes: 852be13f3bd3 ("accel/ivpu: Add PM support")
> Cc: stable@vger.kernel.org
> Reported-by: Daksh Sahni <swizzguy76@gmail.com>
> Closes: https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
> Signed-off-by: Karol Wachowski <karol.wachowski@linux.intel.com>
Reviewed-by: Jeff Hugo <jeff.hugo@oss.qualcomm.com>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v3] accel/ivpu: Defer recovery until probe completes
  2026-11-10 12:15 [PATCH v3] accel/ivpu: Defer recovery until probe completes Karol Wachowski
  2026-11-12 17:35 ` Jeff Hugo
@ 2026-11-15 15:24 ` Toast
  2026-11-16  7:09   ` Karol Wachowski
  1 sibling, 1 reply; 4+ messages in thread
From: Toast @ 2026-11-15 15:24 UTC (permalink / raw)
  To: Karol Wachowski
  Cc: dri-devel, Andrzej Kacprowski, Oded Gabbay, Jeff Hugo,
	dawid.osuchowski, stable

Tested v3 on Meteor Lake (Core Ultra 5 125H, vpu_37xx firmware 20260219)
on top of v7.4-rc1 (d48723711bae):

- 4 kexec runs with the NPU runtime-suspended (one from a 7.0 kernel,
  three from 7.4): every run hit ATS_ERR during probe, recovery_work was
  not queued, probe failed with -ETIMEDOUT, and there was no oops
- FLR plus rebind brings the NPU back, and it autosuspends afterwards
- debugfs force_recovery after a successful probe recovers the NPU, and
  it returns to runtime-suspended

None of the runs had probe succeed after the error IRQ, so the deferred
recovery in ivpu_pm_enable() isn't covered by this.

One question: v2 had "Cc: stable@vger.kernel.org # v6.10+" and v3 dropped
the annotation. disable_work()/enable_work() only exist since v6.10 and
the Fixes: commit is in v6.3, so should the "# v6.10+" come back?

Tested-by: Daksh Sahni <swizzguy76@gmail.com>

On Tue, Nov 10, 2026 at 5:45 PM Karol Wachowski
<karol.wachowski@linux.intel.com> wrote:
>
> An error IRQ that fires after ivpu_boot() enables interrupts queues
> recovery_work during probe. If probe then fails, the device is torn
> down while the work runs, and the work faults on the unmapped BARs:
>
>   BUG: unable to handle page fault for address: ffffcfc6680200f4
>   Workqueue: events_unbound ivpu_pm_recovery_work [intel_vpu]
>   RIP: 0010:ivpu_hw_ip_ipc_rx_addr_get+0xc/0xb0 [intel_vpu]
>   Call Trace:
>    ivpu_wait_for_ready+0x155/0x220 [intel_vpu]
>    ivpu_boot+0x9e/0x270 [intel_vpu]
>    ivpu_resume+0xa4/0xd0 [intel_vpu]
>    ivpu_pm_reset_complete+0x2c/0x90 [intel_vpu]
>    ivpu_pm_recovery_work+0xf6/0x140 [intel_vpu]
>
> If probe succeeds, the recovery runs concurrently with the rest of
> probe, and its pm_runtime_disable() discards the initial autosuspend
> request, leaving the NPU active until first use. Any recovery that
> starts while the NPU is idle loses the pending autosuspend the same way.
>
> Keep recovery_work disabled from ivpu_pm_init() until ivpu_pm_enable().
> A trigger during probe still runs diagnostics and masks interrupts, and
> ivpu_pm_enable() runs the deferred recovery before the initial
> autosuspend request. A trigger racing with ivpu_pm_enable() can cause
> a second, harmless recovery. Request autosuspend at the end of
> ivpu_pm_reset_complete() so the NPU suspends again after every reset.
>
> Fixes: 852be13f3bd3 ("accel/ivpu: Add PM support")
> Cc: stable@vger.kernel.org
> Reported-by: Daksh Sahni <swizzguy76@gmail.com>
> Closes: https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
> Signed-off-by: Karol Wachowski <karol.wachowski@linux.intel.com>
> ---
>
> Notes:
>     v3:
>     - Request autosuspend at the end of ivpu_pm_reset_complete().
>       pm_runtime_disable() in ivpu_pm_reset_begin() cancels a pending
>       autosuspend request and pm_runtime_enable() does not re-issue it, so
>       a recovery racing with the initial autosuspend request in
>       ivpu_pm_enable() left the NPU active until first use (Sashiko).
>       A trigger racing with ivpu_pm_enable() can still queue recovery
>       twice. The second run is a full reset that leaves the device in a
>       consistent state, and autosuspend is now re-requested after each
>       reset.
>     - Replace pm_runtime_mark_last_busy() with pm_request_autosuspend() in
>       ivpu_pm_reset_complete(); it marks the device busy itself.
>     - Shorten the commit message.
>     - v2: https://lore.kernel.org/r/20261110085003.3527014-1-karol.wachowski@linux.intel.com
>
>     v2:
>     - Rework of Daksh's "accel/ivpu: Disable recovery work on probe failure":
>       https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
>     - Keep recovery_work disabled from ivpu_pm_init() until ivpu_pm_enable()
>       instead of disabling it on the probe error path, which is too late as
>       recovery may already run concurrently with ivpu_boot() (Sashiko)
>     - Run a recovery requested during probe from ivpu_pm_enable() and wait
>       for it before the initial autosuspend request, so the NPU is reset
>       before use and the autosuspend is not lost when probe succeeds
>
>     v1:
>     - Link: https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
>
>  drivers/accel/ivpu/ivpu_pm.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/accel/ivpu/ivpu_pm.c b/drivers/accel/ivpu/ivpu_pm.c
> index aa813262253e..e4b79c946ca0 100644
> --- a/drivers/accel/ivpu/ivpu_pm.c
> +++ b/drivers/accel/ivpu/ivpu_pm.c
> @@ -150,8 +150,8 @@ static void ivpu_pm_reset_complete(struct ivpu_device *vdev)
>         up_write(&vdev->pm->reset_lock);
>         atomic_set(&vdev->pm->reset_pending, 0);
>
> -       pm_runtime_mark_last_busy(vdev->drm.dev);
>         pm_runtime_enable(vdev->drm.dev);
> +       pm_request_autosuspend(vdev->drm.dev);
>  }
>
>  static void ivpu_pm_recovery_work(struct work_struct *work)
> @@ -415,6 +415,7 @@ void ivpu_pm_init(struct ivpu_device *vdev)
>         atomic_set(&pm->engine_reset_counter, 0);
>
>         INIT_WORK(&pm->recovery_work, ivpu_pm_recovery_work);
> +       disable_work(&pm->recovery_work);
>         INIT_DELAYED_WORK(&pm->job_timeout_work, ivpu_job_timeout_work);
>
>         if (ivpu_disable_recovery)
> @@ -439,6 +440,12 @@ void ivpu_pm_enable(struct ivpu_device *vdev)
>  {
>         struct device *dev = vdev->drm.dev;
>
> +       enable_work(&vdev->pm->recovery_work);
> +       if (atomic_read(&vdev->pm->reset_pending)) {
> +               queue_work(system_dfl_wq, &vdev->pm->recovery_work);
> +               flush_work(&vdev->pm->recovery_work);
> +       }
> +
>         pm_runtime_allow(dev);
>         pm_runtime_put_autosuspend(dev);
>  }
>
> base-commit: 54d3ff1b5ac08d0e28029b93a1719ddd9ff670a4
> --
> 2.43.0
>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v3] accel/ivpu: Defer recovery until probe completes
  2026-11-15 15:24 ` Toast
@ 2026-11-16  7:09   ` Karol Wachowski
  0 siblings, 0 replies; 4+ messages in thread
From: Karol Wachowski @ 2026-11-16  7:09 UTC (permalink / raw)
  To: Toast
  Cc: dri-devel, Andrzej Kacprowski, Oded Gabbay, Jeff Hugo,
	dawid.osuchowski, stable

On 15-Nov-26 16:24, Toast wrote:
> Tested v3 on Meteor Lake (Core Ultra 5 125H, vpu_37xx firmware 20260219)
> on top of v7.4-rc1 (d48723711bae):
> 
> - 4 kexec runs with the NPU runtime-suspended (one from a 7.0 kernel,
>    three from 7.4): every run hit ATS_ERR during probe, recovery_work was
>    not queued, probe failed with -ETIMEDOUT, and there was no oops
> - FLR plus rebind brings the NPU back, and it autosuspends afterwards
> - debugfs force_recovery after a successful probe recovers the NPU, and
>    it returns to runtime-suspended
> 
> None of the runs had probe succeed after the error IRQ, so the deferred
> recovery in ivpu_pm_enable() isn't covered by this.
> 
> One question: v2 had"Cc: stable@vger.kernel.org # v6.10+" and v3 dropped
> the annotation. disable_work()/enable_work() only exist since v6.10 and
> the Fixes: commit is in v6.3, so should the "# v6.10+" come back?

Thanks for testing, and good catch. I added "# v6.10+" to the stable tag 
when applying.

Applied to drm-misc-fixes.

> 
> Tested-by: Daksh Sahni<swizzguy76@gmail.com>
> 
> On Tue, Nov 10, 2026 at 5:45 PM Karol Wachowski
> <karol.wachowski@linux.intel.com> wrote:
>> An error IRQ that fires after ivpu_boot() enables interrupts queues
>> recovery_work during probe. If probe then fails, the device is torn
>> down while the work runs, and the work faults on the unmapped BARs:
>>
>>    BUG: unable to handle page fault for address: ffffcfc6680200f4
>>    Workqueue: events_unbound ivpu_pm_recovery_work [intel_vpu]
>>    RIP: 0010:ivpu_hw_ip_ipc_rx_addr_get+0xc/0xb0 [intel_vpu]
>>    Call Trace:
>>     ivpu_wait_for_ready+0x155/0x220 [intel_vpu]
>>     ivpu_boot+0x9e/0x270 [intel_vpu]
>>     ivpu_resume+0xa4/0xd0 [intel_vpu]
>>     ivpu_pm_reset_complete+0x2c/0x90 [intel_vpu]
>>     ivpu_pm_recovery_work+0xf6/0x140 [intel_vpu]
>>
>> If probe succeeds, the recovery runs concurrently with the rest of
>> probe, and its pm_runtime_disable() discards the initial autosuspend
>> request, leaving the NPU active until first use. Any recovery that
>> starts while the NPU is idle loses the pending autosuspend the same way.
>>
>> Keep recovery_work disabled from ivpu_pm_init() until ivpu_pm_enable().
>> A trigger during probe still runs diagnostics and masks interrupts, and
>> ivpu_pm_enable() runs the deferred recovery before the initial
>> autosuspend request. A trigger racing with ivpu_pm_enable() can cause
>> a second, harmless recovery. Request autosuspend at the end of
>> ivpu_pm_reset_complete() so the NPU suspends again after every reset.
>>
>> Fixes: 852be13f3bd3 ("accel/ivpu: Add PM support")
>> Cc:stable@vger.kernel.org
>> Reported-by: Daksh Sahni<swizzguy76@gmail.com>
>> Closes:https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
>> Signed-off-by: Karol Wachowski<karol.wachowski@linux.intel.com>
>> ---
>>
>> Notes:
>>      v3:
>>      - Request autosuspend at the end of ivpu_pm_reset_complete().
>>        pm_runtime_disable() in ivpu_pm_reset_begin() cancels a pending
>>        autosuspend request and pm_runtime_enable() does not re-issue it, so
>>        a recovery racing with the initial autosuspend request in
>>        ivpu_pm_enable() left the NPU active until first use (Sashiko).
>>        A trigger racing with ivpu_pm_enable() can still queue recovery
>>        twice. The second run is a full reset that leaves the device in a
>>        consistent state, and autosuspend is now re-requested after each
>>        reset.
>>      - Replace pm_runtime_mark_last_busy() with pm_request_autosuspend() in
>>        ivpu_pm_reset_complete(); it marks the device busy itself.
>>      - Shorten the commit message.
>>      - v2:https://lore.kernel.org/r/20261110085003.3527014-1- 
>> karol.wachowski@linux.intel.com
>>
>>      v2:
>>      - Rework of Daksh's "accel/ivpu: Disable recovery work on probe failure":
>>        https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
>>      - Keep recovery_work disabled from ivpu_pm_init() until ivpu_pm_enable()
>>        instead of disabling it on the probe error path, which is too late as
>>        recovery may already run concurrently with ivpu_boot() (Sashiko)
>>      - Run a recovery requested during probe from ivpu_pm_enable() and wait
>>        for it before the initial autosuspend request, so the NPU is reset
>>        before use and the autosuspend is not lost when probe succeeds
>>
>>      v1:
>>      - Link:https://lore.kernel.org/r/20261107174215.15818-1-swizzguy76@gmail.com
>>
>>   drivers/accel/ivpu/ivpu_pm.c | 9 ++++++++-
>>   1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/accel/ivpu/ivpu_pm.c b/drivers/accel/ivpu/ivpu_pm.c
>> index aa813262253e..e4b79c946ca0 100644
>> --- a/drivers/accel/ivpu/ivpu_pm.c
>> +++ b/drivers/accel/ivpu/ivpu_pm.c
>> @@ -150,8 +150,8 @@ static void ivpu_pm_reset_complete(struct ivpu_device *vdev)
>>          up_write(&vdev->pm->reset_lock);
>>          atomic_set(&vdev->pm->reset_pending, 0);
>>
>> -       pm_runtime_mark_last_busy(vdev->drm.dev);
>>          pm_runtime_enable(vdev->drm.dev);
>> +       pm_request_autosuspend(vdev->drm.dev);
>>   }
>>
>>   static void ivpu_pm_recovery_work(struct work_struct *work)
>> @@ -415,6 +415,7 @@ void ivpu_pm_init(struct ivpu_device *vdev)
>>          atomic_set(&pm->engine_reset_counter, 0);
>>
>>          INIT_WORK(&pm->recovery_work, ivpu_pm_recovery_work);
>> +       disable_work(&pm->recovery_work);
>>          INIT_DELAYED_WORK(&pm->job_timeout_work, ivpu_job_timeout_work);
>>
>>          if (ivpu_disable_recovery)
>> @@ -439,6 +440,12 @@ void ivpu_pm_enable(struct ivpu_device *vdev)
>>   {
>>          struct device *dev = vdev->drm.dev;
>>
>> +       enable_work(&vdev->pm->recovery_work);
>> +       if (atomic_read(&vdev->pm->reset_pending)) {
>> +               queue_work(system_dfl_wq, &vdev->pm->recovery_work);
>> +               flush_work(&vdev->pm->recovery_work);
>> +       }
>> +
>>          pm_runtime_allow(dev);
>>          pm_runtime_put_autosuspend(dev);
>>   }
>>
>> base-commit: 54d3ff1b5ac08d0e28029b93a1719ddd9ff670a4
>> --
>> 2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-16  7:10 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-10 12:15 [PATCH v3] accel/ivpu: Defer recovery until probe completes Karol Wachowski
2026-11-12 17:35 ` Jeff Hugo
2026-11-15 15:24 ` Toast
2026-11-16  7:09   ` Karol Wachowski

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox