* [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
@ 2026-11-16 20:40 Mario Limonciello
2026-11-16 21:13 ` sashiko-bot
2026-11-17 9:37 ` Christian König
0 siblings, 2 replies; 8+ messages in thread
From: Mario Limonciello @ 2026-11-16 20:40 UTC (permalink / raw)
To: Christian König
Cc: amd-gfx, Logatharshan Thothiralingam, Mario Limonciello
At suspend, amdgpu_fence_driver_hw_fini() only waits for the last fence
emitted to each ring. Jobs still queued in a scheduler entity have not
reached run_job() yet and so have no hardware fence. Userspace is frozen
at this point, but such jobs can become runnable later, for example once
a dependency signals, and the scheduler then pushes them into the ring
after the hardware wait has finished and before the IP blocks are torn
down. On resume the ring write pointer is reset, those jobs never
signal and the GFX ring times out and hangs.
This was observed as a rare (about 2 in 400 cycles) GFX hang on S3 resume
on Picasso.
Flush the context and VM scheduler entities of every open file
before evicting resources, mirroring what amdgpu_flush() does on file
close. Like for any other dma_fence there is no timeout, the wait is
not bounded.
Reported-by: Logatharshan Thothiralingam <Logatharshan.Thothiralingam@amd.com>
Closes: https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2@SA1SPRMB0073.namprd12.prod.outlook.com/
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 19a566187dfb5..d18348116d3f1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
}
}
+/**
+ * amdgpu_device_flush_file_entities - drain scheduler entities of open files
+ * @adev: amdgpu device object
+ *
+ * Wait for the scheduler entities owned by all open files (the context
+ * entities used for command submission and the VM entities used for page
+ * table updates) to run out of queued jobs, so that nothing can slip into
+ * the rings after the hardware fences were waited on at suspend time.
+ */
+static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev)
+{
+ struct drm_device *ddev = adev_to_drm(adev);
+ struct drm_file *file;
+
+ guard(mutex)(&ddev->filelist_mutex);
+ list_for_each_entry(file, &ddev->filelist, lhead) {
+ struct amdgpu_fpriv *fpriv = file->driver_priv;
+
+ amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
+ MAX_SCHEDULE_TIMEOUT);
+ amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
+ }
+}
+
/**
* amdgpu_device_suspend - initiate device suspend
*
@@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
if (r)
goto unwind_ip_phase1;
+ amdgpu_device_flush_file_entities(adev);
+
r = amdgpu_device_evict_resources(adev);
if (r)
goto unwind_userq;
---
base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
change-id: 20261116-superm1-drain-submissions-17d4414ab3b9
Best regards,
--
Mario Limonciello <mario.limonciello@amd.com>
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
2026-11-16 20:40 [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend Mario Limonciello
@ 2026-11-16 21:13 ` sashiko-bot
2026-11-16 21:21 ` Mario Limonciello
2026-11-17 9:37 ` Christian König
1 sibling, 1 reply; 8+ messages in thread
From: sashiko-bot @ 2026-11-16 21:13 UTC (permalink / raw)
To: Mario Limonciello; +Cc: amd-gfx
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] drm/amdgpu: deadlock in amdgpu_device_flush_file_entities() on
suspend
- [Medium] drm/amdgpu: race in amdgpu_device_flush_file_entities()
before run_job()
--
commit 2d3632cb05b8d04352d9590da3fabb5b096f1b19
Author: Mario Limonciello <mario.limonciello@amd.com>
drm/amdgpu: drain queued scheduler jobs before suspend
This commit adds amdgpu_device_flush_file_entities() to flush the context
and VM scheduler entities of all open DRM files during
amdgpu_device_suspend() before evicting resources.
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 19a566187dfb5..d18348116d3f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
> }
> }
>
> +/**
> + * amdgpu_device_flush_file_entities - drain scheduler entities of open files
> + * @adev: amdgpu device object
> + *
> + * Wait for the scheduler entities owned by all open files (the context
> + * entities used for command submission and the VM entities used for page
> + * table updates) to run out of queued jobs, so that nothing can slip into
> + * the rings after the hardware fences were waited on at suspend time.
> + */
> +static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev)
> +{
> + struct drm_device *ddev = adev_to_drm(adev);
> + struct drm_file *file;
> +
> + guard(mutex)(&ddev->filelist_mutex);
> + list_for_each_entry(file, &ddev->filelist, lhead) {
> + struct amdgpu_fpriv *fpriv = file->driver_priv;
> +
> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
> + MAX_SCHEDULE_TIMEOUT);
> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
> + }
> +}
[Severity: Medium]
Can drm_sched_entity_flush() return while the scheduler worker is still in
the middle of popping a job and before sched->ops->run_job() has emitted its
hardware fence?
CPU0 (submit_wq) CPU1 (suspend)
drm_sched_run_job_work()
drm_sched_entity_pop_job()
spsc_queue_pop(&entity->job_queue)
drm_sched_rq_pop_entity(entity)
amdgpu_device_suspend()
amdgpu_device_flush_file_entities()
...
drm_sched_entity_flush()
drm_sched_entity_is_idle() == true
amdgpu_device_evict_resources()
amdgpu_fence_driver_hw_fini()
amdgpu_fence_wait_empty()
sched->ops->run_job(sched_job)
amdgpu_job_run()
amdgpu_ib_schedule()
amdgpu_fence_emit()
amdgpu_ring_commit() <-- ring written after fence wait
complete_all(&entity->entity_idle)
drm_sched_entity_is_idle() reports the entity idle as soon as
spsc_queue_pop() empties entity->job_queue, before drm_sched_run_job_work()
calls sched->ops->run_job() and complete_all(&entity->entity_idle). On APUs
such as Picasso where amdgpu_device_evict_resources() returns immediately
during S3 suspend, amdgpu_fence_driver_hw_fini() can finish waiting on the
ring before the popped job emits its fence and commits the ring write
pointer, leaving the ring hang on resume window open.
> +
> /**
> * amdgpu_device_suspend - initiate device suspend
> *
> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
> if (r)
> goto unwind_ip_phase1;
>
> + amdgpu_device_flush_file_entities(adev);
[Severity: High]
Can calling amdgpu_device_flush_file_entities() here deadlock runtime or
system suspend when an entity has a job waiting on an unsignaled dependency?
drivers/gpu/drm/scheduler/sched_entity.c:drm_sched_entity_flush() {
...
if (current->flags & PF_EXITING) {
if (timeout)
ret = wait_event_timeout(
sched->job_scheduled,
drm_sched_entity_is_idle(entity),
timeout);
} else {
wait_event_killable(sched->job_scheduled,
drm_sched_entity_is_idle(entity));
}
...
}
Because the suspend task or runtime PM worker does not have PF_EXITING set,
drm_sched_entity_flush() ignores the timeout and sleeps indefinitely in
wait_event_killable() while holding ddev->filelist_mutex.
In amdgpu_cs_ioctl(), a runtime PM reference is only held during the ioctl
itself, and per-job runtime PM references are only taken in
amdgpu_fence_emit() when a job reaches run_job(). If a client submits a job
with an unsignaled syncobj dependency, runtime autosuspend can enter
amdgpu_pmops_runtime_suspend() -> amdgpu_device_suspend() and block in
amdgpu_device_flush_file_entities() while the device status is
RPM_SUSPENDING. When another thread or process then calls amdgpu_cs_ioctl()
to submit the work that would signal that syncobj, its runtime PM resume
waits for RPM_SUSPENDING to complete, deadlocking both tasks.
Killing the process owning the blocked entity does not break the wait
either: amdgpu_flush() times out after 2 seconds and calls
drm_sched_entity_kill(), which drains the entity without waking
sched->job_scheduled, and the exiting task then blocks on
ddev->filelist_mutex in drm_close_helper(). During system suspend,
amdgpu_userq_suspend() also runs right before
amdgpu_device_flush_file_entities() and unmaps usermode queues while
userspace is already frozen, so any entity job waiting on a usermode queue
or userspace fence can never make progress.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-superm1-drain-submissions-v1-1-e3ffee977a28@amd.com?part=1
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
2026-11-16 21:13 ` sashiko-bot
@ 2026-11-16 21:21 ` Mario Limonciello
2026-11-17 9:08 ` Christian König
0 siblings, 1 reply; 8+ messages in thread
From: Mario Limonciello @ 2026-11-16 21:21 UTC (permalink / raw)
To: Christian König; +Cc: amd-gfx
On 11/16/26 15:13, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential
> issues to consider:
> - [High] drm/amdgpu: deadlock in amdgpu_device_flush_file_entities() on
> suspend
> - [Medium] drm/amdgpu: race in amdgpu_device_flush_file_entities()
> before run_job()
> --
>
> commit 2d3632cb05b8d04352d9590da3fabb5b096f1b19
> Author: Mario Limonciello <mario.limonciello@amd.com>
>
> drm/amdgpu: drain queued scheduler jobs before suspend
>
> This commit adds amdgpu_device_flush_file_entities() to flush the context
> and VM scheduler entities of all open DRM files during
> amdgpu_device_suspend() before evicting resources.
>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> index 19a566187dfb5..d18348116d3f1 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
>> }
>> }
>>
>> +/**
>> + * amdgpu_device_flush_file_entities - drain scheduler entities of open files
>> + * @adev: amdgpu device object
>> + *
>> + * Wait for the scheduler entities owned by all open files (the context
>> + * entities used for command submission and the VM entities used for page
>> + * table updates) to run out of queued jobs, so that nothing can slip into
>> + * the rings after the hardware fences were waited on at suspend time.
>> + */
>> +static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev)
>> +{
>> + struct drm_device *ddev = adev_to_drm(adev);
>> + struct drm_file *file;
>> +
>> + guard(mutex)(&ddev->filelist_mutex);
>> + list_for_each_entry(file, &ddev->filelist, lhead) {
>> + struct amdgpu_fpriv *fpriv = file->driver_priv;
>> +
>> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
>> + MAX_SCHEDULE_TIMEOUT);
>> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
>> + }
>> +}
>
> [Severity: Medium]
> Can drm_sched_entity_flush() return while the scheduler worker is still in
> the middle of popping a job and before sched->ops->run_job() has emitted its
> hardware fence?
>
> CPU0 (submit_wq) CPU1 (suspend)
> drm_sched_run_job_work()
> drm_sched_entity_pop_job()
> spsc_queue_pop(&entity->job_queue)
> drm_sched_rq_pop_entity(entity)
> amdgpu_device_suspend()
> amdgpu_device_flush_file_entities()
> ...
> drm_sched_entity_flush()
> drm_sched_entity_is_idle() == true
> amdgpu_device_evict_resources()
> amdgpu_fence_driver_hw_fini()
> amdgpu_fence_wait_empty()
> sched->ops->run_job(sched_job)
> amdgpu_job_run()
> amdgpu_ib_schedule()
> amdgpu_fence_emit()
> amdgpu_ring_commit() <-- ring written after fence wait
> complete_all(&entity->entity_idle)
>
> drm_sched_entity_is_idle() reports the entity idle as soon as
> spsc_queue_pop() empties entity->job_queue, before drm_sched_run_job_work()
> calls sched->ops->run_job() and complete_all(&entity->entity_idle). On APUs
> such as Picasso where amdgpu_device_evict_resources() returns immediately
> during S3 suspend, amdgpu_fence_driver_hw_fini() can finish waiting on the
> ring before the popped job emits its fence and commits the ring write
> pointer, leaving the ring hang on resume window open.
>
>> +
>> /**
>> * amdgpu_device_suspend - initiate device suspend
>> *
>> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
>> if (r)
>> goto unwind_ip_phase1;
>>
>> + amdgpu_device_flush_file_entities(adev);
>
> [Severity: High]
> Can calling amdgpu_device_flush_file_entities() here deadlock runtime or
> system suspend when an entity has a job waiting on an unsignaled dependency?
>
> drivers/gpu/drm/scheduler/sched_entity.c:drm_sched_entity_flush() {
> ...
> if (current->flags & PF_EXITING) {
> if (timeout)
> ret = wait_event_timeout(
> sched->job_scheduled,
> drm_sched_entity_is_idle(entity),
> timeout);
> } else {
> wait_event_killable(sched->job_scheduled,
> drm_sched_entity_is_idle(entity));
> }
> ...
> }
>
> Because the suspend task or runtime PM worker does not have PF_EXITING set,
> drm_sched_entity_flush() ignores the timeout and sleeps indefinitely in
> wait_event_killable() while holding ddev->filelist_mutex.
>
> In amdgpu_cs_ioctl(), a runtime PM reference is only held during the ioctl
> itself, and per-job runtime PM references are only taken in
> amdgpu_fence_emit() when a job reaches run_job(). If a client submits a job
> with an unsignaled syncobj dependency, runtime autosuspend can enter
> amdgpu_pmops_runtime_suspend() -> amdgpu_device_suspend() and block in
> amdgpu_device_flush_file_entities() while the device status is
> RPM_SUSPENDING. When another thread or process then calls amdgpu_cs_ioctl()
> to submit the work that would signal that syncobj, its runtime PM resume
> waits for RPM_SUSPENDING to complete, deadlocking both tasks.
>
> Killing the process owning the blocked entity does not break the wait
> either: amdgpu_flush() times out after 2 seconds and calls
> drm_sched_entity_kill(), which drains the entity without waking
> sched->job_scheduled, and the exiting task then blocks on
> ddev->filelist_mutex in drm_close_helper(). During system suspend,
> amdgpu_userq_suspend() also runs right before
> amdgpu_device_flush_file_entities() and unmaps usermode queues while
> userspace is already frozen, so any entity job waiting on a usermode queue
> or userspace fence can never make progress.
>
I'd say that these findings both look real and this being able to
potentially deadlock the system means it's not the right solution.
How about instead we stop schedulers with drm_sched_wqueue_stop() on
each of the rings? Restart then in any unwind and resume paths. That's
kinda like how SDMA already works from what I can tell.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
2026-11-16 21:21 ` Mario Limonciello
@ 2026-11-17 9:08 ` Christian König
0 siblings, 0 replies; 8+ messages in thread
From: Christian König @ 2026-11-17 9:08 UTC (permalink / raw)
To: Mario Limonciello; +Cc: amd-gfx
On 11/16/26 22:21, Mario Limonciello wrote:
> On 11/16/26 15:13, sashiko-bot@kernel.org wrote:
>> Thank you for your contribution! Sashiko AI review found 2 potential
>> issues to consider:
>> - [High] drm/amdgpu: deadlock in amdgpu_device_flush_file_entities() on
>> suspend
>> - [Medium] drm/amdgpu: race in amdgpu_device_flush_file_entities()
>> before run_job()
>> --
>>
>> commit 2d3632cb05b8d04352d9590da3fabb5b096f1b19
>> Author: Mario Limonciello <mario.limonciello@amd.com>
>>
>> drm/amdgpu: drain queued scheduler jobs before suspend
>>
>> This commit adds amdgpu_device_flush_file_entities() to flush the context
>> and VM scheduler entities of all open DRM files during
>> amdgpu_device_suspend() before evicting resources.
>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> index 19a566187dfb5..d18348116d3f1 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
>>> }
>>> }
>>> +/**
>>> + * amdgpu_device_flush_file_entities - drain scheduler entities of open files
>>> + * @adev: amdgpu device object
>>> + *
>>> + * Wait for the scheduler entities owned by all open files (the context
>>> + * entities used for command submission and the VM entities used for page
>>> + * table updates) to run out of queued jobs, so that nothing can slip into
>>> + * the rings after the hardware fences were waited on at suspend time.
>>> + */
>>> +static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev)
>>> +{
>>> + struct drm_device *ddev = adev_to_drm(adev);
>>> + struct drm_file *file;
>>> +
>>> + guard(mutex)(&ddev->filelist_mutex);
>>> + list_for_each_entry(file, &ddev->filelist, lhead) {
>>> + struct amdgpu_fpriv *fpriv = file->driver_priv;
>>> +
>>> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
>>> + MAX_SCHEDULE_TIMEOUT);
>>> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
>>> + }
>>> +}
>>
>> [Severity: Medium]
>> Can drm_sched_entity_flush() return while the scheduler worker is still in
>> the middle of popping a job and before sched->ops->run_job() has emitted its
>> hardware fence?
>>
>> CPU0 (submit_wq) CPU1 (suspend)
>> drm_sched_run_job_work()
>> drm_sched_entity_pop_job()
>> spsc_queue_pop(&entity->job_queue)
>> drm_sched_rq_pop_entity(entity)
>> amdgpu_device_suspend()
>> amdgpu_device_flush_file_entities()
>> ...
>> drm_sched_entity_flush()
>> drm_sched_entity_is_idle() == true
>> amdgpu_device_evict_resources()
>> amdgpu_fence_driver_hw_fini()
>> amdgpu_fence_wait_empty()
>> sched->ops->run_job(sched_job)
>> amdgpu_job_run()
>> amdgpu_ib_schedule()
>> amdgpu_fence_emit()
>> amdgpu_ring_commit() <-- ring written after fence wait
>> complete_all(&entity->entity_idle)
>>
>> drm_sched_entity_is_idle() reports the entity idle as soon as
>> spsc_queue_pop() empties entity->job_queue, before drm_sched_run_job_work()
>> calls sched->ops->run_job() and complete_all(&entity->entity_idle). On APUs
>> such as Picasso where amdgpu_device_evict_resources() returns immediately
>> during S3 suspend, amdgpu_fence_driver_hw_fini() can finish waiting on the
>> ring before the popped job emits its fence and commits the ring write
>> pointer, leaving the ring hang on resume window open.
>>
>>> +
>>> /**
>>> * amdgpu_device_suspend - initiate device suspend
>>> *
>>> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
>>> if (r)
>>> goto unwind_ip_phase1;
>>> + amdgpu_device_flush_file_entities(adev);
>>
>> [Severity: High]
>> Can calling amdgpu_device_flush_file_entities() here deadlock runtime or
>> system suspend when an entity has a job waiting on an unsignaled dependency?
>>
>> drivers/gpu/drm/scheduler/sched_entity.c:drm_sched_entity_flush() {
>> ...
>> if (current->flags & PF_EXITING) {
>> if (timeout)
>> ret = wait_event_timeout(
>> sched->job_scheduled,
>> drm_sched_entity_is_idle(entity),
>> timeout);
>> } else {
>> wait_event_killable(sched->job_scheduled,
>> drm_sched_entity_is_idle(entity));
>> }
>> ...
>> }
>>
>> Because the suspend task or runtime PM worker does not have PF_EXITING set,
>> drm_sched_entity_flush() ignores the timeout and sleeps indefinitely in
>> wait_event_killable() while holding ddev->filelist_mutex.
>>
>> In amdgpu_cs_ioctl(), a runtime PM reference is only held during the ioctl
>> itself, and per-job runtime PM references are only taken in
>> amdgpu_fence_emit() when a job reaches run_job(). If a client submits a job
>> with an unsignaled syncobj dependency, runtime autosuspend can enter
>> amdgpu_pmops_runtime_suspend() -> amdgpu_device_suspend() and block in
>> amdgpu_device_flush_file_entities() while the device status is
>> RPM_SUSPENDING. When another thread or process then calls amdgpu_cs_ioctl()
>> to submit the work that would signal that syncobj, its runtime PM resume
>> waits for RPM_SUSPENDING to complete, deadlocking both tasks.
>>
>> Killing the process owning the blocked entity does not break the wait
>> either: amdgpu_flush() times out after 2 seconds and calls
>> drm_sched_entity_kill(), which drains the entity without waking
>> sched->job_scheduled, and the exiting task then blocks on
>> ddev->filelist_mutex in drm_close_helper(). During system suspend,
>> amdgpu_userq_suspend() also runs right before
>> amdgpu_device_flush_file_entities() and unmaps usermode queues while
>> userspace is already frozen, so any entity job waiting on a usermode queue
>> or userspace fence can never make progress.
>>
>
> I'd say that these findings both look real and this being able to potentially deadlock the system means it's not the right solution.
That's just AI nonsense.
All dma_fences *MUST* be signaled before suspending and runtime PM shouldn't be able to suspend the HW while there is still work queued up.
What could be is that runtime PM is not correctly implemented here, but that is an independent problem to what we try to fix here.
> How about instead we stop schedulers with drm_sched_wqueue_stop() on each of the rings? Restart then in any unwind and resume paths. That's kinda like how SDMA already works from what I can tell.
Suspending with pending work is completely broken and can trivially deadlock. So we should never call drm_sched_wqueue_stop() from the suspend path.
What we could do is to issue a warning from the run_job callback when we are already suspended.
Regards,
Christian.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
2026-11-16 20:40 [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend Mario Limonciello
2026-11-16 21:13 ` sashiko-bot
@ 2026-11-17 9:37 ` Christian König
2026-11-17 13:39 ` Thothiralingam, Logatharshan
1 sibling, 1 reply; 8+ messages in thread
From: Christian König @ 2026-11-17 9:37 UTC (permalink / raw)
To: Mario Limonciello; +Cc: amd-gfx, Logatharshan Thothiralingam
On 11/16/26 21:40, Mario Limonciello wrote:
> At suspend, amdgpu_fence_driver_hw_fini() only waits for the last fence
> emitted to each ring. Jobs still queued in a scheduler entity have not
> reached run_job() yet and so have no hardware fence. Userspace is frozen
> at this point, but such jobs can become runnable later, for example once
> a dependency signals, and the scheduler then pushes them into the ring
> after the hardware wait has finished and before the IP blocks are torn
> down. On resume the ring write pointer is reset, those jobs never
> signal and the GFX ring times out and hangs.
>
> This was observed as a rare (about 2 in 400 cycles) GFX hang on S3 resume
> on Picasso.
>
> Flush the context and VM scheduler entities of every open file
> before evicting resources, mirroring what amdgpu_flush() does on file
> close. Like for any other dma_fence there is no timeout, the wait is
> not bounded.
>
> Reported-by: Logatharshan Thothiralingam <Logatharshan.Thothiralingam@amd.com>
> Closes: https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2@SA1SPRMB0073.namprd12.prod.outlook.com/
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26 ++++++++++++++++++++++++++
> 1 file changed, 26 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 19a566187dfb5..d18348116d3f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
> }
> }
>
> +/**
> + * amdgpu_device_flush_file_entities - drain scheduler entities of open files
> + * @adev: amdgpu device object
> + *
> + * Wait for the scheduler entities owned by all open files (the context
> + * entities used for command submission and the VM entities used for page
> + * table updates) to run out of queued jobs, so that nothing can slip into
> + * the rings after the hardware fences were waited on at suspend time.
> + */
> +static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev)
> +{
> + struct drm_device *ddev = adev_to_drm(adev);
> + struct drm_file *file;
> +
> + guard(mutex)(&ddev->filelist_mutex);
> + list_for_each_entry(file, &ddev->filelist, lhead) {
> + struct amdgpu_fpriv *fpriv = file->driver_priv;
> +
> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
> + MAX_SCHEDULE_TIMEOUT);
That is actually superfluous and even a little bit dangerous.
amdgpu_ctx_mgr_entity_flush() is only supposed to be called from the process which has the file descriptor open and is closing it.
We use that to detect if that process using the file descriptor was killed and we should discard queued up work instead of waiting for it to finish executing.
> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
That here should be enough to wait for all work to complete.
Regards,
Christian.
> + }
> +}
> +
> /**
> * amdgpu_device_suspend - initiate device suspend
> *
> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
> if (r)
> goto unwind_ip_phase1;
>
> + amdgpu_device_flush_file_entities(adev);
> +
> r = amdgpu_device_evict_resources(adev);
> if (r)
> goto unwind_userq;
>
> ---
> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
> change-id: 20261116-superm1-drain-submissions-17d4414ab3b9
>
> Best regards,
> --
> Mario Limonciello <mario.limonciello@amd.com>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
2026-11-17 9:37 ` Christian König
@ 2026-11-17 13:39 ` Thothiralingam, Logatharshan
2026-11-17 17:52 ` Thothiralingam, Logatharshan
0 siblings, 1 reply; 8+ messages in thread
From: Thothiralingam, Logatharshan @ 2026-11-17 13:39 UTC (permalink / raw)
To: Koenig, Christian, Limonciello, Mario; +Cc: amd-gfx@lists.freedesktop.org
AMD General
Hi Chistian,
I looked a the amdgpu_vm_wait_idle() function and noticed that starting commit 1f02f20 (https://github.com/eseiker/kernel-asahi/commit/1f02f2044bda1db1fd995bc35961ab075fa7b5a2) the wait on the VM's root page table fence list is removed. So correct me if I am wrong, this alone won't wait on the gfx jobs right?
So should I add this before the idle call?
dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT);
Thanks,
Tharshan
-----Original Message-----
From: Koenig, Christian <Christian.Koenig@amd.com>
Sent: Tuesday, November 17, 2026 4:37 AM
To: Limonciello, Mario <Mario.Limonciello@amd.com>
Cc: amd-gfx@lists.freedesktop.org; Thothiralingam, Logatharshan <Logatharshan.Thothiralingam@amd.com>
Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
On 11/16/26 21:40, Mario Limonciello wrote:
> At suspend, amdgpu_fence_driver_hw_fini() only waits for the last
> fence emitted to each ring. Jobs still queued in a scheduler entity
> have not reached run_job() yet and so have no hardware fence.
> Userspace is frozen at this point, but such jobs can become runnable
> later, for example once a dependency signals, and the scheduler then
> pushes them into the ring after the hardware wait has finished and
> before the IP blocks are torn down. On resume the ring write pointer
> is reset, those jobs never signal and the GFX ring times out and hangs.
>
> This was observed as a rare (about 2 in 400 cycles) GFX hang on S3
> resume on Picasso.
>
> Flush the context and VM scheduler entities of every open file before
> evicting resources, mirroring what amdgpu_flush() does on file close.
> Like for any other dma_fence there is no timeout, the wait is not
> bounded.
>
> Reported-by: Logatharshan Thothiralingam
> <Logatharshan.Thothiralingam@amd.com>
> Closes:
> https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2@
> SA1SPRMB0073.namprd12.prod.outlook.com/
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26
> ++++++++++++++++++++++++++
> 1 file changed, 26 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 19a566187dfb5..d18348116d3f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
> }
> }
>
> +/**
> + * amdgpu_device_flush_file_entities - drain scheduler entities of
> +open files
> + * @adev: amdgpu device object
> + *
> + * Wait for the scheduler entities owned by all open files (the
> +context
> + * entities used for command submission and the VM entities used for
> +page
> + * table updates) to run out of queued jobs, so that nothing can slip
> +into
> + * the rings after the hardware fences were waited on at suspend time.
> + */
> +static void amdgpu_device_flush_file_entities(struct amdgpu_device
> +*adev) {
> + struct drm_device *ddev = adev_to_drm(adev);
> + struct drm_file *file;
> +
> + guard(mutex)(&ddev->filelist_mutex);
> + list_for_each_entry(file, &ddev->filelist, lhead) {
> + struct amdgpu_fpriv *fpriv = file->driver_priv;
> +
> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
> + MAX_SCHEDULE_TIMEOUT);
That is actually superfluous and even a little bit dangerous.
amdgpu_ctx_mgr_entity_flush() is only supposed to be called from the process which has the file descriptor open and is closing it.
We use that to detect if that process using the file descriptor was killed and we should discard queued up work instead of waiting for it to finish executing.
> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
That here should be enough to wait for all work to complete.
Regards,
Christian.
> + }
> +}
> +
> /**
> * amdgpu_device_suspend - initiate device suspend
> *
> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
> if (r)
> goto unwind_ip_phase1;
>
> + amdgpu_device_flush_file_entities(adev);
> +
> r = amdgpu_device_evict_resources(adev);
> if (r)
> goto unwind_userq;
>
> ---
> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
> change-id: 20261116-superm1-drain-submissions-17d4414ab3b9
>
> Best regards,
> --
> Mario Limonciello <mario.limonciello@amd.com>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
2026-11-17 13:39 ` Thothiralingam, Logatharshan
@ 2026-11-17 17:52 ` Thothiralingam, Logatharshan
2026-11-17 20:08 ` Thothiralingam, Logatharshan
0 siblings, 1 reply; 8+ messages in thread
From: Thothiralingam, Logatharshan @ 2026-11-17 17:52 UTC (permalink / raw)
To: Koenig, Christian, Limonciello, Mario; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1: Type: text/plain, Size: 5311 bytes --]
AMD General
Hi Chistian,
Just having the amdgpu_vm_wait_idle() doesn't solve the issue. I tested it and I do see the reset. I have attached a patch with the dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT); I am building it now to test. Please let me know your thoughts
Thanks,
Tharshan
-----Original Message-----
From: Thothiralingam, Logatharshan
Sent: Tuesday, November 17, 2026 8:39 AM
To: Koenig, Christian <Christian.Koenig@amd.com>; Limonciello, Mario <Mario.Limonciello@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
Hi Chistian,
I looked a the amdgpu_vm_wait_idle() function and noticed that starting commit 1f02f20 (https://github.com/eseiker/kernel-asahi/commit/1f02f2044bda1db1fd995bc35961ab075fa7b5a2) the wait on the VM's root page table fence list is removed. So correct me if I am wrong, this alone won't wait on the gfx jobs right?
So should I add this before the idle call?
dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT);
Thanks,
Tharshan
-----Original Message-----
From: Koenig, Christian <Christian.Koenig@amd.com>
Sent: Tuesday, November 17, 2026 4:37 AM
To: Limonciello, Mario <Mario.Limonciello@amd.com>
Cc: amd-gfx@lists.freedesktop.org; Thothiralingam, Logatharshan <Logatharshan.Thothiralingam@amd.com>
Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
On 11/16/26 21:40, Mario Limonciello wrote:
> At suspend, amdgpu_fence_driver_hw_fini() only waits for the last
> fence emitted to each ring. Jobs still queued in a scheduler entity
> have not reached run_job() yet and so have no hardware fence.
> Userspace is frozen at this point, but such jobs can become runnable
> later, for example once a dependency signals, and the scheduler then
> pushes them into the ring after the hardware wait has finished and
> before the IP blocks are torn down. On resume the ring write pointer
> is reset, those jobs never signal and the GFX ring times out and hangs.
>
> This was observed as a rare (about 2 in 400 cycles) GFX hang on S3
> resume on Picasso.
>
> Flush the context and VM scheduler entities of every open file before
> evicting resources, mirroring what amdgpu_flush() does on file close.
> Like for any other dma_fence there is no timeout, the wait is not
> bounded.
>
> Reported-by: Logatharshan Thothiralingam
> <Logatharshan.Thothiralingam@amd.com>
> Closes:
> https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2@
> SA1SPRMB0073.namprd12.prod.outlook.com/
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26
> ++++++++++++++++++++++++++
> 1 file changed, 26 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 19a566187dfb5..d18348116d3f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
> }
> }
>
> +/**
> + * amdgpu_device_flush_file_entities - drain scheduler entities of
> +open files
> + * @adev: amdgpu device object
> + *
> + * Wait for the scheduler entities owned by all open files (the
> +context
> + * entities used for command submission and the VM entities used for
> +page
> + * table updates) to run out of queued jobs, so that nothing can slip
> +into
> + * the rings after the hardware fences were waited on at suspend time.
> + */
> +static void amdgpu_device_flush_file_entities(struct amdgpu_device
> +*adev) {
> + struct drm_device *ddev = adev_to_drm(adev);
> + struct drm_file *file;
> +
> + guard(mutex)(&ddev->filelist_mutex);
> + list_for_each_entry(file, &ddev->filelist, lhead) {
> + struct amdgpu_fpriv *fpriv = file->driver_priv;
> +
> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
> + MAX_SCHEDULE_TIMEOUT);
That is actually superfluous and even a little bit dangerous.
amdgpu_ctx_mgr_entity_flush() is only supposed to be called from the process which has the file descriptor open and is closing it.
We use that to detect if that process using the file descriptor was killed and we should discard queued up work instead of waiting for it to finish executing.
> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
That here should be enough to wait for all work to complete.
Regards,
Christian.
> + }
> +}
> +
> /**
> * amdgpu_device_suspend - initiate device suspend
> *
> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
> if (r)
> goto unwind_ip_phase1;
>
> + amdgpu_device_flush_file_entities(adev);
> +
> r = amdgpu_device_evict_resources(adev);
> if (r)
> goto unwind_userq;
>
> ---
> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
> change-id: 20261116-superm1-drain-submissions-17d4414ab3b9
>
> Best regards,
> --
> Mario Limonciello <mario.limonciello@amd.com>
>
[-- Attachment #2: 0001-drm-amdgpu-wait-for-queued-user-submissions-at-suspe.patch --]
[-- Type: application/octet-stream, Size: 2994 bytes --]
From 91efd69977a918e28bc40ffb0e19a2cbe7cd9769 Mon Sep 17 00:00:00 2001
From: Logatharshan Thothiralingam <Logatharshan.Thothiralingam@amd.com>
Date: Thu, 12 Nov 2026 14:31:45 -0500
Subject: [PATCH] drm/amdgpu: wait for queued user submissions at suspend
amdgpu_device_suspend() doesn't wait on user jobs that are still queued in
a scheduler entity. Those jobs can reach the rings after the fence wait,
while the device is already being put to sleep. This can lead to a GPU hang
or panic on S3 suspend.
Add amdgpu_device_wait_for_user_submissions(), which walks all open DRM
files and waits for each VM to become idle before suspend continues.
Signed-off-by: Logatharshan Thothiralingam <Logatharshan.Thothiralingam@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 ++++++++++++++++++++++
2 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
index afedea02188d..98c2613f9203 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
@@ -989,4 +989,4 @@ void amdgpu_ctx_mgr_usage(struct amdgpu_ctx_mgr *mgr,
}
}
mutex_unlock(&mgr->lock);
-}
+}
\ No newline at end of file
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 49107475af61..084f6cd26c3e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -5196,6 +5196,35 @@ void amdgpu_device_complete(struct drm_device *dev)
}
}
+/**
+ * amdgpu_device_wait_for_vms_idle - wait for queued user submissions
+ * @adev: amdgpu device object
+ *
+ * Wait for the VMs of all open files to become idle, so that nothing can slip
+ * into the rings after the hardware fences were waited on at suspend time.
+ *
+ * Every user submission adds its scheduler fence to the root PD reservation
+ * object when it is submitted, so waiting for it covers jobs which are still
+ * queued in an entity as well as jobs on the hardware.
+ * Page table updates are handled by amdgpu_vm_wait_idle().
+ */
+static void amdgpu_device_wait_for_vms_idle(struct amdgpu_device *adev)
+{
+ struct drm_device *ddev = adev_to_drm(adev);
+ struct drm_file *file;
+
+ guard(mutex)(&ddev->filelist_mutex);
+ list_for_each_entry(file, &ddev->filelist, lhead) {
+ struct amdgpu_fpriv *fpriv = file->driver_priv;
+ struct amdgpu_vm *vm = &fpriv->vm;
+
+ dma_resv_wait_timeout(vm->root.bo->tbo.base.resv,
+ DMA_RESV_USAGE_READ, false,
+ MAX_SCHEDULE_TIMEOUT);
+ amdgpu_vm_wait_idle(vm, MAX_SCHEDULE_TIMEOUT);
+ }
+}
+
/**
* amdgpu_device_suspend - initiate device suspend
*
@@ -5213,6 +5242,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)
return 0;
+
+ amdgpu_device_wait_for_vms_idle(adev);
adev->in_suspend = true;
--
2.55.0.windows.5
^ permalink raw reply related [flat|nested] 8+ messages in thread
* RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
2026-11-17 17:52 ` Thothiralingam, Logatharshan
@ 2026-11-17 20:08 ` Thothiralingam, Logatharshan
0 siblings, 0 replies; 8+ messages in thread
From: Thothiralingam, Logatharshan @ 2026-11-17 20:08 UTC (permalink / raw)
To: Koenig, Christian, Limonciello, Mario; +Cc: amd-gfx@lists.freedesktop.org
[-- Attachment #1: Type: text/plain, Size: 5958 bytes --]
AMD General
Hi Christian,
So adding dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT); before the amdgpu_vm_wait_idle() fixes that gfx reset. So it seems like amdgpu_vm_wait_idle() alone is not enough.
Attached the patch again. Please let me know if it's good.
Thanks,
Tharshan
-----Original Message-----
From: Thothiralingam, Logatharshan
Sent: Tuesday, November 17, 2026 12:52 PM
To: Koenig, Christian <Christian.Koenig@amd.com>; Limonciello, Mario <Mario.Limonciello@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
Hi Chistian,
Just having the amdgpu_vm_wait_idle() doesn't solve the issue. I tested it and I do see the reset. I have attached a patch with the dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT); I am building it now to test. Please let me know your thoughts
Thanks,
Tharshan
-----Original Message-----
From: Thothiralingam, Logatharshan
Sent: Tuesday, November 17, 2026 8:39 AM
To: Koenig, Christian <Christian.Koenig@amd.com>; Limonciello, Mario <Mario.Limonciello@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: RE: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
Hi Chistian,
I looked a the amdgpu_vm_wait_idle() function and noticed that starting commit 1f02f20 (https://github.com/eseiker/kernel-asahi/commit/1f02f2044bda1db1fd995bc35961ab075fa7b5a2) the wait on the VM's root page table fence list is removed. So correct me if I am wrong, this alone won't wait on the gfx jobs right?
So should I add this before the idle call?
dma_resv_wait_timeout(vm->root.bo->tbo.base.resv, DMA_RESV_USAGE_READ, false,MAX_SCHEDULE_TIMEOUT);
Thanks,
Tharshan
-----Original Message-----
From: Koenig, Christian <Christian.Koenig@amd.com>
Sent: Tuesday, November 17, 2026 4:37 AM
To: Limonciello, Mario <Mario.Limonciello@amd.com>
Cc: amd-gfx@lists.freedesktop.org; Thothiralingam, Logatharshan <Logatharshan.Thothiralingam@amd.com>
Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
On 11/16/26 21:40, Mario Limonciello wrote:
> At suspend, amdgpu_fence_driver_hw_fini() only waits for the last
> fence emitted to each ring. Jobs still queued in a scheduler entity
> have not reached run_job() yet and so have no hardware fence.
> Userspace is frozen at this point, but such jobs can become runnable
> later, for example once a dependency signals, and the scheduler then
> pushes them into the ring after the hardware wait has finished and
> before the IP blocks are torn down. On resume the ring write pointer
> is reset, those jobs never signal and the GFX ring times out and hangs.
>
> This was observed as a rare (about 2 in 400 cycles) GFX hang on S3
> resume on Picasso.
>
> Flush the context and VM scheduler entities of every open file before
> evicting resources, mirroring what amdgpu_flush() does on file close.
> Like for any other dma_fence there is no timeout, the wait is not
> bounded.
>
> Reported-by: Logatharshan Thothiralingam
> <Logatharshan.Thothiralingam@amd.com>
> Closes:
> https://lore.kernel.org/amd-gfx/SA1SPRMB0073292C113C6B98D9282092F87F2@
> SA1SPRMB0073.namprd12.prod.outlook.com/
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 26
> ++++++++++++++++++++++++++
> 1 file changed, 26 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 19a566187dfb5..d18348116d3f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
> }
> }
>
> +/**
> + * amdgpu_device_flush_file_entities - drain scheduler entities of
> +open files
> + * @adev: amdgpu device object
> + *
> + * Wait for the scheduler entities owned by all open files (the
> +context
> + * entities used for command submission and the VM entities used for
> +page
> + * table updates) to run out of queued jobs, so that nothing can slip
> +into
> + * the rings after the hardware fences were waited on at suspend time.
> + */
> +static void amdgpu_device_flush_file_entities(struct amdgpu_device
> +*adev) {
> + struct drm_device *ddev = adev_to_drm(adev);
> + struct drm_file *file;
> +
> + guard(mutex)(&ddev->filelist_mutex);
> + list_for_each_entry(file, &ddev->filelist, lhead) {
> + struct amdgpu_fpriv *fpriv = file->driver_priv;
> +
> + amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
> + MAX_SCHEDULE_TIMEOUT);
That is actually superfluous and even a little bit dangerous.
amdgpu_ctx_mgr_entity_flush() is only supposed to be called from the process which has the file descriptor open and is closing it.
We use that to detect if that process using the file descriptor was killed and we should discard queued up work instead of waiting for it to finish executing.
> + amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
That here should be enough to wait for all work to complete.
Regards,
Christian.
> + }
> +}
> +
> /**
> * amdgpu_device_suspend - initiate device suspend
> *
> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
> if (r)
> goto unwind_ip_phase1;
>
> + amdgpu_device_flush_file_entities(adev);
> +
> r = amdgpu_device_evict_resources(adev);
> if (r)
> goto unwind_userq;
>
> ---
> base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
> change-id: 20261116-superm1-drain-submissions-17d4414ab3b9
>
> Best regards,
> --
> Mario Limonciello <mario.limonciello@amd.com>
>
[-- Attachment #2: 0001-drm-amdgpu-wait-for-queued-user-submissions-at-suspe.patch --]
[-- Type: application/octet-stream, Size: 2994 bytes --]
From 91efd69977a918e28bc40ffb0e19a2cbe7cd9769 Mon Sep 17 00:00:00 2001
From: Logatharshan Thothiralingam <Logatharshan.Thothiralingam@amd.com>
Date: Thu, 12 Nov 2026 14:31:45 -0500
Subject: [PATCH] drm/amdgpu: wait for queued user submissions at suspend
amdgpu_device_suspend() doesn't wait on user jobs that are still queued in
a scheduler entity. Those jobs can reach the rings after the fence wait,
while the device is already being put to sleep. This can lead to a GPU hang
or panic on S3 suspend.
Add amdgpu_device_wait_for_user_submissions(), which walks all open DRM
files and waits for each VM to become idle before suspend continues.
Signed-off-by: Logatharshan Thothiralingam <Logatharshan.Thothiralingam@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 ++++++++++++++++++++++
2 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
index afedea02188d..98c2613f9203 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
@@ -989,4 +989,4 @@ void amdgpu_ctx_mgr_usage(struct amdgpu_ctx_mgr *mgr,
}
}
mutex_unlock(&mgr->lock);
-}
+}
\ No newline at end of file
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 49107475af61..084f6cd26c3e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -5196,6 +5196,35 @@ void amdgpu_device_complete(struct drm_device *dev)
}
}
+/**
+ * amdgpu_device_wait_for_vms_idle - wait for queued user submissions
+ * @adev: amdgpu device object
+ *
+ * Wait for the VMs of all open files to become idle, so that nothing can slip
+ * into the rings after the hardware fences were waited on at suspend time.
+ *
+ * Every user submission adds its scheduler fence to the root PD reservation
+ * object when it is submitted, so waiting for it covers jobs which are still
+ * queued in an entity as well as jobs on the hardware.
+ * Page table updates are handled by amdgpu_vm_wait_idle().
+ */
+static void amdgpu_device_wait_for_vms_idle(struct amdgpu_device *adev)
+{
+ struct drm_device *ddev = adev_to_drm(adev);
+ struct drm_file *file;
+
+ guard(mutex)(&ddev->filelist_mutex);
+ list_for_each_entry(file, &ddev->filelist, lhead) {
+ struct amdgpu_fpriv *fpriv = file->driver_priv;
+ struct amdgpu_vm *vm = &fpriv->vm;
+
+ dma_resv_wait_timeout(vm->root.bo->tbo.base.resv,
+ DMA_RESV_USAGE_READ, false,
+ MAX_SCHEDULE_TIMEOUT);
+ amdgpu_vm_wait_idle(vm, MAX_SCHEDULE_TIMEOUT);
+ }
+}
+
/**
* amdgpu_device_suspend - initiate device suspend
*
@@ -5213,6 +5242,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)
return 0;
+
+ amdgpu_device_wait_for_vms_idle(adev);
adev->in_suspend = true;
--
2.55.0.windows.5
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-17 20:08 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 20:40 [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend Mario Limonciello
2026-11-16 21:13 ` sashiko-bot
2026-11-16 21:21 ` Mario Limonciello
2026-11-17 9:08 ` Christian König
2026-11-17 9:37 ` Christian König
2026-11-17 13:39 ` Thothiralingam, Logatharshan
2026-11-17 17:52 ` Thothiralingam, Logatharshan
2026-11-17 20:08 ` Thothiralingam, Logatharshan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.