All of lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Mario Limonciello" <mario.limonciello@amd.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend
Date: Mon, 16 Nov 2026 21:13:42 +0000	[thread overview]
Message-ID: <sashiko-outbox-217693@kernel.org> (raw)
In-Reply-To: <20261116-superm1-drain-submissions-v1-1-e3ffee977a28@amd.com>

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] drm/amdgpu: deadlock in amdgpu_device_flush_file_entities() on
  suspend
- [Medium] drm/amdgpu: race in amdgpu_device_flush_file_entities()
  before run_job()
--

commit 2d3632cb05b8d04352d9590da3fabb5b096f1b19
Author: Mario Limonciello <mario.limonciello@amd.com>

drm/amdgpu: drain queued scheduler jobs before suspend

This commit adds amdgpu_device_flush_file_entities() to flush the context
and VM scheduler entities of all open DRM files during
amdgpu_device_suspend() before evicting resources.

> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 19a566187dfb5..d18348116d3f1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -4538,6 +4538,30 @@ void amdgpu_device_complete(struct drm_device *dev)
>  	}
>  }
>  
> +/**
> + * amdgpu_device_flush_file_entities - drain scheduler entities of open files
> + * @adev: amdgpu device object
> + *
> + * Wait for the scheduler entities owned by all open files (the context
> + * entities used for command submission and the VM entities used for page
> + * table updates) to run out of queued jobs, so that nothing can slip into
> + * the rings after the hardware fences were waited on at suspend time.
> + */
> +static void amdgpu_device_flush_file_entities(struct amdgpu_device *adev)
> +{
> +	struct drm_device *ddev = adev_to_drm(adev);
> +	struct drm_file *file;
> +
> +	guard(mutex)(&ddev->filelist_mutex);
> +	list_for_each_entry(file, &ddev->filelist, lhead) {
> +		struct amdgpu_fpriv *fpriv = file->driver_priv;
> +
> +		amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr,
> +					    MAX_SCHEDULE_TIMEOUT);
> +		amdgpu_vm_wait_idle(&fpriv->vm, MAX_SCHEDULE_TIMEOUT);
> +	}
> +}

[Severity: Medium]
Can drm_sched_entity_flush() return while the scheduler worker is still in
the middle of popping a job and before sched->ops->run_job() has emitted its
hardware fence?

CPU0 (submit_wq)                  CPU1 (suspend)
drm_sched_run_job_work()
  drm_sched_entity_pop_job()
    spsc_queue_pop(&entity->job_queue)
    drm_sched_rq_pop_entity(entity)
                                  amdgpu_device_suspend()
                                    amdgpu_device_flush_file_entities()
                                      ...
                                        drm_sched_entity_flush()
                                          drm_sched_entity_is_idle() == true
                                    amdgpu_device_evict_resources()
                                    amdgpu_fence_driver_hw_fini()
                                      amdgpu_fence_wait_empty()
  sched->ops->run_job(sched_job)
    amdgpu_job_run()
      amdgpu_ib_schedule()
        amdgpu_fence_emit()
        amdgpu_ring_commit()  <-- ring written after fence wait
  complete_all(&entity->entity_idle)

drm_sched_entity_is_idle() reports the entity idle as soon as
spsc_queue_pop() empties entity->job_queue, before drm_sched_run_job_work()
calls sched->ops->run_job() and complete_all(&entity->entity_idle).  On APUs
such as Picasso where amdgpu_device_evict_resources() returns immediately
during S3 suspend, amdgpu_fence_driver_hw_fini() can finish waiting on the
ring before the popped job emits its fence and commits the ring write
pointer, leaving the ring hang on resume window open.

> +
>  /**
>   * amdgpu_device_suspend - initiate device suspend
>   *
> @@ -4587,6 +4611,8 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients)
>  	if (r)
>  		goto unwind_ip_phase1;
>  
> +	amdgpu_device_flush_file_entities(adev);

[Severity: High]
Can calling amdgpu_device_flush_file_entities() here deadlock runtime or
system suspend when an entity has a job waiting on an unsignaled dependency?

drivers/gpu/drm/scheduler/sched_entity.c:drm_sched_entity_flush() {
	...
	if (current->flags & PF_EXITING) {
		if (timeout)
			ret = wait_event_timeout(
					sched->job_scheduled,
					drm_sched_entity_is_idle(entity),
					timeout);
	} else {
		wait_event_killable(sched->job_scheduled,
				    drm_sched_entity_is_idle(entity));
	}
	...
}

Because the suspend task or runtime PM worker does not have PF_EXITING set,
drm_sched_entity_flush() ignores the timeout and sleeps indefinitely in
wait_event_killable() while holding ddev->filelist_mutex.

In amdgpu_cs_ioctl(), a runtime PM reference is only held during the ioctl
itself, and per-job runtime PM references are only taken in
amdgpu_fence_emit() when a job reaches run_job().  If a client submits a job
with an unsignaled syncobj dependency, runtime autosuspend can enter
amdgpu_pmops_runtime_suspend() -> amdgpu_device_suspend() and block in
amdgpu_device_flush_file_entities() while the device status is
RPM_SUSPENDING.  When another thread or process then calls amdgpu_cs_ioctl()
to submit the work that would signal that syncobj, its runtime PM resume
waits for RPM_SUSPENDING to complete, deadlocking both tasks.

Killing the process owning the blocked entity does not break the wait
either: amdgpu_flush() times out after 2 seconds and calls
drm_sched_entity_kill(), which drains the entity without waking
sched->job_scheduled, and the exiting task then blocks on
ddev->filelist_mutex in drm_close_helper().  During system suspend,
amdgpu_userq_suspend() also runs right before
amdgpu_device_flush_file_entities() and unmaps usermode queues while
userspace is already frozen, so any entity job waiting on a usermode queue
or userspace fence can never make progress.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-superm1-drain-submissions-v1-1-e3ffee977a28@amd.com?part=1

  reply	other threads:[~2026-11-16 21:13 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 20:40 [PATCH] drm/amdgpu: drain queued scheduler jobs before suspend Mario Limonciello
2026-11-16 21:13 ` sashiko-bot [this message]
2026-11-16 21:21   ` Mario Limonciello
2026-11-17  9:08     ` Christian König
2026-11-17  9:37 ` Christian König
2026-11-17 13:39   ` Thothiralingam, Logatharshan
2026-11-17 17:52     ` Thothiralingam, Logatharshan
2026-11-17 20:08       ` Thothiralingam, Logatharshan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-217693@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=mario.limonciello@amd.com \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.