From: "Timur Kristóf" <timur.kristof@gmail.com>
To: Jesse Zhang <Jesse.Zhang@amd.com>, amd-gfx@lists.freedesktop.org
Cc: Alexander.Deucher@amd.com,
Christian Koenig <christian.koenig@amd.com>,
Lijo.Lazar@amd.com, Mario.Novales@amd.com, Manu.Rastogi@amd.com,
Jiongyu.Zhu@amd.com
Subject: Re: [PATCH v4 08/13] drm/amdgpu/userq: add helpers to keep the user queues of one engine across a reset
Date: Thu, 19 Nov 2026 12:36:38 +0100 [thread overview]
Message-ID: <1ec4c59a581ecb4b9753da5fcf06ca17b63b9f70.camel@gmail.com> (raw)
In-Reply-To: <20261105030612.726542-8-Jesse.Zhang@amd.com>
On Thu, 2026-11-05 at 11:05 +0800, Jesse Zhang wrote:
> An engine reset, eg. of the gfx engine, only resets the queues of one
> engine of the GFX IP; the queues of the other engines aren't reset.
> amdgpu_userq_pre_reset() and amdgpu_userq_post_reset() handle all the
> user queues for a full reset. Add separate helpers for the queues of
> one
> engine, which keep them in the state they had:
>
> - amdgpu_userq_save_hqds(), called after the MES suspend, saves the
> HQD
> of the mapped queues still on a HQD: the reset clears their HQD,
> and
> MES couldn't preempt them into their MQD. The queues MES preempted
> have their state in their MQD already.
> - amdgpu_userq_restore_hqds(), called after the reset but before MES
> resumes, unmaps those queues, writes the saved HQD into their MQD
> and
> maps them again: MES doesn't map a queue whose HQD the reset
> cleared
> again on its own. The queues keep their state and their work,
> except
> the one the pipe executed: its work in flight depends on pipe state
> the reset cleared, it is dropped and its fences signaled. The queue
> that hung, if any, is unmapped and hung.
> - amdgpu_userq_clear_hqds() drops the saved HQDs for the full reset
> that
> follows a failed engine reset.
>
> The helpers walk the queues and unmap and map them as
> amdgpu_userq_pre_reset() and amdgpu_userq_post_reset() do. The saved
> HQDs are kept in the gfx IP, in saved_gfx_hqds[], and matched to a
> queue
> by its doorbell and MQD address.
>
> v2: Drop the user queue reset lock and the walk over the queues of
> one
> type (Christian).
> v4: Rename the helpers and the saved HQD array (Timur).
>
> Signed-off-by: Jesse Zhang <Jesse.Zhang@amd.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h | 3 +
> drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 107
> ++++++++++++++++++++++
> drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h | 4 +
> 3 files changed, 114 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
> index 1c23e97fd04d..79cb6be75329 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
> @@ -515,6 +515,9 @@ struct amdgpu_gfx {
>
> /* reset mask */
> uint32_t gfx_supported_reset;
> + /* HQDs of the gfx user queues, saved by an engine reset */
> + struct
> amdgpu_gfx_saved_hqd saved_gfx_hqds[AMDGPU_MAX_GFX_QUEUES];
> + u32 num_saved_gfx_hqds;
> uint32_t compute_supported_reset;
>
> /* gfx off */
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> index ccf0ff490ac2..7d0c4e8795b4 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> @@ -1594,3 +1594,110 @@ int amdgpu_userq_post_reset(struct
> amdgpu_device *adev, bool vram_lost)
>
> return ret;
> }
> +
> +/**
> + * amdgpu_userq_save_hqds - save the HQDs of the user queues before
> an engine reset
> + * @adev: amdgpu device
> + * @queue_type: AMDGPU_HW_IP_* type of the queues on the reset
> engine
> + */
> +void amdgpu_userq_save_hqds(struct amdgpu_device *adev, u32
> queue_type)
> +{
> + struct amdgpu_gfx_saved_hqd hqd;
> + struct amdgpu_usermode_queue *queue;
> + unsigned long queue_id;
> +
> + adev->gfx.num_saved_gfx_hqds = 0;
> + if (!adev->gfx.funcs->save_hqd)
> + return;
> +
> + xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> + if (adev->gfx.num_saved_gfx_hqds >=
> AMDGPU_MAX_GFX_QUEUES)
> + break;
> + if (queue->queue_type != queue_type ||
> + queue->state != AMDGPU_USERQ_STATE_MAPPED)
> + continue;
> + if (adev->gfx.funcs->save_hqd(adev, queue-
> >doorbell_index, &hqd) &&
> + hqd.on_hqd)
> + adev->gfx.saved_gfx_hqds[adev-
> >gfx.num_saved_gfx_hqds++] = hqd;
> + }
> +}
> +
> +static struct amdgpu_gfx_saved_hqd *
> +amdgpu_userq_find_saved_hqd(struct amdgpu_device *adev,
> + struct amdgpu_usermode_queue *queue)
> +{
> + struct amdgpu_gfx_saved_hqd *hqd;
> + u32 i;
> +
> + for (i = 0; i < adev->gfx.num_saved_gfx_hqds; i++) {
> + hqd = &adev->gfx.saved_gfx_hqds[i];
> + /* Check the MQD too: a new queue may reuse the
> doorbell. */
> + if (hqd->doorbell_index == queue->doorbell_index &&
> + hqd->mqd_addr == queue->mqd.gpu_addr)
> + return hqd;
> + }
> + return NULL;
> +}
> +
> +/**
> + * amdgpu_userq_restore_hqds - restore the user queues after an
> engine reset
> + * @adev: amdgpu device
> + * @queue_type: AMDGPU_HW_IP_* type of the queues on the reset
> engine
> + * @guilty: the queue that hung, or NULL
> + *
> + * MES doesn't map a queue whose HQD the reset cleared again on its
> own: unmap
> + * it from MES, restore its MQD, and map it again.
> + */
> +void amdgpu_userq_restore_hqds(struct amdgpu_device *adev, u32
> queue_type,
> + struct amdgpu_usermode_queue *guilty)
> +{
> + const struct amdgpu_userq_funcs *userq_funcs = adev-
> >userq_funcs[queue_type];
> + struct amdgpu_usermode_queue *queue;
> + struct amdgpu_gfx_saved_hqd *hqd;
> + unsigned long queue_id;
> + int r;
> +
> + xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> + if (queue->queue_type != queue_type ||
> + queue->state != AMDGPU_USERQ_STATE_MAPPED)
> + continue;
> +
> + if (queue == guilty) {
> + userq_funcs->unmap(queue);
> + amdgpu_userq_fence_driver_force_completion(q
> ueue);
> + trace_amdgpu_userq_state_changed(queue,
> AMDGPU_USERQ_STATE_HUNG);
> + queue->state = AMDGPU_USERQ_STATE_HUNG;
> + continue;
> + }
> +
> + hqd = amdgpu_userq_find_saved_hqd(adev, queue);
> + if (!hqd)
> + continue;
> +
> + userq_funcs->unmap(queue);
> + if (adev->gfx.funcs->restore_hqd(adev, queue-
> >mqd.cpu_ptr,
> + queue->userq_prop-
> >queue_size,
> + hqd))
> + amdgpu_userq_fence_driver_force_completion_w
> ith_seqno(queue,
> +
> hqd->wptr);
> + r = userq_funcs->map(queue);
> + if (r) {
> + dev_err(adev->dev, "Failed to remap user
> queue %llu: %d\n",
> + queue->doorbell_index, r);
> + amdgpu_userq_fence_driver_force_completion(q
> ueue);
> + trace_amdgpu_userq_state_changed(queue,
> AMDGPU_USERQ_STATE_HUNG);
> + queue->state = AMDGPU_USERQ_STATE_HUNG;
> + }
> + }
> + adev->gfx.num_saved_gfx_hqds = 0;
> +}
> +
> +/**
> + * amdgpu_userq_clear_hqds - drop the saved HQDs after a failed
> engine reset
> + * @adev: amdgpu device
> + * @queue_type: AMDGPU_HW_IP_* type of the queues on the reset
> engine
> + */
> +void amdgpu_userq_clear_hqds(struct amdgpu_device *adev, u32
> queue_type)
> +{
> + adev->gfx.num_saved_gfx_hqds = 0;
> +}
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h
> index 488dc21d7c81..b5195031877d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h
> @@ -181,6 +181,10 @@ int
> amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device
> *adev,
> void amdgpu_userq_reset_work(struct work_struct *work);
> void amdgpu_userq_pre_reset(struct amdgpu_device *adev);
> int amdgpu_userq_post_reset(struct amdgpu_device *adev, bool
> vram_lost);
> +void amdgpu_userq_save_hqds(struct amdgpu_device *adev, u32
> queue_type);
> +void amdgpu_userq_restore_hqds(struct amdgpu_device *adev, u32
> queue_type,
> + struct amdgpu_usermode_queue
> *guilty);
> +void amdgpu_userq_clear_hqds(struct amdgpu_device *adev, u32
> queue_type);
> void amdgpu_userq_start_hang_detect_work(struct
> amdgpu_usermode_queue *queue);
> void amdgpu_userq_process_fence_irq(struct amdgpu_device *adev, u32
> doorbell);
>
next prev parent reply other threads:[~2026-11-19 11:36 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 3:05 [PATCH v4 01/13] drm/amdgpu: pass the device to the multi ring reset helpers Jesse Zhang
2026-11-05 3:05 ` [PATCH v4 02/13] drm/amdgpu/ring: During multi-queue reset, only call begin helper if needed Jesse Zhang
2026-11-11 23:52 ` Daniel Campos Ramos
2026-11-19 11:34 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 03/13] drm/amdgpu: add an engine scope to the IP block soft reset Jesse Zhang
2026-11-19 11:35 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 04/13] drm/amdgpu/gfx: add callbacks to save and restore a gfx HQD Jesse Zhang
2026-11-19 11:35 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 05/13] drm/amdgpu/gfx12: " Jesse Zhang
2026-11-19 11:35 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 06/13] drm/amdgpu/gfx11: " Jesse Zhang
2026-11-19 11:36 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 07/13] drm/amdgpu/userq: add a helper to signal the fences of a queue up to a point Jesse Zhang
2026-11-19 11:36 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 08/13] drm/amdgpu/userq: add helpers to keep the user queues of one engine across a reset Jesse Zhang
2026-11-19 11:36 ` Timur Kristóf [this message]
2026-11-05 3:05 ` [PATCH v4 09/13] drm/amdgpu/gfx12: add a gfx engine soft reset Jesse Zhang
2026-11-19 11:42 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 10/13] drm/amdgpu/gfx11: " Jesse Zhang
2026-11-19 11:47 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 11/13] drm/amdgpu/userq: reset the gfx engine when a gfx user queue reset fails Jesse Zhang
2026-11-19 11:47 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 12/13] drm/amdgpu/gfx12: enable the gfx engine soft reset Jesse Zhang
2026-11-19 11:47 ` Timur Kristóf
2026-11-05 3:05 ` [PATCH v4 13/13] drm/amdgpu/gfx11: " Jesse Zhang
2026-11-19 11:47 ` Timur Kristóf
2026-11-10 2:06 ` [PATCH v4 01/13] drm/amdgpu: pass the device to the multi ring reset helpers Zhang, Jesse(Jie)
2026-11-11 15:28 ` Timur Kristóf
2026-11-19 11:33 ` Timur Kristóf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1ec4c59a581ecb4b9753da5fcf06ca17b63b9f70.camel@gmail.com \
--to=timur.kristof@gmail.com \
--cc=Alexander.Deucher@amd.com \
--cc=Jesse.Zhang@amd.com \
--cc=Jiongyu.Zhu@amd.com \
--cc=Lijo.Lazar@amd.com \
--cc=Manu.Rastogi@amd.com \
--cc=Mario.Novales@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=christian.koenig@amd.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.