* [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir
@ 2026-11-20 17:24 Christoph Berliner
2026-11-20 17:24 ` [PATCH 1/2] drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check Christoph Berliner
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Christoph Berliner @ 2026-11-20 17:24 UTC (permalink / raw)
To: amd-gfx
Cc: alexander.deucher, christian.koenig, Jiqian.Chen,
Christoph Berliner
On Renoir/Cezanne (GC 9.3.0) a MODE2 reset reports success, graphics and
the kernel compute queues come back, but KFD stays dead until reboot:
every new KFD queue runs into "qcm fence wait loop timeout expired"
and triggers the next reset. Reported in
https://gitlab.freedesktop.org/drm/amd/-/issues/6177, sent here as
asked there.
Commit c3988a7ad479 ("drm/amdgpu/gfx9: Fix Ring and IB test fail after
mode2") already found that MODE2 does not reset the HQD registers on
gfx9 APUs and deactivates the KCQ HQDs before the reset. The HIQ used by
KFD survives as well, though: kq_uninitialize() skips destroy_mqd()
during a reset because it cannot take the reset domain semaphore. After
MODE2 the MEC restarts with the stale HIQ and the HWS no longer answers.
The same trylock pattern on the MES side is discussed in
https://gitlab.freedesktop.org/drm/amd/-/issues/6053.
Patch 1 is a small cleanup plus a fix for the timeout check of the
existing dequeue loop. Patch 2 deactivates all active compute HQDs
(except the KIQ) before MODE2, restricted to GC 9.3.0 because that is
the only gfx9 APU I could test. Raven/Picasso likely need the same, but
I don't know this code well enough to widen the condition without
hardware to test on.
Tested on a Ryzen 5 5600G (gfx90c), MEC firmware 0x1e3, HWS,
amdgpu.noretry=1. An instrumented build used the old path for the first
MODE2 and the new path for later ones, so both were compared in the
same boot:
old path: HIQ still ACTIVE with the old MQD/PQ after MODE2; next HIP
program -> "qcm fence wait loop timeout", GPU hang, reset
new path: only the KIQ active at resume, HIP/rocBLAS tests pass
Since then the patches run as the boot-time amdgpu module on the
Proxmox 7.0.14 kernels (-29 and now -30): requested reset with 9 HQDs
deactivated and the KIQ skipped, HIP/rocBLAS work before and after.
On amd-staging-drm-next (8dd68a795) the series applies with offsets
and builds without warnings (W=1).
The analysis and the patches were prepared with an AI coding assistant
(see the Assisted-by tags); I tested them on the hardware described
above.
Christoph Berliner (2):
drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check
drm/amdgpu/gfx9: deactivate all compute HQDs before MODE2 on Renoir
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 105 ++++++++++++++++++++------
1 file changed, 82 insertions(+), 23 deletions(-)
--
2.56.1
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/2] drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check
2026-11-20 17:24 [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir Christoph Berliner
@ 2026-11-20 17:24 ` Christoph Berliner
2026-11-20 17:24 ` [PATCH 2/2] drm/amdgpu/gfx9: deactivate all compute HQDs before MODE2 on Renoir Christoph Berliner
2026-11-20 19:25 ` [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after " Alex Deucher
2 siblings, 0 replies; 4+ messages in thread
From: Christoph Berliner @ 2026-11-20 17:24 UTC (permalink / raw)
To: amd-gfx
Cc: alexander.deucher, christian.koenig, Jiqian.Chen,
Christoph Berliner
gfx_v9_0_deactivate_kcq_hqd() polls CP_HQD_ACTIVE for adev->usec_timeout
iterations, but then compares the loop counter against
AMDGPU_MAX_USEC_TIMEOUT. Unless both values happen to be equal, the
manual CP_HQD_ACTIVE=0 fallback for a timed-out dequeue request is never
reached. Compare against the actual loop bound.
Move the per-queue deactivation into gfx_v9_0_deactivate_hqd() so it can
be reused for queues that are not kernel compute rings, and use
dev_dbg() instead of DRM_DEBUG() while at it. No functional change other
than the timeout check.
Fixes: c3988a7ad479 ("drm/amdgpu/gfx9: Fix Ring and IB test fail after mode2")
Assisted-by: Claude Code
Signed-off-by: Christoph Berliner <caberliner@gmail.com>
---
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 50 ++++++++++++++++-----------
1 file changed, 29 insertions(+), 21 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
index 9d87ebbc4..b60b6fa88 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
@@ -4076,35 +4076,43 @@ static int gfx_v9_0_hw_init(struct amdgpu_ip_block *ip_block)
return r;
}
+/* caller must hold srbm_mutex and have selected me/pipe/queue */
+static bool gfx_v9_0_deactivate_hqd(struct amdgpu_device *adev,
+ u32 me, u32 pipe, u32 queue)
+{
+ int j;
+
+ /* disable the queue if it's active */
+ if (!(RREG32_SOC15(GC, 0, mmCP_HQD_ACTIVE) & CP_HQD_ACTIVE__ACTIVE_MASK))
+ return false;
+
+ WREG32_SOC15(GC, 0, mmCP_HQD_DEQUEUE_REQUEST, 1);
+ for (j = 0; j < adev->usec_timeout; j++) {
+ if (!(RREG32_SOC15(GC, 0, mmCP_HQD_ACTIVE) &
+ CP_HQD_ACTIVE__ACTIVE_MASK))
+ break;
+ udelay(1);
+ }
+ if (j == adev->usec_timeout) {
+ dev_dbg(adev->dev, "comp_%u_%u_%u dequeue request failed.\n",
+ me, pipe, queue);
+ /* Manual disable if dequeue request times out */
+ WREG32_SOC15(GC, 0, mmCP_HQD_ACTIVE, 0);
+ }
+ WREG32_SOC15(GC, 0, mmCP_HQD_DEQUEUE_REQUEST, 0);
+
+ return true;
+}
+
static void gfx_v9_0_deactivate_kcq_hqd(struct amdgpu_device *adev)
{
amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
for (int i = 0; i < adev->gfx.num_compute_rings; i++) {
- u32 tmp;
struct amdgpu_ring *ring = &adev->gfx.compute_ring[i];
mutex_lock(&adev->srbm_mutex);
soc15_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0, 0);
- tmp = RREG32_SOC15(GC, 0, mmCP_HQD_ACTIVE);
- /* disable the queue if it's active */
- if (tmp & CP_HQD_ACTIVE__ACTIVE_MASK) {
- int j;
-
- WREG32_SOC15(GC, 0, mmCP_HQD_DEQUEUE_REQUEST, 1);
- for (j = 0; j < adev->usec_timeout; j++) {
- tmp = RREG32_SOC15(GC, 0, mmCP_HQD_ACTIVE);
- if (!(tmp & CP_HQD_ACTIVE__ACTIVE_MASK))
- break;
- udelay(1);
- }
- if (j == AMDGPU_MAX_USEC_TIMEOUT) {
- DRM_DEBUG("comp_%u_%u_%u dequeue request failed.\n",
- ring->me, ring->pipe, ring->queue);
- /* Manual disable if dequeue request times out */
- WREG32_SOC15(GC, 0, mmCP_HQD_ACTIVE, 0);
- }
- WREG32_SOC15(GC, 0, mmCP_HQD_DEQUEUE_REQUEST, 0);
- }
+ gfx_v9_0_deactivate_hqd(adev, ring->me, ring->pipe, ring->queue);
soc15_grbm_select(adev, 0, 0, 0, 0, 0);
mutex_unlock(&adev->srbm_mutex);
}
--
2.56.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 2/2] drm/amdgpu/gfx9: deactivate all compute HQDs before MODE2 on Renoir
2026-11-20 17:24 [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir Christoph Berliner
2026-11-20 17:24 ` [PATCH 1/2] drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check Christoph Berliner
@ 2026-11-20 17:24 ` Christoph Berliner
2026-11-20 19:25 ` [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after " Alex Deucher
2 siblings, 0 replies; 4+ messages in thread
From: Christoph Berliner @ 2026-11-20 17:24 UTC (permalink / raw)
To: amd-gfx
Cc: alexander.deucher, christian.koenig, Jiqian.Chen,
Christoph Berliner
On Renoir (GC 9.3.0) MODE2 does not reset the CP HQD registers. Commit
c3988a7ad479 ("drm/amdgpu/gfx9: Fix Ring and IB test fail after mode2")
deactivates the HQDs of the kernel compute queues before MODE2, but the
HQDs owned by KFD stay active as well:
- the HIQ is never destroyed during a GPU reset, because
kq_uninitialize() skips destroy_mqd() when it cannot take the reset
domain semaphore, which amdgpu_device_gpu_recover() holds for write
before calling amdgpu_amdkfd_pre_reset();
- user queues mapped by the HWS stay active when the preceding unmap
failed (which is the usual reason for the reset).
After MODE2 the MEC is unhalted with the stale HIQ. The reset reports
success and graphics/KCQs work again, but the HWS no longer answers:
the next KFD queue creation runs into
amdgpu: qcm fence wait loop timeout expired
amdgpu: The cp might be in an unrecoverable state due to an unsuccessful queues preemption
amdgpu: process pid ... DQM create queue type 0 failed. ret -62
which triggers another reset, and KFD stays unusable until reboot.
On Renoir, deactivate every active HQD of all MECs before MODE2 except
the KIQ, which gfx_v9_0_kiq_init_register() reprograms on resume. The
KIQ is also identified by its MQD address, because on Renoir the HQD
registers of MEC2 pipes 2/3 read back as those of pipes 0/1 and the KIQ
would otherwise get a dequeue request through the alias.
Tested on Ryzen 5 5600G (Cezanne, gfx90c, GC 9.3.0), MEC firmware
0x1e3, HWS enabled, amdgpu.noretry=1, with an instrumented build that
uses the old path for the first MODE2 and the new one afterwards (same
boot):
- MODE2 #1 (old path): after the reset, before KIQ init, the HIQ
(MEC2 pipe 0 queue 0) is still ACTIVE with its old MQD/PQ; the next
HIP program fails with the messages above (GPU hang) and KFD triggers
MODE2 #2.
- MODE2 #2 (new path, first revision without the alias check): all
active HQDs deactivated, only the KIQ active at resume; HIP and
rocBLAS tests pass afterwards.
This version (with the KIQ alias check) was then tested on its own,
loaded at boot: a requested reset deactivated 9 HQDs (8 KCQs + HIQ),
skipped the KIQ seen through the pipe alias, and HIP/rocBLAS tests
passed before and after the reset.
Other gfx9 APUs (Raven/Picasso, gfx902/gfx909) probably have the same
issue, since kq_uninitialize() behaves the same there, but they keep
the existing KCQ-only behaviour because this could not be tested on
them.
Fixes: c3988a7ad479 ("drm/amdgpu/gfx9: Fix Ring and IB test fail after mode2")
Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/6177
Assisted-by: Claude Code
Signed-off-by: Christoph Berliner <caberliner@gmail.com>
---
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 55 ++++++++++++++++++++++++++-
1 file changed, 53 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
index b60b6fa88..59c0cbd86 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
@@ -4119,6 +4119,53 @@ static void gfx_v9_0_deactivate_kcq_hqd(struct amdgpu_device *adev)
amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
}
+/*
+ * Renoir: MODE2 does not reset the CP HQD registers. Besides the KCQs, the
+ * HQDs owned by KFD (HIQ and the user queues mapped by the HWS) also stay
+ * active, because KFD cannot unmap/destroy them while the reset domain is
+ * locked. After MODE2 the MEC then resumes with stale HQDs and the HWS no
+ * longer answers (qcm fence timeout on the next KFD queue). Deactivate every
+ * active HQD of all MECs except the KIQ, which is reprogrammed on resume.
+ */
+static void gfx_v9_0_deactivate_all_hqd(struct amdgpu_device *adev)
+{
+ struct amdgpu_ring *kiq_ring = &adev->gfx.kiq[0].ring;
+ u64 kiq_mqd_addr = kiq_ring->mqd_gpu_addr & ~0x3ULL;
+ u32 me, pipe, queue, n = 0;
+ u64 mqd_addr;
+
+ amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
+ mutex_lock(&adev->srbm_mutex);
+ for (me = 1; me <= adev->gfx.mec.num_mec; me++) {
+ for (pipe = 0; pipe < adev->gfx.mec.num_pipe_per_mec; pipe++) {
+ for (queue = 0; queue < adev->gfx.mec.num_queue_per_pipe;
+ queue++) {
+ if (me == kiq_ring->me && pipe == kiq_ring->pipe &&
+ queue == kiq_ring->queue)
+ continue;
+ soc15_grbm_select(adev, me, pipe, queue, 0, 0);
+ /*
+ * On Renoir the HQD registers of MEC2 pipes 2/3
+ * read back as those of pipes 0/1, so the KIQ
+ * shows up a second time. Skip it by its MQD
+ * address instead of sending it a dequeue.
+ */
+ mqd_addr = ((u64)RREG32_SOC15(GC, 0, mmCP_MQD_BASE_ADDR_HI) << 32) |
+ RREG32_SOC15(GC, 0, mmCP_MQD_BASE_ADDR);
+ if (mqd_addr == kiq_mqd_addr)
+ continue;
+ if (gfx_v9_0_deactivate_hqd(adev, me, pipe, queue))
+ n++;
+ }
+ }
+ }
+ soc15_grbm_select(adev, 0, 0, 0, 0, 0);
+ mutex_unlock(&adev->srbm_mutex);
+ amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
+
+ dev_dbg(adev->dev, "deactivated %u active compute HQDs before MODE2\n", n);
+}
+
static int gfx_v9_0_hw_fini(struct amdgpu_ip_block *ip_block)
{
struct amdgpu_device *adev = ip_block->adev;
@@ -4144,8 +4191,12 @@ static int gfx_v9_0_hw_fini(struct amdgpu_ip_block *ip_block)
}
if ((adev->flags & AMD_IS_APU) && amdgpu_in_reset(adev) &&
- amdgpu_asic_reset_method(adev) == AMD_RESET_METHOD_MODE2)
- gfx_v9_0_deactivate_kcq_hqd(adev);
+ amdgpu_asic_reset_method(adev) == AMD_RESET_METHOD_MODE2) {
+ if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 3, 0))
+ gfx_v9_0_deactivate_all_hqd(adev);
+ else
+ gfx_v9_0_deactivate_kcq_hqd(adev);
+ }
/* Use deinitialize sequence from CAIL when unbinding device from driver,
* otherwise KIQ is hanging when binding back
--
2.56.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir
2026-11-20 17:24 [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir Christoph Berliner
2026-11-20 17:24 ` [PATCH 1/2] drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check Christoph Berliner
2026-11-20 17:24 ` [PATCH 2/2] drm/amdgpu/gfx9: deactivate all compute HQDs before MODE2 on Renoir Christoph Berliner
@ 2026-11-20 19:25 ` Alex Deucher
2 siblings, 0 replies; 4+ messages in thread
From: Alex Deucher @ 2026-11-20 19:25 UTC (permalink / raw)
To: Christoph Berliner
Cc: amd-gfx, alexander.deucher, christian.koenig, Jiqian.Chen
On Fri, Nov 20, 2026 at 12:46 PM Christoph Berliner
<caberliner@gmail.com> wrote:
>
> On Renoir/Cezanne (GC 9.3.0) a MODE2 reset reports success, graphics and
> the kernel compute queues come back, but KFD stays dead until reboot:
> every new KFD queue runs into "qcm fence wait loop timeout expired"
> and triggers the next reset. Reported in
> https://gitlab.freedesktop.org/drm/amd/-/issues/6177, sent here as
> asked there.
>
> Commit c3988a7ad479 ("drm/amdgpu/gfx9: Fix Ring and IB test fail after
> mode2") already found that MODE2 does not reset the HQD registers on
> gfx9 APUs and deactivates the KCQ HQDs before the reset. The HIQ used by
> KFD survives as well, though: kq_uninitialize() skips destroy_mqd()
> during a reset because it cannot take the reset domain semaphore. After
> MODE2 the MEC restarts with the stale HIQ and the HWS no longer answers.
> The same trylock pattern on the MES side is discussed in
> https://gitlab.freedesktop.org/drm/amd/-/issues/6053.
>
> Patch 1 is a small cleanup plus a fix for the timeout check of the
> existing dequeue loop. Patch 2 deactivates all active compute HQDs
> (except the KIQ) before MODE2, restricted to GC 9.3.0 because that is
> the only gfx9 APU I could test. Raven/Picasso likely need the same, but
> I don't know this code well enough to widen the condition without
> hardware to test on.
>
> Tested on a Ryzen 5 5600G (gfx90c), MEC firmware 0x1e3, HWS,
> amdgpu.noretry=1. An instrumented build used the old path for the first
> MODE2 and the new path for later ones, so both were compared in the
> same boot:
>
> old path: HIQ still ACTIVE with the old MQD/PQ after MODE2; next HIP
> program -> "qcm fence wait loop timeout", GPU hang, reset
> new path: only the KIQ active at resume, HIP/rocBLAS tests pass
>
> Since then the patches run as the boot-time amdgpu module on the
> Proxmox 7.0.14 kernels (-29 and now -30): requested reset with 9 HQDs
> deactivated and the KIQ skipped, HIP/rocBLAS work before and after.
> On amd-staging-drm-next (8dd68a795) the series applies with offsets
> and builds without warnings (W=1).
>
> The analysis and the patches were prepared with an AI coding assistant
> (see the Assisted-by tags); I tested them on the hardware described
> above.
>
> Christoph Berliner (2):
> drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check
> drm/amdgpu/gfx9: deactivate all compute HQDs before MODE2 on Renoir
I would just squash this into one patch to deactivate the HQDs on all
queues. No need to make this renoir specific. It should not hurt
anything to do this on all gfx9 APUs.
Alex
>
> drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 105 ++++++++++++++++++++------
> 1 file changed, 82 insertions(+), 23 deletions(-)
>
> --
> 2.56.1
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-20 19:25 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 17:24 [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir Christoph Berliner
2026-11-20 17:24 ` [PATCH 1/2] drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check Christoph Berliner
2026-11-20 17:24 ` [PATCH 2/2] drm/amdgpu/gfx9: deactivate all compute HQDs before MODE2 on Renoir Christoph Berliner
2026-11-20 19:25 ` [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after " Alex Deucher
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.