From: Christoph Berliner <caberliner@gmail.com>
To: amd-gfx@lists.freedesktop.org
Cc: alexander.deucher@amd.com, christian.koenig@amd.com,
Jiqian.Chen@amd.com, Christoph Berliner <caberliner@gmail.com>
Subject: [PATCH 2/2] drm/amdgpu/gfx9: deactivate all compute HQDs before MODE2 on Renoir
Date: Fri, 20 Nov 2026 18:24:07 +0100 [thread overview]
Message-ID: <20261120172407.64651-3-caberliner@gmail.com> (raw)
In-Reply-To: <20261120172407.64651-1-caberliner@gmail.com>
On Renoir (GC 9.3.0) MODE2 does not reset the CP HQD registers. Commit
c3988a7ad479 ("drm/amdgpu/gfx9: Fix Ring and IB test fail after mode2")
deactivates the HQDs of the kernel compute queues before MODE2, but the
HQDs owned by KFD stay active as well:
- the HIQ is never destroyed during a GPU reset, because
kq_uninitialize() skips destroy_mqd() when it cannot take the reset
domain semaphore, which amdgpu_device_gpu_recover() holds for write
before calling amdgpu_amdkfd_pre_reset();
- user queues mapped by the HWS stay active when the preceding unmap
failed (which is the usual reason for the reset).
After MODE2 the MEC is unhalted with the stale HIQ. The reset reports
success and graphics/KCQs work again, but the HWS no longer answers:
the next KFD queue creation runs into
amdgpu: qcm fence wait loop timeout expired
amdgpu: The cp might be in an unrecoverable state due to an unsuccessful queues preemption
amdgpu: process pid ... DQM create queue type 0 failed. ret -62
which triggers another reset, and KFD stays unusable until reboot.
On Renoir, deactivate every active HQD of all MECs before MODE2 except
the KIQ, which gfx_v9_0_kiq_init_register() reprograms on resume. The
KIQ is also identified by its MQD address, because on Renoir the HQD
registers of MEC2 pipes 2/3 read back as those of pipes 0/1 and the KIQ
would otherwise get a dequeue request through the alias.
Tested on Ryzen 5 5600G (Cezanne, gfx90c, GC 9.3.0), MEC firmware
0x1e3, HWS enabled, amdgpu.noretry=1, with an instrumented build that
uses the old path for the first MODE2 and the new one afterwards (same
boot):
- MODE2 #1 (old path): after the reset, before KIQ init, the HIQ
(MEC2 pipe 0 queue 0) is still ACTIVE with its old MQD/PQ; the next
HIP program fails with the messages above (GPU hang) and KFD triggers
MODE2 #2.
- MODE2 #2 (new path, first revision without the alias check): all
active HQDs deactivated, only the KIQ active at resume; HIP and
rocBLAS tests pass afterwards.
This version (with the KIQ alias check) was then tested on its own,
loaded at boot: a requested reset deactivated 9 HQDs (8 KCQs + HIQ),
skipped the KIQ seen through the pipe alias, and HIP/rocBLAS tests
passed before and after the reset.
Other gfx9 APUs (Raven/Picasso, gfx902/gfx909) probably have the same
issue, since kq_uninitialize() behaves the same there, but they keep
the existing KCQ-only behaviour because this could not be tested on
them.
Fixes: c3988a7ad479 ("drm/amdgpu/gfx9: Fix Ring and IB test fail after mode2")
Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/6177
Assisted-by: Claude Code
Signed-off-by: Christoph Berliner <caberliner@gmail.com>
---
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 55 ++++++++++++++++++++++++++-
1 file changed, 53 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
index b60b6fa88..59c0cbd86 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
@@ -4119,6 +4119,53 @@ static void gfx_v9_0_deactivate_kcq_hqd(struct amdgpu_device *adev)
amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
}
+/*
+ * Renoir: MODE2 does not reset the CP HQD registers. Besides the KCQs, the
+ * HQDs owned by KFD (HIQ and the user queues mapped by the HWS) also stay
+ * active, because KFD cannot unmap/destroy them while the reset domain is
+ * locked. After MODE2 the MEC then resumes with stale HQDs and the HWS no
+ * longer answers (qcm fence timeout on the next KFD queue). Deactivate every
+ * active HQD of all MECs except the KIQ, which is reprogrammed on resume.
+ */
+static void gfx_v9_0_deactivate_all_hqd(struct amdgpu_device *adev)
+{
+ struct amdgpu_ring *kiq_ring = &adev->gfx.kiq[0].ring;
+ u64 kiq_mqd_addr = kiq_ring->mqd_gpu_addr & ~0x3ULL;
+ u32 me, pipe, queue, n = 0;
+ u64 mqd_addr;
+
+ amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
+ mutex_lock(&adev->srbm_mutex);
+ for (me = 1; me <= adev->gfx.mec.num_mec; me++) {
+ for (pipe = 0; pipe < adev->gfx.mec.num_pipe_per_mec; pipe++) {
+ for (queue = 0; queue < adev->gfx.mec.num_queue_per_pipe;
+ queue++) {
+ if (me == kiq_ring->me && pipe == kiq_ring->pipe &&
+ queue == kiq_ring->queue)
+ continue;
+ soc15_grbm_select(adev, me, pipe, queue, 0, 0);
+ /*
+ * On Renoir the HQD registers of MEC2 pipes 2/3
+ * read back as those of pipes 0/1, so the KIQ
+ * shows up a second time. Skip it by its MQD
+ * address instead of sending it a dequeue.
+ */
+ mqd_addr = ((u64)RREG32_SOC15(GC, 0, mmCP_MQD_BASE_ADDR_HI) << 32) |
+ RREG32_SOC15(GC, 0, mmCP_MQD_BASE_ADDR);
+ if (mqd_addr == kiq_mqd_addr)
+ continue;
+ if (gfx_v9_0_deactivate_hqd(adev, me, pipe, queue))
+ n++;
+ }
+ }
+ }
+ soc15_grbm_select(adev, 0, 0, 0, 0, 0);
+ mutex_unlock(&adev->srbm_mutex);
+ amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
+
+ dev_dbg(adev->dev, "deactivated %u active compute HQDs before MODE2\n", n);
+}
+
static int gfx_v9_0_hw_fini(struct amdgpu_ip_block *ip_block)
{
struct amdgpu_device *adev = ip_block->adev;
@@ -4144,8 +4191,12 @@ static int gfx_v9_0_hw_fini(struct amdgpu_ip_block *ip_block)
}
if ((adev->flags & AMD_IS_APU) && amdgpu_in_reset(adev) &&
- amdgpu_asic_reset_method(adev) == AMD_RESET_METHOD_MODE2)
- gfx_v9_0_deactivate_kcq_hqd(adev);
+ amdgpu_asic_reset_method(adev) == AMD_RESET_METHOD_MODE2) {
+ if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 3, 0))
+ gfx_v9_0_deactivate_all_hqd(adev);
+ else
+ gfx_v9_0_deactivate_kcq_hqd(adev);
+ }
/* Use deinitialize sequence from CAIL when unbinding device from driver,
* otherwise KIQ is hanging when binding back
--
2.56.1
next prev parent reply other threads:[~2026-11-20 17:38 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 17:24 [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir Christoph Berliner
2026-11-20 17:24 ` [PATCH 1/2] drm/amdgpu/gfx9: factor out HQD deactivation and fix its timeout check Christoph Berliner
2026-11-20 17:24 ` Christoph Berliner [this message]
2026-11-20 19:25 ` [PATCH 0/2] drm/amdgpu/gfx9: make KFD usable again after MODE2 on Renoir Alex Deucher
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120172407.64651-3-caberliner@gmail.com \
--to=caberliner@gmail.com \
--cc=Jiqian.Chen@amd.com \
--cc=alexander.deucher@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=christian.koenig@amd.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.