* [PATCH v2] drm/amdkfd: Move gpu_recovery location
@ 2026-11-16 7:50 Amber Lin
2026-11-16 8:13 ` sashiko-bot
0 siblings, 1 reply; 2+ messages in thread
From: Amber Lin @ 2026-11-16 7:50 UTC (permalink / raw)
To: amd-gfx, alexander.deucher; +Cc: jesse.zhang, Amber Lin
amdgpu_gfx_reset_mes_compute combines two features into one:
1) hung queue/pipe reset - suspend all/queue reset/resume all
2) bad queue removal - suspend_all/remove_queue/resume_all
GPU recovery aims on 1)resetting hung queues/pipes. When gpu_recovery is
disabled, we need to keep 2) going. Therefore move the gpu_recovery
check from KFD's reset_queues_mes into amdgpu_gfx_reset_mes_compute
where queue reset is done.
Signed-off-by: Amber Lin <amber.lin@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c | 10 +++++++---
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 9 +++------
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
index c5a2dcff5058..c30fecc653b4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
@@ -2594,14 +2594,17 @@ int amdgpu_gfx_reset_mes_compute(struct amdgpu_device *adev,
* reset detect will return the array of bad queue doorbells
*/
r = amdgpu_mes_suspend(adev, xcc_id);
+ dev_dbg(adev->dev, "Suspend all returns %d. xcc_id %d\n", r, xcc_id);
+
+ if (!amdgpu_gpu_recovery)
+ goto after_reset;
+
/* if suspend all success, it should no hang queue */
- if (!r) {
- dev_dbg(adev->dev, "Suspend all succeeds. xcc_id %d\n", xcc_id);
+ if (!r)
/* always reset the KCQ/userq since we need to signal the fence
* and we could be stuck in a loop which is preemptable.
*/
goto fence_reset;
- }
r = amdgpu_mes_detect_and_reset_hung_queues(adev, AMDGPU_RING_TYPE_COMPUTE,
true, &num_hung, adev->gfx.mec.mes_hung_db_array, xcc_id);
if (r) {
@@ -2662,6 +2665,7 @@ int amdgpu_gfx_reset_mes_compute(struct amdgpu_device *adev,
goto out;
}
+after_reset:
/* MES doesn't detect any hung queue but we have a known bad queue
* and it is not KCQ
*/
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 86b70748617b..b77cfd8ba907 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -471,11 +471,6 @@ static int reset_queues_mes(struct device_queue_manager *dqm, struct queue *q)
if (WARN_ON(dqm->detect_hang_count > 0))
return 0;
- if (!amdgpu_gpu_recovery) {
- r = -ENOTRECOVERABLE;
- goto fail;
- }
-
memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));
queue_input.doorbell_offset = q->properties.doorbell_off;
queue_input.gang_context_addr = q->gang_ctx_gpu_addr;
@@ -3609,7 +3604,9 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel
list_for_each_entry(q, &qpd->queues_list, list) {
if (q->doorbell_id == doorbell_id && q->properties.is_active) {
- reset_queues_mes(dqm, q);
+ if (reset_queues_mes(dqm, q))
+ dev_dbg(knode->adev->dev,
+ "Fail to remove bad queue. doorbell_id 0x%x\n", q->doorbell_id);
q->properties.is_evicted = true;
q->properties.is_active = false;
decrement_queue_count(dqm, qpd, q);
--
2.43.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH v2] drm/amdkfd: Move gpu_recovery location
2026-11-16 7:50 [PATCH v2] drm/amdkfd: Move gpu_recovery location Amber Lin
@ 2026-11-16 8:13 ` sashiko-bot
0 siblings, 0 replies; 2+ messages in thread
From: sashiko-bot @ 2026-11-16 8:13 UTC (permalink / raw)
To: Amber Lin, amd-gfx
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] drm/amdgpu: broken !amdgpu_gpu_recovery flow in
amdgpu_gfx_reset_mes_compute()
--
commit 8a38bbe4db1b13fc0dffaa46e16598a2c6c11936
Author: Amber Lin <amber.lin@amd.com>
drm/amdkfd: Move gpu_recovery location
This commit moves the amdgpu_gpu_recovery check from reset_queues_mes()
into amdgpu_gfx_reset_mes_compute() so bad queue removal still runs when
GPU recovery is disabled.
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> index c5a2dcff5058a..c30fecc653b4b 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
> @@ -2594,14 +2594,17 @@ int amdgpu_gfx_reset_mes_compute(struct amdgpu_device *adev,
> * reset detect will return the array of bad queue doorbells
> */
> r = amdgpu_mes_suspend(adev, xcc_id);
> + dev_dbg(adev->dev, "Suspend all returns %d. xcc_id %d\n", r, xcc_id);
> +
> + if (!amdgpu_gpu_recovery)
> + goto after_reset;
> +
> /* if suspend all success, it should no hang queue */
> - if (!r) {
> - dev_dbg(adev->dev, "Suspend all succeeds. xcc_id %d\n", xcc_id);
> + if (!r)
> /* always reset the KCQ/userq since we need to signal the fence
> * and we could be stuck in a loop which is preemptable.
> */
> goto fence_reset;
> - }
> r = amdgpu_mes_detect_and_reset_hung_queues(adev, AMDGPU_RING_TYPE_COMPUTE,
> true, &num_hung, adev->gfx.mec.mes_hung_db_array, xcc_id);
> if (r) {
> @@ -2662,6 +2665,7 @@ int amdgpu_gfx_reset_mes_compute(struct amdgpu_device *adev,
> goto out;
> }
>
> +after_reset:
> /* MES doesn't detect any hung queue but we have a known bad queue
> * and it is not KCQ
> */
[Severity: High]
Can jumping to after_reset when !amdgpu_gpu_recovery leave all kernel
compute ring DRM schedulers permanently stopped when a queue hangs?
Before calling amdgpu_mes_suspend(), amdgpu_gfx_reset_mes_compute() stops
the compute ring schedulers via amdgpu_gfx_reset_stop_compute_scheds().
When a queue is hung, amdgpu_mes_suspend() fails with a non-zero r, but
if !amdgpu_gpu_recovery is set, this jumps straight to after_reset with
num_hung == 0:
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c:amdgpu_gfx_reset_mes_compute() {
...
after_reset:
if (!num_hung && queue_input && !ring) {
...
r = adev->mes.funcs->remove_hw_queue(&adev->mes, queue_input);
...
}
out:
amdgpu_mes_resume(adev, xcc_id);
ring_err = r;
...
if (!ring_err)
amdgpu_gfx_reset_start_compute_scheds(adev, ring);
...
}
Because suspend_all failed, the hung queue was not preempted and
remove_hw_queue() fails as well (or if another queue on the XCC caused
suspend_all to fail, remove_hw_queue() can succeed and overwrite the
non-zero suspend error r with 0). When r is non-zero, ring_err is
non-zero and amdgpu_gfx_reset_start_compute_scheds() is skipped; since no
full GPU reset follows when !amdgpu_gpu_recovery (or when called from
kfd_dqm_suspend_bad_queue_mes()), the compute schedulers are never
restarted.
Also, if amdgpu_gfx_reset_mes_compute() is called for a KCQ or KGD user
queue (where queue_input is NULL) and amdgpu_mes_suspend() succeeds,
jumping to after_reset skips fence_reset and returns 0 without resetting
the ring or user queue.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116075047.35150-1-amber.lin@amd.com?part=1
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-11-16 8:14 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 7:50 [PATCH v2] drm/amdkfd: Move gpu_recovery location Amber Lin
2026-11-16 8:13 ` sashiko-bot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox