* GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701
@ 2026-07-03 12:44 Bert Karwatzki
2026-07-05 0:35 ` [Re] " Bert Karwatzki
0 siblings, 1 reply; 4+ messages in thread
From: Bert Karwatzki @ 2026-07-03 12:44 UTC (permalink / raw)
To: Alex Deucher
Cc: Bert Karwatzki, linux-kernel, amd-gfx, linux-next, Jesse Zhang,
Amber Lin, Mario Limonciello
When running the ROCm hsa selftests on debian unstable (with ROCm packages from experimental)
with linux next-20260701, the GPU will reset with the following error message:
$ /usr/libexec/rocm/libhsa-runtime64-tests/run-tests
[ T2548] amdgpu 0000:03:00.0: failed to suspend gangs from MES
[ T2548] amdgpu 0000:03:00.0: MES might be in unrecoverable state, issue a GPU reset
[ T360] amdgpu 0000:03:00.0: GPU reset begin!. Source: 3
[T28252] amdgpu 0000:03:00.0: Failed to evict queue 0
[T28252] amdgpu: Failed to quiesce KFD
[ T360] amdgpu 0000:03:00.0: Dumping IP State
[ T360] amdgpu 0000:03:00.0: Dumping IP State Completed
[ T360] amdgpu 0000:03:00.0: MODE1 reset
[ T360] amdgpu 0000:03:00.0: GPU mode1 reset
[ T360] amdgpu 0000:03:00.0: GPU smu mode1 reset
[ T360] amdgpu 0000:03:00.0: GPU reset succeeded, trying to resume
[drm] PCIE GART of 512M enabled (table at 0x00000083DAB00000).
[drm] AMDGPU device coredump file has been created
[drm] Check your /sys/class/drm/card0/device/devcoredump/data
[ T360] amdgpu 0000:03:00.0: VRAM is lost due to GPU reset!
[ T360] amdgpu 0000:03:00.0: PSP is resuming...
[ T360] amdgpu 0000:03:00.0: RAS: optional ras ta ucode is not available
[ T360] amdgpu 0000:03:00.0: RAP: optional rap ta ucode is not available
[ T360] amdgpu 0000:03:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
[ T360] amdgpu 0000:03:00.0: SMU is resuming...
[ T360] amdgpu 0000:03:00.0: SMU is resumed successfully!
[ T360] amdgpu 0000:03:00.0: program CP_MES_CNTL : 0x4000000
[ T360] amdgpu 0000:03:00.0: program CP_MES_CNTL : 0xc000000
[drm] DMUB hardware initialized: version=0x00010300
[ T360] amdgpu 0000:03:00.0: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ T360] amdgpu 0000:03:00.0: ring sdma0 uses VM inv eng 9 on hub 0
[ T360] amdgpu 0000:03:00.0: ring sdma1 uses VM inv eng 10 on hub 0
[ T360] amdgpu 0000:03:00.0: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[ T360] amdgpu 0000:03:00.0: ring jpeg_dec uses VM inv eng 1 on hub 8
[ T360] amdgpu 0000:03:00.0: GPU reset(1) succeeded!
[drm] device wedged, but no recovery needed
[ T2050] amdgpu 0000:03:00.0: VM memory stats for proc xfwm4(2105) task xfwm4:cs0(2050) is non-zero when fini
[ T1932] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T2293] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T1929] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T1930] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T2263] amdgpu 0000:03:00.0: VM memory stats for proc firefox-esr(3483) task firefox-es:cs0(2263) is non-zero when fini
[ T3590] amdgpu 0000:03:00.0: VM memory stats for proc RDD Process(4755) task firefox-es:cs0(3590) is non-zero when fini
[ T1469] amdgpu 0000:03:00.0: VM memory stats for proc Xorg(1495) task Xorg:cs0(1469) is non-zero when fini
As this does not occur on v7.2-rc1 I bisected the issue and found
commit f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
as responsible.
Unfortunately the does not revert cleanly in linux-next-20260701, so I had to revert these
commits to fix the issue in next-20260701:
b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes")
a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset")
a4e4d945cba8 ("drm/amdgpu/gfx: defer per-queue helper_end until after MES resume")
f401a2633e02 ("drm/amdgpu: Remove faulty queue before resume")
f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
GPU used:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 44 [Radeon RX 9060 XT] [1002:7590] (rev c0)
CPU used:
model name : AMD Ryzen 9 9950X 16-Core Processor
Installed ROCm packages (from debian experimental)
bert@homer:~$ dpkg -l | grep 7.2.4
ii hip-utils 7.2.4-1~exp1 amd64 utilities for HIP language development
ii hipcc 7.2.4+dfsg-1~exp1 amd64 C++ Runtime API and Kernel Language for AMD and NVIDIA GPUs
ii libamd-comgr-dev 7.2.4+dfsg-1~exp1 amd64 ROCm code object manager - development package
ii libamd-comgr3:amd64 7.2.4+dfsg-1~exp1 amd64 ROCm code object manager
ii libamdhip64-7:amd64 7.2.4-1~exp1 amd64 HIP runtime for AMD GPUs - library
ii libamdhip64-dev 7.2.4-1~exp1 amd64 HIP runtime for AMD GPUs - headers
ii libhiprtc-builtins7:amd64 7.2.4-1~exp1 amd64 HIP runtime compilation library - builtins
ii libhiprtc7:amd64 7.2.4-1~exp1 amd64 HIP runtime compilation library
ii libhsa-runtime-dev:amd64 7.2.4+dfsg-1~exp1 amd64 HSA Runtime API and runtime for ROCm - development files
ii libhsa-runtime64-1:amd64 7.2.4+dfsg-1~exp1 amd64 HSA Runtime API and runtime for ROCm
ii libhsa-runtime64-tests 7.2.4+dfsg-1~exp1 amd64 HSA Runtime test suites for ROCm
ii librocblas-dev 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - headers
ii librocblas-doc 7.2.4-1~exp1 all ROCm library for basic linear algebra - documentation
ii librocblas5 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - library
ii librocblas5-bench 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - benchmarks
ii librocblas5-tests 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - tests
ii librocblas5-tests-data 7.2.4-1~exp1 all ROCm library for basic linear algebra - test data
ii librocm-core-dev 7.2.4-1~exp1 amd64 provides methods to get information about installed ROCm - headers
ii librocm-core1 7.2.4-1~exp1 amd64 provides methods to get information about installed ROCm
ii librocm-smi64-7 7.2.4-1~exp1 amd64 ROCm System Management Interface (ROCm SMI) library
ii librocsolver-dev 7.2.4-1~exp1 amd64 ROCm library for numerical linear algebra - headers
ii librocsolver0 7.2.4-1~exp1 amd64 ROCm library for numerical linear algebra - library
ii librocsolver0-tests 7.2.4-1~exp1 amd64 ROCm library for numerical linear algebra - tests
ii libroctx64-4 7.2.4-1~exp1 amd64 ROCm library for profiler annotations - library
ii rocm-device-libs-22 7.2.4+dfsg-1~exp1 amd64 AMD specific device-side language runtime libraries
ii rocm-opencl-icd:amd64 7.2.4-1~exp1 amd64 AMD ROCm implementation of the OpenCL API - ICD runtime
ii rocminfo 7.2.4-1~exp1 amd64 ROCm Application for Reporting System Info
Bert Karwatzki
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Re] GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701
2026-07-03 12:44 GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701 Bert Karwatzki
@ 2026-07-05 0:35 ` Bert Karwatzki
2026-07-05 22:45 ` Bert Karwatzki
0 siblings, 1 reply; 4+ messages in thread
From: Bert Karwatzki @ 2026-07-05 0:35 UTC (permalink / raw)
To: Alex Deucher
Cc: Bert Karwatzki, linux-kernel, amd-gfx, linux-next, Jesse Zhang,
Amber Lin, Mario Limonciello
I identified the problem in
f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
it's reset_queues_mes() being called unconditionally. f94bbd648bb4 can be fixed
like this:
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 6054c8e216b8..ed1ffa8b1743 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -484,15 +484,18 @@ static int suspend_all_queues_mes(struct device_queue_manager *dqm)
if (!down_read_trylock(&adev->reset_domain->sem))
return -EIO;
+ r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1);
- if (!reset_queues_mes(dqm)) {
- r = 0;
- goto out;
- }
+ if (r) {
+ if (!reset_queues_mes(dqm)) {
+ r = 0;
+ goto out;
+ }
- dev_err(adev->dev, "failed to suspend gangs from MES\n");
- dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
- kfd_hws_hang(dqm);
+ dev_err(adev->dev, "failed to suspend gangs from MES\n");
+ dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
+ kfd_hws_hang(dqm);
+ }
out:
up_read(&adev->reset_domain->sem);
But when I try to fix next-20260701 like this
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 5c9dfb0c424f..5a78b1504f8c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -493,7 +493,10 @@ static int recover_bad_queue_mes(struct device_queue_manager *dqm, struct queue
if (!down_read_trylock(&adev->reset_domain->sem))
return -EIO;
- r = reset_queues_mes(dqm, q);
+ r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1);
+
+ if (r)
+ r = reset_queues_mes(dqm, q);
up_read(&adev->reset_domain->sem);
return r;
I still get GPUVM errors (but no GPU reset) and when running
$ /usr/libexec/rocm/libhsa-runtime64-tests/run-tests
[ 146.577245] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0)
[ 146.577247] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10
[ 146.577248] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A
[ 146.577249] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5)
[ 146.577249] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0
[ 146.577250] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5
[ 146.577250] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3
[ 146.577250] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1
[ 146.577251] [ T418] amdgpu 0000:03:00.0: RW: 0x0
[ 146.577606] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0)
[ 146.577608] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10
[ 146.577609] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A
[ 146.577609] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5)
[ 146.577610] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0
[ 146.577611] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5
[ 146.577611] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3
[ 146.577611] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1
[ 146.577612] [ T418] amdgpu 0000:03:00.0: RW: 0x0
This means there's at least another error in these commits (reverting all these in
next-20260701 fixes the issue)
b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes")
a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset")
a4e4d945cba8 ("drm/amdgpu/gfx: defer per-queue helper_end until after MES resume")
f401a2633e02 ("drm/amdgpu: Remove faulty queue before resume")
f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
Bert Karwatzki
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Re] GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701
2026-07-05 0:35 ` [Re] " Bert Karwatzki
@ 2026-07-05 22:45 ` Bert Karwatzki
2026-07-06 12:46 ` Bert Karwatzki
0 siblings, 1 reply; 4+ messages in thread
From: Bert Karwatzki @ 2026-07-05 22:45 UTC (permalink / raw)
To: Alex Deucher
Cc: linux-kernel, amd-gfx, linux-next, Jesse Zhang, Amber Lin,
Mario Limonciello, spasswolf
Am Sonntag, dem 05.07.2026 um 02:35 +0200 schrieb Bert Karwatzki:
> I identified the problem in
> f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
> it's reset_queues_mes() being called unconditionally. f94bbd648bb4 can be fixed
> like this:
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> index 6054c8e216b8..ed1ffa8b1743 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> @@ -484,15 +484,18 @@ static int suspend_all_queues_mes(struct device_queue_manager *dqm)
> if (!down_read_trylock(&adev->reset_domain->sem))
> return -EIO;
>
> + r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1);
>
> - if (!reset_queues_mes(dqm)) {
> - r = 0;
> - goto out;
> - }
> + if (r) {
> + if (!reset_queues_mes(dqm)) {
> + r = 0;
> + goto out;
> + }
>
> - dev_err(adev->dev, "failed to suspend gangs from MES\n");
> - dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
> - kfd_hws_hang(dqm);
> + dev_err(adev->dev, "failed to suspend gangs from MES\n");
> + dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
> + kfd_hws_hang(dqm);
> + }
> out:
>
> up_read(&adev->reset_domain->sem);
>
>
> But when I try to fix next-20260701 like this
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> index 5c9dfb0c424f..5a78b1504f8c 100644
> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
> @@ -493,7 +493,10 @@ static int recover_bad_queue_mes(struct device_queue_manager *dqm, struct queue
> if (!down_read_trylock(&adev->reset_domain->sem))
> return -EIO;
>
> - r = reset_queues_mes(dqm, q);
> + r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1);
> +
> + if (r)
> + r = reset_queues_mes(dqm, q);
>
> up_read(&adev->reset_domain->sem);
> return r;
>
> I still get GPUVM errors (but no GPU reset) and when running
>
> $ /usr/libexec/rocm/libhsa-runtime64-tests/run-tests
>
> [ 146.577245] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0)
> [ 146.577247] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10
> [ 146.577248] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A
> [ 146.577249] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5)
> [ 146.577249] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0
> [ 146.577250] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5
> [ 146.577250] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3
> [ 146.577250] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1
> [ 146.577251] [ T418] amdgpu 0000:03:00.0: RW: 0x0
> [ 146.577606] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0)
> [ 146.577608] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10
> [ 146.577609] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A
> [ 146.577609] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5)
> [ 146.577610] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0
> [ 146.577611] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5
> [ 146.577611] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3
> [ 146.577611] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1
> [ 146.577612] [ T418] amdgpu 0000:03:00.0: RW: 0x0
>
> This means there's at least another error in these commits (reverting all these in
> next-20260701 fixes the issue)
>
> b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes")
> a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset")
> a4e4d945cba8 ("drm/amdgpu/gfx: defer per-queue helper_end until after MES resume")
> f401a2633e02 ("drm/amdgpu: Remove faulty queue before resume")
> f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
>
> Bert Karwatzki
These GPUVM error are introduce by
a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset")
and can be fixed by a partial revert. a665d09b10af can be fixed by
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 0d95dd941129..0c49fe95a75d 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -491,15 +491,18 @@ static int suspend_all_queues_mes(struct device_queue_manager *dqm, struct queue
if (!down_read_trylock(&adev->reset_domain->sem))
return -EIO;
+ r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1);
- if (!reset_queues_mes(dqm, q)) {
- r = 0;
- goto out;
- }
+ if (r) {
+ if (!reset_queues_mes(dqm, q)) {
+ r = 0;
+ goto out;
+ }
- dev_err(adev->dev, "failed to suspend gangs from MES\n");
- dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
- kfd_hws_hang(dqm);
+ dev_err(adev->dev, "failed to suspend gangs from MES\n");
+ dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
+ kfd_hws_hang(dqm);
+ }
out:
up_read(&adev->reset_domain->sem);
@@ -3239,6 +3242,7 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel
struct kfd_process_device *pdd = NULL;
struct kfd_process *p = kfd_lookup_process_by_pasid(pasid, &pdd);
struct device_queue_manager *dqm = knode->dqm;
+ struct device *dev = dqm->dev->adev->dev;
struct qcm_process_device *qpd;
struct queue *q = NULL;
int ret = 0;
@@ -3260,6 +3264,12 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel
q->properties.is_active = false;
decrement_queue_count(dqm, qpd, q);
+ /* this will remove the bad queue and sched a GPU reset if needed */
+ ret = remove_queue_mes(dqm, q, qpd);
+ if (ret)
+ dev_err(dev, "Removing bad queue failed");
+ /* resume the good queues */
+ resume_all_queues_mes(dqm);
break;
}
}
This does not work in next-20260701 as commit
b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes")
removes resume_all_queues_mes().
Bert Karwatzki
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Re] GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701
2026-07-05 22:45 ` Bert Karwatzki
@ 2026-07-06 12:46 ` Bert Karwatzki
0 siblings, 0 replies; 4+ messages in thread
From: Bert Karwatzki @ 2026-07-06 12:46 UTC (permalink / raw)
To: Alex Deucher
Cc: linux-kernel, amd-gfx, linux-next, Jesse Zhang, Amber Lin,
Mario Limonciello, spasswolf
I found the real cause of my problems:
bool amdgpu_mes_queue_reset_by_mes_supported(struct amdgpu_device *adev)
{
u32 ip_maj = IP_VERSION_MAJ(amdgpu_ip_version(adev, GC_HWIP, 0));
u32 ip_min = IP_VERSION_MIN(amdgpu_ip_version(adev, GC_HWIP, 0));
u32 mes_sched = adev->mes.sched_version & AMDGPU_MES_VERSION_MASK;
printk(KERN_INFO "%s: ip_maj = %u ip_min = %u mes_sched = 0x%x", __func__, ip_maj, ip_min, mes_sched);
return (ip_maj == 11 && mes_sched >= 0x8c) ||
((ip_maj == 12 && ip_min == 0) && mes_sched >= 0x8d) ||
((ip_maj == 12 && ip_min == 1) && mes_sched >= 0x73);
}
returns false on my machine (because mes_sched is not large enough)
[ T8549] amdgpu_mes_queue_reset_by_mes_supported: ip_maj = 12 ip_min = 0 mes_sched = 0x76
So I skipped the call to amdgpu_mes_queue_reset_by_mes_supported()
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 5c9dfb0c424f..462f20aeb681 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -448,17 +448,22 @@ static int reset_queues_mes(struct device_queue_manager *dqm, struct queue *q)
unsigned int num_hung = 0;
int r = 0;
struct mes_remove_queue_input queue_input;
+ printk(KERN_INFO "entering %s", __func__);
+ /*
if (!amdgpu_mes_queue_reset_by_mes_supported(adev)) {
r = -ENOTRECOVERABLE;
+ printk(KERN_INFO "%s: reset by mes not supported", __func__);
goto fail;
- }
+ }*/
+ printk(KERN_INFO "%s: skip calling amdgpu_mes_queue_reset_by_mes_supported()", __func__);
/* reset should be used only in dqm locked queue reset */
if (WARN_ON(dqm->detect_hang_count > 0))
return 0;
if (!amdgpu_gpu_recovery) {
+ printk(KERN_INFO "%s: gpu recovery not enabled", __func__);
r = -ENOTRECOVERABLE;
goto fail;
}
@@ -470,6 +475,7 @@ static int reset_queues_mes(struct device_queue_manager *dqm, struct queue *q)
queue_input.xcc_id = ffs(dqm->dev->xcc_mask) - 1;
/* pass the known bad queue info to the reset function */
r = amdgpu_gfx_reset_mes_compute(adev, NULL, NULL, NULL, &num_hung, &queue_input);
+ printk(KERN_INFO "%s: amdgpu_gfx_reset_mes_compute() returned %d", __func__, r);
if (r)
goto fail;
@@ -3231,6 +3237,7 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel
struct qcm_process_device *qpd;
struct queue *q = NULL;
int ret = 0;
+ printk(KERN_INFO "entering %s", __func__);
if (!pdd)
return -EINVAL;
@@ -3242,6 +3249,7 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel
list_for_each_entry(q, &qpd->queues_list, list) {
if (q->doorbell_id == doorbell_id && q->properties.is_active) {
+ printk(KERN_INFO "%s: calling recover_bad_queue_mes() for queue %px", __func__, q);
and got this output when running the hsaruntime (which inject illegal opcodes into the
command stream) test:
[ 113.811612] [ T645] [drm:gfx_v12_0_bad_op_irq [amdgpu]] *ERROR* Illegal opcode in command stream
[ 113.811675] [ T2558] entering kfd_dqm_suspend_bad_queue_mes
[ 113.811676] [ T2558] kfd_dqm_suspend_bad_queue_mes: calling recover_bad_queue_mes() for queue ffffa03160950400
[ 113.811676] [ T2558] entering reset_queues_mes
[ 113.811677] [ T2558] reset_queues_mes: skip calling amdgpu_mes_queue_reset_by_mes_supported()
[ 113.811887] [ T2558] reset_queues_mes: amdgpu_gfx_reset_mes_compute() returned 0
So even though amdgpu_mes_queue_reset_by_mes_supported() reported false,
amdgpu_gfx_reset_mes_compute() returns 0, suggesting that resetting actually works here!
Perhaps the minimum required mes_sched version for 12.0 can be relaxed to 0x76 to solve this:
From 6d7af652177063963012eb4df228e99caeb03b31 Mon Sep 17 00:00:00 2001
From: Bert Karwatzki <spasswolf@web.de>
Date: Mon, 6 Jul 2026 14:36:22 +0200
Subject: [PATCH] amdgpu: relax required mes_sched version
This mes_sched version is actually enough on this hardware:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 44 [Radeon RX 9060 XT] [1002:7590] (rev c0)
[ 3.200538] [ T537] amdgpu 0000:03:00.0: initializing kernel modesetting (IP DISCOVERY 0x1002:0x7590 0x148C:0x2437 0xC0).
[ 3.200546] [ T537] amdgpu 0000:03:00.0: register mmio base: 0xDFC00000
[ 3.200547] [ T537] amdgpu 0000:03:00.0: register mmio size: 524288
[ 3.203739] [ T537] amdgpu 0000:03:00.0: detected ip block number 0 <common_v1_0_0> (soc24_common)
[ 3.203740] [ T537] amdgpu 0000:03:00.0: detected ip block number 1 <gmc_v12_0_0> (gmc_v12_0)
[ 3.203741] [ T537] amdgpu 0000:03:00.0: detected ip block number 2 <ih_v7_0_0> (ih_v7_0)
[ 3.203741] [ T537] amdgpu 0000:03:00.0: detected ip block number 3 <psp_v14_0_0> (psp)
[ 3.203742] [ T537] amdgpu 0000:03:00.0: detected ip block number 4 <smu_v14_0_0> (smu)
[ 3.203742] [ T537] amdgpu 0000:03:00.0: detected ip block number 5 <dce_v1_0_0> (dm)
[ 3.203743] [ T537] amdgpu 0000:03:00.0: detected ip block number 6 <gfx_v12_0_0> (gfx_v12_0)
[ 3.203743] [ T537] amdgpu 0000:03:00.0: detected ip block number 7 <sdma_v7_0_0> (sdma_v7_0)
[ 3.203744] [ T537] amdgpu 0000:03:00.0: detected ip block number 8 <vcn_v5_0_0> (vcn_v5_0_0)
[ 3.203744] [ T537] amdgpu 0000:03:00.0: detected ip block number 9 <jpeg_v5_0_0> (jpeg_v5_0_0)
[ 3.203745] [ T537] amdgpu 0000:03:00.0: detected ip block number 10 <mes_v12_0_0> (mes_v12_0)
Signed-off-by: Bert Karwatzki <spasswolf@web.de>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index 6c0dde3786e3..c88fdc8a187d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -869,7 +869,7 @@ bool amdgpu_mes_queue_reset_by_mes_supported(struct amdgpu_device *adev)
u32 mes_sched = adev->mes.sched_version & AMDGPU_MES_VERSION_MASK;
return (ip_maj == 11 && mes_sched >= 0x8c) ||
- ((ip_maj == 12 && ip_min == 0) && mes_sched >= 0x8d) ||
+ ((ip_maj == 12 && ip_min == 0) && mes_sched >= 0x76) ||
((ip_maj == 12 && ip_min == 1) && mes_sched >= 0x73);
}
--
2.53.0
Bert Karwatzki
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-07-06 12:47 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-03 12:44 GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701 Bert Karwatzki
2026-07-05 0:35 ` [Re] " Bert Karwatzki
2026-07-05 22:45 ` Bert Karwatzki
2026-07-06 12:46 ` Bert Karwatzki
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox