* GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701
@ 2026-07-03 12:44 Bert Karwatzki
2026-07-05 0:35 ` [Re] " Bert Karwatzki
0 siblings, 1 reply; 4+ messages in thread
From: Bert Karwatzki @ 2026-07-03 12:44 UTC (permalink / raw)
To: Alex Deucher
Cc: Bert Karwatzki, linux-kernel, amd-gfx, linux-next, Jesse Zhang,
Amber Lin, Mario Limonciello
When running the ROCm hsa selftests on debian unstable (with ROCm packages from experimental)
with linux next-20260701, the GPU will reset with the following error message:
$ /usr/libexec/rocm/libhsa-runtime64-tests/run-tests
[ T2548] amdgpu 0000:03:00.0: failed to suspend gangs from MES
[ T2548] amdgpu 0000:03:00.0: MES might be in unrecoverable state, issue a GPU reset
[ T360] amdgpu 0000:03:00.0: GPU reset begin!. Source: 3
[T28252] amdgpu 0000:03:00.0: Failed to evict queue 0
[T28252] amdgpu: Failed to quiesce KFD
[ T360] amdgpu 0000:03:00.0: Dumping IP State
[ T360] amdgpu 0000:03:00.0: Dumping IP State Completed
[ T360] amdgpu 0000:03:00.0: MODE1 reset
[ T360] amdgpu 0000:03:00.0: GPU mode1 reset
[ T360] amdgpu 0000:03:00.0: GPU smu mode1 reset
[ T360] amdgpu 0000:03:00.0: GPU reset succeeded, trying to resume
[drm] PCIE GART of 512M enabled (table at 0x00000083DAB00000).
[drm] AMDGPU device coredump file has been created
[drm] Check your /sys/class/drm/card0/device/devcoredump/data
[ T360] amdgpu 0000:03:00.0: VRAM is lost due to GPU reset!
[ T360] amdgpu 0000:03:00.0: PSP is resuming...
[ T360] amdgpu 0000:03:00.0: RAS: optional ras ta ucode is not available
[ T360] amdgpu 0000:03:00.0: RAP: optional rap ta ucode is not available
[ T360] amdgpu 0000:03:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
[ T360] amdgpu 0000:03:00.0: SMU is resuming...
[ T360] amdgpu 0000:03:00.0: SMU is resumed successfully!
[ T360] amdgpu 0000:03:00.0: program CP_MES_CNTL : 0x4000000
[ T360] amdgpu 0000:03:00.0: program CP_MES_CNTL : 0xc000000
[drm] DMUB hardware initialized: version=0x00010300
[ T360] amdgpu 0000:03:00.0: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ T360] amdgpu 0000:03:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ T360] amdgpu 0000:03:00.0: ring sdma0 uses VM inv eng 9 on hub 0
[ T360] amdgpu 0000:03:00.0: ring sdma1 uses VM inv eng 10 on hub 0
[ T360] amdgpu 0000:03:00.0: ring vcn_unified_0 uses VM inv eng 0 on hub 8
[ T360] amdgpu 0000:03:00.0: ring jpeg_dec uses VM inv eng 1 on hub 8
[ T360] amdgpu 0000:03:00.0: GPU reset(1) succeeded!
[drm] device wedged, but no recovery needed
[ T2050] amdgpu 0000:03:00.0: VM memory stats for proc xfwm4(2105) task xfwm4:cs0(2050) is non-zero when fini
[ T1932] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T2293] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T1929] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T1930] amdgpu 0000:03:00.0: VM memory stats for proc (0) task (0) is non-zero when fini
[ T2263] amdgpu 0000:03:00.0: VM memory stats for proc firefox-esr(3483) task firefox-es:cs0(2263) is non-zero when fini
[ T3590] amdgpu 0000:03:00.0: VM memory stats for proc RDD Process(4755) task firefox-es:cs0(3590) is non-zero when fini
[ T1469] amdgpu 0000:03:00.0: VM memory stats for proc Xorg(1495) task Xorg:cs0(1469) is non-zero when fini
As this does not occur on v7.2-rc1 I bisected the issue and found
commit f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
as responsible.
Unfortunately the does not revert cleanly in linux-next-20260701, so I had to revert these
commits to fix the issue in next-20260701:
b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes")
a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset")
a4e4d945cba8 ("drm/amdgpu/gfx: defer per-queue helper_end until after MES resume")
f401a2633e02 ("drm/amdgpu: Remove faulty queue before resume")
f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset")
GPU used:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 44 [Radeon RX 9060 XT] [1002:7590] (rev c0)
CPU used:
model name : AMD Ryzen 9 9950X 16-Core Processor
Installed ROCm packages (from debian experimental)
bert@homer:~$ dpkg -l | grep 7.2.4
ii hip-utils 7.2.4-1~exp1 amd64 utilities for HIP language development
ii hipcc 7.2.4+dfsg-1~exp1 amd64 C++ Runtime API and Kernel Language for AMD and NVIDIA GPUs
ii libamd-comgr-dev 7.2.4+dfsg-1~exp1 amd64 ROCm code object manager - development package
ii libamd-comgr3:amd64 7.2.4+dfsg-1~exp1 amd64 ROCm code object manager
ii libamdhip64-7:amd64 7.2.4-1~exp1 amd64 HIP runtime for AMD GPUs - library
ii libamdhip64-dev 7.2.4-1~exp1 amd64 HIP runtime for AMD GPUs - headers
ii libhiprtc-builtins7:amd64 7.2.4-1~exp1 amd64 HIP runtime compilation library - builtins
ii libhiprtc7:amd64 7.2.4-1~exp1 amd64 HIP runtime compilation library
ii libhsa-runtime-dev:amd64 7.2.4+dfsg-1~exp1 amd64 HSA Runtime API and runtime for ROCm - development files
ii libhsa-runtime64-1:amd64 7.2.4+dfsg-1~exp1 amd64 HSA Runtime API and runtime for ROCm
ii libhsa-runtime64-tests 7.2.4+dfsg-1~exp1 amd64 HSA Runtime test suites for ROCm
ii librocblas-dev 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - headers
ii librocblas-doc 7.2.4-1~exp1 all ROCm library for basic linear algebra - documentation
ii librocblas5 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - library
ii librocblas5-bench 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - benchmarks
ii librocblas5-tests 7.2.4-1~exp1 amd64 ROCm library for basic linear algebra - tests
ii librocblas5-tests-data 7.2.4-1~exp1 all ROCm library for basic linear algebra - test data
ii librocm-core-dev 7.2.4-1~exp1 amd64 provides methods to get information about installed ROCm - headers
ii librocm-core1 7.2.4-1~exp1 amd64 provides methods to get information about installed ROCm
ii librocm-smi64-7 7.2.4-1~exp1 amd64 ROCm System Management Interface (ROCm SMI) library
ii librocsolver-dev 7.2.4-1~exp1 amd64 ROCm library for numerical linear algebra - headers
ii librocsolver0 7.2.4-1~exp1 amd64 ROCm library for numerical linear algebra - library
ii librocsolver0-tests 7.2.4-1~exp1 amd64 ROCm library for numerical linear algebra - tests
ii libroctx64-4 7.2.4-1~exp1 amd64 ROCm library for profiler annotations - library
ii rocm-device-libs-22 7.2.4+dfsg-1~exp1 amd64 AMD specific device-side language runtime libraries
ii rocm-opencl-icd:amd64 7.2.4-1~exp1 amd64 AMD ROCm implementation of the OpenCL API - ICD runtime
ii rocminfo 7.2.4-1~exp1 amd64 ROCm Application for Reporting System Info
Bert Karwatzki
^ permalink raw reply [flat|nested] 4+ messages in thread* [Re] GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701 2026-07-03 12:44 GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701 Bert Karwatzki @ 2026-07-05 0:35 ` Bert Karwatzki 2026-07-05 22:45 ` Bert Karwatzki 0 siblings, 1 reply; 4+ messages in thread From: Bert Karwatzki @ 2026-07-05 0:35 UTC (permalink / raw) To: Alex Deucher Cc: Bert Karwatzki, linux-kernel, amd-gfx, linux-next, Jesse Zhang, Amber Lin, Mario Limonciello I identified the problem in f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset") it's reset_queues_mes() being called unconditionally. f94bbd648bb4 can be fixed like this: diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index 6054c8e216b8..ed1ffa8b1743 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -484,15 +484,18 @@ static int suspend_all_queues_mes(struct device_queue_manager *dqm) if (!down_read_trylock(&adev->reset_domain->sem)) return -EIO; + r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1); - if (!reset_queues_mes(dqm)) { - r = 0; - goto out; - } + if (r) { + if (!reset_queues_mes(dqm)) { + r = 0; + goto out; + } - dev_err(adev->dev, "failed to suspend gangs from MES\n"); - dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n"); - kfd_hws_hang(dqm); + dev_err(adev->dev, "failed to suspend gangs from MES\n"); + dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n"); + kfd_hws_hang(dqm); + } out: up_read(&adev->reset_domain->sem); But when I try to fix next-20260701 like this diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index 5c9dfb0c424f..5a78b1504f8c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -493,7 +493,10 @@ static int recover_bad_queue_mes(struct device_queue_manager *dqm, struct queue if (!down_read_trylock(&adev->reset_domain->sem)) return -EIO; - r = reset_queues_mes(dqm, q); + r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1); + + if (r) + r = reset_queues_mes(dqm, q); up_read(&adev->reset_domain->sem); return r; I still get GPUVM errors (but no GPU reset) and when running $ /usr/libexec/rocm/libhsa-runtime64-tests/run-tests [ 146.577245] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0) [ 146.577247] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10 [ 146.577248] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A [ 146.577249] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5) [ 146.577249] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0 [ 146.577250] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5 [ 146.577250] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3 [ 146.577250] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1 [ 146.577251] [ T418] amdgpu 0000:03:00.0: RW: 0x0 [ 146.577606] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0) [ 146.577608] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10 [ 146.577609] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A [ 146.577609] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5) [ 146.577610] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0 [ 146.577611] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5 [ 146.577611] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3 [ 146.577611] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1 [ 146.577612] [ T418] amdgpu 0000:03:00.0: RW: 0x0 This means there's at least another error in these commits (reverting all these in next-20260701 fixes the issue) b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes") a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset") a4e4d945cba8 ("drm/amdgpu/gfx: defer per-queue helper_end until after MES resume") f401a2633e02 ("drm/amdgpu: Remove faulty queue before resume") f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset") Bert Karwatzki ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Re] GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701 2026-07-05 0:35 ` [Re] " Bert Karwatzki @ 2026-07-05 22:45 ` Bert Karwatzki 2026-07-06 12:46 ` Bert Karwatzki 0 siblings, 1 reply; 4+ messages in thread From: Bert Karwatzki @ 2026-07-05 22:45 UTC (permalink / raw) To: Alex Deucher Cc: linux-kernel, amd-gfx, linux-next, Jesse Zhang, Amber Lin, Mario Limonciello, spasswolf Am Sonntag, dem 05.07.2026 um 02:35 +0200 schrieb Bert Karwatzki: > I identified the problem in > f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset") > it's reset_queues_mes() being called unconditionally. f94bbd648bb4 can be fixed > like this: > > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > index 6054c8e216b8..ed1ffa8b1743 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > @@ -484,15 +484,18 @@ static int suspend_all_queues_mes(struct device_queue_manager *dqm) > if (!down_read_trylock(&adev->reset_domain->sem)) > return -EIO; > > + r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1); > > - if (!reset_queues_mes(dqm)) { > - r = 0; > - goto out; > - } > + if (r) { > + if (!reset_queues_mes(dqm)) { > + r = 0; > + goto out; > + } > > - dev_err(adev->dev, "failed to suspend gangs from MES\n"); > - dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n"); > - kfd_hws_hang(dqm); > + dev_err(adev->dev, "failed to suspend gangs from MES\n"); > + dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n"); > + kfd_hws_hang(dqm); > + } > out: > > up_read(&adev->reset_domain->sem); > > > But when I try to fix next-20260701 like this > > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > index 5c9dfb0c424f..5a78b1504f8c 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c > @@ -493,7 +493,10 @@ static int recover_bad_queue_mes(struct device_queue_manager *dqm, struct queue > if (!down_read_trylock(&adev->reset_domain->sem)) > return -EIO; > > - r = reset_queues_mes(dqm, q); > + r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1); > + > + if (r) > + r = reset_queues_mes(dqm, q); > > up_read(&adev->reset_domain->sem); > return r; > > I still get GPUVM errors (but no GPU reset) and when running > > $ /usr/libexec/rocm/libhsa-runtime64-tests/run-tests > > [ 146.577245] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0) > [ 146.577247] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10 > [ 146.577248] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A > [ 146.577249] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5) > [ 146.577249] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0 > [ 146.577250] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5 > [ 146.577250] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3 > [ 146.577250] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1 > [ 146.577251] [ T418] amdgpu 0000:03:00.0: RW: 0x0 > [ 146.577606] [ T418] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:157 vmid:0 pasid:0) > [ 146.577608] [ T418] amdgpu 0000:03:00.0: in page starting at address 0x00000000002ba000 from client 10 > [ 146.577609] [ T418] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00000B3A > [ 146.577609] [ T418] amdgpu 0000:03:00.0: Faulty UTCL2 client ID: CPC (0x5) > [ 146.577610] [ T418] amdgpu 0000:03:00.0: MORE_FAULTS: 0x0 > [ 146.577611] [ T418] amdgpu 0000:03:00.0: WALKER_ERROR: 0x5 > [ 146.577611] [ T418] amdgpu 0000:03:00.0: PERMISSION_FAULTS: 0x3 > [ 146.577611] [ T418] amdgpu 0000:03:00.0: MAPPING_ERROR: 0x1 > [ 146.577612] [ T418] amdgpu 0000:03:00.0: RW: 0x0 > > This means there's at least another error in these commits (reverting all these in > next-20260701 fixes the issue) > > b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes") > a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset") > a4e4d945cba8 ("drm/amdgpu/gfx: defer per-queue helper_end until after MES resume") > f401a2633e02 ("drm/amdgpu: Remove faulty queue before resume") > f94bbd648bb4 ("drm/amdgpu: use a single entry point for mes compute reset") > > Bert Karwatzki These GPUVM error are introduce by a665d09b10af ("drm/amdkfd: Pass known bad queue info to reset") and can be fixed by a partial revert. a665d09b10af can be fixed by diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index 0d95dd941129..0c49fe95a75d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -491,15 +491,18 @@ static int suspend_all_queues_mes(struct device_queue_manager *dqm, struct queue if (!down_read_trylock(&adev->reset_domain->sem)) return -EIO; + r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1); - if (!reset_queues_mes(dqm, q)) { - r = 0; - goto out; - } + if (r) { + if (!reset_queues_mes(dqm, q)) { + r = 0; + goto out; + } - dev_err(adev->dev, "failed to suspend gangs from MES\n"); - dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n"); - kfd_hws_hang(dqm); + dev_err(adev->dev, "failed to suspend gangs from MES\n"); + dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n"); + kfd_hws_hang(dqm); + } out: up_read(&adev->reset_domain->sem); @@ -3239,6 +3242,7 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel struct kfd_process_device *pdd = NULL; struct kfd_process *p = kfd_lookup_process_by_pasid(pasid, &pdd); struct device_queue_manager *dqm = knode->dqm; + struct device *dev = dqm->dev->adev->dev; struct qcm_process_device *qpd; struct queue *q = NULL; int ret = 0; @@ -3260,6 +3264,12 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel q->properties.is_active = false; decrement_queue_count(dqm, qpd, q); + /* this will remove the bad queue and sched a GPU reset if needed */ + ret = remove_queue_mes(dqm, q, qpd); + if (ret) + dev_err(dev, "Removing bad queue failed"); + /* resume the good queues */ + resume_all_queues_mes(dqm); break; } } This does not work in next-20260701 as commit b789664e3e30 ("drm/amdkfd: Clean up suspend_all and resume_all mes") removes resume_all_queues_mes(). Bert Karwatzki ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Re] GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701 2026-07-05 22:45 ` Bert Karwatzki @ 2026-07-06 12:46 ` Bert Karwatzki 0 siblings, 0 replies; 4+ messages in thread From: Bert Karwatzki @ 2026-07-06 12:46 UTC (permalink / raw) To: Alex Deucher Cc: linux-kernel, amd-gfx, linux-next, Jesse Zhang, Amber Lin, Mario Limonciello, spasswolf I found the real cause of my problems: bool amdgpu_mes_queue_reset_by_mes_supported(struct amdgpu_device *adev) { u32 ip_maj = IP_VERSION_MAJ(amdgpu_ip_version(adev, GC_HWIP, 0)); u32 ip_min = IP_VERSION_MIN(amdgpu_ip_version(adev, GC_HWIP, 0)); u32 mes_sched = adev->mes.sched_version & AMDGPU_MES_VERSION_MASK; printk(KERN_INFO "%s: ip_maj = %u ip_min = %u mes_sched = 0x%x", __func__, ip_maj, ip_min, mes_sched); return (ip_maj == 11 && mes_sched >= 0x8c) || ((ip_maj == 12 && ip_min == 0) && mes_sched >= 0x8d) || ((ip_maj == 12 && ip_min == 1) && mes_sched >= 0x73); } returns false on my machine (because mes_sched is not large enough) [ T8549] amdgpu_mes_queue_reset_by_mes_supported: ip_maj = 12 ip_min = 0 mes_sched = 0x76 So I skipped the call to amdgpu_mes_queue_reset_by_mes_supported() diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index 5c9dfb0c424f..462f20aeb681 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -448,17 +448,22 @@ static int reset_queues_mes(struct device_queue_manager *dqm, struct queue *q) unsigned int num_hung = 0; int r = 0; struct mes_remove_queue_input queue_input; + printk(KERN_INFO "entering %s", __func__); + /* if (!amdgpu_mes_queue_reset_by_mes_supported(adev)) { r = -ENOTRECOVERABLE; + printk(KERN_INFO "%s: reset by mes not supported", __func__); goto fail; - } + }*/ + printk(KERN_INFO "%s: skip calling amdgpu_mes_queue_reset_by_mes_supported()", __func__); /* reset should be used only in dqm locked queue reset */ if (WARN_ON(dqm->detect_hang_count > 0)) return 0; if (!amdgpu_gpu_recovery) { + printk(KERN_INFO "%s: gpu recovery not enabled", __func__); r = -ENOTRECOVERABLE; goto fail; } @@ -470,6 +475,7 @@ static int reset_queues_mes(struct device_queue_manager *dqm, struct queue *q) queue_input.xcc_id = ffs(dqm->dev->xcc_mask) - 1; /* pass the known bad queue info to the reset function */ r = amdgpu_gfx_reset_mes_compute(adev, NULL, NULL, NULL, &num_hung, &queue_input); + printk(KERN_INFO "%s: amdgpu_gfx_reset_mes_compute() returned %d", __func__, r); if (r) goto fail; @@ -3231,6 +3237,7 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel struct qcm_process_device *qpd; struct queue *q = NULL; int ret = 0; + printk(KERN_INFO "entering %s", __func__); if (!pdd) return -EINVAL; @@ -3242,6 +3249,7 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel list_for_each_entry(q, &qpd->queues_list, list) { if (q->doorbell_id == doorbell_id && q->properties.is_active) { + printk(KERN_INFO "%s: calling recover_bad_queue_mes() for queue %px", __func__, q); and got this output when running the hsaruntime (which inject illegal opcodes into the command stream) test: [ 113.811612] [ T645] [drm:gfx_v12_0_bad_op_irq [amdgpu]] *ERROR* Illegal opcode in command stream [ 113.811675] [ T2558] entering kfd_dqm_suspend_bad_queue_mes [ 113.811676] [ T2558] kfd_dqm_suspend_bad_queue_mes: calling recover_bad_queue_mes() for queue ffffa03160950400 [ 113.811676] [ T2558] entering reset_queues_mes [ 113.811677] [ T2558] reset_queues_mes: skip calling amdgpu_mes_queue_reset_by_mes_supported() [ 113.811887] [ T2558] reset_queues_mes: amdgpu_gfx_reset_mes_compute() returned 0 So even though amdgpu_mes_queue_reset_by_mes_supported() reported false, amdgpu_gfx_reset_mes_compute() returns 0, suggesting that resetting actually works here! Perhaps the minimum required mes_sched version for 12.0 can be relaxed to 0x76 to solve this: From 6d7af652177063963012eb4df228e99caeb03b31 Mon Sep 17 00:00:00 2001 From: Bert Karwatzki <spasswolf@web.de> Date: Mon, 6 Jul 2026 14:36:22 +0200 Subject: [PATCH] amdgpu: relax required mes_sched version This mes_sched version is actually enough on this hardware: 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 44 [Radeon RX 9060 XT] [1002:7590] (rev c0) [ 3.200538] [ T537] amdgpu 0000:03:00.0: initializing kernel modesetting (IP DISCOVERY 0x1002:0x7590 0x148C:0x2437 0xC0). [ 3.200546] [ T537] amdgpu 0000:03:00.0: register mmio base: 0xDFC00000 [ 3.200547] [ T537] amdgpu 0000:03:00.0: register mmio size: 524288 [ 3.203739] [ T537] amdgpu 0000:03:00.0: detected ip block number 0 <common_v1_0_0> (soc24_common) [ 3.203740] [ T537] amdgpu 0000:03:00.0: detected ip block number 1 <gmc_v12_0_0> (gmc_v12_0) [ 3.203741] [ T537] amdgpu 0000:03:00.0: detected ip block number 2 <ih_v7_0_0> (ih_v7_0) [ 3.203741] [ T537] amdgpu 0000:03:00.0: detected ip block number 3 <psp_v14_0_0> (psp) [ 3.203742] [ T537] amdgpu 0000:03:00.0: detected ip block number 4 <smu_v14_0_0> (smu) [ 3.203742] [ T537] amdgpu 0000:03:00.0: detected ip block number 5 <dce_v1_0_0> (dm) [ 3.203743] [ T537] amdgpu 0000:03:00.0: detected ip block number 6 <gfx_v12_0_0> (gfx_v12_0) [ 3.203743] [ T537] amdgpu 0000:03:00.0: detected ip block number 7 <sdma_v7_0_0> (sdma_v7_0) [ 3.203744] [ T537] amdgpu 0000:03:00.0: detected ip block number 8 <vcn_v5_0_0> (vcn_v5_0_0) [ 3.203744] [ T537] amdgpu 0000:03:00.0: detected ip block number 9 <jpeg_v5_0_0> (jpeg_v5_0_0) [ 3.203745] [ T537] amdgpu 0000:03:00.0: detected ip block number 10 <mes_v12_0_0> (mes_v12_0) Signed-off-by: Bert Karwatzki <spasswolf@web.de> --- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c index 6c0dde3786e3..c88fdc8a187d 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c @@ -869,7 +869,7 @@ bool amdgpu_mes_queue_reset_by_mes_supported(struct amdgpu_device *adev) u32 mes_sched = adev->mes.sched_version & AMDGPU_MES_VERSION_MASK; return (ip_maj == 11 && mes_sched >= 0x8c) || - ((ip_maj == 12 && ip_min == 0) && mes_sched >= 0x8d) || + ((ip_maj == 12 && ip_min == 0) && mes_sched >= 0x76) || ((ip_maj == 12 && ip_min == 1) && mes_sched >= 0x73); } -- 2.53.0 Bert Karwatzki ^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-07-06 12:47 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-03 12:44 GPU reset when running the ROCm hsa runtime tests on gfx12 and next-20260701 Bert Karwatzki 2026-07-05 0:35 ` [Re] " Bert Karwatzki 2026-07-05 22:45 ` Bert Karwatzki 2026-07-06 12:46 ` Bert Karwatzki
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox