[PATCH] drm/amdgpu/userq: fix indefinite fence wait during GPU reset

[PATCH] drm/amdgpu/userq: fix indefinite fence wait during GPU reset

[Jesse Zhang]

pre_reset only force-completes fences of MAPPED queues. A queue in any
other state (e.g. mid-eviction) keeps its last_fence pending; after a
GPU reset that fence never signals, so the eviction/suspend worker and
process teardown (amdgpu_evf_mgr_flush_suspend) wait on it forever and
wedge the machine:

  INFO: task kworker/6:28 blocked for more than 120 seconds.
  Workqueue: events amdgpu_eviction_fence_suspend_worker [amdgpu]
  Call Trace:
   dma_fence_wait_timeout+0x7e/0x130
   amdgpu_userq_evict+0x67/0x140 [amdgpu]
   amdgpu_eviction_fence_suspend_worker+0xd8/0x160 [amdgpu]
   process_scheduled_works+0xa6/0x420

Force-complete every queue's fence regardless of state. The unmap and
mark-hung step stays gated on MAPPED, since unmapping a queue that is
not mapped is invalid.

Fixes: 290f46cf5726 ("drm/amdgpu: Implement user queue reset functionality")
Signed-off-by: Jesse Zhang <Jesse.Zhang@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 25 +++++++++++++----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
index aa5cc5642e87..6aa75da27f91 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
@@ -1424,18 +1424,21 @@ void amdgpu_userq_pre_reset(struct amdgpu_device *adev)
 
 	/* TODO: We probably need a new lock for the queue state */
 	xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
-		if (queue->state != AMDGPU_USERQ_STATE_MAPPED)
-			continue;
-
-		trace_amdgpu_userq_state_start(queue);
-		userq_funcs = adev->userq_funcs[queue->queue_type];
-		userq_funcs->unmap(queue);
-		/* just mark all queues as hung at this point.
-		 * if unmap succeeds, we could map again
-		 * in amdgpu_userq_post_reset() if vram is not lost
+		if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
+			trace_amdgpu_userq_state_start(queue);
+			userq_funcs = adev->userq_funcs[queue->queue_type];
+			userq_funcs->unmap(queue);
+			/* just mark all queues as hung at this point.
+			 * if unmap succeeds, we could map again
+			 * in amdgpu_userq_post_reset() if vram is not lost
+			 */
+			trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
+			queue->state = AMDGPU_USERQ_STATE_HUNG;
+		}
+		/* Force-complete any pending fence regardless of queue state so
+		 * that eviction/suspend and queue teardown waiters don't block
+		 * forever on a fence that will never signal after the reset.
 		 */
-		trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
-		queue->state = AMDGPU_USERQ_STATE_HUNG;
 		amdgpu_userq_fence_driver_force_completion(queue);
 	}
 }
-- 
2.49.0

Re: [PATCH] drm/amdgpu/userq: fix indefinite fence wait during GPU reset

[Christian König]

On 7/10/26 12:01, Jesse Zhang wrote:
> pre_reset only force-completes fences of MAPPED queues. A queue in any
> other state (e.g. mid-eviction) keeps its last_fence pending; after a
> GPU reset that fence never signals, so the eviction/suspend worker and
> process teardown (amdgpu_evf_mgr_flush_suspend) wait on it forever and
> wedge the machine:
> 
>   INFO: task kworker/6:28 blocked for more than 120 seconds.
>   Workqueue: events amdgpu_eviction_fence_suspend_worker [amdgpu]
>   Call Trace:
>    dma_fence_wait_timeout+0x7e/0x130
>    amdgpu_userq_evict+0x67/0x140 [amdgpu]
>    amdgpu_eviction_fence_suspend_worker+0xd8/0x160 [amdgpu]
>    process_scheduled_works+0xa6/0x420
> 
> Force-complete every queue's fence regardless of state. The unmap and
> mark-hung step stays gated on MAPPED, since unmapping a queue that is
> not mapped is invalid.
> 
> Fixes: 290f46cf5726 ("drm/amdgpu: Implement user queue reset functionality")
> Signed-off-by: Jesse Zhang <Jesse.Zhang@amd.com>

Reviewed-by: Christian König <christian.koenig@amd.com>

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 25 +++++++++++++----------
>  1 file changed, 14 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> index aa5cc5642e87..6aa75da27f91 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> @@ -1424,18 +1424,21 @@ void amdgpu_userq_pre_reset(struct amdgpu_device *adev)
>  
>  	/* TODO: We probably need a new lock for the queue state */
>  	xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> -		if (queue->state != AMDGPU_USERQ_STATE_MAPPED)
> -			continue;
> -
> -		trace_amdgpu_userq_state_start(queue);
> -		userq_funcs = adev->userq_funcs[queue->queue_type];
> -		userq_funcs->unmap(queue);
> -		/* just mark all queues as hung at this point.
> -		 * if unmap succeeds, we could map again
> -		 * in amdgpu_userq_post_reset() if vram is not lost
> +		if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
> +			trace_amdgpu_userq_state_start(queue);
> +			userq_funcs = adev->userq_funcs[queue->queue_type];
> +			userq_funcs->unmap(queue);
> +			/* just mark all queues as hung at this point.
> +			 * if unmap succeeds, we could map again
> +			 * in amdgpu_userq_post_reset() if vram is not lost
> +			 */
> +			trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
> +			queue->state = AMDGPU_USERQ_STATE_HUNG;
> +		}
> +		/* Force-complete any pending fence regardless of queue state so
> +		 * that eviction/suspend and queue teardown waiters don't block
> +		 * forever on a fence that will never signal after the reset.
>  		 */
> -		trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
> -		queue->state = AMDGPU_USERQ_STATE_HUNG;
>  		amdgpu_userq_fence_driver_force_completion(queue);
>  	}
>  }