AMD-GFX Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Alex Deucher" <alexander.deucher@amd.com>,
	amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 08/11] drm/amdgpu/userq: add handling for set_gang
Date: Wed, 18 Nov 2026 22:54:06 +0000	[thread overview]
Message-ID: <sashiko-outbox-222131@kernel.org> (raw)
In-Reply-To: <20261118223712.2518580-8-alexander.deucher@amd.com>

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] drm/amdgpu/userq: missing reset_domain lock in
  amdgpu_userq_set_gang_helper()
- [Medium] drm/amdgpu/userq: invalid set_gang call in
  amdgpu_userq_post_reset()
--

commit c119a0781eef4bb00ce5e6c3043c15d765972cac
Author: Alex Deucher <alexander.deucher@amd.com>

drm/amdgpu/userq: add handling for set_gang

This commit adds gang primary/secondary pointers to struct
amdgpu_usermode_queue and introduces amdgpu_userq_set_gang_helper() to
re-establish queue gangs during user queue restore, resume, and post-reset
recovery.

> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> index ed6f3b67e8720..3afd17af28548 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> @@ -303,6 +303,27 @@ static bool amdgpu_userq_buffer_vas_mapped(struct amdgpu_usermode_queue *queue)
>  	return true;
>  }
>  
> +static int amdgpu_userq_set_gang_helper(struct amdgpu_usermode_queue *queue)
> +{
> +	struct amdgpu_usermode_queue *secondary_queue = queue->gang_secondary;
> +	struct amdgpu_userq_mgr *uq_mgr = queue->userq_mgr;
> +	struct amdgpu_device *adev = uq_mgr->adev;
> +	const struct amdgpu_userq_funcs *userq_funcs =
> +		adev->userq_funcs[queue->queue_type];
> +	int r;
> +
> +	if (secondary_queue &&
> +	    (queue->state == AMDGPU_USERQ_STATE_MAPPED ||
> +	     queue->state == AMDGPU_USERQ_STATE_PREEMPTED) &&
> +	    (secondary_queue->state == AMDGPU_USERQ_STATE_MAPPED ||
> +	     secondary_queue->state == AMDGPU_USERQ_STATE_PREEMPTED)) {
> +		r = userq_funcs->set_gang(queue, secondary_queue);
> +		if (r)
> +			return r;
> +	}
> +	return 0;
> +}
> +
>  static int amdgpu_userq_preempt_helper(struct amdgpu_usermode_queue *queue)
>  {
>  	struct amdgpu_userq_mgr *uq_mgr = queue->userq_mgr;
> @@ -968,6 +989,11 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
>  		if (r)
>  			ret = r;
>  	}
> +	xa_for_each(&uq_mgr->userq_xa, queue_id, queue) {
> +		r = amdgpu_userq_set_gang_helper(queue);
> +		if (r)
> +			ret = r;
> +	}
>  	mutex_unlock(&uq_mgr->userq_mutex);
>  
>  	if (ret)
> @@ -1446,6 +1472,13 @@ int amdgpu_userq_resume(struct amdgpu_device *adev)
>  		if (r)
>  			return r;
>  	}
> +	xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> +		uqm = queue->userq_mgr;
> +		guard(mutex)(&uqm->userq_mutex);
> +		r = amdgpu_userq_set_gang_helper(queue);
> +		if (r)
> +			return r;
> +	}
>  
>  	return 0;
>  }

[Severity: Medium]
Should callers on the non-reset paths hold a read lock on
adev->reset_domain->sem across amdgpu_userq_set_gang_helper()?

In amdgpu_userq_restore_all() and amdgpu_userq_resume(), the first loop calls
amdgpu_userq_map_helper(), which takes and releases adev->reset_domain->sem
per queue:

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c:amdgpu_userq_map_helper() {
    ...
	/* The reset changes the queue state and the MES is unusable meanwhile */
	guard(rwsem_read)(&adev->reset_domain->sem);
    ...
}

When the second loop calls amdgpu_userq_set_gang_helper(),
adev->reset_domain->sem is no longer held.  If a GPU reset starts between the
two loops, amdgpu_userq_pre_reset() and amdgpu_userq_post_reset() mutate
queue->state, secondary_queue->state, and gang_ctx_array_index under the
write side of adev->reset_domain->sem without taking uq_mgr->userq_mutex:

kworker (uq_mgr->resume_work)     GPU reset thread
amdgpu_userq_restore_all()
  mutex_lock(&uq_mgr->userq_mutex)
  amdgpu_userq_map_helper(queue)
    down_read(&reset_domain->sem)
    ...
    up_read(&reset_domain->sem)
                                  down_write(&reset_domain->sem)
                                  amdgpu_userq_pre_reset()
                                    userq_funcs->unmap(queue)
                                    queue->state = AMDGPU_USERQ_STATE_HUNG
  amdgpu_userq_set_gang_helper()
    userq_funcs->set_gang()       <-- submits to MES during reset

Calling userq_funcs->set_gang() without adev->reset_domain->sem held also
triggers lockdep_assert_held(&adev->reset_domain->sem) in
amdgpu_device_skip_hw_access() when ringing the MES doorbell.  Note that
amdgpu_userq_post_reset() also calls amdgpu_userq_set_gang_helper() while
already holding adev->reset_domain->sem for write, so taking the read lock
inside amdgpu_userq_set_gang_helper() itself would deadlock the reset path.

> @@ -1591,6 +1624,13 @@ int amdgpu_userq_post_reset(struct amdgpu_device *adev, bool vram_lost)
>  			queue->state = AMDGPU_USERQ_STATE_MAPPED;
>  		}
>  	}
> +	xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> +		r = amdgpu_userq_set_gang_helper(queue);
> +		if (r) {
> +			dev_err(adev->dev, "Failed to gang queue %ld\n", queue_id);
> +			ret = r;
> +		}
> +	}
>  
>  	return ret;
>  }

[Severity: Medium]
Can amdgpu_userq_post_reset() call userq_funcs->set_gang() on queues that were
not re-added to MES after the reset?

During a GPU reset, amdgpu_userq_pre_reset() only transitions
AMDGPU_USERQ_STATE_MAPPED queues to AMDGPU_USERQ_STATE_HUNG, leaving any
preempted queues in AMDGPU_USERQ_STATE_PREEMPTED while MES hardware state is
wiped.  In amdgpu_userq_post_reset(), the first loop only remaps queues whose
state is AMDGPU_USERQ_STATE_HUNG when !vram_lost.

Because the second loop runs unconditionally (even when vram_lost is true)
and amdgpu_userq_set_gang_helper() accepts queues in
AMDGPU_USERQ_STATE_PREEMPTED, it will issue MES SET_GANG_SUBMIT commands for
preempted queues that have not been mapped in MES after the reset, causing
MES command failures and failing amdgpu_userq_post_reset().

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261118223712.2518580-1-alexander.deucher@amd.com?part=8

  reply	other threads:[~2026-11-18 22:54 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 22:37 [PATCH 01/11] drm/amdgpu/userq: Add INFO IOCTL query for doorbell ranges Alex Deucher
2026-11-18 22:37 ` [PATCH 02/11] drm/amdgpu/mes11: add API for gang submit Alex Deucher
2026-11-18 22:37 ` [PATCH 03/11] drm/amdgpu/mes: add callback " Alex Deucher
2026-11-18 22:37 ` [PATCH 04/11] drm/amdgpu/mes11: add support for setting " Alex Deucher
2026-11-18 22:37 ` [PATCH 05/11] drm/amdgpu/mes12: " Alex Deucher
2026-11-18 22:37 ` [PATCH 06/11] drm/amdgpu/userq: add callback for gang setup Alex Deucher
2026-11-18 22:37 ` [PATCH 07/11] drm/amdgpu/userq/mes: implement set gang submit Alex Deucher
2026-11-18 22:44   ` sashiko-bot
2026-11-18 22:37 ` [PATCH 08/11] drm/amdgpu/userq: add handling for set_gang Alex Deucher
2026-11-18 22:54   ` sashiko-bot [this message]
2026-11-18 22:37 ` [PATCH 09/11] drm/amdgpu: add UAPI to create user queue gangs Alex Deucher
2026-11-18 22:37 ` [PATCH 10/11] drm/amdgpu/userq: enable support for creating gangs Alex Deucher
2026-11-18 22:58   ` sashiko-bot
2026-11-18 22:37 ` [PATCH 11/11] drm/amdgpu: bump driver version for userq updates Alex Deucher
2026-11-18 23:04 ` [PATCH 01/11] drm/amdgpu/userq: Add INFO IOCTL query for doorbell ranges sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-222131@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=alexander.deucher@amd.com \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox