AMD-GFX Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Alex Deucher" <alexander.deucher@amd.com>,
	amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 10/11] drm/amdgpu/userq: enable support for creating gangs
Date: Wed, 18 Nov 2026 22:58:24 +0000	[thread overview]
Message-ID: <sashiko-outbox-222136@kernel.org> (raw)
In-Reply-To: <20261118223712.2518580-10-alexander.deucher@amd.com>

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] drm/amdgpu/userq: lock order inversion in
  amdgpu_userq_create_gang()
- [High] drm/amdgpu/userq: race between amdgpu_userq_destroy() and
  post_reset()
- [Medium] drm/amdgpu/userq: missing INVALID_VA check in
  amdgpu_userq_create_gang()
--

commit 8106c51d4a0229062b3c666f8f9004011c7d6501
Author: Alex Deucher <alexander.deucher@amd.com>

drm/amdgpu/userq: enable support for creating gangs

This commit adds the AMDGPU_USERQ_OP_CREATE_GANG ioctl operation to link
primary and secondary usermode queues into a gang and cleans up the gang
links when a queue is destroyed.

> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> index 3afd17af28548..388385380d8bc 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> @@ -566,6 +566,7 @@ amdgpu_userq_destroy(struct amdgpu_userq_mgr *uq_mgr, struct amdgpu_usermode_que
>  {
>  	struct amdgpu_device *adev = uq_mgr->adev;
>  	const struct amdgpu_userq_funcs *uq_funcs = adev->userq_funcs[queue->queue_type];
> +	struct amdgpu_usermode_queue *other_gang_queue;
>  	int r = 0;
>  
>  	trace_amdgpu_userq_destroy_start(queue);
> @@ -585,6 +586,17 @@ amdgpu_userq_destroy(struct amdgpu_userq_mgr *uq_mgr, struct amdgpu_usermode_que
>  	debugfs_remove_recursive(queue->debugfs_queue);
>  #endif
>  	r = amdgpu_userq_unmap_helper(queue);
> +	/* clean up gangs */
> +	if (queue->gang_primary) {
> +		other_gang_queue = queue->gang_primary;
> +		other_gang_queue->gang_secondary = NULL;
> +		queue->gang_primary = NULL;
> +	}
> +	if (queue->gang_secondary) {
> +		other_gang_queue = queue->gang_secondary;
> +		other_gang_queue->gang_primary = NULL;
> +		queue->gang_secondary = NULL;
> +	}
>  	atomic_dec(&uq_mgr->userq_count[queue->queue_type]);
>  	amdgpu_userq_fence_driver_free(queue);
>  	queue->fence_drv = NULL;

[Severity: High]
Can amdgpu_userq_destroy() race with amdgpu_userq_post_reset() on the
partner queue's gang_secondary pointer and cause a use-after-free?

amdgpu_userq_destroy() holds uq_mgr->userq_mutex, while
amdgpu_userq_detach_doorbell() and amdgpu_userq_unmap_helper() each take
and release adev->reset_domain->sem individually.  During a GPU reset,
amdgpu_userq_post_reset() iterates adev->userq_doorbell_xa under
down_write(&adev->reset_domain->sem) without taking uq_mgr->userq_mutex:

CPU0                                  CPU1 (GPU reset)
amdgpu_userq_destroy(secondary)
  mutex_lock(&uq_mgr->userq_mutex)
  amdgpu_userq_detach_doorbell()
  amdgpu_userq_unmap_helper()
    /* releases reset_domain->sem */
                                      down_write(&adev->reset_domain->sem)
                                      amdgpu_userq_post_reset()
                                        amdgpu_userq_set_gang_helper(primary)
                                          sec = primary->gang_secondary
  primary->gang_secondary = NULL
  mutex_unlock(&uq_mgr->userq_mutex)
  uq_funcs->mqd_destroy(secondary)
  kfree(secondary)
                                          if (sec->state == ...)
                                            <-- use-after-free

In addition, if the reset begins between amdgpu_userq_detach_doorbell()
and amdgpu_userq_unmap_helper(), secondary is already erased from
adev->userq_doorbell_xa while still in AMDGPU_USERQ_STATE_MAPPED, and
amdgpu_userq_destroy() blocks on reset_domain->sem inside
amdgpu_userq_unmap_helper() before clearing primary->gang_secondary.
amdgpu_userq_post_reset() then remaps primary and calls
userq_funcs->set_gang(primary, secondary) on a secondary queue that was
never remapped after the reset and is about to be destroyed.

> @@ -838,6 +850,106 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args)
>  	return r;
>  }
>  
> +static int
> +amdgpu_userq_create_gang(struct drm_file *filp, union drm_amdgpu_userq *args)
> +{
> +	struct amdgpu_usermode_queue *primary_queue, *secondary_queue;
> +	struct amdgpu_fpriv *fpriv = filp->driver_priv;
> +	struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr;
> +	const struct amdgpu_userq_funcs *userq_funcs;
> +	struct amdgpu_device *adev = uq_mgr->adev;
> +	int primary_queue_id = args->in_cg.primary_queue_id;
> +	int secondary_queue_id = args->in_cg.secondary_queue_id;
> +	int r;
> +
> +	/* The reset changes the queue state and the MES is unusable meanwhile */
> +	guard(rwsem_read)(&adev->reset_domain->sem);
> +
> +	mutex_lock(&uq_mgr->userq_mutex);

[Severity: High]
Does taking adev->reset_domain->sem before uq_mgr->userq_mutex invert the
lock hierarchy and risk an ABBA deadlock?

amdgpu_lockdep_init() and the rest of amdgpu_userq.c establish
uq_mgr->userq_mutex outer to adev->reset_domain->sem:

CPU0                              CPU1
----                              ----
lock(&uq_mgr->userq_mutex)
                                  down_read(&adev->reset_domain->sem)
down_read(&adev->reset_domain->sem)
                                  lock(&uq_mgr->userq_mutex)

 *** DEADLOCK ***

CPU0: amdgpu_userq_destroy() -> amdgpu_userq_unmap_helper()
CPU1: amdgpu_userq_create_gang()
(when a GPU reset writer is queued on adev->reset_domain->sem)

Also, because guard(rwsem_read)(&adev->reset_domain->sem) has function
scope, it is still held at the out label when amdgpu_userq_put() runs.
If a concurrent AMDGPU_USERQ_OP_FREE dropped the queue's base reference,
amdgpu_userq_put() enters amdgpu_userq_destroy() while holding
reset_domain->sem, waiting on cancel_delayed_work_sync() under the reset
semaphore and taking reset_domain->sem recursively in
amdgpu_userq_detach_doorbell() and amdgpu_userq_unmap_helper().

[ ... ]

> +	if (primary_queue->state == AMDGPU_USERQ_STATE_HUNG) {
> +		drm_file_err(uq_mgr->file, "Primary queue is hung\n");
> +		r = -EINVAL;
> +		goto out;
> +	}
> +
> +	if ((secondary_queue->queue_type != AMDGPU_HW_IP_GFX) &&
> +	    (secondary_queue->queue_type != AMDGPU_HW_IP_COMPUTE) &&
> +	    (secondary_queue->queue_type != AMDGPU_HW_IP_DMA)) {
> +		drm_file_err(uq_mgr->file, "Invalid gang secondary queue type\n");
> +		r = -EINVAL;
> +		goto out;
> +	}
> +
> +	if (secondary_queue->gang_primary || secondary_queue->gang_secondary) {
> +		drm_file_err(uq_mgr->file, "Secondary queue already in a gang\n");
> +		r = -EINVAL;
> +		goto out;
> +	}
> +
> +	if (secondary_queue->state == AMDGPU_USERQ_STATE_HUNG) {
> +		drm_file_err(uq_mgr->file, "Secondary queue is hung\n");
> +		r = -EINVAL;
> +		goto out;
> +	}
> +
> +	userq_funcs = adev->userq_funcs[primary_queue->queue_type];
> +	if ((primary_queue->state == AMDGPU_USERQ_STATE_MAPPED ||
> +	     primary_queue->state == AMDGPU_USERQ_STATE_PREEMPTED) &&
> +	    (secondary_queue->state == AMDGPU_USERQ_STATE_MAPPED ||
> +	     secondary_queue->state == AMDGPU_USERQ_STATE_PREEMPTED))
> +		r = userq_funcs->set_gang(primary_queue, secondary_queue);
> +	else
> +		/* if the queues are currently unmapped, the gang state will get
> +		 * updated when they get mapped.
> +		 */
> +		r = 0;

[Severity: Medium]
Should amdgpu_userq_create_gang() also reject queues in
AMDGPU_USERQ_STATE_INVALID_VA?

If a queue was marked AMDGPU_USERQ_STATE_INVALID_VA in
amdgpu_userq_restore_all(), amdgpu_userq_map_helper() will never remap it
(it only maps queues in AMDGPU_USERQ_STATE_UNMAPPED).  Here it passes the
HUNG checks and falls into the else branch with r = 0, linking a dead
queue into the partner queue's gang pointer without ever programming the
gang in MES.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261118223712.2518580-1-alexander.deucher@amd.com?part=10

  reply	other threads:[~2026-11-18 22:58 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 22:37 [PATCH 01/11] drm/amdgpu/userq: Add INFO IOCTL query for doorbell ranges Alex Deucher
2026-11-18 22:37 ` [PATCH 02/11] drm/amdgpu/mes11: add API for gang submit Alex Deucher
2026-11-18 22:37 ` [PATCH 03/11] drm/amdgpu/mes: add callback " Alex Deucher
2026-11-18 22:37 ` [PATCH 04/11] drm/amdgpu/mes11: add support for setting " Alex Deucher
2026-11-18 22:37 ` [PATCH 05/11] drm/amdgpu/mes12: " Alex Deucher
2026-11-18 22:37 ` [PATCH 06/11] drm/amdgpu/userq: add callback for gang setup Alex Deucher
2026-11-18 22:37 ` [PATCH 07/11] drm/amdgpu/userq/mes: implement set gang submit Alex Deucher
2026-11-18 22:44   ` sashiko-bot
2026-11-18 22:37 ` [PATCH 08/11] drm/amdgpu/userq: add handling for set_gang Alex Deucher
2026-11-18 22:54   ` sashiko-bot
2026-11-18 22:37 ` [PATCH 09/11] drm/amdgpu: add UAPI to create user queue gangs Alex Deucher
2026-11-18 22:37 ` [PATCH 10/11] drm/amdgpu/userq: enable support for creating gangs Alex Deucher
2026-11-18 22:58   ` sashiko-bot [this message]
2026-11-18 22:37 ` [PATCH 11/11] drm/amdgpu: bump driver version for userq updates Alex Deucher
2026-11-18 23:04 ` [PATCH 01/11] drm/amdgpu/userq: Add INFO IOCTL query for doorbell ranges sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-222136@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=alexander.deucher@amd.com \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox