All of lore.kernel.org
 help / color / mirror / Atom feed
From: sashiko-bot@kernel.org
To: "Alex Deucher" <alexander.deucher@amd.com>,
	amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 08/11] drm/amdgpu/userq: add handling for set_gang
Date: Wed, 18 Nov 2026 22:54:06 +0000	[thread overview]
Message-ID: <sashiko-outbox-222131@kernel.org> (raw)
In-Reply-To: <20261118223712.2518580-8-alexander.deucher@amd.com>

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] drm/amdgpu/userq: missing reset_domain lock in
  amdgpu_userq_set_gang_helper()
- [Medium] drm/amdgpu/userq: invalid set_gang call in
  amdgpu_userq_post_reset()
--

commit c119a0781eef4bb00ce5e6c3043c15d765972cac
Author: Alex Deucher <alexander.deucher@amd.com>

drm/amdgpu/userq: add handling for set_gang

This commit adds gang primary/secondary pointers to struct
amdgpu_usermode_queue and introduces amdgpu_userq_set_gang_helper() to
re-establish queue gangs during user queue restore, resume, and post-reset
recovery.

> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> index ed6f3b67e8720..3afd17af28548 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
> @@ -303,6 +303,27 @@ static bool amdgpu_userq_buffer_vas_mapped(struct amdgpu_usermode_queue *queue)
>  	return true;
>  }
>  
> +static int amdgpu_userq_set_gang_helper(struct amdgpu_usermode_queue *queue)
> +{
> +	struct amdgpu_usermode_queue *secondary_queue = queue->gang_secondary;
> +	struct amdgpu_userq_mgr *uq_mgr = queue->userq_mgr;
> +	struct amdgpu_device *adev = uq_mgr->adev;
> +	const struct amdgpu_userq_funcs *userq_funcs =
> +		adev->userq_funcs[queue->queue_type];
> +	int r;
> +
> +	if (secondary_queue &&
> +	    (queue->state == AMDGPU_USERQ_STATE_MAPPED ||
> +	     queue->state == AMDGPU_USERQ_STATE_PREEMPTED) &&
> +	    (secondary_queue->state == AMDGPU_USERQ_STATE_MAPPED ||
> +	     secondary_queue->state == AMDGPU_USERQ_STATE_PREEMPTED)) {
> +		r = userq_funcs->set_gang(queue, secondary_queue);
> +		if (r)
> +			return r;
> +	}
> +	return 0;
> +}
> +
>  static int amdgpu_userq_preempt_helper(struct amdgpu_usermode_queue *queue)
>  {
>  	struct amdgpu_userq_mgr *uq_mgr = queue->userq_mgr;
> @@ -968,6 +989,11 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
>  		if (r)
>  			ret = r;
>  	}
> +	xa_for_each(&uq_mgr->userq_xa, queue_id, queue) {
> +		r = amdgpu_userq_set_gang_helper(queue);
> +		if (r)
> +			ret = r;
> +	}
>  	mutex_unlock(&uq_mgr->userq_mutex);
>  
>  	if (ret)
> @@ -1446,6 +1472,13 @@ int amdgpu_userq_resume(struct amdgpu_device *adev)
>  		if (r)
>  			return r;
>  	}
> +	xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> +		uqm = queue->userq_mgr;
> +		guard(mutex)(&uqm->userq_mutex);
> +		r = amdgpu_userq_set_gang_helper(queue);
> +		if (r)
> +			return r;
> +	}
>  
>  	return 0;
>  }

[Severity: Medium]
Should callers on the non-reset paths hold a read lock on
adev->reset_domain->sem across amdgpu_userq_set_gang_helper()?

In amdgpu_userq_restore_all() and amdgpu_userq_resume(), the first loop calls
amdgpu_userq_map_helper(), which takes and releases adev->reset_domain->sem
per queue:

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c:amdgpu_userq_map_helper() {
    ...
	/* The reset changes the queue state and the MES is unusable meanwhile */
	guard(rwsem_read)(&adev->reset_domain->sem);
    ...
}

When the second loop calls amdgpu_userq_set_gang_helper(),
adev->reset_domain->sem is no longer held.  If a GPU reset starts between the
two loops, amdgpu_userq_pre_reset() and amdgpu_userq_post_reset() mutate
queue->state, secondary_queue->state, and gang_ctx_array_index under the
write side of adev->reset_domain->sem without taking uq_mgr->userq_mutex:

kworker (uq_mgr->resume_work)     GPU reset thread
amdgpu_userq_restore_all()
  mutex_lock(&uq_mgr->userq_mutex)
  amdgpu_userq_map_helper(queue)
    down_read(&reset_domain->sem)
    ...
    up_read(&reset_domain->sem)
                                  down_write(&reset_domain->sem)
                                  amdgpu_userq_pre_reset()
                                    userq_funcs->unmap(queue)
                                    queue->state = AMDGPU_USERQ_STATE_HUNG
  amdgpu_userq_set_gang_helper()
    userq_funcs->set_gang()       <-- submits to MES during reset

Calling userq_funcs->set_gang() without adev->reset_domain->sem held also
triggers lockdep_assert_held(&adev->reset_domain->sem) in
amdgpu_device_skip_hw_access() when ringing the MES doorbell.  Note that
amdgpu_userq_post_reset() also calls amdgpu_userq_set_gang_helper() while
already holding adev->reset_domain->sem for write, so taking the read lock
inside amdgpu_userq_set_gang_helper() itself would deadlock the reset path.

> @@ -1591,6 +1624,13 @@ int amdgpu_userq_post_reset(struct amdgpu_device *adev, bool vram_lost)
>  			queue->state = AMDGPU_USERQ_STATE_MAPPED;
>  		}
>  	}
> +	xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
> +		r = amdgpu_userq_set_gang_helper(queue);
> +		if (r) {
> +			dev_err(adev->dev, "Failed to gang queue %ld\n", queue_id);
> +			ret = r;
> +		}
> +	}
>  
>  	return ret;
>  }

[Severity: Medium]
Can amdgpu_userq_post_reset() call userq_funcs->set_gang() on queues that were
not re-added to MES after the reset?

During a GPU reset, amdgpu_userq_pre_reset() only transitions
AMDGPU_USERQ_STATE_MAPPED queues to AMDGPU_USERQ_STATE_HUNG, leaving any
preempted queues in AMDGPU_USERQ_STATE_PREEMPTED while MES hardware state is
wiped.  In amdgpu_userq_post_reset(), the first loop only remaps queues whose
state is AMDGPU_USERQ_STATE_HUNG when !vram_lost.

Because the second loop runs unconditionally (even when vram_lost is true)
and amdgpu_userq_set_gang_helper() accepts queues in
AMDGPU_USERQ_STATE_PREEMPTED, it will issue MES SET_GANG_SUBMIT commands for
preempted queues that have not been mapped in MES after the reset, causing
MES command failures and failing amdgpu_userq_post_reset().

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261118223712.2518580-1-alexander.deucher@amd.com?part=8

  reply	other threads:[~2026-11-18 22:54 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 22:37 [PATCH 01/11] drm/amdgpu/userq: Add INFO IOCTL query for doorbell ranges Alex Deucher
2026-11-18 22:37 ` [PATCH 02/11] drm/amdgpu/mes11: add API for gang submit Alex Deucher
2026-11-18 22:37 ` [PATCH 03/11] drm/amdgpu/mes: add callback " Alex Deucher
2026-11-18 22:37 ` [PATCH 04/11] drm/amdgpu/mes11: add support for setting " Alex Deucher
2026-11-18 22:37 ` [PATCH 05/11] drm/amdgpu/mes12: " Alex Deucher
2026-11-18 22:37 ` [PATCH 06/11] drm/amdgpu/userq: add callback for gang setup Alex Deucher
2026-11-18 22:37 ` [PATCH 07/11] drm/amdgpu/userq/mes: implement set gang submit Alex Deucher
2026-11-18 22:44   ` sashiko-bot
2026-11-18 22:37 ` [PATCH 08/11] drm/amdgpu/userq: add handling for set_gang Alex Deucher
2026-11-18 22:54   ` sashiko-bot [this message]
2026-11-18 22:37 ` [PATCH 09/11] drm/amdgpu: add UAPI to create user queue gangs Alex Deucher
2026-11-20  5:15   ` Mohan Marimuthu, Yogesh
2026-11-20 19:28     ` Alex Deucher
2026-11-18 22:37 ` [PATCH 10/11] drm/amdgpu/userq: enable support for creating gangs Alex Deucher
2026-11-18 22:58   ` sashiko-bot
2026-11-18 22:37 ` [PATCH 11/11] drm/amdgpu: bump driver version for userq updates Alex Deucher
2026-11-18 23:04 ` [PATCH 01/11] drm/amdgpu/userq: Add INFO IOCTL query for doorbell ranges sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=sashiko-outbox-222131@kernel.org \
    --to=sashiko-bot@kernel.org \
    --cc=alexander.deucher@amd.com \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.