All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Sasha Levin <sashal@kernel.org>,
	Philip Yang <Philip.Yang@amd.com>,
	Felix.Kuehling@amd.com, Xinhui.Pan@amd.com,
	amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org,
	daniel@ffwll.ch, Alex Deucher <alexander.deucher@amd.com>,
	airlied@gmail.com, christian.koenig@amd.com
Subject: [PATCH AUTOSEL 6.1 17/27] drm/amdkfd: Fix lock dependency warning with srcu
Date: Sun, 28 Jan 2024 11:14:02 -0500	[thread overview]
Message-ID: <20240128161424.203600-17-sashal@kernel.org> (raw)
In-Reply-To: <20240128161424.203600-1-sashal@kernel.org>

From: Philip Yang <Philip.Yang@amd.com>

[ Upstream commit 2a9de42e8d3c82c6990d226198602be44f43f340 ]

======================================================
WARNING: possible circular locking dependency detected
6.5.0-kfd-yangp #2289 Not tainted
------------------------------------------------------
kworker/0:2/996 is trying to acquire lock:
        (srcu){.+.+}-{0:0}, at: __synchronize_srcu+0x5/0x1a0

but task is already holding lock:
        ((work_completion)(&svms->deferred_list_work)){+.+.}-{0:0}, at:
	process_one_work+0x211/0x560

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #3 ((work_completion)(&svms->deferred_list_work)){+.+.}-{0:0}:
        __flush_work+0x88/0x4f0
        svm_range_list_lock_and_flush_work+0x3d/0x110 [amdgpu]
        svm_range_set_attr+0xd6/0x14c0 [amdgpu]
        kfd_ioctl+0x1d1/0x630 [amdgpu]
        __x64_sys_ioctl+0x88/0xc0

-> #2 (&info->lock#2){+.+.}-{3:3}:
        __mutex_lock+0x99/0xc70
        amdgpu_amdkfd_gpuvm_restore_process_bos+0x54/0x740 [amdgpu]
        restore_process_helper+0x22/0x80 [amdgpu]
        restore_process_worker+0x2d/0xa0 [amdgpu]
        process_one_work+0x29b/0x560
        worker_thread+0x3d/0x3d0

-> #1 ((work_completion)(&(&process->restore_work)->work)){+.+.}-{0:0}:
        __flush_work+0x88/0x4f0
        __cancel_work_timer+0x12c/0x1c0
        kfd_process_notifier_release_internal+0x37/0x1f0 [amdgpu]
        __mmu_notifier_release+0xad/0x240
        exit_mmap+0x6a/0x3a0
        mmput+0x6a/0x120
        do_exit+0x322/0xb90
        do_group_exit+0x37/0xa0
        __x64_sys_exit_group+0x18/0x20
        do_syscall_64+0x38/0x80

-> #0 (srcu){.+.+}-{0:0}:
        __lock_acquire+0x1521/0x2510
        lock_sync+0x5f/0x90
        __synchronize_srcu+0x4f/0x1a0
        __mmu_notifier_release+0x128/0x240
        exit_mmap+0x6a/0x3a0
        mmput+0x6a/0x120
        svm_range_deferred_list_work+0x19f/0x350 [amdgpu]
        process_one_work+0x29b/0x560
        worker_thread+0x3d/0x3d0

other info that might help us debug this:
Chain exists of:
  srcu --> &info->lock#2 --> (work_completion)(&svms->deferred_list_work)

Possible unsafe locking scenario:

        CPU0                    CPU1
        ----                    ----
        lock((work_completion)(&svms->deferred_list_work));
                        lock(&info->lock#2);
			lock((work_completion)(&svms->deferred_list_work));
        sync(srcu);

Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 4ecc4be1a910..5188c4d2e7c0 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -2241,8 +2241,10 @@ static void svm_range_deferred_list_work(struct work_struct *work)
 		mutex_unlock(&svms->lock);
 		mmap_write_unlock(mm);
 
-		/* Pairs with mmget in svm_range_add_list_work */
-		mmput(mm);
+		/* Pairs with mmget in svm_range_add_list_work. If dropping the
+		 * last mm refcount, schedule release work to avoid circular locking
+		 */
+		mmput_async(mm);
 
 		spin_lock(&svms->deferred_list_lock);
 	}
-- 
2.43.0


WARNING: multiple messages have this Message-ID (diff)
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Philip Yang <Philip.Yang@amd.com>,
	Felix Kuehling <felix.kuehling@amd.com>,
	Alex Deucher <alexander.deucher@amd.com>,
	Sasha Levin <sashal@kernel.org>,
	Felix.Kuehling@amd.com, christian.koenig@amd.com,
	Xinhui.Pan@amd.com, airlied@gmail.com, daniel@ffwll.ch,
	amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org
Subject: [PATCH AUTOSEL 6.1 17/27] drm/amdkfd: Fix lock dependency warning with srcu
Date: Sun, 28 Jan 2024 11:14:02 -0500	[thread overview]
Message-ID: <20240128161424.203600-17-sashal@kernel.org> (raw)
In-Reply-To: <20240128161424.203600-1-sashal@kernel.org>

From: Philip Yang <Philip.Yang@amd.com>

[ Upstream commit 2a9de42e8d3c82c6990d226198602be44f43f340 ]

======================================================
WARNING: possible circular locking dependency detected
6.5.0-kfd-yangp #2289 Not tainted
------------------------------------------------------
kworker/0:2/996 is trying to acquire lock:
        (srcu){.+.+}-{0:0}, at: __synchronize_srcu+0x5/0x1a0

but task is already holding lock:
        ((work_completion)(&svms->deferred_list_work)){+.+.}-{0:0}, at:
	process_one_work+0x211/0x560

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #3 ((work_completion)(&svms->deferred_list_work)){+.+.}-{0:0}:
        __flush_work+0x88/0x4f0
        svm_range_list_lock_and_flush_work+0x3d/0x110 [amdgpu]
        svm_range_set_attr+0xd6/0x14c0 [amdgpu]
        kfd_ioctl+0x1d1/0x630 [amdgpu]
        __x64_sys_ioctl+0x88/0xc0

-> #2 (&info->lock#2){+.+.}-{3:3}:
        __mutex_lock+0x99/0xc70
        amdgpu_amdkfd_gpuvm_restore_process_bos+0x54/0x740 [amdgpu]
        restore_process_helper+0x22/0x80 [amdgpu]
        restore_process_worker+0x2d/0xa0 [amdgpu]
        process_one_work+0x29b/0x560
        worker_thread+0x3d/0x3d0

-> #1 ((work_completion)(&(&process->restore_work)->work)){+.+.}-{0:0}:
        __flush_work+0x88/0x4f0
        __cancel_work_timer+0x12c/0x1c0
        kfd_process_notifier_release_internal+0x37/0x1f0 [amdgpu]
        __mmu_notifier_release+0xad/0x240
        exit_mmap+0x6a/0x3a0
        mmput+0x6a/0x120
        do_exit+0x322/0xb90
        do_group_exit+0x37/0xa0
        __x64_sys_exit_group+0x18/0x20
        do_syscall_64+0x38/0x80

-> #0 (srcu){.+.+}-{0:0}:
        __lock_acquire+0x1521/0x2510
        lock_sync+0x5f/0x90
        __synchronize_srcu+0x4f/0x1a0
        __mmu_notifier_release+0x128/0x240
        exit_mmap+0x6a/0x3a0
        mmput+0x6a/0x120
        svm_range_deferred_list_work+0x19f/0x350 [amdgpu]
        process_one_work+0x29b/0x560
        worker_thread+0x3d/0x3d0

other info that might help us debug this:
Chain exists of:
  srcu --> &info->lock#2 --> (work_completion)(&svms->deferred_list_work)

Possible unsafe locking scenario:

        CPU0                    CPU1
        ----                    ----
        lock((work_completion)(&svms->deferred_list_work));
                        lock(&info->lock#2);
			lock((work_completion)(&svms->deferred_list_work));
        sync(srcu);

Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 4ecc4be1a910..5188c4d2e7c0 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -2241,8 +2241,10 @@ static void svm_range_deferred_list_work(struct work_struct *work)
 		mutex_unlock(&svms->lock);
 		mmap_write_unlock(mm);
 
-		/* Pairs with mmget in svm_range_add_list_work */
-		mmput(mm);
+		/* Pairs with mmget in svm_range_add_list_work. If dropping the
+		 * last mm refcount, schedule release work to avoid circular locking
+		 */
+		mmput_async(mm);
 
 		spin_lock(&svms->deferred_list_lock);
 	}
-- 
2.43.0


  parent reply	other threads:[~2024-01-28 16:15 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-01-28 16:13 [PATCH AUTOSEL 6.1 01/27] PCI: Only override AMD USB controller if required Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 02/27] PCI: switchtec: Fix stdev_release() crash after surprise hot remove Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 03/27] perf cs-etm: Bump minimum OpenCSD version to ensure a bugfix is present Sasha Levin
2024-01-28 16:13   ` Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 04/27] xhci: fix possible null pointer deref during xhci urb enqueue Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 05/27] usb: hub: Replace hardcoded quirk value with BIT() macro Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 06/27] usb: hub: Add quirk to decrease IN-ep poll interval for Microchip USB491x hub Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 07/27] selftests/sgx: Fix linker script asserts Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 08/27] tty: allow TIOCSLCKTRMIOS with CAP_CHECKPOINT_RESTORE Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 09/27] fs/kernfs/dir: obey S_ISGID Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 10/27] spmi: mediatek: Fix UAF on device remove Sasha Levin
2024-01-28 16:13   ` Sasha Levin
2024-01-28 16:13   ` Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 11/27] PCI: Fix 64GT/s effective data rate calculation Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 12/27] PCI/AER: Decode Requester ID when no error info found Sasha Levin
2024-01-28 16:13   ` Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 13/27] 9p: Fix initialisation of netfs_inode for 9p Sasha Levin
2024-01-28 16:13 ` [PATCH AUTOSEL 6.1 14/27] misc: lis3lv02d_i2c: Add missing setting of the reg_ctrl callback Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 15/27] libsubcmd: Fix memory leak in uniq() Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 16/27] drm/amdkfd: Fix lock dependency warning Sasha Levin
2024-01-28 16:14   ` Sasha Levin
2024-01-28 16:14 ` Sasha Levin [this message]
2024-01-28 16:14   ` [PATCH AUTOSEL 6.1 17/27] drm/amdkfd: Fix lock dependency warning with srcu Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 18/27] virtio_net: Fix "‘%d’ directive writing between 1 and 11 bytes into a region of size 10" warnings Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 19/27] blk-mq: fix IO hang from sbitmap wakeup race Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 20/27] ceph: reinitialize mds feature bit even when session in open Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 21/27] ceph: fix deadlock or deadcode of misusing dget() Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 22/27] ceph: fix invalid pointer access if get_quota_realm return ERR_PTR Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 23/27] drm/amd/powerplay: Fix kzalloc parameter 'ATOM_Tonga_PPM_Table' in 'get_platform_power_management_table()' Sasha Levin
2024-01-28 16:14   ` Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 24/27] drm/amdgpu: Fix with right return code '-EIO' in 'amdgpu_gmc_vram_checking()' Sasha Levin
2024-01-28 16:14   ` Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 25/27] drm/amdgpu: Release 'adev->pm.fw' before return in 'amdgpu_device_need_post()' Sasha Levin
2024-01-28 16:14   ` Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 26/27] drm/amdkfd: Fix 'node' NULL check in 'svm_range_get_range_boundaries()' Sasha Levin
2024-01-28 16:14   ` Sasha Levin
2024-01-28 16:14 ` [PATCH AUTOSEL 6.1 27/27] i2c: rk3x: Adjust mask/value offset for i2c2 on rv1126 Sasha Levin
2024-01-28 16:14   ` Sasha Levin
2024-01-28 16:14   ` Sasha Levin
2024-01-28 23:19   ` Tim Lunn
2024-01-28 23:19     ` Tim Lunn
2024-01-28 23:19     ` Tim Lunn
2024-02-22 12:37     ` Sasha Levin
2024-02-22 12:37       ` Sasha Levin
2024-02-22 12:37       ` Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240128161424.203600-17-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=Felix.Kuehling@amd.com \
    --cc=Philip.Yang@amd.com \
    --cc=Xinhui.Pan@amd.com \
    --cc=airlied@gmail.com \
    --cc=alexander.deucher@amd.com \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=christian.koenig@amd.com \
    --cc=daniel@ffwll.ch \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.