Re: [PATCH V4] nvme-tcp: teardown circular lockng fixes

Hannes Reinecke
2026-04-14T10:09:12Z

On 4/13/26 19:16, Chaitanya Kulkarni wrote:
> When a controller reset is triggered via sysfs (by writing to
> /sys/class/nvme/<nvmedev>/reset_controller), the reset work tears down
> and re-establishes all queues. The socket release using fput() defers
> the actual cleanup to task_work delayed_fput workqueue. This deferred
> cleanup can race with the subsequent queue re-allocation during reset,
> potentially leading to use-after-free or resource conflicts.
>
> Replace fput() with __fput_sync() to ensure synchronous socket release,
> guaranteeing that all socket resources are fully cleaned up before the
> function returns. This prevents races during controller reset where
> new queue setup may begin before the old socket is fully released.
>
> * Call chain during reset:
> nvme_reset_ctrl_work()
> -> nvme_tcp_teardown_ctrl()
> -> nvme_tcp_teardown_io_queues()
> -> nvme_tcp_free_io_queues()
> -> nvme_tcp_free_queue() <-- fput() -> __fput_sync()
> -> nvme_tcp_teardown_admin_queue()
> -> nvme_tcp_free_admin_queue()
> -> nvme_tcp_free_queue() <-- fput() -> __fput_sync()
> -> nvme_tcp_setup_ctrl() <-- race with deferred fput
>
> memalloc_noreclaim_save() sets PF_MEMALLOC which is intended for tasks
> performing memory reclaim work that need reserve access. While PF_MEMALLOC
> prevents the task from entering direct reclaim (causing __need_reclaim() to
> return false), it does not strip __GFP_IO from gfp flags. The allocator can
> therefore still trigger writeback I/O when __GFP_IO remains set, which is
> unsafe when the caller holds block layer locks.
>
> Switch to memalloc_noio_save() which sets PF_MEMALLOC_NOIO. This causes
> current_gfp_context() to strip __GFP_IO|__GFP_FS from every allocation in
> the scope, making it safe to allocate memory while holding elevator_lock and
> set->srcu.
>
> * The issue can be reproduced using blktests:
>
> nvme_trtype=tcp ./check nvme/005
> blktests (master) # nvme_trtype=tcp ./check nvme/005
> nvme/005 (tr=tcp) (reset local loopback target) [failed]
> runtime 0.725s ... 0.798s
> something found in dmesg:
> [ 108.473940] run blktests nvme/005 at 2025-11-22 16:12:20
>
> [...]
> ...
> (See '/root/blktests/results/nodev_tr_tcp/nvme/005.dmesg' for the entire message)
> blktests (master) # cat /root/blktests/results/nodev_tr_tcp/nvme/005.dmesg
> [ 108.473940] run blktests nvme/005 at 2025-11-22 16:12:20
> [ 108.526983] loop0: detected capacity change from 0 to 2097152
> [ 108.555606] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
> [ 108.572531] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
> [ 108.613061] nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
> [ 108.616832] nvme nvme0: creating 48 I/O queues.
> [ 108.630791] nvme nvme0: mapped 48/0/0 default/read/poll queues.
> [ 108.661892] nvme nvme0: new ctrl: NQN "blktests-subsystem-1", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> [ 108.746639] nvmet: Created nvm controller 2 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
> [ 108.748466] nvme nvme0: creating 48 I/O queues.
> [ 108.802984] nvme nvme0: mapped 48/0/0 default/read/poll queues.
> [ 108.829983] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1"
> [ 108.854288] block nvme0n1: no available path - failing I/O
> [ 108.854344] block nvme0n1: no available path - failing I/O
> [ 108.854373] Buffer I/O error on dev nvme0n1, logical block 1, async page read
>
> [ 108.891693] ======================================================
> [ 108.895912] WARNING: possible circular locking dependency detected
> [ 108.900184] 6.17.0nvme+ #3 Tainted: G N
> [ 108.903913] ------------------------------------------------------
> [ 108.908171] nvme/2734 is trying to acquire lock:
> [ 108.911957] ffff88810210e610 (set->srcu){.+.+}-{0:0}, at: __synchronize_srcu+0x17/0x170
> [ 108.917587]
> but task is already holding lock:
> [ 108.921570] ffff88813abea198 (&q->elevator_lock){+.+.}-{4:4}, at: elevator_change+0xa8/0x1c0
> [ 108.927361]
> which lock already depends on the new lock.
>
> [ 108.933018]
> the existing dependency chain (in reverse order) is:
> [ 108.938223]
> -> #4 (&q->elevator_lock){+.+.}-{4:4}:
> [ 108.942988] __mutex_lock+0xa2/0x1150
> [ 108.945873] elevator_change+0xa8/0x1c0
> [ 108.948925] elv_iosched_store+0xdf/0x140
> [ 108.952043] kernfs_fop_write_iter+0x16a/0x220
> [ 108.955367] vfs_write+0x378/0x520
> [ 108.957598] ksys_write+0x67/0xe0
> [ 108.959721] do_syscall_64+0x76/0xbb0
> [ 108.962052] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 108.965145]
> -> #3 (&q->q_usage_counter(io)){++++}-{0:0}:
> [ 108.968923] blk_alloc_queue+0x30e/0x350
> [ 108.972117] blk_mq_alloc_queue+0x61/0xd0
> [ 108.974677] scsi_alloc_sdev+0x2a0/0x3e0
> [ 108.977092] scsi_probe_and_add_lun+0x1bd/0x430
> [ 108.979921] __scsi_add_device+0x109/0x120
> [ 108.982504] ata_scsi_scan_host+0x97/0x1c0
> [ 108.984365] async_run_entry_fn+0x2d/0x130
> [ 108.986109] process_one_work+0x20e/0x630
> [ 108.987830] worker_thread+0x184/0x330
> [ 108.989473] kthread+0x10a/0x250
> [ 108.990852] ret_from_fork+0x297/0x300
> [ 108.992491] ret_from_fork_asm+0x1a/0x30
> [ 108.994159]
> -> #2 (fs_reclaim){+.+.}-{0:0}:
> [ 108.996320] fs_reclaim_acquire+0x99/0xd0
> [ 108.998058] kmem_cache_alloc_node_noprof+0x4e/0x3c0
> [ 109.000123] __alloc_skb+0x15f/0x190
> [ 109.002195] tcp_send_active_reset+0x3f/0x1e0
> [ 109.004038] tcp_disconnect+0x50b/0x720
> [ 109.005695] __tcp_close+0x2b8/0x4b0
> [ 109.007227] tcp_close+0x20/0x80
> [ 109.008663] inet_release+0x31/0x60
> [ 109.010175] __sock_release+0x3a/0xc0
> [ 109.011778] sock_close+0x14/0x20
> [ 109.013263] __fput+0xee/0x2c0
> [ 109.014673] delayed_fput+0x31/0x50
> [ 109.016183] process_one_work+0x20e/0x630
> [ 109.017897] worker_thread+0x184/0x330
> [ 109.019543] kthread+0x10a/0x250
> [ 109.020929] ret_from_fork+0x297/0x300
> [ 109.022565] ret_from_fork_asm+0x1a/0x30
> [ 109.024194]
> -> #1 (sk_lock-AF_INET-NVME){+.+.}-{0:0}:
> [ 109.026634] lock_sock_nested+0x2e/0x70
> [ 109.028251] tcp_sendmsg+0x1a/0x40
> [ 109.029783] sock_sendmsg+0xed/0x110
> [ 109.031321] nvme_tcp_try_send_cmd_pdu+0x13e/0x260 [nvme_tcp]
> [ 109.034263] nvme_tcp_try_send+0xb3/0x330 [nvme_tcp]
> [ 109.036375] nvme_tcp_queue_rq+0x342/0x3d0 [nvme_tcp]
> [ 109.038528] blk_mq_dispatch_rq_list+0x297/0x800
> [ 109.040448] __blk_mq_sched_dispatch_requests+0x3db/0x5f0
> [ 109.042677] blk_mq_sched_dispatch_requests+0x29/0x70
> [ 109.044787] blk_mq_run_work_fn+0x76/0x1b0
> [ 109.046535] process_one_work+0x20e/0x630
> [ 109.048245] worker_thread+0x184/0x330
> [ 109.049890] kthread+0x10a/0x250
> [ 109.051331] ret_from_fork+0x297/0x300
> [ 109.053024] ret_from_fork_asm+0x1a/0x30
> [ 109.054740]
> -> #0 (set->srcu){.+.+}-{0:0}:
> [ 109.056850] __lock_acquire+0x1468/0x2210
> [ 109.058614] lock_sync+0xa5/0x110
> [ 109.060048] __synchronize_srcu+0x49/0x170
> [ 109.061802] elevator_switch+0xc9/0x330
> [ 109.063950] elevator_change+0x128/0x1c0
> [ 109.065675] elevator_set_none+0x4c/0x90
> [ 109.067316] blk_unregister_queue+0xa8/0x110
> [ 109.069165] __del_gendisk+0x14e/0x3c0
> [ 109.070824] del_gendisk+0x75/0xa0
> [ 109.072328] nvme_ns_remove+0xf2/0x230 [nvme_core]
> [ 109.074365] nvme_remove_namespaces+0xf2/0x150 [nvme_core]
> [ 109.076652] nvme_do_delete_ctrl+0x71/0x90 [nvme_core]
> [ 109.078775] nvme_delete_ctrl_sync+0x3b/0x50 [nvme_core]
> [ 109.081009] nvme_sysfs_delete+0x34/0x40 [nvme_core]
> [ 109.083082] kernfs_fop_write_iter+0x16a/0x220
> [ 109.085009] vfs_write+0x378/0x520
> [ 109.086539] ksys_write+0x67/0xe0
> [ 109.087982] do_syscall_64+0x76/0xbb0
> [ 109.089577] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 109.091665]
> other info that might help us debug this:
>
> [ 109.095478] Chain exists of:
> set->srcu --> &q->q_usage_counter(io) --> &q->elevator_lock
>
> [ 109.099544] Possible unsafe locking scenario:
>
> [ 109.101708] CPU0 CPU1
> [ 109.103402] ---- ----
> [ 109.105103] lock(&q->elevator_lock);
> [ 109.106530] lock(&q->q_usage_counter(io));
> [ 109.109022] lock(&q->elevator_lock);
> [ 109.111391] sync(set->srcu);
> [ 109.112586]
> *** DEADLOCK ***
>
> [ 109.114772] 5 locks held by nvme/2734:
> [ 109.116189] #0: ffff888101925410 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x67/0xe0
> [ 109.119143] #1: ffff88817a914e88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x10f/0x220
> [ 109.123141] #2: ffff8881046313f8 (kn->active#185){++++}-{0:0}, at: sysfs_remove_file_self+0x26/0x50
> [ 109.126543] #3: ffff88810470e1d0 (&set->update_nr_hwq_lock){++++}-{4:4}, at: del_gendisk+0x6d/0xa0
> [ 109.129891] #4: ffff88813abea198 (&q->elevator_lock){+.+.}-{4:4}, at: elevator_change+0xa8/0x1c0
> [ 109.133149]
> stack backtrace:
> [ 109.134817] CPU: 6 UID: 0 PID: 2734 Comm: nvme Tainted: G N 6.17.0nvme+ #3 PREEMPT(voluntary)
> [ 109.134819] Tainted: [N]=TEST
> [ 109.134820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
> [ 109.134821] Call Trace:
> [ 109.134823] <TASK>
> [ 109.134824] dump_stack_lvl+0x75/0xb0
> [ 109.134828] print_circular_bug+0x26a/0x330
> [ 109.134831] check_noncircular+0x12f/0x150
> [ 109.134834] __lock_acquire+0x1468/0x2210
> [ 109.134837] ? __synchronize_srcu+0x17/0x170
> [ 109.134838] lock_sync+0xa5/0x110
> [ 109.134840] ? __synchronize_srcu+0x17/0x170
> [ 109.134842] __synchronize_srcu+0x49/0x170
> [ 109.134843] ? mark_held_locks+0x49/0x80
> [ 109.134845] ? _raw_spin_unlock_irqrestore+0x2d/0x60
> [ 109.134847] ? kvm_clock_get_cycles+0x14/0x30
> [ 109.134853] ? ktime_get_mono_fast_ns+0x36/0xb0
> [ 109.134858] elevator_switch+0xc9/0x330
> [ 109.134860] elevator_change+0x128/0x1c0
> [ 109.134862] ? kernfs_put.part.0+0x86/0x290
> [ 109.134864] elevator_set_none+0x4c/0x90
> [ 109.134866] blk_unregister_queue+0xa8/0x110
> [ 109.134868] __del_gendisk+0x14e/0x3c0
> [ 109.134870] del_gendisk+0x75/0xa0
> [ 109.134872] nvme_ns_remove+0xf2/0x230 [nvme_core]
> [ 109.134879] nvme_remove_namespaces+0xf2/0x150 [nvme_core]
> [ 109.134887] nvme_do_delete_ctrl+0x71/0x90 [nvme_core]
> [ 109.134893] nvme_delete_ctrl_sync+0x3b/0x50 [nvme_core]
> [ 109.134899] nvme_sysfs_delete+0x34/0x40 [nvme_core]
> [ 109.134905] kernfs_fop_write_iter+0x16a/0x220
> [ 109.134908] vfs_write+0x378/0x520
> [ 109.134911] ksys_write+0x67/0xe0
> [ 109.134913] do_syscall_64+0x76/0xbb0
> [ 109.134915] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 109.134916] RIP: 0033:0x7fd68a737317
> [ 109.134917] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
> [ 109.134919] RSP: 002b:00007ffded1546d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> [ 109.134920] RAX: ffffffffffffffda RBX: 000000000054f7e0 RCX: 00007fd68a737317
> [ 109.134921] RDX: 0000000000000001 RSI: 00007fd68a855719 RDI: 0000000000000003
> [ 109.134921] RBP: 0000000000000003 R08: 0000000030407850 R09: 00007fd68a7cd4e0
> [ 109.134922] R10: 00007fd68a65b130 R11: 0000000000000246 R12: 00007fd68a855719
> [ 109.134923] R13: 00000000304074c0 R14: 00000000304074c0 R15: 0000000030408660
> [ 109.134926] </TASK>
> [ 109.962756] Key type psk unregistered
>
>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> ---
>
Reviewed-by: Hannes Reinecke <hare@suse.de>

Cheers,

Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich