Re: [PATCH 00/33 v3] cpuset/isolation: Honour kthreads preferred affinity

Johannes Thumshirn
2025-10-14T12:17:20Z

On 10/14/25 1:29 PM, shinichiro.kawasaki@wdc.com wrote:
> Dear patch submitter,
>
> Blktests CI has tested the following submission:
> Status: FAILURE
> Name: [00/33,v3] cpuset/isolation: Honour kthreads preferred affinity
> Patchwork: https://patchwork.kernel.org/project/linux-block/list/?series=1010948&state=*
> Run record: https://github.com/linux-blktests/linux-block/actions/runs/18488587693
>
>
> Failed test cases: nvme/005
>
>
>
The testcase failure looks unrelated to me:

[ 4886.107169] nvmet: Created nvm controller 2 for subsystem
blktests-subsystem-1 for NQN
nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[ 4886.108997] WARNING: possible circular locking dependency detected
[ 4886.109010] 6.18.0-rc1 #1 Tainted: G            E
[ 4886.109018] ------------------------------------------------------
[ 4886.109024] kworker/3:2/13970 is trying to acquire lock:
[ 4886.109032] ffffffff8e5e1c00 (fs_reclaim){+.+.}-{0:0}, at:
kmem_cache_alloc_node_noprof+0x61/0x7e0
[ 4886.116754] nvme nvme2: creating 4 I/O queues.
[ 4886.117019]
but task is already holding lock:
[ 4886.117025] ffff888108367518 (sk_lock-AF_INET-NVME){+.+.}-{0:0}, at:
tcp_close+0x18/0xc0
[ 4886.130429]
which lock already depends on the new lock.

[ 4886.133999]
the existing dependency chain (in reverse order) is:
[ 4886.135777]
-> #4 (sk_lock-AF_INET-NVME){+.+.}-{0:0}:
[ 4886.137477]        __lock_acquire+0x56d/0xc20
[ 4886.138439]        lock_acquire.part.0+0xb7/0x220
[ 4886.139386]        lock_sock_nested+0x3b/0xe0
[ 4886.140330]        tcp_sendmsg+0x1d/0x40
[ 4886.141239]        sock_sendmsg+0x2fb/0x410
[ 4886.142139]        nvme_tcp_try_send_cmd_pdu+0x5a0/0xc10 [nvme_tcp]
[ 4886.143208]        nvme_tcp_try_send+0x1b6/0x9b0 [nvme_tcp]
[ 4886.144220]        nvme_tcp_queue_rq+0x449/0x6d0 [nvme_tcp]
[ 4886.145223]        blk_mq_dispatch_rq_list+0x38f/0x1450
[ 4886.146214]        __blk_mq_sched_dispatch_requests+0x1a9/0x4b0
[ 4886.147231]        blk_mq_sched_dispatch_requests+0xac/0x150
[ 4886.148264]        blk_mq_run_work_fn+0x125/0x2b0
[ 4886.149176]        process_one_work+0x854/0x1460
[ 4886.150073]        worker_thread+0x5f3/0xfe0
[ 4886.150915]        kthread+0x3a5/0x760
[ 4886.151712]        ret_from_fork+0x2d3/0x3d0
[ 4886.152577]        ret_from_fork_asm+0x1a/0x30
[ 4886.153428]
-> #3 (set->srcu){.+.+}-{0:0}:
[ 4886.154836]        __lock_acquire+0x56d/0xc20
[ 4886.155673]        lock_sync.part.0+0x41/0x80
[ 4886.156513]        __synchronize_srcu+0xa7/0x240
[ 4886.157367]        elevator_switch+0x2c7/0x650
[ 4886.158196]        elevator_change+0x2db/0x550
[ 4886.159019]        elevator_set_none+0xa3/0xf0
[ 4886.159836]        blk_unregister_queue+0x144/0x2c0
[ 4886.160684]        __del_gendisk+0x265/0xa20
[ 4886.161466]        del_gendisk+0x105/0x190
[ 4886.162228]        loop_add+0x5f5/0xb10 [loop]
[ 4886.163029]        loop_validate_file+0xc5/0x410 [loop]
[ 4886.163951]        configfs_rmdir+0x69f/0xac0
[ 4886.164748]        vfs_rmdir+0x1a5/0x5b0
[ 4886.165492]        do_rmdir+0x276/0x330
[ 4886.166203]        __x64_sys_unlinkat+0x16b/0x1e0
[ 4886.166978]        do_syscall_64+0x97/0x800
[ 4886.167738]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 4886.168585]
-> #2 (&q->elevator_lock){+.+.}-{4:4}:
[ 4886.169841]        __lock_acquire+0x56d/0xc20
[ 4886.170575]        lock_acquire.part.0+0xb7/0x220
[ 4886.171331]        __mutex_lock+0x1a6/0x2110
[ 4886.172061]        elevator_change+0x12d/0x550
[ 4886.172788]        elv_iosched_store+0x27b/0x2f0
[ 4886.173522]        queue_attr_store+0x232/0x2f0
[ 4886.174244]        kernfs_fop_write_iter+0x3df/0x5f0
[ 4886.175004]        vfs_write+0x518/0x1090
[ 4886.175697]        ksys_write+0xf9/0x1d0
[ 4886.176392]        do_syscall_64+0x97/0x800
[ 4886.177086]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 4886.177894]
-> #1 (&q->q_usage_counter(io)#4){++++}-{0:0}:
[ 4886.179155]        __lock_acquire+0x56d/0xc20
[ 4886.179861]        lock_acquire.part.0+0xb7/0x220
[ 4886.180611]        blk_alloc_queue+0x5c0/0x700
[ 4886.181328]        blk_mq_alloc_queue+0x14b/0x230
[ 4886.182061]        scsi_alloc_sdev+0x841/0xc60
[ 4886.182795]        scsi_probe_and_add_lun+0x4d6/0xbb0
[ 4886.183566]        __scsi_add_device+0x1c4/0x1f0
[ 4886.184313]        ata_scsi_scan_host+0x13d/0x390
[ 4886.185059]        async_run_entry_fn+0x9a/0x4f0
[ 4886.185809]        process_one_work+0x854/0x1460
[ 4886.186538]        worker_thread+0x5f3/0xfe0
[ 4886.187233]        kthread+0x3a5/0x760
[ 4886.187885]        ret_from_fork+0x2d3/0x3d0
[ 4886.188597]        ret_from_fork_asm+0x1a/0x30
[ 4886.189308]
-> #0 (fs_reclaim){+.+.}-{0:0}:
[ 4886.190445]        check_prev_add+0xeb/0xca0
[ 4886.191144]        validate_chain+0x46e/0x560
[ 4886.191845]        __lock_acquire+0x56d/0xc20
[ 4886.192558]        lock_acquire.part.0+0xb7/0x220
[ 4886.193287]        fs_reclaim_acquire+0xc9/0x110
[ 4886.194018]        kmem_cache_alloc_node_noprof+0x61/0x7e0
[ 4886.194811]        __alloc_skb+0x1e6/0x2d0
[ 4886.195502]        tcp_send_active_reset+0x8a/0x700
[ 4886.196261]        tcp_disconnect+0x1689/0x1f20
[ 4886.196982]        __tcp_close+0x7d7/0xee0
[ 4886.197665]        tcp_close+0x23/0xc0
[ 4886.198323]        inet_release+0x109/0x230
[ 4886.199021]        __sock_release+0xa6/0x250
[ 4886.199720]        sock_close+0x18/0x20
[ 4886.200393]        __fput+0x372/0xaa0
[ 4886.201039]        delayed_fput+0x6d/0x90
[ 4886.201712]        process_one_work+0x854/0x1460
[ 4886.202438]        worker_thread+0x5f3/0xfe0
[ 4886.203131]        kthread+0x3a5/0x760
[ 4886.203785]        ret_from_fork+0x2d3/0x3d0
[ 4886.204498]        ret_from_fork_asm+0x1a/0x30
[ 4886.205201]
other info that might help us debug this:

[ 4886.206825] Chain exists of:
fs_reclaim --> set->srcu --> sk_lock-AF_INET-NVME

[ 4886.208710]  Possible unsafe locking scenario:

[ 4886.209881]        CPU0                    CPU1
[ 4886.210600]        ----                    ----
[ 4886.211315]   lock(sk_lock-AF_INET-NVME);
[ 4886.212016]                                lock(set->srcu);
[ 4886.212818] lock(sk_lock-AF_INET-NVME);
[ 4886.213692]   lock(fs_reclaim);
[ 4886.214303]
*** DEADLOCK ***

[ 4886.215761] 4 locks held by kworker/3:2/13970:
[ 4886.216476]  #0: ffff88810020f548
((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0xf11/0x1460
[ 4886.217600]  #1: ffff8881e326fd10
((delayed_fput_work).work){+.+.}-{0:0}, at: process_one_work+0x7dc/0x1460
[ 4886.218749]  #2: ffff88811caa87c8
(&sb->s_type->i_mutex_key#9){+.+.}-{4:4}, at: __sock_release+0x7b/0x250
[ 4886.219893]  #3: ffff888108367518 (sk_lock-AF_INET-NVME){+.+.}-{0:0},
at: tcp_close+0x18/0xc0
[ 4886.220966]