[bug report] blktests block/022 failed with possible circular locking dependency detected at: __kernfs_remove+0x275/0x710 and del_gendisk+0xfd/0x190

[bug report] blktests block/022 failed with possible circular locking dependency detected at: __kernfs_remove+0x275/0x710 and del_gendisk+0xfd/0x190

[Yi Zhang]

Hi

I found blktests block/022 failed on the latest
linux-block/for-next[1], please help check it and let me know if you
need any info/testing for it, thanks.

[1]
commit b82c5e4f1bcb3fa78ba3c95a0d198f55553c0fc1
Merge: 187d9a5a18b4 d47f7c173262
Author: Jens Axboe <axboe@kernel.dk>
Date:   Mon Mar 2 14:04:16 2026 -0700

    Merge branch 'for-7.1/block' into for-next

dmesg:
[ 1329.227229] run blktests block/022 at 2026-03-03 13:51:07
[ 1329.838331] null_blk: module loaded
--snip--
[ 1349.577627] null_blk: disk 1 created
[ 1349.590297] ======================================================
[ 1349.596474] WARNING: possible circular locking dependency detected
[ 1349.602654] 7.0.0-rc2+ #1 Not tainted
[ 1349.606320] ------------------------------------------------------
[ 1349.612499] check/14895 is trying to acquire lock:
[ 1349.617291] ffff8883142961f8 (kn->active#104){++++}-{0:0}, at:
__kernfs_remove+0x275/0x710
[ 1349.625576]
               but task is already holding lock:
[ 1349.631410] ffffffffc1fe7db0
(&set->update_nr_hwq_lock){++++}-{4:4}, at: del_gendisk+0xfd/0x190
[ 1349.640130]
               which lock already depends on the new lock.

[ 1349.648300]
               the existing dependency chain (in reverse order) is:
[ 1349.655780]
               -> #1 (&set->update_nr_hwq_lock){++++}-{4:4}:
[ 1349.662670]        __lock_acquire+0x58c/0xbd0
[ 1349.667037]        lock_acquire.part.0+0xbd/0x260
[ 1349.671742]        down_read+0xa2/0x490
[ 1349.675581]        elv_iosched_store+0x31e/0x3b0
[ 1349.680203]        queue_attr_store+0x232/0x2f0
[ 1349.684741]        kernfs_fop_write_iter+0x3df/0x5f0
[ 1349.689711]        vfs_write+0x525/0xfd0
[ 1349.693644]        ksys_write+0xf9/0x1d0
[ 1349.697578]        do_syscall_64+0x13a/0x1520
[ 1349.701947]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1349.707528]
               -> #0 (kn->active#104){++++}-{0:0}:
[ 1349.713551]        check_prev_add+0xf1/0xc80
[ 1349.717832]        validate_chain+0x481/0x560
[ 1349.722200]        __lock_acquire+0x58c/0xbd0
[ 1349.726568]        lock_acquire.part.0+0xbd/0x260
[ 1349.731281]        kernfs_drain+0x3cf/0x4a0
[ 1349.735466]        __kernfs_remove+0x275/0x710
[ 1349.739914]        kernfs_remove_by_name_ns+0x9c/0x100
[ 1349.745062]        remove_files+0x8c/0x1a0
[ 1349.749170]        sysfs_remove_group+0x7b/0x170
[ 1349.753797]        sysfs_remove_groups+0x53/0xa0
[ 1349.758424]        __kobject_del+0x7d/0x1e0
[ 1349.762610]        kobject_del+0x35/0x50
[ 1349.766536]        blk_unregister_queue+0x112/0x2b0
[ 1349.771417]        __del_gendisk+0x265/0x9e0
[ 1349.775696]        del_gendisk+0x105/0x190
[ 1349.779796]        null_del_dev.part.0+0x165/0x480 [null_blk]
[ 1349.785551]        nullb_device_power_store+0x1a2/0x280 [null_blk]
[ 1349.791737]        configfs_write_iter+0x2ac/0x460
[ 1349.796530]        vfs_write+0x525/0xfd0
[ 1349.800456]        ksys_write+0xf9/0x1d0
[ 1349.804383]        do_syscall_64+0x13a/0x1520
[ 1349.808742]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1349.814315]
               other info that might help us debug this:

[ 1349.822314]  Possible unsafe locking scenario:

[ 1349.828232]        CPU0                    CPU1
[ 1349.832766]        ----                    ----
[ 1349.837296]   rlock(&set->update_nr_hwq_lock);
[ 1349.841752]                                lock(kn->active#104);
[ 1349.847767]                                lock(&set->update_nr_hwq_lock);
[ 1349.854647]   lock(kn->active#104);
[ 1349.858148]
                *** DEADLOCK ***

[ 1349.864068] 5 locks held by check/14895:
[ 1349.867994]  #0: ffff8882807d0440 (sb_writers#16){.+.+}-{0:0}, at:
ksys_write+0xf9/0x1d0
[ 1349.876106]  #1: ffff8881c5d6fa90 (&buffer->mutex){+.+.}-{4:4}, at:
configfs_write_iter+0x71/0x460
[ 1349.885085]  #2: ffff8881a17a6280 (&p->frag_sem){++++}-{4:4}, at:
configfs_write_iter+0x1d9/0x460
[ 1349.893975]  #3: ffffffffc1fe7eb0 (&lock){+.+.}-{4:4}, at:
nullb_device_power_store+0xab/0x280 [null_blk]
[ 1349.903562]  #4: ffffffffc1fe7db0
(&set->update_nr_hwq_lock){++++}-{4:4}, at: del_gendisk+0xfd/0x190
[ 1349.912712]
               stack backtrace:
[ 1349.917076] CPU: 1 UID: 0 PID: 14895 Comm: check Not tainted
7.0.0-rc2+ #1 PREEMPT(full)
[ 1349.917082] Hardware name: Dell Inc. PowerEdge R6515/07PXPY, BIOS
2.21.1 09/24/2025
[ 1349.917086] Call Trace:
[ 1349.917090]  <TASK>
[ 1349.917094]  dump_stack_lvl+0x6f/0xb0
[ 1349.917103]  print_circular_bug.cold+0x38/0x45
[ 1349.917113]  check_noncircular+0x146/0x160
[ 1349.917130]  check_prev_add+0xf1/0xc80
[ 1349.917134]  ? alloc_chain_hlocks+0x8/0x1d0
[ 1349.917139]  ? srso_return_thunk+0x5/0x5f
[ 1349.917144]  ? add_chain_cache+0x11c/0x300
[ 1349.917152]  validate_chain+0x481/0x560
[ 1349.917157]  ? srso_return_thunk+0x5/0x5f
[ 1349.917162]  ? mark_lock+0x2e3/0x3d0
[ 1349.917172]  __lock_acquire+0x58c/0xbd0
[ 1349.917178]  ? srso_return_thunk+0x5/0x5f
[ 1349.917187]  lock_acquire.part.0+0xbd/0x260
[ 1349.917192]  ? __kernfs_remove+0x275/0x710
[ 1349.917201]  ? srso_return_thunk+0x5/0x5f
[ 1349.917205]  ? rcu_is_watching+0x15/0xb0
[ 1349.917211]  ? srso_return_thunk+0x5/0x5f
[ 1349.917215]  ? lock_acquire+0x159/0x180
[ 1349.917225]  kernfs_drain+0x3cf/0x4a0
[ 1349.917231]  ? __kernfs_remove+0x275/0x710
[ 1349.917239]  ? __pfx_kernfs_drain+0x10/0x10
[ 1349.917245]  ? srso_return_thunk+0x5/0x5f
[ 1349.917249]  ? __lock_release.isra.0+0x1a2/0x2c0
[ 1349.917258]  ? kernfs_root+0xb2/0x1c0
[ 1349.917263]  ? srso_return_thunk+0x5/0x5f
[ 1349.917269]  ? srso_return_thunk+0x5/0x5f
[ 1349.917274]  ? kernfs_root+0xbc/0x1c0
[ 1349.917283]  __kernfs_remove+0x275/0x710
[ 1349.917289]  ? kernfs_find_ns+0x1a1/0x390
[ 1349.917299]  kernfs_remove_by_name_ns+0x9c/0x100
[ 1349.917307]  remove_files+0x8c/0x1a0
[ 1349.917316]  sysfs_remove_group+0x7b/0x170
[ 1349.917324]  sysfs_remove_groups+0x53/0xa0
[ 1349.917332]  __kobject_del+0x7d/0x1e0
[ 1349.917340]  kobject_del+0x35/0x50
[ 1349.917346]  blk_unregister_queue+0x112/0x2b0
[ 1349.917354]  __del_gendisk+0x265/0x9e0
[ 1349.917362]  ? down_read+0xbf/0x490
[ 1349.917367]  ? local_clock_noinstr+0xd/0xe0
[ 1349.917374]  ? __pfx___del_gendisk+0x10/0x10
[ 1349.917378]  ? __lock_release.isra.0+0x1a2/0x2c0
[ 1349.917384]  ? srso_return_thunk+0x5/0x5f
[ 1349.917390]  ? srso_return_thunk+0x5/0x5f
[ 1349.917395]  ? __up_write+0x27e/0x4e0
[ 1349.917400]  ? del_gendisk+0xb4/0x190
[ 1349.917409]  del_gendisk+0x105/0x190
[ 1349.917417]  null_del_dev.part.0+0x165/0x480 [null_blk]
[ 1349.917434]  nullb_device_power_store+0x1a2/0x280 [null_blk]
[ 1349.917448]  ? __pfx_nullb_device_power_store+0x10/0x10 [null_blk]
[ 1349.917460]  ? srso_untrain_ret+0x2/0x2
[ 1349.917473]  configfs_write_iter+0x2ac/0x460
[ 1349.917484]  vfs_write+0x525/0xfd0
[ 1349.917495]  ? __pfx_vfs_write+0x10/0x10
[ 1349.917507]  ? srso_return_thunk+0x5/0x5f
[ 1349.917511]  ? __lock_acquire+0x58c/0xbd0
[ 1349.917521]  ? srso_return_thunk+0x5/0x5f
[ 1349.917526]  ? srso_return_thunk+0x5/0x5f
[ 1349.917530]  ? find_held_lock+0x32/0x90
[ 1349.917541]  ksys_write+0xf9/0x1d0
[ 1349.917547]  ? __pfx_ksys_write+0x10/0x10
[ 1349.917553]  ? srso_return_thunk+0x5/0x5f
[ 1349.917557]  ? lockdep_hardirqs_on+0x78/0x100
[ 1349.917563]  ? srso_return_thunk+0x5/0x5f
[ 1349.917568]  ? do_syscall_64+0xde/0x1520
[ 1349.917577]  do_syscall_64+0x13a/0x1520
[ 1349.917582]  ? rcu_is_watching+0x15/0xb0
[ 1349.917589]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1349.917593]  ? srso_return_thunk+0x5/0x5f
[ 1349.917598]  ? lockdep_hardirqs_on+0x78/0x100
[ 1349.917603]  ? srso_return_thunk+0x5/0x5f
[ 1349.917607]  ? do_syscall_64+0x212/0x1520
[ 1349.917616]  ? srso_return_thunk+0x5/0x5f
[ 1349.917620]  ? __lock_acquire+0x58c/0xbd0
[ 1349.917630]  ? srso_return_thunk+0x5/0x5f
[ 1349.917634]  ? lock_acquire.part.0+0xbd/0x260
[ 1349.917641]  ? srso_return_thunk+0x5/0x5f
[ 1349.917645]  ? find_held_lock+0x32/0x90
[ 1349.917651]  ? set_close_on_exec+0x5c/0x1d0
[ 1349.917658]  ? srso_return_thunk+0x5/0x5f
[ 1349.917662]  ? srso_return_thunk+0x5/0x5f
[ 1349.917667]  ? find_held_lock+0x32/0x90
[ 1349.917672]  ? srso_return_thunk+0x5/0x5f
[ 1349.917676]  ? local_clock_noinstr+0xd/0xe0
[ 1349.917682]  ? srso_return_thunk+0x5/0x5f
[ 1349.917687]  ? __lock_release.isra.0+0x1a2/0x2c0
[ 1349.917693]  ? do_fcntl+0x58e/0xde0
[ 1349.917701]  ? srso_return_thunk+0x5/0x5f
[ 1349.917705]  ? do_raw_spin_unlock+0x58/0x1f0
[ 1349.917712]  ? srso_return_thunk+0x5/0x5f
[ 1349.917716]  ? _raw_spin_unlock+0x2d/0x50
[ 1349.917722]  ? srso_return_thunk+0x5/0x5f
[ 1349.917726]  ? do_fcntl+0x58e/0xde0
[ 1349.917733]  ? __pfx_do_fcntl+0x10/0x10
[ 1349.917741]  ? srso_return_thunk+0x5/0x5f
[ 1349.917746]  ? srso_return_thunk+0x5/0x5f
[ 1349.917750]  ? find_held_lock+0x32/0x90
[ 1349.917759]  ? __lock_release.isra.0+0x1a2/0x2c0
[ 1349.917765]  ? srso_return_thunk+0x5/0x5f
[ 1349.917769]  ? srso_return_thunk+0x5/0x5f
[ 1349.917773]  ? selinux_file_fcntl+0x1c/0xf0
[ 1349.917782]  ? srso_return_thunk+0x5/0x5f
[ 1349.917787]  ? rcu_is_watching+0x15/0xb0
[ 1349.917793]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1349.917797]  ? srso_return_thunk+0x5/0x5f
[ 1349.917802]  ? lockdep_hardirqs_on+0x78/0x100
[ 1349.917807]  ? srso_return_thunk+0x5/0x5f
[ 1349.917812]  ? do_syscall_64+0x212/0x1520
[ 1349.917816]  ? srso_return_thunk+0x5/0x5f
[ 1349.917821]  ? do_syscall_64+0x212/0x1520
[ 1349.917825]  ? srso_return_thunk+0x5/0x5f
[ 1349.917830]  ? do_syscall_64+0x212/0x1520
[ 1349.917834]  ? srso_return_thunk+0x5/0x5f
[ 1349.917839]  ? do_syscall_64+0x212/0x1520
[ 1349.917849]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1349.917854] RIP: 0033:0x7f18a1d9d544
[ 1349.917860] Code: 89 02 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00
00 00 0f 1f 40 00 f3 0f 1e fa 80 3d a5 cb 0d 00 00 74 13 b8 01 00 00
00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24
18 48
[ 1349.917864] RSP: 002b:00007ffcc4f4aca8 EFLAGS: 00000202 ORIG_RAX:
0000000000000001
[ 1349.917870] RAX: ffffffffffffffda RBX: 00007f18a1e735c0 RCX: 00007f18a1d9d544
[ 1349.917873] RDX: 0000000000000002 RSI: 00005646e17c3970 RDI: 0000000000000001
[ 1349.917877] RBP: 0000000000000002 R08: 0000000000000073 R09: 00000000ffffffff
[ 1349.917880] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
[ 1349.917883] R13: 00005646e17c3970 R14: 0000000000000002 R15: 00007f18a1e70f00
[ 1349.917902]  </TASK>


-- 
Best Regards,
  Yi Zhang

Re: [bug report] blktests block/022 failed with possible circular locking dependency detected at: __kernfs_remove+0x275/0x710 and del_gendisk+0xfd/0x190

[Ming Lei]

On Wed, Mar 04, 2026 at 03:10:06PM +0800, Yi Zhang wrote:
> Hi
> 
> I found blktests block/022 failed on the latest
> linux-block/for-next[1], please help check it and let me know if you
> need any info/testing for it, thanks.
> 
> [1]
> commit b82c5e4f1bcb3fa78ba3c95a0d198f55553c0fc1
> Merge: 187d9a5a18b4 d47f7c173262
> Author: Jens Axboe <axboe@kernel.dk>
> Date:   Mon Mar 2 14:04:16 2026 -0700
> 
>     Merge branch 'for-7.1/block' into for-next
> 
> dmesg:
> [ 1329.227229] run blktests block/022 at 2026-03-03 13:51:07
> [ 1329.838331] null_blk: module loaded
> --snip--

...

> [ 1349.822314]  Possible unsafe locking scenario:
> 
> [ 1349.828232]        CPU0                    CPU1
> [ 1349.832766]        ----                    ----
> [ 1349.837296]   rlock(&set->update_nr_hwq_lock);
> [ 1349.841752]                                lock(kn->active#104);
> [ 1349.847767]                                lock(&set->update_nr_hwq_lock);
> [ 1349.854647]   lock(kn->active#104);
> [ 1349.858148]
>                 *** DEADLOCK ***

Looks one real deadlock risk, can you try the following patch?


diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index f3b1968c80ce..55a1bbfef7d4 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -78,8 +78,14 @@ queue_requests_store(struct gendisk *disk, const char *page, size_t count)
 	/*
 	 * Serialize updating nr_requests with concurrent queue_requests_store()
 	 * and switching elevator.
+	 *
+	 * Use trylock to avoid circular lock dependency with kernfs active
+	 * reference during concurrent disk deletion:
+	 *   update_nr_hwq_lock -> kn->active (via del_gendisk -> kobject_del)
+	 *   kn->active -> update_nr_hwq_lock (via this sysfs write path)
 	 */
-	down_write(&set->update_nr_hwq_lock);
+	if (!down_write_trylock(&set->update_nr_hwq_lock))
+		return -EBUSY;
 
 	if (nr == q->nr_requests)
 		goto unlock;
diff --git a/block/elevator.c b/block/elevator.c
index ebe2a1fcf011..3bcd37c2aa34 100644
--- a/block/elevator.c
+++ b/block/elevator.c
@@ -807,7 +807,16 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,
 	elv_iosched_load_module(ctx.name);
 	ctx.type = elevator_find_get(ctx.name);
 
-	down_read(&set->update_nr_hwq_lock);
+	/*
+	 * Use trylock to avoid circular lock dependency with kernfs active
+	 * reference during concurrent disk deletion:
+	 *   update_nr_hwq_lock -> kn->active (via del_gendisk -> kobject_del)
+	 *   kn->active -> update_nr_hwq_lock (via this sysfs write path)
+	 */
+	if (!down_read_trylock(&set->update_nr_hwq_lock)) {
+		ret = -EBUSY;
+		goto out;
+	}
 	if (!blk_queue_no_elv_switch(q)) {
 		ret = elevator_change(q, &ctx);
 		if (!ret)
@@ -817,6 +826,7 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,
 	}
 	up_read(&set->update_nr_hwq_lock);
 
+out:
 	if (ctx.type)
 		elevator_put(ctx.type);
 	return ret;


Thanks,
Ming

Re: [bug report] blktests block/022 failed with possible circular locking dependency detected at: __kernfs_remove+0x275/0x710 and del_gendisk+0xfd/0x190

[Yi Zhang]

On Wed, Mar 4, 2026 at 6:16 PM Ming Lei <ming.lei@redhat.com> wrote:
>
> On Wed, Mar 04, 2026 at 03:10:06PM +0800, Yi Zhang wrote:
> > Hi
> >
> > I found blktests block/022 failed on the latest
> > linux-block/for-next[1], please help check it and let me know if you
> > need any info/testing for it, thanks.
> >
> > [1]
> > commit b82c5e4f1bcb3fa78ba3c95a0d198f55553c0fc1
> > Merge: 187d9a5a18b4 d47f7c173262
> > Author: Jens Axboe <axboe@kernel.dk>
> > Date:   Mon Mar 2 14:04:16 2026 -0700
> >
> >     Merge branch 'for-7.1/block' into for-next
> >
> > dmesg:
> > [ 1329.227229] run blktests block/022 at 2026-03-03 13:51:07
> > [ 1329.838331] null_blk: module loaded
> > --snip--
>
> ...
>
> > [ 1349.822314]  Possible unsafe locking scenario:
> >
> > [ 1349.828232]        CPU0                    CPU1
> > [ 1349.832766]        ----                    ----
> > [ 1349.837296]   rlock(&set->update_nr_hwq_lock);
> > [ 1349.841752]                                lock(kn->active#104);
> > [ 1349.847767]                                lock(&set->update_nr_hwq_lock);
> > [ 1349.854647]   lock(kn->active#104);
> > [ 1349.858148]
> >                 *** DEADLOCK ***
>
> Looks one real deadlock risk, can you try the following patch?
>


Yes, the issue was fixed with the patch, thanks.

Tested-by: Yi Zhang <yi.zhang@redhat.com>

>
> diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
> index f3b1968c80ce..55a1bbfef7d4 100644
> --- a/block/blk-sysfs.c
> +++ b/block/blk-sysfs.c
> @@ -78,8 +78,14 @@ queue_requests_store(struct gendisk *disk, const char *page, size_t count)
>         /*
>          * Serialize updating nr_requests with concurrent queue_requests_store()
>          * and switching elevator.
> +        *
> +        * Use trylock to avoid circular lock dependency with kernfs active
> +        * reference during concurrent disk deletion:
> +        *   update_nr_hwq_lock -> kn->active (via del_gendisk -> kobject_del)
> +        *   kn->active -> update_nr_hwq_lock (via this sysfs write path)
>          */
> -       down_write(&set->update_nr_hwq_lock);
> +       if (!down_write_trylock(&set->update_nr_hwq_lock))
> +               return -EBUSY;
>
>         if (nr == q->nr_requests)
>                 goto unlock;
> diff --git a/block/elevator.c b/block/elevator.c
> index ebe2a1fcf011..3bcd37c2aa34 100644
> --- a/block/elevator.c
> +++ b/block/elevator.c
> @@ -807,7 +807,16 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,
>         elv_iosched_load_module(ctx.name);
>         ctx.type = elevator_find_get(ctx.name);
>
> -       down_read(&set->update_nr_hwq_lock);
> +       /*
> +        * Use trylock to avoid circular lock dependency with kernfs active
> +        * reference during concurrent disk deletion:
> +        *   update_nr_hwq_lock -> kn->active (via del_gendisk -> kobject_del)
> +        *   kn->active -> update_nr_hwq_lock (via this sysfs write path)
> +        */
> +       if (!down_read_trylock(&set->update_nr_hwq_lock)) {
> +               ret = -EBUSY;
> +               goto out;
> +       }
>         if (!blk_queue_no_elv_switch(q)) {
>                 ret = elevator_change(q, &ctx);
>                 if (!ret)
> @@ -817,6 +826,7 @@ ssize_t elv_iosched_store(struct gendisk *disk, const char *buf,
>         }
>         up_read(&set->update_nr_hwq_lock);
>
> +out:
>         if (ctx.type)
>                 elevator_put(ctx.type);
>         return ret;
>
>
> Thanks,
> Ming
>


-- 
Best Regards,
  Yi Zhang