From: kernel test robot <oliver.sang@intel.com>
To: Yu Kuai <yukuai@fnnas.com>
Cc: <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
<linux-block@vger.kernel.org>, <axboe@kernel.dk>,
<linux-kernel@vger.kernel.org>, <tj@kernel.org>,
<nilay@linux.ibm.com>, <ming.lei@redhat.com>, <yukuai@fnnas.com>,
<oliver.sang@intel.com>
Subject: Re: [PATCH RESEND 2/5] blk-wbt: fix incorrect lock order for rq_qos_mutex and freeze queue
Date: Wed, 19 Nov 2025 14:55:52 +0800 [thread overview]
Message-ID: <202511191340.643afc3a-lkp@intel.com> (raw)
In-Reply-To: <20251116041024.120500-3-yukuai@fnnas.com>
Hello,
kernel test robot noticed "WARNING:possible_circular_locking_dependency_detected" on:
commit: 9b76049c7ab17a3352a58ee216f444769e216c5c ("[PATCH RESEND 2/5] blk-wbt: fix incorrect lock order for rq_qos_mutex and freeze queue")
url: https://github.com/intel-lab-lkp/linux/commits/Yu-Kuai/block-blk-rq-qos-add-a-new-helper-rq_qos_add_freezed/20251116-121353
base: https://git.kernel.org/cgit/linux/kernel/git/axboe/linux.git for-next
patch link: https://lore.kernel.org/all/20251116041024.120500-3-yukuai@fnnas.com/
patch subject: [PATCH RESEND 2/5] blk-wbt: fix incorrect lock order for rq_qos_mutex and freeze queue
in testcase: boot
config: x86_64-rhel-9.4-kselftests
compiler: gcc-14
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 32G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202511191340.643afc3a-lkp@intel.com
[ 36.217408][ T108] WARNING: possible circular locking dependency detected
[ 36.218277][ T108] 6.18.0-rc5-00238-g9b76049c7ab1 #1 Not tainted
[ 36.219067][ T108] ------------------------------------------------------
[ 36.219956][ T108] udevd/108 is trying to acquire lock:
[ 36.220622][ T108] ffff88813b4b6a40 (&q->debugfs_mutex){+.+.}-{4:4}, at: rq_qos_add_freezed (block/blk-rq-qos.c:345)
[ 36.221938][ T108]
[ 36.221938][ T108] but task is already holding lock:
[ 36.222851][ T108] ffff88813b4b63e0 (&q->rq_qos_mutex){+.+.}-{4:4}, at: wbt_init (block/blk-wbt.c:929)
[ 36.223964][ T108]
[ 36.223964][ T108] which lock already depends on the new lock.
[ 36.223964][ T108]
[ 36.225282][ T108]
[ 36.225282][ T108] the existing dependency chain (in reverse order) is:
[ 36.226380][ T108]
[ 36.226380][ T108] -> #4 (&q->rq_qos_mutex){+.+.}-{4:4}:
[ 36.228833][ T108] __lock_acquire (kernel/locking/lockdep.c:5237 (discriminator 1))
[ 36.230959][ T108] lock_acquire (include/linux/preempt.h:471 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) kernel/locking/lockdep.c:5831 (discriminator 2))
[ 36.233086][ T108] __mutex_lock (arch/x86/include/asm/jump_label.h:36 include/trace/events/lock.h:95 kernel/locking/mutex.c:600 kernel/locking/mutex.c:760)
[ 36.234852][ T108] wbt_init (block/blk-wbt.c:929)
[ 36.236509][ T108] wbt_enable_default (include/linux/blk-mq.h:960 block/blk-wbt.c:731)
[ 36.238663][ T108] blk_register_queue (block/blk-sysfs.c:948)
[ 36.240650][ T108] __add_disk (block/genhd.c:528)
[ 36.242626][ T108] add_disk_fwnode (block/genhd.c:598)
[ 36.244660][ T108] sr_probe (drivers/scsi/sr.c:703) sr_mod
[ 36.246799][ T108] really_probe (drivers/base/dd.c:581 drivers/base/dd.c:659)
[ 36.248845][ T108] __driver_probe_device (drivers/base/dd.c:801)
[ 36.250799][ T108] driver_probe_device (drivers/base/dd.c:831)
[ 36.252778][ T108] __driver_attach (drivers/base/dd.c:1218)
[ 36.254752][ T108] bus_for_each_dev (drivers/base/bus.c:369)
[ 36.256776][ T108] bus_add_driver (drivers/base/bus.c:678)
[ 36.258650][ T108] driver_register (drivers/base/driver.c:249)
[ 36.261124][ T108] init_sr (drivers/scsi/sr.c:152) sr_mod
[ 36.262938][ T108] do_one_initcall (init/main.c:1283)
[ 36.264802][ T108] do_init_module (kernel/module/main.c:3039)
[ 36.266560][ T108] load_module (kernel/module/main.c:3509)
[ 36.268367][ T108] init_module_from_file (kernel/module/main.c:3701)
[ 36.270150][ T108] idempotent_init_module (kernel/module/main.c:3713)
[ 36.272050][ T108] __x64_sys_finit_module (include/linux/file.h:62 (discriminator 1) include/linux/file.h:83 (discriminator 1) kernel/module/main.c:3736 (discriminator 1) kernel/module/main.c:3723 (discriminator 1) kernel/module/main.c:3723 (discriminator 1))
[ 36.273921][ T108] do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1) arch/x86/entry/syscall_64.c:94 (discriminator 1))
[ 36.275672][ T108] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[ 36.277584][ T108]
[ 36.277584][ T108] -> #3 (&q->q_usage_counter(io)){++++}-{0:0}:
[ 36.280765][ T108] __lock_acquire (kernel/locking/lockdep.c:5237 (discriminator 1))
[ 36.282560][ T108] lock_acquire (include/linux/preempt.h:471 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) kernel/locking/lockdep.c:5831 (discriminator 2))
[ 36.284435][ T108] blk_alloc_queue (block/blk-core.c:462 (discriminator 1))
[ 36.286153][ T108] blk_mq_alloc_queue (block/blk-mq.c:4405)
[ 36.287937][ T108] scsi_alloc_sdev (drivers/scsi/scsi_scan.c:339)
[ 36.289536][ T108] scsi_probe_and_add_lun (drivers/scsi/scsi_scan.c:1211)
[ 36.291280][ T108] __scsi_add_device (drivers/scsi/scsi_scan.c:1625)
[ 36.292952][ T108] ata_scsi_scan_host (drivers/ata/libata-scsi.c:4577 (discriminator 1)) libata
[ 36.295269][ T108] async_run_entry_fn (arch/x86/include/asm/jump_label.h:36 kernel/async.c:131)
[ 36.297162][ T108] process_one_work (arch/x86/include/asm/jump_label.h:36 include/trace/events/workqueue.h:110 kernel/workqueue.c:3268)
[ 36.299000][ T108] worker_thread (kernel/workqueue.c:3340 (discriminator 2) kernel/workqueue.c:3427 (discriminator 2))
[ 36.300695][ T108] kthread (kernel/kthread.c:463)
[ 36.302155][ T108] ret_from_fork (arch/x86/kernel/process.c:164)
[ 36.303743][ T108] ret_from_fork_asm (arch/x86/entry/entry_64.S:255)
[ 36.305346][ T108]
[ 36.305346][ T108] -> #2 (fs_reclaim){+.+.}-{0:0}:
[ 36.307985][ T108] __lock_acquire (kernel/locking/lockdep.c:5237 (discriminator 1))
[ 36.309621][ T108] lock_acquire (include/linux/preempt.h:471 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) kernel/locking/lockdep.c:5831 (discriminator 2))
[ 36.311256][ T108] fs_reclaim_acquire (mm/page_alloc.c:4270 mm/page_alloc.c:4283)
[ 36.312913][ T108] kmem_cache_alloc_lru_noprof (include/linux/sched/mm.h:319 mm/slub.c:4925 mm/slub.c:5260 mm/slub.c:5303)
[ 36.314550][ T108] __d_alloc (fs/dcache.c:1692)
[ 36.316022][ T108] d_alloc_parallel (fs/dcache.c:2549)
[ 36.317503][ T108] __lookup_slow (fs/namei.c:1801)
[ 36.319003][ T108] simple_start_creating (fs/libfs.c:2304 (discriminator 1))
[ 36.320529][ T108] debugfs_start_creating+0x4f/0xe0
[ 36.322219][ T108] debugfs_create_dir (fs/debugfs/inode.c:374 (discriminator 1) fs/debugfs/inode.c:581 (discriminator 1))
[ 36.323776][ T108] pinctrl_init (drivers/pinctrl/core.c:2028 (discriminator 1) drivers/pinctrl/core.c:2420 (discriminator 1))
[ 36.325323][ T108] do_one_initcall (init/main.c:1283)
[ 36.326866][ T108] do_initcalls (init/main.c:1344 (discriminator 3) init/main.c:1361 (discriminator 3))
[ 36.328322][ T108] kernel_init_freeable (init/main.c:1597)
[ 36.329944][ T108] kernel_init (init/main.c:1485)
[ 36.331399][ T108] ret_from_fork (arch/x86/kernel/process.c:164)
[ 36.332850][ T108] ret_from_fork_asm (arch/x86/entry/entry_64.S:255)
[ 36.334399][ T108]
[ 36.334399][ T108] -> #1 (&sb->s_type->i_mutex_key#3){+.+.}-{4:4}:
[ 36.337887][ T108] __lock_acquire (kernel/locking/lockdep.c:5237 (discriminator 1))
[ 36.339409][ T108] lock_acquire (include/linux/preempt.h:471 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) kernel/locking/lockdep.c:5831 (discriminator 2))
[ 36.341061][ T108] down_write (arch/x86/include/asm/preempt.h:80 (discriminator 10) kernel/locking/rwsem.c:1315 (discriminator 10) kernel/locking/rwsem.c:1326 (discriminator 10) kernel/locking/rwsem.c:1591 (discriminator 10))
[ 36.342598][ T108] simple_start_creating (fs/libfs.c:2299)
[ 36.344192][ T108] debugfs_start_creating+0x4f/0xe0
[ 36.345959][ T108] debugfs_create_dir (fs/debugfs/inode.c:374 (discriminator 1) fs/debugfs/inode.c:581 (discriminator 1))
[ 36.347535][ T108] blk_register_queue (block/blk-sysfs.c:928 (discriminator 1))
[ 36.349145][ T108] __add_disk (block/genhd.c:528)
[ 36.350605][ T108] add_disk_fwnode (block/genhd.c:598)
[ 36.352129][ T108] sr_probe (drivers/scsi/sr.c:703) sr_mod
[ 36.353727][ T108] really_probe (drivers/base/dd.c:581 drivers/base/dd.c:659)
[ 36.355225][ T108] __driver_probe_device (drivers/base/dd.c:801)
[ 36.356818][ T108] driver_probe_device (drivers/base/dd.c:831)
[ 36.358406][ T108] __driver_attach (drivers/base/dd.c:1218)
[ 36.359984][ T108] bus_for_each_dev (drivers/base/bus.c:369)
[ 36.361462][ T108] bus_add_driver (drivers/base/bus.c:678)
[ 36.362950][ T108] driver_register (drivers/base/driver.c:249)
[ 36.364384][ T108] init_sr (drivers/scsi/sr.c:152) sr_mod
[ 36.365900][ T108] do_one_initcall (init/main.c:1283)
[ 36.367417][ T108] do_init_module (kernel/module/main.c:3039)
[ 36.368926][ T108] load_module (kernel/module/main.c:3509)
[ 36.370434][ T108] init_module_from_file (kernel/module/main.c:3701)
[ 36.372090][ T108] idempotent_init_module (kernel/module/main.c:3713)
[ 36.373729][ T108] __x64_sys_finit_module (include/linux/file.h:62 (discriminator 1) include/linux/file.h:83 (discriminator 1) kernel/module/main.c:3736 (discriminator 1) kernel/module/main.c:3723 (discriminator 1) kernel/module/main.c:3723 (discriminator 1))
[ 36.375279][ T108] do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1) arch/x86/entry/syscall_64.c:94 (discriminator 1))
[ 36.376796][ T108] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[ 36.378301][ T108]
[ 36.378301][ T108] -> #0 (&q->debugfs_mutex){+.+.}-{4:4}:
[ 36.381043][ T108] check_prev_add (kernel/locking/lockdep.c:3166 (discriminator 2))
[ 36.382551][ T108] validate_chain (kernel/locking/lockdep.c:3285 kernel/locking/lockdep.c:3908)
[ 36.384101][ T108] __lock_acquire (kernel/locking/lockdep.c:5237 (discriminator 1))
[ 36.385608][ T108] lock_acquire (include/linux/preempt.h:471 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) include/trace/events/lock.h:24 (discriminator 2) kernel/locking/lockdep.c:5831 (discriminator 2))
[ 36.387119][ T108] __mutex_lock (arch/x86/include/asm/jump_label.h:36 include/trace/events/lock.h:95 kernel/locking/mutex.c:600 kernel/locking/mutex.c:760)
[ 36.388585][ T108] rq_qos_add_freezed (block/blk-rq-qos.c:345)
[ 36.390240][ T108] wbt_init (block/blk-wbt.c:930)
[ 36.391782][ T108] wbt_enable_default (include/linux/blk-mq.h:960 block/blk-wbt.c:731)
[ 36.393333][ T108] blk_register_queue (block/blk-sysfs.c:948)
[ 36.394867][ T108] __add_disk (block/genhd.c:528)
[ 36.396410][ T108] add_disk_fwnode (block/genhd.c:598)
[ 36.398009][ T108] sr_probe (drivers/scsi/sr.c:703) sr_mod
[ 36.399514][ T108] really_probe (drivers/base/dd.c:581 drivers/base/dd.c:659)
[ 36.401035][ T108] __driver_probe_device (drivers/base/dd.c:801)
[ 36.402511][ T108] driver_probe_device (drivers/base/dd.c:831)
[ 36.403990][ T108] __driver_attach (drivers/base/dd.c:1218)
[ 36.405471][ T108] bus_for_each_dev (drivers/base/bus.c:369)
[ 36.407062][ T108] bus_add_driver (drivers/base/bus.c:678)
[ 36.408549][ T108] driver_register (drivers/base/driver.c:249)
[ 36.410081][ T108] init_sr (drivers/scsi/sr.c:152) sr_mod
[ 36.411625][ T108] do_one_initcall (init/main.c:1283)
[ 36.413166][ T108] do_init_module (kernel/module/main.c:3039)
[ 36.414682][ T108] load_module (kernel/module/main.c:3509)
[ 36.416196][ T108] init_module_from_file (kernel/module/main.c:3701)
[ 36.417822][ T108] idempotent_init_module (kernel/module/main.c:3713)
[ 36.419453][ T108] __x64_sys_finit_module (include/linux/file.h:62 (discriminator 1) include/linux/file.h:83 (discriminator 1) kernel/module/main.c:3736 (discriminator 1) kernel/module/main.c:3723 (discriminator 1) kernel/module/main.c:3723 (discriminator 1))
[ 36.421144][ T108] do_syscall_64 (arch/x86/entry/syscall_64.c:63 (discriminator 1) arch/x86/entry/syscall_64.c:94 (discriminator 1))
[ 36.422741][ T108] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[ 36.424468][ T108]
[ 36.424468][ T108] other info that might help us debug this:
[ 36.424468][ T108]
[ 36.428405][ T108] Chain exists of:
[ 36.428405][ T108] &q->debugfs_mutex --> &q->q_usage_counter(io) --> &q->rq_qos_mutex
[ 36.428405][ T108]
[ 36.432967][ T108] Possible unsafe locking scenario:
[ 36.432967][ T108]
[ 36.435485][ T108] CPU0 CPU1
[ 36.437074][ T108] ---- ----
[ 36.438592][ T108] lock(&q->rq_qos_mutex);
[ 36.440094][ T108] lock(&q->q_usage_counter(io));
[ 36.441920][ T108] lock(&q->rq_qos_mutex);
[ 36.443715][ T108] lock(&q->debugfs_mutex);
[ 36.445174][ T108]
[ 36.445174][ T108] *** DEADLOCK ***
[ 36.445174][ T108]
[ 36.448803][ T108] 6 locks held by udevd/108:
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251119/202511191340.643afc3a-lkp@intel.com
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
next prev parent reply other threads:[~2025-11-19 6:56 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-11-16 4:10 [PATCH RESEND 0/5] block/blk-rq-qos: fix incorrect lock order for rq_qos_mutex and freeze queue Yu Kuai
2025-11-16 4:10 ` [PATCH RESEND 1/5] block/blk-rq-qos: add a new helper rq_qos_add_freezed() Yu Kuai
2025-11-17 10:10 ` Nilay Shroff
2025-11-17 11:01 ` Ming Lei
2025-11-17 11:13 ` Nilay Shroff
2025-11-17 11:30 ` Ming Lei
2025-11-17 11:39 ` Yu Kuai
2025-11-17 11:54 ` Ming Lei
2025-11-17 11:59 ` Yu Kuai
2025-11-17 23:32 ` Bart Van Assche
2025-11-16 4:10 ` [PATCH RESEND 2/5] blk-wbt: fix incorrect lock order for rq_qos_mutex and freeze queue Yu Kuai
2025-11-17 10:11 ` Nilay Shroff
2025-11-19 6:55 ` kernel test robot [this message]
2025-11-16 4:10 ` [PATCH RESEND 3/5] blk-iocost: " Yu Kuai
2025-11-17 10:11 ` Nilay Shroff
2025-11-16 4:10 ` [PATCH RESEND 4/5] blk-iolatency: " Yu Kuai
2025-11-17 10:12 ` Nilay Shroff
2025-11-16 4:10 ` [PATCH RESEND 5/5] block/blk-rq-qos: cleanup rq_qos_add() Yu Kuai
2025-11-17 10:13 ` Nilay Shroff
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=202511191340.643afc3a-lkp@intel.com \
--to=oliver.sang@intel.com \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@intel.com \
--cc=ming.lei@redhat.com \
--cc=nilay@linux.ibm.com \
--cc=oe-lkp@lists.linux.dev \
--cc=tj@kernel.org \
--cc=yukuai@fnnas.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox