* [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
@ 2024-12-10 14:41 Nilay Shroff
2024-12-11 12:00 ` Ming Lei
2024-12-13 17:52 ` Jens Axboe
0 siblings, 2 replies; 6+ messages in thread
From: Nilay Shroff @ 2024-12-10 14:41 UTC (permalink / raw)
To: linux-block
Cc: Nilay Shroff, kjain, hch, axboe, ritesh.list, ming.lei, gjoyce
For storing a value to a queue attribute, the queue_attr_store function
first freezes the queue (->q_usage_counter(io)) and then acquire
->sysfs_lock. This seems not correct as the usual ordering should be to
acquire ->sysfs_lock before freezing the queue. This incorrect ordering
causes the following lockdep splat which we are able to reproduce always
simply by accessing /sys/kernel/debug file using ls command:
[ 57.597146] WARNING: possible circular locking dependency detected
[ 57.597154] 6.12.0-10553-gb86545e02e8c #20 Tainted: G W
[ 57.597162] ------------------------------------------------------
[ 57.597168] ls/4605 is trying to acquire lock:
[ 57.597176] c00000003eb56710 (&mm->mmap_lock){++++}-{4:4}, at: __might_fault+0x58/0xc0
[ 57.597200]
but task is already holding lock:
[ 57.597207] c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
[ 57.597226]
which lock already depends on the new lock.
[ 57.597233]
the existing dependency chain (in reverse order) is:
[ 57.597241]
-> #5 (&sb->s_type->i_mutex_key#3){++++}-{4:4}:
[ 57.597255] down_write+0x6c/0x18c
[ 57.597264] start_creating+0xb4/0x24c
[ 57.597274] debugfs_create_dir+0x2c/0x1e8
[ 57.597283] blk_register_queue+0xec/0x294
[ 57.597292] add_disk_fwnode+0x2e4/0x548
[ 57.597302] brd_alloc+0x2c8/0x338
[ 57.597309] brd_init+0x100/0x178
[ 57.597317] do_one_initcall+0x88/0x3e4
[ 57.597326] kernel_init_freeable+0x3cc/0x6e0
[ 57.597334] kernel_init+0x34/0x1cc
[ 57.597342] ret_from_kernel_user_thread+0x14/0x1c
[ 57.597350]
-> #4 (&q->debugfs_mutex){+.+.}-{4:4}:
[ 57.597362] __mutex_lock+0xfc/0x12a0
[ 57.597370] blk_register_queue+0xd4/0x294
[ 57.597379] add_disk_fwnode+0x2e4/0x548
[ 57.597388] brd_alloc+0x2c8/0x338
[ 57.597395] brd_init+0x100/0x178
[ 57.597402] do_one_initcall+0x88/0x3e4
[ 57.597410] kernel_init_freeable+0x3cc/0x6e0
[ 57.597418] kernel_init+0x34/0x1cc
[ 57.597426] ret_from_kernel_user_thread+0x14/0x1c
[ 57.597434]
-> #3 (&q->sysfs_lock){+.+.}-{4:4}:
[ 57.597446] __mutex_lock+0xfc/0x12a0
[ 57.597454] queue_attr_store+0x9c/0x110
[ 57.597462] sysfs_kf_write+0x70/0xb0
[ 57.597471] kernfs_fop_write_iter+0x1b0/0x2ac
[ 57.597480] vfs_write+0x3dc/0x6e8
[ 57.597488] ksys_write+0x84/0x140
[ 57.597495] system_call_exception+0x130/0x360
[ 57.597504] system_call_common+0x160/0x2c4
[ 57.597516]
-> #2 (&q->q_usage_counter(io)#21){++++}-{0:0}:
[ 57.597530] __submit_bio+0x5ec/0x828
[ 57.597538] submit_bio_noacct_nocheck+0x1e4/0x4f0
[ 57.597547] iomap_readahead+0x2a0/0x448
[ 57.597556] xfs_vm_readahead+0x28/0x3c
[ 57.597564] read_pages+0x88/0x41c
[ 57.597571] page_cache_ra_unbounded+0x1ac/0x2d8
[ 57.597580] filemap_get_pages+0x188/0x984
[ 57.597588] filemap_read+0x13c/0x4bc
[ 57.597596] xfs_file_buffered_read+0x88/0x17c
[ 57.597605] xfs_file_read_iter+0xac/0x158
[ 57.597614] vfs_read+0x2d4/0x3b4
[ 57.597622] ksys_read+0x84/0x144
[ 57.597629] system_call_exception+0x130/0x360
[ 57.597637] system_call_common+0x160/0x2c4
[ 57.597647]
-> #1 (mapping.invalidate_lock#2){++++}-{4:4}:
[ 57.597661] down_read+0x6c/0x220
[ 57.597669] filemap_fault+0x870/0x100c
[ 57.597677] xfs_filemap_fault+0xc4/0x18c
[ 57.597684] __do_fault+0x64/0x164
[ 57.597693] __handle_mm_fault+0x1274/0x1dac
[ 57.597702] handle_mm_fault+0x248/0x484
[ 57.597711] ___do_page_fault+0x428/0xc0c
[ 57.597719] hash__do_page_fault+0x30/0x68
[ 57.597727] do_hash_fault+0x90/0x35c
[ 57.597736] data_access_common_virt+0x210/0x220
[ 57.597745] _copy_from_user+0xf8/0x19c
[ 57.597754] sel_write_load+0x178/0xd54
[ 57.597762] vfs_write+0x108/0x6e8
[ 57.597769] ksys_write+0x84/0x140
[ 57.597777] system_call_exception+0x130/0x360
[ 57.597785] system_call_common+0x160/0x2c4
[ 57.597794]
-> #0 (&mm->mmap_lock){++++}-{4:4}:
[ 57.597806] __lock_acquire+0x17cc/0x2330
[ 57.597814] lock_acquire+0x138/0x400
[ 57.597822] __might_fault+0x7c/0xc0
[ 57.597830] filldir64+0xe8/0x390
[ 57.597839] dcache_readdir+0x80/0x2d4
[ 57.597846] iterate_dir+0xd8/0x1d4
[ 57.597855] sys_getdents64+0x88/0x2d4
[ 57.597864] system_call_exception+0x130/0x360
[ 57.597872] system_call_common+0x160/0x2c4
[ 57.597881]
other info that might help us debug this:
[ 57.597888] Chain exists of:
&mm->mmap_lock --> &q->debugfs_mutex --> &sb->s_type->i_mutex_key#3
[ 57.597905] Possible unsafe locking scenario:
[ 57.597911] CPU0 CPU1
[ 57.597917] ---- ----
[ 57.597922] rlock(&sb->s_type->i_mutex_key#3);
[ 57.597932] lock(&q->debugfs_mutex);
[ 57.597940] lock(&sb->s_type->i_mutex_key#3);
[ 57.597950] rlock(&mm->mmap_lock);
[ 57.597958]
*** DEADLOCK ***
[ 57.597965] 2 locks held by ls/4605:
[ 57.597971] #0: c0000000137c12f8 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos+0xcc/0x154
[ 57.597989] #1: c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
Prevent the above lockdep warning by acquiring ->sysfs_lock before
freezing the queue while storing a queue attribute in queue_attr_store
function. Later, we also found[1] another function __blk_mq_update_nr_
hw_queues where we first freeze queue and then acquire the ->sysfs_lock.
So we've also updated lock ordering in __blk_mq_update_nr_hw_queues
function and ensured that in all code paths we follow the correct lock
ordering i.e. acquire ->sysfs_lock before freezing the queue.
[1] https://lore.kernel.org/all/CAFj5m9Ke8+EHKQBs_Nk6hqd=LGXtk4mUxZUN5==ZcCjnZSBwHw@mail.gmail.com/
Reported-by: kjain@linux.ibm.com
Fixes: af2814149883 ("block: freeze the queue in queue_attr_store")
Tested-by: kjain@linux.ibm.com
Cc: hch@lst.de
Cc: axboe@kernel.dk
Cc: ritesh.list@gmail.com
Cc: ming.lei@redhat.com
Cc: gjoyce@linux.ibm.com
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
Changes from v1:
- Fix lock ordering in __blk_mq_update_nr_hw_queues (Ming Lei)
---
block/blk-mq-sysfs.c | 16 ++++++----------
block/blk-mq.c | 29 ++++++++++++++++++-----------
block/blk-sysfs.c | 4 ++--
3 files changed, 26 insertions(+), 23 deletions(-)
diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c
index 156e9bb07abf..cd5ea6eaa76b 100644
--- a/block/blk-mq-sysfs.c
+++ b/block/blk-mq-sysfs.c
@@ -275,15 +275,13 @@ void blk_mq_sysfs_unregister_hctxs(struct request_queue *q)
struct blk_mq_hw_ctx *hctx;
unsigned long i;
- mutex_lock(&q->sysfs_dir_lock);
+ lockdep_assert_held(&q->sysfs_dir_lock);
+
if (!q->mq_sysfs_init_done)
- goto unlock;
+ return;
queue_for_each_hw_ctx(q, hctx, i)
blk_mq_unregister_hctx(hctx);
-
-unlock:
- mutex_unlock(&q->sysfs_dir_lock);
}
int blk_mq_sysfs_register_hctxs(struct request_queue *q)
@@ -292,9 +290,10 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
unsigned long i;
int ret = 0;
- mutex_lock(&q->sysfs_dir_lock);
+ lockdep_assert_held(&q->sysfs_dir_lock);
+
if (!q->mq_sysfs_init_done)
- goto unlock;
+ return ret;
queue_for_each_hw_ctx(q, hctx, i) {
ret = blk_mq_register_hctx(hctx);
@@ -302,8 +301,5 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
break;
}
-unlock:
- mutex_unlock(&q->sysfs_dir_lock);
-
return ret;
}
diff --git a/block/blk-mq.c b/block/blk-mq.c
index aa340b097b6e..cccfc6dada7e 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4455,7 +4455,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
unsigned long i, j;
/* protect against switching io scheduler */
- mutex_lock(&q->sysfs_lock);
+ lockdep_assert_held(&q->sysfs_lock);
+
for (i = 0; i < set->nr_hw_queues; i++) {
int old_node;
int node = blk_mq_get_hctx_node(set, i);
@@ -4488,7 +4489,6 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
xa_for_each_start(&q->hctx_table, j, hctx, j)
blk_mq_exit_hctx(q, set, hctx, j);
- mutex_unlock(&q->sysfs_lock);
/* unregister cpuhp callbacks for exited hctxs */
blk_mq_remove_hw_queues_cpuhp(q);
@@ -4520,10 +4520,14 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
xa_init(&q->hctx_table);
+ mutex_lock(&q->sysfs_lock);
+
blk_mq_realloc_hw_ctxs(set, q);
if (!q->nr_hw_queues)
goto err_hctxs;
+ mutex_unlock(&q->sysfs_lock);
+
INIT_WORK(&q->timeout_work, blk_mq_timeout_work);
blk_queue_rq_timeout(q, set->timeout ? set->timeout : 30 * HZ);
@@ -4542,6 +4546,7 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
return 0;
err_hctxs:
+ mutex_unlock(&q->sysfs_lock);
blk_mq_release(q);
err_exit:
q->mq_ops = NULL;
@@ -4922,12 +4927,12 @@ static bool blk_mq_elv_switch_none(struct list_head *head,
return false;
/* q->elevator needs protection from ->sysfs_lock */
- mutex_lock(&q->sysfs_lock);
+ lockdep_assert_held(&q->sysfs_lock);
/* the check has to be done with holding sysfs_lock */
if (!q->elevator) {
kfree(qe);
- goto unlock;
+ goto out;
}
INIT_LIST_HEAD(&qe->node);
@@ -4937,9 +4942,7 @@ static bool blk_mq_elv_switch_none(struct list_head *head,
__elevator_get(qe->type);
list_add(&qe->node, head);
elevator_disable(q);
-unlock:
- mutex_unlock(&q->sysfs_lock);
-
+out:
return true;
}
@@ -4968,11 +4971,9 @@ static void blk_mq_elv_switch_back(struct list_head *head,
list_del(&qe->node);
kfree(qe);
- mutex_lock(&q->sysfs_lock);
elevator_switch(q, t);
/* drop the reference acquired in blk_mq_elv_switch_none */
elevator_put(t);
- mutex_unlock(&q->sysfs_lock);
}
static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
@@ -4992,8 +4993,11 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
if (set->nr_maps == 1 && nr_hw_queues == set->nr_hw_queues)
return;
- list_for_each_entry(q, &set->tag_list, tag_set_list)
+ list_for_each_entry(q, &set->tag_list, tag_set_list) {
+ mutex_lock(&q->sysfs_dir_lock);
+ mutex_lock(&q->sysfs_lock);
blk_mq_freeze_queue(q);
+ }
/*
* Switch IO scheduler to 'none', cleaning up the data associated
* with the previous scheduler. We will switch back once we are done
@@ -5049,8 +5053,11 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
list_for_each_entry(q, &set->tag_list, tag_set_list)
blk_mq_elv_switch_back(&head, q);
- list_for_each_entry(q, &set->tag_list, tag_set_list)
+ list_for_each_entry(q, &set->tag_list, tag_set_list) {
blk_mq_unfreeze_queue(q);
+ mutex_unlock(&q->sysfs_lock);
+ mutex_unlock(&q->sysfs_dir_lock);
+ }
/* Free the excess tags when nr_hw_queues shrink. */
for (i = set->nr_hw_queues; i < prev_nr_hw_queues; i++)
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 4241aea84161..f648b112782f 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -706,11 +706,11 @@ queue_attr_store(struct kobject *kobj, struct attribute *attr,
if (entry->load_module)
entry->load_module(disk, page, length);
- blk_mq_freeze_queue(q);
mutex_lock(&q->sysfs_lock);
+ blk_mq_freeze_queue(q);
res = entry->store(disk, page, length);
- mutex_unlock(&q->sysfs_lock);
blk_mq_unfreeze_queue(q);
+ mutex_unlock(&q->sysfs_lock);
return res;
}
--
2.45.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
@ 2024-12-11 8:19 kernel test robot
0 siblings, 0 replies; 6+ messages in thread
From: kernel test robot @ 2024-12-11 8:19 UTC (permalink / raw)
To: oe-kbuild; +Cc: lkp, Julia Lawall
BCC: lkp@intel.com
CC: oe-kbuild-all@lists.linux.dev
In-Reply-To: <20241210144222.1066229-1-nilay@linux.ibm.com>
References: <20241210144222.1066229-1-nilay@linux.ibm.com>
TO: Nilay Shroff <nilay@linux.ibm.com>
TO: linux-block@vger.kernel.org
CC: Nilay Shroff <nilay@linux.ibm.com>
CC: kjain@linux.ibm.com
CC: hch@lst.de
CC: axboe@kernel.dk
CC: ritesh.list@gmail.com
CC: ming.lei@redhat.com
CC: gjoyce@linux.ibm.com
Hi Nilay,
kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.13-rc2 next-20241211]
[cannot apply to axboe-block/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nilay-Shroff/block-Fix-potential-deadlock-while-freezing-queue-and-acquiring-sysfs_lock/20241210-224435
base: linus/master
patch link: https://lore.kernel.org/r/20241210144222.1066229-1-nilay%40linux.ibm.com
patch subject: [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
:::::: branch date: 17 hours ago
:::::: commit date: 17 hours ago
config: x86_64-randconfig-101-20241211 (https://download.01.org/0day-ci/archive/20241211/202412111553.XzRIWedS-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Reported-by: Julia Lawall <julia.lawall@inria.fr>
| Closes: https://lore.kernel.org/r/202412111553.XzRIWedS-lkp@intel.com/
cocci warnings: (new ones prefixed by >>)
>> block/blk-mq.c:4997:2-12: second lock on line 4997
block/blk-mq.c:4998:2-12: second lock on line 4998
--
>> block/blk-mq.c:5065:0-1: preceding lock on line 4997
block/blk-mq.c:5065:0-1: preceding lock on line 4998
vim +4997 block/blk-mq.c
d48ece209f82c9c Jianchao Wang 2018-08-21 4978
e4dc2b32df5573b Keith Busch 2017-05-30 4979 static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
e4dc2b32df5573b Keith Busch 2017-05-30 4980 int nr_hw_queues)
868f2f0b72068a0 Keith Busch 2015-12-17 4981 {
868f2f0b72068a0 Keith Busch 2015-12-17 4982 struct request_queue *q;
d48ece209f82c9c Jianchao Wang 2018-08-21 4983 LIST_HEAD(head);
6be6d1124197133 Chengming Zhou 2023-09-08 4984 int prev_nr_hw_queues = set->nr_hw_queues;
6be6d1124197133 Chengming Zhou 2023-09-08 4985 int i;
868f2f0b72068a0 Keith Busch 2015-12-17 4986
705cda97ee3abb6 Bart Van Assche 2017-04-07 4987 lockdep_assert_held(&set->tag_list_lock);
705cda97ee3abb6 Bart Van Assche 2017-04-07 4988
392546aed220090 Jens Axboe 2018-10-29 4989 if (set->nr_maps == 1 && nr_hw_queues > nr_cpu_ids)
868f2f0b72068a0 Keith Busch 2015-12-17 4990 nr_hw_queues = nr_cpu_ids;
fe35ec58f0d3392 Weiping Zhang 2020-06-17 4991 if (nr_hw_queues < 1)
fe35ec58f0d3392 Weiping Zhang 2020-06-17 4992 return;
fe35ec58f0d3392 Weiping Zhang 2020-06-17 4993 if (set->nr_maps == 1 && nr_hw_queues == set->nr_hw_queues)
868f2f0b72068a0 Keith Busch 2015-12-17 4994 return;
868f2f0b72068a0 Keith Busch 2015-12-17 4995
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 4996 list_for_each_entry(q, &set->tag_list, tag_set_list) {
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 @4997 mutex_lock(&q->sysfs_dir_lock);
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 4998 mutex_lock(&q->sysfs_lock);
868f2f0b72068a0 Keith Busch 2015-12-17 4999 blk_mq_freeze_queue(q);
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 5000 }
d48ece209f82c9c Jianchao Wang 2018-08-21 5001 /*
d48ece209f82c9c Jianchao Wang 2018-08-21 5002 * Switch IO scheduler to 'none', cleaning up the data associated
d48ece209f82c9c Jianchao Wang 2018-08-21 5003 * with the previous scheduler. We will switch back once we are done
d48ece209f82c9c Jianchao Wang 2018-08-21 5004 * updating the new sw to hw queue mappings.
d48ece209f82c9c Jianchao Wang 2018-08-21 5005 */
d48ece209f82c9c Jianchao Wang 2018-08-21 5006 list_for_each_entry(q, &set->tag_list, tag_set_list)
d48ece209f82c9c Jianchao Wang 2018-08-21 5007 if (!blk_mq_elv_switch_none(&head, q))
d48ece209f82c9c Jianchao Wang 2018-08-21 5008 goto switch_back;
868f2f0b72068a0 Keith Busch 2015-12-17 5009
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5010 list_for_each_entry(q, &set->tag_list, tag_set_list) {
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5011 blk_mq_debugfs_unregister_hctxs(q);
eaa870f97544668 Christoph Hellwig 2022-06-28 5012 blk_mq_sysfs_unregister_hctxs(q);
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5013 }
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5014
ee9d55210c2fe40 Christoph Hellwig 2022-11-09 5015 if (blk_mq_realloc_tag_set_tags(set, nr_hw_queues) < 0)
f7e76dbc24df695 Bart Van Assche 2019-10-25 5016 goto reregister;
f7e76dbc24df695 Bart Van Assche 2019-10-25 5017
e01ad46d53b5972 Jianchao Wang 2018-10-12 5018 fallback:
aa880ad690ab6d4 Weiping Zhang 2020-05-13 5019 blk_mq_update_queue_map(set);
868f2f0b72068a0 Keith Busch 2015-12-17 5020 list_for_each_entry(q, &set->tag_list, tag_set_list) {
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5021 struct queue_limits lim;
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5022
868f2f0b72068a0 Keith Busch 2015-12-17 5023 blk_mq_realloc_hw_ctxs(set, q);
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5024
e01ad46d53b5972 Jianchao Wang 2018-10-12 5025 if (q->nr_hw_queues != set->nr_hw_queues) {
a846a8e6c9a5949 Ye Bin 2021-11-08 5026 int i = prev_nr_hw_queues;
a846a8e6c9a5949 Ye Bin 2021-11-08 5027
e01ad46d53b5972 Jianchao Wang 2018-10-12 5028 pr_warn("Increasing nr_hw_queues to %d fails, fallback to %d\n",
e01ad46d53b5972 Jianchao Wang 2018-10-12 5029 nr_hw_queues, prev_nr_hw_queues);
a846a8e6c9a5949 Ye Bin 2021-11-08 5030 for (; i < set->nr_hw_queues; i++)
a846a8e6c9a5949 Ye Bin 2021-11-08 5031 __blk_mq_free_map_and_rqs(set, i);
a846a8e6c9a5949 Ye Bin 2021-11-08 5032
e01ad46d53b5972 Jianchao Wang 2018-10-12 5033 set->nr_hw_queues = prev_nr_hw_queues;
e01ad46d53b5972 Jianchao Wang 2018-10-12 5034 goto fallback;
e01ad46d53b5972 Jianchao Wang 2018-10-12 5035 }
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5036 lim = queue_limits_start_update(q);
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5037 if (blk_mq_can_poll(set))
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5038 lim.features |= BLK_FEAT_POLL;
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5039 else
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5040 lim.features &= ~BLK_FEAT_POLL;
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5041 if (queue_limits_commit_update(q, &lim) < 0)
8023e144f9d6e35 Christoph Hellwig 2024-06-17 5042 pr_warn("updating the poll flag failed\n");
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5043 blk_mq_map_swqueue(q);
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5044 }
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5045
f7e76dbc24df695 Bart Van Assche 2019-10-25 5046 reregister:
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5047 list_for_each_entry(q, &set->tag_list, tag_set_list) {
eaa870f97544668 Christoph Hellwig 2022-06-28 5048 blk_mq_sysfs_register_hctxs(q);
477e19dedc9d3e1 Jianchao Wang 2018-10-12 5049 blk_mq_debugfs_register_hctxs(q);
868f2f0b72068a0 Keith Busch 2015-12-17 5050 }
868f2f0b72068a0 Keith Busch 2015-12-17 5051
d48ece209f82c9c Jianchao Wang 2018-08-21 5052 switch_back:
d48ece209f82c9c Jianchao Wang 2018-08-21 5053 list_for_each_entry(q, &set->tag_list, tag_set_list)
d48ece209f82c9c Jianchao Wang 2018-08-21 5054 blk_mq_elv_switch_back(&head, q);
d48ece209f82c9c Jianchao Wang 2018-08-21 5055
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 5056 list_for_each_entry(q, &set->tag_list, tag_set_list) {
868f2f0b72068a0 Keith Busch 2015-12-17 5057 blk_mq_unfreeze_queue(q);
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 5058 mutex_unlock(&q->sysfs_lock);
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 5059 mutex_unlock(&q->sysfs_dir_lock);
38bfa3ab3c72c15 Nilay Shroff 2024-12-10 5060 }
6be6d1124197133 Chengming Zhou 2023-09-08 5061
6be6d1124197133 Chengming Zhou 2023-09-08 5062 /* Free the excess tags when nr_hw_queues shrink. */
6be6d1124197133 Chengming Zhou 2023-09-08 5063 for (i = set->nr_hw_queues; i < prev_nr_hw_queues; i++)
6be6d1124197133 Chengming Zhou 2023-09-08 5064 __blk_mq_free_map_and_rqs(set, i);
868f2f0b72068a0 Keith Busch 2015-12-17 @5065 }
e4dc2b32df5573b Keith Busch 2017-05-30 5066
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
2024-12-10 14:41 [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock Nilay Shroff
@ 2024-12-11 12:00 ` Ming Lei
2024-12-11 12:32 ` Nilay Shroff
2024-12-13 17:52 ` Jens Axboe
1 sibling, 1 reply; 6+ messages in thread
From: Ming Lei @ 2024-12-11 12:00 UTC (permalink / raw)
To: Nilay Shroff; +Cc: linux-block, kjain, hch, axboe, ritesh.list, gjoyce
On Tue, Dec 10, 2024 at 08:11:43PM +0530, Nilay Shroff wrote:
> For storing a value to a queue attribute, the queue_attr_store function
> first freezes the queue (->q_usage_counter(io)) and then acquire
> ->sysfs_lock. This seems not correct as the usual ordering should be to
> acquire ->sysfs_lock before freezing the queue. This incorrect ordering
> causes the following lockdep splat which we are able to reproduce always
> simply by accessing /sys/kernel/debug file using ls command:
>
> [ 57.597146] WARNING: possible circular locking dependency detected
> [ 57.597154] 6.12.0-10553-gb86545e02e8c #20 Tainted: G W
> [ 57.597162] ------------------------------------------------------
> [ 57.597168] ls/4605 is trying to acquire lock:
> [ 57.597176] c00000003eb56710 (&mm->mmap_lock){++++}-{4:4}, at: __might_fault+0x58/0xc0
> [ 57.597200]
> but task is already holding lock:
> [ 57.597207] c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
> [ 57.597226]
> which lock already depends on the new lock.
>
> [ 57.597233]
> the existing dependency chain (in reverse order) is:
> [ 57.597241]
> -> #5 (&sb->s_type->i_mutex_key#3){++++}-{4:4}:
> [ 57.597255] down_write+0x6c/0x18c
> [ 57.597264] start_creating+0xb4/0x24c
> [ 57.597274] debugfs_create_dir+0x2c/0x1e8
> [ 57.597283] blk_register_queue+0xec/0x294
> [ 57.597292] add_disk_fwnode+0x2e4/0x548
> [ 57.597302] brd_alloc+0x2c8/0x338
> [ 57.597309] brd_init+0x100/0x178
> [ 57.597317] do_one_initcall+0x88/0x3e4
> [ 57.597326] kernel_init_freeable+0x3cc/0x6e0
> [ 57.597334] kernel_init+0x34/0x1cc
> [ 57.597342] ret_from_kernel_user_thread+0x14/0x1c
> [ 57.597350]
> -> #4 (&q->debugfs_mutex){+.+.}-{4:4}:
> [ 57.597362] __mutex_lock+0xfc/0x12a0
> [ 57.597370] blk_register_queue+0xd4/0x294
> [ 57.597379] add_disk_fwnode+0x2e4/0x548
> [ 57.597388] brd_alloc+0x2c8/0x338
> [ 57.597395] brd_init+0x100/0x178
> [ 57.597402] do_one_initcall+0x88/0x3e4
> [ 57.597410] kernel_init_freeable+0x3cc/0x6e0
> [ 57.597418] kernel_init+0x34/0x1cc
> [ 57.597426] ret_from_kernel_user_thread+0x14/0x1c
> [ 57.597434]
> -> #3 (&q->sysfs_lock){+.+.}-{4:4}:
> [ 57.597446] __mutex_lock+0xfc/0x12a0
> [ 57.597454] queue_attr_store+0x9c/0x110
> [ 57.597462] sysfs_kf_write+0x70/0xb0
> [ 57.597471] kernfs_fop_write_iter+0x1b0/0x2ac
> [ 57.597480] vfs_write+0x3dc/0x6e8
> [ 57.597488] ksys_write+0x84/0x140
> [ 57.597495] system_call_exception+0x130/0x360
> [ 57.597504] system_call_common+0x160/0x2c4
> [ 57.597516]
> -> #2 (&q->q_usage_counter(io)#21){++++}-{0:0}:
> [ 57.597530] __submit_bio+0x5ec/0x828
> [ 57.597538] submit_bio_noacct_nocheck+0x1e4/0x4f0
> [ 57.597547] iomap_readahead+0x2a0/0x448
> [ 57.597556] xfs_vm_readahead+0x28/0x3c
> [ 57.597564] read_pages+0x88/0x41c
> [ 57.597571] page_cache_ra_unbounded+0x1ac/0x2d8
> [ 57.597580] filemap_get_pages+0x188/0x984
> [ 57.597588] filemap_read+0x13c/0x4bc
> [ 57.597596] xfs_file_buffered_read+0x88/0x17c
> [ 57.597605] xfs_file_read_iter+0xac/0x158
> [ 57.597614] vfs_read+0x2d4/0x3b4
> [ 57.597622] ksys_read+0x84/0x144
> [ 57.597629] system_call_exception+0x130/0x360
> [ 57.597637] system_call_common+0x160/0x2c4
> [ 57.597647]
> -> #1 (mapping.invalidate_lock#2){++++}-{4:4}:
> [ 57.597661] down_read+0x6c/0x220
> [ 57.597669] filemap_fault+0x870/0x100c
> [ 57.597677] xfs_filemap_fault+0xc4/0x18c
> [ 57.597684] __do_fault+0x64/0x164
> [ 57.597693] __handle_mm_fault+0x1274/0x1dac
> [ 57.597702] handle_mm_fault+0x248/0x484
> [ 57.597711] ___do_page_fault+0x428/0xc0c
> [ 57.597719] hash__do_page_fault+0x30/0x68
> [ 57.597727] do_hash_fault+0x90/0x35c
> [ 57.597736] data_access_common_virt+0x210/0x220
> [ 57.597745] _copy_from_user+0xf8/0x19c
> [ 57.597754] sel_write_load+0x178/0xd54
> [ 57.597762] vfs_write+0x108/0x6e8
> [ 57.597769] ksys_write+0x84/0x140
> [ 57.597777] system_call_exception+0x130/0x360
> [ 57.597785] system_call_common+0x160/0x2c4
> [ 57.597794]
> -> #0 (&mm->mmap_lock){++++}-{4:4}:
> [ 57.597806] __lock_acquire+0x17cc/0x2330
> [ 57.597814] lock_acquire+0x138/0x400
> [ 57.597822] __might_fault+0x7c/0xc0
> [ 57.597830] filldir64+0xe8/0x390
> [ 57.597839] dcache_readdir+0x80/0x2d4
> [ 57.597846] iterate_dir+0xd8/0x1d4
> [ 57.597855] sys_getdents64+0x88/0x2d4
> [ 57.597864] system_call_exception+0x130/0x360
> [ 57.597872] system_call_common+0x160/0x2c4
> [ 57.597881]
> other info that might help us debug this:
>
> [ 57.597888] Chain exists of:
> &mm->mmap_lock --> &q->debugfs_mutex --> &sb->s_type->i_mutex_key#3
>
> [ 57.597905] Possible unsafe locking scenario:
>
> [ 57.597911] CPU0 CPU1
> [ 57.597917] ---- ----
> [ 57.597922] rlock(&sb->s_type->i_mutex_key#3);
> [ 57.597932] lock(&q->debugfs_mutex);
> [ 57.597940] lock(&sb->s_type->i_mutex_key#3);
> [ 57.597950] rlock(&mm->mmap_lock);
> [ 57.597958]
> *** DEADLOCK ***
>
> [ 57.597965] 2 locks held by ls/4605:
> [ 57.597971] #0: c0000000137c12f8 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos+0xcc/0x154
> [ 57.597989] #1: c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
>
> Prevent the above lockdep warning by acquiring ->sysfs_lock before
> freezing the queue while storing a queue attribute in queue_attr_store
> function. Later, we also found[1] another function __blk_mq_update_nr_
> hw_queues where we first freeze queue and then acquire the ->sysfs_lock.
> So we've also updated lock ordering in __blk_mq_update_nr_hw_queues
> function and ensured that in all code paths we follow the correct lock
> ordering i.e. acquire ->sysfs_lock before freezing the queue.
>
> [1] https://lore.kernel.org/all/CAFj5m9Ke8+EHKQBs_Nk6hqd=LGXtk4mUxZUN5==ZcCjnZSBwHw@mail.gmail.com/
>
> Reported-by: kjain@linux.ibm.com
> Fixes: af2814149883 ("block: freeze the queue in queue_attr_store")
> Tested-by: kjain@linux.ibm.com
> Cc: hch@lst.de
> Cc: axboe@kernel.dk
> Cc: ritesh.list@gmail.com
> Cc: ming.lei@redhat.com
> Cc: gjoyce@linux.ibm.com
> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
> ---
> Changes from v1:
> - Fix lock ordering in __blk_mq_update_nr_hw_queues (Ming Lei)
> ---
> block/blk-mq-sysfs.c | 16 ++++++----------
> block/blk-mq.c | 29 ++++++++++++++++++-----------
> block/blk-sysfs.c | 4 ++--
> 3 files changed, 26 insertions(+), 23 deletions(-)
>
> diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c
> index 156e9bb07abf..cd5ea6eaa76b 100644
> --- a/block/blk-mq-sysfs.c
> +++ b/block/blk-mq-sysfs.c
> @@ -275,15 +275,13 @@ void blk_mq_sysfs_unregister_hctxs(struct request_queue *q)
> struct blk_mq_hw_ctx *hctx;
> unsigned long i;
>
> - mutex_lock(&q->sysfs_dir_lock);
> + lockdep_assert_held(&q->sysfs_dir_lock);
> +
> if (!q->mq_sysfs_init_done)
> - goto unlock;
> + return;
>
> queue_for_each_hw_ctx(q, hctx, i)
> blk_mq_unregister_hctx(hctx);
> -
> -unlock:
> - mutex_unlock(&q->sysfs_dir_lock);
> }
>
> int blk_mq_sysfs_register_hctxs(struct request_queue *q)
> @@ -292,9 +290,10 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
> unsigned long i;
> int ret = 0;
>
> - mutex_lock(&q->sysfs_dir_lock);
> + lockdep_assert_held(&q->sysfs_dir_lock);
> +
> if (!q->mq_sysfs_init_done)
> - goto unlock;
> + return ret;
>
> queue_for_each_hw_ctx(q, hctx, i) {
> ret = blk_mq_register_hctx(hctx);
> @@ -302,8 +301,5 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
> break;
> }
>
> -unlock:
> - mutex_unlock(&q->sysfs_dir_lock);
> -
> return ret;
> }
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index aa340b097b6e..cccfc6dada7e 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -4455,7 +4455,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
> unsigned long i, j;
>
> /* protect against switching io scheduler */
> - mutex_lock(&q->sysfs_lock);
> + lockdep_assert_held(&q->sysfs_lock);
> +
> for (i = 0; i < set->nr_hw_queues; i++) {
> int old_node;
> int node = blk_mq_get_hctx_node(set, i);
> @@ -4488,7 +4489,6 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
>
> xa_for_each_start(&q->hctx_table, j, hctx, j)
> blk_mq_exit_hctx(q, set, hctx, j);
> - mutex_unlock(&q->sysfs_lock);
>
> /* unregister cpuhp callbacks for exited hctxs */
> blk_mq_remove_hw_queues_cpuhp(q);
> @@ -4520,10 +4520,14 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
>
> xa_init(&q->hctx_table);
>
> + mutex_lock(&q->sysfs_lock);
> +
> blk_mq_realloc_hw_ctxs(set, q);
> if (!q->nr_hw_queues)
> goto err_hctxs;
>
> + mutex_unlock(&q->sysfs_lock);
> +
> INIT_WORK(&q->timeout_work, blk_mq_timeout_work);
> blk_queue_rq_timeout(q, set->timeout ? set->timeout : 30 * HZ);
>
> @@ -4542,6 +4546,7 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
> return 0;
>
> err_hctxs:
> + mutex_unlock(&q->sysfs_lock);
The change in blk_mq_init_allocated_queue() isn't necessary, since queue
kobj isn't exposed yet, otherwise, feel free to add:
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Thanks,
Ming
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
2024-12-11 12:00 ` Ming Lei
@ 2024-12-11 12:32 ` Nilay Shroff
2024-12-11 12:49 ` Ming Lei
0 siblings, 1 reply; 6+ messages in thread
From: Nilay Shroff @ 2024-12-11 12:32 UTC (permalink / raw)
To: Ming Lei; +Cc: linux-block, kjain, hch, axboe, ritesh.list, gjoyce
On 12/11/24 17:30, Ming Lei wrote:
> On Tue, Dec 10, 2024 at 08:11:43PM +0530, Nilay Shroff wrote:
>> For storing a value to a queue attribute, the queue_attr_store function
>> first freezes the queue (->q_usage_counter(io)) and then acquire
>> ->sysfs_lock. This seems not correct as the usual ordering should be to
>> acquire ->sysfs_lock before freezing the queue. This incorrect ordering
>> causes the following lockdep splat which we are able to reproduce always
>> simply by accessing /sys/kernel/debug file using ls command:
>>
>> [ 57.597146] WARNING: possible circular locking dependency detected
>> [ 57.597154] 6.12.0-10553-gb86545e02e8c #20 Tainted: G W
>> [ 57.597162] ------------------------------------------------------
>> [ 57.597168] ls/4605 is trying to acquire lock:
>> [ 57.597176] c00000003eb56710 (&mm->mmap_lock){++++}-{4:4}, at: __might_fault+0x58/0xc0
>> [ 57.597200]
>> but task is already holding lock:
>> [ 57.597207] c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
>> [ 57.597226]
>> which lock already depends on the new lock.
>>
>> [ 57.597233]
>> the existing dependency chain (in reverse order) is:
>> [ 57.597241]
>> -> #5 (&sb->s_type->i_mutex_key#3){++++}-{4:4}:
>> [ 57.597255] down_write+0x6c/0x18c
>> [ 57.597264] start_creating+0xb4/0x24c
>> [ 57.597274] debugfs_create_dir+0x2c/0x1e8
>> [ 57.597283] blk_register_queue+0xec/0x294
>> [ 57.597292] add_disk_fwnode+0x2e4/0x548
>> [ 57.597302] brd_alloc+0x2c8/0x338
>> [ 57.597309] brd_init+0x100/0x178
>> [ 57.597317] do_one_initcall+0x88/0x3e4
>> [ 57.597326] kernel_init_freeable+0x3cc/0x6e0
>> [ 57.597334] kernel_init+0x34/0x1cc
>> [ 57.597342] ret_from_kernel_user_thread+0x14/0x1c
>> [ 57.597350]
>> -> #4 (&q->debugfs_mutex){+.+.}-{4:4}:
>> [ 57.597362] __mutex_lock+0xfc/0x12a0
>> [ 57.597370] blk_register_queue+0xd4/0x294
>> [ 57.597379] add_disk_fwnode+0x2e4/0x548
>> [ 57.597388] brd_alloc+0x2c8/0x338
>> [ 57.597395] brd_init+0x100/0x178
>> [ 57.597402] do_one_initcall+0x88/0x3e4
>> [ 57.597410] kernel_init_freeable+0x3cc/0x6e0
>> [ 57.597418] kernel_init+0x34/0x1cc
>> [ 57.597426] ret_from_kernel_user_thread+0x14/0x1c
>> [ 57.597434]
>> -> #3 (&q->sysfs_lock){+.+.}-{4:4}:
>> [ 57.597446] __mutex_lock+0xfc/0x12a0
>> [ 57.597454] queue_attr_store+0x9c/0x110
>> [ 57.597462] sysfs_kf_write+0x70/0xb0
>> [ 57.597471] kernfs_fop_write_iter+0x1b0/0x2ac
>> [ 57.597480] vfs_write+0x3dc/0x6e8
>> [ 57.597488] ksys_write+0x84/0x140
>> [ 57.597495] system_call_exception+0x130/0x360
>> [ 57.597504] system_call_common+0x160/0x2c4
>> [ 57.597516]
>> -> #2 (&q->q_usage_counter(io)#21){++++}-{0:0}:
>> [ 57.597530] __submit_bio+0x5ec/0x828
>> [ 57.597538] submit_bio_noacct_nocheck+0x1e4/0x4f0
>> [ 57.597547] iomap_readahead+0x2a0/0x448
>> [ 57.597556] xfs_vm_readahead+0x28/0x3c
>> [ 57.597564] read_pages+0x88/0x41c
>> [ 57.597571] page_cache_ra_unbounded+0x1ac/0x2d8
>> [ 57.597580] filemap_get_pages+0x188/0x984
>> [ 57.597588] filemap_read+0x13c/0x4bc
>> [ 57.597596] xfs_file_buffered_read+0x88/0x17c
>> [ 57.597605] xfs_file_read_iter+0xac/0x158
>> [ 57.597614] vfs_read+0x2d4/0x3b4
>> [ 57.597622] ksys_read+0x84/0x144
>> [ 57.597629] system_call_exception+0x130/0x360
>> [ 57.597637] system_call_common+0x160/0x2c4
>> [ 57.597647]
>> -> #1 (mapping.invalidate_lock#2){++++}-{4:4}:
>> [ 57.597661] down_read+0x6c/0x220
>> [ 57.597669] filemap_fault+0x870/0x100c
>> [ 57.597677] xfs_filemap_fault+0xc4/0x18c
>> [ 57.597684] __do_fault+0x64/0x164
>> [ 57.597693] __handle_mm_fault+0x1274/0x1dac
>> [ 57.597702] handle_mm_fault+0x248/0x484
>> [ 57.597711] ___do_page_fault+0x428/0xc0c
>> [ 57.597719] hash__do_page_fault+0x30/0x68
>> [ 57.597727] do_hash_fault+0x90/0x35c
>> [ 57.597736] data_access_common_virt+0x210/0x220
>> [ 57.597745] _copy_from_user+0xf8/0x19c
>> [ 57.597754] sel_write_load+0x178/0xd54
>> [ 57.597762] vfs_write+0x108/0x6e8
>> [ 57.597769] ksys_write+0x84/0x140
>> [ 57.597777] system_call_exception+0x130/0x360
>> [ 57.597785] system_call_common+0x160/0x2c4
>> [ 57.597794]
>> -> #0 (&mm->mmap_lock){++++}-{4:4}:
>> [ 57.597806] __lock_acquire+0x17cc/0x2330
>> [ 57.597814] lock_acquire+0x138/0x400
>> [ 57.597822] __might_fault+0x7c/0xc0
>> [ 57.597830] filldir64+0xe8/0x390
>> [ 57.597839] dcache_readdir+0x80/0x2d4
>> [ 57.597846] iterate_dir+0xd8/0x1d4
>> [ 57.597855] sys_getdents64+0x88/0x2d4
>> [ 57.597864] system_call_exception+0x130/0x360
>> [ 57.597872] system_call_common+0x160/0x2c4
>> [ 57.597881]
>> other info that might help us debug this:
>>
>> [ 57.597888] Chain exists of:
>> &mm->mmap_lock --> &q->debugfs_mutex --> &sb->s_type->i_mutex_key#3
>>
>> [ 57.597905] Possible unsafe locking scenario:
>>
>> [ 57.597911] CPU0 CPU1
>> [ 57.597917] ---- ----
>> [ 57.597922] rlock(&sb->s_type->i_mutex_key#3);
>> [ 57.597932] lock(&q->debugfs_mutex);
>> [ 57.597940] lock(&sb->s_type->i_mutex_key#3);
>> [ 57.597950] rlock(&mm->mmap_lock);
>> [ 57.597958]
>> *** DEADLOCK ***
>>
>> [ 57.597965] 2 locks held by ls/4605:
>> [ 57.597971] #0: c0000000137c12f8 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos+0xcc/0x154
>> [ 57.597989] #1: c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
>>
>> Prevent the above lockdep warning by acquiring ->sysfs_lock before
>> freezing the queue while storing a queue attribute in queue_attr_store
>> function. Later, we also found[1] another function __blk_mq_update_nr_
>> hw_queues where we first freeze queue and then acquire the ->sysfs_lock.
>> So we've also updated lock ordering in __blk_mq_update_nr_hw_queues
>> function and ensured that in all code paths we follow the correct lock
>> ordering i.e. acquire ->sysfs_lock before freezing the queue.
>>
>> [1] https://lore.kernel.org/all/CAFj5m9Ke8+EHKQBs_Nk6hqd=LGXtk4mUxZUN5==ZcCjnZSBwHw@mail.gmail.com/
>>
>> Reported-by: kjain@linux.ibm.com
>> Fixes: af2814149883 ("block: freeze the queue in queue_attr_store")
>> Tested-by: kjain@linux.ibm.com
>> Cc: hch@lst.de
>> Cc: axboe@kernel.dk
>> Cc: ritesh.list@gmail.com
>> Cc: ming.lei@redhat.com
>> Cc: gjoyce@linux.ibm.com
>> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
>> ---
>> Changes from v1:
>> - Fix lock ordering in __blk_mq_update_nr_hw_queues (Ming Lei)
>> ---
>> block/blk-mq-sysfs.c | 16 ++++++----------
>> block/blk-mq.c | 29 ++++++++++++++++++-----------
>> block/blk-sysfs.c | 4 ++--
>> 3 files changed, 26 insertions(+), 23 deletions(-)
>>
>> diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c
>> index 156e9bb07abf..cd5ea6eaa76b 100644
>> --- a/block/blk-mq-sysfs.c
>> +++ b/block/blk-mq-sysfs.c
>> @@ -275,15 +275,13 @@ void blk_mq_sysfs_unregister_hctxs(struct request_queue *q)
>> struct blk_mq_hw_ctx *hctx;
>> unsigned long i;
>>
>> - mutex_lock(&q->sysfs_dir_lock);
>> + lockdep_assert_held(&q->sysfs_dir_lock);
>> +
>> if (!q->mq_sysfs_init_done)
>> - goto unlock;
>> + return;
>>
>> queue_for_each_hw_ctx(q, hctx, i)
>> blk_mq_unregister_hctx(hctx);
>> -
>> -unlock:
>> - mutex_unlock(&q->sysfs_dir_lock);
>> }
>>
>> int blk_mq_sysfs_register_hctxs(struct request_queue *q)
>> @@ -292,9 +290,10 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
>> unsigned long i;
>> int ret = 0;
>>
>> - mutex_lock(&q->sysfs_dir_lock);
>> + lockdep_assert_held(&q->sysfs_dir_lock);
>> +
>> if (!q->mq_sysfs_init_done)
>> - goto unlock;
>> + return ret;
>>
>> queue_for_each_hw_ctx(q, hctx, i) {
>> ret = blk_mq_register_hctx(hctx);
>> @@ -302,8 +301,5 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
>> break;
>> }
>>
>> -unlock:
>> - mutex_unlock(&q->sysfs_dir_lock);
>> -
>> return ret;
>> }
>> diff --git a/block/blk-mq.c b/block/blk-mq.c
>> index aa340b097b6e..cccfc6dada7e 100644
>> --- a/block/blk-mq.c
>> +++ b/block/blk-mq.c
>> @@ -4455,7 +4455,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
>> unsigned long i, j;
>>
>> /* protect against switching io scheduler */
>> - mutex_lock(&q->sysfs_lock);
>> + lockdep_assert_held(&q->sysfs_lock);
>> +
>> for (i = 0; i < set->nr_hw_queues; i++) {
>> int old_node;
>> int node = blk_mq_get_hctx_node(set, i);
>> @@ -4488,7 +4489,6 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
>>
>> xa_for_each_start(&q->hctx_table, j, hctx, j)
>> blk_mq_exit_hctx(q, set, hctx, j);
>> - mutex_unlock(&q->sysfs_lock);
>>
>> /* unregister cpuhp callbacks for exited hctxs */
>> blk_mq_remove_hw_queues_cpuhp(q);
>> @@ -4520,10 +4520,14 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
>>
>> xa_init(&q->hctx_table);
>>
>> + mutex_lock(&q->sysfs_lock);
>> +
>> blk_mq_realloc_hw_ctxs(set, q);
>> if (!q->nr_hw_queues)
>> goto err_hctxs;
>>
>> + mutex_unlock(&q->sysfs_lock);
>> +
>> INIT_WORK(&q->timeout_work, blk_mq_timeout_work);
>> blk_queue_rq_timeout(q, set->timeout ? set->timeout : 30 * HZ);
>>
>> @@ -4542,6 +4546,7 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
>> return 0;
>>
>> err_hctxs:
>> + mutex_unlock(&q->sysfs_lock);
>
> The change in blk_mq_init_allocated_queue() isn't necessary, since queue
> kobj isn't exposed yet, otherwise, feel free to add:
Yeah agreed but the reason I added the ->sysfs_lock is because now
blk_mq_realloc_hw_ctxs() expects the ->sysfs_lock is held before
it's invoked. The reason being when blk_mq_realloc_hw_ctxs() is
invoked from __blk_mq_update_nr_hw_queues(), the ->sysfs_lock must be
held.
>
> Reviewed-by: Ming Lei <ming.lei@redhat.com>
>
>
Thanks,
--Nilay
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
2024-12-11 12:32 ` Nilay Shroff
@ 2024-12-11 12:49 ` Ming Lei
0 siblings, 0 replies; 6+ messages in thread
From: Ming Lei @ 2024-12-11 12:49 UTC (permalink / raw)
To: Nilay Shroff; +Cc: linux-block, kjain, hch, axboe, ritesh.list, gjoyce
On Wed, Dec 11, 2024 at 8:32 PM Nilay Shroff <nilay@linux.ibm.com> wrote:
>
>
>
> On 12/11/24 17:30, Ming Lei wrote:
> > On Tue, Dec 10, 2024 at 08:11:43PM +0530, Nilay Shroff wrote:
> >> For storing a value to a queue attribute, the queue_attr_store function
> >> first freezes the queue (->q_usage_counter(io)) and then acquire
> >> ->sysfs_lock. This seems not correct as the usual ordering should be to
> >> acquire ->sysfs_lock before freezing the queue. This incorrect ordering
> >> causes the following lockdep splat which we are able to reproduce always
> >> simply by accessing /sys/kernel/debug file using ls command:
> >>
> >> [ 57.597146] WARNING: possible circular locking dependency detected
> >> [ 57.597154] 6.12.0-10553-gb86545e02e8c #20 Tainted: G W
> >> [ 57.597162] ------------------------------------------------------
> >> [ 57.597168] ls/4605 is trying to acquire lock:
> >> [ 57.597176] c00000003eb56710 (&mm->mmap_lock){++++}-{4:4}, at: __might_fault+0x58/0xc0
> >> [ 57.597200]
> >> but task is already holding lock:
> >> [ 57.597207] c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
> >> [ 57.597226]
> >> which lock already depends on the new lock.
> >>
> >> [ 57.597233]
> >> the existing dependency chain (in reverse order) is:
> >> [ 57.597241]
> >> -> #5 (&sb->s_type->i_mutex_key#3){++++}-{4:4}:
> >> [ 57.597255] down_write+0x6c/0x18c
> >> [ 57.597264] start_creating+0xb4/0x24c
> >> [ 57.597274] debugfs_create_dir+0x2c/0x1e8
> >> [ 57.597283] blk_register_queue+0xec/0x294
> >> [ 57.597292] add_disk_fwnode+0x2e4/0x548
> >> [ 57.597302] brd_alloc+0x2c8/0x338
> >> [ 57.597309] brd_init+0x100/0x178
> >> [ 57.597317] do_one_initcall+0x88/0x3e4
> >> [ 57.597326] kernel_init_freeable+0x3cc/0x6e0
> >> [ 57.597334] kernel_init+0x34/0x1cc
> >> [ 57.597342] ret_from_kernel_user_thread+0x14/0x1c
> >> [ 57.597350]
> >> -> #4 (&q->debugfs_mutex){+.+.}-{4:4}:
> >> [ 57.597362] __mutex_lock+0xfc/0x12a0
> >> [ 57.597370] blk_register_queue+0xd4/0x294
> >> [ 57.597379] add_disk_fwnode+0x2e4/0x548
> >> [ 57.597388] brd_alloc+0x2c8/0x338
> >> [ 57.597395] brd_init+0x100/0x178
> >> [ 57.597402] do_one_initcall+0x88/0x3e4
> >> [ 57.597410] kernel_init_freeable+0x3cc/0x6e0
> >> [ 57.597418] kernel_init+0x34/0x1cc
> >> [ 57.597426] ret_from_kernel_user_thread+0x14/0x1c
> >> [ 57.597434]
> >> -> #3 (&q->sysfs_lock){+.+.}-{4:4}:
> >> [ 57.597446] __mutex_lock+0xfc/0x12a0
> >> [ 57.597454] queue_attr_store+0x9c/0x110
> >> [ 57.597462] sysfs_kf_write+0x70/0xb0
> >> [ 57.597471] kernfs_fop_write_iter+0x1b0/0x2ac
> >> [ 57.597480] vfs_write+0x3dc/0x6e8
> >> [ 57.597488] ksys_write+0x84/0x140
> >> [ 57.597495] system_call_exception+0x130/0x360
> >> [ 57.597504] system_call_common+0x160/0x2c4
> >> [ 57.597516]
> >> -> #2 (&q->q_usage_counter(io)#21){++++}-{0:0}:
> >> [ 57.597530] __submit_bio+0x5ec/0x828
> >> [ 57.597538] submit_bio_noacct_nocheck+0x1e4/0x4f0
> >> [ 57.597547] iomap_readahead+0x2a0/0x448
> >> [ 57.597556] xfs_vm_readahead+0x28/0x3c
> >> [ 57.597564] read_pages+0x88/0x41c
> >> [ 57.597571] page_cache_ra_unbounded+0x1ac/0x2d8
> >> [ 57.597580] filemap_get_pages+0x188/0x984
> >> [ 57.597588] filemap_read+0x13c/0x4bc
> >> [ 57.597596] xfs_file_buffered_read+0x88/0x17c
> >> [ 57.597605] xfs_file_read_iter+0xac/0x158
> >> [ 57.597614] vfs_read+0x2d4/0x3b4
> >> [ 57.597622] ksys_read+0x84/0x144
> >> [ 57.597629] system_call_exception+0x130/0x360
> >> [ 57.597637] system_call_common+0x160/0x2c4
> >> [ 57.597647]
> >> -> #1 (mapping.invalidate_lock#2){++++}-{4:4}:
> >> [ 57.597661] down_read+0x6c/0x220
> >> [ 57.597669] filemap_fault+0x870/0x100c
> >> [ 57.597677] xfs_filemap_fault+0xc4/0x18c
> >> [ 57.597684] __do_fault+0x64/0x164
> >> [ 57.597693] __handle_mm_fault+0x1274/0x1dac
> >> [ 57.597702] handle_mm_fault+0x248/0x484
> >> [ 57.597711] ___do_page_fault+0x428/0xc0c
> >> [ 57.597719] hash__do_page_fault+0x30/0x68
> >> [ 57.597727] do_hash_fault+0x90/0x35c
> >> [ 57.597736] data_access_common_virt+0x210/0x220
> >> [ 57.597745] _copy_from_user+0xf8/0x19c
> >> [ 57.597754] sel_write_load+0x178/0xd54
> >> [ 57.597762] vfs_write+0x108/0x6e8
> >> [ 57.597769] ksys_write+0x84/0x140
> >> [ 57.597777] system_call_exception+0x130/0x360
> >> [ 57.597785] system_call_common+0x160/0x2c4
> >> [ 57.597794]
> >> -> #0 (&mm->mmap_lock){++++}-{4:4}:
> >> [ 57.597806] __lock_acquire+0x17cc/0x2330
> >> [ 57.597814] lock_acquire+0x138/0x400
> >> [ 57.597822] __might_fault+0x7c/0xc0
> >> [ 57.597830] filldir64+0xe8/0x390
> >> [ 57.597839] dcache_readdir+0x80/0x2d4
> >> [ 57.597846] iterate_dir+0xd8/0x1d4
> >> [ 57.597855] sys_getdents64+0x88/0x2d4
> >> [ 57.597864] system_call_exception+0x130/0x360
> >> [ 57.597872] system_call_common+0x160/0x2c4
> >> [ 57.597881]
> >> other info that might help us debug this:
> >>
> >> [ 57.597888] Chain exists of:
> >> &mm->mmap_lock --> &q->debugfs_mutex --> &sb->s_type->i_mutex_key#3
> >>
> >> [ 57.597905] Possible unsafe locking scenario:
> >>
> >> [ 57.597911] CPU0 CPU1
> >> [ 57.597917] ---- ----
> >> [ 57.597922] rlock(&sb->s_type->i_mutex_key#3);
> >> [ 57.597932] lock(&q->debugfs_mutex);
> >> [ 57.597940] lock(&sb->s_type->i_mutex_key#3);
> >> [ 57.597950] rlock(&mm->mmap_lock);
> >> [ 57.597958]
> >> *** DEADLOCK ***
> >>
> >> [ 57.597965] 2 locks held by ls/4605:
> >> [ 57.597971] #0: c0000000137c12f8 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos+0xcc/0x154
> >> [ 57.597989] #1: c0000018e27c6810 (&sb->s_type->i_mutex_key#3){++++}-{4:4}, at: iterate_dir+0x94/0x1d4
> >>
> >> Prevent the above lockdep warning by acquiring ->sysfs_lock before
> >> freezing the queue while storing a queue attribute in queue_attr_store
> >> function. Later, we also found[1] another function __blk_mq_update_nr_
> >> hw_queues where we first freeze queue and then acquire the ->sysfs_lock.
> >> So we've also updated lock ordering in __blk_mq_update_nr_hw_queues
> >> function and ensured that in all code paths we follow the correct lock
> >> ordering i.e. acquire ->sysfs_lock before freezing the queue.
> >>
> >> [1] https://lore.kernel.org/all/CAFj5m9Ke8+EHKQBs_Nk6hqd=LGXtk4mUxZUN5==ZcCjnZSBwHw@mail.gmail.com/
> >>
> >> Reported-by: kjain@linux.ibm.com
> >> Fixes: af2814149883 ("block: freeze the queue in queue_attr_store")
> >> Tested-by: kjain@linux.ibm.com
> >> Cc: hch@lst.de
> >> Cc: axboe@kernel.dk
> >> Cc: ritesh.list@gmail.com
> >> Cc: ming.lei@redhat.com
> >> Cc: gjoyce@linux.ibm.com
> >> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
> >> ---
> >> Changes from v1:
> >> - Fix lock ordering in __blk_mq_update_nr_hw_queues (Ming Lei)
> >> ---
> >> block/blk-mq-sysfs.c | 16 ++++++----------
> >> block/blk-mq.c | 29 ++++++++++++++++++-----------
> >> block/blk-sysfs.c | 4 ++--
> >> 3 files changed, 26 insertions(+), 23 deletions(-)
> >>
> >> diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c
> >> index 156e9bb07abf..cd5ea6eaa76b 100644
> >> --- a/block/blk-mq-sysfs.c
> >> +++ b/block/blk-mq-sysfs.c
> >> @@ -275,15 +275,13 @@ void blk_mq_sysfs_unregister_hctxs(struct request_queue *q)
> >> struct blk_mq_hw_ctx *hctx;
> >> unsigned long i;
> >>
> >> - mutex_lock(&q->sysfs_dir_lock);
> >> + lockdep_assert_held(&q->sysfs_dir_lock);
> >> +
> >> if (!q->mq_sysfs_init_done)
> >> - goto unlock;
> >> + return;
> >>
> >> queue_for_each_hw_ctx(q, hctx, i)
> >> blk_mq_unregister_hctx(hctx);
> >> -
> >> -unlock:
> >> - mutex_unlock(&q->sysfs_dir_lock);
> >> }
> >>
> >> int blk_mq_sysfs_register_hctxs(struct request_queue *q)
> >> @@ -292,9 +290,10 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
> >> unsigned long i;
> >> int ret = 0;
> >>
> >> - mutex_lock(&q->sysfs_dir_lock);
> >> + lockdep_assert_held(&q->sysfs_dir_lock);
> >> +
> >> if (!q->mq_sysfs_init_done)
> >> - goto unlock;
> >> + return ret;
> >>
> >> queue_for_each_hw_ctx(q, hctx, i) {
> >> ret = blk_mq_register_hctx(hctx);
> >> @@ -302,8 +301,5 @@ int blk_mq_sysfs_register_hctxs(struct request_queue *q)
> >> break;
> >> }
> >>
> >> -unlock:
> >> - mutex_unlock(&q->sysfs_dir_lock);
> >> -
> >> return ret;
> >> }
> >> diff --git a/block/blk-mq.c b/block/blk-mq.c
> >> index aa340b097b6e..cccfc6dada7e 100644
> >> --- a/block/blk-mq.c
> >> +++ b/block/blk-mq.c
> >> @@ -4455,7 +4455,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
> >> unsigned long i, j;
> >>
> >> /* protect against switching io scheduler */
> >> - mutex_lock(&q->sysfs_lock);
> >> + lockdep_assert_held(&q->sysfs_lock);
> >> +
> >> for (i = 0; i < set->nr_hw_queues; i++) {
> >> int old_node;
> >> int node = blk_mq_get_hctx_node(set, i);
> >> @@ -4488,7 +4489,6 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
> >>
> >> xa_for_each_start(&q->hctx_table, j, hctx, j)
> >> blk_mq_exit_hctx(q, set, hctx, j);
> >> - mutex_unlock(&q->sysfs_lock);
> >>
> >> /* unregister cpuhp callbacks for exited hctxs */
> >> blk_mq_remove_hw_queues_cpuhp(q);
> >> @@ -4520,10 +4520,14 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
> >>
> >> xa_init(&q->hctx_table);
> >>
> >> + mutex_lock(&q->sysfs_lock);
> >> +
> >> blk_mq_realloc_hw_ctxs(set, q);
> >> if (!q->nr_hw_queues)
> >> goto err_hctxs;
> >>
> >> + mutex_unlock(&q->sysfs_lock);
> >> +
> >> INIT_WORK(&q->timeout_work, blk_mq_timeout_work);
> >> blk_queue_rq_timeout(q, set->timeout ? set->timeout : 30 * HZ);
> >>
> >> @@ -4542,6 +4546,7 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
> >> return 0;
> >>
> >> err_hctxs:
> >> + mutex_unlock(&q->sysfs_lock);
> >
> > The change in blk_mq_init_allocated_queue() isn't necessary, since queue
> > kobj isn't exposed yet, otherwise, feel free to add:
> Yeah agreed but the reason I added the ->sysfs_lock is because now
> blk_mq_realloc_hw_ctxs() expects the ->sysfs_lock is held before
> it's invoked. The reason being when blk_mq_realloc_hw_ctxs() is
> invoked from __blk_mq_update_nr_hw_queues(), the ->sysfs_lock must be
> held.
>
Fair enough, now I am fine:
Reviewed-by: Ming Lei <ming.lei@redhat.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
2024-12-10 14:41 [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock Nilay Shroff
2024-12-11 12:00 ` Ming Lei
@ 2024-12-13 17:52 ` Jens Axboe
1 sibling, 0 replies; 6+ messages in thread
From: Jens Axboe @ 2024-12-13 17:52 UTC (permalink / raw)
To: linux-block, Nilay Shroff; +Cc: kjain, hch, ritesh.list, ming.lei, gjoyce
On Tue, 10 Dec 2024 20:11:43 +0530, Nilay Shroff wrote:
> For storing a value to a queue attribute, the queue_attr_store function
> first freezes the queue (->q_usage_counter(io)) and then acquire
> ->sysfs_lock. This seems not correct as the usual ordering should be to
> acquire ->sysfs_lock before freezing the queue. This incorrect ordering
> causes the following lockdep splat which we are able to reproduce always
> simply by accessing /sys/kernel/debug file using ls command:
>
> [...]
Applied, thanks!
[1/1] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock
commit: be26ba96421ab0a8fa2055ccf7db7832a13c44d2
Best regards,
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-12-13 17:52 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-10 14:41 [PATCHv2] block: Fix potential deadlock while freezing queue and acquiring sysfs_lock Nilay Shroff
2024-12-11 12:00 ` Ming Lei
2024-12-11 12:32 ` Nilay Shroff
2024-12-11 12:49 ` Ming Lei
2024-12-13 17:52 ` Jens Axboe
-- strict thread matches above, loose matches on Subject: below --
2024-12-11 8:19 kernel test robot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.