* [PATCH] nvme-multipath: fix suspicious RCU usage warning
@ 2025-06-30 7:48 Geliang Tang
2025-06-30 9:30 ` Hannes Reinecke
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Geliang Tang @ 2025-06-30 7:48 UTC (permalink / raw)
To: Hannes Reinecke, Keith Busch, Jens Axboe, Christoph Hellwig,
Sagi Grimberg, Nilay Shroff
Cc: geliang, Geliang Tang, linux-nvme
From: Geliang Tang <tanggeliang@kylinos.cn>
When I run the NVME over TCP test in virtme-ng, I get the following
"suspicious RCU usage" warning in nvme_mpath_add_sysfs_link():
'''
[ 5.024557][ T44] nvmet: Created nvm controller 1 for subsystem nqn.2025-06.org.nvmexpress.mptcp for NQN nqn.2014-08.org.nvmexpress:uuid:f7f6b5e0-ff97-4894-98ac-c85309e0bc77.
[ 5.027401][ T183] nvme nvme0: creating 2 I/O queues.
[ 5.029017][ T183] nvme nvme0: mapped 2/0/0 default/read/poll queues.
[ 5.032587][ T183] nvme nvme0: new ctrl: NQN "nqn.2025-06.org.nvmexpress.mptcp", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:f7f6b5e0-ff97-4894-98ac-c85309e0bc77
[ 5.042214][ T25]
[ 5.042440][ T25] =============================
[ 5.042579][ T25] WARNING: suspicious RCU usage
[ 5.042705][ T25] 6.16.0-rc3+ #23 Not tainted
[ 5.042812][ T25] -----------------------------
[ 5.042934][ T25] drivers/nvme/host/multipath.c:1203 RCU-list traversed in non-reader section!!
[ 5.043111][ T25]
[ 5.043111][ T25] other info that might help us debug this:
[ 5.043111][ T25]
[ 5.043341][ T25]
[ 5.043341][ T25] rcu_scheduler_active = 2, debug_locks = 1
[ 5.043502][ T25] 3 locks held by kworker/u9:0/25:
[ 5.043615][ T25] #0: ffff888008730948 ((wq_completion)async){+.+.}-{0:0}, at: process_one_work+0x7ed/0x1350
[ 5.043830][ T25] #1: ffffc900001afd40 ((work_completion)(&entry->work)){+.+.}-{0:0}, at: process_one_work+0xcf3/0x1350
[ 5.044084][ T25] #2: ffff888013ee0020 (&head->srcu){.+.+}-{0:0}, at: nvme_mpath_add_sysfs_link.part.0+0xb4/0x3a0
[ 5.044300][ T25]
[ 5.044300][ T25] stack backtrace:
[ 5.044439][ T25] CPU: 0 UID: 0 PID: 25 Comm: kworker/u9:0 Not tainted 6.16.0-rc3+ #23 PREEMPT(full)
[ 5.044441][ T25] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 5.044442][ T25] Workqueue: async async_run_entry_fn
[ 5.044445][ T25] Call Trace:
[ 5.044446][ T25] <TASK>
[ 5.044449][ T25] dump_stack_lvl+0x6f/0xb0
[ 5.044453][ T25] lockdep_rcu_suspicious.cold+0x4f/0xb1
[ 5.044457][ T25] nvme_mpath_add_sysfs_link.part.0+0x2fb/0x3a0
[ 5.044459][ T25] ? queue_work_on+0x90/0xf0
[ 5.044461][ T25] ? lockdep_hardirqs_on+0x78/0x110
[ 5.044466][ T25] nvme_mpath_set_live+0x1e9/0x4f0
[ 5.044470][ T25] nvme_mpath_add_disk+0x240/0x2f0
[ 5.044472][ T25] ? __pfx_nvme_mpath_add_disk+0x10/0x10
[ 5.044475][ T25] ? add_disk_fwnode+0x361/0x580
[ 5.044480][ T25] nvme_alloc_ns+0x81c/0x17c0
[ 5.044483][ T25] ? kasan_quarantine_put+0x104/0x240
[ 5.044487][ T25] ? __pfx_nvme_alloc_ns+0x10/0x10
[ 5.044495][ T25] ? __pfx_nvme_find_get_ns+0x10/0x10
[ 5.044496][ T25] ? rcu_read_lock_any_held+0x45/0xa0
[ 5.044498][ T25] ? validate_chain+0x232/0x4f0
[ 5.044503][ T25] nvme_scan_ns+0x4c8/0x810
[ 5.044506][ T25] ? __pfx_nvme_scan_ns+0x10/0x10
[ 5.044508][ T25] ? find_held_lock+0x2b/0x80
[ 5.044512][ T25] ? ktime_get+0x16d/0x220
[ 5.044517][ T25] ? kvm_clock_get_cycles+0x18/0x30
[ 5.044520][ T25] ? __pfx_nvme_scan_ns_async+0x10/0x10
[ 5.044522][ T25] async_run_entry_fn+0x97/0x560
[ 5.044523][ T25] ? rcu_is_watching+0x12/0xc0
[ 5.044526][ T25] process_one_work+0xd3c/0x1350
[ 5.044532][ T25] ? __pfx_process_one_work+0x10/0x10
[ 5.044536][ T25] ? assign_work+0x16c/0x240
[ 5.044539][ T25] worker_thread+0x4da/0xd50
[ 5.044545][ T25] ? __pfx_worker_thread+0x10/0x10
[ 5.044546][ T25] kthread+0x356/0x5c0
[ 5.044548][ T25] ? __pfx_kthread+0x10/0x10
[ 5.044549][ T25] ? ret_from_fork+0x1b/0x2e0
[ 5.044552][ T25] ? __lock_release.isra.0+0x5d/0x180
[ 5.044553][ T25] ? ret_from_fork+0x1b/0x2e0
[ 5.044555][ T25] ? rcu_is_watching+0x12/0xc0
[ 5.044557][ T25] ? __pfx_kthread+0x10/0x10
[ 5.044559][ T25] ret_from_fork+0x218/0x2e0
[ 5.044561][ T25] ? __pfx_kthread+0x10/0x10
[ 5.044562][ T25] ret_from_fork_asm+0x1a/0x30
[ 5.044570][ T25] </TASK>
'''
This patch uses sleepable RCU version of helper list_for_each_entry_srcu()
instead of list_for_each_entry_rcu() to fix it.
Fixes: 4dbd2b2ebe4c ("nvme-multipath: Add visibility for round-robin io-policy")
Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
---
drivers/nvme/host/multipath.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index e040e467f9fa..5e11b0fdf250 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -1200,7 +1200,8 @@ void nvme_mpath_add_sysfs_link(struct nvme_ns_head *head)
*/
srcu_idx = srcu_read_lock(&head->srcu);
- list_for_each_entry_rcu(ns, &head->list, siblings) {
+ list_for_each_entry_srcu(ns, &head->list, siblings,
+ srcu_read_lock_held(&head->srcu)) {
/*
* Ensure that ns path disk node is already added otherwise we
* may get invalid kobj name for target
--
2.48.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] nvme-multipath: fix suspicious RCU usage warning
2025-06-30 7:48 [PATCH] nvme-multipath: fix suspicious RCU usage warning Geliang Tang
@ 2025-06-30 9:30 ` Hannes Reinecke
2025-06-30 10:45 ` Nilay Shroff
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Hannes Reinecke @ 2025-06-30 9:30 UTC (permalink / raw)
To: Geliang Tang, Keith Busch, Jens Axboe, Christoph Hellwig,
Sagi Grimberg, Nilay Shroff
Cc: Geliang Tang, linux-nvme
On 6/30/25 09:48, Geliang Tang wrote:
> From: Geliang Tang <tanggeliang@kylinos.cn>
>
> When I run the NVME over TCP test in virtme-ng, I get the following
> "suspicious RCU usage" warning in nvme_mpath_add_sysfs_link():
>
> '''
> [ 5.024557][ T44] nvmet: Created nvm controller 1 for subsystem nqn.2025-06.org.nvmexpress.mptcp for NQN nqn.2014-08.org.nvmexpress:uuid:f7f6b5e0-ff97-4894-98ac-c85309e0bc77.
> [ 5.027401][ T183] nvme nvme0: creating 2 I/O queues.
> [ 5.029017][ T183] nvme nvme0: mapped 2/0/0 default/read/poll queues.
> [ 5.032587][ T183] nvme nvme0: new ctrl: NQN "nqn.2025-06.org.nvmexpress.mptcp", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:f7f6b5e0-ff97-4894-98ac-c85309e0bc77
> [ 5.042214][ T25]
> [ 5.042440][ T25] =============================
> [ 5.042579][ T25] WARNING: suspicious RCU usage
> [ 5.042705][ T25] 6.16.0-rc3+ #23 Not tainted
> [ 5.042812][ T25] -----------------------------
> [ 5.042934][ T25] drivers/nvme/host/multipath.c:1203 RCU-list traversed in non-reader section!!
> [ 5.043111][ T25]
> [ 5.043111][ T25] other info that might help us debug this:
> [ 5.043111][ T25]
> [ 5.043341][ T25]
> [ 5.043341][ T25] rcu_scheduler_active = 2, debug_locks = 1
> [ 5.043502][ T25] 3 locks held by kworker/u9:0/25:
> [ 5.043615][ T25] #0: ffff888008730948 ((wq_completion)async){+.+.}-{0:0}, at: process_one_work+0x7ed/0x1350
> [ 5.043830][ T25] #1: ffffc900001afd40 ((work_completion)(&entry->work)){+.+.}-{0:0}, at: process_one_work+0xcf3/0x1350
> [ 5.044084][ T25] #2: ffff888013ee0020 (&head->srcu){.+.+}-{0:0}, at: nvme_mpath_add_sysfs_link.part.0+0xb4/0x3a0
> [ 5.044300][ T25]
> [ 5.044300][ T25] stack backtrace:
> [ 5.044439][ T25] CPU: 0 UID: 0 PID: 25 Comm: kworker/u9:0 Not tainted 6.16.0-rc3+ #23 PREEMPT(full)
> [ 5.044441][ T25] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [ 5.044442][ T25] Workqueue: async async_run_entry_fn
> [ 5.044445][ T25] Call Trace:
> [ 5.044446][ T25] <TASK>
> [ 5.044449][ T25] dump_stack_lvl+0x6f/0xb0
> [ 5.044453][ T25] lockdep_rcu_suspicious.cold+0x4f/0xb1
> [ 5.044457][ T25] nvme_mpath_add_sysfs_link.part.0+0x2fb/0x3a0
> [ 5.044459][ T25] ? queue_work_on+0x90/0xf0
> [ 5.044461][ T25] ? lockdep_hardirqs_on+0x78/0x110
> [ 5.044466][ T25] nvme_mpath_set_live+0x1e9/0x4f0
> [ 5.044470][ T25] nvme_mpath_add_disk+0x240/0x2f0
> [ 5.044472][ T25] ? __pfx_nvme_mpath_add_disk+0x10/0x10
> [ 5.044475][ T25] ? add_disk_fwnode+0x361/0x580
> [ 5.044480][ T25] nvme_alloc_ns+0x81c/0x17c0
> [ 5.044483][ T25] ? kasan_quarantine_put+0x104/0x240
> [ 5.044487][ T25] ? __pfx_nvme_alloc_ns+0x10/0x10
> [ 5.044495][ T25] ? __pfx_nvme_find_get_ns+0x10/0x10
> [ 5.044496][ T25] ? rcu_read_lock_any_held+0x45/0xa0
> [ 5.044498][ T25] ? validate_chain+0x232/0x4f0
> [ 5.044503][ T25] nvme_scan_ns+0x4c8/0x810
> [ 5.044506][ T25] ? __pfx_nvme_scan_ns+0x10/0x10
> [ 5.044508][ T25] ? find_held_lock+0x2b/0x80
> [ 5.044512][ T25] ? ktime_get+0x16d/0x220
> [ 5.044517][ T25] ? kvm_clock_get_cycles+0x18/0x30
> [ 5.044520][ T25] ? __pfx_nvme_scan_ns_async+0x10/0x10
> [ 5.044522][ T25] async_run_entry_fn+0x97/0x560
> [ 5.044523][ T25] ? rcu_is_watching+0x12/0xc0
> [ 5.044526][ T25] process_one_work+0xd3c/0x1350
> [ 5.044532][ T25] ? __pfx_process_one_work+0x10/0x10
> [ 5.044536][ T25] ? assign_work+0x16c/0x240
> [ 5.044539][ T25] worker_thread+0x4da/0xd50
> [ 5.044545][ T25] ? __pfx_worker_thread+0x10/0x10
> [ 5.044546][ T25] kthread+0x356/0x5c0
> [ 5.044548][ T25] ? __pfx_kthread+0x10/0x10
> [ 5.044549][ T25] ? ret_from_fork+0x1b/0x2e0
> [ 5.044552][ T25] ? __lock_release.isra.0+0x5d/0x180
> [ 5.044553][ T25] ? ret_from_fork+0x1b/0x2e0
> [ 5.044555][ T25] ? rcu_is_watching+0x12/0xc0
> [ 5.044557][ T25] ? __pfx_kthread+0x10/0x10
> [ 5.044559][ T25] ret_from_fork+0x218/0x2e0
> [ 5.044561][ T25] ? __pfx_kthread+0x10/0x10
> [ 5.044562][ T25] ret_from_fork_asm+0x1a/0x30
> [ 5.044570][ T25] </TASK>
> '''
>
> This patch uses sleepable RCU version of helper list_for_each_entry_srcu()
> instead of list_for_each_entry_rcu() to fix it.
>
> Fixes: 4dbd2b2ebe4c ("nvme-multipath: Add visibility for round-robin io-policy")
> Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
> ---
> drivers/nvme/host/multipath.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index e040e467f9fa..5e11b0fdf250 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -1200,7 +1200,8 @@ void nvme_mpath_add_sysfs_link(struct nvme_ns_head *head)
> */
> srcu_idx = srcu_read_lock(&head->srcu);
>
> - list_for_each_entry_rcu(ns, &head->list, siblings) {
> + list_for_each_entry_srcu(ns, &head->list, siblings,
> + srcu_read_lock_held(&head->srcu)) {
> /*
> * Ensure that ns path disk node is already added otherwise we
> * may get invalid kobj name for target
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] nvme-multipath: fix suspicious RCU usage warning
2025-06-30 7:48 [PATCH] nvme-multipath: fix suspicious RCU usage warning Geliang Tang
2025-06-30 9:30 ` Hannes Reinecke
@ 2025-06-30 10:45 ` Nilay Shroff
2025-06-30 13:44 ` Christoph Hellwig
2025-06-30 16:44 ` Keith Busch
3 siblings, 0 replies; 5+ messages in thread
From: Nilay Shroff @ 2025-06-30 10:45 UTC (permalink / raw)
To: Geliang Tang, Hannes Reinecke, Keith Busch, Jens Axboe,
Christoph Hellwig, Sagi Grimberg
Cc: Geliang Tang, linux-nvme
On 6/30/25 1:18 PM, Geliang Tang wrote:
> This patch uses sleepable RCU version of helper list_for_each_entry_srcu()
> instead of list_for_each_entry_rcu() to fix it.
>
> Fixes: 4dbd2b2ebe4c ("nvme-multipath: Add visibility for round-robin io-policy")
> Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>
Looks good to me:
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] nvme-multipath: fix suspicious RCU usage warning
2025-06-30 7:48 [PATCH] nvme-multipath: fix suspicious RCU usage warning Geliang Tang
2025-06-30 9:30 ` Hannes Reinecke
2025-06-30 10:45 ` Nilay Shroff
@ 2025-06-30 13:44 ` Christoph Hellwig
2025-06-30 16:44 ` Keith Busch
3 siblings, 0 replies; 5+ messages in thread
From: Christoph Hellwig @ 2025-06-30 13:44 UTC (permalink / raw)
To: Geliang Tang
Cc: Hannes Reinecke, Keith Busch, Jens Axboe, Christoph Hellwig,
Sagi Grimberg, Nilay Shroff, Geliang Tang, linux-nvme
Thanks,
applied to nvme-6.16.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] nvme-multipath: fix suspicious RCU usage warning
2025-06-30 7:48 [PATCH] nvme-multipath: fix suspicious RCU usage warning Geliang Tang
` (2 preceding siblings ...)
2025-06-30 13:44 ` Christoph Hellwig
@ 2025-06-30 16:44 ` Keith Busch
3 siblings, 0 replies; 5+ messages in thread
From: Keith Busch @ 2025-06-30 16:44 UTC (permalink / raw)
To: Geliang Tang
Cc: Hannes Reinecke, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
Nilay Shroff, Geliang Tang, linux-nvme
On Mon, Jun 30, 2025 at 03:48:32PM +0800, Geliang Tang wrote:
> This patch uses sleepable RCU version of helper list_for_each_entry_srcu()
> instead of list_for_each_entry_rcu() to fix it.
Looks good.
Reviewed-by: Keith Busch <kbusch@kernel.org>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2025-06-30 17:53 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-06-30 7:48 [PATCH] nvme-multipath: fix suspicious RCU usage warning Geliang Tang
2025-06-30 9:30 ` Hannes Reinecke
2025-06-30 10:45 ` Nilay Shroff
2025-06-30 13:44 ` Christoph Hellwig
2025-06-30 16:44 ` Keith Busch
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox