* [RFC PATCH 1/5] nbd: validate the backend identifier under config_lock
2026-11-07 6:46 [RFC PATCH 0/5] nbd: allow parallel genetlink operations Yang Erkun
@ 2026-11-07 6:46 ` Yang Erkun
2026-11-07 6:46 ` [RFC PATCH 2/5] nbd: decrement nbd_total_devices when a device is removed Yang Erkun
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Yang Erkun @ 2026-11-07 6:46 UTC (permalink / raw)
To: axboe, josef, xiubli, prasanna.kalever, ming.lei, yukuai, nilay
Cc: linux-block, nbd, yangerkun, wangkefeng.wang, leijitang, yi.zhang,
chengzhihao1, houtao1, wozizhi
nbd_genl_reconfigure() reads nbd->backend with only nbd_index_mutex
held, while nbd_config_put() frees it under config_lock. A teardown
running concurrently triggers:
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: Oops: 0000 [#1] SMP KASAN NOPTI
CPU: 0 UID: 0 PID: 819 Comm: a.out Not tainted 7.3.0-dirty #11 PREEMPT(lazy)
RIP: 0010:strlen+0x4/0x30
...
Call Trace:
nla_strcmp+0x20/0x100
nbd_genl_reconfigure+0x1f7/0xd00 [nbd]
genl_family_rcv_msg_doit+0x1e5/0x2c0
genl_family_rcv_msg+0x35f/0x5b0
genl_rcv_msg+0xa3/0x130
netlink_rcv_skb+0x12a/0x390
genl_rcv+0x28/0x40
netlink_unicast+0x7b2/0xb30
netlink_sendmsg+0x6fe/0xb90
__sys_sendto+0x325/0x380
__x64_sys_sendto+0xe0/0x1c0
do_syscall_64+0xc1/0x5e0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
Validate the identifier after taking config_lock and a config
reference. A device that still has a backend identifier but is no
longer bound, for example because it was disconnected while the block
device stayed open, now reports "not configured" instead of a backend
mismatch.
Fixes: 6497ef8df568 ("nbd: provide a way for userspace processes to identify device backends")
Assisted-by: LLM
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
drivers/block/nbd.c | 32 ++++++++++++++++----------------
1 file changed, 16 insertions(+), 16 deletions(-)
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index ffce519bf008..fbade6ae4481 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -2401,22 +2401,6 @@ static int nbd_genl_reconfigure(struct sk_buff *skb, struct genl_info *info)
pr_err("couldn't find a device at index %d\n", index);
return -EINVAL;
}
- if (nbd->backend) {
- if (info->attrs[NBD_ATTR_BACKEND_IDENTIFIER]) {
- if (nla_strcmp(info->attrs[NBD_ATTR_BACKEND_IDENTIFIER],
- nbd->backend)) {
- mutex_unlock(&nbd_index_mutex);
- dev_err(nbd_to_dev(nbd),
- "backend image doesn't match with %s\n",
- nbd->backend);
- return -EINVAL;
- }
- } else {
- mutex_unlock(&nbd_index_mutex);
- dev_err(nbd_to_dev(nbd), "must specify backend\n");
- return -EINVAL;
- }
- }
if (!refcount_inc_not_zero(&nbd->refs)) {
mutex_unlock(&nbd_index_mutex);
pr_err("device at index %d is going down\n", index);
@@ -2441,6 +2425,22 @@ static int nbd_genl_reconfigure(struct sk_buff *skb, struct genl_info *info)
goto out;
}
+ if (nbd->backend) {
+ if (!info->attrs[NBD_ATTR_BACKEND_IDENTIFIER]) {
+ dev_err(nbd_to_dev(nbd), "must specify backend\n");
+ ret = -EINVAL;
+ goto out;
+ }
+ if (nla_strcmp(info->attrs[NBD_ATTR_BACKEND_IDENTIFIER],
+ nbd->backend)) {
+ dev_err(nbd_to_dev(nbd),
+ "backend image doesn't match with %s\n",
+ nbd->backend);
+ ret = -EINVAL;
+ goto out;
+ }
+ }
+
ret = nbd_genl_size_set(info, nbd);
if (ret)
goto out;
--
2.52.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [RFC PATCH 2/5] nbd: decrement nbd_total_devices when a device is removed
2026-11-07 6:46 [RFC PATCH 0/5] nbd: allow parallel genetlink operations Yang Erkun
2026-11-07 6:46 ` [RFC PATCH 1/5] nbd: validate the backend identifier under config_lock Yang Erkun
@ 2026-11-07 6:46 ` Yang Erkun
2026-11-07 6:46 ` [RFC PATCH 3/5] nbd: publish devices atomically Yang Erkun
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Yang Erkun @ 2026-11-07 6:46 UTC (permalink / raw)
To: axboe, josef, xiubli, prasanna.kalever, ming.lei, yukuai, nilay
Cc: linux-block, nbd, yangerkun, wangkefeng.wang, leijitang, yi.zhang,
chengzhihao1, houtao1, wozizhi
nbd_total_devices is incremented in nbd_dev_add() but never decremented,
so it ends up larger than the number of devices left in nbd_index_idr.
nbd_genl_status() sizes the reply from it, so status allocates for
devices that no longer exist.
Decrement it in nbd_dev_remove() under nbd_index_mutex.
Fixes: 47d902b90a32 ("nbd: add a status netlink command")
Assisted-by: LLM
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
drivers/block/nbd.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index fbade6ae4481..1448ee63fab5 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -275,6 +275,7 @@ static void nbd_dev_remove(struct nbd_device *nbd)
*/
mutex_lock(&nbd_index_mutex);
idr_remove(&nbd_index_idr, nbd->index);
+ nbd_total_devices--;
mutex_unlock(&nbd_index_mutex);
destroy_workqueue(nbd->recv_workq);
put_disk(disk);
--
2.52.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [RFC PATCH 3/5] nbd: publish devices atomically
2026-11-07 6:46 [RFC PATCH 0/5] nbd: allow parallel genetlink operations Yang Erkun
2026-11-07 6:46 ` [RFC PATCH 1/5] nbd: validate the backend identifier under config_lock Yang Erkun
2026-11-07 6:46 ` [RFC PATCH 2/5] nbd: decrement nbd_total_devices when a device is removed Yang Erkun
@ 2026-11-07 6:46 ` Yang Erkun
2026-11-07 6:46 ` [RFC PATCH 4/5] nbd: hold config_lock while updating nr_hw_queues Yang Erkun
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Yang Erkun @ 2026-11-07 6:46 UTC (permalink / raw)
To: axboe, josef, xiubli, prasanna.kalever, ming.lei, yukuai, nilay
Cc: linux-block, nbd, yangerkun, wangkefeng.wang, leijitang, yi.zhang,
chengzhihao1, houtao1, wozizhi
nbd_dev_add() inserts the device into nbd_index_idr as soon as it has
an index, but sets nbd->refs and nbd_total_devices only after add_disk(),
the latter without nbd_index_mutex. With parallel_ops enabled in a later
patch, a status walk can then report the half-initialized device while
nbd_total_devices does not account for it yet, and overflow the reply
(-EMSGSIZE).
Set nbd->refs and nbd_total_devices together under nbd_index_mutex, and
skip devices with a zero refcount in populate_nbd_status(). Then
nbd_total_devices is an upper bound on the devices status reports, and a
reply sized from it can't overflow.
Assisted-by: LLM
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
drivers/block/nbd.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 1448ee63fab5..8c2cb312f209 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -2029,10 +2029,15 @@ static struct nbd_device *nbd_dev_add(int index, unsigned int refs,
goto out_free_work;
/*
- * Now publish the device.
+ * Now publish the device. nbd->refs and nbd_total_devices are
+ * updated together under nbd_index_mutex, so a concurrent idr
+ * lookup either sees a fully initialised device or one that it has
+ * to skip because its refcount is still zero.
*/
+ mutex_lock(&nbd_index_mutex);
refcount_set(&nbd->refs, refs);
nbd_total_devices++;
+ mutex_unlock(&nbd_index_mutex);
return nbd;
out_free_work:
@@ -2537,6 +2542,16 @@ static int populate_nbd_status(struct nbd_device *nbd, struct sk_buff *reply)
u8 connected = 0;
int ret;
+ /*
+ * nbd_dev_add() inserts the device into the idr before it is fully
+ * initialised and only publishes it (sets nbd->refs) after
+ * add_disk(); skip a device that is not published yet or is already
+ * going away, so that the walk never reports a device that
+ * nbd_total_devices does not account for.
+ */
+ if (!refcount_read(&nbd->refs))
+ return 0;
+
/* This is a little racey, but for status it's ok. The
* reason we don't take a ref here is because we can't
* take a ref in the index == -1 case as we would need
--
2.52.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [RFC PATCH 4/5] nbd: hold config_lock while updating nr_hw_queues
2026-11-07 6:46 [RFC PATCH 0/5] nbd: allow parallel genetlink operations Yang Erkun
` (2 preceding siblings ...)
2026-11-07 6:46 ` [RFC PATCH 3/5] nbd: publish devices atomically Yang Erkun
@ 2026-11-07 6:46 ` Yang Erkun
2026-11-07 6:46 ` [RFC PATCH 5/5] nbd: allow genetlink operations to run in parallel Yang Erkun
2026-11-17 4:36 ` [RFC PATCH 0/5] nbd: allow parallel genetlink operations yangerkun
5 siblings, 0 replies; 7+ messages in thread
From: Yang Erkun @ 2026-11-07 6:46 UTC (permalink / raw)
To: axboe, josef, xiubli, prasanna.kalever, ming.lei, yukuai, nilay
Cc: linux-block, nbd, yangerkun, wangkefeng.wang, leijitang, yi.zhang,
chengzhihao1, houtao1, wozizhi
Link [1] reports a possible deadlock, and the discussion there shows that
it is impossible. In order to silence the warning, commit 8b428f42f3ed
("nbd: fix lockdep deadlock warning") released nbd->config_lock before
blk_mq_update_nr_hw_queues(), and then relocked it.
But releasing the lock has a real cost. While nbd_start_device() runs
without config_lock, a concurrent disconnect can shut down the sockets,
and setup continues and reports success with a config that is already
dead. This patch fixes it in another way, by using mutex_lock_nested().
Only nbd_genl_connect() and the NBD_DO_IT path in nbd_ioctl() reach
nbd_start_device(), so only these take the separate subclass; the other
ioctls keep the default one and stay fully checked by lockdep.
Fixes: 8b428f42f3ed ("nbd: fix lockdep deadlock warning")
Assisted-by: LLM
Link: https://lore.kernel.org/all/6855034f.a00a0220.137b3.0031.GAE@google.com/T/#u
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
drivers/block/nbd.c | 44 +++++++++++++++++++++++++++++++-------------
1 file changed, 31 insertions(+), 13 deletions(-)
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 8c2cb312f209..cf2a47fc572d 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -1510,6 +1510,31 @@ static void nbd_config_put(struct nbd_device *nbd)
}
}
+/*
+ * nbd_start_device() calls blk_mq_update_nr_hw_queues() with config_lock
+ * held. del_gendisk() takes the tag set's update_nr_hwq_lock and then
+ * disk->open_mutex, while ->open()/->release() run with disk->open_mutex
+ * held and take config_lock (nbd_open(), and nbd_release() through
+ * nbd_disconnect_and_put() or nbd_config_put()), so lockdep reports the
+ * class-level cycle
+ *
+ * config_lock -> update_nr_hwq_lock -> open_mutex -> config_lock
+ *
+ * as a possible deadlock. It cannot happen: nbd_start_device() is called
+ * with an nbd reference held, and the open/release side takes config_lock
+ * while the open file still holds one, so nbd->refs cannot reach zero and
+ * nbd_dev_remove_work() -> del_gendisk() cannot run concurrently. In
+ * addition, nbd_config_put() drops config_lock before nbd_put(), so the
+ * teardown path never calls del_gendisk() with config_lock held.
+ *
+ * Only the acquisitions that can reach nbd_start_device() are annotated:
+ * nbd_genl_connect(), and nbd_ioctl() for NBD_DO_IT (which ends up in
+ * nbd_start_device_ioctl()). No other ioctl reaches it, so they keep the
+ * default subclass and stay fully checked. This only changes the lockdep
+ * annotation, not the locking itself.
+ */
+#define NBD_CONFIG_LOCK_SETUP 1
+
static int nbd_start_device(struct nbd_device *nbd)
{
struct nbd_config *config = nbd->config;
@@ -1526,17 +1551,7 @@ static int nbd_start_device(struct nbd_device *nbd)
return -EINVAL;
}
-retry:
- mutex_unlock(&nbd->config_lock);
- blk_mq_update_nr_hw_queues(&nbd->tag_set, num_connections);
- mutex_lock(&nbd->config_lock);
-
- /* if another code path updated nr_hw_queues, retry until succeed */
- if (num_connections != config->num_connections) {
- num_connections = config->num_connections;
- goto retry;
- }
-
+ blk_mq_update_nr_hw_queues(&nbd->tag_set, config->num_connections);
nbd->pid = task_pid_nr(current);
nbd_parse_flags(nbd);
@@ -1695,7 +1710,10 @@ static int nbd_ioctl(struct block_device *bdev, blk_mode_t mode,
if (_IOC_TYPE(cmd) != 0xab)
return -EINVAL;
- mutex_lock(&nbd->config_lock);
+ if (cmd == NBD_DO_IT)
+ mutex_lock_nested(&nbd->config_lock, NBD_CONFIG_LOCK_SETUP);
+ else
+ mutex_lock(&nbd->config_lock);
/* Don't allow ioctl operations on a nbd device that was created with
* netlink, unless it's DISCONNECT or CLEAR_SOCK, which are fine.
@@ -2233,7 +2251,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
}
}
- mutex_lock(&nbd->config_lock);
+ mutex_lock_nested(&nbd->config_lock, NBD_CONFIG_LOCK_SETUP);
if (refcount_read(&nbd->config_refs)) {
mutex_unlock(&nbd->config_lock);
nbd_put(nbd);
--
2.52.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [RFC PATCH 5/5] nbd: allow genetlink operations to run in parallel
2026-11-07 6:46 [RFC PATCH 0/5] nbd: allow parallel genetlink operations Yang Erkun
` (3 preceding siblings ...)
2026-11-07 6:46 ` [RFC PATCH 4/5] nbd: hold config_lock while updating nr_hw_queues Yang Erkun
@ 2026-11-07 6:46 ` Yang Erkun
2026-11-17 4:36 ` [RFC PATCH 0/5] nbd: allow parallel genetlink operations yangerkun
5 siblings, 0 replies; 7+ messages in thread
From: Yang Erkun @ 2026-11-07 6:46 UTC (permalink / raw)
To: axboe, josef, xiubli, prasanna.kalever, ming.lei, yukuai, nilay
Cc: linux-block, nbd, yangerkun, wangkefeng.wang, leijitang, yi.zhang,
chengzhihao1, houtao1, wozizhi
Each nbd doit only touches one device's state, and the global state
(nbd_index_idr and nbd_total_devices) is accessed briefly under
nbd_index_mutex. genl_mutex nevertheless serializes the whole genetlink
doit path.
The cost that matters is teardown: nbd_disconnect_and_put() calls
flush_workqueue() while holding genl_mutex, so a disconnect blocks every
other device for the duration of the flush. Enable parallel_ops; the
preceding patches make sure no handler relies on genl_mutex for
correctness.
Test with a minimal netlink client in an 8-vCPU KVM guest, 32 devices x
10 connect/disconnect cycles, median of 3 runs:
parallel_ops=0: 378 ms
parallel_ops=1: 27 ms
Connect-only and status do not improve, since their critical sections
are short.
Suggested-by: Xiu Jianfeng <xiujianfeng@huawei.com>
Assisted-by: LLM
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
drivers/block/nbd.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index cf2a47fc572d..6a7060773e27 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -2542,6 +2542,7 @@ static struct genl_family nbd_genl_family __ro_after_init = {
.hdrsize = 0,
.name = NBD_GENL_FAMILY_NAME,
.version = NBD_GENL_VERSION,
+ .parallel_ops = 1,
.module = THIS_MODULE,
.small_ops = nbd_connect_genl_ops,
.n_small_ops = ARRAY_SIZE(nbd_connect_genl_ops),
--
2.52.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [RFC PATCH 0/5] nbd: allow parallel genetlink operations
2026-11-07 6:46 [RFC PATCH 0/5] nbd: allow parallel genetlink operations Yang Erkun
` (4 preceding siblings ...)
2026-11-07 6:46 ` [RFC PATCH 5/5] nbd: allow genetlink operations to run in parallel Yang Erkun
@ 2026-11-17 4:36 ` yangerkun
5 siblings, 0 replies; 7+ messages in thread
From: yangerkun @ 2026-11-17 4:36 UTC (permalink / raw)
To: axboe, josef, xiubli, prasanna.kalever, ming.lei, yukuai, nilay
Cc: linux-block, nbd, wangkefeng.wang, leijitang, yi.zhang,
chengzhihao1, houtao1, wozizhi, xiujianfeng, echo.chenlin
Hi all,
Any advise for this series?
Thanks,
Erkun.
在 2026/11/7 14:46, Yang Erkun 写道:
> Each nbd doit only touches the state of one device; the only global
> state is nbd_index_idr, accessed briefly under nbd_index_mutex.
> genl_mutex nevertheless serializes the whole genetlink doit path, and
> nbd_disconnect_and_put() sleeps in flush_workqueue() while holding it,
> so one device's teardown blocks every other device. This series
> enables parallel_ops for nbd.
>
> Patches 1, 2 and 4 fix bugs that are reachable without parallel_ops;
> patch 3 makes device publication and the status walk consistent for
> concurrent genl handlers. Patch 5 enables parallel_ops.
>
> A minimal netlink client on an 8-vCPU KVM guest, 32 devices x 10
> connect/disconnect cycles, median of 3 runs: 390 ms -> 25 ms.
> Connect-only and status do not change.
>
> Yang Erkun (5):
> nbd: validate the backend identifier under config_lock
> nbd: decrement nbd_total_devices when a device is removed
> nbd: publish devices atomically
> nbd: hold config_lock while updating nr_hw_queues
> nbd: allow genetlink operations to run in parallel
>
> drivers/block/nbd.c | 95 +++++++++++++++++++++++++++++++--------------
> 1 file changed, 65 insertions(+), 30 deletions(-)
>
^ permalink raw reply [flat|nested] 7+ messages in thread