[PATCH v3 0/6] nbd: eliminate queue freeze/unfreeze overhead in connection setup

[PATCH v3 0/6] nbd: eliminate queue freeze/unfreeze overhead in connection setup

[Yang Erkun]

This series eliminates the blk_mq_freeze_queue()/unfreeze_queue()
overhead that currently occurs during nbd connection setup(nbd provide
ondemand and was using in AI sandbox e2b[1] to mount and attach rootfs).
Each freeze cycle will introduce a RCU grace which was significant
with multi-core system.

time nbd-client --name myexport --connections 96 127.0.0.1 1234

After this patchset:
real    0m0.090s
user    0m0.004s
sys     0m0.018s

Without this patchset:
real    0m2.195s
user    0m0.005s
sys     0m0.022s

v2->v3:
1. Stop using xarray, and protect it by rejecting late NBD_SET_SOCKS
2. Eliminate freeze in nbd_set_size when device setup
3. Add module parameter to eliminate freeze for pre-created device

v1->v2:
1. Rewrite cover letter
2. Eliminate freeze call from blk_mq_update_nr_hw_queues in
   nbd_start_device for netlink path


[1]. https://github.com/e2b-dev/e2b

Long Li (1):
  nbd: simplify find_fallback() by removing redundant logic

Yang Erkun (5):
  nbd: disallow NBD_SET_SOCK on an active device
  nbd: remove queue freeze in nbd_add_socket
  nbd: set nr_hw_queues at device creation to skip queue freeze
  nbd: skip queue freeze when setting size at device startup
  nbd: add nr_hw_queues module parameter for pre-created devices

 drivers/block/nbd.c | 119 ++++++++++++++++++++++++++++----------------
 1 file changed, 76 insertions(+), 43 deletions(-)

-- 
2.52.0

[PATCH v3 1/6] nbd: simplify find_fallback() by removing redundant logic

[Yang Erkun]

From: Long Li <leo.lilong@huawei.com>

Remove the intermediate new_index variable and return -1 directly.
The second conditional checking nsock->fallback_index validity is the
logical inverse of the first, so drop it and let execution fall through
naturally. Consolidate the two identical dev_err_ratelimited() + return
paths into a single no_fallback label to reduce duplication.

Signed-off-by: Long Li <leo.lilong@huawei.com>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
---
 drivers/block/nbd.c | 37 ++++++++++++++-----------------------
 1 file changed, 14 insertions(+), 23 deletions(-)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 8f10762e90ef..b1a5acd57426 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -1061,40 +1061,31 @@ static int find_fallback(struct nbd_device *nbd, int index)
 	int new_index = -1;
 	struct nbd_sock *nsock = config->socks[index];
 	int fallback = nsock->fallback_index;
+	int i;
 
 	if (test_bit(NBD_RT_DISCONNECTED, &config->runtime_flags))
 		return new_index;
 
-	if (config->num_connections <= 1) {
-		dev_err_ratelimited(disk_to_dev(nbd->disk),
-				    "Dead connection, failed to find a fallback\n");
-		return new_index;
-	}
+	if (config->num_connections <= 1)
+		goto no_fallback;
 
 	if (fallback >= 0 && fallback < config->num_connections &&
 	    !config->socks[fallback]->dead)
 		return fallback;
 
-	if (nsock->fallback_index < 0 ||
-	    nsock->fallback_index >= config->num_connections ||
-	    config->socks[nsock->fallback_index]->dead) {
-		int i;
-		for (i = 0; i < config->num_connections; i++) {
-			if (i == index)
-				continue;
-			if (!config->socks[i]->dead) {
-				new_index = i;
-				break;
-			}
-		}
-		nsock->fallback_index = new_index;
-		if (new_index < 0) {
-			dev_err_ratelimited(disk_to_dev(nbd->disk),
-					    "Dead connection, failed to find a fallback\n");
-			return new_index;
+	for (i = 0; i < config->num_connections; i++) {
+		if (i != index && !config->socks[i]->dead) {
+			new_index = i;
+			break;
 		}
 	}
-	new_index = nsock->fallback_index;
+	nsock->fallback_index = new_index;
+	if (new_index >= 0)
+		return new_index;
+
+no_fallback:
+	dev_err_ratelimited(disk_to_dev(nbd->disk),
+			    "Dead connection, failed to find a fallback\n");
 	return new_index;
 }
 
-- 
2.52.0

[PATCH v3 2/6] nbd: disallow NBD_SET_SOCK on an active device

[Yang Erkun]

We cannot add a socket to an already running nbd device, the reconfigure
for netlink can only active dead socket. But for ioctl path, we can call
NBD_SET_SOCK after NBD_DO_IT, reject this using nbd->pid which has been
setted when NBD_DO_IT. Besides, it is the root cause for commit
b98e762e3d71 ("nbd: freeze the queue while we're adding connections").

Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
 drivers/block/nbd.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index b1a5acd57426..a15553ab4b97 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -1302,6 +1302,13 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
 		goto put_socket;
 	}
 
+	if (nbd->pid) {
+		dev_err(disk_to_dev(nbd->disk),
+			"Cannot add socket to a running device\n");
+		err = -EBUSY;
+		goto put_socket;
+	}
+
 	nsock = kzalloc_obj(*nsock);
 	if (!nsock) {
 		err = -ENOMEM;
-- 
2.52.0

[PATCH v3 3/6] nbd: remove queue freeze in nbd_add_socket

[Yang Erkun]

nbd_add_socket can never run concurrently with inflight I/O:

 - netlink path: nbd_genl_connect calls nbd_add_socket before
   nbd_start_device, so no I/O can happened when invoking nbd_add_socket,
   nbd_genl_reconfigure cannot too since it won't call nbd_add_socket

 - ioctl path: NBD_SET_SOCK cannot be called after NBD_DO_IT with
   the previous commit, so capability of nbd will keep 0 while
   invoking NBD_SET_SOCK

Removing the freeze in nbd_add_socket to speed up nbd device startup.

Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
 drivers/block/nbd.c | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index a15553ab4b97..0755b7046ed4 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -1272,7 +1272,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
 	struct socket *sock;
 	struct nbd_sock **socks;
 	struct nbd_sock *nsock;
-	unsigned int memflags;
 	int err;
 
 	/* Arg will be cast to int, check it to avoid overflow */
@@ -1283,12 +1282,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
 		return err;
 	nbd_reclassify_socket(sock);
 
-	/*
-	 * We need to make sure we don't get any errant requests while we're
-	 * reallocating the ->socks array.
-	 */
-	memflags = blk_mq_freeze_queue(nbd->disk->queue);
-
 	if (!netlink && !nbd->task_setup &&
 	    !test_bit(NBD_RT_BOUND, &config->runtime_flags))
 		nbd->task_setup = current;
@@ -1335,12 +1328,10 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
 	INIT_WORK(&nsock->work, nbd_pending_cmd_work);
 	socks[config->num_connections++] = nsock;
 	atomic_inc(&config->live_connections);
-	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
 
 	return 0;
 
 put_socket:
-	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
 	sockfd_put(sock);
 	return err;
 }
-- 
2.52.0

[PATCH v3 4/6] nbd: set nr_hw_queues at device creation to skip queue freeze

[Yang Erkun]

There still be queue freeze call when nbd_start_device invoking
blk_mq_update_nr_hw_queues. For netlink path, we can obtain the actual
number of connections before calling nbd_dev_add in nbd_genl_connect,
which can helps remove this queue freeze.

However, nbd devices created with a fixed nbds_max may still require
this freezing because the real connection count is unknown.

Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
 drivers/block/nbd.c | 39 +++++++++++++++++++++++++++++++++++----
 1 file changed, 35 insertions(+), 4 deletions(-)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 0755b7046ed4..400f638e832e 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -1930,7 +1930,8 @@ static const struct blk_mq_ops nbd_mq_ops = {
 	.timeout	= nbd_xmit_timeout,
 };
 
-static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
+static struct nbd_device *nbd_dev_add(int index, unsigned int refs,
+				       int nr_hw_queues)
 {
 	struct queue_limits lim = {
 		.max_hw_sectors		= 65536,
@@ -1947,7 +1948,7 @@ static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
 		goto out;
 
 	nbd->tag_set.ops = &nbd_mq_ops;
-	nbd->tag_set.nr_hw_queues = 1;
+	nbd->tag_set.nr_hw_queues = nr_hw_queues;
 	nbd->tag_set.queue_depth = 128;
 	nbd->tag_set.numa_node = NUMA_NO_NODE;
 	nbd->tag_set.cmd_size = sizeof(struct nbd_cmd);
@@ -2070,6 +2071,35 @@ static const struct nla_policy nbd_sock_policy[NBD_SOCK_MAX + 1] = {
 	[NBD_SOCK_FD]			=	{ .type = NLA_U32 },
 };
 
+/*
+ * Count the number of socket FDs in the NBD_ATTR_SOCKETS netlink attribute.
+ * This is used to determine the correct nr_hw_queues before creating the
+ * nbd device, so that blk_mq_update_nr_hw_queues (and its RCU grace period
+ * overhead) can be avoided entirely.
+ */
+static int nbd_genl_count_sockets(struct genl_info *info)
+{
+	struct nlattr *attr;
+	int rem, count = 0;
+
+	if (!info->attrs[NBD_ATTR_SOCKETS])
+		return 0;
+
+	nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS], rem) {
+		struct nlattr *socks[NBD_SOCK_MAX + 1];
+
+		if (nla_type(attr) != NBD_SOCK_ITEM)
+			continue;
+		if (nla_parse_nested_deprecated(socks, NBD_SOCK_MAX,
+						  attr, nbd_sock_policy,
+						  info->extack) != 0)
+			continue;
+		if (socks[NBD_SOCK_FD])
+			count++;
+	}
+	return count;
+}
+
 /* We don't use this right now since we don't parse the incoming list, but we
  * still want it here so userspace knows what to expect.
  */
@@ -2101,6 +2131,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
 	struct nbd_device *nbd;
 	struct nbd_config *config;
 	int index = -1;
+	int num_connections = nbd_genl_count_sockets(info);
 	int ret;
 	bool put_dev = false;
 
@@ -2148,7 +2179,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
 	mutex_unlock(&nbd_index_mutex);
 
 	if (!nbd) {
-		nbd = nbd_dev_add(index, 2);
+		nbd = nbd_dev_add(index, 2, num_connections);
 		if (IS_ERR(nbd)) {
 			pr_err("failed to add new device\n");
 			return PTR_ERR(nbd);
@@ -2715,7 +2746,7 @@ static int __init nbd_init(void)
 	nbd_dbg_init();
 
 	for (i = 0; i < nbds_max; i++)
-		nbd_dev_add(i, 1);
+		nbd_dev_add(i, 1, 1);
 	return 0;
 }
 
-- 
2.52.0

[PATCH v3 5/6] nbd: skip queue freeze when setting size at device startup

[Yang Erkun]

Commit 242a49e5c878 ("nbd: freeze the queue for queue limits updates")
introduce queue freeze/unfreeze in nbd_set_size to avoid inflight
commands see this inconsistent limits. However, this cannot be happened
when device setup since the capacity is still 0.

time nbd-client --name myexport --connections 96 127.0.0.1 1234

Before this patchset:
real    0m2.195s
user    0m0.005s
sys     0m0.022s

After this patchset:
real    0m0.090s
user    0m0.004s
sys     0m0.018s

Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
 drivers/block/nbd.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 400f638e832e..2c7b09c70da2 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -331,7 +331,8 @@ static void nbd_mark_nsock_dead(struct nbd_device *nbd, struct nbd_sock *nsock,
 	nsock->sent = 0;
 }
 
-static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
+static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize,
+			bool freeze)
 {
 	struct queue_limits lim;
 	int error;
@@ -371,7 +372,13 @@ static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
 
 	lim.logical_block_size = blksize;
 	lim.physical_block_size = blksize;
-	error = queue_limits_commit_update_frozen(nbd->disk->queue, &lim);
+
+	if (freeze)
+		error = queue_limits_commit_update_frozen(nbd->disk->queue,
+				&lim);
+	else
+		error = queue_limits_commit_update(nbd->disk->queue, &lim);
+
 	if (error)
 		return error;
 
@@ -1563,7 +1570,7 @@ static int nbd_start_device(struct nbd_device *nbd)
 		args->index = i;
 		queue_work(nbd->recv_workq, &args->work);
 	}
-	return nbd_set_size(nbd, config->bytesize, nbd_blksize(config));
+	return nbd_set_size(nbd, config->bytesize, nbd_blksize(config), false);
 }
 
 static int nbd_start_device_ioctl(struct nbd_device *nbd)
@@ -1631,13 +1638,13 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd,
 	case NBD_SET_SOCK:
 		return nbd_add_socket(nbd, arg, false);
 	case NBD_SET_BLKSIZE:
-		return nbd_set_size(nbd, config->bytesize, arg);
+		return nbd_set_size(nbd, config->bytesize, arg, true);
 	case NBD_SET_SIZE:
-		return nbd_set_size(nbd, arg, nbd_blksize(config));
+		return nbd_set_size(nbd, arg, nbd_blksize(config), true);
 	case NBD_SET_SIZE_BLOCKS:
 		if (check_shl_overflow(arg, config->blksize_bits, &bytesize))
 			return -EINVAL;
-		return nbd_set_size(nbd, bytesize, nbd_blksize(config));
+		return nbd_set_size(nbd, bytesize, nbd_blksize(config), true);
 	case NBD_SET_TIMEOUT:
 		nbd_set_cmd_timeout(nbd, arg);
 		return 0;
@@ -2122,7 +2129,7 @@ static int nbd_genl_size_set(struct genl_info *info, struct nbd_device *nbd)
 		bsize = nla_get_u64(info->attrs[NBD_ATTR_BLOCK_SIZE_BYTES]);
 
 	if (bytes != config->bytesize || bsize != nbd_blksize(config))
-		return nbd_set_size(nbd, bytes, bsize);
+		return nbd_set_size(nbd, bytes, bsize, true);
 	return 0;
 }
 
-- 
2.52.0

[PATCH v3 6/6] nbd: add nr_hw_queues module parameter for pre-created devices

[Yang Erkun]

Previous commit can help remove all freeze for netlink newly created nbd
device. But for the devices pre-created at module load(nbds_max default
as 16), the nr_hw_queues was setting default as 1, then ioctl/netlink
path will set the real connection count, and blk_mq_update_nr_hw_queues
in nbd_start_device will introduce freeze.

Add an nr_hw_queues module parameter so that users who know their
expected connection count can pre-created devices with the right queue
count to avoid this freeze too.

Signed-off-by: Yang Erkun <yangerkun@huawei.com>
---
 drivers/block/nbd.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 2c7b09c70da2..f918c9efa9b2 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -166,6 +166,7 @@ static struct dentry *nbd_dbg_dir;
 
 static unsigned int nbds_max = 16;
 static int max_part = 16;
+static int nr_hw_queues = 1;
 static int part_shift;
 
 static int nbd_dev_dbg_init(struct nbd_device *nbd);
@@ -2752,8 +2753,10 @@ static int __init nbd_init(void)
 	}
 	nbd_dbg_init();
 
+	if (nr_hw_queues < 1)
+		nr_hw_queues = 1;
 	for (i = 0; i < nbds_max; i++)
-		nbd_dev_add(i, 1, 1);
+		nbd_dev_add(i, 1, nr_hw_queues);
 	return 0;
 }
 
@@ -2814,3 +2817,6 @@ module_param(nbds_max, int, 0444);
 MODULE_PARM_DESC(nbds_max, "number of network block devices to initialize (default: 16)");
 module_param(max_part, int, 0444);
 MODULE_PARM_DESC(max_part, "number of partitions per device (default: 16)");
+module_param(nr_hw_queues, int, 0444);
+MODULE_PARM_DESC(nr_hw_queues,
+"number of hardware queues for devices pre-created at module load (default: 1). ");
-- 
2.52.0

Re: [PATCH v3 0/6] nbd: eliminate queue freeze/unfreeze overhead in connection setup

[yangerkun]

gently ping for this series...

在 2026/7/13 14:56, Yang Erkun 写道:
> This series eliminates the blk_mq_freeze_queue()/unfreeze_queue()
> overhead that currently occurs during nbd connection setup(nbd provide
> ondemand and was using in AI sandbox e2b[1] to mount and attach rootfs).
> Each freeze cycle will introduce a RCU grace which was significant
> with multi-core system.
> 
> time nbd-client --name myexport --connections 96 127.0.0.1 1234
> 
> After this patchset:
> real    0m0.090s
> user    0m0.004s
> sys     0m0.018s
> 
> Without this patchset:
> real    0m2.195s
> user    0m0.005s
> sys     0m0.022s
> 
> v2->v3:
> 1. Stop using xarray, and protect it by rejecting late NBD_SET_SOCKS
> 2. Eliminate freeze in nbd_set_size when device setup
> 3. Add module parameter to eliminate freeze for pre-created device
> 
> v1->v2:
> 1. Rewrite cover letter
> 2. Eliminate freeze call from blk_mq_update_nr_hw_queues in
>     nbd_start_device for netlink path
> 
> 
> [1]. https://github.com/e2b-dev/e2b
> 
> Long Li (1):
>    nbd: simplify find_fallback() by removing redundant logic
> 
> Yang Erkun (5):
>    nbd: disallow NBD_SET_SOCK on an active device
>    nbd: remove queue freeze in nbd_add_socket
>    nbd: set nr_hw_queues at device creation to skip queue freeze
>    nbd: skip queue freeze when setting size at device startup
>    nbd: add nr_hw_queues module parameter for pre-created devices
> 
>   drivers/block/nbd.c | 119 ++++++++++++++++++++++++++++----------------
>   1 file changed, 76 insertions(+), 43 deletions(-)
> 

Re: [PATCH v3 2/6] nbd: disallow NBD_SET_SOCK on an active device

[yu kuai]

Hi,

在 2026/7/13 14:56, Yang Erkun 写道:
> We cannot add a socket to an already running nbd device, the reconfigure
> for netlink can only active dead socket. But for ioctl path, we can call
> NBD_SET_SOCK after NBD_DO_IT, reject this using nbd->pid which has been
> setted when NBD_DO_IT. Besides, it is the root cause for commit
> b98e762e3d71 ("nbd: freeze the queue while we're adding connections").
>
> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
> ---
>   drivers/block/nbd.c | 7 +++++++
>   1 file changed, 7 insertions(+)
>
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index b1a5acd57426..a15553ab4b97 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -1302,6 +1302,13 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>   		goto put_socket;
>   	}
>   
> +	if (nbd->pid) {
> +		dev_err(disk_to_dev(nbd->disk),
> +			"Cannot add socket to a running device\n");
> +		err = -EBUSY;
> +		goto put_socket;
> +	}

I think the checking if fine, but I'd like to add the checking before nbd_get_socket() and
return -EBUSY directly.

> +
>   	nsock = kzalloc_obj(*nsock);
>   	if (!nsock) {
>   		err = -ENOMEM;

-- 
Thanks,
Kuai

Re: [PATCH v3 3/6] nbd: remove queue freeze in nbd_add_socket

[yu kuai]

Hi,

在 2026/7/13 14:56, Yang Erkun 写道:
> nbd_add_socket can never run concurrently with inflight I/O:
>
>   - netlink path: nbd_genl_connect calls nbd_add_socket before
>     nbd_start_device, so no I/O can happened when invoking nbd_add_socket,
>     nbd_genl_reconfigure cannot too since it won't call nbd_add_socket
>
>   - ioctl path: NBD_SET_SOCK cannot be called after NBD_DO_IT with
>     the previous commit, so capability of nbd will keep 0 while
>     invoking NBD_SET_SOCK
>
> Removing the freeze in nbd_add_socket to speed up nbd device startup.

Check nbd_open(), the pid is not checked there, which means the nbd device
can be opened after add_disk() succeed. Normally read/write IO will not pass
bio_check_eod() checking, however, there can be special zero sized bio that
might still be issued to nbd device. So I think there should be a pid checking
in nbd_open() for this patch, and I'm not aware of any special IO mush be handled
before nbd_start_device(). Otherwise, the freeze is still necessary since
nbd_handle_cmd() will deference config->socks[] directly.

>
> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
> ---
>   drivers/block/nbd.c | 9 ---------
>   1 file changed, 9 deletions(-)
>
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index a15553ab4b97..0755b7046ed4 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -1272,7 +1272,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>   	struct socket *sock;
>   	struct nbd_sock **socks;
>   	struct nbd_sock *nsock;
> -	unsigned int memflags;
>   	int err;
>   
>   	/* Arg will be cast to int, check it to avoid overflow */
> @@ -1283,12 +1282,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>   		return err;
>   	nbd_reclassify_socket(sock);
>   
> -	/*
> -	 * We need to make sure we don't get any errant requests while we're
> -	 * reallocating the ->socks array.
> -	 */
> -	memflags = blk_mq_freeze_queue(nbd->disk->queue);
> -
>   	if (!netlink && !nbd->task_setup &&
>   	    !test_bit(NBD_RT_BOUND, &config->runtime_flags))
>   		nbd->task_setup = current;
> @@ -1335,12 +1328,10 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>   	INIT_WORK(&nsock->work, nbd_pending_cmd_work);
>   	socks[config->num_connections++] = nsock;
>   	atomic_inc(&config->live_connections);
> -	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>   
>   	return 0;
>   
>   put_socket:
> -	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>   	sockfd_put(sock);
>   	return err;
>   }

-- 
Thanks,
Kuai

Re: [PATCH v3 4/6] nbd: set nr_hw_queues at device creation to skip queue freeze

[yu kuai]

Hi,

在 2026/7/13 14:56, Yang Erkun 写道:
> There still be queue freeze call when nbd_start_device invoking
> blk_mq_update_nr_hw_queues. For netlink path, we can obtain the actual
> number of connections before calling nbd_dev_add in nbd_genl_connect,
> which can helps remove this queue freeze.
>
> However, nbd devices created with a fixed nbds_max may still require
> this freezing because the real connection count is unknown.
>
> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
> ---
>   drivers/block/nbd.c | 39 +++++++++++++++++++++++++++++++++++----
>   1 file changed, 35 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index 0755b7046ed4..400f638e832e 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -1930,7 +1930,8 @@ static const struct blk_mq_ops nbd_mq_ops = {
>   	.timeout	= nbd_xmit_timeout,
>   };
>   
> -static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
> +static struct nbd_device *nbd_dev_add(int index, unsigned int refs,
> +				       int nr_hw_queues)
>   {
>   	struct queue_limits lim = {
>   		.max_hw_sectors		= 65536,
> @@ -1947,7 +1948,7 @@ static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
>   		goto out;
>   
>   	nbd->tag_set.ops = &nbd_mq_ops;
> -	nbd->tag_set.nr_hw_queues = 1;
> +	nbd->tag_set.nr_hw_queues = nr_hw_queues;
>   	nbd->tag_set.queue_depth = 128;
>   	nbd->tag_set.numa_node = NUMA_NO_NODE;
>   	nbd->tag_set.cmd_size = sizeof(struct nbd_cmd);
> @@ -2070,6 +2071,35 @@ static const struct nla_policy nbd_sock_policy[NBD_SOCK_MAX + 1] = {
>   	[NBD_SOCK_FD]			=	{ .type = NLA_U32 },
>   };
>   
> +/*
> + * Count the number of socket FDs in the NBD_ATTR_SOCKETS netlink attribute.
> + * This is used to determine the correct nr_hw_queues before creating the
> + * nbd device, so that blk_mq_update_nr_hw_queues (and its RCU grace period
> + * overhead) can be avoided entirely.
> + */
> +static int nbd_genl_count_sockets(struct genl_info *info)
> +{
> +	struct nlattr *attr;
> +	int rem, count = 0;
> +
> +	if (!info->attrs[NBD_ATTR_SOCKETS])
> +		return 0;
> +
> +	nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS], rem) {
> +		struct nlattr *socks[NBD_SOCK_MAX + 1];
> +
> +		if (nla_type(attr) != NBD_SOCK_ITEM)
> +			continue;

nbd_genl_connect() will fail in this case.

> +		if (nla_parse_nested_deprecated(socks, NBD_SOCK_MAX,
> +						  attr, nbd_sock_policy,
> +						  info->extack) != 0)
> +			continue;
> +		if (socks[NBD_SOCK_FD])
> +			count++;
> +	}
> +	return count;
> +}

Personally I don't like to copy code from existed NBD_ATTR_SOCKETS handling. Do
you consider moving NBD_ATTR_SOCKETS handling forward or factor out a common helper?

> +
>   /* We don't use this right now since we don't parse the incoming list, but we
>    * still want it here so userspace knows what to expect.
>    */
> @@ -2101,6 +2131,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
>   	struct nbd_device *nbd;
>   	struct nbd_config *config;
>   	int index = -1;
> +	int num_connections = nbd_genl_count_sockets(info);
>   	int ret;
>   	bool put_dev = false;
>   
> @@ -2148,7 +2179,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
>   	mutex_unlock(&nbd_index_mutex);
>   
>   	if (!nbd) {
> -		nbd = nbd_dev_add(index, 2);
> +		nbd = nbd_dev_add(index, 2, num_connections);
>   		if (IS_ERR(nbd)) {
>   			pr_err("failed to add new device\n");
>   			return PTR_ERR(nbd);
> @@ -2715,7 +2746,7 @@ static int __init nbd_init(void)
>   	nbd_dbg_init();
>   
>   	for (i = 0; i < nbds_max; i++)
> -		nbd_dev_add(i, 1);
> +		nbd_dev_add(i, 1, 1);
>   	return 0;
>   }

This approach will only be useful when nbd device is created the first time.
If NBD_ATTR_INDEX is set and nbd device is found, or NBD_ATTR_IDNEX is not set
and nbd_find_get_unused() found an unused nbd device, blk_mq_update_nr_hw_queues()
is still required. As you can see, nbds_max is default to 16 and all these devices
will be created with nr_hw_queues as 1.

>   

-- 
Thanks,
Kuai

Re: [PATCH v3 5/6] nbd: skip queue freeze when setting size at device startup

[yu kuai]

Hi,

在 2026/7/13 14:56, Yang Erkun 写道:
> Commit 242a49e5c878 ("nbd: freeze the queue for queue limits updates")
> introduce queue freeze/unfreeze in nbd_set_size to avoid inflight
> commands see this inconsistent limits. However, this cannot be happened
> when device setup since the capacity is still 0.

Same reason above, 0 capacity disk will only reject non-zero sized bio from
bio_check_eod().

>
> time nbd-client --name myexport --connections 96 127.0.0.1 1234
>
> Before this patchset:
> real    0m2.195s
> user    0m0.005s
> sys     0m0.022s
>
> After this patchset:
> real    0m0.090s
> user    0m0.004s
> sys     0m0.018s
>
> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
> ---
>   drivers/block/nbd.c | 21 ++++++++++++++-------
>   1 file changed, 14 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index 400f638e832e..2c7b09c70da2 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -331,7 +331,8 @@ static void nbd_mark_nsock_dead(struct nbd_device *nbd, struct nbd_sock *nsock,
>   	nsock->sent = 0;
>   }
>   
> -static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
> +static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize,
> +			bool freeze)
>   {
>   	struct queue_limits lim;
>   	int error;
> @@ -371,7 +372,13 @@ static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
>   
>   	lim.logical_block_size = blksize;
>   	lim.physical_block_size = blksize;
> -	error = queue_limits_commit_update_frozen(nbd->disk->queue, &lim);
> +
> +	if (freeze)
> +		error = queue_limits_commit_update_frozen(nbd->disk->queue,
> +				&lim);
> +	else
> +		error = queue_limits_commit_update(nbd->disk->queue, &lim);
> +
>   	if (error)
>   		return error;
>   
> @@ -1563,7 +1570,7 @@ static int nbd_start_device(struct nbd_device *nbd)
>   		args->index = i;
>   		queue_work(nbd->recv_workq, &args->work);
>   	}
> -	return nbd_set_size(nbd, config->bytesize, nbd_blksize(config));
> +	return nbd_set_size(nbd, config->bytesize, nbd_blksize(config), false);
>   }
>   
>   static int nbd_start_device_ioctl(struct nbd_device *nbd)
> @@ -1631,13 +1638,13 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd,
>   	case NBD_SET_SOCK:
>   		return nbd_add_socket(nbd, arg, false);
>   	case NBD_SET_BLKSIZE:
> -		return nbd_set_size(nbd, config->bytesize, arg);
> +		return nbd_set_size(nbd, config->bytesize, arg, true);
>   	case NBD_SET_SIZE:
> -		return nbd_set_size(nbd, arg, nbd_blksize(config));
> +		return nbd_set_size(nbd, arg, nbd_blksize(config), true);
>   	case NBD_SET_SIZE_BLOCKS:
>   		if (check_shl_overflow(arg, config->blksize_bits, &bytesize))
>   			return -EINVAL;
> -		return nbd_set_size(nbd, bytesize, nbd_blksize(config));
> +		return nbd_set_size(nbd, bytesize, nbd_blksize(config), true);
>   	case NBD_SET_TIMEOUT:
>   		nbd_set_cmd_timeout(nbd, arg);
>   		return 0;
> @@ -2122,7 +2129,7 @@ static int nbd_genl_size_set(struct genl_info *info, struct nbd_device *nbd)
>   		bsize = nla_get_u64(info->attrs[NBD_ATTR_BLOCK_SIZE_BYTES]);
>   
>   	if (bytes != config->bytesize || bsize != nbd_blksize(config))
> -		return nbd_set_size(nbd, bytes, bsize);
> +		return nbd_set_size(nbd, bytes, bsize, true);

This is still called from nbd_genl_connect().

Will it be much simpler to just check pid in nbd_set_size()?

>   	return 0;
>   }
>   

-- 
Thanks,
Kuai

Re: [PATCH v3 6/6] nbd: add nr_hw_queues module parameter for pre-created devices

[yu kuai]

Hi,

在 2026/7/13 14:56, Yang Erkun 写道:
> Previous commit can help remove all freeze for netlink newly created nbd
> device. But for the devices pre-created at module load(nbds_max default
> as 16), the nr_hw_queues was setting default as 1, then ioctl/netlink
> path will set the real connection count, and blk_mq_update_nr_hw_queues
> in nbd_start_device will introduce freeze.
>
> Add an nr_hw_queues module parameter so that users who know their
> expected connection count can pre-created devices with the right queue
> count to avoid this freeze too.

I'm fine with this, but this is misleading. Please also add comment to
emphasize this module parameter is just to avoid queue freeze if user configure
the exact number of socks as this new module parameter. Otherwise, the real
nr_hw_queues is still depend on the real number of user configured socks.

>
> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
> ---
>   drivers/block/nbd.c | 8 +++++++-
>   1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index 2c7b09c70da2..f918c9efa9b2 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -166,6 +166,7 @@ static struct dentry *nbd_dbg_dir;
>   
>   static unsigned int nbds_max = 16;
>   static int max_part = 16;
> +static int nr_hw_queues = 1;
>   static int part_shift;
>   
>   static int nbd_dev_dbg_init(struct nbd_device *nbd);
> @@ -2752,8 +2753,10 @@ static int __init nbd_init(void)
>   	}
>   	nbd_dbg_init();
>   
> +	if (nr_hw_queues < 1)
> +		nr_hw_queues = 1;
>   	for (i = 0; i < nbds_max; i++)
> -		nbd_dev_add(i, 1, 1);
> +		nbd_dev_add(i, 1, nr_hw_queues);
>   	return 0;
>   }
>   
> @@ -2814,3 +2817,6 @@ module_param(nbds_max, int, 0444);
>   MODULE_PARM_DESC(nbds_max, "number of network block devices to initialize (default: 16)");
>   module_param(max_part, int, 0444);
>   MODULE_PARM_DESC(max_part, "number of partitions per device (default: 16)");
> +module_param(nr_hw_queues, int, 0444);
> +MODULE_PARM_DESC(nr_hw_queues,
> +"number of hardware queues for devices pre-created at module load (default: 1). ");

-- 
Thanks,
Kuai

Re: [PATCH v3 2/6] nbd: disallow NBD_SET_SOCK on an active device

[yangerkun]

在 2026/7/22 11:03, yu kuai 写道:
> Hi,
> 
> 在 2026/7/13 14:56, Yang Erkun 写道:
>> We cannot add a socket to an already running nbd device, the reconfigure
>> for netlink can only active dead socket. But for ioctl path, we can call
>> NBD_SET_SOCK after NBD_DO_IT, reject this using nbd->pid which has been
>> setted when NBD_DO_IT. Besides, it is the root cause for commit
>> b98e762e3d71 ("nbd: freeze the queue while we're adding connections").
>>
>> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
>> ---
>>    drivers/block/nbd.c | 7 +++++++
>>    1 file changed, 7 insertions(+)
>>
>> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
>> index b1a5acd57426..a15553ab4b97 100644
>> --- a/drivers/block/nbd.c
>> +++ b/drivers/block/nbd.c
>> @@ -1302,6 +1302,13 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>>    		goto put_socket;
>>    	}
>>    
>> +	if (nbd->pid) {
>> +		dev_err(disk_to_dev(nbd->disk),
>> +			"Cannot add socket to a running device\n");
>> +		err = -EBUSY;
>> +		goto put_socket;
>> +	}
> 
> I think the checking if fine, but I'd like to add the checking before nbd_get_socket() and
> return -EBUSY directly.

OK.
> 
>> +
>>    	nsock = kzalloc_obj(*nsock);
>>    	if (!nsock) {
>>    		err = -ENOMEM;
> 

Re: [PATCH v3 3/6] nbd: remove queue freeze in nbd_add_socket

[yu kuai]

Hi,

在 2026/7/22 11:17, yu kuai 写道:
> Hi,
>
> 在 2026/7/13 14:56, Yang Erkun 写道:
>> nbd_add_socket can never run concurrently with inflight I/O:
>>
>>    - netlink path: nbd_genl_connect calls nbd_add_socket before
>>      nbd_start_device, so no I/O can happened when invoking nbd_add_socket,
>>      nbd_genl_reconfigure cannot too since it won't call nbd_add_socket
>>
>>    - ioctl path: NBD_SET_SOCK cannot be called after NBD_DO_IT with
>>      the previous commit, so capability of nbd will keep 0 while
>>      invoking NBD_SET_SOCK
>>
>> Removing the freeze in nbd_add_socket to speed up nbd device startup.
> Check nbd_open(), the pid is not checked there, which means the nbd device
> can be opened after add_disk() succeed. Normally read/write IO will not pass
> bio_check_eod() checking, however, there can be special zero sized bio that
> might still be issued to nbd device. So I think there should be a pid checking
> in nbd_open() for this patch, and I'm not aware of any special IO mush be handled
> before nbd_start_device(). Otherwise, the freeze is still necessary since
> nbd_handle_cmd() will deference config->socks[] directly.

Sorry this is a mistake, nbd_open() must succeed before pid is set, because of ioctl
like NBD_SET_SOCK and NED_DO_IT :( I just forgot this simple case.

>
>> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
>> ---
>>    drivers/block/nbd.c | 9 ---------
>>    1 file changed, 9 deletions(-)
>>
>> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
>> index a15553ab4b97..0755b7046ed4 100644
>> --- a/drivers/block/nbd.c
>> +++ b/drivers/block/nbd.c
>> @@ -1272,7 +1272,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>>    	struct socket *sock;
>>    	struct nbd_sock **socks;
>>    	struct nbd_sock *nsock;
>> -	unsigned int memflags;
>>    	int err;
>>    
>>    	/* Arg will be cast to int, check it to avoid overflow */
>> @@ -1283,12 +1282,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>>    		return err;
>>    	nbd_reclassify_socket(sock);
>>    
>> -	/*
>> -	 * We need to make sure we don't get any errant requests while we're
>> -	 * reallocating the ->socks array.
>> -	 */
>> -	memflags = blk_mq_freeze_queue(nbd->disk->queue);
>> -
>>    	if (!netlink && !nbd->task_setup &&
>>    	    !test_bit(NBD_RT_BOUND, &config->runtime_flags))
>>    		nbd->task_setup = current;
>> @@ -1335,12 +1328,10 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>>    	INIT_WORK(&nsock->work, nbd_pending_cmd_work);
>>    	socks[config->num_connections++] = nsock;
>>    	atomic_inc(&config->live_connections);
>> -	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>>    
>>    	return 0;
>>    
>>    put_socket:
>> -	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>>    	sockfd_put(sock);
>>    	return err;
>>    }

-- 
Thanks,
Kuai

Re: [PATCH v3 3/6] nbd: remove queue freeze in nbd_add_socket

[yangerkun]

在 2026/7/22 14:35, yu kuai 写道:
> Hi,
> 
> 在 2026/7/22 11:17, yu kuai 写道:
>> Hi,
>>
>> 在 2026/7/13 14:56, Yang Erkun 写道:
>>> nbd_add_socket can never run concurrently with inflight I/O:
>>>
>>>     - netlink path: nbd_genl_connect calls nbd_add_socket before
>>>       nbd_start_device, so no I/O can happened when invoking nbd_add_socket,
>>>       nbd_genl_reconfigure cannot too since it won't call nbd_add_socket
>>>
>>>     - ioctl path: NBD_SET_SOCK cannot be called after NBD_DO_IT with
>>>       the previous commit, so capability of nbd will keep 0 while
>>>       invoking NBD_SET_SOCK
>>>
>>> Removing the freeze in nbd_add_socket to speed up nbd device startup.
>> Check nbd_open(), the pid is not checked there, which means the nbd device
>> can be opened after add_disk() succeed. Normally read/write IO will not pass
>> bio_check_eod() checking, however, there can be special zero sized bio that
>> might still be issued to nbd device. So I think there should be a pid checking
>> in nbd_open() for this patch, and I'm not aware of any special IO mush be handled
>> before nbd_start_device(). Otherwise, the freeze is still necessary since
>> nbd_handle_cmd() will deference config->socks[] directly.
> 
> Sorry this is a mistake, nbd_open() must succeed before pid is set, because of ioctl
> like NBD_SET_SOCK and NED_DO_IT :( I just forgot this simple case.

nbd can only accept request from bio, so bio_check_eod will reject them all?

> 
>>
>>> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
>>> ---
>>>     drivers/block/nbd.c | 9 ---------
>>>     1 file changed, 9 deletions(-)
>>>
>>> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
>>> index a15553ab4b97..0755b7046ed4 100644
>>> --- a/drivers/block/nbd.c
>>> +++ b/drivers/block/nbd.c
>>> @@ -1272,7 +1272,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>>>     	struct socket *sock;
>>>     	struct nbd_sock **socks;
>>>     	struct nbd_sock *nsock;
>>> -	unsigned int memflags;
>>>     	int err;
>>>     
>>>     	/* Arg will be cast to int, check it to avoid overflow */
>>> @@ -1283,12 +1282,6 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>>>     		return err;
>>>     	nbd_reclassify_socket(sock);
>>>     
>>> -	/*
>>> -	 * We need to make sure we don't get any errant requests while we're
>>> -	 * reallocating the ->socks array.
>>> -	 */
>>> -	memflags = blk_mq_freeze_queue(nbd->disk->queue);
>>> -
>>>     	if (!netlink && !nbd->task_setup &&
>>>     	    !test_bit(NBD_RT_BOUND, &config->runtime_flags))
>>>     		nbd->task_setup = current;
>>> @@ -1335,12 +1328,10 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
>>>     	INIT_WORK(&nsock->work, nbd_pending_cmd_work);
>>>     	socks[config->num_connections++] = nsock;
>>>     	atomic_inc(&config->live_connections);
>>> -	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>>>     
>>>     	return 0;
>>>     
>>>     put_socket:
>>> -	blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>>>     	sockfd_put(sock);
>>>     	return err;
>>>     }
> 

Re: [PATCH v3 3/6] nbd: remove queue freeze in nbd_add_socket

[yangerkun]

在 2026/7/22 16:45, yangerkun 写道:
> 
> 
> 在 2026/7/22 14:35, yu kuai 写道:
>> Hi,
>>
>> 在 2026/7/22 11:17, yu kuai 写道:
>>> Hi,
>>>
>>> 在 2026/7/13 14:56, Yang Erkun 写道:
>>>> nbd_add_socket can never run concurrently with inflight I/O:
>>>>
>>>>     - netlink path: nbd_genl_connect calls nbd_add_socket before
>>>>       nbd_start_device, so no I/O can happened when invoking 
>>>> nbd_add_socket,
>>>>       nbd_genl_reconfigure cannot too since it won't call 
>>>> nbd_add_socket
>>>>
>>>>     - ioctl path: NBD_SET_SOCK cannot be called after NBD_DO_IT with
>>>>       the previous commit, so capability of nbd will keep 0 while
>>>>       invoking NBD_SET_SOCK
>>>>
>>>> Removing the freeze in nbd_add_socket to speed up nbd device startup.
>>> Check nbd_open(), the pid is not checked there, which means the nbd 
>>> device
>>> can be opened after add_disk() succeed. Normally read/write IO will 
>>> not pass
>>> bio_check_eod() checking, however, there can be special zero sized 
>>> bio that
>>> might still be issued to nbd device. So I think there should be a pid 
>>> checking
>>> in nbd_open() for this patch, and I'm not aware of any special IO 
>>> mush be handled
>>> before nbd_start_device(). Otherwise, the freeze is still necessary 
>>> since
>>> nbd_handle_cmd() will deference config->socks[] directly.
>>
>> Sorry this is a mistake, nbd_open() must succeed before pid is set, 
>> because of ioctl
>> like NBD_SET_SOCK and NED_DO_IT :( I just forgot this simple case.
> 
> nbd can only accept request from bio, so bio_check_eod will reject them 
> all?

Sorry, for read/write bio, bio_check_eod will reject them, for flush 
only bio, if (!bdev_write_cache(bdev)) in submit_bio_noacct will reject 
it. So it seems safe here.

> 
>>
>>>
>>>> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
>>>> ---
>>>>     drivers/block/nbd.c | 9 ---------
>>>>     1 file changed, 9 deletions(-)
>>>>
>>>> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
>>>> index a15553ab4b97..0755b7046ed4 100644
>>>> --- a/drivers/block/nbd.c
>>>> +++ b/drivers/block/nbd.c
>>>> @@ -1272,7 +1272,6 @@ static int nbd_add_socket(struct nbd_device 
>>>> *nbd, unsigned long arg,
>>>>         struct socket *sock;
>>>>         struct nbd_sock **socks;
>>>>         struct nbd_sock *nsock;
>>>> -    unsigned int memflags;
>>>>         int err;
>>>>         /* Arg will be cast to int, check it to avoid overflow */
>>>> @@ -1283,12 +1282,6 @@ static int nbd_add_socket(struct nbd_device 
>>>> *nbd, unsigned long arg,
>>>>             return err;
>>>>         nbd_reclassify_socket(sock);
>>>> -    /*
>>>> -     * We need to make sure we don't get any errant requests while 
>>>> we're
>>>> -     * reallocating the ->socks array.
>>>> -     */
>>>> -    memflags = blk_mq_freeze_queue(nbd->disk->queue);
>>>> -
>>>>         if (!netlink && !nbd->task_setup &&
>>>>             !test_bit(NBD_RT_BOUND, &config->runtime_flags))
>>>>             nbd->task_setup = current;
>>>> @@ -1335,12 +1328,10 @@ static int nbd_add_socket(struct nbd_device 
>>>> *nbd, unsigned long arg,
>>>>         INIT_WORK(&nsock->work, nbd_pending_cmd_work);
>>>>         socks[config->num_connections++] = nsock;
>>>>         atomic_inc(&config->live_connections);
>>>> -    blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>>>>         return 0;
>>>>     put_socket:
>>>> -    blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
>>>>         sockfd_put(sock);
>>>>         return err;
>>>>     }
>>
> 
> 

Re: [PATCH v3 4/6] nbd: set nr_hw_queues at device creation to skip queue freeze

[yangerkun]

Hi,

在 2026/7/22 11:38, yu kuai 写道:
> Hi,
> 
> 在 2026/7/13 14:56, Yang Erkun 写道:
>> There still be queue freeze call when nbd_start_device invoking
>> blk_mq_update_nr_hw_queues. For netlink path, we can obtain the actual
>> number of connections before calling nbd_dev_add in nbd_genl_connect,
>> which can helps remove this queue freeze.
>>
>> However, nbd devices created with a fixed nbds_max may still require
>> this freezing because the real connection count is unknown.
>>
>> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
>> ---
>>    drivers/block/nbd.c | 39 +++++++++++++++++++++++++++++++++++----
>>    1 file changed, 35 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
>> index 0755b7046ed4..400f638e832e 100644
>> --- a/drivers/block/nbd.c
>> +++ b/drivers/block/nbd.c
>> @@ -1930,7 +1930,8 @@ static const struct blk_mq_ops nbd_mq_ops = {
>>    	.timeout	= nbd_xmit_timeout,
>>    };
>>    
>> -static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
>> +static struct nbd_device *nbd_dev_add(int index, unsigned int refs,
>> +				       int nr_hw_queues)
>>    {
>>    	struct queue_limits lim = {
>>    		.max_hw_sectors		= 65536,
>> @@ -1947,7 +1948,7 @@ static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
>>    		goto out;
>>    
>>    	nbd->tag_set.ops = &nbd_mq_ops;
>> -	nbd->tag_set.nr_hw_queues = 1;
>> +	nbd->tag_set.nr_hw_queues = nr_hw_queues;
>>    	nbd->tag_set.queue_depth = 128;
>>    	nbd->tag_set.numa_node = NUMA_NO_NODE;
>>    	nbd->tag_set.cmd_size = sizeof(struct nbd_cmd);
>> @@ -2070,6 +2071,35 @@ static const struct nla_policy nbd_sock_policy[NBD_SOCK_MAX + 1] = {
>>    	[NBD_SOCK_FD]			=	{ .type = NLA_U32 },
>>    };
>>    
>> +/*
>> + * Count the number of socket FDs in the NBD_ATTR_SOCKETS netlink attribute.
>> + * This is used to determine the correct nr_hw_queues before creating the
>> + * nbd device, so that blk_mq_update_nr_hw_queues (and its RCU grace period
>> + * overhead) can be avoided entirely.
>> + */
>> +static int nbd_genl_count_sockets(struct genl_info *info)
>> +{
>> +	struct nlattr *attr;
>> +	int rem, count = 0;
>> +
>> +	if (!info->attrs[NBD_ATTR_SOCKETS])
>> +		return 0;
>> +
>> +	nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS], rem) {
>> +		struct nlattr *socks[NBD_SOCK_MAX + 1];
>> +
>> +		if (nla_type(attr) != NBD_SOCK_ITEM)
>> +			continue;
> 
> nbd_genl_connect() will fail in this case.
> 
>> +		if (nla_parse_nested_deprecated(socks, NBD_SOCK_MAX,
>> +						  attr, nbd_sock_policy,
>> +						  info->extack) != 0)
>> +			continue;
>> +		if (socks[NBD_SOCK_FD])
>> +			count++;
>> +	}
>> +	return count;
>> +}
> 
> Personally I don't like to copy code from existed NBD_ATTR_SOCKETS handling. Do
> you consider moving NBD_ATTR_SOCKETS handling forward or factor out a common helper?

OK, I will try it.

> 
>> +
>>    /* We don't use this right now since we don't parse the incoming list, but we
>>     * still want it here so userspace knows what to expect.
>>     */
>> @@ -2101,6 +2131,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
>>    	struct nbd_device *nbd;
>>    	struct nbd_config *config;
>>    	int index = -1;
>> +	int num_connections = nbd_genl_count_sockets(info);
>>    	int ret;
>>    	bool put_dev = false;
>>    
>> @@ -2148,7 +2179,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
>>    	mutex_unlock(&nbd_index_mutex);
>>    
>>    	if (!nbd) {
>> -		nbd = nbd_dev_add(index, 2);
>> +		nbd = nbd_dev_add(index, 2, num_connections);
>>    		if (IS_ERR(nbd)) {
>>    			pr_err("failed to add new device\n");
>>    			return PTR_ERR(nbd);
>> @@ -2715,7 +2746,7 @@ static int __init nbd_init(void)
>>    	nbd_dbg_init();
>>    
>>    	for (i = 0; i < nbds_max; i++)
>> -		nbd_dev_add(i, 1);
>> +		nbd_dev_add(i, 1, 1);
>>    	return 0;
>>    }
> 
> This approach will only be useful when nbd device is created the first time.
> If NBD_ATTR_INDEX is set and nbd device is found, or NBD_ATTR_IDNEX is not set
> and nbd_find_get_unused() found an unused nbd device, blk_mq_update_nr_hw_queues()
> is still required. As you can see, nbds_max is default to 16 and all these devices
> will be created with nr_hw_queues as 1.

Yes, this patch indeed only addresses the freeze/unfreeze issue when 
creating a nbd device for the first time through netlink. The later 
patch addresses the nbds_max case. I will update the commit message 
later to make it more accurate.

> 
>>    
> 

Re: [PATCH v3 5/6] nbd: skip queue freeze when setting size at device startup

[yangerkun]

在 2026/7/22 11:49, yu kuai 写道:
> Hi,
> 
> 在 2026/7/13 14:56, Yang Erkun 写道:
>> Commit 242a49e5c878 ("nbd: freeze the queue for queue limits updates")
>> introduce queue freeze/unfreeze in nbd_set_size to avoid inflight
>> commands see this inconsistent limits. However, this cannot be happened
>> when device setup since the capacity is still 0.
> 
> Same reason above, 0 capacity disk will only reject non-zero sized bio from
> bio_check_eod().

Mabye we have reached an agreement on this issue? Flush bio will be 
reject since not enable write cache.

> 
>>
>> time nbd-client --name myexport --connections 96 127.0.0.1 1234
>>
>> Before this patchset:
>> real    0m2.195s
>> user    0m0.005s
>> sys     0m0.022s
>>
>> After this patchset:
>> real    0m0.090s
>> user    0m0.004s
>> sys     0m0.018s
>>
>> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
>> ---
>>    drivers/block/nbd.c | 21 ++++++++++++++-------
>>    1 file changed, 14 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
>> index 400f638e832e..2c7b09c70da2 100644
>> --- a/drivers/block/nbd.c
>> +++ b/drivers/block/nbd.c
>> @@ -331,7 +331,8 @@ static void nbd_mark_nsock_dead(struct nbd_device *nbd, struct nbd_sock *nsock,
>>    	nsock->sent = 0;
>>    }
>>    
>> -static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
>> +static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize,
>> +			bool freeze)
>>    {
>>    	struct queue_limits lim;
>>    	int error;
>> @@ -371,7 +372,13 @@ static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
>>    
>>    	lim.logical_block_size = blksize;
>>    	lim.physical_block_size = blksize;
>> -	error = queue_limits_commit_update_frozen(nbd->disk->queue, &lim);
>> +
>> +	if (freeze)
>> +		error = queue_limits_commit_update_frozen(nbd->disk->queue,
>> +				&lim);
>> +	else
>> +		error = queue_limits_commit_update(nbd->disk->queue, &lim);
>> +
>>    	if (error)
>>    		return error;
>>    
>> @@ -1563,7 +1570,7 @@ static int nbd_start_device(struct nbd_device *nbd)
>>    		args->index = i;
>>    		queue_work(nbd->recv_workq, &args->work);
>>    	}
>> -	return nbd_set_size(nbd, config->bytesize, nbd_blksize(config));
>> +	return nbd_set_size(nbd, config->bytesize, nbd_blksize(config), false);
>>    }
>>    
>>    static int nbd_start_device_ioctl(struct nbd_device *nbd)
>> @@ -1631,13 +1638,13 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd,
>>    	case NBD_SET_SOCK:
>>    		return nbd_add_socket(nbd, arg, false);
>>    	case NBD_SET_BLKSIZE:
>> -		return nbd_set_size(nbd, config->bytesize, arg);
>> +		return nbd_set_size(nbd, config->bytesize, arg, true);
>>    	case NBD_SET_SIZE:
>> -		return nbd_set_size(nbd, arg, nbd_blksize(config));
>> +		return nbd_set_size(nbd, arg, nbd_blksize(config), true);
>>    	case NBD_SET_SIZE_BLOCKS:
>>    		if (check_shl_overflow(arg, config->blksize_bits, &bytesize))
>>    			return -EINVAL;
>> -		return nbd_set_size(nbd, bytesize, nbd_blksize(config));
>> +		return nbd_set_size(nbd, bytesize, nbd_blksize(config), true);
>>    	case NBD_SET_TIMEOUT:
>>    		nbd_set_cmd_timeout(nbd, arg);
>>    		return 0;
>> @@ -2122,7 +2129,7 @@ static int nbd_genl_size_set(struct genl_info *info, struct nbd_device *nbd)
>>    		bsize = nla_get_u64(info->attrs[NBD_ATTR_BLOCK_SIZE_BYTES]);
>>    
>>    	if (bytes != config->bytesize || bsize != nbd_blksize(config))
>> -		return nbd_set_size(nbd, bytes, bsize);
>> +		return nbd_set_size(nbd, bytes, bsize, true);
> 
> This is still called from nbd_genl_connect().

nbd->pid check exist in nbd_set_size, and nbd_genl_connect call 
nbd_genl_size_set before nbd_start_device, which has not set nbd->pid, 
so path from nbd_genl_connect won't work. But for path from 
nbd_genl_reconnect, we need this protect...

> 
> Will it be much simpler to just check pid in nbd_set_size()?
> 
>>    	return 0;
>>    }
>>    
> 

Re: [PATCH v3 6/6] nbd: add nr_hw_queues module parameter for pre-created devices

[yangerkun]

在 2026/7/22 11:53, yu kuai 写道:
> Hi,
> 
> 在 2026/7/13 14:56, Yang Erkun 写道:
>> Previous commit can help remove all freeze for netlink newly created nbd
>> device. But for the devices pre-created at module load(nbds_max default
>> as 16), the nr_hw_queues was setting default as 1, then ioctl/netlink
>> path will set the real connection count, and blk_mq_update_nr_hw_queues
>> in nbd_start_device will introduce freeze.
>>
>> Add an nr_hw_queues module parameter so that users who know their
>> expected connection count can pre-created devices with the right queue
>> count to avoid this freeze too.
> 
> I'm fine with this, but this is misleading. Please also add comment to
> emphasize this module parameter is just to avoid queue freeze if user configure
> the exact number of socks as this new module parameter. Otherwise, the real
> nr_hw_queues is still depend on the real number of user configured socks.

OK, will add this comment in next version! Thanks a lot for your careful 
review!

> 
>>
>> Signed-off-by: Yang Erkun <yangerkun@huawei.com>
>> ---
>>    drivers/block/nbd.c | 8 +++++++-
>>    1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
>> index 2c7b09c70da2..f918c9efa9b2 100644
>> --- a/drivers/block/nbd.c
>> +++ b/drivers/block/nbd.c
>> @@ -166,6 +166,7 @@ static struct dentry *nbd_dbg_dir;
>>    
>>    static unsigned int nbds_max = 16;
>>    static int max_part = 16;
>> +static int nr_hw_queues = 1;
>>    static int part_shift;
>>    
>>    static int nbd_dev_dbg_init(struct nbd_device *nbd);
>> @@ -2752,8 +2753,10 @@ static int __init nbd_init(void)
>>    	}
>>    	nbd_dbg_init();
>>    
>> +	if (nr_hw_queues < 1)
>> +		nr_hw_queues = 1;
>>    	for (i = 0; i < nbds_max; i++)
>> -		nbd_dev_add(i, 1, 1);
>> +		nbd_dev_add(i, 1, nr_hw_queues);
>>    	return 0;
>>    }
>>    
>> @@ -2814,3 +2817,6 @@ module_param(nbds_max, int, 0444);
>>    MODULE_PARM_DESC(nbds_max, "number of network block devices to initialize (default: 16)");
>>    module_param(max_part, int, 0444);
>>    MODULE_PARM_DESC(max_part, "number of partitions per device (default: 16)");
>> +module_param(nr_hw_queues, int, 0444);
>> +MODULE_PARM_DESC(nr_hw_queues,
>> +"number of hardware queues for devices pre-created at module load (default: 1). ");
>