[PATCH 0/3] null_blk: documentation, refactor and warning

[PATCH 0/3] null_blk: documentation, refactor and warning

[Matias Bjorling]

Hi Jens,

I took the liberty to write up some documentation for the blk_null driver.

Additionally, factored the init code paths to use the standard error return
codes. Shuffled some code to aid sharing between the block-layer instantiations.
And at last added a warning when submit_queues parameter is ignored.

Thanks,
Matias

Matias Bjorling (3):
  null_blk: documentation
  null_blk: refactor init and init errors code paths
  null_blk: warning on ignored submit_queues param

 Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++
 drivers/block/null_blk.c         | 66 +++++++++++++++++++++++--------------
 2 files changed, 112 insertions(+), 25 deletions(-)
 create mode 100644 Documentation/block/null_blk.txt

-- 
1.8.3.2

[PATCH 1/3] null_blk: documentation

[Matias Bjorling]

Add description of module and its parameters.

Signed-off-by: Matias Bjorling <m@bjorling.me>
---
 Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 Documentation/block/null_blk.txt

diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt
new file mode 100644
index 0000000..9e1b047
--- /dev/null
+++ b/Documentation/block/null_blk.txt
@@ -0,0 +1,71 @@
+Null block device driver
+================================================================================
+
+I. Overview
+
+The null block device (/dev/nullb*) is used for benchmarking the various
+block-layer implementations. It emulates a block device of X gigabytes in size.
+The following instances are possible:
+
+  Single-queue block-layer
+    - Request-based.
+    - Single submission queue per device.
+    - Implements IO scheduling algorithms (CFQ, Deadline, noop).
+  Multi-queue block-layer
+    - Request-based.
+    - Configurable submission queues per device.
+  No block-layer (Known as bio-based)
+    - Bio-based. IO requests are submitted directly to the device driver.
+    - Directly accepts bio data structure and returns them.
+
+All of them has a completion queue for each core in the system.
+
+II. Module parameters applicable for all instances:
+
+queue_mode=[0-2]: Default: 2-Multi-queue
+  Selects which block-layer the module should instantiate with.
+
+  0: Bio-based.
+  1: Single-queue.
+  2: Multi-queue.
+
+home_node=[0--nr_nodes]: Default: NUMA_NO_NODE
+  Selects what socket the data structures is allocated from.
+
+gb=[Size in GB]: Default: 250GB
+  The size of the device reported to the system.
+
+bs=[Block size (in bytes)]: Default: 512 bytes
+  The block size reported to the system.
+
+nr_devices=[Num. devices]: Default: 2
+  Number of block devices instantiated. They are instantiated as /dev/nullb0,
+  etc.
+
+irq_mode=[0-2]: Default: Soft-irq
+  The completion mode used for completing IOs to the block-layer.
+
+  0: None.
+  1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead
+     when IOs are issued from another socket than the home the device is
+     connected to.
+  2: Timer: Waits a specific period (completion_nsec) for each IO before
+     completion.
+
+completion_nsec=[Num. ns]: Default: 10.000ns
+  Combined with irq_mode=2 (timer). The time each completion event must wait.
+
+submit_queues=[0..nr_cpus]:
+  The number of submission queues attached to the device driver. If unset, it
+  defaults to 1 on single-queue and bio-based instances. For multi-queue,
+  its ignored when use_per_node_hctx module parameter is 1.
+
+hw_queue_depth=[0..qdepth]: Defaults: 64
+  The hardware queue depth of the device.
+
+III: Multi-queue specific parameters
+
+use_per_node_hctx=[0/1]: Defaults: 1
+  If 1, the multi-queue block layer is instantiated with a hardware dispatch
+  queue for each CPU node in the system. If 0, it is instantiated with the
+  number of queues defined in the submit_queues parameter.
-- 
1.8.3.2

[PATCH 2/3] null_blk: refactor init and init errors code paths

[Matias Bjorling]

Simplify the initialization logic of the three block-layers.

- The queue initialization is split into two parts. This allows reuse of
  code when initializing the sq-, bio- and mq-based layers.
- Set submit_queues default value to 0 and always set it at init time.
- Simplify the init error code paths.

Signed-off-by: Matias Bjorling <m@bjorling.me>
---
 drivers/block/null_blk.c | 63 +++++++++++++++++++++++++++++-------------------
 1 file changed, 38 insertions(+), 25 deletions(-)

diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index f370fc1..f0aeb2a 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -65,7 +65,7 @@ enum {
 	NULL_Q_MQ		= 2,
 };
 
-static int submit_queues = 1;
+static int submit_queues;
 module_param(submit_queues, int, S_IRUGO);
 MODULE_PARM_DESC(submit_queues, "Number of submission queues");
 
@@ -355,16 +355,24 @@ static void null_free_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_index)
 	kfree(hctx);
 }
 
+static void null_init_queue(struct nullb *nullb, struct nullb_queue *nq)
+{
+	BUG_ON(!nullb);
+	BUG_ON(!nq);
+
+	init_waitqueue_head(&nq->wait);
+	nq->queue_depth = nullb->queue_depth;
+}
+
 static int null_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
 			  unsigned int index)
 {
 	struct nullb *nullb = data;
 	struct nullb_queue *nq = &nullb->queues[index];
 
-	init_waitqueue_head(&nq->wait);
-	nq->queue_depth = nullb->queue_depth;
-	nullb->nr_queues++;
 	hctx->driver_data = nq;
+	null_init_queue(nullb, nq);
+	nullb->nr_queues++;
 
 	return 0;
 }
@@ -417,13 +425,13 @@ static int setup_commands(struct nullb_queue *nq)
 
 	nq->cmds = kzalloc(nq->queue_depth * sizeof(*cmd), GFP_KERNEL);
 	if (!nq->cmds)
-		return 1;
+		return -ENOMEM;
 
 	tag_size = ALIGN(nq->queue_depth, BITS_PER_LONG) / BITS_PER_LONG;
 	nq->tag_map = kzalloc(tag_size * sizeof(unsigned long), GFP_KERNEL);
 	if (!nq->tag_map) {
 		kfree(nq->cmds);
-		return 1;
+		return -ENOMEM;
 	}
 
 	for (i = 0; i < nq->queue_depth; i++) {
@@ -454,33 +462,37 @@ static void cleanup_queues(struct nullb *nullb)
 
 static int setup_queues(struct nullb *nullb)
 {
-	struct nullb_queue *nq;
-	int i;
-
-	nullb->queues = kzalloc(submit_queues * sizeof(*nq), GFP_KERNEL);
+	nullb->queues = kzalloc(submit_queues * sizeof(struct nullb_queue),
+								GFP_KERNEL);
 	if (!nullb->queues)
-		return 1;
+		return -ENOMEM;
 
 	nullb->nr_queues = 0;
 	nullb->queue_depth = hw_queue_depth;
 
-	if (queue_mode == NULL_Q_MQ)
-		return 0;
+	return 0;
+}
+
+static int init_driver_queues(struct nullb *nullb)
+{
+	struct nullb_queue *nq;
+	int i, ret = 0;
 
 	for (i = 0; i < submit_queues; i++) {
 		nq = &nullb->queues[i];
-		init_waitqueue_head(&nq->wait);
-		nq->queue_depth = hw_queue_depth;
-		if (setup_commands(nq))
-			break;
+
+		null_init_queue(nullb, nq);
+
+		ret = setup_commands(nq);
+		if (ret)
+			goto err_queue;
 		nullb->nr_queues++;
 	}
 
-	if (i == submit_queues)
-		return 0;
-
+	return 0;
+err_queue:
 	cleanup_queues(nullb);
-	return 1;
+	return ret;
 }
 
 static int null_add_dev(void)
@@ -495,9 +507,6 @@ static int null_add_dev(void)
 
 	spin_lock_init(&nullb->lock);
 
-	if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
-		submit_queues = nr_online_nodes;
-
 	if (setup_queues(nullb))
 		goto err;
 
@@ -518,11 +527,13 @@ static int null_add_dev(void)
 	} else if (queue_mode == NULL_Q_BIO) {
 		nullb->q = blk_alloc_queue_node(GFP_KERNEL, home_node);
 		blk_queue_make_request(nullb->q, null_queue_bio);
+		init_driver_queues(nullb);
 	} else {
 		nullb->q = blk_init_queue_node(null_request_fn, &nullb->lock, home_node);
 		blk_queue_prep_rq(nullb->q, null_rq_prep_fn);
 		if (nullb->q)
 			blk_queue_softirq_done(nullb->q, null_softirq_done_fn);
+		init_driver_queues(nullb);
 	}
 
 	if (!nullb->q)
@@ -579,7 +590,9 @@ static int __init null_init(void)
 	}
 #endif
 
-	if (submit_queues > nr_cpu_ids)
+	if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
+		submit_queues = nr_online_nodes;
+	else if (submit_queues > nr_cpu_ids)
 		submit_queues = nr_cpu_ids;
 	else if (!submit_queues)
 		submit_queues = 1;
-- 
1.8.3.2

[PATCH 3/3] null_blk: warning on ignored submit_queues param

[Matias Bjorling]

Let the user know when the number of submission queues are being
ignored.

Signed-off-by: Matias Bjorling <m@bjorling.me>
---
 drivers/block/null_blk.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index f0aeb2a..8f2e7c3 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -590,9 +590,12 @@ static int __init null_init(void)
 	}
 #endif
 
-	if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
+	if (queue_mode == NULL_Q_MQ && use_per_node_hctx) {
+		if (submit_queues > 0)
+			pr_warn("null_blk: submit_queues param is set to %u.",
+							nr_online_nodes);
 		submit_queues = nr_online_nodes;
-	else if (submit_queues > nr_cpu_ids)
+	} else if (submit_queues > nr_cpu_ids)
 		submit_queues = nr_cpu_ids;
 	else if (!submit_queues)
 		submit_queues = 1;
-- 
1.8.3.2

Re: [PATCH 1/3] null_blk: documentation

[Randy Dunlap]

On 12/18/13 04:41, Matias Bjorling wrote:
> Add description of module and its parameters.
> 
> Signed-off-by: Matias Bjorling <m@bjorling.me>
> ---
>  Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 71 insertions(+)
>  create mode 100644 Documentation/block/null_blk.txt
> 
> diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt
> new file mode 100644
> index 0000000..9e1b047
> --- /dev/null
> +++ b/Documentation/block/null_blk.txt
> @@ -0,0 +1,71 @@
> +Null block device driver
> +================================================================================
> +
> +I. Overview
> +
> +The null block device (/dev/nullb*) is used for benchmarking the various
> +block-layer implementations. It emulates a block device of X gigabytes in size.
> +The following instances are possible:
> +
> +  Single-queue block-layer
> +    - Request-based.
> +    - Single submission queue per device.
> +    - Implements IO scheduling algorithms (CFQ, Deadline, noop).
> +  Multi-queue block-layer
> +    - Request-based.
> +    - Configurable submission queues per device.
> +  No block-layer (Known as bio-based)
> +    - Bio-based. IO requests are submitted directly to the device driver.
> +    - Directly accepts bio data structure and returns them.
> +
> +All of them has a completion queue for each core in the system.

               have

The text sometimes uses "core" vs. "socket" vs. "cpu(s)".
Did you double-check all of these uses?  Thanks.


> +
> +II. Module parameters applicable for all instances:
> +
> +queue_mode=[0-2]: Default: 2-Multi-queue
> +  Selects which block-layer the module should instantiate with.
> +
> +  0: Bio-based.
> +  1: Single-queue.
> +  2: Multi-queue.
> +
> +home_node=[0--nr_nodes]: Default: NUMA_NO_NODE
> +  Selects what socket the data structures is allocated from.

                                             are

> +
> +gb=[Size in GB]: Default: 250GB
> +  The size of the device reported to the system.
> +
> +bs=[Block size (in bytes)]: Default: 512 bytes
> +  The block size reported to the system.
> +
> +nr_devices=[Num. devices]: Default: 2

               Number

> +  Number of block devices instantiated. They are instantiated as /dev/nullb0,
> +  etc.
> +
> +irq_mode=[0-2]: Default: Soft-irq

                   Default: 1-Soft-irq

> +  The completion mode used for completing IOs to the block-layer.
> +
> +  0: None.
> +  1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead

                       IPI

> +     when IOs are issued from another socket than the home the device is
> +     connected to.
> +  2: Timer: Waits a specific period (completion_nsec) for each IO before
> +     completion.
> +
> +completion_nsec=[Num. ns]: Default: 10.000ns

                    Number

> +  Combined with irq_mode=2 (timer). The time each completion event must wait.
> +
> +submit_queues=[0..nr_cpus]:
> +  The number of submission queues attached to the device driver. If unset, it
> +  defaults to 1 on single-queue and bio-based instances. For multi-queue,
> +  its ignored when use_per_node_hctx module parameter is 1.

     it is

> +
> +hw_queue_depth=[0..qdepth]: Defaults: 64

                               Default:

> +  The hardware queue depth of the device.
> +
> +III: Multi-queue specific parameters
> +
> +use_per_node_hctx=[0/1]: Defaults: 1

                            Default:

> +  If 1, the multi-queue block layer is instantiated with a hardware dispatch
> +  queue for each CPU node in the system. If 0, it is instantiated with the
> +  number of queues defined in the submit_queues parameter.
> 


-- 
~Randy

Re: [PATCH 0/3] null_blk: documentation, refactor and warning

[Jens Axboe]

On Wed, Dec 18 2013, Matias Bjorling wrote:
> Hi Jens,
> 
> I took the liberty to write up some documentation for the blk_null driver.
> 
> Additionally, factored the init code paths to use the standard error return
> codes. Shuffled some code to aid sharing between the block-layer instantiations.
> And at last added a warning when submit_queues parameter is ignored.

Thanks Matias, applied! We still need a patch to not default to per-node
hctx, though. And to be more clever when submit_queues is given and
per-node is true.

-- 
Jens Axboe