* [PATCH 1/3] null_blk: corrections to documentation
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
@ 2013-12-20 23:10 ` Matias Bjorling
2013-12-21 3:13 ` Randy Dunlap
2013-12-20 23:11 ` [PATCH 2/3] null_blk: set use_per_node_hctx param to false Matias Bjorling
` (2 subsequent siblings)
3 siblings, 1 reply; 6+ messages in thread
From: Matias Bjorling @ 2013-12-20 23:10 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel, Matias Bjørling
From: Matias Bjørling <m@bjorling.me>
Randy Dunlap reported a couple of grammar errors and unfortunate usages of
socket/node/core.
Signed-off-by: Matias Bjorling <m@bjorling.me>
---
Documentation/block/null_blk.txt | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt
index 9e1b047..5603dad 100644
--- a/Documentation/block/null_blk.txt
+++ b/Documentation/block/null_blk.txt
@@ -18,7 +18,7 @@ The following instances are possible:
- Bio-based. IO requests are submitted directly to the device driver.
- Directly accepts bio data structure and returns them.
-All of them has a completion queue for each core in the system.
+All of them have a completion queue for each core in the system.
II. Module parameters applicable for all instances:
@@ -30,7 +30,7 @@ queue_mode=[0-2]: Default: 2-Multi-queue
2: Multi-queue.
home_node=[0--nr_nodes]: Default: NUMA_NO_NODE
- Selects what socket the data structures is allocated from.
+ Selects what CPU node the data structures are allocated from.
gb=[Size in GB]: Default: 250GB
The size of the device reported to the system.
@@ -38,34 +38,34 @@ gb=[Size in GB]: Default: 250GB
bs=[Block size (in bytes)]: Default: 512 bytes
The block size reported to the system.
-nr_devices=[Num. devices]: Default: 2
+nr_devices=[Number of devices]: Default: 2
Number of block devices instantiated. They are instantiated as /dev/nullb0,
etc.
-irq_mode=[0-2]: Default: Soft-irq
+irq_mode=[0-2]: Default: 1-Soft-irq
The completion mode used for completing IOs to the block-layer.
0: None.
- 1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead
- when IOs are issued from another socket than the home the device is
+ 1: Soft-irq. Uses IPI to complete IOs across CPU nodes. Simulates the overhead
+ when IOs are issued from another CPU node than the home the device is
connected to.
2: Timer: Waits a specific period (completion_nsec) for each IO before
completion.
-completion_nsec=[Num. ns]: Default: 10.000ns
+completion_nsec=[ns]: Default: 10.000ns
Combined with irq_mode=2 (timer). The time each completion event must wait.
submit_queues=[0..nr_cpus]:
The number of submission queues attached to the device driver. If unset, it
defaults to 1 on single-queue and bio-based instances. For multi-queue,
- its ignored when use_per_node_hctx module parameter is 1.
+ it is ignored when use_per_node_hctx module parameter is 1.
-hw_queue_depth=[0..qdepth]: Defaults: 64
+hw_queue_depth=[0..qdepth]: Default: 64
The hardware queue depth of the device.
III: Multi-queue specific parameters
-use_per_node_hctx=[0/1]: Defaults: 1
+use_per_node_hctx=[0/1]: Default: 1
If 1, the multi-queue block layer is instantiated with a hardware dispatch
queue for each CPU node in the system. If 0, it is instantiated with the
number of queues defined in the submit_queues parameter.
--
1.8.3.2
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 2/3] null_blk: set use_per_node_hctx param to false
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
2013-12-20 23:10 ` [PATCH 1/3] null_blk: corrections to documentation Matias Bjorling
@ 2013-12-20 23:11 ` Matias Bjorling
2013-12-20 23:11 ` [PATCH 3/3] null_blk: support submit_queues on use_per_node_hctx Matias Bjorling
2013-12-21 16:30 ` [PATCH 0/3] null_blk: documentation, defaults and mapping Jens Axboe
3 siblings, 0 replies; 6+ messages in thread
From: Matias Bjorling @ 2013-12-20 23:11 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel, Matias Bjørling
From: Matias Bjørling <m@bjorling.me>
The defaults for the module is to instantiate itself with blk-mq and a
submit queue for each CPU node in the system.
To save resources, initialize instead with a single submit queue.
Signed-off-by: Matias Bjorling <m@bjorling.me>
---
Documentation/block/null_blk.txt | 9 +++++----
drivers/block/null_blk.c | 4 ++--
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt
index 5603dad..b2830b4 100644
--- a/Documentation/block/null_blk.txt
+++ b/Documentation/block/null_blk.txt
@@ -65,7 +65,8 @@ hw_queue_depth=[0..qdepth]: Default: 64
III: Multi-queue specific parameters
-use_per_node_hctx=[0/1]: Default: 1
- If 1, the multi-queue block layer is instantiated with a hardware dispatch
- queue for each CPU node in the system. If 0, it is instantiated with the
- number of queues defined in the submit_queues parameter.
+use_per_node_hctx=[0/1]: Default: 0
+ 0: The number of submit queues are set to the value of the submit_queues
+ parameter.
+ 1: The multi-queue block layer is instantiated with a hardware dispatch
+ queue for each CPU node in the system.
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 8f2e7c3..9b0311b 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -101,9 +101,9 @@ static int hw_queue_depth = 64;
module_param(hw_queue_depth, int, S_IRUGO);
MODULE_PARM_DESC(hw_queue_depth, "Queue depth for each hardware queue. Default: 64");
-static bool use_per_node_hctx = true;
+static bool use_per_node_hctx = false;
module_param(use_per_node_hctx, bool, S_IRUGO);
-MODULE_PARM_DESC(use_per_node_hctx, "Use per-node allocation for hardware context queues. Default: true");
+MODULE_PARM_DESC(use_per_node_hctx, "Use per-node allocation for hardware context queues. Default: false");
static void put_tag(struct nullb_queue *nq, unsigned int tag)
{
--
1.8.3.2
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 3/3] null_blk: support submit_queues on use_per_node_hctx
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
2013-12-20 23:10 ` [PATCH 1/3] null_blk: corrections to documentation Matias Bjorling
2013-12-20 23:11 ` [PATCH 2/3] null_blk: set use_per_node_hctx param to false Matias Bjorling
@ 2013-12-20 23:11 ` Matias Bjorling
2013-12-21 16:30 ` [PATCH 0/3] null_blk: documentation, defaults and mapping Jens Axboe
3 siblings, 0 replies; 6+ messages in thread
From: Matias Bjorling @ 2013-12-20 23:11 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel, Matias Bjørling
From: Matias Bjørling <m@bjorling.me>
In the case of both the submit_queues param and use_per_node_hctx param
are used. We limit the number af submit_queues to the number of online
nodes.
If the submit_queues is a multiple of nr_online_nodes, its trivial. Simply map
them to the nodes. For example: 8 submit queues are mapped as node0[0,1],
node1[2,3], ...
If uneven, we are left with an uneven number of submit_queues that must be
mapped. These are mapped toward the first node and onward. E.g. 5
submit queues mapped onto 4 nodes are mapped as node0[0,1], node1[2], ...
Signed-off-by: Matias Bjorling <m@bjorling.me>
---
drivers/block/null_blk.c | 39 +++++++++++++++++++++++++++++++++++----
1 file changed, 35 insertions(+), 4 deletions(-)
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 9b0311b..528f4e4 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -1,4 +1,5 @@
#include <linux/module.h>
+
#include <linux/moduleparam.h>
#include <linux/sched.h>
#include <linux/fs.h>
@@ -346,8 +347,37 @@ static int null_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq)
static struct blk_mq_hw_ctx *null_alloc_hctx(struct blk_mq_reg *reg, unsigned int hctx_index)
{
- return kzalloc_node(sizeof(struct blk_mq_hw_ctx), GFP_KERNEL,
- hctx_index);
+ int b_size = DIV_ROUND_UP(reg->nr_hw_queues, nr_online_nodes);
+ int tip = (reg->nr_hw_queues % nr_online_nodes);
+ int node = 0, i, n;
+
+ /*
+ * Split submit queues evenly wrt to the number of nodes. If uneven,
+ * fill the first buckets with one extra, until the rest is filled with
+ * no extra.
+ */
+ for (i = 0, n = 1; i < hctx_index; i++, n++) {
+ if (n % b_size == 0) {
+ n = 0;
+ node++;
+
+ tip--;
+ if (!tip)
+ b_size = reg->nr_hw_queues / nr_online_nodes;
+ }
+ }
+
+ /*
+ * A node might not be online, therefore map the relative node id to the
+ * real node id.
+ */
+ for_each_online_node(n) {
+ if (!node)
+ break;
+ node--;
+ }
+
+ return kzalloc_node(sizeof(struct blk_mq_hw_ctx), GFP_KERNEL, n);
}
static void null_free_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_index)
@@ -591,10 +621,11 @@ static int __init null_init(void)
#endif
if (queue_mode == NULL_Q_MQ && use_per_node_hctx) {
- if (submit_queues > 0)
+ if (submit_queues < nr_online_nodes) {
pr_warn("null_blk: submit_queues param is set to %u.",
nr_online_nodes);
- submit_queues = nr_online_nodes;
+ submit_queues = nr_online_nodes;
+ }
} else if (submit_queues > nr_cpu_ids)
submit_queues = nr_cpu_ids;
else if (!submit_queues)
--
1.8.3.2
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH 0/3] null_blk: documentation, defaults and mapping
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
` (2 preceding siblings ...)
2013-12-20 23:11 ` [PATCH 3/3] null_blk: support submit_queues on use_per_node_hctx Matias Bjorling
@ 2013-12-21 16:30 ` Jens Axboe
3 siblings, 0 replies; 6+ messages in thread
From: Jens Axboe @ 2013-12-21 16:30 UTC (permalink / raw)
To: Matias Bjorling; +Cc: linux-kernel
On Sat, Dec 21 2013, Matias Bjorling wrote:
> Hi,
>
> These three patches cover:
>
> * Incorporated the feedback from Randy Dunlap into documentation.
> Can be merged with the previous documentation commit (6824518).
> * Set use_per_node_hctx to false per default to save resources.
> * Allow submit_queues and use_per_node_hctx to be used simultanesly.
> If the submit_queues is a multiple of nr_online_nodes, its trivial. Simply map
> them to the nodes. For example: 8 submit queues are mapped as node0[0,1],
> node1[2,3], ...
> If uneven, we are left with an uneven number of submit_queues that must be
> mapped. These are mapped toward the first node and onward. E.g. 5
> submit queues mapped onto 4 nodes are mapped as node0[0,1], node1[2], ...
Thanks, applied.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread