* [PATCH 0/3] null_blk: documentation, defaults and mapping
@ 2013-12-20 23:10 Matias Bjorling
2013-12-20 23:10 ` [PATCH 1/3] null_blk: corrections to documentation Matias Bjorling
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Matias Bjorling @ 2013-12-20 23:10 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel, Matias Bjorling
Hi,
These three patches cover:
* Incorporated the feedback from Randy Dunlap into documentation.
Can be merged with the previous documentation commit (6824518).
* Set use_per_node_hctx to false per default to save resources.
* Allow submit_queues and use_per_node_hctx to be used simultanesly.
If the submit_queues is a multiple of nr_online_nodes, its trivial. Simply map
them to the nodes. For example: 8 submit queues are mapped as node0[0,1],
node1[2,3], ...
If uneven, we are left with an uneven number of submit_queues that must be
mapped. These are mapped toward the first node and onward. E.g. 5
submit queues mapped onto 4 nodes are mapped as node0[0,1], node1[2], ...
Matias Bjørling (3):
null_blk: corrections to documentation
null_blk: set use_per_node_hctx param to false
null_blk: support submit_queues on use_per_node_hctx
Documentation/block/null_blk.txt | 27 +++++++++++++------------
drivers/block/null_blk.c | 43 ++++++++++++++++++++++++++++++++++------
2 files changed, 51 insertions(+), 19 deletions(-)
--
1.8.3.2
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/3] null_blk: corrections to documentation
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
@ 2013-12-20 23:10 ` Matias Bjorling
2013-12-21 3:13 ` Randy Dunlap
2013-12-20 23:11 ` [PATCH 2/3] null_blk: set use_per_node_hctx param to false Matias Bjorling
` (2 subsequent siblings)
3 siblings, 1 reply; 6+ messages in thread
From: Matias Bjorling @ 2013-12-20 23:10 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel, Matias Bjørling
From: Matias Bjørling <m@bjorling.me>
Randy Dunlap reported a couple of grammar errors and unfortunate usages of
socket/node/core.
Signed-off-by: Matias Bjorling <m@bjorling.me>
---
Documentation/block/null_blk.txt | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt
index 9e1b047..5603dad 100644
--- a/Documentation/block/null_blk.txt
+++ b/Documentation/block/null_blk.txt
@@ -18,7 +18,7 @@ The following instances are possible:
- Bio-based. IO requests are submitted directly to the device driver.
- Directly accepts bio data structure and returns them.
-All of them has a completion queue for each core in the system.
+All of them have a completion queue for each core in the system.
II. Module parameters applicable for all instances:
@@ -30,7 +30,7 @@ queue_mode=[0-2]: Default: 2-Multi-queue
2: Multi-queue.
home_node=[0--nr_nodes]: Default: NUMA_NO_NODE
- Selects what socket the data structures is allocated from.
+ Selects what CPU node the data structures are allocated from.
gb=[Size in GB]: Default: 250GB
The size of the device reported to the system.
@@ -38,34 +38,34 @@ gb=[Size in GB]: Default: 250GB
bs=[Block size (in bytes)]: Default: 512 bytes
The block size reported to the system.
-nr_devices=[Num. devices]: Default: 2
+nr_devices=[Number of devices]: Default: 2
Number of block devices instantiated. They are instantiated as /dev/nullb0,
etc.
-irq_mode=[0-2]: Default: Soft-irq
+irq_mode=[0-2]: Default: 1-Soft-irq
The completion mode used for completing IOs to the block-layer.
0: None.
- 1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead
- when IOs are issued from another socket than the home the device is
+ 1: Soft-irq. Uses IPI to complete IOs across CPU nodes. Simulates the overhead
+ when IOs are issued from another CPU node than the home the device is
connected to.
2: Timer: Waits a specific period (completion_nsec) for each IO before
completion.
-completion_nsec=[Num. ns]: Default: 10.000ns
+completion_nsec=[ns]: Default: 10.000ns
Combined with irq_mode=2 (timer). The time each completion event must wait.
submit_queues=[0..nr_cpus]:
The number of submission queues attached to the device driver. If unset, it
defaults to 1 on single-queue and bio-based instances. For multi-queue,
- its ignored when use_per_node_hctx module parameter is 1.
+ it is ignored when use_per_node_hctx module parameter is 1.
-hw_queue_depth=[0..qdepth]: Defaults: 64
+hw_queue_depth=[0..qdepth]: Default: 64
The hardware queue depth of the device.
III: Multi-queue specific parameters
-use_per_node_hctx=[0/1]: Defaults: 1
+use_per_node_hctx=[0/1]: Default: 1
If 1, the multi-queue block layer is instantiated with a hardware dispatch
queue for each CPU node in the system. If 0, it is instantiated with the
number of queues defined in the submit_queues parameter.
--
1.8.3.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/3] null_blk: set use_per_node_hctx param to false
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
2013-12-20 23:10 ` [PATCH 1/3] null_blk: corrections to documentation Matias Bjorling
@ 2013-12-20 23:11 ` Matias Bjorling
2013-12-20 23:11 ` [PATCH 3/3] null_blk: support submit_queues on use_per_node_hctx Matias Bjorling
2013-12-21 16:30 ` [PATCH 0/3] null_blk: documentation, defaults and mapping Jens Axboe
3 siblings, 0 replies; 6+ messages in thread
From: Matias Bjorling @ 2013-12-20 23:11 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel, Matias Bjørling
From: Matias Bjørling <m@bjorling.me>
The defaults for the module is to instantiate itself with blk-mq and a
submit queue for each CPU node in the system.
To save resources, initialize instead with a single submit queue.
Signed-off-by: Matias Bjorling <m@bjorling.me>
---
Documentation/block/null_blk.txt | 9 +++++----
drivers/block/null_blk.c | 4 ++--
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt
index 5603dad..b2830b4 100644
--- a/Documentation/block/null_blk.txt
+++ b/Documentation/block/null_blk.txt
@@ -65,7 +65,8 @@ hw_queue_depth=[0..qdepth]: Default: 64
III: Multi-queue specific parameters
-use_per_node_hctx=[0/1]: Default: 1
- If 1, the multi-queue block layer is instantiated with a hardware dispatch
- queue for each CPU node in the system. If 0, it is instantiated with the
- number of queues defined in the submit_queues parameter.
+use_per_node_hctx=[0/1]: Default: 0
+ 0: The number of submit queues are set to the value of the submit_queues
+ parameter.
+ 1: The multi-queue block layer is instantiated with a hardware dispatch
+ queue for each CPU node in the system.
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 8f2e7c3..9b0311b 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -101,9 +101,9 @@ static int hw_queue_depth = 64;
module_param(hw_queue_depth, int, S_IRUGO);
MODULE_PARM_DESC(hw_queue_depth, "Queue depth for each hardware queue. Default: 64");
-static bool use_per_node_hctx = true;
+static bool use_per_node_hctx = false;
module_param(use_per_node_hctx, bool, S_IRUGO);
-MODULE_PARM_DESC(use_per_node_hctx, "Use per-node allocation for hardware context queues. Default: true");
+MODULE_PARM_DESC(use_per_node_hctx, "Use per-node allocation for hardware context queues. Default: false");
static void put_tag(struct nullb_queue *nq, unsigned int tag)
{
--
1.8.3.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/3] null_blk: support submit_queues on use_per_node_hctx
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
2013-12-20 23:10 ` [PATCH 1/3] null_blk: corrections to documentation Matias Bjorling
2013-12-20 23:11 ` [PATCH 2/3] null_blk: set use_per_node_hctx param to false Matias Bjorling
@ 2013-12-20 23:11 ` Matias Bjorling
2013-12-21 16:30 ` [PATCH 0/3] null_blk: documentation, defaults and mapping Jens Axboe
3 siblings, 0 replies; 6+ messages in thread
From: Matias Bjorling @ 2013-12-20 23:11 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel, Matias Bjørling
From: Matias Bjørling <m@bjorling.me>
In the case of both the submit_queues param and use_per_node_hctx param
are used. We limit the number af submit_queues to the number of online
nodes.
If the submit_queues is a multiple of nr_online_nodes, its trivial. Simply map
them to the nodes. For example: 8 submit queues are mapped as node0[0,1],
node1[2,3], ...
If uneven, we are left with an uneven number of submit_queues that must be
mapped. These are mapped toward the first node and onward. E.g. 5
submit queues mapped onto 4 nodes are mapped as node0[0,1], node1[2], ...
Signed-off-by: Matias Bjorling <m@bjorling.me>
---
drivers/block/null_blk.c | 39 +++++++++++++++++++++++++++++++++++----
1 file changed, 35 insertions(+), 4 deletions(-)
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 9b0311b..528f4e4 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -1,4 +1,5 @@
#include <linux/module.h>
+
#include <linux/moduleparam.h>
#include <linux/sched.h>
#include <linux/fs.h>
@@ -346,8 +347,37 @@ static int null_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq)
static struct blk_mq_hw_ctx *null_alloc_hctx(struct blk_mq_reg *reg, unsigned int hctx_index)
{
- return kzalloc_node(sizeof(struct blk_mq_hw_ctx), GFP_KERNEL,
- hctx_index);
+ int b_size = DIV_ROUND_UP(reg->nr_hw_queues, nr_online_nodes);
+ int tip = (reg->nr_hw_queues % nr_online_nodes);
+ int node = 0, i, n;
+
+ /*
+ * Split submit queues evenly wrt to the number of nodes. If uneven,
+ * fill the first buckets with one extra, until the rest is filled with
+ * no extra.
+ */
+ for (i = 0, n = 1; i < hctx_index; i++, n++) {
+ if (n % b_size == 0) {
+ n = 0;
+ node++;
+
+ tip--;
+ if (!tip)
+ b_size = reg->nr_hw_queues / nr_online_nodes;
+ }
+ }
+
+ /*
+ * A node might not be online, therefore map the relative node id to the
+ * real node id.
+ */
+ for_each_online_node(n) {
+ if (!node)
+ break;
+ node--;
+ }
+
+ return kzalloc_node(sizeof(struct blk_mq_hw_ctx), GFP_KERNEL, n);
}
static void null_free_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_index)
@@ -591,10 +621,11 @@ static int __init null_init(void)
#endif
if (queue_mode == NULL_Q_MQ && use_per_node_hctx) {
- if (submit_queues > 0)
+ if (submit_queues < nr_online_nodes) {
pr_warn("null_blk: submit_queues param is set to %u.",
nr_online_nodes);
- submit_queues = nr_online_nodes;
+ submit_queues = nr_online_nodes;
+ }
} else if (submit_queues > nr_cpu_ids)
submit_queues = nr_cpu_ids;
else if (!submit_queues)
--
1.8.3.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/3] null_blk: corrections to documentation
2013-12-20 23:10 ` [PATCH 1/3] null_blk: corrections to documentation Matias Bjorling
@ 2013-12-21 3:13 ` Randy Dunlap
0 siblings, 0 replies; 6+ messages in thread
From: Randy Dunlap @ 2013-12-21 3:13 UTC (permalink / raw)
To: Matias Bjorling, axboe; +Cc: linux-kernel
On 12/20/13 15:10, Matias Bjorling wrote:
> From: Matias Bjørling <m@bjorling.me>
>
> Randy Dunlap reported a couple of grammar errors and unfortunate usages of
> socket/node/core.
>
> Signed-off-by: Matias Bjorling <m@bjorling.me>
> ---
Thanks, looks good.
Acked-by: Randy Dunlap <rdunlap@infradead.org>
--
~Randy
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/3] null_blk: documentation, defaults and mapping
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
` (2 preceding siblings ...)
2013-12-20 23:11 ` [PATCH 3/3] null_blk: support submit_queues on use_per_node_hctx Matias Bjorling
@ 2013-12-21 16:30 ` Jens Axboe
3 siblings, 0 replies; 6+ messages in thread
From: Jens Axboe @ 2013-12-21 16:30 UTC (permalink / raw)
To: Matias Bjorling; +Cc: linux-kernel
On Sat, Dec 21 2013, Matias Bjorling wrote:
> Hi,
>
> These three patches cover:
>
> * Incorporated the feedback from Randy Dunlap into documentation.
> Can be merged with the previous documentation commit (6824518).
> * Set use_per_node_hctx to false per default to save resources.
> * Allow submit_queues and use_per_node_hctx to be used simultanesly.
> If the submit_queues is a multiple of nr_online_nodes, its trivial. Simply map
> them to the nodes. For example: 8 submit queues are mapped as node0[0,1],
> node1[2,3], ...
> If uneven, we are left with an uneven number of submit_queues that must be
> mapped. These are mapped toward the first node and onward. E.g. 5
> submit queues mapped onto 4 nodes are mapped as node0[0,1], node1[2], ...
Thanks, applied.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-12-21 16:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-20 23:10 [PATCH 0/3] null_blk: documentation, defaults and mapping Matias Bjorling
2013-12-20 23:10 ` [PATCH 1/3] null_blk: corrections to documentation Matias Bjorling
2013-12-21 3:13 ` Randy Dunlap
2013-12-20 23:11 ` [PATCH 2/3] null_blk: set use_per_node_hctx param to false Matias Bjorling
2013-12-20 23:11 ` [PATCH 3/3] null_blk: support submit_queues on use_per_node_hctx Matias Bjorling
2013-12-21 16:30 ` [PATCH 0/3] null_blk: documentation, defaults and mapping Jens Axboe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.