[PATCH] null_blk: mem garbage on NUMA systems during init

[PATCH] null_blk: mem garbage on NUMA systems during init

[Matias Bjorling]

For NUMA systems, initializing the blk-mq layer and using per node hctx. We
initialize submit queues to 1, while blk-mq nr_hw_queues is initialized to the
number of NUMA nodes.

This makes the null_init_hctx function overwrite memory outside of what it
allocated. In my case it lead to writing garbage into struct request_queue's
mq_map.

Signed-off-by: Matias Bjorling <m@bjorling.me>
---
 drivers/block/null_blk.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index ea192ec..f370fc1 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -495,23 +495,23 @@ static int null_add_dev(void)
 
 	spin_lock_init(&nullb->lock);
 
+	if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
+		submit_queues = nr_online_nodes;
+
 	if (setup_queues(nullb))
 		goto err;
 
 	if (queue_mode == NULL_Q_MQ) {
 		null_mq_reg.numa_node = home_node;
 		null_mq_reg.queue_depth = hw_queue_depth;
+		null_mq_reg.nr_hw_queues = submit_queues;
 
 		if (use_per_node_hctx) {
 			null_mq_reg.ops->alloc_hctx = null_alloc_hctx;
 			null_mq_reg.ops->free_hctx = null_free_hctx;
-
-			null_mq_reg.nr_hw_queues = nr_online_nodes;
 		} else {
 			null_mq_reg.ops->alloc_hctx = blk_mq_alloc_single_hw_queue;
 			null_mq_reg.ops->free_hctx = blk_mq_free_single_hw_queue;
-
-			null_mq_reg.nr_hw_queues = submit_queues;
 		}
 
 		nullb->q = blk_mq_init_queue(&null_mq_reg, nullb);
-- 
1.8.3.2

Re: [PATCH] null_blk: mem garbage on NUMA systems during init

[Linus Torvalds]

Jens, ping?

Patch looks fine, and it looks completely bogus to set

     null_mq_reg.nr_hw_queues

to 'nr_online_nodes', when we've potentially done "setup_queues()"
with just a single queue.

So I'm going to apply this as-is, I'm just wondering why I haven't
heard anything from you..

                     Linus


On Tue, Dec 10, 2013 at 7:50 AM, Matias Bjorling <m@bjorling.me> wrote:
> For NUMA systems, initializing the blk-mq layer and using per node hctx. We
> initialize submit queues to 1, while blk-mq nr_hw_queues is initialized to the
> number of NUMA nodes.
>
> This makes the null_init_hctx function overwrite memory outside of what it
> allocated. In my case it lead to writing garbage into struct request_queue's
> mq_map.
>
> Signed-off-by: Matias Bjorling <m@bjorling.me>
> ---
>  drivers/block/null_blk.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
> index ea192ec..f370fc1 100644
> --- a/drivers/block/null_blk.c
> +++ b/drivers/block/null_blk.c
> @@ -495,23 +495,23 @@ static int null_add_dev(void)
>
>         spin_lock_init(&nullb->lock);
>
> +       if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
> +               submit_queues = nr_online_nodes;
> +
>         if (setup_queues(nullb))
>                 goto err;
>
>         if (queue_mode == NULL_Q_MQ) {
>                 null_mq_reg.numa_node = home_node;
>                 null_mq_reg.queue_depth = hw_queue_depth;
> +               null_mq_reg.nr_hw_queues = submit_queues;
>
>                 if (use_per_node_hctx) {
>                         null_mq_reg.ops->alloc_hctx = null_alloc_hctx;
>                         null_mq_reg.ops->free_hctx = null_free_hctx;
> -
> -                       null_mq_reg.nr_hw_queues = nr_online_nodes;
>                 } else {
>                         null_mq_reg.ops->alloc_hctx = blk_mq_alloc_single_hw_queue;
>                         null_mq_reg.ops->free_hctx = blk_mq_free_single_hw_queue;
> -
> -                       null_mq_reg.nr_hw_queues = submit_queues;
>                 }
>
>                 nullb->q = blk_mq_init_queue(&null_mq_reg, nullb);
> --
> 1.8.3.2
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] null_blk: mem garbage on NUMA systems during init

[Jens Axboe]

On Sun, Dec 15 2013, Linus Torvalds wrote:
> Jens, ping?
> 
> Patch looks fine, and it looks completely bogus to set
> 
>      null_mq_reg.nr_hw_queues
> 
> to 'nr_online_nodes', when we've potentially done "setup_queues()"
> with just a single queue.
> 
> So I'm going to apply this as-is, I'm just wondering why I haven't
> heard anything from you..

Sorry, I talked with Matias on IM about it, hence no lkml reference to
it. It is queued up. You can apply as-is too, that's fine too.

-- 
Jens Axboe