* [PATCH] blk-mq: Fix uninitialized kobject at CPU hotplugging
@ 2014-12-10 15:38 Takashi Iwai
2014-12-10 16:01 ` Jens Axboe
0 siblings, 1 reply; 4+ messages in thread
From: Takashi Iwai @ 2014-12-10 15:38 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-kernel
When a CPU is hotplugged, the current blk-mq spews a warning like:
kobject '(null)' (ffffe8ffffc8b5d8): tried to add an uninitialized object, something is seriously wrong.
CPU: 1 PID: 1386 Comm: systemd-udevd Not tainted 3.18.0-rc7-2.g088d59b-default #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014
0000000000000000 0000000000000002 ffffffff81605f07 ffffe8ffffc8b5d8
ffffffff8132c7a0 ffff88023341d370 0000000000000020 ffff8800bb05bd58
ffff8800bb05bd08 000000000000a0a0 000000003f441940 0000000000000007
Call Trace:
[<ffffffff81005306>] dump_trace+0x86/0x330
[<ffffffff81005644>] show_stack_log_lvl+0x94/0x170
[<ffffffff81006d21>] show_stack+0x21/0x50
[<ffffffff81605f07>] dump_stack+0x41/0x51
[<ffffffff8132c7a0>] kobject_add+0xa0/0xb0
[<ffffffff8130aee1>] blk_mq_register_hctx+0x91/0xb0
[<ffffffff8130b82e>] blk_mq_sysfs_register+0x3e/0x60
[<ffffffff81309298>] blk_mq_queue_reinit_notify+0xf8/0x190
[<ffffffff8107cfdc>] notifier_call_chain+0x4c/0x70
[<ffffffff8105fd23>] cpu_notify+0x23/0x50
[<ffffffff81060037>] _cpu_up+0x157/0x170
[<ffffffff810600d9>] cpu_up+0x89/0xb0
[<ffffffff815fa5b5>] cpu_subsys_online+0x35/0x80
[<ffffffff814323cd>] device_online+0x5d/0xa0
[<ffffffff81432485>] online_store+0x75/0x80
[<ffffffff81236a5a>] kernfs_fop_write+0xda/0x150
[<ffffffff811c5532>] vfs_write+0xb2/0x1f0
[<ffffffff811c5f42>] SyS_write+0x42/0xb0
[<ffffffff8160c4ed>] system_call_fastpath+0x16/0x1b
[<00007f0132fb24e0>] 0x7f0132fb24e0
This is indeed because of an uninitialized kobject for blk_mq_ctx.
The blk_mq_ctx kobjects are initialized in blk_mq_sysfs_init(), but it
goes loop over hctx_for_each_ctx(), i.e. it initializes only for
online CPUs. Thus, when a CPU is hotplugged, the ctx for the newly
onlined CPU is registered without initialization.
This patch fixes the issue by initializing the all ctx kobjects
belonging to each queue.
Bugzilla: https://bugzilla.novell.com/show_bug.cgi?id=908794
Cc: <stable@vger.kernel.org>
Signed-off-by: Takashi Iwai <tiwai@suse.de>
---
block/blk-mq-sysfs.c | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c
index 371d8800b48a..1630a20d5dcf 100644
--- a/block/blk-mq-sysfs.c
+++ b/block/blk-mq-sysfs.c
@@ -390,16 +390,15 @@ static void blk_mq_sysfs_init(struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
struct blk_mq_ctx *ctx;
- int i, j;
+ int i;
kobject_init(&q->mq_kobj, &blk_mq_ktype);
- queue_for_each_hw_ctx(q, hctx, i) {
+ queue_for_each_hw_ctx(q, hctx, i)
kobject_init(&hctx->kobj, &blk_mq_hw_ktype);
- hctx_for_each_ctx(hctx, ctx, j)
- kobject_init(&ctx->kobj, &blk_mq_ctx_ktype);
- }
+ queue_for_each_ctx(q, ctx, i)
+ kobject_init(&ctx->kobj, &blk_mq_ctx_ktype);
}
/* see blk_register_queue() */
--
2.2.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] blk-mq: Fix uninitialized kobject at CPU hotplugging
2014-12-10 15:38 [PATCH] blk-mq: Fix uninitialized kobject at CPU hotplugging Takashi Iwai
@ 2014-12-10 16:01 ` Jens Axboe
2014-12-10 16:03 ` Takashi Iwai
0 siblings, 1 reply; 4+ messages in thread
From: Jens Axboe @ 2014-12-10 16:01 UTC (permalink / raw)
To: Takashi Iwai; +Cc: linux-kernel
On 12/10/2014 08:38 AM, Takashi Iwai wrote:
> When a CPU is hotplugged, the current blk-mq spews a warning like:
>
> kobject '(null)' (ffffe8ffffc8b5d8): tried to add an uninitialized object, something is seriously wrong.
> CPU: 1 PID: 1386 Comm: systemd-udevd Not tainted 3.18.0-rc7-2.g088d59b-default #1
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014
> 0000000000000000 0000000000000002 ffffffff81605f07 ffffe8ffffc8b5d8
> ffffffff8132c7a0 ffff88023341d370 0000000000000020 ffff8800bb05bd58
> ffff8800bb05bd08 000000000000a0a0 000000003f441940 0000000000000007
> Call Trace:
> [<ffffffff81005306>] dump_trace+0x86/0x330
> [<ffffffff81005644>] show_stack_log_lvl+0x94/0x170
> [<ffffffff81006d21>] show_stack+0x21/0x50
> [<ffffffff81605f07>] dump_stack+0x41/0x51
> [<ffffffff8132c7a0>] kobject_add+0xa0/0xb0
> [<ffffffff8130aee1>] blk_mq_register_hctx+0x91/0xb0
> [<ffffffff8130b82e>] blk_mq_sysfs_register+0x3e/0x60
> [<ffffffff81309298>] blk_mq_queue_reinit_notify+0xf8/0x190
> [<ffffffff8107cfdc>] notifier_call_chain+0x4c/0x70
> [<ffffffff8105fd23>] cpu_notify+0x23/0x50
> [<ffffffff81060037>] _cpu_up+0x157/0x170
> [<ffffffff810600d9>] cpu_up+0x89/0xb0
> [<ffffffff815fa5b5>] cpu_subsys_online+0x35/0x80
> [<ffffffff814323cd>] device_online+0x5d/0xa0
> [<ffffffff81432485>] online_store+0x75/0x80
> [<ffffffff81236a5a>] kernfs_fop_write+0xda/0x150
> [<ffffffff811c5532>] vfs_write+0xb2/0x1f0
> [<ffffffff811c5f42>] SyS_write+0x42/0xb0
> [<ffffffff8160c4ed>] system_call_fastpath+0x16/0x1b
> [<00007f0132fb24e0>] 0x7f0132fb24e0
>
> This is indeed because of an uninitialized kobject for blk_mq_ctx.
> The blk_mq_ctx kobjects are initialized in blk_mq_sysfs_init(), but it
> goes loop over hctx_for_each_ctx(), i.e. it initializes only for
> online CPUs. Thus, when a CPU is hotplugged, the ctx for the newly
> onlined CPU is registered without initialization.
>
> This patch fixes the issue by initializing the all ctx kobjects
> belonging to each queue.
Thanks, this looks good. I wonder why I haven't hit this yet.
--
Jens Axboe
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] blk-mq: Fix uninitialized kobject at CPU hotplugging
2014-12-10 16:01 ` Jens Axboe
@ 2014-12-10 16:03 ` Takashi Iwai
2014-12-10 16:13 ` Jens Axboe
0 siblings, 1 reply; 4+ messages in thread
From: Takashi Iwai @ 2014-12-10 16:03 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-kernel
At Wed, 10 Dec 2014 09:01:05 -0700,
Jens Axboe wrote:
>
> On 12/10/2014 08:38 AM, Takashi Iwai wrote:
> > When a CPU is hotplugged, the current blk-mq spews a warning like:
> >
> > kobject '(null)' (ffffe8ffffc8b5d8): tried to add an uninitialized object, something is seriously wrong.
> > CPU: 1 PID: 1386 Comm: systemd-udevd Not tainted 3.18.0-rc7-2.g088d59b-default #1
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014
> > 0000000000000000 0000000000000002 ffffffff81605f07 ffffe8ffffc8b5d8
> > ffffffff8132c7a0 ffff88023341d370 0000000000000020 ffff8800bb05bd58
> > ffff8800bb05bd08 000000000000a0a0 000000003f441940 0000000000000007
> > Call Trace:
> > [<ffffffff81005306>] dump_trace+0x86/0x330
> > [<ffffffff81005644>] show_stack_log_lvl+0x94/0x170
> > [<ffffffff81006d21>] show_stack+0x21/0x50
> > [<ffffffff81605f07>] dump_stack+0x41/0x51
> > [<ffffffff8132c7a0>] kobject_add+0xa0/0xb0
> > [<ffffffff8130aee1>] blk_mq_register_hctx+0x91/0xb0
> > [<ffffffff8130b82e>] blk_mq_sysfs_register+0x3e/0x60
> > [<ffffffff81309298>] blk_mq_queue_reinit_notify+0xf8/0x190
> > [<ffffffff8107cfdc>] notifier_call_chain+0x4c/0x70
> > [<ffffffff8105fd23>] cpu_notify+0x23/0x50
> > [<ffffffff81060037>] _cpu_up+0x157/0x170
> > [<ffffffff810600d9>] cpu_up+0x89/0xb0
> > [<ffffffff815fa5b5>] cpu_subsys_online+0x35/0x80
> > [<ffffffff814323cd>] device_online+0x5d/0xa0
> > [<ffffffff81432485>] online_store+0x75/0x80
> > [<ffffffff81236a5a>] kernfs_fop_write+0xda/0x150
> > [<ffffffff811c5532>] vfs_write+0xb2/0x1f0
> > [<ffffffff811c5f42>] SyS_write+0x42/0xb0
> > [<ffffffff8160c4ed>] system_call_fastpath+0x16/0x1b
> > [<00007f0132fb24e0>] 0x7f0132fb24e0
> >
> > This is indeed because of an uninitialized kobject for blk_mq_ctx.
> > The blk_mq_ctx kobjects are initialized in blk_mq_sysfs_init(), but it
> > goes loop over hctx_for_each_ctx(), i.e. it initializes only for
> > online CPUs. Thus, when a CPU is hotplugged, the ctx for the newly
> > onlined CPU is registered without initialization.
> >
> > This patch fixes the issue by initializing the all ctx kobjects
> > belonging to each queue.
>
> Thanks, this looks good. I wonder why I haven't hit this yet.
Maybe you did test CPU hot removal before adding?
Takashi
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] blk-mq: Fix uninitialized kobject at CPU hotplugging
2014-12-10 16:03 ` Takashi Iwai
@ 2014-12-10 16:13 ` Jens Axboe
0 siblings, 0 replies; 4+ messages in thread
From: Jens Axboe @ 2014-12-10 16:13 UTC (permalink / raw)
To: Takashi Iwai; +Cc: linux-kernel
On 12/10/2014 09:03 AM, Takashi Iwai wrote:
> At Wed, 10 Dec 2014 09:01:05 -0700,
> Jens Axboe wrote:
>>
>> On 12/10/2014 08:38 AM, Takashi Iwai wrote:
>>> When a CPU is hotplugged, the current blk-mq spews a warning like:
>>>
>>> kobject '(null)' (ffffe8ffffc8b5d8): tried to add an uninitialized object, something is seriously wrong.
>>> CPU: 1 PID: 1386 Comm: systemd-udevd Not tainted 3.18.0-rc7-2.g088d59b-default #1
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_171129-lamiak 04/01/2014
>>> 0000000000000000 0000000000000002 ffffffff81605f07 ffffe8ffffc8b5d8
>>> ffffffff8132c7a0 ffff88023341d370 0000000000000020 ffff8800bb05bd58
>>> ffff8800bb05bd08 000000000000a0a0 000000003f441940 0000000000000007
>>> Call Trace:
>>> [<ffffffff81005306>] dump_trace+0x86/0x330
>>> [<ffffffff81005644>] show_stack_log_lvl+0x94/0x170
>>> [<ffffffff81006d21>] show_stack+0x21/0x50
>>> [<ffffffff81605f07>] dump_stack+0x41/0x51
>>> [<ffffffff8132c7a0>] kobject_add+0xa0/0xb0
>>> [<ffffffff8130aee1>] blk_mq_register_hctx+0x91/0xb0
>>> [<ffffffff8130b82e>] blk_mq_sysfs_register+0x3e/0x60
>>> [<ffffffff81309298>] blk_mq_queue_reinit_notify+0xf8/0x190
>>> [<ffffffff8107cfdc>] notifier_call_chain+0x4c/0x70
>>> [<ffffffff8105fd23>] cpu_notify+0x23/0x50
>>> [<ffffffff81060037>] _cpu_up+0x157/0x170
>>> [<ffffffff810600d9>] cpu_up+0x89/0xb0
>>> [<ffffffff815fa5b5>] cpu_subsys_online+0x35/0x80
>>> [<ffffffff814323cd>] device_online+0x5d/0xa0
>>> [<ffffffff81432485>] online_store+0x75/0x80
>>> [<ffffffff81236a5a>] kernfs_fop_write+0xda/0x150
>>> [<ffffffff811c5532>] vfs_write+0xb2/0x1f0
>>> [<ffffffff811c5f42>] SyS_write+0x42/0xb0
>>> [<ffffffff8160c4ed>] system_call_fastpath+0x16/0x1b
>>> [<00007f0132fb24e0>] 0x7f0132fb24e0
>>>
>>> This is indeed because of an uninitialized kobject for blk_mq_ctx.
>>> The blk_mq_ctx kobjects are initialized in blk_mq_sysfs_init(), but it
>>> goes loop over hctx_for_each_ctx(), i.e. it initializes only for
>>> online CPUs. Thus, when a CPU is hotplugged, the ctx for the newly
>>> onlined CPU is registered without initialization.
>>>
>>> This patch fixes the issue by initializing the all ctx kobjects
>>> belonging to each queue.
>>
>> Thanks, this looks good. I wonder why I haven't hit this yet.
>
> Maybe you did test CPU hot removal before adding?
I've definitely tested all cases, but changes in this area may have
happened on the side since that testing was done. In any case, the patch
makes it saner, just initializing hw and sw ctx's separately.
--
Jens Axboe
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-12-10 16:13 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-12-10 15:38 [PATCH] blk-mq: Fix uninitialized kobject at CPU hotplugging Takashi Iwai
2014-12-10 16:01 ` Jens Axboe
2014-12-10 16:03 ` Takashi Iwai
2014-12-10 16:13 ` Jens Axboe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.