kernel: smp: WARNING at kernel/smp.c:673

kernel: smp: WARNING at kernel/smp.c:673

[Sasha Levin]

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next
kernel, I've stumbled on the following spew:


[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
[ 1226.703128] Modules linked in:
[ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
[ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
[ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
[ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
[ 1226.710966] Call Trace:
[ 1226.711417] dump_stack (lib/dump_stack.c:52)
[ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
[ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
[ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
[ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
[ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
[ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
[ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
[ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
[ 1226.727263] memory_notify (drivers/base/memory.c:177)
[ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
[ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
[ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
[ 1226.735208] device_offline (drivers/base/core.c:1428)
[ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
[ 1226.737720] dev_attr_store (drivers/base/core.c:138)
[ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
[ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
[ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
[ 1226.745095] do_readv_writev (fs/read_write.c:840)
[ 1226.752769] vfs_writev (fs/read_write.c:879)
[ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
[ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)

No time to dig too much into this today, will work on it tomorrow.


Thanks,
Sasha

Re: kernel: smp: WARNING at kernel/smp.c:673

[Joonsoo Kim]

On Sun, Aug 03, 2014 at 10:57:02PM -0400, Sasha Levin wrote:
> Hi all,
> 
> While fuzzing with trinity inside a KVM tools guest running the latest -next
> kernel, I've stumbled on the following spew:
> 
> 
> [ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
> [ 1226.703128] Modules linked in:
> [ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
> [ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
> [ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
> [ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
> [ 1226.710966] Call Trace:
> [ 1226.711417] dump_stack (lib/dump_stack.c:52)
> [ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
> [ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
> [ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
> [ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
> [ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
> [ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
> [ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
> [ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
> [ 1226.727263] memory_notify (drivers/base/memory.c:177)
> [ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
> [ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
> [ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
> [ 1226.735208] device_offline (drivers/base/core.c:1428)
> [ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
> [ 1226.737720] dev_attr_store (drivers/base/core.c:138)
> [ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
> [ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
> [ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
> [ 1226.745095] do_readv_writev (fs/read_write.c:840)
> [ 1226.752769] vfs_writev (fs/read_write.c:879)
> [ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
> [ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
> 
> No time to dig too much into this today, will work on it tomorrow.

I think that that WARN_ON_ONCE() in kernel/smp.c:673 is wrong.
smp_call_function_single() returns 0 if success so it should be
WARN_ON_ONCE(ret) rather than WARN_ON_ONCE(!ret).

Thanks.

Re: kernel: smp: WARNING at kernel/smp.c:673

[Gilad Ben-Yossef]

Hi Sasha and Kim,

I believe Kim is right and the WARN_ON_ONCE condition should be negated.
Furthermore, I'm  pretty sure I'm the one to blame for this bug.

Good catch and thank you both.

(Un)fortunately, I'm in a vacation in an island resort in Thailand at
the moment and the beer is much better than the Internet connection.

I'll do my best to prepare, test and send a patch, but if someone
beats me to it before my return in 10 days I will be happy to review.

Many thanks,
Gilad



On Aug 4, 2014 1:41 PM, "Joonsoo Kim" <iamjoonsoo.kim@lge.com> wrote:
>
> On Sun, Aug 03, 2014 at 10:57:02PM -0400, Sasha Levin wrote:
> > Hi all,
> >
> > While fuzzing with trinity inside a KVM tools guest running the latest -next
> > kernel, I've stumbled on the following spew:
> >
> >
> > [ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
> > [ 1226.703128] Modules linked in:
> > [ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
> > [ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
> > [ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
> > [ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
> > [ 1226.710966] Call Trace:
> > [ 1226.711417] dump_stack (lib/dump_stack.c:52)
> > [ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
> > [ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
> > [ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
> > [ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
> > [ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
> > [ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
> > [ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
> > [ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
> > [ 1226.727263] memory_notify (drivers/base/memory.c:177)
> > [ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
> > [ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
> > [ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
> > [ 1226.735208] device_offline (drivers/base/core.c:1428)
> > [ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
> > [ 1226.737720] dev_attr_store (drivers/base/core.c:138)
> > [ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
> > [ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
> > [ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
> > [ 1226.745095] do_readv_writev (fs/read_write.c:840)
> > [ 1226.752769] vfs_writev (fs/read_write.c:879)
> > [ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
> > [ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
> >
> > No time to dig too much into this today, will work on it tomorrow.
>
> I think that that WARN_ON_ONCE() in kernel/smp.c:673 is wrong.
> smp_call_function_single() returns 0 if success so it should be
> WARN_ON_ONCE(ret) rather than WARN_ON_ONCE(!ret).
>
> Thanks.

Re: kernel: smp: WARNING at kernel/smp.c:673

[Christoph Lameter]

On Sun, 3 Aug 2014, Sasha Levin wrote:

> While fuzzing with trinity inside a KVM tools guest running the latest -next
> kernel, I've stumbled on the following spew:

Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from
the cpu notifier?

TJ?

Re: kernel: smp: WARNING at kernel/smp.c:673

[Tejun Heo]

On Mon, Aug 04, 2014 at 10:00:39AM -0500, Christoph Lameter wrote:
> On Sun, 3 Aug 2014, Sasha Levin wrote:
> 
> > While fuzzing with trinity inside a KVM tools guest running the latest -next
> > kernel, I've stumbled on the following spew:
> 
> Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from
> the cpu notifier?

Hah?  I don't see anything cpuset related there.

-- 
tejun

Re: kernel: smp: WARNING at kernel/smp.c:673

[Christoph Lameter]

On Mon, 4 Aug 2014, Tejun Heo wrote:

> On Mon, Aug 04, 2014 at 10:00:39AM -0500, Christoph Lameter wrote:
> > On Sun, 3 Aug 2014, Sasha Levin wrote:
> >
> > > While fuzzing with trinity inside a KVM tools guest running the latest -next
> > > kernel, I've stumbled on the following spew:
> >
> > Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from
> > the cpu notifier?
>
> Hah?  I don't see anything cpuset related there.

Yuck. I meant cgroup code. There were some patches that were suggesting a
shrink operation when taking down a cgroup. Does that also apply to per
cpu downing?

Re: kernel: smp: WARNING at kernel/smp.c:673

[Tejun Heo]

On Mon, Aug 04, 2014 at 10:21:47AM -0500, Christoph Lameter wrote:
> On Mon, 4 Aug 2014, Tejun Heo wrote:
> 
> > On Mon, Aug 04, 2014 at 10:00:39AM -0500, Christoph Lameter wrote:
> > > On Sun, 3 Aug 2014, Sasha Levin wrote:
> > >
> > > > While fuzzing with trinity inside a KVM tools guest running the latest -next
> > > > kernel, I've stumbled on the following spew:
> > >
> > > Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from
> > > the cpu notifier?
> >
> > Hah?  I don't see anything cpuset related there.
> 
> Yuck. I meant cgroup code. There were some patches that were suggesting a
> shrink operation when taking down a cgroup. Does that also apply to per
> cpu downing?

Heh, I have no idea what you're talking about.  cgroup currently
doesn't do anything like that.  It doesn't even make use of the cpu
hotplug hooks.

Thanks.

-- 
tejun

Re: kernel: smp: WARNING at kernel/smp.c:673

[Christoph Lameter]

On Mon, 4 Aug 2014, Tejun Heo wrote:

> > Yuck. I meant cgroup code. There were some patches that were suggesting a
> > shrink operation when taking down a cgroup. Does that also apply to per
> > cpu downing?
>
> Heh, I have no idea what you're talking about.  cgroup currently
> doesn't do anything like that.  It doesn't even make use of the cpu
> hotplug hooks.

Something is running a kmem_cache_shrink from cpu down hooks. Thats weird.

Re: kernel: smp: WARNING at kernel/smp.c:673

[Sasha Levin]

On 08/04/2014 12:00 PM, Christoph Lameter wrote:
> On Mon, 4 Aug 2014, Tejun Heo wrote:
> 
>>> Yuck. I meant cgroup code. There were some patches that were suggesting a
>>> shrink operation when taking down a cgroup. Does that also apply to per
>>> cpu downing?
>>
>> Heh, I have no idea what you're talking about.  cgroup currently
>> doesn't do anything like that.  It doesn't even make use of the cpu
>> hotplug hooks.
> 
> Something is running a kmem_cache_shrink from cpu down hooks. Thats weird.
> 

That's not cpu down, that's mem down.


Thanks,
Sasha

Re: kernel: smp: WARNING at kernel/smp.c:673

[Andrew Morton]

On Mon, 4 Aug 2014 15:48:23 +0900 Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:

> > While fuzzing with trinity inside a KVM tools guest running the latest -next
> > kernel, I've stumbled on the following spew:
> > 
> > 
> > [ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
> > [ 1226.703128] Modules linked in:
> > [ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
> > [ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
> > [ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
> > [ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
> > [ 1226.710966] Call Trace:
> > [ 1226.711417] dump_stack (lib/dump_stack.c:52)
> > [ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
> > [ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
> > [ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
> > [ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
> > [ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
> > [ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
> > [ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
> > [ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
> > [ 1226.727263] memory_notify (drivers/base/memory.c:177)
> > [ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
> > [ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
> > [ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
> > [ 1226.735208] device_offline (drivers/base/core.c:1428)
> > [ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
> > [ 1226.737720] dev_attr_store (drivers/base/core.c:138)
> > [ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
> > [ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
> > [ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
> > [ 1226.745095] do_readv_writev (fs/read_write.c:840)
> > [ 1226.752769] vfs_writev (fs/read_write.c:879)
> > [ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
> > [ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
> > 
> > No time to dig too much into this today, will work on it tomorrow.
> 
> I think that that WARN_ON_ONCE() in kernel/smp.c:673 is wrong.
> smp_call_function_single() returns 0 if success so it should be
> WARN_ON_ONCE(ret) rather than WARN_ON_ONCE(!ret).

Surely true.


From: Andrew Morton <akpm@linux-foundation.org>
Subject: kernel/smp.c:on_each_cpu_cond(): fix warning in fallback path

The rarely-executed memry-allocation-failed callback path generates a
WARN_ON_ONCE() when smp_call_function_single() succeeds.  Presumably it's
supposed to warn on failures.

Reported-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Gilad Ben-Yossef <gilad@benyossef.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Tejun Heo <htejun@gmail.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 kernel/smp.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff -puN kernel/smp.c~a kernel/smp.c
--- a/kernel/smp.c~a
+++ a/kernel/smp.c
@@ -661,7 +661,7 @@ void on_each_cpu_cond(bool (*cond_func)(
 			if (cond_func(cpu, info)) {
 				ret = smp_call_function_single(cpu, func,
 								info, wait);
-				WARN_ON_ONCE(!ret);
+				WARN_ON_ONCE(ret);
 			}
 		preempt_enable();
 	}
_