[PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Andreas Herrmann]

When I've used slub_debug kernel option (e.g.
"slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
seen a panic like:

  Highbank #setenv bootargs console=ttyAMA0 root=/dev/sda2 kgdboc.kgdboc=ttyAMA0,115200 slub_debug=,kmalloc-4096 earlyprintk=ttyAMA0
  ...
  Unable to handle kernel NULL pointer dereference at virtual address 00000000
  pgd = c0004000
  [00000000] *pgd=00000000
  Internal error: Oops: 5 [#1] SMP ARM
  Modules linked in:
  CPU: 0 PID: 0 Comm: swapper Tainted: G        W    3.12.0-00048-gbe408cd #314
  task: c0898360 ti: c088a000 task.ti: c088a000
  PC is at strncmp+0x1c/0x84
  LR is at kmem_cache_flags.isra.46.part.47+0x44/0x60
  pc : [<c02c6da0>]    lr : [<c0110a3c>]    psr: 200001d3
  sp : c088bea8  ip : c088beb8  fp : c088beb4
  r10: 00000000  r9 : 413fc090  r8 : 00000001
  r7 : 00000000  r6 : c2984a08  r5 : c0966e78  r4 : 00000000
  r3 : 0000006b  r2 : 0000000c  r1 : 00000000  r0 : c2984a08
  Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel
  Control: 10c5387d  Table: 0000404a  DAC: 00000015
  Process swapper (pid: 0, stack limit = 0xc088a248)
  Stack: (0xc088bea8 to 0xc088c000)
  bea0:                   c088bed4 c088beb8 c0110a3c c02c6d90 c0966e78 00000040
  bec0: ef001f00 00000040 c088bf14 c088bed8 c0112070 c0110a04 00000005 c010fac8
  bee0: c088bf5c c088bef0 c010fac8 ef001f00 00000040 00000000 00000040 00000001
  bf00: 413fc090 00000000 c088bf34 c088bf18 c0839190 c0112040 00000000 ef001f00
  bf20: 00000000 00000000 c088bf54 c088bf38 c0839200 c083914c 00000006 c0961c4c
  bf40: c0961c28 00000000 c088bf7c c088bf58 c08392ac c08391c0 c08a2ed8 c0966e78
  bf60: c086b874 c08a3f50 c0961c28 00000001 c088bfb4 c088bf80 c083b258 c0839248
  bf80: 2f800000 0f000000 c08935b4 ffffffff c08cd400 ffffffff c08cd400 c0868408
  bfa0: c29849c0 00000000 c088bff4 c088bfb8 c0824974 c083b1e4 ffffffff ffffffff
  bfc0: c08245c0 00000000 00000000 c0868408 00000000 10c5387d c0892bcc c0868404
  bfe0: c0899440 0000406a 00000000 c088bff8 00008074 c0824824 00000000 00000000
  [<c02c6da0>] (strncmp+0x1c/0x84) from [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60)
  [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60) from [<c0112070>] (__kmem_cache_create+0x3c/0x410)
  [<c0112070>] (__kmem_cache_create+0x3c/0x410) from [<c0839190>] (create_boot_cache+0x50/0x74)
  [<c0839190>] (create_boot_cache+0x50/0x74) from [<c0839200>] (create_kmalloc_cache+0x4c/0x88)
  [<c0839200>] (create_kmalloc_cache+0x4c/0x88) from [<c08392ac>] (create_kmalloc_caches+0x70/0x114)
  [<c08392ac>] (create_kmalloc_caches+0x70/0x114) from [<c083b258>] (kmem_cache_init+0x80/0xe0)
  [<c083b258>] (kmem_cache_init+0x80/0xe0) from [<c0824974>] (start_kernel+0x15c/0x318)
  [<c0824974>] (start_kernel+0x15c/0x318) from [<00008074>] (0x8074)
  Code: e3520000 01a00002 089da800 e5d03000 (e5d1c000)
  ---[ end trace 1b75b31a2719ed1d ]---
  Kernel panic - not syncing: Fatal exception

Problem is that slub_debug option is not parsed before
create_boot_cache is called. Solve this by changing slub_debug to
early_param.

Kernels 3.11, 3.10 are also affected.
I am not sure about older kernels.

Cc: stable@vger.kernel.org    #3.10 and 3.11
Signed-off-by: Andreas Herrmann <andreas.herrmann@calxeda.com>
---
 mm/slub.c |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

Patch is against v3.12-48-gbe408cd.

Regards,
Andreas

diff --git a/mm/slub.c b/mm/slub.c
index c3eb3d3..596c58a 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1205,10 +1205,9 @@ check_slabs:
 	if (*str == ',')
 		slub_debug_slabs = str + 1;
 out:
-	return 1;
+	return 0;
 }
-
-__setup("slub_debug", setup_slub_debug);
+early_param("slub_debug", setup_slub_debug);
 
 static unsigned long kmem_cache_flags(unsigned long object_size,
 	unsigned long flags, const char *name,
-- 
1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Christoph Lameter]

On Wed, 6 Nov 2013, Andreas Herrmann wrote:

> When I've used slub_debug kernel option (e.g.
> "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
> seen a panic like:

Hmmm.. That looks like its due to some slabs not having names
during early boot. kmem_cache_flags is called with NULL as a parameter.

Are you sure that this fixes the issue? Looks like the
kmem_cache_flag function should fail regardless of how early you set it.

AFAICT the right fix would be:


Subject: slub: Handle NULL parameter in kmem_cache_flags

kmem_cache_flags may be called with NULL parameter during early boot.
Skip the test in that case.

Signed-off-by: Christoph Lameter <cl@linux.com>

Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c	2013-10-15 13:55:44.000000000 -0500
+++ linux/mm/slub.c	2013-11-06 13:09:21.810583134 -0600
@@ -1217,7 +1217,7 @@ static unsigned long kmem_cache_flags(un
 	/*
 	 * Enable debugging if selected on the kernel commandline.
 	 */
-	if (slub_debug && (!slub_debug_slabs ||
+	if (slub_debug && name && (!slub_debug_slabs ||
 		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs))))
 		flags |= slub_debug;



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Andreas Herrmann]

On Wed, Nov 06, 2013 at 02:16:33PM -0500, Christoph Lameter wrote:
> On Wed, 6 Nov 2013, Andreas Herrmann wrote:
> 
> > When I've used slub_debug kernel option (e.g.
> > "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
> > seen a panic like:
> 
> Hmmm.. That looks like its due to some slabs not having names
> during early boot. kmem_cache_flags is called with NULL as a parameter.
> 
> Are you sure that this fixes the issue?

?

Sure, of course I've tested my patch as I required slub debugging.
Panic was gone and debugging of the selected slab enabled.


Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Andreas Herrmann]

On Wed, Nov 06, 2013 at 02:16:33PM -0500, Christoph Lameter wrote:
> On Wed, 6 Nov 2013, Andreas Herrmann wrote:
> 
> > When I've used slub_debug kernel option (e.g.
> > "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
> > seen a panic like:
> 
> Hmmm.. That looks like its due to some slabs not having names
> during early boot. kmem_cache_flags is called with NULL as a parameter.

That's because the slub_debug parameter is not evaluated before
kmem_cache_flags is called.

Older kernels didn't show this problem. I think the sequence of those
calls has changed. Not sure what patch set has made that change.

> Are you sure that this fixes the issue? Looks like the
> kmem_cache_flag function should fail regardless of how early you set it.
> 
> AFAICT the right fix would be:

That would avoid the panic but I guess it won't enable slub debugging.

However I'll test this.


Andreas

> Subject: slub: Handle NULL parameter in kmem_cache_flags
> 
> kmem_cache_flags may be called with NULL parameter during early boot.
> Skip the test in that case.
> 
> Signed-off-by: Christoph Lameter <cl@linux.com>
> 
> Index: linux/mm/slub.c
> ===================================================================
> --- linux.orig/mm/slub.c	2013-10-15 13:55:44.000000000 -0500
> +++ linux/mm/slub.c	2013-11-06 13:09:21.810583134 -0600
> @@ -1217,7 +1217,7 @@ static unsigned long kmem_cache_flags(un
>  	/*
>  	 * Enable debugging if selected on the kernel commandline.
>  	 */
> -	if (slub_debug && (!slub_debug_slabs ||
> +	if (slub_debug && name && (!slub_debug_slabs ||
>  		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs))))
>  		flags |= slub_debug;
> 
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Andreas Herrmann]

On Wed, Nov 06, 2013 at 08:54:17PM +0100, Andreas Herrmann wrote:
> On Wed, Nov 06, 2013 at 02:16:33PM -0500, Christoph Lameter wrote:
> > On Wed, 6 Nov 2013, Andreas Herrmann wrote:
> > 
> > > When I've used slub_debug kernel option (e.g.
> > > "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
> > > seen a panic like:
> > 
> > Hmmm.. That looks like its due to some slabs not having names
> > during early boot. kmem_cache_flags is called with NULL as a parameter.
> 
> That's because the slub_debug parameter is not evaluated before
> kmem_cache_flags is called.
> 
> Older kernels didn't show this problem. I think the sequence of those
> calls has changed. Not sure what patch set has made that change.

Please ignore this comment.
I revisisted the code and of course you are right.
Hmm, now wondering why my patch covered the panic.

> > Are you sure that this fixes the issue? Looks like the
> > kmem_cache_flag function should fail regardless of how early you set it.
> > 
> > AFAICT the right fix would be:
> 
> That would avoid the panic but I guess it won't enable slub debugging.

Ok, the latter is bogus. Of course debugging is switched on.

> However I'll test this.

Your patch fixed it.


Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Andreas Herrmann]

On Wed, Nov 06, 2013 at 09:34:29PM +0100, Andreas Herrmann wrote:
> On Wed, Nov 06, 2013 at 08:54:17PM +0100, Andreas Herrmann wrote:
> > On Wed, Nov 06, 2013 at 02:16:33PM -0500, Christoph Lameter wrote:
> > > On Wed, 6 Nov 2013, Andreas Herrmann wrote:
> > > 
> > > > When I've used slub_debug kernel option (e.g.
> > > > "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
> > > > seen a panic like:
> > > 
> > > Hmmm.. That looks like its due to some slabs not having names
> > > during early boot. kmem_cache_flags is called with NULL as a parameter.
> > 
> > That's because the slub_debug parameter is not evaluated before
> > kmem_cache_flags is called.
> > 
> > Older kernels didn't show this problem. I think the sequence of those
> > calls has changed. Not sure what patch set has made that change.
> 
> Please ignore this comment.
> I revisisted the code and of course you are right.
> Hmm, now wondering why my patch covered the panic.

Arrgh, my patch changed slub_debug_slabs to be NULL.
That is why the panic didn't happen.

Would be nice, if your patch is pushed upstream asap.


Thanks,

Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Christoph Lameter]

On Wed, 6 Nov 2013, Andreas Herrmann wrote:

> Would be nice, if your patch is pushed upstream asap.

Ok so this is a

Tested-by: Andreas Herrmann <andreas.herrmann@calxeda.com>

I think?

BTW Calxeda is a great product. Hope you get 64 bit running soon.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Andreas Herrmann]

On Wed, Nov 06, 2013 at 04:38:10PM -0500, Christoph Lameter wrote:
> On Wed, 6 Nov 2013, Andreas Herrmann wrote:
> 
> > Would be nice, if your patch is pushed upstream asap.
> 
> Ok so this is a
> 
> Tested-by: Andreas Herrmann <andreas.herrmann@calxeda.com>
> 
> I think?

Yes.

> BTW Calxeda is a great product. Hope you get 64 bit running soon.


Thanks,

Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Andreas Herrmann]

On Thu, Nov 07, 2013 at 09:27:32AM +0100, Andreas Herrmann wrote:
> On Wed, Nov 06, 2013 at 04:38:10PM -0500, Christoph Lameter wrote:
> > On Wed, 6 Nov 2013, Andreas Herrmann wrote:
> > 
> > > Would be nice, if your patch is pushed upstream asap.
> > 
> > Ok so this is a
> > 
> > Tested-by: Andreas Herrmann <andreas.herrmann@calxeda.com>
> > 
> > I think?
> 
> Yes.

And for sake of completeness. Here is some debug output with a kernel
that had your "slub: Handle NULL parameter in kmem_cache_flags" patch
applied. And of course there were a couple of unnamed slabs:

  ...
         .bss : 0xc089fd80 - 0xc094cc4c   ( 692 kB)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc90c): kmem_cache_node
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc91c): kmem_cache
a?? slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
  ...

The third one is wheree the panic happened w/o the fix.


Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Christoph Lameter]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1568 bytes --]

On Thu, 7 Nov 2013, Andreas Herrmann wrote:

> And for sake of completeness. Here is some debug output with a kernel
> that had your "slub: Handle NULL parameter in kmem_cache_flags" patch
> applied. And of course there were a couple of unnamed slabs:
>
>   ...
>          .bss : 0xc089fd80 - 0xc094cc4c   ( 692 kB)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc90c): kmem_cache_node
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc91c): kmem_cache
> a?? slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
>   SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
>   ...

Well yes on bootstrap the slabs are initially created without a name.
Later the name is provided. Before the introduction of the common kmalloc
code the kmalloc caches had a generic name instead of NULL. I was not sure
that this was a valid thing to do so I put NULL in there so that we can
catch the uses.

Re: [PATCH] mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

[Christoph Lameter]

There is still something not optimal with the patch since it would disable
debuggin on the kmalloc stack. Ccheck name for NULL only if
slub_debug_slabs.


Subject: slub: Handle NULL parameter in kmem_cache_flags V2

V1->V2
 - flags need to be applied regardless if !slub_debug_slabs

kmem_cache_flags may be called with NULL parameter during early boot.
Skip the test in that case.

Signed-off-by: Christoph Lameter <cl@linux.com>

Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c	2013-11-07 10:25:49.669706521 -0600
+++ linux/mm/slub.c	2013-11-07 10:27:49.130394661 -0600
@@ -1217,8 +1217,8 @@ static unsigned long kmem_cache_flags(un
 	/*
 	 * Enable debugging if selected on the kernel commandline.
 	 */
-	if (slub_debug && (!slub_debug_slabs ||
-		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs))))
+	if (slub_debug && (!slub_debug_slabs || (name &&
+		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs)))))
 		flags |= slub_debug;

 	return flags;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>