Slab BUG with DEBUG_* options

Slab BUG with DEBUG_* options

[Meelis Roos]

I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM 
and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most 
other debug options) and got the following BUG and hang on startup. This 
happened originally with 3.11-rc2-00058 where my bisection of 
another problem lead, but I retested 3.12 to have the same BUG in the 
same place.

kernel BUG at mm/slab.c:2391!
              \|/ ____ \|/
              "@'/ .. \`@"
              /_| \__/ |_\
                 \__U_/
swapper(0): Kernel bad sw trap 5 [#1]
CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty #127
task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y: 00000000    Not tainted
TPC: <__kmem_cache_create+0x374/0x480>
g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3: 00000000008b5120
g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7: 0000000000000000
o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3: 0000000000000000
o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc: 00000000004f57cc
RPC: <__kmem_cache_create+0x36c/0x480>
l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3: fffff8001e819fd8
l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7: fffff8001e8300a0
i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3: 0000000000000034
i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7: 0000000000907464
I7: <create_boot_cache+0x4c/0x84>
Call Trace:
 [0000000000907464] create_boot_cache+0x4c/0x84
 [00000000009074d0] create_kmalloc_cache+0x34/0x60
 [0000000000907540] create_kmalloc_caches+0x44/0x168
 [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
 [00000000008fc658] start_kernel+0x18c/0x370
 [0000000000761df4] tlb_fixup_done+0x88/0x94
 [0000000000000000]           (null)
Disabling lock debugging due to kernel taint
Caller[0000000000907464]: create_boot_cache+0x4c/0x84
Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
Caller[00000000008fc658]: start_kernel+0x18c/0x370
Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
Caller[0000000000000000]:           (null)
Instruction DUMP: 92102957  7ffccb35  90122308 <91d02005> 90100018  4009b371  920f20d0  ba922000  02480006 
Kernel panic - not syncing: Attempted to kill the idle task!
Press Stop-A (L1-A) to return to the boot prom

The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().

I instrumented the code and found the following:

__kmem_cache_create: starting, size=248, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 248 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=96, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 96 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=192, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 192 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=32, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 32 because of redzoning
__kmem_cache_create: aligned size to 32
__kmem_cache_create: num=226, slab_size=960
__kmem_cache_create: starting, size=64, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 64 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52

With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation 
with this size fails. I do not know slab internals so I can not tell if 
this just happens because of the debug paths, or is it a real problem 
without the debug options too.

-- 
Meelis Roos (mroos@linux.ee)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[David Rientjes]

On Sat, 30 Nov 2013, Meelis Roos wrote:

> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM 
> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most 
> other debug options) and got the following BUG and hang on startup. This 
> happened originally with 3.11-rc2-00058 where my bisection of 
> another problem lead, but I retested 3.12 to have the same BUG in the 
> same place.
> 
> kernel BUG at mm/slab.c:2391!
>               \|/ ____ \|/
>               "@'/ .. \`@"
>               /_| \__/ |_\
>                  \__U_/
> swapper(0): Kernel bad sw trap 5 [#1]
> CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty #127
> task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
> TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y: 00000000    Not tainted
> TPC: <__kmem_cache_create+0x374/0x480>
> g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3: 00000000008b5120
> g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7: 0000000000000000
> o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3: 0000000000000000
> o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc: 00000000004f57cc
> RPC: <__kmem_cache_create+0x36c/0x480>
> l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3: fffff8001e819fd8
> l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7: fffff8001e8300a0
> i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3: 0000000000000034
> i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7: 0000000000907464
> I7: <create_boot_cache+0x4c/0x84>
> Call Trace:
>  [0000000000907464] create_boot_cache+0x4c/0x84
>  [00000000009074d0] create_kmalloc_cache+0x34/0x60
>  [0000000000907540] create_kmalloc_caches+0x44/0x168
>  [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
>  [00000000008fc658] start_kernel+0x18c/0x370
>  [0000000000761df4] tlb_fixup_done+0x88/0x94
>  [0000000000000000]           (null)
> Disabling lock debugging due to kernel taint
> Caller[0000000000907464]: create_boot_cache+0x4c/0x84
> Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
> Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
> Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
> Caller[00000000008fc658]: start_kernel+0x18c/0x370
> Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
> Caller[0000000000000000]:           (null)
> Instruction DUMP: 92102957  7ffccb35  90122308 <91d02005> 90100018  4009b371  920f20d0  ba922000  02480006 
> Kernel panic - not syncing: Attempted to kill the idle task!
> Press Stop-A (L1-A) to return to the boot prom
> 
> The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
> 
> I instrumented the code and found the following:
> 
> __kmem_cache_create: starting, size=248, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 248 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=96, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 96 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=192, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 192 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=32, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 32 because of redzoning
> __kmem_cache_create: aligned size to 32
> __kmem_cache_create: num=226, slab_size=960
> __kmem_cache_create: starting, size=64, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 64 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
> __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
> 
> With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation 
> with this size fails. I do not know slab internals so I can not tell if 
> this just happens because of the debug paths, or is it a real problem 
> without the debug options too.
> 

Sounds like a problem with create_kmalloc_caches(), what are the values 
for KMALLOC_SHIFT_LOW and KMALLOC_SHIFT_HIGH?

It's interesting you report this as starting with 3.11-rc2 because this 
changed in 3.9, what kernels were tested before 3.11-rc2 if any?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Christoph Lameter]

Does this patch from 3.13 fix it?

commit c6f58d9b362b45c52afebe4342c9137d0dabe47f
Author: Christoph Lameter <cl@linux.com>
Date:   Thu Nov 7 16:29:15 2013 +0000

    slub: Handle NULL parameter in kmem_cache_flags

    Andreas Herrmann writes:

      When I've used slub_debug kernel option (e.g.
      "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've
      seen a panic like:

        Highbank #setenv bootargs console=ttyAMA0 root=/dev/sda2 kgdboc.kgdboc=ttyAMA0,115200 slub_debug=,kmalloc-4096 earlypr
        ...
        Unable to handle kernel NULL pointer dereference at virtual address 00000000
        pgd = c0004000
        [00000000] *pgd=00000000
        Internal error: Oops: 5 [#1] SMP ARM
        Modules linked in:
        CPU: 0 PID: 0 Comm: swapper Tainted: G        W    3.12.0-00048-gbe408cd #314
        task: c0898360 ti: c088a000 task.ti: c088a000
        PC is at strncmp+0x1c/0x84
        LR is at kmem_cache_flags.isra.46.part.47+0x44/0x60
        pc : [<c02c6da0>]    lr : [<c0110a3c>]    psr: 200001d3
        sp : c088bea8  ip : c088beb8  fp : c088beb4
        r10: 00000000  r9 : 413fc090  r8 : 00000001
        r7 : 00000000  r6 : c2984a08  r5 : c0966e78  r4 : 00000000
        r3 : 0000006b  r2 : 0000000c  r1 : 00000000  r0 : c2984a08
        Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel
        Control: 10c5387d  Table: 0000404a  DAC: 00000015
        Process swapper (pid: 0, stack limit = 0xc088a248)
        Stack: (0xc088bea8 to 0xc088c000)
        bea0:                   c088bed4 c088beb8 c0110a3c c02c6d90 c0966e78 00000040
        bec0: ef001f00 00000040 c088bf14 c088bed8 c0112070 c0110a04 00000005 c010fac8
        bee0: c088bf5c c088bef0 c010fac8 ef001f00 00000040 00000000 00000040 00000001
        bf00: 413fc090 00000000 c088bf34 c088bf18 c0839190 c0112040 00000000 ef001f00
        bf20: 00000000 00000000 c088bf54 c088bf38 c0839200 c083914c 00000006 c0961c4c
        bf40: c0961c28 00000000 c088bf7c c088bf58 c08392ac c08391c0 c08a2ed8 c0966e78
        bf60: c086b874 c08a3f50 c0961c28 00000001 c088bfb4 c088bf80 c083b258 c0839248
        bf80: 2f800000 0f000000 c08935b4 ffffffff c08cd400 ffffffff c08cd400 c0868408
        bfa0: c29849c0 00000000 c088bff4 c088bfb8 c0824974 c083b1e4 ffffffff ffffffff
        bfc0: c08245c0 00000000 00000000 c0868408 00000000 10c5387d c0892bcc c0868404
        bfe0: c0899440 0000406a 00000000 c088bff8 00008074 c0824824 00000000 00000000
        [<c02c6da0>] (strncmp+0x1c/0x84) from [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60)
        [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60) from [<c0112070>] (__kmem_cache_create+0x3c/0x410)
        [<c0112070>] (__kmem_cache_create+0x3c/0x410) from [<c0839190>] (create_boot_cache+0x50/0x74)
        [<c0839190>] (create_boot_cache+0x50/0x74) from [<c0839200>] (create_kmalloc_cache+0x4c/0x88)
        [<c0839200>] (create_kmalloc_cache+0x4c/0x88) from [<c08392ac>] (create_kmalloc_caches+0x70/0x114)
        [<c08392ac>] (create_kmalloc_caches+0x70/0x114) from [<c083b258>] (kmem_cache_init+0x80/0xe0)
        [<c083b258>] (kmem_cache_init+0x80/0xe0) from [<c0824974>] (start_kernel+0x15c/0x318)
        [<c0824974>] (start_kernel+0x15c/0x318) from [<00008074>] (0x8074)
        Code: e3520000 01a00002 089da800 e5d03000 (e5d1c000)
        ---[ end trace 1b75b31a2719ed1d ]---
        Kernel panic - not syncing: Fatal exception

      Problem is that slub_debug option is not parsed before
      create_boot_cache is called. Solve this by changing slub_debug to
      early_param.

      Kernels 3.11, 3.10 are also affected.  I am not sure about older
      kernels.

    Christoph Lameter explains:

      kmem_cache_flags may be called with NULL parameter during early boot.
      Skip the test in that case.

    Cc: stable@vger.kernel.org # 3.10 and 3.11
    Reported-by: Andreas Herrmann <andreas.herrmann@calxeda.com>
    Signed-off-by: Christoph Lameter <cl@linux.com>
    Signed-off-by: Pekka Enberg <penberg@kernel.org>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Meelis Roos]

> Does this patch from 3.13 fix it?
> 
> commit c6f58d9b362b45c52afebe4342c9137d0dabe47f
> Author: Christoph Lameter <cl@linux.com>
> Date:   Thu Nov 7 16:29:15 2013 +0000
> 
>     slub: Handle NULL parameter in kmem_cache_flags

I do not think so - it is for slub but this machine uses slab.

-- 
Meelis Roos (mroos@linux.ee)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Meelis Roos]

> > I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM 
> > and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most 
> > other debug options) and got the following BUG and hang on startup. This 
> > happened originally with 3.11-rc2-00058 where my bisection of 
> > another problem lead, but I retested 3.12 to have the same BUG in the 
> > same place.
> > 
> > kernel BUG at mm/slab.c:2391!
[...

> > The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
> > 
> > I instrumented the code and found the following:
> > 
> > __kmem_cache_create: starting, size=248, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 248 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: starting, size=96, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 96 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: starting, size=192, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 192 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: starting, size=32, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 32 because of redzoning
> > __kmem_cache_create: aligned size to 32
> > __kmem_cache_create: num=226, slab_size=960
> > __kmem_cache_create: starting, size=64, flags=8192
> > __kmem_cache_create: now flags=76800
> > __kmem_cache_create: aligned size to 64 because of redzoning
> > __kmem_cache_create: pagealloc debug, setting size to 8192
> > __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
> > __kmem_cache_create: aligned size to 8192
> > __kmem_cache_create: num=1, slab_size=64
> > __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
> > __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
> > 
> > With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation 
> > with this size fails. I do not know slab internals so I can not tell if 
> > this just happens because of the debug paths, or is it a real problem 
> > without the debug options too.
> > 
> 
> Sounds like a problem with create_kmalloc_caches(), what are the values 
> for KMALLOC_SHIFT_LOW and KMALLOC_SHIFT_HIGH?

KMALLOC_SHIFT_LOW=5, KMALLOC_SHIFT_HIGH=23
 
> It's interesting you report this as starting with 3.11-rc2 because this 
> changed in 3.9, what kernels were tested before 3.11-rc2 if any?

No other kernels were tested with slab debug - but all relsese kernels 
and most rc-s starting with rc2 or rc3 eacg time were tested without 
slab debug.

-- 
Meelis Roos (mroos@linux.ee)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Pekka Enberg]

On 11/30/2013 01:42 PM, Meelis Roos wrote:
> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> other debug options) and got the following BUG and hang on startup. This
> happened originally with 3.11-rc2-00058 where my bisection of
> another problem lead, but I retested 3.12 to have the same BUG in the
> same place.
>
> kernel BUG at mm/slab.c:2391!
>                \|/ ____ \|/
>                "@'/ .. \`@"
>                /_| \__/ |_\
>                   \__U_/
> swapper(0): Kernel bad sw trap 5 [#1]
> CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty #127
> task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
> TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y: 00000000    Not tainted
> TPC: <__kmem_cache_create+0x374/0x480>
> g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3: 00000000008b5120
> g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7: 0000000000000000
> o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3: 0000000000000000
> o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc: 00000000004f57cc
> RPC: <__kmem_cache_create+0x36c/0x480>
> l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3: fffff8001e819fd8
> l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7: fffff8001e8300a0
> i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3: 0000000000000034
> i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7: 0000000000907464
> I7: <create_boot_cache+0x4c/0x84>
> Call Trace:
>   [0000000000907464] create_boot_cache+0x4c/0x84
>   [00000000009074d0] create_kmalloc_cache+0x34/0x60
>   [0000000000907540] create_kmalloc_caches+0x44/0x168
>   [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
>   [00000000008fc658] start_kernel+0x18c/0x370
>   [0000000000761df4] tlb_fixup_done+0x88/0x94
>   [0000000000000000]           (null)
> Disabling lock debugging due to kernel taint
> Caller[0000000000907464]: create_boot_cache+0x4c/0x84
> Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
> Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
> Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
> Caller[00000000008fc658]: start_kernel+0x18c/0x370
> Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
> Caller[0000000000000000]:           (null)
> Instruction DUMP: 92102957  7ffccb35  90122308 <91d02005> 90100018  4009b371  920f20d0  ba922000  02480006
> Kernel panic - not syncing: Attempted to kill the idle task!
> Press Stop-A (L1-A) to return to the boot prom
>
> The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
>
> I instrumented the code and found the following:
>
> __kmem_cache_create: starting, size=248, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 248 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=96, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 96 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=192, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 192 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=32, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 32 because of redzoning
> __kmem_cache_create: aligned size to 32
> __kmem_cache_create: num=226, slab_size=960
> __kmem_cache_create: starting, size=64, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 64 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
> __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
>
> With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation
> with this size fails. I do not know slab internals so I can not tell if
> this just happens because of the debug paths, or is it a real problem
> without the debug options too.
>

There was a rather large change to mm/slab.c that changed it to use 
'struct page' like SLUB. Perhaps slab debugging was broken in the 
process. Joonsoo, does the problem Meelis describes ring a bell?

                         Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Joonsoo Kim]

2013/12/3 Pekka Enberg <penberg@iki.fi>:
> On 11/30/2013 01:42 PM, Meelis Roos wrote:
>>
>> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
>> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
>> other debug options) and got the following BUG and hang on startup. This
>> happened originally with 3.11-rc2-00058 where my bisection of
>> another problem lead, but I retested 3.12 to have the same BUG in the
>> same place.
>>
>> kernel BUG at mm/slab.c:2391!
>>                \|/ ____ \|/
>>                "@'/ .. \`@"
>>                /_| \__/ |_\
>>                   \__U_/
>> swapper(0): Kernel bad sw trap 5 [#1]
>> CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty
>> #127
>> task: 00000000008ac468 ti: 000000000089c000 task.ti: 000000000089c000
>> TSTATE: 0000004480e01606 TPC: 00000000004f57d4 TNPC: 00000000004f57d8 Y:
>> 00000000    Not tainted
>> TPC: <__kmem_cache_create+0x374/0x480>
>> g0: 00000000000000f8 g1: 00000000008bb400 g2: 000000000002780b g3:
>> 00000000008b5120
>> g4: 00000000008ac468 g5: 0000000000000000 g6: 000000000089c000 g7:
>> 0000000000000000
>> o0: 0000000000845f08 o1: 0000000000000957 o2: ffffffffffffffe0 o3:
>> 0000000000000000
>> o4: 0000000000002004 o5: 0000000000000000 sp: 000000000089f301 ret_pc:
>> 00000000004f57cc
>> RPC: <__kmem_cache_create+0x36c/0x480>
>> l0: fffff8001e812040 l1: fffff8001e819f80 l2: fffff8001e819fb8 l3:
>> fffff8001e819fd8
>> l4: 0000000000000001 l5: fffff8001e819fc8 l6: 0000000000845f08 l7:
>> fffff8001e8300a0
>> i0: fffff8001e831fa0 i1: 0000000080002800 i2: 0000000080000000 i3:
>> 0000000000000034
>> i4: 0000000000000000 i5: 0000000000002000 i6: 000000000089f3b1 i7:
>> 0000000000907464
>> I7: <create_boot_cache+0x4c/0x84>
>> Call Trace:
>>   [0000000000907464] create_boot_cache+0x4c/0x84
>>   [00000000009074d0] create_kmalloc_cache+0x34/0x60
>>   [0000000000907540] create_kmalloc_caches+0x44/0x168
>>   [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
>>   [00000000008fc658] start_kernel+0x18c/0x370
>>   [0000000000761df4] tlb_fixup_done+0x88/0x94
>>   [0000000000000000]           (null)
>> Disabling lock debugging due to kernel taint
>> Caller[0000000000907464]: create_boot_cache+0x4c/0x84
>> Caller[00000000009074d0]: create_kmalloc_cache+0x34/0x60
>> Caller[0000000000907540]: create_kmalloc_caches+0x44/0x168
>> Caller[0000000000908dfc]: kmem_cache_init+0x1d0/0x1e0
>> Caller[00000000008fc658]: start_kernel+0x18c/0x370
>> Caller[0000000000761df4]: tlb_fixup_done+0x88/0x94
>> Caller[0000000000000000]:           (null)
>> Instruction DUMP: 92102957  7ffccb35  90122308 <91d02005> 90100018
>> 4009b371  920f20d0  ba922000  02480006
>> Kernel panic - not syncing: Attempted to kill the idle task!
>> Press Stop-A (L1-A) to return to the boot prom
>>
>> The line shows that __kmem_cache_create gets a NULL from kmalloc_slab().
>>
>> I instrumented the code and found the following:
>>
>> __kmem_cache_create: starting, size=248, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 248 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: starting, size=96, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 96 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: starting, size=192, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 192 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: starting, size=32, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 32 because of redzoning
>> __kmem_cache_create: aligned size to 32
>> __kmem_cache_create: num=226, slab_size=960
>> __kmem_cache_create: starting, size=64, flags=8192
>> __kmem_cache_create: now flags=76800
>> __kmem_cache_create: aligned size to 64 because of redzoning
>> __kmem_cache_create: pagealloc debug, setting size to 8192
>> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
>> __kmem_cache_create: aligned size to 8192
>> __kmem_cache_create: num=1, slab_size=64
>> __kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
>> __kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
>>
>> With slab size 64, it turns on CFLGS_OFF_SLAB and off slab allocation
>> with this size fails. I do not know slab internals so I can not tell if
>> this just happens because of the debug paths, or is it a real problem
>> without the debug options too.
>>
>
> There was a rather large change to mm/slab.c that changed it to use 'struct
> page' like SLUB. Perhaps slab debugging was broken in the process. Joonsoo,
> does the problem Meelis describes ring a bell?

Hello, Pekka.

No. He report that BUG() is triggered on v3.11-rc2 and v3.12.
And my recent change is merged into v3.13-rc1 as you know. :)

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Pekka Enberg]

On 12/03/2013 02:25 PM, Joonsoo Kim wrote:
> No. He report that BUG() is triggered on v3.11-rc2 and v3.12.
> And my recent change is merged into v3.13-rc1 as you know. :)

Hah, I guess my eyesight isn't what it used to be, I could have sworn it 
said v3.13-rc2... Thanks anyway, Joonsoo!

                             Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Christoph Lameter]

On Sat, 30 Nov 2013, Meelis Roos wrote:

On Sat, 30 Nov 2013, Meelis Roos wrote:

> I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> other debug options) and got the following BUG and hang on startup. This
> happened originally with 3.11-rc2-00058 where my bisection of
> another problem lead, but I retested 3.12 to have the same BUG in the
> same place.

Hmmm. With CONFIG_DEBUG_PAGEALLOC *and* DEBUG_SLAB you would get a pretty
strange configuration with massive sizes of slabs.

> kernel BUG at mm/slab.c:2391!

Ok so this means that we are trying to create a cache with off slab
management during bootstrap which should not happen.

> __kmem_cache_create: starting, size=248, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 248 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=96, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 96 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=192, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 192 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: aligned size to 8192
> __kmem_cache_create: num=1, slab_size=64
> __kmem_cache_create: starting, size=32, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 32 because of redzoning
> __kmem_cache_create: aligned size to 32
> __kmem_cache_create: num=226, slab_size=960
> __kmem_cache_create: starting, size=64, flags=8192
> __kmem_cache_create: now flags=76800
> __kmem_cache_create: aligned size to 64 because of redzoning
> __kmem_cache_create: pagealloc debug, setting size to 8192
> __kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192

We should not be switching on CFLGS_OFF_SLAB here because the
kmalloc array does not contain the necessary entries yet.

Does this fix it? We may need a more sophisticated fix from someone who
knows how handle CONFIG_DEBUG_PAGEALLOC.



Subject: slab: Do not use off slab metadata for CONFIG_DEBUG_PAGEALLOC

Signed-off-by: Christoph Lameter <cl@linux.com>

Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c	2013-12-03 09:44:20.463144282 -0600
+++ linux/mm/slab.c	2013-12-03 09:45:09.321786608 -0600
@@ -2243,13 +2243,15 @@ __kmem_cache_create (struct kmem_cache *
 	 * it too early on. Always use on-slab management when
 	 * SLAB_NOLEAKTRACE to avoid recursive calls into kmemleak)
 	 */
+#ifndef CONFIG_DEBUG_PAGEALLOC
 	if ((size >= (PAGE_SIZE >> 3)) && !slab_early_init &&
-	    !(flags & SLAB_NOLEAKTRACE))
+	    !(flags & SLAB_NOLEAKTRACE) )
 		/*
 		 * Size is large, assume best to place the slab management obj
 		 * off-slab (should allow better packing of objs).
 		 */
 		flags |= CFLGS_OFF_SLAB;
+#endif

 	size = ALIGN(size, cachep->align);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Meelis Roos]

> > I am debugging a reboot problem on Sun Ultra 5 (sparc64) with 512M RAM
> > and turned on DEBUG_PAGEALLOC DEBUG_SLAB and DEBUG_SLAB_LEAK (and most
> > other debug options) and got the following BUG and hang on startup. This
> > happened originally with 3.11-rc2-00058 where my bisection of
> > another problem lead, but I retested 3.12 to have the same BUG in the
> > same place.
> 
> Hmmm. With CONFIG_DEBUG_PAGEALLOC *and* DEBUG_SLAB you would get a pretty
> strange configuration with massive sizes of slabs.
> 
> > kernel BUG at mm/slab.c:2391!
> 
> Ok so this means that we are trying to create a cache with off slab
> management during bootstrap which should not happen.
[...]
> We should not be switching on CFLGS_OFF_SLAB here because the
> kmalloc array does not contain the necessary entries yet.
> 
> Does this fix it? We may need a more sophisticated fix from someone who
> knows how handle CONFIG_DEBUG_PAGEALLOC.

No:

Kernel panic - not syncing: Creation of kmalloc slab (null) size=8388608 
failed. Reason -7

CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc2-00058-g20bafb3-dirty 
#134
Call Trace:
 [000000000076416c] panic+0xb4/0x22c
 [0000000000907488] create_boot_cache+0x70/0x84
 [00000000009074d0] create_kmalloc_cache+0x34/0x60
 [0000000000907540] create_kmalloc_caches+0x44/0x168
 [0000000000908dfc] kmem_cache_init+0x1d0/0x1e0
 [00000000008fc658] start_kernel+0x18c/0x370
 [0000000000761db4] tlb_fixup_done+0x88/0x94
 [0000000000000000]           (null)

Am I just running out of memory perhaps?

Will try the other patch soon.

-- 
Meelis Roos (mroos@linux.ee)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Christoph Lameter]

On Tue, 3 Dec 2013, Meelis Roos wrote:

> Kernel panic - not syncing: Creation of kmalloc slab (null) size=8388608
> failed. Reason -7

Same here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Christoph Lameter]

On Tue, 3 Dec 2013, Christoph Lameter wrote:

> Subject: slab: Do not use off slab metadata for CONFIG_DEBUG_PAGEALLOC
>
> Signed-off-by: Christoph Lameter <cl@linux.com>

The patch fails here at the largest slab which requires off slab
management because otherwise the order becomes too high.

The fundamental problem is that debugging increases the size of the slab
significantly so that even small slabs require slab management. But
the kmalloc slabs used for the sizes required for management objects have not
been created yet.

Slab bootstrap relies on smaller slabs not requiring slab management
slabs. Once it gets to larger sizes then smaller slab management objects
may be allocated.

However, in this case the slab sizes are required for management at early
boot after slab_early_init has been set to 0 are not available.

If the slab management sizes required match other slab caches sizes that
were already created at boot then bootstrap will succeed regardless.

I have another patch here (that I cannot test since I cannot run sparc
code) that simply changes the determination for switching slab management
to base the decision not on the final size of the slab (which is always
large in the debugging cases here) but on the initial object size. For
small objects < PAGESIZE/8 this should avoid the use of slab management
even in the debugging case.

Subject: slab: Do not use slab management for slabs with smaller objects

Use the object size to make the off slab decision instead of the final
size of the slab objects (which is large in case of
CONFIG_PAGEALLOC_DEBUG).

Signed-off-by: Christoph Lameter <cl@linux.com>

Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c	2013-12-03 10:48:42.625823157 -0600
+++ linux/mm/slab.c	2013-12-03 10:49:06.755152653 -0600
@@ -2243,8 +2243,8 @@ __kmem_cache_create (struct kmem_cache *
 	 * it too early on. Always use on-slab management when
 	 * SLAB_NOLEAKTRACE to avoid recursive calls into kmemleak)
 	 */
-	if ((size >= (PAGE_SIZE >> 3)) && !slab_early_init &&
-	    !(flags & SLAB_NOLEAKTRACE))
+	if ((cachep->size >= (PAGE_SIZE >> 3)) && !slab_early_init &&
+	    !(flags & SLAB_NOLEAKTRACE) )
 		/*
 		 * Size is large, assume best to place the slab management obj
 		 * off-slab (should allow better packing of objs).


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Meelis Roos]

> I have another patch here (that I cannot test since I cannot run sparc
> code) that simply changes the determination for switching slab management
> to base the decision not on the final size of the slab (which is always
> large in the debugging cases here) but on the initial object size. For
> small objects < PAGESIZE/8 this should avoid the use of slab management
> even in the debugging case.
> 
> Subject: slab: Do not use slab management for slabs with smaller objects
> 
> Use the object size to make the off slab decision instead of the final
> size of the slab objects (which is large in case of
> CONFIG_PAGEALLOC_DEBUG).

Tested it. seems to hang after switching to another console. Before 
that, slabs are initialized successfully, I verified it with my previous 
debug printk sprinkle patch. Many allocations are still off slab - is 
that OK?

Memory: 493968K/523456K available (3521K kernel code, 343K rwdata, 1176K rodata, 264K init, 9803K bss, 29488K reserved)
__kmem_cache_create: starting, size=248, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 248 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=96, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 96 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=192, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 192 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=32, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 32 because of redzoning
__kmem_cache_create: aligned size to 32
__kmem_cache_create: num=226, slab_size=960
__kmem_cache_create: starting, size=64, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 64 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=128, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 128 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=256, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 256 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=512, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 512 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: starting, size=1024, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 1024 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=2048, flags=8192
__kmem_cache_create: now flags=76800
__kmem_cache_create: aligned size to 2048 because of redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=4096, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8192, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=16384, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=16384
__kmem_cache_create: aligned size to 16384
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=16384, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=32768, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=32768
__kmem_cache_create: aligned size to 32768
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=32768, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=65536, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=65536
__kmem_cache_create: aligned size to 65536
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=65536, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=131072, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=131072
__kmem_cache_create: aligned size to 131072
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=131072, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=262144, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=262144
__kmem_cache_create: aligned size to 262144
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=262144, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=524288, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=524288
__kmem_cache_create: aligned size to 524288
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=524288, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=1048576, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=1048576
__kmem_cache_create: aligned size to 1048576
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=1048576, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=2097152, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=2097152
__kmem_cache_create: aligned size to 2097152
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=2097152, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=4194304, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=4194304
__kmem_cache_create: aligned size to 4194304
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=4194304, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8388608, flags=8192
__kmem_cache_create: now flags=10240
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8388608
__kmem_cache_create: aligned size to 8388608
__kmem_cache_create: num=1, slab_size=64
__kmem_cache_create: CFLGS_OFF_SLAB, size=8388608, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8192, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=8192
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=8192, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=8192
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=16384, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=16384
__kmem_cache_create: aligned size to 16384
__kmem_cache_create: num=1, slab_size=16384
__kmem_cache_create: CFLGS_OFF_SLAB, size=16384, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=32768, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=32768
__kmem_cache_create: aligned size to 32768
__kmem_cache_create: num=1, slab_size=32768
__kmem_cache_create: CFLGS_OFF_SLAB, size=32768, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=65536, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=65536
__kmem_cache_create: aligned size to 65536
__kmem_cache_create: num=1, slab_size=65536
__kmem_cache_create: CFLGS_OFF_SLAB, size=65536, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=131072, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=131072
__kmem_cache_create: aligned size to 131072
__kmem_cache_create: num=1, slab_size=131072
__kmem_cache_create: CFLGS_OFF_SLAB, size=131072, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=262144, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=262144
__kmem_cache_create: aligned size to 262144
__kmem_cache_create: num=1, slab_size=262144
__kmem_cache_create: CFLGS_OFF_SLAB, size=262144, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=524288, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=524288
__kmem_cache_create: aligned size to 524288
__kmem_cache_create: num=1, slab_size=524288
__kmem_cache_create: CFLGS_OFF_SLAB, size=524288, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=1048576, flags=0
__kmem_cache_create: now flags=2048
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=1048576
__kmem_cache_create: aligned size to 1048576
__kmem_cache_create: num=1, slab_size=1048576
__kmem_cache_create: CFLGS_OFF_SLAB, size=1048576, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=2112, flags=262144
__kmem_cache_create: now flags=330752
__kmem_cache_create: aligned size to 2112 because of redzoning
__kmem_cache_create: aligned size to 2128 because of redzoning, take 2
__kmem_cache_create: increased size to 2136 because user store and redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: turning on CFLGS_OFF_SLAB, size=8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=56
__kmem_cache_create: CFLGS_OFF_SLAB, size=8192, slab_size=52
__kmem_cache_create: CFLGS_OFF_SLAB, allocating slab 52
__kmem_cache_create: starting, size=560, flags=393216
__kmem_cache_create: now flags=461824
__kmem_cache_create: aligned size to 560 because of redzoning
__kmem_cache_create: aligned size to 576 because of redzoning, take 2
__kmem_cache_create: increased size to 584 because user store and redzoning
__kmem_cache_create: pagealloc debug, setting size to 8192
__kmem_cache_create: aligned size to 8192
__kmem_cache_create: num=1, slab_size=56
NR_IRQS:255
kmemleak: Kernel memory leak detector disabled
clocksource: mult[2c71c72] shift[24]
clockevent: mult[5c28f5c3] shift[32]
Console: colour dummy device 80x25
console [tty0] enabled, bootconsole disabled

-- 
Meelis Roos (mroos@linux.ee)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Christoph Lameter]

On Tue, 3 Dec 2013, Meelis Roos wrote:

> Tested it. seems to hang after switching to another console. Before
> that, slabs are initialized successfully, I verified it with my previous
> debug printk sprinkle patch. Many allocations are still off slab - is
> that OK?

Yes that was the intend. Only exempt the small ones.

> console [tty0] enabled, bootconsole disabled

Looks like the bootstrap worked.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab BUG with DEBUG_* options

[Meelis Roos]

(Added 3 addresses to CC from my RED state exception thread since this 
is related)

> On Tue, 3 Dec 2013, Meelis Roos wrote:
> 
> > Tested it. seems to hang after switching to another console. Before
> > that, slabs are initialized successfully, I verified it with my previous
> > debug printk sprinkle patch. Many allocations are still off slab - is
> > that OK?
> 
> Yes that was the intend. Only exempt the small ones.
> 
> > console [tty0] enabled, bootconsole disabled
> 
> Looks like the bootstrap worked.

But the configuration should work fine with this console setup - with no 
slab debug options, it booted fine... I decided to do more tests.

In short, tests about 3.11-rc2-00058:

clean kernel: boots OK, RED state on shutdown (the actual problem I am 
chasing)

clean kernel, slab debug: mm crash

your second slab patch, slab debug: OK - this one shows that the RED 
state problem went away too which is good but strange

clean kernel, your second slab patch: OK - no RED state

Following another lead I had discovered that I can make the RED state 
problem go away if I switch tty ldata allocation from vmalloc to 
kmalloc. Tests with that:

ldata alloc change only, no slab debug: OK (works around RED state 
somehow)

ldata alloc change + slab debug: mm crash, can not test for RED state

ldata alloc change + your second slab patch + slab debug: hang on boot 
after
console [tty0] enabled, bootconsole disabled
(after that, I should see all the dmesg again on serial but I do not).

ldata alloc change + your second slab patch + no slab debug: OK

So, in short:

your slab patch 2 seems to cure both slab debug startup crash and the 
RED state problem in this specific version of the kernel. However, it is 
still mystery to me why tty ldata alloc change vmalloc->kmalloc would 
break but that may to be in the serial field - will do more tests with 
this patch applied and newer kernels.

-- 
Meelis Roos (mroos@linux.ee)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>