Slab initialisation problems on MN10300

Slab initialisation problems on MN10300

[David Howells]

Hi,

I'm running into a BUG_ON() when trying to boot an MN10300 arch kernel.  The
kernel is UP, non-NUMA and is using SLAB.  With gdb attached to the kernel, I
see the following backtrace:

	(gdb) bt
	#0  0x90258041 in setup_cpu_cache (cachep=0x93c00130) at mm/slab.c:2103
	#1  0x900977d7 in kmem_cache_create (name=0x9026de9d "size-64", size=64, align=16, flags=270336, 
	    ctor=0) at mm/slab.c:2384
	#2  0x9029e959 in kmem_cache_init () at mm/slab.c:1548
	#3  0x902987aa in start_kernel () at init/main.c:618
	#4  0x9000122f in __no_parameters () at arch/mn10300/kernel/head.S:209
	#5  0x9000122f in __no_parameters () at arch/mn10300/kernel/head.S:209

The offending line is this:

		} else {
			int node;
			for_each_online_node(node) {
				cachep->nodelists[node] =
				    kmalloc_node(sizeof(struct kmem_list3),
						GFP_KERNEL, node);
  >>>>>>			BUG_ON(!cachep->nodelists[node]);
				kmem_list3_init(cachep->nodelists[node]);
			}
		}

This is line 2103 of mm/slab.c in setup_cpu_cache().

Using gdb, I can see that node is 0 and cachep->nodelists[0] is NULL before
the kmalloc_node().

Looking in malloc_sizes[], I see:

	(gdb) p malloc_sizes[0]
	$18 = {cs_size = 0x20, cs_cachep = 0x93c000e0}
	(gdb) p malloc_sizes[1]
	$19 = {cs_size = 0x40, cs_cachep = 0x0}
	(gdb) p malloc_sizes[2]
	$20 = {cs_size = 0x60, cs_cachep = 0x0}
	(gdb) p malloc_sizes[3]
	$21 = {cs_size = 0x80, cs_cachep = 0x0}

and sizeof(struct kmem_list3) is 52, which is going to get rounded up to 64 by
kmalloc_node().  This means that it's going to attempt to allocate out of the
64-byte kmalloc slab, which is what the kernel is currently setting up, so the
allocation fails.

I have to wonder if this comment in setup_cpu_cache() is actually correct:

		/*
		 * Note: the first kmem_cache_create must create the cache
		 * that's used by kmalloc(24), otherwise the creation of
		 * further caches will BUG().
		 */

Perhaps it's no longer 24, but something bigger.  The first pass through
setup_cpu_cache() is done for the 32-byte kmalloc slab, with g_cpucache_up set
to NONE.  The second pass is done for the 64-byte kmalloc slab with
g_cpucache_up set to PARTIAL_L3.  It is the second pass that fails.

The first pass is special cased.  malloc_sizes[0]->cs_cachep is set after the
first pass.

The second pass calls kmalloc() on sizeof(struct arraycache_init), which is 20
and succeeds.  It then calls kmalloc_node() on sizeof(struct kmem_list3),
which is 52 and fails.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab initialisation problems on MN10300

[Pekka Enberg]

Hi David,

What kernel version is this?

On Feb 18, 2008 6:07 PM, David Howells <dhowells@redhat.com> wrote:
>	(gdb) bt
>	#0  0x90258041 in setup_cpu_cache (cachep=0x93c00130) at mm/slab.c:2103
>	#1  0x900977d7 in kmem_cache_create (name=0x9026de9d "size-64",
size=64, align=16, flags=270336,
>              ctor=0) at mm/slab.c:2384
>	#2  0x9029e959 in kmem_cache_init () at mm/slab.c:1548

So we've already set up caches for struct arraycache_init (INDEX_AC)
and struct kmem_list3 (INDEX_L3) here and trying to initialize rest of
the caches.

On Feb 18, 2008 6:07 PM, David Howells <dhowells@redhat.com> wrote:
>	#3  0x902987aa in start_kernel () at init/main.c:618
>	#4  0x9000122f in __no_parameters () at arch/mn10300/kernel/head.S:209
>	#5  0x9000122f in __no_parameters () at arch/mn10300/kernel/head.S:209

But then:

On Feb 18, 2008 6:07 PM, David Howells <dhowells@redhat.com> wrote:
> and sizeof(struct kmem_list3) is 52, which is going to get rounded up to 64 by
> kmalloc_node().  This means that it's going to attempt to allocate out of the
> 64-byte kmalloc slab, which is what the kernel is currently setting up, so the
> allocation fails.

doesn't make any sense as we should have already initialized the cache
for sizeof(struct kmem_list3) (denoted by INDEX_L3).

Also:

On Feb 18, 2008 6:07 PM, David Howells <dhowells@redhat.com> wrote:
> Perhaps it's no longer 24, but something bigger.  The first pass through
> setup_cpu_cache() is done for the 32-byte kmalloc slab, with g_cpucache_up set
> to NONE.  The second pass is done for the 64-byte kmalloc slab with
> g_cpucache_up set to PARTIAL_L3.  It is the second pass that fails.

If you didn't see PARTIAL_AC state at all, SLAB thinks INDEX_AC and
INDEX_L3 are equal. However,

On Feb 18, 2008 6:07 PM, David Howells <dhowells@redhat.com> wrote:
> The second pass calls kmalloc() on sizeof(struct arraycache_init), which is 20
> and succeeds.  It then calls kmalloc_node() on sizeof(struct kmem_list3),
> which is 52 and fails.

would put struct arraycache_init to kmalloc-32 and struct kmem_list3
to kmalloc-64. So are INDEX_AC and INDEX_L3 really equivalent? To
which cache do they refer to?

And if this broke recently, you might want to try and see if commit
556a169dab38b5100df6f4a45b655dddd3db94c1 ("slab: fix bootstrap on
memoryless node") is at fault here by reverting it.

				Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab initialisation problems on MN10300

[Pekka Enberg]

Hi,

On Feb 18, 2008 7:18 PM, Pekka Enberg <penberg@cs.helsinki.fi> wrote:
> And if this broke recently, you might want to try and see if commit
> 556a169dab38b5100df6f4a45b655dddd3db94c1 ("slab: fix bootstrap on
> memoryless node") is at fault here by reverting it.

Hmm, I double-checked the patch and it probably isn't the cause here.
It's just that we haven't changed SLAB bootstrap all that much except
for this patch. One thing that I thought of was ARCH_KMALLOC_MINALIGN
which is set to some fairly big values on some MIPS architectures
(MN10300 is one, right?) but reading the code I wasn't able to see
what could go wrong with that either...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab initialisation problems on MN10300

[David Howells]

Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> One thing that I thought of was ARCH_KMALLOC_MINALIGN
> which is set to some fairly big values on some MIPS architectures
> (MN10300 is one, right?)

No.  MN10300 is not MIPS.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab initialisation problems on MN10300

[David Howells]

Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> would put struct arraycache_init to kmalloc-32 and struct kmem_list3
> to kmalloc-64. So are INDEX_AC and INDEX_L3 really equivalent? To
> which cache do they refer to?

(gdb) p sizeof(struct arraycache_init)
$1 = 20
(gdb) p sizeof(struct kmem_list3)
$2 = 52

However, the compiler has eliminated the test:

		if (INDEX_AC == INDEX_L3)

even though it's compiled with -O0.

This is odd.  I'll have to investigate the preprocessor output.

> And if this broke recently, you might want to try and see if commit
> 556a169dab38b5100df6f4a45b655dddd3db94c1 ("slab: fix bootstrap on
> memoryless node") is at fault here by reverting it.

Well, the MN10300 arch worked in -mm, but no longer works now that the patches
have been merged into Linus's tree.  Bisecting is probably not an option.

Thanks, anyway.  I've got something to investigate.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab initialisation problems on MN10300

[Pekka Enberg]

David Howells wrote:
> (gdb) p sizeof(struct arraycache_init)
> $1 = 20
> (gdb) p sizeof(struct kmem_list3)
> $2 = 52
> 
> However, the compiler has eliminated the test:
> 
> 		if (INDEX_AC == INDEX_L3)
> 
> even though it's compiled with -O0.
> 
> This is odd.  I'll have to investigate the preprocessor output.

What's PAGE_SIZE for the architecture? If it's something other than 4KB, 
the size 32 cache is not there which makes both use the 64 one. However, 
that should work too, so maybe there's some GCC bug here for your 
architecture?

			Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab initialisation problems on MN10300

[David Howells]

Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> What's PAGE_SIZE for the architecture? If it's something other than 4KB, the
> size 32 cache is not there which makes both use the 64 one. However, that
> should work too, so maybe there's some GCC bug here for your architecture?

It's 4K.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Slab initialisation problems on MN10300

[David Howells]

Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> If you didn't see PARTIAL_AC state at all, SLAB thinks INDEX_AC and
> INDEX_L3 are equal. However,

Ah...  The problem is that index_of() behaves differently under -O0 rather
than -O1, -O2 or -Os.

I was using -O0 so that I could debug another problem using GDB on the kernel.
However, this appears to mean that __builtin_constant_p() inside an inline
function is always false, even if the function is actually inlined because of
__always_inline.

I'd commented out the __bad_size() calls because they went to places that
don't exist, and so the -O0 kernel wouldn't link.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>