Re: Oops in slab.c in CentOS kernel, looking for ideas -- correction, it's in slub.c

[Chris Friesen <chris.friesen@windriver.com>]

On 09/27/2016 11:14 PM, Joonsoo Kim wrote:
> On Tue, Sep 27, 2016 at 12:33:08PM -0600, Chris Friesen wrote:
>> On 09/27/2016 10:12 AM, Chris Friesen wrote:

>>> Basically it appears that __mpol_dup() is failing because the value of
>>> c->freelist in slab_alloc_node() is corrupt, causing the call to
>>> get_freepointer_safe(s, object) to Oops because it tries to dereference
>>> "object + s->offset".  (Where s->offset is zero.)
>>>
>>> In the trace, "kmem_cache_alloc+0x87" maps to the following assembly:
>>>     0xffffffff8118be17 <+135>:   mov    (%r12,%rax,1),%rbx
>>>
>>> This corresponds to this line in get_freepointer():
>>> 	return *(void **)(object + s->offset);
>>>
>>> In the assembly code, R12 is "object", and RAX is s->offset.
>>>
>>> So the question becomes, why is "object" (which corresponds to c->freelist)
>>> corrupt?
>>>
>>> Looking at the value of R12 (0x1ada8000), it's nonzero but also not a
>>> valid pointer. Does the value mean anything to you?  (I'm not really
>>> a memory subsystem guy, so I'm hoping you might have some ideas.)
>>>
>>> Do you have any suggestions on how to track down what's going on here?
>
> Please run with kernel parameter "slub_debug=F" or something.
> See Documentation/vm/slub.txt.

I enabled /sys/kernel/slab/numa_policy/sanity_checks, but that's only going to 
maybe help if I can cause another CPU to get into the bad state.

I created a kernel module to walk the list of objects starting at 
__this_cpu_ptr(policy_cache->cpu_slab)->freelist.

All other cpus had a freelist value of NULL, or else they pointed at a linked 
list which eventually ended with a NULL pointer.  ("s->offset" is 0, so 
get_freepointer() just dereferences "object")  For example:

cpu: 45, object: ffff88046d483cd8->ffff88046d483de0->ffff88046d483ee8->NULL
cpu: 46, object: NULL

In the case of CPU 48, the value of 
__this_cpu_ptr(policy_cache->cpu_slab)->freelist was good, but dereferencing it 
gave an invalid address:

cpu: 48, object: ffff8804102f0528->000000001ada8000


In the code path that causes problems we call mpol_new(), which calls 
kmem_cache_alloc(policy_cache, GFP_KERNEL) and consumes the object at 
0xffff8804102f0528.  This results in 
__this_cpu_ptr(policy_cache->cpu_slab)->freelist being set to 
0x000000001ada8000.   Then we fork, which calls __mpol_dup() which calls 
kmem_cache_alloc(policy_cache, GFP_KERNEL) with 'object' set to 
0x000000001ada8000, which segfaults when we try to dereference it in 
get_freepointer().

So how do items get added to the freelist?  Do they always get added at the 
head, or is there a path where they could get added at the tail?

Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>