Re: Kernel 4.1.6 Panic due to slab corruption

[Nikolay Borisov <kernel@kyup.com>]

On 09/09/2015 05:01 PM, Christoph Lameter wrote:
> On Wed, 9 Sep 2015, Nikolay Borisov wrote:
> 
>> [root@kernighan vm]# ./slabinfo -da kmalloc-32
>> Cannot write to dma-kmalloc-32/sanity
>> [root@kernighan vm]# ./slabinfo -dF kmalloc-32
>> Cannot write to dma-kmalloc-32/sanity
>> [root@kernighan vm]# ./slabinfo -dz kmalloc-32
>> kmalloc-32 not empty cannot enable redzoning
>> [root@kernighan vm]# ./slabinfo -dp kmalloc-32
>> kmalloc-32 not empty cannot enable poisoning
>> [root@kernighan vm]# ./slabinfo -du kmalloc-32
>> kmalloc-32 not empty cannot enable tracking
>> [root@kernighan vm]# ./slabinfo -dt ^kmalloc-32$
>> kmalloc-32 can only enable trace for one slab at a time
> 
> 
> Hmmmm.. Whats the problem here?
> 
> christoph@gentwo:/sys/kernel/slab/kmalloc-32$ ls -l
> total 0
> -r-------- 1 root root 4096 Sep  9 08:57 aliases
> -r-------- 1 root root 4096 Sep  9 08:57 align
> -r-------- 1 root root 4096 Sep  9 08:57 alloc_calls
> -r-------- 1 root root 4096 Sep  9 08:57 cache_dma
> -rw------- 1 root root 4096 Sep  9 08:57 cpu_partial
> -r-------- 1 root root 4096 Sep  9 08:57 cpu_slabs
> -r-------- 1 root root 4096 Sep  9 08:57 ctor
> -r-------- 1 root root 4096 Sep  9 08:57 destroy_by_rcu
> -r-------- 1 root root 4096 Sep  9 08:57 free_calls
> -r-------- 1 root root 4096 Sep  9 08:57 hwcache_align
> -rw------- 1 root root 4096 Sep  9 08:57 min_partial
> -r-------- 1 root root 4096 Sep  9 08:57 objects
> -r-------- 1 root root 4096 Sep  9 08:57 object_size
> -r-------- 1 root root 4096 Sep  9 08:57 objects_partial
> -r-------- 1 root root 4096 Sep  9 08:57 objs_per_slab
> -rw------- 1 root root 4096 Sep  9 08:57 order
> -r-------- 1 root root 4096 Sep  9 08:57 partial
> -rw------- 1 root root 4096 Sep  9 08:57 poison
> -rw------- 1 root root 4096 Sep  9 08:57 reclaim_account
> -rw------- 1 root root 4096 Sep  9 08:57 red_zone
> -rw------- 1 root root 4096 Sep  9 08:57 remote_node_defrag_ratio
> -r-------- 1 root root 4096 Sep  9 08:57 reserved
> -rw------- 1 root root 4096 Sep  9 08:57 sanity_checks
> -rw------- 1 root root 4096 Sep  9 08:57 shrink
> -r-------- 1 root root 4096 Sep  9 08:57 slabs
> -r-------- 1 root root 4096 Sep  9 08:57 slabs_cpu_partial
> -r-------- 1 root root 4096 Sep  9 08:57 slab_size
> -rw------- 1 root root 4096 Sep  9 08:57 store_user
> -r-------- 1 root root 4096 Sep  9 08:57 total_objects
> -rw------- 1 root root 4096 Sep  9 08:57 trace
> -rw------- 1 root root 4096 Sep  9 08:57 validate
> 
> Try
> 
> 	echo 1 >santy_checks

[root@kernighan linux-stable]# cd /sys/kernel/slab/kmalloc-32/
[root@kernighan kmalloc-32]# echo 1 > sanity_checks
[root@kernighan kmalloc-32]# cat sanity_checks
1

So this works as expected when set by echo. Just for testing I then
tried the following:
[root@kernighan kmalloc-32]# slabinfo -d- kmalloc-32
kmalloc-32 not empty cannot disable sanity checks

[root@kernighan kmalloc-32]# echo 0 > sanity_checks
[root@kernighan kmalloc-32]# slabinfo -d- kmalloc-32

So turns out slabinfo fails where the raw sys interface succeeds, strange?



> 
> 
>>
>> I did however had success with enabling tracing but couldn't see where
>> the output is produced - tried dmesg and the ftrace buffer but nothing
>> turned up.
> 
> dmesg is the output channel for tracing.
> 
> What does:
> 
> 	echo 1 >trace
> 
> do? Could crash the sysem due to overload of messages.

Didn't have that much luck with this one:
[root@kernighan kmalloc-32]# dmesg -c > /dev/null
[root@kernighan kmalloc-32]# echo 1 > trace
-bash: echo: write error: Invalid argument

> 
>> But it seems it is not possible to enable any debugging whatsoever, so I
>> will restor to doing it at boot time. In this case can you advice which
>> options might not result in very high performance degradation - I'm
>> thinking of sanity checking and maybe redzoning?
> 
> Sanity checking is ok. But I would think you should be fine with enabling
> full debugging on the particular caches of interest.

I was just thinking that if enabling debug options disables merging this
means it won't be sufficient to enable debugging on kmalloc-32 but
rather before enabling debugging I do need to check which caches were
aliased and enable debugging on those as well, correct?


Regards,
Nikolay