Re: Slab memory leak in JFFS2 filesystems

[Artem Bityutskiy <dedekind1@gmail.com>]

On Fri, 2011-02-25 at 11:11 -0600, Johns Daniel wrote:
> On Fri, Feb 25, 2011 at 10:27 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> > On Fri, 2011-02-25 at 10:12 -0600, Johns Daniel wrote:
> >> On Fri, Feb 25, 2011 at 6:38 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> >> > On Thu, 2011-02-24 at 18:41 -0600, Johns Daniel wrote:
> >> >> I have discovered a kernel memory leak associated with JFFS2
> >> >> filesystems. I have verified the leak in kernels 2.6.28 and 2.6.36 on
> >> >> a Freescale PowerPC board using this script:
> >> >>
> >> >> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
> >> >>    cat /proc/slabinfo |grep "dentry\|size-64 "; sleep 1; /bin/rm $FN; done
> >> >
> >> > Please, check whether they go away after:
> >> >
> >> > echo 3 > /proc/sys/vm/drop_caches
> >> >
> >> > See Documentation/sysctl/vm.txt for more information about what this
> >> > means.
> >>
> >> Thanks for that suggestion, Artem! Here is what I tried:
> >
> > Hi, you can try to play with kmemleak - this is a kernel feature which
> > slows down the system a lot but is great in catching memory leaks. It
> > may have false positives sometimes, though. You can read about kmemleak
> > in the Documentation/ directory. I think if there are leaks in JFFS2 -
> > kmemleak would spot them.
> >
> 
> Unfortunately, the kmemleak feature is not supported on PPC even in
> 2.6.36. And I don't have a supported system available with the JFFS2
> filesystem.

Well, I cannot help you with JFFS2, sorry. Just few ideas, may be you'll
find them helpful.

I can suggest you thought to run a test on a PC + nandsim (or mtdram if
you have NOR). If it is possible, compile a similar kernel for PC and
test with the simulator. If you can reproduce the issue, you will have
kmemleak.

> One more data point. After running the script like this:
> 
> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
>    echo 3 > /proc/sys/vm/drop_caches; sleep 1; \
>    grep  "dentry\|size-64 " /proc/slabinfo; /bin/rm $FN; done
> 
> it looks like the leak may only be in "size-64" (and not "dentry").

Well, this comes from kmalloc(33-64) AFAIU. You can just instrument your
kernel yourself - add a small piece of code to kmalloc for the "size-64"
case. Make this code to do the following:

Define something like:

struct mem_user {
	void *addr;
	void *caller_addr;
}

You can pre-allocate few megs of bootmem and use that memory for these
objects.

Then, plug some code to kmalloc which will for create "struct mem_user"
object for each allocation and insert it into an RB-tree indexed by
'addr', where the 'addr' is the address of the allocated memory, so it
will be the key.

On kfree() - delete corresponding object from the RB-tree.

This way you will always be able to see who made an allocation. You can
add a debugfs file and print the list of memory users. You can store not
only caller address, but also whole or partial stackdump.

I did like this in UBI and even submitted the code in the first UBI
submittion - you can find it in LKML.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)