Slab memory leak in JFFS2 filesystems

Slab memory leak in JFFS2 filesystems

[Johns Daniel]

I have discovered a kernel memory leak associated with JFFS2
filesystems. I have verified the leak in kernels 2.6.28 and 2.6.36 on
a Freescale PowerPC board using this script:

while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
   cat /proc/slabinfo |grep "dentry\|size-64 "; sleep 1; /bin/rm $FN; done

The "dentry" and "size-64" slabs grow continuously with this script --
and the memory is not reclaimed even after the script is stopped.

Basically, JFFS2 does not do a complete slab cleanup when a file is
deleted. If the old filename is reused, you don't notice the leak
because it is very small. However, if you use a new filename each
time, the memory leak is pretty obvious.

Is this a JFFS2 feature or is this a bug?
If it is a bug, does anybody have a fix for it?

-- Johns

Re: Slab memory leak in JFFS2 filesystems

[Artem Bityutskiy]

On Thu, 2011-02-24 at 18:41 -0600, Johns Daniel wrote:
> I have discovered a kernel memory leak associated with JFFS2
> filesystems. I have verified the leak in kernels 2.6.28 and 2.6.36 on
> a Freescale PowerPC board using this script:
> 
> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
>    cat /proc/slabinfo |grep "dentry\|size-64 "; sleep 1; /bin/rm $FN; done

Please, check whether they go away after:

echo 3 > /proc/sys/vm/drop_caches

See Documentation/sysctl/vm.txt for more information about what this
means.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: Slab memory leak in JFFS2 filesystems

[Johns Daniel]

On Fri, Feb 25, 2011 at 6:38 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Thu, 2011-02-24 at 18:41 -0600, Johns Daniel wrote:
>> I have discovered a kernel memory leak associated with JFFS2
>> filesystems. I have verified the leak in kernels 2.6.28 and 2.6.36 on
>> a Freescale PowerPC board using this script:
>>
>> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
>>    cat /proc/slabinfo |grep "dentry\|size-64 "; sleep 1; /bin/rm $FN; done
>
> Please, check whether they go away after:
>
> echo 3 > /proc/sys/vm/drop_caches
>
> See Documentation/sysctl/vm.txt for more information about what this
> means.

Thanks for that suggestion, Artem! Here is what I tried:

# grep "dentry\|size-64 " /proc/slabinfo
dentry              1398   1410    128   30    1 : tunables  120   60
  0 : slabdata     47     47      0
size-64             1235   1239     64   59    1 : tunables  120   60
  0 : slabdata     21     21      0
# echo 3 > /proc/sys/vm/drop_caches
# grep "dentry\|size-64 " /proc/slabinfo
dentry               332    780    128   30    1 : tunables  120   60
  0 : slabdata     26     26      0
size-64             1213   1239     64   59    1 : tunables  120   60
  0 : slabdata     21     21      0

Then I ran this script for 10 minutes:
# while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
     grep "dentry\|size-64 " /proc/slabinfo; sleep 1; /bin/rm $FN; done

I wasn't sure whether the effect of the "drop_caches" command was
"persistent". So I ran the command once more:
# echo 3 > /proc/sys/vm/drop_caches

About 20 minutes after I stopped the test script, here is the status:
# grep "dentry\|size-64 " /proc/slabinfo
dentry               387    780    128   30    1 : tunables  120   60
  0 : slabdata     26     26      0
size-64             1865   1888     64   59    1 : tunables  120   60
  0 : slabdata     32     32      0

Clearly, there is some growth in the slabs that cannot be explained. I
will also give you two counterpoints:

1.) If I run this similar test script on the same system, the slab
memory does not grow at all (regardless of the drop_caches setting):

while :; do FN=$(touch /jffs2fs/TMP.XXXXXXXX); \
   grep "dentry\|size-64 " /proc/slabinfo; sleep 1; /bin/rm /data/TMP.*; done

2.) This bug brought one of our production systems in the field to its
knee in less than 3 months. If the memory was reclaimable, I don't
think this slab memory growth would have exhausted available free
memory in the system to the point that other processes were unable to
allocate memory.

Any help would be appreciated!

-- Johns

>
> --
> Best Regards,
> Artem Bityutskiy (Артём Битюцкий)
>
>

Re: Slab memory leak in JFFS2 filesystems

[Artem Bityutskiy]

On Fri, 2011-02-25 at 10:12 -0600, Johns Daniel wrote:
> On Fri, Feb 25, 2011 at 6:38 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> > On Thu, 2011-02-24 at 18:41 -0600, Johns Daniel wrote:
> >> I have discovered a kernel memory leak associated with JFFS2
> >> filesystems. I have verified the leak in kernels 2.6.28 and 2.6.36 on
> >> a Freescale PowerPC board using this script:
> >>
> >> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
> >>    cat /proc/slabinfo |grep "dentry\|size-64 "; sleep 1; /bin/rm $FN; done
> >
> > Please, check whether they go away after:
> >
> > echo 3 > /proc/sys/vm/drop_caches
> >
> > See Documentation/sysctl/vm.txt for more information about what this
> > means.
> 
> Thanks for that suggestion, Artem! Here is what I tried:

Hi, you can try to play with kmemleak - this is a kernel feature which
slows down the system a lot but is great in catching memory leaks. It
may have false positives sometimes, though. You can read about kmemleak
in the Documentation/ directory. I think if there are leaks in JFFS2 -
kmemleak would spot them.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Slab memory leak in JFFS2 filesystems

[Johns Daniel]

On Fri, Feb 25, 2011 at 10:27 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Fri, 2011-02-25 at 10:12 -0600, Johns Daniel wrote:
>> On Fri, Feb 25, 2011 at 6:38 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
>> > On Thu, 2011-02-24 at 18:41 -0600, Johns Daniel wrote:
>> >> I have discovered a kernel memory leak associated with JFFS2
>> >> filesystems. I have verified the leak in kernels 2.6.28 and 2.6.36 on
>> >> a Freescale PowerPC board using this script:
>> >>
>> >> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
>> >>    cat /proc/slabinfo |grep "dentry\|size-64 "; sleep 1; /bin/rm $FN; done
>> >
>> > Please, check whether they go away after:
>> >
>> > echo 3 > /proc/sys/vm/drop_caches
>> >
>> > See Documentation/sysctl/vm.txt for more information about what this
>> > means.
>>
>> Thanks for that suggestion, Artem! Here is what I tried:
>
> Hi, you can try to play with kmemleak - this is a kernel feature which
> slows down the system a lot but is great in catching memory leaks. It
> may have false positives sometimes, though. You can read about kmemleak
> in the Documentation/ directory. I think if there are leaks in JFFS2 -
> kmemleak would spot them.
>

Unfortunately, the kmemleak feature is not supported on PPC even in
2.6.36. And I don't have a supported system available with the JFFS2
filesystem.

One more data point. After running the script like this:

while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
   echo 3 > /proc/sys/vm/drop_caches; sleep 1; \
   grep  "dentry\|size-64 " /proc/slabinfo; /bin/rm $FN; done

it looks like the leak may only be in "size-64" (and not "dentry").

> --
> Best Regards,
> Artem Bityutskiy (Битюцкий Артём)
>
>

Re: Slab memory leak in JFFS2 filesystems

[Artem Bityutskiy]

On Fri, 2011-02-25 at 11:11 -0600, Johns Daniel wrote:
> On Fri, Feb 25, 2011 at 10:27 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> > On Fri, 2011-02-25 at 10:12 -0600, Johns Daniel wrote:
> >> On Fri, Feb 25, 2011 at 6:38 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> >> > On Thu, 2011-02-24 at 18:41 -0600, Johns Daniel wrote:
> >> >> I have discovered a kernel memory leak associated with JFFS2
> >> >> filesystems. I have verified the leak in kernels 2.6.28 and 2.6.36 on
> >> >> a Freescale PowerPC board using this script:
> >> >>
> >> >> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
> >> >>    cat /proc/slabinfo |grep "dentry\|size-64 "; sleep 1; /bin/rm $FN; done
> >> >
> >> > Please, check whether they go away after:
> >> >
> >> > echo 3 > /proc/sys/vm/drop_caches
> >> >
> >> > See Documentation/sysctl/vm.txt for more information about what this
> >> > means.
> >>
> >> Thanks for that suggestion, Artem! Here is what I tried:
> >
> > Hi, you can try to play with kmemleak - this is a kernel feature which
> > slows down the system a lot but is great in catching memory leaks. It
> > may have false positives sometimes, though. You can read about kmemleak
> > in the Documentation/ directory. I think if there are leaks in JFFS2 -
> > kmemleak would spot them.
> >
> 
> Unfortunately, the kmemleak feature is not supported on PPC even in
> 2.6.36. And I don't have a supported system available with the JFFS2
> filesystem.

Well, I cannot help you with JFFS2, sorry. Just few ideas, may be you'll
find them helpful.

I can suggest you thought to run a test on a PC + nandsim (or mtdram if
you have NOR). If it is possible, compile a similar kernel for PC and
test with the simulator. If you can reproduce the issue, you will have
kmemleak.

> One more data point. After running the script like this:
> 
> while :; do FN=$(mktemp /jffs2fs/TMP.XXXXXXXX); \
>    echo 3 > /proc/sys/vm/drop_caches; sleep 1; \
>    grep  "dentry\|size-64 " /proc/slabinfo; /bin/rm $FN; done
> 
> it looks like the leak may only be in "size-64" (and not "dentry").

Well, this comes from kmalloc(33-64) AFAIU. You can just instrument your
kernel yourself - add a small piece of code to kmalloc for the "size-64"
case. Make this code to do the following:

Define something like:

struct mem_user {
	void *addr;
	void *caller_addr;
}

You can pre-allocate few megs of bootmem and use that memory for these
objects.

Then, plug some code to kmalloc which will for create "struct mem_user"
object for each allocation and insert it into an RB-tree indexed by
'addr', where the 'addr' is the address of the allocated memory, so it
will be the key.

On kfree() - delete corresponding object from the RB-tree.

This way you will always be able to see who made an allocation. You can
add a debugfs file and print the list of memory users. You can store not
only caller address, but also whole or partial stackdump.

I did like this in UBI and even submitted the code in the first UBI
submittion - you can find it in LKML.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)