Bug report: slab error at module removal

Bug report: slab error at module removal

[Mathieu Segaud]

I don't know if this is the right place, but I just saw a kernel message
in my dmesg.  

The box is a test box with 7 Reiser4 FSes.
5 of them are LVM2 logical volumes. And the device-mapper
is used to crypt root partition (cipher AES).
All Reiser4 partitions have been converted to 1.0.0 layout/format,
and were consistent at boot time.
After several hours of running some moderatly high load stuff 
on each of them (big transfert of data, long repacker runs...).
I decided to umount them and unload reiser4.ko module.
dmesg gave this message:

slab error in kmem_cache_destroy(): cache `plugin_set': Can't free all objects
 [<c0135c41>] kmem_cache_destroy+0xd1/0x130
 [<f99bf57a>] plugin_set_done+0xa/0x30 [reiser4]
 [<f99b9194>] shutdown_reiser4+0xf4/0x210 [reiser4]
 [<c012a004>] sys_delete_module+0x144/0x180
 [<c013ea9e>] unmap_vma_list+0xe/0x20
 [<c013edcf>] do_munmap+0xef/0x150
 [<c0103afd>] sysenter_past_esp+0x52/0x71
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name
SLAB: cache with size 64 has lost its name

The FSes are still consistent (fsck'ed with no --fix needed).
Unfortunatly, I turned off debugging options to test latest snapshot
performance. I can rebuild the module with debugging options on,
and try to give you a more accurate report to help find the memory leak.

Regards,

-- 
Mathieu Segaud

Re: Bug report: slab error at module removal

[Lamont R. Peterson]

[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]

On Fri, 2004-08-13 at 22:26, Mathieu Segaud wrote:
> I don't know if this is the right place, but I just saw a kernel message
> in my dmesg.  
[SNIP]
> After several hours of running some moderatly high load stuff 
> on each of them (big transfert of data, long repacker runs...).
> I decided to umount them and unload reiser4.ko module.
> dmesg gave this message:

I thought that building reiser4 as a module was still not ready.  Is
that true?

Is it safe to build and load as a module but does not remove cleanly
yet?  Is that the reason reiser4 as a kernel module was not ready?

> slab error in kmem_cache_destroy(): cache `plugin_set': Can't free all objects
>  [<c0135c41>] kmem_cache_destroy+0xd1/0x130
>  [<f99bf57a>] plugin_set_done+0xa/0x30 [reiser4]
>  [<f99b9194>] shutdown_reiser4+0xf4/0x210 [reiser4]
>  [<c012a004>] sys_delete_module+0x144/0x180
>  [<c013ea9e>] unmap_vma_list+0xe/0x20
>  [<c013edcf>] do_munmap+0xef/0x150
>  [<c0103afd>] sysenter_past_esp+0x52/0x71
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> 
> The FSes are still consistent (fsck'ed with no --fix needed).
> Unfortunatly, I turned off debugging options to test latest snapshot
> performance. I can rebuild the module with debugging options on,
> and try to give you a more accurate report to help find the memory leak.
> 
> Regards,
-- 
Lamont R. Peterson <lamont@gurulabs.com>
Senior Instructor
Guru Labs http://www.gurulabs.com/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Bug report: slab error at module removal

[Alex Zarochentsev]

On Sat, Aug 14, 2004 at 12:51:42PM -0600, Lamont R. Peterson wrote:
> On Fri, 2004-08-13 at 22:26, Mathieu Segaud wrote:
> > I don't know if this is the right place, but I just saw a kernel message
> > in my dmesg.  
> [SNIP]
> > After several hours of running some moderatly high load stuff 
> > on each of them (big transfert of data, long repacker runs...).
> > I decided to umount them and unload reiser4.ko module.
> > dmesg gave this message:
> 
> I thought that building reiser4 as a module was still not ready.  Is
> that true?
> 
> Is it safe to build and load as a module but does not remove cleanly
> yet?  Is that the reason reiser4 as a kernel module was not ready?

reiser4-as-module is supposed to work :) If it doesn't please report the bug to
the mailing list. 

> > slab error in kmem_cache_destroy(): cache `plugin_set': Can't free all objects
> >  [<c0135c41>] kmem_cache_destroy+0xd1/0x130
> >  [<f99bf57a>] plugin_set_done+0xa/0x30 [reiser4]
> >  [<f99b9194>] shutdown_reiser4+0xf4/0x210 [reiser4]
> >  [<c012a004>] sys_delete_module+0x144/0x180
> >  [<c013ea9e>] unmap_vma_list+0xe/0x20
> >  [<c013edcf>] do_munmap+0xef/0x150
> >  [<c0103afd>] sysenter_past_esp+0x52/0x71
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > SLAB: cache with size 64 has lost its name
> > 
> > The FSes are still consistent (fsck'ed with no --fix needed).
> > Unfortunatly, I turned off debugging options to test latest snapshot
> > performance. I can rebuild the module with debugging options on,
> > and try to give you a more accurate report to help find the memory leak.
> > 
> > Regards,
> -- 
> Lamont R. Peterson <lamont@gurulabs.com>
> Senior Instructor
> Guru Labs http://www.gurulabs.com/



-- 
Alex.

Re: Bug report: slab error at module removal

[Alex Zarochentsev]

On Sat, Aug 14, 2004 at 06:26:36AM +0200, Mathieu Segaud wrote:
> 
> I don't know if this is the right place, but I just saw a kernel message
> in my dmesg.  
> 
> The box is a test box with 7 Reiser4 FSes.
> 5 of them are LVM2 logical volumes. And the device-mapper
> is used to crypt root partition (cipher AES).
> All Reiser4 partitions have been converted to 1.0.0 layout/format,
> and were consistent at boot time.
> After several hours of running some moderatly high load stuff 
> on each of them (big transfert of data, long repacker runs...).
> I decided to umount them and unload reiser4.ko module.
> dmesg gave this message:
> 
> slab error in kmem_cache_destroy(): cache `plugin_set': Can't free all objects
>  [<c0135c41>] kmem_cache_destroy+0xd1/0x130
>  [<f99bf57a>] plugin_set_done+0xa/0x30 [reiser4]
>  [<f99b9194>] shutdown_reiser4+0xf4/0x210 [reiser4]
>  [<c012a004>] sys_delete_module+0x144/0x180
>  [<c013ea9e>] unmap_vma_list+0xe/0x20
>  [<c013edcf>] do_munmap+0xef/0x150
>  [<c0103afd>] sysenter_past_esp+0x52/0x71
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> SLAB: cache with size 64 has lost its name
> 
> The FSes are still consistent (fsck'ed with no --fix needed).
> Unfortunatly, I turned off debugging options to test latest snapshot
> performance. I can rebuild the module with debugging options on,
> and try to give you a more accurate report to help find the memory leak.

can you try the following patch:

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2004/08/15 20:07:10+04:00 zam@crimson.namesys.com 
#   plugin_set_done: first clean the hash table and free all elements, then remove the slab cache.
# 
# plugin/plugin_set.c
#   2004/08/15 20:07:07+04:00 zam@crimson.namesys.com +1 -1
#   plugin_set_done: first clean the hash table and free all elements, then remove the slab cache.
# 
diff -Nru a/plugin/plugin_set.c b/plugin/plugin_set.c
--- a/plugin/plugin_set.c	Sun Aug 15 20:12:29 2004
+++ b/plugin/plugin_set.c	Sun Aug 15 20:12:29 2004
@@ -329,8 +329,8 @@
 reiser4_internal void plugin_set_done(void)
 {
 	/* NOTE: scan hash table and recycle all objects. */
-	kmem_cache_destroy(plugin_set_slab);
 	ps_hash_done(&ps_table);
+	kmem_cache_destroy(plugin_set_slab);
 }
 
 
> 
> Regards,
> 
> -- 
> Mathieu Segaud
> 

-- 
Alex.

Re: Bug report: slab error at module removal

[Mathieu Segaud]

Alex Zarochentsev <zam@namesys.com> writes:

> On Sat, Aug 14, 2004 at 06:26:36AM +0200, Mathieu Segaud wrote:

>> slab error in kmem_cache_destroy(): cache `plugin_set': Can't free all objects
>>  [<c0135c41>] kmem_cache_destroy+0xd1/0x130
>>  [<f99bf57a>] plugin_set_done+0xa/0x30 [reiser4]
>>  [<f99b9194>] shutdown_reiser4+0xf4/0x210 [reiser4]
>>  [<c012a004>] sys_delete_module+0x144/0x180
>>  [<c013ea9e>] unmap_vma_list+0xe/0x20
>>  [<c013edcf>] do_munmap+0xef/0x150
>>  [<c0103afd>] sysenter_past_esp+0x52/0x71

This does not seem to fix every leak...
I got the same message at module removal (this time the fs had been mounted,
one of them was used to compile lirc, and each one was then umounted)

> can you try the following patch:
>
> # This is a BitKeeper generated diff -Nru style patch.
> #
> # ChangeSet
> #   2004/08/15 20:07:10+04:00 zam@crimson.namesys.com 
> #   plugin_set_done: first clean the hash table and free all elements, then remove the slab cache.
> # 
> # plugin/plugin_set.c
> #   2004/08/15 20:07:07+04:00 zam@crimson.namesys.com +1 -1
> #   plugin_set_done: first clean the hash table and free all elements, then remove the slab cache.
> # 
> diff -Nru a/plugin/plugin_set.c b/plugin/plugin_set.c
> --- a/plugin/plugin_set.c	Sun Aug 15 20:12:29 2004
> +++ b/plugin/plugin_set.c	Sun Aug 15 20:12:29 2004
> @@ -329,8 +329,8 @@
>  reiser4_internal void plugin_set_done(void)
>  {
>  	/* NOTE: scan hash table and recycle all objects. */
> -	kmem_cache_destroy(plugin_set_slab);
>  	ps_hash_done(&ps_table);
> +	kmem_cache_destroy(plugin_set_slab);
>  }
>  

I forgot again to reenable DEBUG options. I will give it a try this night.

-- 
Mathieu Segaud

Re: Bug report: slab error at module removal

[Alex Zarochentsev]

On Sun, Aug 15, 2004 at 10:01:06PM +0200, Mathieu Segaud wrote:
> Alex Zarochentsev <zam@namesys.com> writes:
> 
> > On Sat, Aug 14, 2004 at 06:26:36AM +0200, Mathieu Segaud wrote:
> 
> >> slab error in kmem_cache_destroy(): cache `plugin_set': Can't free all objects
> >>  [<c0135c41>] kmem_cache_destroy+0xd1/0x130
> >>  [<f99bf57a>] plugin_set_done+0xa/0x30 [reiser4]
> >>  [<f99b9194>] shutdown_reiser4+0xf4/0x210 [reiser4]
> >>  [<c012a004>] sys_delete_module+0x144/0x180
> >>  [<c013ea9e>] unmap_vma_list+0xe/0x20
> >>  [<c013edcf>] do_munmap+0xef/0x150
> >>  [<c0103afd>] sysenter_past_esp+0x52/0x71
> 
> This does not seem to fix every leak...

ah, sorry, the fix was wrong, try this one:

diff -Nru a/plugin/plugin_set.c b/plugin/plugin_set.c
--- a/plugin/plugin_set.c	Mon Aug 16 18:21:38 2004
+++ b/plugin/plugin_set.c	Mon Aug 16 18:21:38 2004
@@ -328,7 +328,12 @@
 
 reiser4_internal void plugin_set_done(void)
 {
-	/* NOTE: scan hash table and recycle all objects. */
+	plugin_set * cur, * next;
+
+	for_all_in_htable(&ps_table, ps, cur, next) {
+		ps_hash_remove(&ps_table, cur);
+		kmem_cache_free(plugin_set_slab, cur);
+	}
 	kmem_cache_destroy(plugin_set_slab);
 	ps_hash_done(&ps_table);
 }

> I got the same message at module removal (this time the fs had been mounted,
> one of them was used to compile lirc, and each one was then umounted)
> 
> > can you try the following patch:
> >
> > # This is a BitKeeper generated diff -Nru style patch.
> > #
> > # ChangeSet
> > #   2004/08/15 20:07:10+04:00 zam@crimson.namesys.com 
> > #   plugin_set_done: first clean the hash table and free all elements, then remove the slab cache.
> > # 
> > # plugin/plugin_set.c
> > #   2004/08/15 20:07:07+04:00 zam@crimson.namesys.com +1 -1
> > #   plugin_set_done: first clean the hash table and free all elements, then remove the slab cache.
> > # 
> > diff -Nru a/plugin/plugin_set.c b/plugin/plugin_set.c
> > --- a/plugin/plugin_set.c	Sun Aug 15 20:12:29 2004
> > +++ b/plugin/plugin_set.c	Sun Aug 15 20:12:29 2004
> > @@ -329,8 +329,8 @@
> >  reiser4_internal void plugin_set_done(void)
> >  {
> >  	/* NOTE: scan hash table and recycle all objects. */
> > -	kmem_cache_destroy(plugin_set_slab);
> >  	ps_hash_done(&ps_table);
> > +	kmem_cache_destroy(plugin_set_slab);
> >  }
> >  
> 
> I forgot again to reenable DEBUG options. I will give it a try this night.
> 
> -- 
> Mathieu Segaud
> 

-- 
Alex.

Re: Bug report: slab error at module removal

[Mathieu Segaud]

Alex Zarochentsev <zam@namesys.com> writes:

> ah, sorry, the fix was wrong, try this one:

This time, module removal issues no slab error.
Thanks a lot. It will sit in my patchset till next snapshot :-)

Regards,

-- 
Mathieu Segaud