BUG: slub creates kmalloc slabs with refcount=0

BUG: slub creates kmalloc slabs with refcount=0

[Paul Hargrove]

[-- Attachment #1: Type: text/plain, Size: 1457 bytes --]

I have a 3.7.1 kernel on x86-86
It is configured with
  CONFIG_SLUB=y
  CONFIG_SLUB_DEBUG=y

I have an out-of-tree module calling KMEM_CACHE for an 8-byte struct:
        cr_pdata_cachep = KMEM_CACHE(cr_pdata_s,0);
        if (!cr_pdata_cachep) goto no_pdata_cachep;
        printk(KERN_ERR "@ refcount = %d name = '%s'\n",
cr_pdata_cachep->refcount, cr_pdata_cachep->name);

The output of the printk, below, shows that the request has been merged
with the built-in 8-byte kmalloc pool, BUT the resulting refcount is 1,
rather than 2 (or more):
    @ refcount = 1 name = 'kmalloc-8'

This results in a very unhappy kernel when the module calls
    kmem_cache_destroy(cr_pdata_cachep);
at rmmod time, resulting is messages like
    BUG kmalloc-8 (Tainted: G           O): Objects remaining in kmalloc-96
on kmem_cache_close()

A quick look through mm/slub.c appears to confirm my suspicion that
"s->refcount" is never incremented for the built-in kmalloc-* caches.
 However, I leave it to the experts to determine where the increment
belongs.

FWIW: I am currently passing SLAB_POISON for the flags argument to
KMEM_CACHE() as a work-around (it prevents merging and, if I understand
correctly, has no overhead in a non-debug build).

-Paul

-- 
Paul H. Hargrove                          PHHargrove@lbl.gov
Future Technologies Group
Computer and Data Sciences Department     Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900

[-- Attachment #2: Type: text/html, Size: 2092 bytes --]

Re: BUG: slub creates kmalloc slabs with refcount=0

[Paul Hargrove]

[-- Attachment #1: Type: text/plain, Size: 2319 bytes --]

Just to be clear the "BUG...." line I quoted contains an error of my own.

It should say "kmalloc-8" twice rather than having "kmalloc-96" toward the
end.  This is the result of a cut-and-paste error (and brain already on
vacation).  The module ALSO has a larger struct the cache for which gets
merged with the kmalloc-96 cache.  I grabbed the first BUG line from the
dmesg output and replaced "96" with "8" when composing the email, but
missed the second occurrence.

Merry Christmas,
-Paul


On Mon, Dec 24, 2012 at 4:55 PM, Paul Hargrove <phhargrove@lbl.gov> wrote:

>
> I have a 3.7.1 kernel on x86-86
> It is configured with
>   CONFIG_SLUB=y
>   CONFIG_SLUB_DEBUG=y
>
> I have an out-of-tree module calling KMEM_CACHE for an 8-byte struct:
>         cr_pdata_cachep = KMEM_CACHE(cr_pdata_s,0);
>         if (!cr_pdata_cachep) goto no_pdata_cachep;
>         printk(KERN_ERR "@ refcount = %d name = '%s'\n",
> cr_pdata_cachep->refcount, cr_pdata_cachep->name);
>
> The output of the printk, below, shows that the request has been merged
> with the built-in 8-byte kmalloc pool, BUT the resulting refcount is 1,
> rather than 2 (or more):
>     @ refcount = 1 name = 'kmalloc-8'
>
> This results in a very unhappy kernel when the module calls
>     kmem_cache_destroy(cr_pdata_cachep);
> at rmmod time, resulting is messages like
>     BUG kmalloc-8 (Tainted: G           O): Objects remaining in
> kmalloc-96 on kmem_cache_close()
>
> A quick look through mm/slub.c appears to confirm my suspicion that
> "s->refcount" is never incremented for the built-in kmalloc-* caches.
>  However, I leave it to the experts to determine where the increment
> belongs.
>
> FWIW: I am currently passing SLAB_POISON for the flags argument to
> KMEM_CACHE() as a work-around (it prevents merging and, if I understand
> correctly, has no overhead in a non-debug build).
>
> -Paul
>
> --
> Paul H. Hargrove                          PHHargrove@lbl.gov
> Future Technologies Group
> Computer and Data Sciences Department     Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900
>



-- 
Paul H. Hargrove                          PHHargrove@lbl.gov
Future Technologies Group
Computer and Data Sciences Department     Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900

[-- Attachment #2: Type: text/html, Size: 3586 bytes --]

[PATCH] slub: assign refcount for kmalloc_caches

[Joonsoo Kim]

commit cce89f4f6911286500cf7be0363f46c9b0a12ce0('Move kmem_cache
refcounting to common code') moves some refcount manipulation code to
common code. Unfortunately, it also removed refcount assignment for
kmalloc_caches. So, kmalloc_caches's refcount is initially 0.
This makes errornous situation.

Paul Hargrove report that when he create a 8-byte kmem_cache and
destory it, he encounter below message.
'Objects remaining in kmalloc-8 on kmem_cache_close()'

8-byte kmem_cache merge with 8-byte kmalloc cache and refcount is
increased by one. So, resulting refcount is 1. When destory it, it hit
refcount = 0, then kmem_cache_close() is executed and error message is
printed.

This patch assign initial refcount 1 to kmalloc_caches, so fix this
errornous situtation.

Cc: <stable@vger.kernel.org> # v3.7
Cc: Christoph Lameter <cl@linux.com>
Reported-by: Paul Hargrove <phhargrove@lbl.gov>
Signed-off-by: Joonsoo Kim <js1304@gmail.com>

diff --git a/mm/slub.c b/mm/slub.c
index a0d6984..321afab 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3279,6 +3279,7 @@ static struct kmem_cache *__init create_kmalloc_cache(const char *name,
 	if (kmem_cache_open(s, flags))
 		goto panic;
 
+	s->refcount = 1;
 	list_add(&s->list, &slab_caches);
 	return s;
 
-- 
1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: assign refcount for kmalloc_caches

[JoonSoo Kim]

2012/12/26 Joonsoo Kim <js1304@gmail.com>:
> commit cce89f4f6911286500cf7be0363f46c9b0a12ce0('Move kmem_cache
> refcounting to common code') moves some refcount manipulation code to
> common code. Unfortunately, it also removed refcount assignment for
> kmalloc_caches. So, kmalloc_caches's refcount is initially 0.
> This makes errornous situation.
>
> Paul Hargrove report that when he create a 8-byte kmem_cache and
> destory it, he encounter below message.
> 'Objects remaining in kmalloc-8 on kmem_cache_close()'
>
> 8-byte kmem_cache merge with 8-byte kmalloc cache and refcount is
> increased by one. So, resulting refcount is 1. When destory it, it hit
> refcount = 0, then kmem_cache_close() is executed and error message is
> printed.
>
> This patch assign initial refcount 1 to kmalloc_caches, so fix this
> errornous situtation.
>
> Cc: <stable@vger.kernel.org> # v3.7
> Cc: Christoph Lameter <cl@linux.com>
> Reported-by: Paul Hargrove <phhargrove@lbl.gov>
> Signed-off-by: Joonsoo Kim <js1304@gmail.com>
>
> diff --git a/mm/slub.c b/mm/slub.c
> index a0d6984..321afab 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3279,6 +3279,7 @@ static struct kmem_cache *__init create_kmalloc_cache(const char *name,
>         if (kmem_cache_open(s, flags))
>                 goto panic;
>
> +       s->refcount = 1;
>         list_add(&s->list, &slab_caches);
>         return s;
>
> --
> 1.7.9.5
>

I missed some explanation.
In v3.8-rc1, this problem is already solved.
See create_kmalloc_cache() in mm/slab_common.c.
So this patch is just for v3.7 stable.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: assign refcount for kmalloc_caches

[Paul Hargrove]

[-- Attachment #1: Type: text/plain, Size: 2073 bytes --]

I just had a look at patch-3.7.2-rc1, and this change doesn't appear to
have made it in yet.
Am I missing something?

-Paul


On Tue, Dec 25, 2012 at 7:30 AM, JoonSoo Kim <js1304@gmail.com> wrote:

> 2012/12/26 Joonsoo Kim <js1304@gmail.com>:
> > commit cce89f4f6911286500cf7be0363f46c9b0a12ce0('Move kmem_cache
> > refcounting to common code') moves some refcount manipulation code to
> > common code. Unfortunately, it also removed refcount assignment for
> > kmalloc_caches. So, kmalloc_caches's refcount is initially 0.
> > This makes errornous situation.
> >
> > Paul Hargrove report that when he create a 8-byte kmem_cache and
> > destory it, he encounter below message.
> > 'Objects remaining in kmalloc-8 on kmem_cache_close()'
> >
> > 8-byte kmem_cache merge with 8-byte kmalloc cache and refcount is
> > increased by one. So, resulting refcount is 1. When destory it, it hit
> > refcount = 0, then kmem_cache_close() is executed and error message is
> > printed.
> >
> > This patch assign initial refcount 1 to kmalloc_caches, so fix this
> > errornous situtation.
> >
> > Cc: <stable@vger.kernel.org> # v3.7
> > Cc: Christoph Lameter <cl@linux.com>
> > Reported-by: Paul Hargrove <phhargrove@lbl.gov>
> > Signed-off-by: Joonsoo Kim <js1304@gmail.com>
> >
> > diff --git a/mm/slub.c b/mm/slub.c
> > index a0d6984..321afab 100644
> > --- a/mm/slub.c
> > +++ b/mm/slub.c
> > @@ -3279,6 +3279,7 @@ static struct kmem_cache *__init
> create_kmalloc_cache(const char *name,
> >         if (kmem_cache_open(s, flags))
> >                 goto panic;
> >
> > +       s->refcount = 1;
> >         list_add(&s->list, &slab_caches);
> >         return s;
> >
> > --
> > 1.7.9.5
> >
>
> I missed some explanation.
> In v3.8-rc1, this problem is already solved.
> See create_kmalloc_cache() in mm/slab_common.c.
> So this patch is just for v3.7 stable.
>



-- 
Paul H. Hargrove                          PHHargrove@lbl.gov
Future Technologies Group
Computer and Data Sciences Department     Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900

[-- Attachment #2: Type: text/html, Size: 3094 bytes --]

Re: [PATCH] slub: assign refcount for kmalloc_caches

[Joonsoo Kim]

On Thu, Jan 10, 2013 at 08:47:39PM -0800, Paul Hargrove wrote:
> I just had a look at patch-3.7.2-rc1, and this change doesn't appear to
> have made it in yet.
> Am I missing something?
> 
> -Paul

I try to check it.
Ccing to Greg.

Hello, Pekka and Greg.

v3.8-rcX has already fixed by another stuff, but it is not simple change.
So I made a new patch and sent it.

How this kind of patch (only for stable v3.7) go into stable tree?
through Pekka's slab tree? or send it to Greg, directly?

I don't know how to submit this kind of patch to stable tree exactly.
Could anyone help me?

Thanks.

> On Tue, Dec 25, 2012 at 7:30 AM, JoonSoo Kim <js1304@gmail.com> wrote:
> 
> > 2012/12/26 Joonsoo Kim <js1304@gmail.com>:
> > > commit cce89f4f6911286500cf7be0363f46c9b0a12ce0('Move kmem_cache
> > > refcounting to common code') moves some refcount manipulation code to
> > > common code. Unfortunately, it also removed refcount assignment for
> > > kmalloc_caches. So, kmalloc_caches's refcount is initially 0.
> > > This makes errornous situation.
> > >
> > > Paul Hargrove report that when he create a 8-byte kmem_cache and
> > > destory it, he encounter below message.
> > > 'Objects remaining in kmalloc-8 on kmem_cache_close()'
> > >
> > > 8-byte kmem_cache merge with 8-byte kmalloc cache and refcount is
> > > increased by one. So, resulting refcount is 1. When destory it, it hit
> > > refcount = 0, then kmem_cache_close() is executed and error message is
> > > printed.
> > >
> > > This patch assign initial refcount 1 to kmalloc_caches, so fix this
> > > errornous situtation.
> > >
> > > Cc: <stable@vger.kernel.org> # v3.7
> > > Cc: Christoph Lameter <cl@linux.com>
> > > Reported-by: Paul Hargrove <phhargrove@lbl.gov>
> > > Signed-off-by: Joonsoo Kim <js1304@gmail.com>
> > >
> > > diff --git a/mm/slub.c b/mm/slub.c
> > > index a0d6984..321afab 100644
> > > --- a/mm/slub.c
> > > +++ b/mm/slub.c
> > > @@ -3279,6 +3279,7 @@ static struct kmem_cache *__init
> > create_kmalloc_cache(const char *name,
> > >         if (kmem_cache_open(s, flags))
> > >                 goto panic;
> > >
> > > +       s->refcount = 1;
> > >         list_add(&s->list, &slab_caches);
> > >         return s;
> > >
> > > --
> > > 1.7.9.5
> > >
> >
> > I missed some explanation.
> > In v3.8-rc1, this problem is already solved.
> > See create_kmalloc_cache() in mm/slab_common.c.
> > So this patch is just for v3.7 stable.
> >
> 
> 
> 
> -- 
> Paul H. Hargrove                          PHHargrove@lbl.gov
> Future Technologies Group
> Computer and Data Sciences Department     Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: assign refcount for kmalloc_caches

[Greg Kroah-Hartman]

On Fri, Jan 11, 2013 at 04:52:54PM +0900, Joonsoo Kim wrote:
> On Thu, Jan 10, 2013 at 08:47:39PM -0800, Paul Hargrove wrote:
> > I just had a look at patch-3.7.2-rc1, and this change doesn't appear to
> > have made it in yet.
> > Am I missing something?
> > 
> > -Paul
> 
> I try to check it.
> Ccing to Greg.
> 
> Hello, Pekka and Greg.
> 
> v3.8-rcX has already fixed by another stuff, but it is not simple change.
> So I made a new patch and sent it.
> 
> How this kind of patch (only for stable v3.7) go into stable tree?
> through Pekka's slab tree? or send it to Greg, directly?
> 
> I don't know how to submit this kind of patch to stable tree exactly.
> Could anyone help me?

Please redo it, and send it to stable@vger.kernel.org, and say exactly
why it isn't in Linus's tree, and that it should only be applied to
3.7-stable.

thanks,

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: assign refcount for kmalloc_caches

[CAI Qian]

----- Original Message -----
> From: "Greg Kroah-Hartman" <gregkh@linuxfoundation.org>
> To: "Joonsoo Kim" <iamjoonsoo.kim@lge.com>
> Cc: "Paul Hargrove" <phhargrove@lbl.gov>, "Pekka Enberg" <penberg@kernel.org>, linux-kernel@vger.kernel.org,
> linux-mm@kvack.org, "Christoph Lameter" <cl@linux.com>
> Sent: Tuesday, January 15, 2013 3:23:36 AM
> Subject: Re: [PATCH] slub: assign refcount for kmalloc_caches
> 
> On Fri, Jan 11, 2013 at 04:52:54PM +0900, Joonsoo Kim wrote:
> > On Thu, Jan 10, 2013 at 08:47:39PM -0800, Paul Hargrove wrote:
> > > I just had a look at patch-3.7.2-rc1, and this change doesn't
> > > appear to
> > > have made it in yet.
> > > Am I missing something?
> > > 
> > > -Paul
> > 
> > I try to check it.
> > Ccing to Greg.
> > 
> > Hello, Pekka and Greg.
> > 
> > v3.8-rcX has already fixed by another stuff, but it is not simple
> > change.
> > So I made a new patch and sent it.
> > 
> > How this kind of patch (only for stable v3.7) go into stable tree?
> > through Pekka's slab tree? or send it to Greg, directly?
> > 
> > I don't know how to submit this kind of patch to stable tree
> > exactly.
> > Could anyone help me?
> 
> Please redo it, and send it to stable@vger.kernel.org, and say
> exactly
> why it isn't in Linus's tree, and that it should only be applied to
> 3.7-stable.
I also met this during the testing, so I'll re-send it then.
> 
> thanks,
> 
> greg k-h
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: assign refcount for kmalloc_caches

[Joonsoo Kim]

On Thu, Jan 24, 2013 at 10:32:32PM -0500, CAI Qian wrote:
> 
> 
> ----- Original Message -----
> > From: "Greg Kroah-Hartman" <gregkh@linuxfoundation.org>
> > To: "Joonsoo Kim" <iamjoonsoo.kim@lge.com>
> > Cc: "Paul Hargrove" <phhargrove@lbl.gov>, "Pekka Enberg" <penberg@kernel.org>, linux-kernel@vger.kernel.org,
> > linux-mm@kvack.org, "Christoph Lameter" <cl@linux.com>
> > Sent: Tuesday, January 15, 2013 3:23:36 AM
> > Subject: Re: [PATCH] slub: assign refcount for kmalloc_caches
> > 
> > On Fri, Jan 11, 2013 at 04:52:54PM +0900, Joonsoo Kim wrote:
> > > On Thu, Jan 10, 2013 at 08:47:39PM -0800, Paul Hargrove wrote:
> > > > I just had a look at patch-3.7.2-rc1, and this change doesn't
> > > > appear to
> > > > have made it in yet.
> > > > Am I missing something?
> > > > 
> > > > -Paul
> > > 
> > > I try to check it.
> > > Ccing to Greg.
> > > 
> > > Hello, Pekka and Greg.
> > > 
> > > v3.8-rcX has already fixed by another stuff, but it is not simple
> > > change.
> > > So I made a new patch and sent it.
> > > 
> > > How this kind of patch (only for stable v3.7) go into stable tree?
> > > through Pekka's slab tree? or send it to Greg, directly?
> > > 
> > > I don't know how to submit this kind of patch to stable tree
> > > exactly.
> > > Could anyone help me?
> > 
> > Please redo it, and send it to stable@vger.kernel.org, and say
> > exactly
> > why it isn't in Linus's tree, and that it should only be applied to
> > 3.7-stable.
> I also met this during the testing, so I'll re-send it then.

Hello, CAI Qian.

I totally forget this.
Thanks for this work.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: assign refcount for kmalloc_caches

[Christoph Lameter]

On Wed, 26 Dec 2012, Joonsoo Kim wrote:

> This patch assign initial refcount 1 to kmalloc_caches, so fix this
> errornous situtation.

Ok Only for 3.7:

Acked-by: Christoph Lameter <cl@linux.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: BUG: slub creates kmalloc slabs with refcount=0

[JoonSoo Kim]

Hello, Paul.

2012/12/25 Paul Hargrove <phhargrove@lbl.gov>:
>
> I have a 3.7.1 kernel on x86-86
> It is configured with
>   CONFIG_SLUB=y
>   CONFIG_SLUB_DEBUG=y
>
> I have an out-of-tree module calling KMEM_CACHE for an 8-byte struct:
>         cr_pdata_cachep = KMEM_CACHE(cr_pdata_s,0);
>         if (!cr_pdata_cachep) goto no_pdata_cachep;
>         printk(KERN_ERR "@ refcount = %d name = '%s'\n",
> cr_pdata_cachep->refcount, cr_pdata_cachep->name);
>
> The output of the printk, below, shows that the request has been merged with
> the built-in 8-byte kmalloc pool, BUT the resulting refcount is 1, rather
> than 2 (or more):
>     @ refcount = 1 name = 'kmalloc-8'
>
> This results in a very unhappy kernel when the module calls
>     kmem_cache_destroy(cr_pdata_cachep);
> at rmmod time, resulting is messages like
>     BUG kmalloc-8 (Tainted: G           O): Objects remaining in kmalloc-96
> on kmem_cache_close()
>
> A quick look through mm/slub.c appears to confirm my suspicion that
> "s->refcount" is never incremented for the built-in kmalloc-* caches.
> However, I leave it to the experts to determine where the increment belongs.
>
> FWIW: I am currently passing SLAB_POISON for the flags argument to
> KMEM_CACHE() as a work-around (it prevents merging and, if I understand
> correctly, has no overhead in a non-debug build).
>
> -Paul
>
> --
> Paul H. Hargrove                          PHHargrove@lbl.gov
> Future Technologies Group
> Computer and Data Sciences Department     Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900

My e-mail client's 'Reply to message ID' is not working properly.
I sent a patch('slub:assign refcount for kmalloc_caches') for fixing
this and Cc'ed you.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>