[PATCH] slab: optimize constant-size kzalloc calls

[PATCH] slab: optimize constant-size kzalloc calls

[Pekka Enberg]

From: Pekka Enberg <penberg@cs.helsinki.fi>

As suggested by Eric Dumazet, this patch optimizes kzalloc() calls
that pass a compile-time constant size. Please note that the patch
increases kernel text slightly (~200 bytes for defconfig on x86).

This patch requires the kmem_cache_zalloc() patches I sent earlier.

Signed-off-by: Pekka Enberg <penberg@cs.helsinki.fi>

---

 include/linux/slab.h |   30 +++++++++++++++++++++++++++---
 mm/util.c            |    6 +++---
 2 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index b595c09..db3b302 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -109,7 +109,30 @@ found:
 	return __kmalloc(size, flags);
 }
 
-extern void *kzalloc(size_t, gfp_t);
+extern void *__kzalloc(size_t, gfp_t);
+
+static inline void *kzalloc(size_t size, gfp_t flags)
+{
+	if (__builtin_constant_p(size)) {
+		int i = 0;
+#define CACHE(x) \
+		if (size <= x) \
+			goto found; \
+		else \
+			i++;
+#include "kmalloc_sizes.h"
+#undef CACHE
+		{
+			extern void __you_cannot_kzalloc_that_much(void);
+			__you_cannot_kzalloc_that_much();
+		}
+found:
+		return kmem_cache_zalloc((flags & GFP_DMA) ?
+			malloc_sizes[i].cs_dmacachep :
+			malloc_sizes[i].cs_cachep, flags);
+	}
+	return __kzalloc(size, flags);
+}
 
 /**
  * kcalloc - allocate memory for an array. The memory is set to zero.
@@ -160,14 +183,14 @@ void *kmem_cache_zalloc(struct kmem_cach
 void kmem_cache_free(struct kmem_cache *c, void *b);
 const char *kmem_cache_name(struct kmem_cache *);
 void *kmalloc(size_t size, gfp_t flags);
-void *kzalloc(size_t size, gfp_t flags);
+void *__kzalloc(size_t size, gfp_t flags);
 void kfree(const void *m);
 unsigned int ksize(const void *m);
 unsigned int kmem_cache_size(struct kmem_cache *c);
 
 static inline void *kcalloc(size_t n, size_t size, gfp_t flags)
 {
-	return kzalloc(n * size, flags);
+	return __kzalloc(n * size, flags);
 }
 
 #define kmem_cache_shrink(d) (0)
@@ -175,6 +198,7 @@ static inline void *kcalloc(size_t n, si
 #define kmem_ptr_validate(a, b) (0)
 #define kmem_cache_alloc_node(c, f, n) kmem_cache_alloc(c, f)
 #define kmalloc_node(s, f, n) kmalloc(s, f)
+#define kzalloc(s, f) __kzalloc(s, f)
 
 #endif /* CONFIG_SLOB */
 
diff --git a/mm/util.c b/mm/util.c
index 5f4bb59..fd78ee4 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -3,18 +3,18 @@
 #include <linux/module.h>
 
 /**
- * kzalloc - allocate memory. The memory is set to zero.
+ * __kzalloc - allocate memory. The memory is set to zero.
  * @size: how many bytes of memory are required.
  * @flags: the type of memory to allocate.
  */
-void *kzalloc(size_t size, gfp_t flags)
+void *__kzalloc(size_t size, gfp_t flags)
 {
 	void *ret = kmalloc(size, flags);
 	if (ret)
 		memset(ret, 0, size);
 	return ret;
 }
-EXPORT_SYMBOL(kzalloc);
+EXPORT_SYMBOL(__kzalloc);
 
 /*
  * kstrdup - allocate space for and copy an existing string

Re: [PATCH] slab: optimize constant-size kzalloc calls

[Andrew Morton]

Pekka Enberg <penberg@cs.helsinki.fi> wrote:
>
> As suggested by Eric Dumazet, this patch optimizes kzalloc() calls
>  that pass a compile-time constant size.

Logical.

> Please note that the patch
>  increases kernel text slightly (~200 bytes for defconfig on x86)

Why?

Re: [PATCH] slab: optimize constant-size kzalloc calls

[Pekka J Enberg]

Pekka Enberg <penberg@cs.helsinki.fi> wrote:
> > Please note that the patch
> >  increases kernel text slightly (~200 bytes for defconfig on x86)

On Tue, 21 Mar 2006, Andrew Morton wrote:
> Why?

Because of the malloc_sizes lookup. As kindly explained by Eric, we're 
now doing:

	kzalloc(100, GFP_KERNEL);

and with the patch, we will do:

	kmem_cache_zalloc(malloc_sizes[4].cs_dmacachep, GFP_KERNEL);

which explains the the difference. The end result should be faster 
kzalloc(), though.

				Pekka

Re: [PATCH] slab: optimize constant-size kzalloc calls

[Horms]

Hi,

I feel like I mist be dreaming, but this patch, which was inlcuded
in Linus' tree as 40c07ae8daa659b8feb149c84731629386873c16 calls
__you_cannot_kzalloc_that_much(), but that does not seem to exist.

On i386 at least that causes a build failure

# make mrproper
# make allyesconfig
# make 
[snip]
drivers/built-in.o: In function `kzalloc':include/linux/slab.h:128: undefined reference to `__you_cannot_kzalloc_that_much'
:include/linux/slab.h:128: undefined reference to
`__you_cannot_kzalloc_that_much'
make: *** [.tmp_vmlinux1] Error 1

-- 
Horms

Re: [PATCH] slab: optimize constant-size kzalloc calls

[David S. Miller]

From: Horms <horms@verge.net.au>
Date: Wed, 29 Mar 2006 10:57:46 +0900

> I feel like I mist be dreaming, but this patch, which was inlcuded
> in Linus' tree as 40c07ae8daa659b8feb149c84731629386873c16 calls
> __you_cannot_kzalloc_that_much(), but that does not seem to exist.
> 
> On i386 at least that causes a build failure

It's a purposeful build time error introduced so that invalid calls
that specify too large kzalloc() length arguments are caught at build
time.

Re: [PATCH] slab: optimize constant-size kzalloc calls

[Horms]

On Tue, Mar 28, 2006 at 06:06:05PM -0800, David S. Miller wrote:
> From: Horms <horms@verge.net.au>
> Date: Wed, 29 Mar 2006 10:57:46 +0900
> 
> > I feel like I mist be dreaming, but this patch, which was inlcuded
> > in Linus' tree as 40c07ae8daa659b8feb149c84731629386873c16 calls
> > __you_cannot_kzalloc_that_much(), but that does not seem to exist.
> > 
> > On i386 at least that causes a build failure
> 
> It's a purposeful build time error introduced so that invalid calls
> that specify too large kzalloc() length arguments are caught at build
> time.

Thanks, I knew that I had to be missing the point somewhere.
I'll focus my attention on the caller.

-- 
Horms