[patch 1/5] slqb: irq section fix

[patch 1/5] slqb: irq section fix

[Nick Piggin]

slqb: irq section fix

flush_free_list can be called with interrupts enabled, from
kmem_cache_destroy. Fix this.

Signed-off-by: Nick Piggin <npiggin@suse.de>
---
Index: linux-2.6/mm/slqb.c
===================================================================
--- linux-2.6.orig/mm/slqb.c	2009-04-01 00:57:13.000000000 +1100
+++ linux-2.6/mm/slqb.c	2009-04-01 01:02:02.000000000 +1100
@@ -1087,7 +1087,6 @@ static void slab_free_to_remote(struct k
  */
 static void flush_free_list(struct kmem_cache *s, struct kmem_cache_list *l)
 {
-	struct kmem_cache_cpu *c;
 	void **head;
 	int nr;
 
@@ -1100,8 +1099,6 @@ static void flush_free_list(struct kmem_
 	slqb_stat_inc(l, FLUSH_FREE_LIST);
 	slqb_stat_add(l, FLUSH_FREE_LIST_OBJECTS, nr);
 
-	c = get_cpu_slab(s, smp_processor_id());
-
 	l->freelist.nr -= nr;
 	head = l->freelist.head;
 
@@ -1116,6 +1113,10 @@ static void flush_free_list(struct kmem_
 
 #ifdef CONFIG_SMP
 		if (page->list != l) {
+			struct kmem_cache_cpu *c;
+
+			c = get_cpu_slab(s, smp_processor_id());
+
 			slab_free_to_remote(s, page, object, c);
 			slqb_stat_inc(l, FLUSH_FREE_LIST_REMOTE);
 		} else
@@ -2251,6 +2252,7 @@ void kmem_cache_destroy(struct kmem_cach
 	down_write(&slqb_lock);
 	list_del(&s->list);
 
+	local_irq_disable();
 #ifdef CONFIG_SMP
 	for_each_online_cpu(cpu) {
 		struct kmem_cache_cpu *c = get_cpu_slab(s, cpu);
@@ -2297,6 +2299,7 @@ void kmem_cache_destroy(struct kmem_cach
 
 	free_kmem_cache_nodes(s);
 #endif
+	local_irq_enable();
 
 	sysfs_slab_remove(s);
 	up_write(&slqb_lock);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 2/5] slqb: fix compilation warning

[Nick Piggin]

slqb: fix compilation warning

gather_stats is not used if CONFIG_SLQB_SYSFS is not selected. Make
it conditional and avoid the warning.

Signed-off-by: Nick Piggin <npiggin@suse.de>
---
Index: linux-2.6/mm/slqb.c
===================================================================
--- linux-2.6.orig/mm/slqb.c	2009-04-01 03:11:05.000000000 +1100
+++ linux-2.6/mm/slqb.c	2009-04-01 03:11:25.000000000 +1100
@@ -3121,6 +3121,7 @@ static void gather_stats_locked(struct k
 	stats->nr_objects = stats->nr_slabs * s->objects;
 }
 
+#ifdef CONFIG_SLQB_SYSFS
 static void gather_stats(struct kmem_cache *s, struct stats_gather *stats)
 {
 	down_read(&slqb_lock); /* hold off hotplug */
@@ -3128,6 +3129,7 @@ static void gather_stats(struct kmem_cac
 	up_read(&slqb_lock);
 }
 #endif
+#endif
 
 /*
  * The /proc/slabinfo ABI

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 3/5] slqb: cap remote free list size

[Nick Piggin]

slqb: cap remote free list size

SLQB has a design flaw noticed by Yanmin Zhang when doing network packet
stress testing.

My intention with lockless slab lists had been to try to balance
producer/consumer type activity on the object queues just at allocation-time
with the producer and free-time with the consumer. But that breaks down if you
have a huge number of objects in-flight and then free them after activity is
reduced at the producer-side.

Basically just objects being allocated on CPU0 are then freed by CPU1 but then
rely on activity from CPU0 (or periodic trimming) to free them back to the page
allocator. If there is infrequent activity on CPU0, then it can take a long
time for the periodic trimming to free up unused objects.

Fix this by adding a lock to the page list queues and allowing CPU1 to do the
freeing work synchronously if queues get too large. It allows "nice"
producer/consumer type patterns to still fit within the fast object queues,
without the possibility to build up a lot of objects... The spinlock should
not be a big problem for nice workloads, as it is at least an order of
magnitude less frequent than an object allocation/free operation.

Signed-off-by: Nick Piggin <npiggin@suse.de>

Index: linux-2.6/include/linux/slqb_def.h
===================================================================
--- linux-2.6.orig/include/linux/slqb_def.h	2009-04-01 03:10:27.000000000 +1100
+++ linux-2.6/include/linux/slqb_def.h	2009-04-01 04:06:58.000000000 +1100
@@ -77,6 +77,9 @@ struct kmem_cache_list {
 				/* Total number of slabs allocated */
 	unsigned long		nr_slabs;
 
+				/* Protects nr_partial, nr_slabs, and partial */
+	spinlock_t		page_lock;
+
 #ifdef CONFIG_SMP
 	/*
 	 * In the case of per-cpu lists, remote_free is for objects freed by
Index: linux-2.6/mm/slqb.c
===================================================================
--- linux-2.6.orig/mm/slqb.c	2009-04-01 03:11:25.000000000 +1100
+++ linux-2.6/mm/slqb.c	2009-04-01 04:24:05.000000000 +1100
@@ -1089,6 +1089,7 @@ static void flush_free_list(struct kmem_
 {
 	void **head;
 	int nr;
+	int locked = 0;
 
 	nr = l->freelist.nr;
 	if (unlikely(!nr))
@@ -1115,17 +1116,31 @@ static void flush_free_list(struct kmem_
 		if (page->list != l) {
 			struct kmem_cache_cpu *c;
 
+			if (locked) {
+				spin_unlock(&l->page_lock);
+				locked = 0;
+			}
+
 			c = get_cpu_slab(s, smp_processor_id());
 
 			slab_free_to_remote(s, page, object, c);
 			slqb_stat_inc(l, FLUSH_FREE_LIST_REMOTE);
 		} else
 #endif
+		{
+			if (!locked) {
+				spin_lock(&l->page_lock);
+				locked = 1;
+			}
 			free_object_to_page(s, l, page, object);
+		}
 
 		nr--;
 	} while (nr);
 
+	if (locked)
+		spin_unlock(&l->page_lock);
+
 	l->freelist.head = head;
 	if (!l->freelist.nr)
 		l->freelist.tail = NULL;
@@ -1272,6 +1287,21 @@ static noinline void *__cache_list_get_p
 	return object;
 }
 
+static void *cache_list_get_page(struct kmem_cache *s,
+				struct kmem_cache_list *l)
+{
+	void *object;
+
+	if (unlikely(!l->nr_partial))
+		return NULL;
+
+	spin_lock(&l->page_lock);
+	object = __cache_list_get_page(s, l);
+	spin_unlock(&l->page_lock);
+
+	return object;
+}
+
 /*
  * Allocation slowpath. Allocate a new slab page from the page allocator, and
  * put it on the list's partial list. Must be followed by an allocation so
@@ -1315,12 +1345,14 @@ static noinline void *__slab_alloc_page(
 		l = &c->list;
 		page->list = l;
 
+		spin_lock(&l->page_lock);
 		l->nr_slabs++;
 		l->nr_partial++;
 		list_add(&page->lru, &l->partial);
 		slqb_stat_inc(l, ALLOC);
 		slqb_stat_inc(l, ALLOC_SLAB_NEW);
 		object = __cache_list_get_page(s, l);
+		spin_unlock(&l->page_lock);
 	} else {
 #ifdef CONFIG_NUMA
 		struct kmem_cache_node *n;
@@ -1378,7 +1410,7 @@ static void *__remote_slab_alloc_node(st
 
 	object = __cache_list_get_object(s, l);
 	if (unlikely(!object)) {
-		object = __cache_list_get_page(s, l);
+		object = cache_list_get_page(s, l);
 		if (unlikely(!object)) {
 			spin_unlock(&n->list_lock);
 			return __slab_alloc_page(s, gfpflags, node);
@@ -1441,7 +1473,7 @@ try_remote:
 	l = &c->list;
 	object = __cache_list_get_object(s, l);
 	if (unlikely(!object)) {
-		object = __cache_list_get_page(s, l);
+		object = cache_list_get_page(s, l);
 		if (unlikely(!object)) {
 			object = __slab_alloc_page(s, gfpflags, node);
 #ifdef CONFIG_NUMA
@@ -1544,6 +1576,37 @@ static void flush_remote_free_cache(stru
 
 	dst = c->remote_cache_list;
 
+	/*
+	 * Less common case, dst is filling up so free synchronously.
+	 * No point in having remote CPU free thse as it will just
+	 * free them back to the page list anyway.
+	 */
+	if (unlikely(dst->remote_free.list.nr > (slab_hiwater(s) >> 1))) {
+		void **head;
+
+		head = src->head;
+		spin_lock(&dst->page_lock);
+		do {
+			struct slqb_page *page;
+			void **object;
+
+			object = head;
+			VM_BUG_ON(!object);
+			head = get_freepointer(s, object);
+			page = virt_to_head_slqb_page(object);
+
+			free_object_to_page(s, dst, page, object);
+			nr--;
+		} while (nr);
+		spin_unlock(&dst->page_lock);
+
+		src->head = NULL;
+		src->tail = NULL;
+		src->nr = 0;
+
+		return;
+	}
+
 	spin_lock(&dst->remote_free.lock);
 
 	if (!dst->remote_free.list.head)
@@ -1598,7 +1661,7 @@ static noinline void slab_free_to_remote
 	r->tail = object;
 	r->nr++;
 
-	if (unlikely(r->nr > slab_freebatch(s)))
+	if (unlikely(r->nr >= slab_freebatch(s)))
 		flush_remote_free_cache(s, c);
 }
 #endif
@@ -1777,6 +1840,7 @@ static void init_kmem_cache_list(struct 
 	l->nr_partial		= 0;
 	l->nr_slabs		= 0;
 	INIT_LIST_HEAD(&l->partial);
+	spin_lock_init(&l->page_lock);
 
 #ifdef CONFIG_SMP
 	l->remote_free_check	= 0;
@@ -3059,6 +3123,7 @@ static void __gather_stats(void *arg)
 	int i;
 #endif
 
+	spin_lock(&l->page_lock);
 	nr_slabs = l->nr_slabs;
 	nr_partial = l->nr_partial;
 	nr_inuse = (nr_slabs - nr_partial) * s->objects;
@@ -3066,6 +3131,7 @@ static void __gather_stats(void *arg)
 	list_for_each_entry(page, &l->partial, lru) {
 		nr_inuse += page->inuse;
 	}
+	spin_unlock(&l->page_lock);
 
 	spin_lock(&gather->lock);
 	gather->nr_slabs += nr_slabs;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 4/5] slqb: config slab size

[Nick Piggin]

slqb: config slab size

Yanmin Zhang had reported performance increases in a routing stress test
with SLUB using gigantic slab sizes. The theory is either increased TLB
efficiency or reduced page allocator costs. Anyway it is trivial and
basically no overhead to add similar parameters to SLQB to experiment with.

Signed-off-by: Nick Piggin <npiggin@suse.de>
---
Index: linux-2.6/mm/slqb.c
===================================================================
--- linux-2.6.orig/mm/slqb.c	2009-04-15 02:34:09.000000000 +1000
+++ linux-2.6/mm/slqb.c	2009-04-15 02:34:40.000000000 +1000
@@ -56,6 +56,17 @@ static inline void struct_slqb_page_wron
 
 #define PG_SLQB_BIT (1 << PG_slab)
 
+/*
+ * slqb_min_order: minimum allocation order for slabs
+ */
+static int slqb_min_order = 0;
+
+/*
+ * slqb_min_objects: minimum number of objects per slab. Increasing this
+ * will increase the allocation order for slabs with larger objects
+ */
+static int slqb_min_objects = 1;
+
 #ifdef CONFIG_NUMA
 static inline int slab_numa(struct kmem_cache *s)
 {
@@ -856,9 +867,25 @@ check_slabs:
 out:
 	return 1;
 }
-
 __setup("slqb_debug", setup_slqb_debug);
 
+static int __init setup_slqb_min_order(char *str)
+{
+	get_option(&str, &slqb_min_order);
+
+	return 1;
+}
+__setup("slqb_min_order=", setup_slqb_min_order);
+
+static int __init setup_slqb_min_objects(char *str)
+{
+	get_option(&str, &slqb_min_objects);
+
+	return 1;
+}
+
+__setup("slqb_min_objects=", setup_slqb_min_objects);
+
 static unsigned long kmem_cache_flags(unsigned long objsize,
 				unsigned long flags, const char *name,
 				void (*ctor)(void *))
@@ -1758,6 +1785,8 @@ static int slab_order(int size, int max_
 		order = 0;
 	else
 		order = fls(size - 1) - PAGE_SHIFT;
+	if (order < slqb_min_order)
+		order = slqb_min_order;
 
 	while (order <= max_order) {
 		unsigned long slab_size = PAGE_SIZE << order;
@@ -1766,13 +1795,23 @@ static int slab_order(int size, int max_
 
 		objects = slab_size / size;
 		if (!objects)
-			continue;
+			goto next;
+
+		if (order < MAX_ORDER && objects < slqb_min_objects) {
+			/*
+			 * if we don't have enough objects for min_objects,
+			 * then try the next size up. Unless we have reached
+			 * our maximum possible page size.
+			 */
+			goto next;
+		}
 
 		waste = slab_size - (objects * size);
 
 		if (waste * frac <= slab_size)
 			break;
 
+next:
 		order++;
 	}
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 5/5] mm: prompt slqb default for oldconfig

[Nick Piggin]

Hi Pekka,

Well there have been reasonably significant changes both for SLQB and
SLUB that I thought it is better to wait one more round before merging
SLQB. Also, SLQB may not have been getting as much testing as it could
have in -next, due to oldconfig choosing existing config as the default.

Thanks,
Nick
--

Change Kconfig names for slab allocator choices to prod SLQB into being
the default. Hopefully increasing testing base.

Signed-off-by: Nick Piggin <npiggin@suse.de>
---
Index: linux-2.6/init/Kconfig
===================================================================
--- linux-2.6.orig/init/Kconfig	2009-04-15 02:36:05.000000000 +1000
+++ linux-2.6/init/Kconfig	2009-04-15 02:41:25.000000000 +1000
@@ -975,18 +975,23 @@ config COMPAT_BRK
 
 choice
 	prompt "Choose SLAB allocator"
-	default SLQB
+	default SLQB_ALLOCATOR
 	help
 	   This option allows to select a slab allocator.
 
-config SLAB
+config SLAB_ALLOCATOR
 	bool "SLAB"
 	help
 	  The regular slab allocator that is established and known to work
 	  well in all environments. It organizes cache hot objects in
 	  per cpu and per node queues.
 
-config SLUB
+config SLAB
+	bool
+	default y
+	depends on SLAB_ALLOCATOR
+
+config SLUB_ALLOCATOR
 	bool "SLUB (Unqueued Allocator)"
 	help
 	   SLUB is a slab allocator that minimizes cache line usage
@@ -996,11 +1001,21 @@ config SLUB
 	   and has enhanced diagnostics. SLUB is the default choice for
 	   a slab allocator.
 
-config SLQB
+config SLUB
+	bool
+	default y
+	depends on SLUB_ALLOCATOR
+
+config SLQB_ALLOCATOR
 	bool "SLQB (Queued allocator)"
 	help
 	  SLQB is a proposed new slab allocator.
 
+config SLQB
+	bool
+	default y
+	depends on SLQB_ALLOCATOR
+
 config SLOB
 	depends on EMBEDDED
 	bool "SLOB (Simple Allocator)"

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 5/5] mm: prompt slqb default for oldconfig

[Pekka Enberg]

Hi Nick,

On Tue, 2009-04-14 at 18:50 +0200, Nick Piggin wrote:
> Hi Pekka,
> 
> Well there have been reasonably significant changes both for SLQB and
> SLUB that I thought it is better to wait one more round before merging
> SLQB. Also, SLQB may not have been getting as much testing as it could
> have in -next, due to oldconfig choosing existing config as the default.
> 
> Thanks,
> Nick
> --
> 
> Change Kconfig names for slab allocator choices to prod SLQB into being
> the default. Hopefully increasing testing base.
> 
> Signed-off-by: Nick Piggin <npiggin@suse.de>

I went ahead and applied this too. IIRC, Ingo had some ideas how this
should be handled so lets CC him as well.

			Pekka

> ---
> Index: linux-2.6/init/Kconfig
> ===================================================================
> --- linux-2.6.orig/init/Kconfig	2009-04-15 02:36:05.000000000 +1000
> +++ linux-2.6/init/Kconfig	2009-04-15 02:41:25.000000000 +1000
> @@ -975,18 +975,23 @@ config COMPAT_BRK
>  
>  choice
>  	prompt "Choose SLAB allocator"
> -	default SLQB
> +	default SLQB_ALLOCATOR
>  	help
>  	   This option allows to select a slab allocator.
>  
> -config SLAB
> +config SLAB_ALLOCATOR
>  	bool "SLAB"
>  	help
>  	  The regular slab allocator that is established and known to work
>  	  well in all environments. It organizes cache hot objects in
>  	  per cpu and per node queues.
>  
> -config SLUB
> +config SLAB
> +	bool
> +	default y
> +	depends on SLAB_ALLOCATOR
> +
> +config SLUB_ALLOCATOR
>  	bool "SLUB (Unqueued Allocator)"
>  	help
>  	   SLUB is a slab allocator that minimizes cache line usage
> @@ -996,11 +1001,21 @@ config SLUB
>  	   and has enhanced diagnostics. SLUB is the default choice for
>  	   a slab allocator.
>  
> -config SLQB
> +config SLUB
> +	bool
> +	default y
> +	depends on SLUB_ALLOCATOR
> +
> +config SLQB_ALLOCATOR
>  	bool "SLQB (Queued allocator)"
>  	help
>  	  SLQB is a proposed new slab allocator.
>  
> +config SLQB
> +	bool
> +	default y
> +	depends on SLQB_ALLOCATOR
> +
>  config SLOB
>  	depends on EMBEDDED
>  	bool "SLOB (Simple Allocator)"

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 5/5] mm: prompt slqb default for oldconfig

[Ingo Molnar]

* Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> Hi Nick,
> 
> On Tue, 2009-04-14 at 18:50 +0200, Nick Piggin wrote:
> > Hi Pekka,
> > 
> > Well there have been reasonably significant changes both for SLQB and
> > SLUB that I thought it is better to wait one more round before merging
> > SLQB. Also, SLQB may not have been getting as much testing as it could
> > have in -next, due to oldconfig choosing existing config as the default.
> > 
> > Thanks,
> > Nick
> > --
> > 
> > Change Kconfig names for slab allocator choices to prod SLQB into being
> > the default. Hopefully increasing testing base.
> > 
> > Signed-off-by: Nick Piggin <npiggin@suse.de>
> 
> I went ahead and applied this too. IIRC, Ingo had some ideas how 
> this should be handled so lets CC him as well.

I forgot it all already :)

Btw., i'm wondering, why didnt this make it into .30?

	Ingo

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 5/5] mm: prompt slqb default for oldconfig

[Nick Piggin]

On Thu, Apr 16, 2009 at 11:22:38AM +0200, Ingo Molnar wrote:
> 
> * Pekka Enberg <penberg@cs.helsinki.fi> wrote:
> 
> > Hi Nick,
> > 
> > On Tue, 2009-04-14 at 18:50 +0200, Nick Piggin wrote:
> > > Hi Pekka,
> > > 
> > > Well there have been reasonably significant changes both for SLQB and
> > > SLUB that I thought it is better to wait one more round before merging
> > > SLQB. Also, SLQB may not have been getting as much testing as it could
> > > have in -next, due to oldconfig choosing existing config as the default.
> > > 
> > > Thanks,
> > > Nick
> > > --
> > > 
> > > Change Kconfig names for slab allocator choices to prod SLQB into being
> > > the default. Hopefully increasing testing base.
> > > 
> > > Signed-off-by: Nick Piggin <npiggin@suse.de>
> > 
> > I went ahead and applied this too. IIRC, Ingo had some ideas how 
> > this should be handled so lets CC him as well.
> 
> I forgot it all already :)

I think I roughly followed consensus. Basically just rename the choices
so oldconfig will default to Kconfig specified rather than the existing
.config choice. We can just revert this little shim eventually.
 

> Btw., i'm wondering, why didnt this make it into .30?

Well, there was a flurry of activity on SLUB, and also one relatively
significant change to SLQB, so in the interest of getting a more uptodate
comparison, I think it better to wait one more round.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 1/5] slqb: irq section fix

[Pekka Enberg]

On Tue, 2009-04-14 at 18:44 +0200, Nick Piggin wrote:
> slqb: irq section fix
> 
> flush_free_list can be called with interrupts enabled, from
> kmem_cache_destroy. Fix this.
> 
> Signed-off-by: Nick Piggin <npiggin@suse.de>

The series has been applied! Thanks.

			Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>