[PATCH] slub: Avoid irqoff/on in bulk allocation

[PATCH] slub: Avoid irqoff/on in bulk allocation

[Christoph Lameter]

Use the new function that can do allocation while
interrupts are disabled.  Avoids irq on/off sequences.

Signed-off-by: Christoph Lameter <cl@linux.com>

Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c	2015-08-28 14:34:59.377234626 -0500
+++ linux/mm/slub.c	2015-08-28 14:34:59.377234626 -0500
@@ -2823,30 +2823,23 @@ bool kmem_cache_alloc_bulk(struct kmem_c
 		void *object = c->freelist;

 		if (unlikely(!object)) {
-			local_irq_enable();
 			/*
 			 * Invoking slow path likely have side-effect
 			 * of re-populating per CPU c->freelist
 			 */
-			p[i] = __slab_alloc(s, flags, NUMA_NO_NODE,
+			p[i] = ___slab_alloc(s, flags, NUMA_NO_NODE,
 					    _RET_IP_, c);
-			if (unlikely(!p[i])) {
-				__kmem_cache_free_bulk(s, i, p);
-				return false;
-			}
-			local_irq_disable();
+			if (unlikely(!p[i]))
+				goto error;
+
 			c = this_cpu_ptr(s->cpu_slab);
 			continue; /* goto for-loop */
 		}

 		/* kmem_cache debug support */
 		s = slab_pre_alloc_hook(s, flags);
-		if (unlikely(!s)) {
-			__kmem_cache_free_bulk(s, i, p);
-			c->tid = next_tid(c->tid);
-			local_irq_enable();
-			return false;
-		}
+		if (unlikely(!s))
+			goto error;

 		c->freelist = get_freepointer(s, object);
 		p[i] = object;
@@ -2866,6 +2859,11 @@ bool kmem_cache_alloc_bulk(struct kmem_c
 	}

 	return true;
+
+error:
+	__kmem_cache_free_bulk(s, i, p);
+	local_irq_enable();
+	return false;
 }
 EXPORT_SYMBOL(kmem_cache_alloc_bulk);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: Avoid irqoff/on in bulk allocation

[Jesper Dangaard Brouer]

On Fri, 28 Aug 2015 14:44:20 -0500 (CDT)
Christoph Lameter <cl@linux.com> wrote:

> Use the new function that can do allocation while
> interrupts are disabled.  Avoids irq on/off sequences.
> 
> Signed-off-by: Christoph Lameter <cl@linux.com>
> 
> Index: linux/mm/slub.c
> ===================================================================
> --- linux.orig/mm/slub.c	2015-08-28 14:34:59.377234626 -0500
> +++ linux/mm/slub.c	2015-08-28 14:34:59.377234626 -0500
> @@ -2823,30 +2823,23 @@ bool kmem_cache_alloc_bulk(struct kmem_c
>  		void *object = c->freelist;
> 
>  		if (unlikely(!object)) {
> -			local_irq_enable();
>  			/*
>  			 * Invoking slow path likely have side-effect
>  			 * of re-populating per CPU c->freelist
>  			 */
> -			p[i] = __slab_alloc(s, flags, NUMA_NO_NODE,
> +			p[i] = ___slab_alloc(s, flags, NUMA_NO_NODE,
>  					    _RET_IP_, c);
> -			if (unlikely(!p[i])) {
> -				__kmem_cache_free_bulk(s, i, p);
> -				return false;
> -			}
> -			local_irq_disable();
> +			if (unlikely(!p[i]))
> +				goto error;
> +
>  			c = this_cpu_ptr(s->cpu_slab);
>  			continue; /* goto for-loop */
>  		}
> 
>  		/* kmem_cache debug support */
>  		s = slab_pre_alloc_hook(s, flags);
> -		if (unlikely(!s)) {
> -			__kmem_cache_free_bulk(s, i, p);
> -			c->tid = next_tid(c->tid);
> -			local_irq_enable();
> -			return false;
> -		}
> +		if (unlikely(!s))
> +			goto error;
> 
>  		c->freelist = get_freepointer(s, object);
>  		p[i] = object;
> @@ -2866,6 +2859,11 @@ bool kmem_cache_alloc_bulk(struct kmem_c
>  	}
> 
>  	return true;
> +
> +error:
> +	__kmem_cache_free_bulk(s, i, p);

Don't we need to update "tid" here, like:

  c->tid = next_tid(c->tid);

Consider a call to the ordinary kmem_cache_alloc/slab_alloc_node was
in-progress, which get PREEMPT'ed just before it's call to
this_cpu_cmpxchg_double().
 Now, this function gets called and we modify c->freelist, but cannot
get all objects and then fail (goto error).  Although we put-back
objects (via __kmem_cache_free_bulk) don't we want to update c->tid
in-order to make sure the call to this_cpu_cmpxchg_double() retry?

> +	local_irq_enable();
> +	return false;
>  }
>  EXPORT_SYMBOL(kmem_cache_alloc_bulk);


-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: Avoid irqoff/on in bulk allocation

[Christoph Lameter]

On Wed, 2 Sep 2015, Jesper Dangaard Brouer wrote:

> > +error:
> > +	__kmem_cache_free_bulk(s, i, p);
>
> Don't we need to update "tid" here, like:
>
>   c->tid = next_tid(c->tid);
>
> Consider a call to the ordinary kmem_cache_alloc/slab_alloc_node was
> in-progress, which get PREEMPT'ed just before it's call to
> this_cpu_cmpxchg_double().
>  Now, this function gets called and we modify c->freelist, but cannot
> get all objects and then fail (goto error).  Although we put-back
> objects (via __kmem_cache_free_bulk) don't we want to update c->tid
> in-order to make sure the call to this_cpu_cmpxchg_double() retry?

Hmm... I thought that __kmem_cache_free_bulk is run with interrupts
disabled and will invoke the __slab_free which will increment tid if any
objects are freed from the local page.

That occurs before interrupts are reenabled.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: Avoid irqoff/on in bulk allocation

[Jesper Dangaard Brouer]

On Wed, 2 Sep 2015 13:04:08 -0500 (CDT)
Christoph Lameter <cl@linux.com> wrote:

> On Wed, 2 Sep 2015, Jesper Dangaard Brouer wrote:
> 
> > > +error:
> > > +	__kmem_cache_free_bulk(s, i, p);
> >
> > Don't we need to update "tid" here, like:
> >
> >   c->tid = next_tid(c->tid);
> >
> > Consider a call to the ordinary kmem_cache_alloc/slab_alloc_node was
> > in-progress, which get PREEMPT'ed just before it's call to
> > this_cpu_cmpxchg_double().
> >  Now, this function gets called and we modify c->freelist, but cannot
> > get all objects and then fail (goto error).  Although we put-back
> > objects (via __kmem_cache_free_bulk) don't we want to update c->tid
> > in-order to make sure the call to this_cpu_cmpxchg_double() retry?
> 
> Hmm... I thought that __kmem_cache_free_bulk is run with interrupts
> disabled and will invoke the __slab_free which will increment tid if any
> objects are freed from the local page.

Ah, yes.  Fallback __kmem_cache_free_bulk() will invoke slab_free(),
which will have updated c->tid.  The patch is correct.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] slub: Avoid irqoff/on in bulk allocation

[Andrew Morton]

On Fri, 28 Aug 2015 14:44:20 -0500 (CDT) Christoph Lameter <cl@linux.com> wrote:

> Use the new function that can do allocation while
> interrupts are disabled.  Avoids irq on/off sequences.
> 

It's going to increase worst-case irq-off times though.  By how much?  Is
it a good tradeoff?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>