[PATCH v3 0/2] mm/slub: batch partial slab list operations

[PATCH v3 0/2] mm/slub: batch partial slab list operations

[Hao Li]

The loops in get_partial_node_bulk() and __refill_objects_node() perform
individual list operations for each slab. This patchset optimize this.

Patch 1 is a cleanup patch that wraps duplicated code into helper
functions.

Patch 2 reduces the number of list operations by processing slabs in
batches, thereby minimizing the time spent inside the critical section.

Changes in v3:
- Reorder the two patches for better logical flow. (Thanks Vlastimil)
- Introduce set_node_partial_state for symmetry. (Thanks Vlastimil)

Changes in v2:
- Patch 1 applies the same optimization to __refill_objects_node. (Thanks
  Harry and Vlastimil)
- Patch 2 introduces a helper to wraps duplicated code. (Thanks Harry)

Hao Li (2):
  mm/slub: introduce helpers for node partial slab state
  mm/slub: detach and reattach partial slabs in batch

 mm/slub.c | 51 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 37 insertions(+), 14 deletions(-)

base-commit: 6b41dc6749232c14f3e069f8115084a7bbeee402
-- 
2.54.0

[PATCH v3 1/2] mm/slub: introduce helpers for node partial slab state

[Hao Li]

Wrap partial slab count inc/dec and flag set/clear into
helper functions to reduce code duplication.

Note that __add_partial() is called locklessly in
early_kmem_cache_node_alloc(), but since there is no such use case for
removal, __remove_partial() does not exist.

Suggested-by: Harry Yoo <harry@kernel.org>
Signed-off-by: Hao Li <hao.li@linux.dev>
---
 mm/slub.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index b9f10b869914..45aa0a834f97 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3521,15 +3521,21 @@ static inline void slab_clear_node_partial(struct slab *slab)
 /*
  * Management of partially allocated slabs.
  */
+static inline void set_node_partial_state(struct kmem_cache_node *n,
+					struct slab *slab)
+{
+	slab_set_node_partial(slab);
+	n->nr_partial++;
+}
+
 static inline void
 __add_partial(struct kmem_cache_node *n, struct slab *slab, enum add_mode mode)
 {
-	n->nr_partial++;
 	if (mode == ADD_TO_TAIL)
 		list_add_tail(&slab->slab_list, &n->partial);
 	else
 		list_add(&slab->slab_list, &n->partial);
-	slab_set_node_partial(slab);
+	set_node_partial_state(n, slab);
 }
 
 static inline void add_partial(struct kmem_cache_node *n,
@@ -3539,13 +3545,19 @@ static inline void add_partial(struct kmem_cache_node *n,
 	__add_partial(n, slab, mode);
 }
 
+static inline void clear_node_partial_state(struct kmem_cache_node *n,
+					struct slab *slab)
+{
+	slab_clear_node_partial(slab);
+	n->nr_partial--;
+}
+
 static inline void remove_partial(struct kmem_cache_node *n,
 					struct slab *slab)
 {
 	lockdep_assert_held(&n->list_lock);
 	list_del(&slab->slab_list);
-	slab_clear_node_partial(slab);
-	n->nr_partial--;
+	clear_node_partial_state(n, slab);
 }
 
 /*
@@ -8271,8 +8283,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s)
 
 			if (free == slab->objects) {
 				list_move(&slab->slab_list, &discard);
-				slab_clear_node_partial(slab);
-				n->nr_partial--;
+				clear_node_partial_state(n, slab);
 				dec_slabs_node(s, node, slab->objects);
 			} else if (free <= SHRINK_PROMOTE_MAX)
 				list_move(&slab->slab_list, promote + free - 1);
-- 
2.54.0

[PATCH v3 2/2] mm/slub: detach and reattach partial slabs in batch

[Hao Li]

get_partial_node_bulk() moves each selected slab from the node's
partial list to the local pc->slabs list using a remove_partial() and
list_add() pair. In practice, the loop often detaches several adjacent
slabs. Doing this individually repeatedly manipulates list pointers
while holding n->list_lock, which causes unnecessary churn.

To demonstrate this, the counts below show how often single vs. multiple
consecutive slabs are retrieved during a will-it-scale mmap stress test:

consecutive_slabs_count        frequency
= 1                            277345324
= 2                            335238023
= 3                            175717884
>= 4                           88862337

The data confirms that retrieving multiple contiguous slabs is highly
frequent.

To optimize this, track contiguous runs of matching slabs and move each
run in a single operation using list_bulk_move_tail(). This reduces list
pointer churn inside the lock critical section.

Apply the same optimization to __refill_objects_node() when reattaching
leftover partial slabs back to the node's partial list.

The will-it-scale mmap benchmark shows a 2% ~ 5% performance improvement
after applying this patch.

Signed-off-by: Hao Li <hao.li@linux.dev>
---
 mm/slub.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 45aa0a834f97..4fdf5c36a69b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3751,6 +3751,7 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 				  bool allow_spin)
 {
 	struct slab *slab, *slab2;
+	struct slab *first = NULL, *last = NULL;
 	unsigned int total_free = 0;
 	unsigned long flags;
 
@@ -3769,8 +3770,15 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 		struct freelist_counters flc;
 		unsigned int slab_free;
 
-		if (!pfmemalloc_match(slab, pc->flags))
+		if (!pfmemalloc_match(slab, pc->flags)) {
+			if (first) {
+				list_bulk_move_tail(&pc->slabs,
+						    &first->slab_list,
+						    &last->slab_list);
+				first = NULL;
+			}
 			continue;
+		}
 
 		/*
 		 * determine the number of free objects in the slab racily
@@ -3787,15 +3795,20 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 		    && total_free + slab_free > pc->max_objects)
 			break;
 
-		remove_partial(n, slab);
-
-		list_add(&slab->slab_list, &pc->slabs);
+		if (!first)
+			first = slab;
+		last = slab;
+		clear_node_partial_state(n, slab);
 
 		total_free += slab_free;
 		if (total_free >= pc->max_objects)
 			break;
 	}
 
+	if (first)
+		list_bulk_move_tail(&pc->slabs, &first->slab_list,
+				    &last->slab_list);
+
 	spin_unlock_irqrestore(&n->list_lock, flags);
 	return total_free > 0;
 }
@@ -7206,11 +7219,10 @@ __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int mi
 	if (!list_empty(&pc.slabs)) {
 		spin_lock_irqsave(&n->list_lock, flags);
 
-		list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) {
+		list_for_each_entry(slab, &pc.slabs, slab_list)
+			set_node_partial_state(n, slab);
 
-			list_del(&slab->slab_list);
-			add_partial(n, slab, ADD_TO_TAIL);
-		}
+		list_splice_tail(&pc.slabs, &n->partial);
 
 		spin_unlock_irqrestore(&n->list_lock, flags);
 	}
-- 
2.54.0

Re: [PATCH v3 0/2] mm/slub: batch partial slab list operations

[Vlastimil Babka (SUSE)]

On 5/29/26 05:50, Hao Li wrote:
> The loops in get_partial_node_bulk() and __refill_objects_node() perform
> individual list operations for each slab. This patchset optimize this.
> 
> Patch 1 is a cleanup patch that wraps duplicated code into helper
> functions.
> 
> Patch 2 reduces the number of list operations by processing slabs in
> batches, thereby minimizing the time spent inside the critical section.
> 
> Changes in v3:
> - Reorder the two patches for better logical flow. (Thanks Vlastimil)
> - Introduce set_node_partial_state for symmetry. (Thanks Vlastimil)
> 
> Changes in v2:
> - Patch 1 applies the same optimization to __refill_objects_node. (Thanks
>   Harry and Vlastimil)
> - Patch 2 introduces a helper to wraps duplicated code. (Thanks Harry)
> 
> Hao Li (2):
>   mm/slub: introduce helpers for node partial slab state
>   mm/slub: detach and reattach partial slabs in batch
> 
>  mm/slub.c | 51 +++++++++++++++++++++++++++++++++++++--------------
>  1 file changed, 37 insertions(+), 14 deletions(-)

Applied to slab/for-next, thanks!

> base-commit: 6b41dc6749232c14f3e069f8115084a7bbeee402