[PATCH v5] mm/slub: use empty sheaf helpers for oversized sheaves

[PATCH v5] mm/slub: use empty sheaf helpers for oversized sheaves

[hu.shengming]

From: Shengming Hu <hu.shengming@zte.com.cn>

Oversized prefilled sheaves are allocated separately because their
capacity can be larger than the cache's regular sheaf capacity. After
they are flushed, however, they are empty sheaves as well, and should be
released through the same empty-sheaf helper.

Allocate oversized prefilled sheaves with __alloc_empty_sheaf() and free
them with free_empty_sheaf() after a failed prefill or after they are
returned and flushed. This keeps the oversized and pfmemalloc return paths
consistent, including the SLAB_KMALLOC-specific __GFP_NO_OBJ_EXT and
mark_obj_codetag_empty() handling.

Keep the caller-GFP filtering in alloc_empty_sheaf() instead of
__alloc_empty_sheaf(). In particular, do not clear OBJCGS_CLEAR_MASK in
the raw helper, so the oversized prefill path does not unexpectedly drop
caller-provided flags such as __GFP_NOFAIL. The SLAB_KMALLOC-specific
addition of __GFP_NO_OBJ_EXT remains in __alloc_empty_sheaf(), matching
the free_empty_sheaf() assumption.

Since oversized sheaves are now allocated and freed through the empty
sheaf helpers, SHEAF_ALLOC and SHEAF_FREE also account for oversized
sheaves. Update the stat comments accordingly.

Keep the capacity initialization in the oversized prefill path, since
capacity is currently only used for prefilled sheaves

Signed-off-by: Shengming Hu <hu.shengming@zte.com.cn>
---
Changes in v2:
- Rework the change as suggested by Harry.
- Teach __alloc_empty_sheaf() to initialize capacity and pfmemalloc.
- Allocate oversized prefilled sheaves through __alloc_empty_sheaf().
- Free flushed oversized and pfmemalloc sheaves through free_empty_sheaf().
- Link to v1: https://lore.kernel.org/all/20260521195015105Y4zvKHj0TfPZEujixy9Vo@zte.com.cn/

Changes in v3:
- Address Hao's comments:
- Drop the redundant `pfmemalloc` initialization in
  __alloc_empty_sheaf().
- Keep initializing `capacity` and `pfmemalloc` in the normal-sized
  prefill path, since the sheaf may be reused rather than freshly
  allocated.
- Link to v2: https://lore.kernel.org/all/20260522145900248m-nBcy07_SCDk2ATDWfmg@zte.com.cn/

Changes in v4:
- Address comments from Hao and Harry:
- Keep capacity initialization in the oversized prefill path.
- Update SHEAF_ALLOC and SHEAF_FREE comments to mention oversized sheaves.
- Restore the oversized sheaf pfmemalloc comment.
- Link to v3: https://lore.kernel.org/all/20260525211000387LYqTHmxYL900XIB8qwV3h@zte.com.cn/

Changes in v5:
- Address VlastimilBabka's comments:
- Move caller-GFP filtering from __alloc_empty_sheaf() to
  alloc_empty_sheaf():
  - keep the __GFP_NO_OBJ_EXT rejection for regular empty sheaves;
  - keep OBJCGS_CLEAR_MASK clearing out of the raw helper, so oversized
    prefilled sheaves do not lose caller-provided flags such as
    __GFP_NOFAIL.
- Keep the SLAB_KMALLOC-specific __GFP_NO_OBJ_EXT addition in
  __alloc_empty_sheaf(), so allocations still match the
  free_empty_sheaf() and mark_obj_codetag_empty() handling.
- Link to v4: https://lore.kernel.org/all/20260526232429176aHA6oZjTYdhXcZVHc2ZQ-@zte.com.cn/

---
 mm/slub.c | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 04692a6f9128..8663ce4ae1b0 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -362,8 +362,8 @@ enum stat_item {
 	CMPXCHG_DOUBLE_FAIL,	/* Failures of slab freelist update */
 	SHEAF_FLUSH,		/* Objects flushed from a sheaf */
 	SHEAF_REFILL,		/* Objects refilled to a sheaf */
-	SHEAF_ALLOC,		/* Allocation of an empty sheaf */
-	SHEAF_FREE,		/* Freeing of an empty sheaf */
+	SHEAF_ALLOC,		/* Allocation of an empty sheaf including oversized ones */
+	SHEAF_FREE,		/* Freeing of an empty sheaf including oversized ones */
 	BARN_GET,		/* Got full sheaf from barn */
 	BARN_GET_FAIL,		/* Failed to get full sheaf from barn */
 	BARN_PUT,		/* Put full sheaf to barn */
@@ -2762,11 +2762,6 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp,
 	struct slab_sheaf *sheaf;
 	size_t sheaf_size;

-	if (gfp & __GFP_NO_OBJ_EXT)
-		return NULL;
-
-	gfp &= ~OBJCGS_CLEAR_MASK;
-
 	/*
 	 * Prevent recursion to the same cache, or a deep stack of kmallocs of
 	 * varying sizes (sheaf capacity might differ for each kmalloc size
@@ -2791,6 +2786,11 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp,
 static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s,
 						   gfp_t gfp)
 {
+	if (gfp & __GFP_NO_OBJ_EXT)
+		return NULL;
+
+	gfp &= ~OBJCGS_CLEAR_MASK;
+
 	return __alloc_empty_sheaf(s, gfp, s->sheaf_capacity);
 }

@@ -5015,12 +5015,11 @@ kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)

 	if (unlikely(size > s->sheaf_capacity)) {

-		sheaf = kzalloc_flex(*sheaf, objects, size, gfp);
+		sheaf = __alloc_empty_sheaf(s, gfp, size);
 		if (!sheaf)
 			return NULL;

 		stat(s, SHEAF_PREFILL_OVERSIZE);
-		sheaf->cache = s;
 		sheaf->capacity = size;

 		/*
@@ -5029,7 +5028,7 @@ kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
 		 */
 		if (!__kmem_cache_alloc_bulk(s, gfp, size,
 					     &sheaf->objects[0])) {
-			kfree(sheaf);
+			free_empty_sheaf(s, sheaf);
 			return NULL;
 		}

@@ -5097,7 +5096,7 @@ void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
 	if (unlikely((sheaf->capacity != s->sheaf_capacity)
 		     || sheaf->pfmemalloc)) {
 		sheaf_flush_unused(s, sheaf);
-		kfree(sheaf);
+		free_empty_sheaf(s, sheaf);
 		return;
 	}

-- 
2.25.1

Re: [PATCH v5] mm/slub: use empty sheaf helpers for oversized sheaves

[Vlastimil Babka (SUSE)]

On 5/28/26 13:35, hu.shengming@zte.com.cn wrote:
> From: Shengming Hu <hu.shengming@zte.com.cn>
> 
> Oversized prefilled sheaves are allocated separately because their
> capacity can be larger than the cache's regular sheaf capacity. After
> they are flushed, however, they are empty sheaves as well, and should be
> released through the same empty-sheaf helper.
> 
> Allocate oversized prefilled sheaves with __alloc_empty_sheaf() and free
> them with free_empty_sheaf() after a failed prefill or after they are
> returned and flushed. This keeps the oversized and pfmemalloc return paths
> consistent, including the SLAB_KMALLOC-specific __GFP_NO_OBJ_EXT and
> mark_obj_codetag_empty() handling.
> 
> Keep the caller-GFP filtering in alloc_empty_sheaf() instead of
> __alloc_empty_sheaf(). In particular, do not clear OBJCGS_CLEAR_MASK in
> the raw helper, so the oversized prefill path does not unexpectedly drop
> caller-provided flags such as __GFP_NOFAIL. The SLAB_KMALLOC-specific
> addition of __GFP_NO_OBJ_EXT remains in __alloc_empty_sheaf(), matching
> the free_empty_sheaf() assumption.
> 
> Since oversized sheaves are now allocated and freed through the empty
> sheaf helpers, SHEAF_ALLOC and SHEAF_FREE also account for oversized
> sheaves. Update the stat comments accordingly.
> 
> Keep the capacity initialization in the oversized prefill path, since
> capacity is currently only used for prefilled sheaves
> 
> Signed-off-by: Shengming Hu <hu.shengming@zte.com.cn>

Thanks! Applied to slab/for-next

Re: [PATCH v5] mm/slub: use empty sheaf helpers for oversized sheaves

[Harry Yoo]

[-- Attachment #1.1: Type: text/plain, Size: 1617 bytes --]



On 5/28/26 8:35 PM, hu.shengming@zte.com.cn wrote:
> From: Shengming Hu <hu.shengming@zte.com.cn>
> 
> Oversized prefilled sheaves are allocated separately because their
> capacity can be larger than the cache's regular sheaf capacity. After
> they are flushed, however, they are empty sheaves as well, and should be
> released through the same empty-sheaf helper.
> 
> Allocate oversized prefilled sheaves with __alloc_empty_sheaf() and free
> them with free_empty_sheaf() after a failed prefill or after they are
> returned and flushed. This keeps the oversized and pfmemalloc return paths
> consistent, including the SLAB_KMALLOC-specific __GFP_NO_OBJ_EXT and
> mark_obj_codetag_empty() handling.
> 
> Keep the caller-GFP filtering in alloc_empty_sheaf() instead of
> __alloc_empty_sheaf(). In particular, do not clear OBJCGS_CLEAR_MASK in
> the raw helper, so the oversized prefill path does not unexpectedly drop
> caller-provided flags such as __GFP_NOFAIL. The SLAB_KMALLOC-specific
> addition of __GFP_NO_OBJ_EXT remains in __alloc_empty_sheaf(), matching
> the free_empty_sheaf() assumption.
> 
> Since oversized sheaves are now allocated and freed through the empty
> sheaf helpers, SHEAF_ALLOC and SHEAF_FREE also account for oversized
> sheaves. Update the stat comments accordingly.
> 
> Keep the capacity initialization in the oversized prefill path, since
> capacity is currently only used for prefilled sheaves
> 
> Signed-off-by: Shengming Hu <hu.shengming@zte.com.cn>
> ---

Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>

-- 
Cheers,
Harry / Hyeonggon


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v5] mm/slub: use empty sheaf helpers for oversized sheaves

[Hao Li]

On Thu, May 28, 2026 at 07:35:37PM +0800, hu.shengming@zte.com.cn wrote:
> From: Shengming Hu <hu.shengming@zte.com.cn>
> 
> Oversized prefilled sheaves are allocated separately because their
> capacity can be larger than the cache's regular sheaf capacity. After
> they are flushed, however, they are empty sheaves as well, and should be
> released through the same empty-sheaf helper.
> 
> Allocate oversized prefilled sheaves with __alloc_empty_sheaf() and free
> them with free_empty_sheaf() after a failed prefill or after they are
> returned and flushed. This keeps the oversized and pfmemalloc return paths
> consistent, including the SLAB_KMALLOC-specific __GFP_NO_OBJ_EXT and
> mark_obj_codetag_empty() handling.
> 
> Keep the caller-GFP filtering in alloc_empty_sheaf() instead of
> __alloc_empty_sheaf(). In particular, do not clear OBJCGS_CLEAR_MASK in
> the raw helper, so the oversized prefill path does not unexpectedly drop
> caller-provided flags such as __GFP_NOFAIL. The SLAB_KMALLOC-specific
> addition of __GFP_NO_OBJ_EXT remains in __alloc_empty_sheaf(), matching
> the free_empty_sheaf() assumption.
> 
> Since oversized sheaves are now allocated and freed through the empty
> sheaf helpers, SHEAF_ALLOC and SHEAF_FREE also account for oversized
> sheaves. Update the stat comments accordingly.
> 
> Keep the capacity initialization in the oversized prefill path, since
> capacity is currently only used for prefilled sheaves
> 
> Signed-off-by: Shengming Hu <hu.shengming@zte.com.cn>

Reviewed-by: Hao Li <hao.li@linux.dev>

-- 
Thanks,
Hao