Re: [PATCH] mm/slub: refill prefilled sheaves from the barn

[Harry Yoo <harry@kernel.org>]

On Fri, Sep 18, 2026 at 07:41:56PM +0800, Hao Li wrote:
> Currently, when the prefill API refills a non-full sheaf, it takes the
> objects from partial slabs and never from the full sheaves in the barn,
> so once the barn's full list becomes saturated, it stays saturated.
> 
> For objects freed via kfree_rcu(), every RCU sheaf then has to be
> flushed to slabs because the barn's full list has no room.
> 
> To fix this, let the sheaf refill from the barn first, and introduce a
> partial sheaf in the barn, which holds the leftover objects.
> 
> If no partial sheaf exists, take a full sheaf from the barn and return
> it to the caller; the non-full sheaf becomes the partial sheaf, or is
> put on the barn's empty list if it holds no objects.
> 
> If a partial sheaf exists and the non-full sheaf and the partial sheaf
> together reach capacity, the one holding more objects is filled up from
> the other and handed out instead, so no full sheaf needs to be taken
> from the barn.
> 
> If a partial sheaf exists but the two together do not reach capacity,
> take a full sheaf from the barn for the caller; the objects of the
> non-full sheaf are absorbed into the partial sheaf, and the non-full
> sheaf, now empty, is put on the barn's empty list.

Perhaps let's explain why the slab allocator needs this partial sheaf
handling only for prefilled sheaves path?

while reviewing it i was wondering "why is this just not part of
refill_sheaf()"

I assume the reason is: usually we only refill empty sheaves because
if it has at least one object, it can serve the allocation.

> the gain comes from two sides: every full sheaf taken out makes room on
> the barn's full list for a future rcu sheaf, and refilling from the
> barn is cheaper than refilling from partial slabs under list_lock.
> 
> note that putting the non-full sheaf on the full list instead would not
> work: it would occupy room on the full list, so the list would stay
> saturated and rcu_free_sheaf() would still keep flushing.

[..]

> ---
>  mm/slub.c | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 134 insertions(+), 8 deletions(-)
> 
> diff --git a/mm/slub.c b/mm/slub.c
> index 54ec12503357..0b1de6b42b61 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3164,6 +3165,89 @@ static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
>  	return empty;
>  }
>  
> +/*
> + * Exchange @sheaf, which holds fewer objects than requested, for a full one,
> + * keeping the leftover objects in the barn's partial sheaf instead of
> + * flushing them.
> + *
> + * Returns a full sheaf, or NULL if the barn cannot make one.
> + * The returned sheaf might be @sheaf itself or a new one.
> + */
> +static struct slab_sheaf *barn_replace_partial_sheaf(struct kmem_cache *s,
> +						     struct node_barn *barn,
> +						     struct slab_sheaf *sheaf)
> +{
> +	struct slab_sheaf *full = NULL, *partial;
> +	unsigned int to_move;
> +	unsigned long flags;
> +
> +	if (!data_race(barn->nr_full) && !data_race(barn->sheaf_partial))
> +		return NULL;
> +
> +	spin_lock_irqsave(&barn->lock, flags);
> +
> +	partial = barn->sheaf_partial;
> +	if (partial && partial->size + sheaf->size >= s->sheaf_capacity) {
> +		/* Fill the larger one to capacity from the smaller */
> +		if (partial->size > sheaf->size)
> +			swap(partial, sheaf);

Hmm but why switch sheaves when we don't have to?
Sounds like we're losing cache affinity unnecessarily.

I think we should try to refill from barn->sheaf_partial,
or if that's not available, refill from a full sheaf, and then move
the previously-full-sheaf to barn->sheaf_partial or barn->sheaf_empty.

Then we'll never replace the sheaf with a new one.

With that, the control flow could be simplified quite a bit.
Something like this. (Warning: pseudocode, it won't compile)

// refill a sheaf from barn.
// return true when the sheaf becomes full
// return false when the sheaf is not full

bool refill_sheaf_from_barn(s, sheaf) {
	struct node_barn *barn = get_barn(s);
	struct slab_sheaf *partial;
	unsigned int to_move;
	unsigned long flags;

	spin_lock_irqsave(&barn->lock, flags);

	partial = barn->sheaf_partial;
	barn->sheaf_partial = NULL;

	if (!partial && barn->nr_full) {
		// grab one from full list
		partial = [...];
	}

	if (!partial)
		// cannot refill from the barn. the caller will try
		// refilling from n->partial list
		goto done;

	to_move = min(s->sheaf_capacity - sheaf->size, partial->size);
	partial->size -= to_move;
	// copy `to_move` objects from `partial` to `sheaf`
	memcpy(...);
	sheaf->size += to_move;

	if (!partial->size)
		// move it to empty list
	else
		barn->sheaf_partial = partial;

done:
	spin_unlock_irqrestore(&barn->lock, flags);
	return (sheaf->size == s->sheaf_capacity);
}

What do you think?

-- 
Cheers,
Harry / Hyeonggon