From: "Vlastimil Babka (SUSE)" <vbabka@kernel.org>
To: Suren Baghdasaryan <surenb@google.com>,
stable@vger.kernel.org, david@kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Sasha Levin <sashal@kernel.org>, Hao Li <hao.li@linux.dev>,
Harry Yoo <harry@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>,
Roman Gushchin <roman.gushchin@linux.dev>
Subject: Re: [PATCH 7.3.y] mm/slub: refill prefilled sheaves from the barn
Date: Fri, 20 Nov 2026 10:43:35 +0100 [thread overview]
Message-ID: <15168363-7171-4b3d-9b6c-a621b7700aee@kernel.org> (raw)
In-Reply-To: <20261120030833.3188636-1-surenb@google.com>
On 11/20/26 04:08, Suren Baghdasaryan wrote:
> From: Hao Li <hao.li@linux.dev>
>
> commit 1b87ea06a34b213e45e3e0c4effd5043f15046ed upstream.
>
> Currently, when the prefill API refills a non-full sheaf, it takes the
> objects from partial slabs and never from the full sheaves in the barn,
> so once the barn's full list becomes saturated, it stays saturated.
>
> For objects freed via kfree_rcu(), every RCU sheaf then has to be
> flushed to slabs because the barn's full list has no room.
>
> To fix this, let the sheaf refill from the barn first, and introduce a
> partial sheaf in the barn, which holds the leftover objects [1].
>
> Only the prefill path needs the partial sheaf. The generic allocation
> path (__pcs_replace_empty_main()) exchanges an empty sheaf for a full
> one from the barn, so nothing is left over. The prefill path refills a
> sheaf that is not necessarily empty, so taking objects from the barn
> usually leaves leftovers, and the partial sheaf is where they are
> kept. refill_sheaf() only takes objects from partial slabs and never
> involves the partial sheaf.
>
> The sheaf is refilled by copying objects from the partial sheaf, and
> then from a full sheaf taken from the barn if it is still not full. A
> sheaf that objects were copied from stays in the barn as the partial
> sheaf if it still holds objects, or goes on the empty list if it is
> empty. The sheaf being refilled is never replaced.
>
> The gain comes from two sides: every full sheaf taken out makes room on
> the barn's full list for a future RCU sheaf, and refilling from the
> barn is cheaper than refilling from partial slabs under list_lock.
>
> Note that putting the sheaf with the leftover objects on the full list
> instead would not work: it would occupy room on the full list, so the
> list would stay saturated and rcu_free_sheaf() would still keep
> flushing.
>
> An earlier version of this patch swapped sheaves instead, to minimize
> the memcpy overhead. As Harry Yoo pointed out [2], replacing the
> caller's sheaf loses cache affinity, filling it directly is more
> straightforward, and testing showed no measurable difference between
> the two approaches, so the approach he suggested is used.
>
> Tested with will-it-scale mmap1 (192 processes, one-minute runs) on the
> maple_node cache.
>
> throughput: 27778727 -> 34500556 (+24.2%)
>
> metric baseline patched change
> =====================================================================
> alloc_fastpath 54,124 56,696 +4.75%
> alloc_slab 7,100,651 159,739 -97.75%
> barn_get 849 194,460,392 +22904539.81%
> barn_get_fail 2 217 +10750.00%
> barn_put 851 182,306,701 +21422544.07%
> barn_put_fail 260,400,484 145,522,675 -44.12%
> cmpxchg_double_fail 1,029,204 334,228 -67.53%
> free_add_partial 326,694,687 181,935,790 -44.31%
> free_fastpath 13,297 15,611 +17.40%
> free_rcu_sheaf 8,332,839,104 10,490,535,227 +25.89%
> free_remove_partial 7,099,788 158,297 -97.77%
> free_slab 7,099,788 158,297 -97.77%
> free_slowpath 10,969,156 782,851 -92.86%
> objects 15,295 14,849 -2.92%
> objects_partial 15,295 14,849 -2.92%
> partial 1,604 2,336 +45.64%
> sheaf_alloc 137,757,895 141,208,076 +2.50%
> sheaf_flush 8,332,823,922 4,656,725,764 -44.12%
> sheaf_free 137,757,883 141,208,072 +2.50%
> sheaf_prefill_fast 3,337,502,163 4,196,506,116 +25.74%
> sheaf_prefill_slow 685 400 -41.61%
> sheaf_refill 8,343,794,364 4,657,509,724 -44.18%
> sheaf_return_fast 3,337,502,411 4,196,506,382 +25.74%
> sheaf_return_slow 437 134 -69.34%
> slabs 1,604 2,336 +45.64%
> total_objects 102,656 149,504 +45.64%
>
> Here is what the important metric changes mean.
>
> barn_get and barn_put: refills now take full sheaves out of the barn,
> and RCU sheaves are put into the barn again. Before, the barn's full
> list was saturated once and hardly ever consumed.
>
> barn_put_fail: more than half of the RCU sheaves are now put into the
> barn instead of being flushed. The rest are still flushed because they
> arrive while the barn's full list is at its limit.
>
> sheaf_refill and free_add_partial: fewer objects are taken from partial
> slabs to refill sheaves, and fewer slabs are added to the partial list,
> by the same percentage.
>
> alloc_slab and free_slab: the partial list is almost never empty when a
> refill looks at it, so slabs are almost never allocated and freed again
> only to serve refills.
>
> After the test, the maple_node cache holds 1604 slabs without the patch
> and 2336 with it. After a manual shrink
> (echo 1 > /sys/kernel/slab/maple_node/shrink), it holds 485 without
> the patch and 563 with it. So most of the extra slabs are held only by
> objects sitting in sheaves and are returned by a shrink, and what
> remains is a small difference, because objects handed out from the barn
> come from many more slabs than a refill from partial slabs would use.
>
> Link: https://lore.kernel.org/linux-mm/aqPlMzUIw-4g2iOX@fedora/ [1]
> Link: https://lore.kernel.org/linux-mm/aq0ylDidEHa2kg4X@thinkstation/ [2]
> Signed-off-by: Hao Li <hao.li@linux.dev>
> Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
> Link: https://patch.msgid.link/20260921094521.141665-1-hao.li@linux.dev
> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
> Cc: stable@vger.kernel.org
> Signed-off-by: Suren Baghdasaryan <surenb@google.com>
> ---
> Original patch from v7.4-rc1 applies cleanly over stable 7.3.y.
>
> In v7.0-rc1, per-CPU slabs and per-CPU partial lists were removed (see:
> 17c38c88294d, e323b52cf00f, 32c894c7274b). Therefore once barn->nr_full
> hits MAX_FULL_SHEAVES, every refill_sheaf() and sheaf_flush_unused()
> will compete for the node-wide n->list_lock. This patch fixes this.
>
> In pre-7.0 kernels per-CPU slabs and per-CPU partial lists are still
> there and refills use per-CPU c->freelist and per-CPU partial slabs
> without taking n->list_lock. Similarly, do_slab_free() recycles freed
> slabs through per-CPU c->slab and c->partial without taking n->list_lock.
> Therefore this patch has no performance benefit on those older branches.
IOW we want to avoid a regression from 6.18 LTS to 7.3 LTS.
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>
> mm/slub.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 81 insertions(+)
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 544cff39762c..a0f0bdc1e30a 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -422,6 +422,7 @@ struct node_barn {
> spinlock_t lock;
> struct list_head sheaves_full;
> struct list_head sheaves_empty;
> + struct slab_sheaf *sheaf_partial;
> unsigned int nr_full;
> unsigned int nr_empty;
> };
> @@ -3298,6 +3299,7 @@ static void barn_init(struct node_barn *barn)
> spin_lock_init(&barn->lock);
> INIT_LIST_HEAD(&barn->sheaves_full);
> INIT_LIST_HEAD(&barn->sheaves_empty);
> + barn->sheaf_partial = NULL;
> barn->nr_full = 0;
> barn->nr_empty = 0;
> }
> @@ -3315,6 +3317,10 @@ static void barn_shrink(struct kmem_cache *s, struct node_barn *barn)
> barn->nr_full = 0;
> list_splice_init(&barn->sheaves_empty, &empty_list);
> barn->nr_empty = 0;
> + if (barn->sheaf_partial) {
> + list_add(&barn->sheaf_partial->barn_list, &full_list);
> + barn->sheaf_partial = NULL;
> + }
>
> spin_unlock_irqrestore(&barn->lock, flags);
>
> @@ -5073,12 +5079,87 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int nod
> }
> EXPORT_SYMBOL(kmem_cache_alloc_node_noprof);
>
> +/*
> + * Refill @sheaf from the barn: from its partial sheaf first, then from its full
> + * sheaves. A sheaf that objects were copied from stays in the barn as the
> + * partial sheaf if it still holds objects, or goes on the empty list if it is
> + * empty.
> + *
> + * Returns true if the sheaf is now full, at s->sheaf_capacity.
> + * Returns false if the sheaf is still not full.
> + */
> +static bool refill_sheaf_from_barn(struct kmem_cache *s,
> + struct slab_sheaf *sheaf)
> +{
> + struct node_barn *barn = get_barn(s);
> + struct slab_sheaf *src;
> + unsigned int to_move;
> + unsigned long flags;
> +
> + if (!barn)
> + return false;
> +
> + if (!data_race(barn->nr_full) && !data_race(barn->sheaf_partial))
> + return false;
> +
> + spin_lock_irqsave(&barn->lock, flags);
> +
> + /*
> + * The partial sheaf can hold fewer objects than @sheaf needs to
> + * reach capacity, and a sheaf on the full list is not necessarily
> + * full (see the comment in rcu_free_sheaf()), so keep taking from
> + * the barn until @sheaf is full or nothing is left.
> + */
> + while (sheaf->size < s->sheaf_capacity) {
> + src = barn->sheaf_partial;
> + barn->sheaf_partial = NULL;
> + if (!src) {
> + if (!barn->nr_full)
> + break;
> + src = list_first_entry(&barn->sheaves_full,
> + struct slab_sheaf, barn_list);
> + list_del(&src->barn_list);
> + barn->nr_full--;
> + }
> +
> + to_move = min(s->sheaf_capacity - sheaf->size, src->size);
> + src->size -= to_move;
> + memcpy(&sheaf->objects[sheaf->size], &src->objects[src->size],
> + to_move * sizeof(void *));
> + sheaf->size += to_move;
> +
> + if (src->size) {
> + barn->sheaf_partial = src;
> + } else {
> + /*
> + * No empty-limit check: the sheaf put on the empty list
> + * was already in the barn, so the barn holds no more
> + * sheaves than before. barn_replace_empty_sheaf() skips
> + * the check for the same reason.
> + */
> + list_add(&src->barn_list, &barn->sheaves_empty);
> + barn->nr_empty++;
> + }
> + }
> +
> + spin_unlock_irqrestore(&barn->lock, flags);
> +
> + if (sheaf->size < s->sheaf_capacity)
> + return false;
> +
> + stat(s, BARN_GET);
> + return true;
> +}
> +
> static int __prefill_sheaf_pfmemalloc(struct kmem_cache *s,
> struct slab_sheaf *sheaf, gfp_t gfp)
> {
> gfp_t gfp_nomemalloc;
> int ret;
>
> + if (refill_sheaf_from_barn(s, sheaf))
> + return 0;
> +
> gfp_nomemalloc = gfp | __GFP_NOMEMALLOC;
> if (gfp_pfmemalloc_allowed(gfp))
> gfp_nomemalloc |= __GFP_NOWARN;
prev parent reply other threads:[~2026-11-20 9:43 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 3:08 [PATCH 7.3.y] mm/slub: refill prefilled sheaves from the barn Suren Baghdasaryan
2026-11-20 9:43 ` Vlastimil Babka (SUSE) [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=15168363-7171-4b3d-9b6c-a621b7700aee@kernel.org \
--to=vbabka@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=cl@gentwo.org \
--cc=david@kernel.org \
--cc=gregkh@linuxfoundation.org \
--cc=hao.li@linux.dev \
--cc=harry@kernel.org \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=sashal@kernel.org \
--cc=stable@vger.kernel.org \
--cc=surenb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox