From: "Vlastimil Babka (SUSE)" <vbabka@kernel.org>
To: Breno Leitao <leitao@debian.org>, Harry Yoo <harry@kernel.org>,
Hao Li <hao.li@linux.dev>, Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com, Catalin Marinas <catalin.marinas@arm.com>
Subject: Re: [PATCH v2] slab: allow sheaves for SLAB_NOLEAKTRACE caches
Date: Mon, 16 Nov 2026 11:47:09 +0100 [thread overview]
Message-ID: <43dde2e9-24a7-4e09-ba07-d1166963e16b@kernel.org> (raw)
In-Reply-To: <20261113-kmemleak_sheaves-v2-1-8c557c66cec2@debian.org>
On 11/13/26 15:36, Breno Leitao wrote:
> This is a patch that makes kmemleak cheaper to run, by letting
> kmemleak's own metadata cache use a _new_ sheaves type. The flag only
> makes that possible without recursion.
>
> The cost today:
> * every allocation kmemleak tracks needs one kmemleak_object from
> object_cache. An RCU callback frees it later.
>
> Why that's slow:
> * object_cache has no sheaves, so each of those allocations and frees
> goes to the shared slab lists under n->list_lock.
> - On meta's fleet kmemleak kernels, list_lock is the second most
> contended lock, after kmemleak_lock.
> - About 88% of that wait is RCU callbacks freeing kmemleak objects.
>
> Why sheaves aren't allowed:
> * a sheaf is a kmalloc object. Tracking it makes kmemleak allocate
> another kmemleak_object, which needs a sheaf, and so on.
>
> When I reported the recursion in [1], I suggested keeping
> SLAB_NOLEAKTRACE caches away from sheaves, and commit e47c897a2949
> ("slab: add sheaves to most caches") did that. It stopped the
> recursion, but it took the sheaves away from the whole cache, although
> only the kmemleak tracking of the sheaf itself recursed.
>
> Commit 17c38c88294d ("slab: remove cpu (partial) slabs usage from
> allocation paths"), from the same series, then left these caches with
> no per-CPU caching at all.
>
> In hindsight, skipping kmemleak only for the
> sheaf itself, as the new SLAB_ALLOC_NO_LEAKTRACE does, would have been
> the better fix.
>
> The sheaf is allocated with SLAB_ALLOC_NO_RECURSE as well, like kmalloc
> sheaves. Without it, a backing kmalloc cache that needs a new sheaf of
> its own allocates it with SLAB_ALLOC_NO_LEAKTRACE too. kmemleak then
> does not track that sheaf, and can report objects queued in it by
> kfree_rcu() as leaks.
>
> SLAB_ALLOC_NO_RECURSE also skips allocation tagging, so mark the codetag
> of these sheaves empty on free, as is done for kmalloc sheaves.
>
> Link: https://lore.kernel.org/all/aW_NK2NXVgtuzCVH@gmail.com/ [1]
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
> Signed-off-by: Breno Leitao <leitao@debian.org>
Applied to slab/for-next, thanks!
> ---
> Changes in v2:
> - Fix sashiko findings:
> - Set SLAB_ALLOC_NO_RECURSE along with SLAB_ALLOC_NO_LEAKTRACE, so the
> flag stays off the kmalloc cache's own sheaf allocation, per Catalin
> Marinas.
> - Mark the codetag of SLAB_NOLEAKTRACE sheaves empty on free, per
> Catalin Marinas.
> - Link to v1: https://patch.msgid.link/20261112-kmemleak_sheaves-v1-1-e40e3d73b253@debian.org
> ---
> mm/slab.h | 1 +
> mm/slub.c | 22 +++++++++++++---------
> 2 files changed, 14 insertions(+), 9 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index e94ac63d10dc03..7b20f10aa46a2e 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -23,6 +23,7 @@
> #define SLAB_ALLOC_NEW_SLAB 0x02 /* a flag for alloc_slab_obj_exts() */
> #define SLAB_ALLOC_NO_RECURSE 0x04 /* prevent kmalloc() recursion */
> #define SLAB_ALLOC_NO_OBJ_EXT 0x08 /* prevent obj_exts array allocation */
> +#define SLAB_ALLOC_NO_LEAKTRACE 0x10 /* skip kmemleak tracking */
>
> #define SLAB_FREE_DEFAULT 0x00 /* no flags */
> #define SLAB_FREE_NOLOCK 0x01 /* spinning not allowed */
> diff --git a/mm/slub.c b/mm/slub.c
> index 8369853ddca721..b927cd684ec656 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2850,6 +2850,14 @@ static struct slab_sheaf *__alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp,
> if (s->flags & SLAB_KMALLOC)
> alloc_flags |= SLAB_ALLOC_NO_RECURSE;
>
> + /*
> + * kmemleak allocates its metadata from SLAB_NOLEAKTRACE caches, so
> + * tracking their sheaves would recurse into kmemleak. NO_RECURSE stops
> + * NO_LEAKTRACE from reaching the kmalloc cache's own sheaf allocation.
> + */
> + if (s->flags & SLAB_NOLEAKTRACE)
> + alloc_flags |= SLAB_ALLOC_NO_LEAKTRACE | SLAB_ALLOC_NO_RECURSE;
> +
> sheaf_size = struct_size(sheaf, objects, capacity);
> sheaf = kmalloc_flags(sheaf_size, gfp | __GFP_ZERO, alloc_flags, NUMA_NO_NODE);
>
> @@ -2883,7 +2891,7 @@ static void __free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
> * warning, therefore replace NULL with CODETAG_EMPTY to indicate
> * that the extension for this sheaf is expected to be NULL.
> */
> - if (s->flags & SLAB_KMALLOC)
> + if (s->flags & (SLAB_KMALLOC | SLAB_NOLEAKTRACE))
> mark_obj_codetag_empty(sheaf);
>
> VM_WARN_ON_ONCE(sheaf->size > 0);
> @@ -4727,7 +4735,8 @@ bool slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags, size_t size,
> if (init && p[i] && !is_kfence_address(p[i]))
> memset(p[i], 0, zero_size);
>
> - if (alloc_flags_allow_spinning(ac->alloc_flags))
> + if (alloc_flags_allow_spinning(ac->alloc_flags) &&
> + !(ac->alloc_flags & SLAB_ALLOC_NO_LEAKTRACE))
> kmemleak_alloc_recursive(p[i], s->object_size, 1,
> s->flags, init_flags);
> kmsan_slab_alloc(s, p[i], init_flags);
> @@ -7987,13 +7996,8 @@ static unsigned int calculate_sheaf_capacity(struct kmem_cache *s,
> if (IS_ENABLED(CONFIG_SLUB_TINY) || s->flags & SLAB_DEBUG_FLAGS)
> return 0;
>
> - /*
> - * Bootstrap caches can't have sheaves for now (SLAB_NO_SHEAVES).
> - * SLAB_NOLEAKTRACE caches (e.g., kmemleak's object_cache) must not
> - * have sheaves to avoid recursion when sheaf allocation triggers
> - * kmemleak tracking.
> - */
> - if (s->flags & (SLAB_NO_SHEAVES | SLAB_NOLEAKTRACE))
> + /* Bootstrap caches can't have sheaves for now (SLAB_NO_SHEAVES). */
> + if (s->flags & SLAB_NO_SHEAVES)
> return 0;
>
> /*
>
> ---
> base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> change-id: 20261110-kmemleak_sheaves-b3531591ad21
>
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
>
prev parent reply other threads:[~2026-11-16 10:47 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 14:36 [PATCH v2] slab: allow sheaves for SLAB_NOLEAKTRACE caches Breno Leitao
2026-11-13 18:47 ` Catalin Marinas
2026-11-16 10:47 ` Vlastimil Babka (SUSE) [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=43dde2e9-24a7-4e09-ba07-d1166963e16b@kernel.org \
--to=vbabka@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=catalin.marinas@arm.com \
--cc=cl@gentwo.org \
--cc=hao.li@linux.dev \
--cc=harry@kernel.org \
--cc=kernel-team@meta.com \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.