From: "Harry Yoo (Meta)" <harry@kernel.org>
To: Johannes Weiner <hannes@cmpxchg.org>,
Michal Hocko <mhocko@kernel.org>,
Roman Gushchin <roman.gushchin@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
Muchun Song <muchun.song@linux.dev>,
Andrew Morton <akpm@linux-foundation.org>,
Harry Yoo <harry@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Vlastimil Babka <vbabka@kernel.org>, Hao Li <hao.li@linux.dev>,
Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Clark Williams <clrkwllms@kernel.org>,
Steven Rostedt <rostedt@goodmis.org>
Cc: kernel-team@meta.com, cgroups@vger.kernel.org,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
linux-rt-devel@lists.linux.dev
Subject: [PATCH for-next v3 3/7] mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding
Date: Sun, 15 Nov 2026 15:09:37 +0000 [thread overview]
Message-ID: <20261115-kmalloc-nolock-fixes-v3-3-9034dd4ecaab@kernel.org> (raw)
In-Reply-To: <20261115-kmalloc-nolock-fixes-v3-0-9034dd4ecaab@kernel.org>
kfree_nolock() deliberately skips slab free hooks that should not be
called in unknown context. However, there is one more place that
needs the same thing: memcg_alloc_abort_single().
Instead of open coding slab_free_hook(), let it handle
SLAB_FREE_NOLOCK. This is required to correctly handle allocation
failures due to memcg charging.
kfree_nolock() now respects init_on_free which has been ignored.
Besides that, no functional change intended.
Fixes: af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().")
Signed-off-by: Harry Yoo (Meta) <harry@kernel.org>
---
mm/slub.c | 76 +++++++++++++++++++++++++++++++--------------------------------
1 file changed, 37 insertions(+), 39 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 761dc2affea2..7c0adc104988 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2684,11 +2684,14 @@ struct rcu_delayed_free {
* - memcg_slab_post_alloc_hook()
* - alloc_tagging_slab_alloc_hook()
*
+ * slab_free_hook() assumes that memcg_slab_free_hook() and
+ * alloc_tagging_slab_free_hook() are already invoked.
+ *
* Free hooks that must handle missing corresponding alloc hooks:
* - kmemleak_free_recursive()
* - kfence_free()
*
- * Free hooks that have no alloc hook counterpart, and thus safe to call:
+ * Free hooks that have no alloc hook counterpart:
* - debug_check_no_locks_freed()
* - debug_check_no_obj_freed()
* - __kcsan_check_access()
@@ -2698,28 +2701,39 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
{
bool init = free_flags & SLAB_FREE_INIT;
bool after_rcu_delay = free_flags & SLAB_FREE_AFTER_RCU_DELAY;
+ bool allow_spin = free_flags_allow_spinning(free_flags);
/* Are the object contents still accessible? */
bool still_accessible = (s->flags & SLAB_TYPESAFE_BY_RCU) && !after_rcu_delay;
- kmemleak_free_recursive(x, s->flags);
kmsan_slab_free(s, x);
- debug_check_no_locks_freed(x, s->object_size);
+ /* Hooks that acquire locks or kmalloc_nolock() skips alloc hook counterpart */
+ if (likely(allow_spin)) {
+ kmemleak_free_recursive(x, s->flags);
+ debug_check_no_locks_freed(x, s->object_size);
- if (!(s->flags & SLAB_DEBUG_OBJECTS))
- debug_check_no_obj_freed(x, s->object_size);
+ if (!(s->flags & SLAB_DEBUG_OBJECTS))
+ debug_check_no_obj_freed(x, s->object_size);
- /* Use KCSAN to help debug racy use-after-free. */
- if (!still_accessible)
- __kcsan_check_access(x, s->object_size,
- KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT);
+ /* Use KCSAN to help debug racy use-after-free. */
+ if (!still_accessible)
+ __kcsan_check_access(x, s->object_size,
+ KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT);
- if (kfence_free(x))
- return false;
+ if (kfence_free(x))
+ return false;
+ }
/*
* Give KASAN a chance to notice an invalid free operation before we
* modify the object.
+ *
+ * If KASAN finds a bug it will do kasan_report_invalid_free() which
+ * will call raw_spin_lock_irqsave() which is technically unsafe from
+ * NMI, but take chance and report kernel bug. The sequence of
+ * kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI
+ * -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU
+ * is double buggy and deserves to deadlock.
*/
if (kasan_slab_pre_free(s, x))
return false;
@@ -2728,6 +2742,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
if (still_accessible) {
struct rcu_delayed_free *delayed_free;
+ VM_WARN_ON_ONCE(!allow_spin);
delayed_free = kmalloc_obj(*delayed_free, GFP_NOWAIT);
if (delayed_free) {
/*
@@ -2775,8 +2790,11 @@ bool slab_free_hook(struct kmem_cache *s, void *x, unsigned int free_flags)
set_orig_size(s, x, orig_size);
}
- /* KASAN might put x into memory quarantine, delaying its reuse. */
- return !kasan_slab_free(s, x, init, still_accessible, false);
+ /*
+ * KASAN might put x into memory quarantine, delaying its reuse.
+ * Skip quarantine when spinning is not allowed as it uses a spinlock.
+ */
+ return !kasan_slab_free(s, x, init, still_accessible, !allow_spin);
}
static __fastpath_inline
@@ -6935,6 +6953,7 @@ void kfree_nolock(const void *object)
struct slab *slab;
struct kmem_cache *s;
void *x = (void *)object;
+ unsigned int free_flags = SLAB_FREE_NOLOCK;
if (unlikely(ZERO_OR_NULL_PTR(object)))
return;
@@ -6949,33 +6968,12 @@ void kfree_nolock(const void *object)
memcg_slab_free_hook(s, slab, &x, 1);
alloc_tagging_slab_free_hook(s, slab, &x, 1);
- /*
- * Unlike slab_free() do NOT call the following:
- * kmemleak_free_recursive(x, s->flags);
- * debug_check_no_locks_freed(x, s->object_size);
- * debug_check_no_obj_freed(x, s->object_size);
- * __kcsan_check_access(x, s->object_size, ..);
- * kfence_free(x);
- * since they take spinlocks or not safe from any context.
- */
- kmsan_slab_free(s, x);
- /*
- * If KASAN finds a kernel bug it will do kasan_report_invalid_free()
- * which will call raw_spin_lock_irqsave() which is technically
- * unsafe from NMI, but take chance and report kernel bug.
- * The sequence of
- * kasan_report_invalid_free() -> raw_spin_lock_irqsave() -> NMI
- * -> kfree_nolock() -> kasan_report_invalid_free() on the same CPU
- * is double buggy and deserves to deadlock.
- */
- if (kasan_slab_pre_free(s, x))
+
+ if (slab_want_init_on_free(s))
+ free_flags |= SLAB_FREE_INIT;
+
+ if (unlikely(!slab_free_hook(s, x, free_flags)))
return;
- /*
- * memcg, kasan_slab_pre_free are done for 'x'.
- * The only thing left is kasan_poison without quarantine,
- * since kasan quarantine takes locks and not supported from NMI.
- */
- kasan_slab_free(s, x, false, false, /* skip quarantine */true);
if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, x, false)))
return;
--
2.54.0
next prev parent reply other threads:[~2026-11-15 15:10 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-15 15:09 [PATCH for-next v3 0/7] memcg,slab: kmalloc_nolock() fixes Harry Yoo (Meta)
2026-11-15 15:09 ` [PATCH for-next v3 1/7] mm/memcontrol: defer obj_cgroup_release() to irq_work Harry Yoo (Meta)
2026-11-15 15:09 ` [PATCH for-next v3 2/7] mm/slab: pass free_flags to slab_free_hook() Harry Yoo (Meta)
2026-11-15 15:09 ` Harry Yoo (Meta) [this message]
2026-11-16 8:26 ` [PATCH for-next v3 3/7] mm/slab: handle SLAB_FREE_NOLOCK in slab_free_hook() instead of open coding sashiko-bot
2026-11-15 15:09 ` [PATCH for-next v3 4/7] mm/slab: fix a deadlock in memcg_alloc_abort_single() Harry Yoo (Meta)
2026-11-15 15:09 ` [PATCH for-next v3 5/7] mm/slab: serialize deferred_work_barrier() Harry Yoo (Meta)
2026-11-15 15:09 ` [PATCH for-next v3 6/7] mm/slab: refactor free_to_pcs() Harry Yoo (Meta)
2026-11-15 15:09 ` [PATCH for-next v3 7/7] mm/slab: call debug_locks and KCSAN checks in kfree_nolock() Harry Yoo (Meta)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261115-kmalloc-nolock-fixes-v3-3-9034dd4ecaab@kernel.org \
--to=harry@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=ast@kernel.org \
--cc=bigeasy@linutronix.de \
--cc=cgroups@vger.kernel.org \
--cc=cl@gentwo.org \
--cc=clrkwllms@kernel.org \
--cc=hannes@cmpxchg.org \
--cc=hao.li@linux.dev \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=mhocko@kernel.org \
--cc=muchun.song@linux.dev \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=rostedt@goodmis.org \
--cc=shakeel.butt@linux.dev \
--cc=vbabka@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.