* [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage
@ 2026-07-20 12:44 Harry Yoo (Oracle)
2026-07-20 12:44 ` [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs Harry Yoo (Oracle)
` (7 more replies)
0 siblings, 8 replies; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
This series is based on slab/for-next (56aa9b819edc), and is also
available on git.kernel.org [4].
Changes since v3
=================
kfree_rcu_nolock() now falls back to kvfree_call_rcu() via irq work
-------------------------------------------------------------------
While revisiting making the fallback path use percpu lockless list [6]
(as the fallback path can be triggered more easily compared to RFC v2),
realized that we don't really have to reinvent yet another mechanism :)
kvfree_rcu() batching now uses kvfree_rcu_head internally, while
the API remains unchanged for now (to avoid making the series larger).
This allows kfree_rcu_nolock() to fall back to kvfree_rcu batching via
irq work and avoids additional complexity.
Minor changes
-------------
- Added Acked-by, Reviewed-by from Vlastimil and Pedro, thanks!
Oh, but I dropped add Vlastimil's Acked-by: on patch 7 as the fallback
path was changed a bit. See above!
- Was Patch 1: Dropped SW perf counter fallback in slub_kunit test (Alexei)
- Patch 1: Explain why we don't test _nolock APIs with kprobe on !SMP (Pedro)
- Patch 1: Make kprobe and perf tests independent test cases
- Patch 2: Introduce SLAB_FREE_* flags instead of passing allow_spin
(inspired by Pedro). alloc_empty_sheaf() drops __GFP_RECLAIM w/
SLAB_ALLOC_NOLOCK
- Patch 3: Elaborate the comment on !irqs_disabled(() check (Vlastimil)
- Patch 4: Explain why it is tricky to implement kvfree_rcu_barrier() that
works on a specific kmem_cache (Pedro)
- Patch 4: Use llist_for_each_entry_safe() instead of llist_for_each_safe()
(Pedro)
- Patch 7: Fall back when NUMA node doesn't match (Shengming)
- Patch 7: Document that kfree_rcu_nolock() supports only objects allocated
via kmalloc_nolock() (Xiao)
- In this version I didn't adjust Alexei's comment on bypassing perf
event throttling because my tests complained that it's firing too
fast and sometimes it exceeded test timeout because of this. Need more
thoughts on this.
Git range-diff is copied-and-pasted at the end of the email.
Actual Cover Letter
===================
This series improves kmalloc_nolock() and kfree_nolock() coverage
in slub_kunit (patch 1) and introduces kfree_rcu_nolock() for
unknown context as suggested by Alexei Starovoitov.
Unknown context means the caller does not know whether spinning on a lock
is safe (e.g., a BPF program attached to an arbitrary kernel function or
in NMI context).
The slab allocator already supports unknown context via kmalloc_nolock()
and kfree_nolock(), but te slab allocator does not support freeing
objects by RCU in unknown context.
It is not ideal to have completely separate batching for unknown context
because the worst scenario where spinning on a lock would lead to
deadlock is very rare, and in most cases, it is safe to use the
existing mechanism (kfree_rcu_sheaf()).
Since most part of the slab allocator already supports unknown context
and sheaves support batching kvfree_rcu() calls for slab objects,
implement kfree_rcu_nolock() with minimal changes by teaching
kfree_rcu_sheaf() how to support unknown context and making
it a little bit harder to allocate an empty sheaf, instead of making
intrusive changes to the existing kvfree_rcu batching logic.
kfree_rcu_nolock() tries to free the object to the rcu sheaf if
trylock succeeds. Once the rcu sheaf becomes full, it is submitted to
RCU via call_rcu() if spinning is allowed or IRQs are enabled (to avoid
calling call_rcu() in the middle of call_rcu()). Otherwise, call_rcu()
is deferred via irq work.
When there is no sheaf available, kfree_rcu_sheaf() falls back to
defer_kfree_rcu(). It submits the object to kvfree_rcu batching
via irq work. To do this, patch 6 converts kvfree_rcu to use
kvfree_rcu_head without visible changes to the API for now.
Unlike kfree_rcu(), only the 2-argument variant is supported.
This is because the last resort of the 1-arg variant is
synchronize_rcu(), which cannot be used in an unknown context.
As suggested by Alexei Starovoitov, kfree_rcu_nolock() can be used with
struct kvfree_rcu_head (8 bytes), which is smaller than struct rcu_head
(16 bytes).
For more background and future plans, please see [5].
[1] RFC v1: https://lore.kernel.org/linux-mm/20260206093410.160622-1-harry.yoo@oracle.com
[2] RFC v2: https://lore.kernel.org/linux-mm/20260416091022.36823-1-harry@kernel.org
[3] v3: https://lore.kernel.org/linux-mm/20260615-kfree_rcu_nolock-v3-0-70a54f3775bb@kernel.org
[4] https://git.kernel.org/pub/scm/linux/kernel/git/harry/linux.git/log/?h=kfree_rcu_nolock-v4r1
[5] kmalloc_nolock() follow-ups, including kfree_rcu_nolock(),
https://lore.kernel.org/linux-mm/esepccfhqg7m6jo76ns2znj2cnuaepx2xvw5zaygtwohq4psma@563ypprp6rr3
To: Vlastimil Babka <vbabka@kernel.org>
To: Andrew Morton <akpm@linux-foundation.org>
To: Hao Li <hao.li@linux.dev>
To: Christoph Lameter <cl@gentwo.org>
To: David Rientjes <rientjes@google.com>
To: Roman Gushchin <roman.gushchin@linux.dev>
To: Alexei Starovoitov <ast@kernel.org>
To: Andrii Nakryiko <andrii@kernel.org>
To: Puranjay Mohan <puranjay@kernel.org>
To: Amery Hung <ameryhung@gmail.com>
To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
To: Clark Williams <clrkwllms@kernel.org>
To: Steven Rostedt <rostedt@goodmis.org>
To: Paul E. McKenney <paulmck@kernel.org>
To: Frederic Weisbecker <frederic@kernel.org>
To: Neeraj Upadhyay <neeraj.upadhyay@kernel.org>
To: Joel Fernandes <joelagnelf@nvidia.com>
To: Josh Triplett <josh@joshtriplett.org>
To: Boqun Feng <boqun@kernel.org>
To: Uladzislau Rezki <urezki@gmail.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: Lai Jiangshan <jiangshanlai@gmail.com>
To: Zqiang <qiang.zhang@linux.dev>
To: Pedro Falcato <pfalcato@suse.de>
To: Suren Baghdasaryan <surenb@google.com>
To: Shengming Hu <hu.shengming@zte.com.cn>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-rt-devel@lists.linux.dev
Cc: rcu@vger.kernel.org
Cc: bpf@vger.kernel.org
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
1: d9a08a6baf2f < -: ------------ slub_kunit: fall back to SW perf events when HW PMU is not available
2: eba4a1d7445d ! 1: a16d94ea7d79 mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
@@ Commit message
Register a kprobe pre-handler to invoke kmalloc_nolock() and
kfree_nolock() in the middle of the slab allocator. However, do not
- register the handler on UP kernels [1].
+ register the handler on UP kernels because that use case is not
+ well supported [1] in the kernel.
To attach the pre-handler while s->cpu_sheaves->lock or n->list_lock
is held, add a wrapper function for lockdep_assert_held() that calls
@@ Commit message
away its callsites. Otherwise, the compiler may consider the function
does not have any side effect and remove callsites.
+ Compared to using plain kprobe, this has two advantages: 1) it avoids
+ hardcoding function names in the test, and 2) it can trigger those APIs
+ in the middle of a function, where the lock is expected to be held as
+ annotated with lockdep.
+
+ While it was proposed [2] to use kunit function redirection to test
+ this, it is currently infeasible as some lock helpers don't have
+ symbols.
+
+ Factor out the nested loop that calls kmalloc and friends to
+ test_kmalloc_kfree(), and call them in
+ test_kmalloc_kfree_nolock_{perf,kprobe}(), each being an independent
+ test case. During the refactoring, drop alloc_fail handling as it
+ doesn't provide much benefits.
+
+ Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b@kernel.org [2]
+ Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
## lib/tests/slub_kunit.c ##
@@ lib/tests/slub_kunit.c: static void test_krealloc_redzone_zeroing(struct kunit *
-#ifdef CONFIG_PERF_EVENTS
+#if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
-+#define SLUB_KUNIT_TEST_KMALLOC_KFREE_NOLOCK
#define NR_ITERATIONS 1000
#define NR_OBJECTS 1000
static void *objects[NR_OBJECTS];
@@ lib/tests/slub_kunit.c: struct test_nolock_context {
int alloc_fail;
+#ifdef CONFIG_PERF_EVENTS
struct perf_event *event;
- bool is_perf_type_hw;
+#endif
-+#ifdef CONFIG_KPROBES
++#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
+ struct kprobe kprobe;
+#endif
};
-+#ifdef CONFIG_PERF_EVENTS
- static struct perf_event_attr hw_attr = {
- .type = PERF_TYPE_HARDWARE,
- .config = PERF_COUNT_HW_CPU_CYCLES,
-@@ lib/tests/slub_kunit.c: static struct perf_event_attr sw_attr = {
- .sample_freq = 100000,
- };
+-static struct perf_event_attr hw_attr = {
+- .type = PERF_TYPE_HARDWARE,
+- .config = PERF_COUNT_HW_CPU_CYCLES,
+- .size = sizeof(struct perf_event_attr),
+- .pinned = 1,
+- .disabled = 1,
+- .freq = 1,
+- .sample_freq = 100000,
+-};
++static void test_kmalloc_kfree(void)
++{
++ int i, j;
++
++ for (i = 0; i < NR_ITERATIONS; i++) {
++ for (j = 0; j < NR_OBJECTS; j++) {
++ gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
--static void overflow_handler_test_nolock(struct perf_event *event,
-- struct perf_sample_data *data,
-- struct pt_regs *regs)
+-static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
+- struct perf_sample_data *data,
+- struct pt_regs *regs)
++ objects[j] = kmalloc_obj(*objects[j], gfp);
++ if (!objects[j]) {
++ j--;
++ while (j >= 0)
++ kfree(objects[j--]);
++ return;
++ }
++ }
++
++ for (j = 0; j < NR_OBJECTS; j++)
++ kfree(objects[j]);
++ }
++}
++
+static void test_nolock(struct test_nolock_context *ctx)
{
void *objp;
@@ lib/tests/slub_kunit.c: static struct perf_event_attr sw_attr = {
/* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */
gfp = (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT;
-@@ lib/tests/slub_kunit.c: static void overflow_handler_test_nolock(struct perf_event *event,
+@@ lib/tests/slub_kunit.c: static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
+ kfree_nolock(objp);
ctx->callback_count++;
}
++#endif
+-static void test_kmalloc_kfree_nolock(struct kunit *test)
++#ifdef CONFIG_PERF_EVENTS
++static struct perf_event_attr hw_attr = {
++ .type = PERF_TYPE_HARDWARE,
++ .config = PERF_COUNT_HW_CPU_CYCLES,
++ .size = sizeof(struct perf_event_attr),
++ .pinned = 1,
++ .disabled = 1,
++ .freq = 1,
++ .sample_freq = 100000,
++};
++
+static void overflow_handler_test_nolock(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
@@ lib/tests/slub_kunit.c: static void overflow_handler_test_nolock(struct perf_eve
+ test_nolock(ctx);
+}
+
- static bool enable_perf_events(struct test_nolock_context *ctx)
++static bool enable_perf_events(struct test_nolock_context *ctx)
{
+- int i, j;
+- struct test_nolock_context ctx = { .test = test };
struct perf_event *event;
-@@ lib/tests/slub_kunit.c: static void disable_perf_events(struct test_nolock_context *ctx)
- perf_event_disable(ctx->event);
- perf_event_release_kernel(ctx->event);
- }
-+#else
-+static bool enable_perf_events(struct test_nolock_context *ctx) { return false; }
-+static void disable_perf_events(struct test_nolock_context *ctx) { }
+- bool alloc_fail = false;
+
+ event = perf_event_create_kernel_counter(&hw_attr, -1, current,
+- overflow_handler_test_kmalloc_kfree_nolock,
+- &ctx);
++ overflow_handler_test_nolock,
++ ctx);
++
+ if (IS_ERR(event))
+- kunit_skip(test, "Failed to create perf event");
+- ctx.event = event;
+- perf_event_enable(ctx.event);
+- for (i = 0; i < NR_ITERATIONS; i++) {
+- for (j = 0; j < NR_OBJECTS; j++) {
+- gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
++ return false;
+
+- objects[j] = kmalloc(64, gfp);
+- if (!objects[j]) {
+- j--;
+- while (j >= 0)
+- kfree(objects[j--]);
+- alloc_fail = true;
+- goto cleanup;
+- }
+- }
+- for (j = 0; j < NR_OBJECTS; j++)
+- kfree(objects[j]);
+- }
++ ctx->event = event;
++ perf_event_enable(ctx->event);
++ return true;
++}
+
+-cleanup:
+- perf_event_disable(ctx.event);
+- perf_event_release_kernel(ctx.event);
++static void disable_perf_events(struct test_nolock_context *ctx)
++{
++ kunit_info(ctx->test, "HW perf events: callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
++ ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail);
++
++ perf_event_disable(ctx->event);
++ perf_event_release_kernel(ctx->event);
++}
++
++static void test_kmalloc_kfree_nolock_perf(struct kunit *test)
++{
++ struct test_nolock_context ctx = { .test = test };
+
+- kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
+- ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail);
++ if (!enable_perf_events(&ctx))
++ kunit_skip(test, "Failed to enable perf event, skipping");
+
+- if (alloc_fail)
+- kunit_skip(test, "Allocation failed");
++ test_kmalloc_kfree();
++
++ disable_perf_events(&ctx);
++ KUNIT_EXPECT_EQ(test, 0, slab_errors);
++}
+#endif
+
+#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
@@ lib/tests/slub_kunit.c: static void disable_perf_events(struct test_nolock_conte
+ ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail);
+ unregister_kprobe(&ctx->kprobe);
+}
-+#else
-+static bool register_slab_kprobes(struct test_nolock_context *ctx) { return false; }
-+static void unregister_slab_kprobes(struct test_nolock_context *ctx) { }
-+#endif
-
- static void test_kmalloc_kfree_nolock(struct kunit *test)
- {
- int i, j;
-- struct test_nolock_context ctx = { .test = test };
-+ struct test_nolock_context perf_ctx = { .test = test };
-+ struct test_nolock_context kprobe_ctx = { .test = test };
- bool alloc_fail = false;
- bool perf_events_enabled;
-+ bool slab_kprobes_enabled;
-
-- perf_events_enabled = enable_perf_events(&ctx);
-- if (!perf_events_enabled)
-- kunit_skip(test, "Failed to create perf event");
-+ perf_events_enabled = enable_perf_events(&perf_ctx);
-+ slab_kprobes_enabled = register_slab_kprobes(&kprobe_ctx);
+
-+ if (!perf_events_enabled && !slab_kprobes_enabled)
-+ kunit_skip(test, "Failed to enable perf event and kprobe, skipping");
-+ else if (!perf_events_enabled)
-+ kunit_info(test, "Failed to create perf event");
-+ if (!slab_kprobes_enabled)
-+ kunit_info(test, "Failed to register kprobe pre-handler");
-
- for (i = 0; i < NR_ITERATIONS; i++) {
- for (j = 0; j < NR_OBJECTS; j++) {
-@@ lib/tests/slub_kunit.c: static void test_kmalloc_kfree_nolock(struct kunit *test)
- }
-
- cleanup:
-- disable_perf_events(&ctx);
-+ if (perf_events_enabled)
-+ disable_perf_events(&perf_ctx);
-+ if (slab_kprobes_enabled)
-+ unregister_slab_kprobes(&kprobe_ctx);
-
- if (alloc_fail)
- kunit_skip(test, "Allocation failed");
++static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
++{
++ struct test_nolock_context ctx = { .test = test };
++
++ if (!register_slab_kprobes(&ctx))
++ kunit_skip(test, "Failed to register kprobe, skipping");
++
++ test_kmalloc_kfree();
++
++ unregister_slab_kprobes(&ctx);
+ KUNIT_EXPECT_EQ(test, 0, slab_errors);
+ }
+ #endif
@@ lib/tests/slub_kunit.c: static struct kunit_case test_cases[] = {
- KUNIT_CASE(test_kfree_rcu_wq_destroy),
KUNIT_CASE(test_leak_destroy),
KUNIT_CASE(test_krealloc_redzone_zeroing),
--#ifdef CONFIG_PERF_EVENTS
-+#ifdef SLUB_KUNIT_TEST_KMALLOC_KFREE_NOLOCK
- KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock),
+ #ifdef CONFIG_PERF_EVENTS
+- KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock),
++ KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_perf),
++#endif
++#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
++ KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_kprobe),
#endif
{}
+ };
## mm/slub.c ##
@@ mm/slub.c: static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
3: b6661675c309 ! 2: b33e9d9e4028 mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
@@ Commit message
an empty sheaf because the fallback path will be more expensive than
kfree_nolock().
+ Now that slab has internal alloc_flags to describe context, introduce
+ free_flags analogously and convert free_flags to alloc_flags when
+ allocating memory in the free path. alloc_empty_sheaf() now strips
+ __GFP_RECLAIM when SLAB_ALLOC_NOLOCK is specified.
+
When trylock fails or the kernel observes non-NULL pcs->rcu_free after
lock acquisition, free the sheaf instead of putting it to the barn.
This is rare and not worth complicating the code.
@@ Commit message
Since call_rcu() cannot be called in an unknown context,
kfree_rcu_sheaf() fails when the rcu sheaf becomes full.
+ Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5@kernel.org
## mm/slab.h ##
+@@
+ #define SLAB_ALLOC_NEW_SLAB 0x02 /* a flag for alloc_slab_obj_exts() */
+ #define SLAB_ALLOC_NO_RECURSE 0x04 /* prevent kmalloc() recursion */
+
++#define SLAB_FREE_DEFAULT 0x00 /* no flags */
++#define SLAB_FREE_NOLOCK 0x01 /* spinning not allowed */
++
++static inline unsigned int to_alloc_flags(unsigned int free_flags)
++{
++ if (free_flags & SLAB_FREE_NOLOCK)
++ return SLAB_ALLOC_NOLOCK;
++ else
++ return SLAB_ALLOC_DEFAULT;
++}
++
+ static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
+ {
+ return !(alloc_flags & SLAB_ALLOC_NOLOCK);
+ }
+
++static inline bool free_flags_allow_spinning(const unsigned int free_flags)
++{
++ return !(free_flags & SLAB_FREE_NOLOCK);
++}
++
+ void *__kmalloc_flags_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags,
+ unsigned int alloc_flags, int node)
+ __assume_kmalloc_alignment __alloc_size(1);
@@ mm/slab.h: static inline bool is_kmalloc_normal(struct kmem_cache *s)
return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
}
-bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
-+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin);
++bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags);
void flush_all_rcu_sheaves(void);
void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
@@ mm/slab_common.c: static bool kfree_rcu_sheaf(void *obj)
s = slab->slab_cache;
if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
- return __kfree_rcu_sheaf(s, obj);
-+ return __kfree_rcu_sheaf(s, obj, /* allow_spin = */ true);
++ return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
return false;
}
## mm/slub.c ##
@@ mm/slub.c: static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s,
+
+ gfp &= ~OBJCGS_CLEAR_MASK;
+
++ if (alloc_flags & SLAB_ALLOC_NOLOCK)
++ gfp &= ~__GFP_RECLAIM;
++
return __alloc_empty_sheaf(s, gfp, alloc_flags, s->sheaf_capacity);
}
@@ mm/slub.c: static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf
static unsigned int
refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
unsigned int max);
-@@ mm/slub.c: static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn,
- * intended action due to a race or cpu migration. Thus they do not check the
- * empty or full sheaf limits for simplicity.
- */
--
- static void barn_put_empty_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf)
- {
- unsigned long flags;
@@ mm/slub.c: static void rcu_free_sheaf(struct rcu_head *head)
*/
static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG);
-bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
-+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)
++bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
{
struct slub_percpu_sheaves *pcs;
struct slab_sheaf *rcu_sheaf;
++ bool allow_spin = free_flags_allow_spinning(free_flags);
+
+ if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
+ return false;
@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
pcs = this_cpu_ptr(s->cpu_sheaves);
@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
-
struct slab_sheaf *empty;
struct node_barn *barn;
-+ unsigned int alloc_flags = SLAB_ALLOC_DEFAULT;
-+ gfp_t gfp = GFP_NOWAIT;
++ unsigned int alloc_flags = to_alloc_flags(free_flags);
/* Bootstrap or debug cache, fall back */
if (unlikely(!cache_has_sheaves(s))) {
@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
local_unlock(&s->cpu_sheaves->lock);
- empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
-+ if (unlikely(!allow_spin)) {
-+ alloc_flags = SLAB_ALLOC_TRYLOCK;
-+ gfp = 0;
-+ }
-+
-+ empty = alloc_empty_sheaf(s, gfp, alloc_flags);
++ empty = alloc_empty_sheaf(s, GFP_NOWAIT, alloc_flags);
if (!empty)
goto fail;
4: 7ac83bd3f701 < -: ------------ mm/slab: use call_rcu() in unknown context if irqs are enabled
-: ------------ > 3: b4e3709e745a mm/slab: use call_rcu() in unknown context if irqs are enabled
5: 18d3f7fac0db ! 4: a4f48243b29f mm/slab: extend deferred free mechanism to handle rcu sheaves
@@ Commit message
deferred free fallback so that a full rcu sheaf can be submitted to
call_rcu() via the existing IRQ work.
- Since the deferred mechanism does more than deferred free of objects,
+ Since the deferred mechanism does more than deferred freeing of objects,
rename the struct to deferred_percpu_work and adjust names accordingly.
When a sheaf is queued on an IRQ work, it is detached from
@@ Commit message
CPU before calling rcu_barrier().
In the meantime, remove the TODO item as apparently there is no simple
- and effective way to achieve that.
+ and effective way to achieve that. This is because, unlike sheaves,
+ kfree_rcu() batches objects from different caches together.
+ Reviewed-by: Pedro Falcato <pfalcato@suse.de>
+ Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
## mm/slab.h ##
@@ mm/slub.c: void flush_all_rcu_sheaves(void)
rcu_barrier();
}
-@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)
+@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
rcu_sheaf = NULL;
} else {
-- /* call_rcu() disables IRQs to protect percpu data structures */
+- /*
+- * With !allow_spin, we might have interrupted call_rcu()'s
+- * IRQ-disabled critical section. If IRQs are not disabled,
+- * we know that's not the case.
+- */
- if (unlikely(!allow_spin && irqs_disabled())) {
- rcu_sheaf->size--;
- local_unlock(&s->cpu_sheaves->lock);
@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_sp
pcs->rcu_free = NULL;
rcu_sheaf->node = numa_node_id();
}
-@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)
+@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
* we flush before local_unlock to make sure a racing
* flush_all_rcu_sheaves() doesn't miss this sheaf
*/
- if (rcu_sheaf)
- call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+ if (rcu_sheaf) {
-+ /* call_rcu() disables IRQs to protect percpu data structures */
++ /*
++ * With !allow_spin, we might have interrupted call_rcu()'s
++ * IRQ-disabled critical section. If IRQs are not disabled,
++ * we know that's not the case.
++ */
+ if (unlikely(!allow_spin && irqs_disabled())) {
+ struct deferred_percpu_work *dpw;
+
@@ mm/slub.c: static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void
+ struct deferred_percpu_work *dpw;
+ struct llist_head *objs, *rcu_sheaves;
struct llist_node *llnode, *pos, *t;
++ struct slab_sheaf *sheaf, *next;
- if (llist_empty(objs))
- return;
@@ mm/slub.c: static void free_deferred_objects(struct irq_work *work)
}
+
+ llnode = llist_del_all(rcu_sheaves);
-+ llist_for_each_safe(pos, t, llnode) {
-+ struct slab_sheaf *rcu_sheaf = llist_entry(pos, struct slab_sheaf, llnode);
-+
-+ call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
-+ }
++ llist_for_each_entry_safe(sheaf, next, llnode, llnode)
++ call_rcu(&sheaf->rcu_head, rcu_free_sheaf);
}
static void defer_free(struct kmem_cache *s, void *head)
6: e7e9c9c8738c ! 5: d343d20115a6 mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
@@ Metadata
## Commit message ##
- As suggested by Vlastimil Babka, kfree_rcu_sheaf() can be used
+ As suggested by Vlastimil Babka [1], kfree_rcu_sheaf() can be used
on PREEMPT_RT if we always assume spinning is not allowed on PREEMPT_RT.
This is because local_trylock and spinlock_t are safe to use with
- trylock variant as long as the kernel does not spin and the context is
- not NMI and not hardirq.
+ trylock and unlock as long as the kernel does not spin and the context
+ is not NMI and not hardirq.
- Now that __kfree_rcu_sheaf() knows how to handle allow_spin = false,
+ Now that __kfree_rcu_sheaf() knows how to handle SLAB_FREE_NOLOCK,
relax the limitation and try the sheaves path on PREEMPT_RT as well.
Keep the lockdep map on non RT kernels. However, do not use the lockdep
map on PREEMPT_RT to avoid suppressing valid lockdep warnings.
- Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org
+ As pointed by Vlastimil Babka [2], on PREEMPT_RT it is unnecessary to
+ defer call_rcu() under IRQ-disabled section or raw spinlock. However,
+ let us avoid adding more complexity as the scenario is not supposed
+ to be common on PREEMPT_RT, with a hope that call_rcu_nolock() will be
+ soon supported in RCU.
+
+ Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org [1]
+ Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be@kernel.org [2]
@@ mm/slab_common.c: static bool kfree_rcu_sheaf(void *obj)
{
struct kmem_cache *s;
struct slab *slab;
-+ bool allow_spin;
++ unsigned int free_flags = SLAB_FREE_DEFAULT;
+
+ /*
+ * It is not safe to spin on PREEMPT_RT because the kernel might be
+ * holding a raw spinlock and slab acquires sleeping locks.
+ */
-+ allow_spin = !IS_ENABLED(CONFIG_PREEMPT_RT);
++ if (IS_ENABLED(CONFIG_PREEMPT_RT))
++ free_flags = SLAB_FREE_NOLOCK;
if (is_vmalloc_addr(obj))
return false;
@@ mm/slab_common.c: static bool kfree_rcu_sheaf(void *obj)
s = slab->slab_cache;
if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
-- return __kfree_rcu_sheaf(s, obj, /* allow_spin = */ true);
-+ return __kfree_rcu_sheaf(s, obj, allow_spin);
+- return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
++ return __kfree_rcu_sheaf(s, obj, free_flags);
return false;
}
@@ mm/slub.c: static void rcu_free_sheaf(struct rcu_head *head)
*/
static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG);
-@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)
- struct slub_percpu_sheaves *pcs;
+@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
struct slab_sheaf *rcu_sheaf;
+ bool allow_spin = free_flags_allow_spinning(free_flags);
- if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
- return false;
@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_sp
if (!local_trylock(&s->cpu_sheaves->lock))
goto fail;
-@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, bool allow_spin)
+@@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
local_unlock(&s->cpu_sheaves->lock);
stat(s, FREE_RCU_SHEAF);
-: ------------ > 6: 5e34a69a743f mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
7: 883f26a02d3f ! 7: 428e78b8b089 mm/slab: introduce kfree_rcu_nolock()
@@ Commit message
now that kfree_rcu_sheaf() is available on PREEMPT_RT and
__kfree_rcu_sheaf() handles unknown context.
- Unlike k[v]free_rcu(), kfree_rcu_nolock() does not fall back to
- the kvfree_rcu batching when the sheaves path fails, and falls back to
- defer_kfree_rcu() instead. In most cases, the sheaves path is expected
- to succeed and it's unnecessary to add complexity to the existing
- kvfree_rcu batching.
+ kfree_rcu_nolock() falls back to the kvfree_rcu batching when sheaves
+ path fails. In most cases, the sheaves path is expected to succeed
+ and it's unnecessary to add complexity to the existing kvfree_rcu
+ batching by teaching it how to handle unknown context.
Since defer_kfree_rcu() can be called on caches without sheaves, move
deferred_work_barrier() and rcu_barrier() outside the branch in
@@ include/linux/rcupdate.h
@@ include/linux/rcupdate.h: static inline void rcu_read_unlock_migrate(void)
* In mm/slab_common.c, no suitable header to include here.
*/
- void kvfree_call_rcu(struct rcu_head *head, void *ptr);
-+void kfree_call_rcu_nolock(struct rcu_head *head, void *ptr);
+ void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr);
++void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr);
/*
* The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the
@@ include/linux/rcupdate.h: do { \
kvfree_call_rcu(NULL, (void *) (___p)); \
} while (0)
-+/* kfree_rcu_nolock() supports 2-arg variant only */
-+#define kfree_rcu_nolock(ptr, krhf) \
-+do { \
-+ typeof (ptr) ___p = (ptr); \
-+ \
-+ if (___p) { \
-+ BUILD_BUG_ON(offsetof(typeof(*(ptr)), krhf) >= 4096); \
-+ kfree_call_rcu_nolock(&((___p)->krhf), (void *) (___p));\
-+ } \
++/**
++ * kfree_rcu_nolock() - a version of kfree_rcu() that can be called in any context.
++ * @ptr: pointer to kfree for double-argument invocations.
++ * @kvrhf: the name of the struct kvfree_rcu_head within the type of @ptr.
++ *
++ * Objects that are not allocated via kmalloc_nolock() are not supported.
++ * kfree_rcu_nolock() supports 2-arg variant only.
++ */
++#define kfree_rcu_nolock(ptr, kvrhf) \
++do { \
++ typeof (ptr) ___p = (ptr); \
++ \
++ if (___p) { \
++ BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >= 4096); \
++ kfree_call_rcu_nolock(&((___p)->kvrhf), (void *) (___p)); \
++ } \
+} while (0)
+
/*
* Place this after a lock-acquisition primitive to guarantee that
* an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
- ## mm/slab.h ##
-@@ mm/slab.h: void __check_heap_object(const void *ptr, unsigned long n,
- const struct slab *slab, bool to_user);
-
- void deferred_work_barrier(void);
-+void defer_kfree_rcu(struct rcu_head *head);
-
- static inline bool slub_debug_orig_size(struct kmem_cache *s)
- {
-
## mm/slab_common.c ##
@@ mm/slab_common.c: EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
EXPORT_TRACEPOINT_SYMBOL(kfree);
EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free);
-+void kfree_call_rcu_nolock(struct rcu_head *head, void *ptr)
++void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr)
+{
+ struct slab *slab;
+ struct kmem_cache *s;
@@ mm/slab_common.c: EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
+ slab = virt_to_slab(ptr);
+ s = slab->slab_cache;
+
-+ if (__kfree_rcu_sheaf(s, ptr, /* allow_spin = */ false))
-+ return;
++ if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != numa_mem_id()))
++ goto fallback;
++
++ if (unlikely(!__kfree_rcu_sheaf(s, ptr, SLAB_FREE_NOLOCK)))
++ goto fallback;
++
++ return;
+
++fallback:
+ defer_kfree_rcu(head);
+}
+EXPORT_SYMBOL_GPL(kfree_call_rcu_nolock);
@@ mm/slub.c: static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void
- struct llist_head *objs, *rcu_sheaves;
+ struct llist_head *objs, *objs_by_rcu, *rcu_sheaves;
struct llist_node *llnode, *pos, *t;
+ struct slab_sheaf *sheaf, *next;
dpw = container_of(work, struct deferred_percpu_work, work);
rcu_sheaves = &dpw->rcu_sheaves;
@@ mm/slub.c: static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void
llnode = llist_del_all(objs);
llist_for_each_safe(pos, t, llnode) {
@@ mm/slub.c: static void deferred_percpu_work_fn(struct irq_work *work)
-
- call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+ stat(s, FREE_SLOWPATH);
}
-+
+
+ llnode = llist_del_all(objs_by_rcu);
+ llist_for_each_safe(pos, t, llnode) {
-+ struct rcu_head *head = (struct rcu_head *)pos;
++ void *head = pos;
++ void *objp = kvmalloc_obj_start_addr(head);
+
-+ call_rcu(head, kvfree_rcu_cb);
++ kvfree_call_rcu(head, objp);
+ }
- }
-
- static void defer_free(struct kmem_cache *s, void *head)
++
+ llnode = llist_del_all(rcu_sheaves);
+ llist_for_each_entry_safe(sheaf, next, llnode, llnode)
+ call_rcu(&sheaf->rcu_head, rcu_free_sheaf);
@@ mm/slub.c: static void defer_free(struct kmem_cache *s, void *head)
irq_work_queue(&dpw->work);
}
-+void defer_kfree_rcu(struct rcu_head *head)
++void defer_kfree_rcu(struct kvfree_rcu_head *head)
+{
+ struct deferred_percpu_work *dpw;
+
8: c467c0a3cdfa < -: ------------ mm/slab: introduce struct kfree_rcu_head and use in kfree_rcu_nolock()
9: d4cdda36b1c3 < -: ------------ slub_kunit: extend the test for kfree_rcu_nolock()
-: ------------ > 8: f5c8670195d4 slub_kunit: extend the test for kfree_rcu_nolock()
Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a92d9a4@kernel.org [1]
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
Suggested-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
Harry Yoo (Oracle) (8):
mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
mm/slab: use call_rcu() in unknown context if irqs are enabled
mm/slab: extend deferred free mechanism to handle rcu sheaves
mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
mm/slab: introduce kfree_rcu_nolock()
slub_kunit: extend the test for kfree_rcu_nolock()
include/linux/rcupdate.h | 29 +++++-
include/linux/types.h | 8 ++
include/trace/events/rcu.h | 2 +-
lib/tests/slub_kunit.c | 190 ++++++++++++++++++++++++++++-----------
mm/slab.h | 48 +++++++++-
mm/slab_common.c | 62 +++++++++----
mm/slub.c | 220 ++++++++++++++++++++++++++++-----------------
7 files changed, 404 insertions(+), 155 deletions(-)
---
base-commit: 56aa9b819edce4acfa879728af56534c951d1818
change-id: 20260615-kfree_rcu_nolock-e5502555992f
Best regards,
--
Harry Yoo (Oracle) <harry@kernel.org>
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
2026-07-20 12:56 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() Harry Yoo (Oracle)
` (6 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
Since kmalloc_nolock() always fails in NMI and hardirq contexts on
PREEMPT_RT, slub_kunit cannot properly test _nolock() APIs.
Register a kprobe pre-handler to invoke kmalloc_nolock() and
kfree_nolock() in the middle of the slab allocator. However, do not
register the handler on UP kernels because that use case is not
well supported [1] in the kernel.
To attach the pre-handler while s->cpu_sheaves->lock or n->list_lock
is held, add a wrapper function for lockdep_assert_held() that calls
a no-op function slab_attach_kprobe_locked() on debug builds. The
function is optimized away when neither CONFIG_PROVE_LOCKING nor
CONFIG_DEBUG_VM is selected and register_kprobe() fails.
The function calls barrier() to prevent the compiler from optimizing
away its callsites. Otherwise, the compiler may consider the function
does not have any side effect and remove callsites.
Compared to using plain kprobe, this has two advantages: 1) it avoids
hardcoding function names in the test, and 2) it can trigger those APIs
in the middle of a function, where the lock is expected to be held as
annotated with lockdep.
While it was proposed [2] to use kunit function redirection to test
this, it is currently infeasible as some lock helpers don't have
symbols.
Factor out the nested loop that calls kmalloc and friends to
test_kmalloc_kfree(), and call them in
test_kmalloc_kfree_nolock_{perf,kprobe}(), each being an independent
test case. During the refactoring, drop alloc_fail handling as it
doesn't provide much benefits.
Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a92d9a4@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b@kernel.org [2]
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
lib/tests/slub_kunit.c | 167 +++++++++++++++++++++++++++++++++++--------------
mm/slub.c | 36 ++++++++---
2 files changed, 148 insertions(+), 55 deletions(-)
diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
index fa6d31dbca16..a391467c1aa6 100644
--- a/lib/tests/slub_kunit.c
+++ b/lib/tests/slub_kunit.c
@@ -8,6 +8,7 @@
#include <linux/rcupdate.h>
#include <linux/delay.h>
#include <linux/perf_event.h>
+#include <linux/kprobes.h>
#include "../mm/slab.h"
static struct kunit_resource resource;
@@ -292,7 +293,7 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)
kmem_cache_destroy(s);
}
-#ifdef CONFIG_PERF_EVENTS
+#if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
#define NR_ITERATIONS 1000
#define NR_OBJECTS 1000
static void *objects[NR_OBJECTS];
@@ -302,26 +303,40 @@ struct test_nolock_context {
int callback_count;
int alloc_ok;
int alloc_fail;
+#ifdef CONFIG_PERF_EVENTS
struct perf_event *event;
+#endif
+#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
+ struct kprobe kprobe;
+#endif
};
-static struct perf_event_attr hw_attr = {
- .type = PERF_TYPE_HARDWARE,
- .config = PERF_COUNT_HW_CPU_CYCLES,
- .size = sizeof(struct perf_event_attr),
- .pinned = 1,
- .disabled = 1,
- .freq = 1,
- .sample_freq = 100000,
-};
+static void test_kmalloc_kfree(void)
+{
+ int i, j;
+
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ for (j = 0; j < NR_OBJECTS; j++) {
+ gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
-static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
- struct perf_sample_data *data,
- struct pt_regs *regs)
+ objects[j] = kmalloc_obj(*objects[j], gfp);
+ if (!objects[j]) {
+ j--;
+ while (j >= 0)
+ kfree(objects[j--]);
+ return;
+ }
+ }
+
+ for (j = 0; j < NR_OBJECTS; j++)
+ kfree(objects[j]);
+ }
+}
+
+static void test_nolock(struct test_nolock_context *ctx)
{
void *objp;
gfp_t gfp;
- struct test_nolock_context *ctx = event->overflow_handler_context;
/* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */
gfp = (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT;
@@ -335,47 +350,104 @@ static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
kfree_nolock(objp);
ctx->callback_count++;
}
+#endif
-static void test_kmalloc_kfree_nolock(struct kunit *test)
+#ifdef CONFIG_PERF_EVENTS
+static struct perf_event_attr hw_attr = {
+ .type = PERF_TYPE_HARDWARE,
+ .config = PERF_COUNT_HW_CPU_CYCLES,
+ .size = sizeof(struct perf_event_attr),
+ .pinned = 1,
+ .disabled = 1,
+ .freq = 1,
+ .sample_freq = 100000,
+};
+
+static void overflow_handler_test_nolock(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ struct test_nolock_context *ctx = event->overflow_handler_context;
+
+ test_nolock(ctx);
+}
+
+static bool enable_perf_events(struct test_nolock_context *ctx)
{
- int i, j;
- struct test_nolock_context ctx = { .test = test };
struct perf_event *event;
- bool alloc_fail = false;
event = perf_event_create_kernel_counter(&hw_attr, -1, current,
- overflow_handler_test_kmalloc_kfree_nolock,
- &ctx);
+ overflow_handler_test_nolock,
+ ctx);
+
if (IS_ERR(event))
- kunit_skip(test, "Failed to create perf event");
- ctx.event = event;
- perf_event_enable(ctx.event);
- for (i = 0; i < NR_ITERATIONS; i++) {
- for (j = 0; j < NR_OBJECTS; j++) {
- gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
+ return false;
- objects[j] = kmalloc(64, gfp);
- if (!objects[j]) {
- j--;
- while (j >= 0)
- kfree(objects[j--]);
- alloc_fail = true;
- goto cleanup;
- }
- }
- for (j = 0; j < NR_OBJECTS; j++)
- kfree(objects[j]);
- }
+ ctx->event = event;
+ perf_event_enable(ctx->event);
+ return true;
+}
-cleanup:
- perf_event_disable(ctx.event);
- perf_event_release_kernel(ctx.event);
+static void disable_perf_events(struct test_nolock_context *ctx)
+{
+ kunit_info(ctx->test, "HW perf events: callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
+ ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail);
+
+ perf_event_disable(ctx->event);
+ perf_event_release_kernel(ctx->event);
+}
+
+static void test_kmalloc_kfree_nolock_perf(struct kunit *test)
+{
+ struct test_nolock_context ctx = { .test = test };
- kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
- ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail);
+ if (!enable_perf_events(&ctx))
+ kunit_skip(test, "Failed to enable perf event, skipping");
- if (alloc_fail)
- kunit_skip(test, "Allocation failed");
+ test_kmalloc_kfree();
+
+ disable_perf_events(&ctx);
+ KUNIT_EXPECT_EQ(test, 0, slab_errors);
+}
+#endif
+
+#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
+static int slab_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
+{
+ struct test_nolock_context *ctx;
+
+ ctx = container_of(p, struct test_nolock_context, kprobe);
+ test_nolock(ctx);
+ return 0;
+}
+
+static bool register_slab_kprobes(struct test_nolock_context *ctx)
+{
+ ctx->kprobe.symbol_name = "slab_attach_kprobe_locked";
+ ctx->kprobe.pre_handler = slab_kprobe_pre_handler;
+
+ if (register_kprobe(&ctx->kprobe))
+ return false;
+ return true;
+}
+
+static void unregister_slab_kprobes(struct test_nolock_context *ctx)
+{
+ kunit_info(ctx->test, "kprobes: callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
+ ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail);
+ unregister_kprobe(&ctx->kprobe);
+}
+
+static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
+{
+ struct test_nolock_context ctx = { .test = test };
+
+ if (!register_slab_kprobes(&ctx))
+ kunit_skip(test, "Failed to register kprobe, skipping");
+
+ test_kmalloc_kfree();
+
+ unregister_slab_kprobes(&ctx);
KUNIT_EXPECT_EQ(test, 0, slab_errors);
}
#endif
@@ -405,7 +477,10 @@ static struct kunit_case test_cases[] = {
KUNIT_CASE(test_leak_destroy),
KUNIT_CASE(test_krealloc_redzone_zeroing),
#ifdef CONFIG_PERF_EVENTS
- KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock),
+ KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_perf),
+#endif
+#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
+ KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_kprobe),
#endif
{}
};
diff --git a/mm/slub.c b/mm/slub.c
index da9efb040040..e32a68677537 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -908,6 +908,24 @@ static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
}
#endif
+/*
+ * A no-op function used to attach kprobe handlers in slub_kunit tests.
+ * The barrier is needed to prevent the compiler from optimizing out callsites.
+ */
+#if defined(CONFIG_DEBUG_VM) || defined(CONFIG_PROVE_LOCKING)
+static noinline void slab_attach_kprobe_locked(void)
+{
+ barrier();
+}
+#else
+static inline void slab_attach_kprobe_locked(void) { }
+#endif
+
+#define slab_lockdep_assert_held(lock) do { \
+ lockdep_assert_held(lock); \
+ slab_attach_kprobe_locked(); \
+} while (0)
+
#ifdef CONFIG_SLUB_DEBUG
/*
@@ -1665,7 +1683,7 @@ static void add_full(struct kmem_cache *s,
if (!(s->flags & SLAB_STORE_USER))
return;
- lockdep_assert_held(&n->list_lock);
+ slab_lockdep_assert_held(&n->list_lock);
list_add(&slab->slab_list, &n->full);
}
@@ -1674,7 +1692,7 @@ static void remove_full(struct kmem_cache *s, struct kmem_cache_node *n, struct
if (!(s->flags & SLAB_STORE_USER))
return;
- lockdep_assert_held(&n->list_lock);
+ slab_lockdep_assert_held(&n->list_lock);
list_del(&slab->slab_list);
}
@@ -2868,7 +2886,7 @@ static unsigned int __sheaf_flush_main_batch(struct kmem_cache *s)
void *objects[PCS_BATCH_MAX];
struct slab_sheaf *sheaf;
- lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+ slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
pcs = this_cpu_ptr(s->cpu_sheaves);
sheaf = pcs->main;
@@ -3547,7 +3565,7 @@ __add_partial(struct kmem_cache_node *n, struct slab *slab, enum add_mode mode)
static inline void add_partial(struct kmem_cache_node *n,
struct slab *slab, enum add_mode mode)
{
- lockdep_assert_held(&n->list_lock);
+ slab_lockdep_assert_held(&n->list_lock);
__add_partial(n, slab, mode);
}
@@ -3561,7 +3579,7 @@ static inline void clear_node_partial_state(struct kmem_cache_node *n,
static inline void remove_partial(struct kmem_cache_node *n,
struct slab *slab)
{
- lockdep_assert_held(&n->list_lock);
+ slab_lockdep_assert_held(&n->list_lock);
list_del(&slab->slab_list);
clear_node_partial_state(n, slab);
}
@@ -3577,7 +3595,7 @@ static void *alloc_single_from_partial(struct kmem_cache *s,
{
void *object;
- lockdep_assert_held(&n->list_lock);
+ slab_lockdep_assert_held(&n->list_lock);
#ifdef CONFIG_SLUB_DEBUG
if (s->flags & SLAB_CONSISTENCY_CHECKS) {
@@ -4642,7 +4660,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
struct node_barn *barn;
bool allow_spin;
- lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+ slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
/* Bootstrap or debug cache, back off */
if (unlikely(!cache_has_sheaves(s))) {
@@ -5765,7 +5783,7 @@ static void __pcs_install_empty_sheaf(struct kmem_cache *s,
struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty,
struct node_barn *barn)
{
- lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+ slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
/* This is what we expect to find if nobody interrupted us. */
if (likely(!pcs->spare)) {
@@ -5816,7 +5834,7 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
bool put_fail;
restart:
- lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+ slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
/* Bootstrap or debug cache, back off */
if (unlikely(!cache_has_sheaves(s))) {
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
2026-07-20 12:44 ` [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
2026-07-20 13:11 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled Harry Yoo (Oracle)
` (5 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
Teach kfree_rcu_sheaf() how to handle the !allow_spin case. Try to get
an empty sheaf from pcs->spare or the barn even when spinning is not
allowed. Unlike __pcs_replace_full_main(), try harder to allocate
an empty sheaf because the fallback path will be more expensive than
kfree_nolock().
Now that slab has internal alloc_flags to describe context, introduce
free_flags analogously and convert free_flags to alloc_flags when
allocating memory in the free path. alloc_empty_sheaf() now strips
__GFP_RECLAIM when SLAB_ALLOC_NOLOCK is specified.
When trylock fails or the kernel observes non-NULL pcs->rcu_free after
lock acquisition, free the sheaf instead of putting it to the barn.
This is rare and not worth complicating the code.
Since call_rcu() cannot be called in an unknown context,
kfree_rcu_sheaf() fails when the rcu sheaf becomes full.
Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5@kernel.org
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
mm/slab.h | 18 +++++++++++++++++-
mm/slab_common.c | 2 +-
mm/slub.c | 36 ++++++++++++++++++++++++++++--------
3 files changed, 46 insertions(+), 10 deletions(-)
diff --git a/mm/slab.h b/mm/slab.h
index 281a65233795..85ef2ebc9812 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -23,11 +23,27 @@
#define SLAB_ALLOC_NEW_SLAB 0x02 /* a flag for alloc_slab_obj_exts() */
#define SLAB_ALLOC_NO_RECURSE 0x04 /* prevent kmalloc() recursion */
+#define SLAB_FREE_DEFAULT 0x00 /* no flags */
+#define SLAB_FREE_NOLOCK 0x01 /* spinning not allowed */
+
+static inline unsigned int to_alloc_flags(unsigned int free_flags)
+{
+ if (free_flags & SLAB_FREE_NOLOCK)
+ return SLAB_ALLOC_NOLOCK;
+ else
+ return SLAB_ALLOC_DEFAULT;
+}
+
static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
{
return !(alloc_flags & SLAB_ALLOC_NOLOCK);
}
+static inline bool free_flags_allow_spinning(const unsigned int free_flags)
+{
+ return !(free_flags & SLAB_FREE_NOLOCK);
+}
+
void *__kmalloc_flags_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags,
unsigned int alloc_flags, int node)
__assume_kmalloc_alignment __alloc_size(1);
@@ -429,7 +445,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
}
-bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags);
void flush_all_rcu_sheaves(void);
void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
diff --git a/mm/slab_common.c b/mm/slab_common.c
index b6426d7ceec9..e07b4e6d6679 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1605,7 +1605,7 @@ static bool kfree_rcu_sheaf(void *obj)
s = slab->slab_cache;
if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
- return __kfree_rcu_sheaf(s, obj);
+ return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
return false;
}
diff --git a/mm/slub.c b/mm/slub.c
index e32a68677537..0c350274fbff 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2814,10 +2814,14 @@ static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s,
gfp &= ~OBJCGS_CLEAR_MASK;
+ if (alloc_flags & SLAB_ALLOC_NOLOCK)
+ gfp &= ~__GFP_RECLAIM;
+
return __alloc_empty_sheaf(s, gfp, alloc_flags, s->sheaf_capacity);
}
-static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
+static void __free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
+ bool allow_spin)
{
/*
* If the sheaf was created with SLAB_ALLOC_NO_RECURSE flag then its
@@ -2829,11 +2833,20 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
mark_obj_codetag_empty(sheaf);
VM_WARN_ON_ONCE(sheaf->size > 0);
- kfree(sheaf);
+
+ if (likely(allow_spin))
+ kfree(sheaf);
+ else
+ kfree_nolock(sheaf);
stat(s, SHEAF_FREE);
}
+static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
+{
+ __free_empty_sheaf(s, sheaf, /* allow_spin = */ true);
+}
+
static unsigned int
refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
unsigned int max);
@@ -6044,10 +6057,11 @@ static void rcu_free_sheaf(struct rcu_head *head)
*/
static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG);
-bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
{
struct slub_percpu_sheaves *pcs;
struct slab_sheaf *rcu_sheaf;
+ bool allow_spin = free_flags_allow_spinning(free_flags);
if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
return false;
@@ -6060,9 +6074,9 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
pcs = this_cpu_ptr(s->cpu_sheaves);
if (unlikely(!pcs->rcu_free)) {
-
struct slab_sheaf *empty;
struct node_barn *barn;
+ unsigned int alloc_flags = to_alloc_flags(free_flags);
/* Bootstrap or debug cache, fall back */
if (unlikely(!cache_has_sheaves(s))) {
@@ -6082,7 +6096,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
goto fail;
}
- empty = barn_get_empty_sheaf(barn, true);
+ empty = barn_get_empty_sheaf(barn, allow_spin);
if (empty) {
pcs->rcu_free = empty;
@@ -6091,20 +6105,20 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
local_unlock(&s->cpu_sheaves->lock);
- empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
+ empty = alloc_empty_sheaf(s, GFP_NOWAIT, alloc_flags);
if (!empty)
goto fail;
if (!local_trylock(&s->cpu_sheaves->lock)) {
- barn_put_empty_sheaf(barn, empty);
+ __free_empty_sheaf(s, empty, allow_spin);
goto fail;
}
pcs = this_cpu_ptr(s->cpu_sheaves);
if (unlikely(pcs->rcu_free))
- barn_put_empty_sheaf(barn, empty);
+ __free_empty_sheaf(s, empty, allow_spin);
else
pcs->rcu_free = empty;
}
@@ -6122,6 +6136,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
rcu_sheaf = NULL;
} else {
+ if (unlikely(!allow_spin)) {
+ /* call_rcu() cannot be called in an unknown context */
+ rcu_sheaf->size--;
+ local_unlock(&s->cpu_sheaves->lock);
+ goto fail;
+ }
pcs->rcu_free = NULL;
rcu_sheaf->node = numa_node_id();
}
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
2026-07-20 12:44 ` [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs Harry Yoo (Oracle)
2026-07-20 12:44 ` [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
2026-07-20 13:01 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves Harry Yoo (Oracle)
` (4 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
call_rcu() disables IRQs with local_irq_save() to protect its per-cpu
data structures. Therefore, if IRQs are not disabled, they cannot be
corrupted by reentrance into call_rcu(). So fall back to the deferred
path only when !allow_spin && irqs_disabled().
The RCU subsystem does not guarantee this contractually, and this
optimization relies on RCU's implementation details. Ideally, it should
be removed once call_rcu_nolock() is supported by the RCU subsystem.
Link: https://lore.kernel.org/linux-mm/CAADnVQKRVD5ZSnEKbZZU7w86gHbGHUug2pvzpgZTngNS+fg4rw@mail.gmail.com
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
mm/slub.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 0c350274fbff..680fe64aff72 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6136,8 +6136,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
rcu_sheaf = NULL;
} else {
- if (unlikely(!allow_spin)) {
- /* call_rcu() cannot be called in an unknown context */
+ /*
+ * With !allow_spin, we might have interrupted call_rcu()'s
+ * IRQ-disabled critical section. If IRQs are not disabled,
+ * we know that's not the case.
+ */
+ if (unlikely(!allow_spin && irqs_disabled())) {
rcu_sheaf->size--;
local_unlock(&s->cpu_sheaves->lock);
goto fail;
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
` (2 preceding siblings ...)
2026-07-20 12:44 ` [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
2026-07-20 13:03 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT Harry Yoo (Oracle)
` (3 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
__kfree_rcu_sheaf() cannot invoke call_rcu() when spinning is not
allowed and IRQs are disabled. To relax the limitation, extend the
deferred free fallback so that a full rcu sheaf can be submitted to
call_rcu() via the existing IRQ work.
Since the deferred mechanism does more than deferred freeing of objects,
rename the struct to deferred_percpu_work and adjust names accordingly.
When a sheaf is queued on an IRQ work, it is detached from
pcs->rcu_free but call_rcu() is not invoked until the irq_work runs.
To keep the kvfree_rcu barrier's promise, call irq_work_sync() on each
CPU before calling rcu_barrier().
In the meantime, remove the TODO item as apparently there is no simple
and effective way to achieve that. This is because, unlike sheaves,
kfree_rcu() batches objects from different caches together.
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
mm/slab.h | 2 +-
mm/slab_common.c | 7 ++---
mm/slub.c | 85 +++++++++++++++++++++++++++++++++-----------------------
3 files changed, 53 insertions(+), 41 deletions(-)
diff --git a/mm/slab.h b/mm/slab.h
index 85ef2ebc9812..fdd293b3efa5 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -760,7 +760,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
void __check_heap_object(const void *ptr, unsigned long n,
const struct slab *slab, bool to_user);
-void defer_free_barrier(void);
+void deferred_work_barrier(void);
static inline bool slub_debug_orig_size(struct kmem_cache *s)
{
diff --git a/mm/slab_common.c b/mm/slab_common.c
index e07b4e6d6679..9c9ae1384f47 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -551,7 +551,7 @@ void kmem_cache_destroy(struct kmem_cache *s)
}
/* Wait for deferred work from kmalloc/kfree_nolock() */
- defer_free_barrier();
+ deferred_work_barrier();
cpus_read_lock();
mutex_lock(&slab_mutex);
@@ -2113,13 +2113,10 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
cpus_read_lock();
flush_rcu_sheaves_on_cache(s);
cpus_read_unlock();
+ deferred_work_barrier();
rcu_barrier();
}
- /*
- * TODO: Introduce a version of __kvfree_rcu_barrier() that works
- * on a specific slab cache.
- */
__kvfree_rcu_barrier();
}
EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
diff --git a/mm/slub.c b/mm/slub.c
index 680fe64aff72..8afa6b47b1f2 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -418,6 +418,8 @@ struct slab_sheaf {
union {
struct rcu_head rcu_head;
struct list_head barn_list;
+ /* only used to defer call_rcu() in unknown context */
+ struct llist_node llnode;
/* only used for prefilled sheafs */
struct {
unsigned int capacity;
@@ -4077,6 +4079,20 @@ static void flush_all(struct kmem_cache *s)
cpus_read_unlock();
}
+struct deferred_percpu_work {
+ struct llist_head objects;
+ struct llist_head rcu_sheaves;
+ struct irq_work work;
+};
+
+static void deferred_percpu_work_fn(struct irq_work *work);
+
+static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = {
+ .objects = LLIST_HEAD_INIT(objects),
+ .rcu_sheaves = LLIST_HEAD_INIT(rcu_sheaves),
+ .work = IRQ_WORK_INIT(deferred_percpu_work_fn),
+};
+
static void flush_rcu_sheaf(struct work_struct *w)
{
struct slub_percpu_sheaves *pcs;
@@ -4148,6 +4164,7 @@ void flush_all_rcu_sheaves(void)
mutex_unlock(&slab_mutex);
cpus_read_unlock();
+ deferred_work_barrier();
rcu_barrier();
}
@@ -6136,16 +6153,6 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
rcu_sheaf = NULL;
} else {
- /*
- * With !allow_spin, we might have interrupted call_rcu()'s
- * IRQ-disabled critical section. If IRQs are not disabled,
- * we know that's not the case.
- */
- if (unlikely(!allow_spin && irqs_disabled())) {
- rcu_sheaf->size--;
- local_unlock(&s->cpu_sheaves->lock);
- goto fail;
- }
pcs->rcu_free = NULL;
rcu_sheaf->node = numa_node_id();
}
@@ -6154,8 +6161,22 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
* we flush before local_unlock to make sure a racing
* flush_all_rcu_sheaves() doesn't miss this sheaf
*/
- if (rcu_sheaf)
- call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+ if (rcu_sheaf) {
+ /*
+ * With !allow_spin, we might have interrupted call_rcu()'s
+ * IRQ-disabled critical section. If IRQs are not disabled,
+ * we know that's not the case.
+ */
+ if (unlikely(!allow_spin && irqs_disabled())) {
+ struct deferred_percpu_work *dpw;
+
+ dpw = this_cpu_ptr(&deferred_percpu_work);
+ if (llist_add(&rcu_sheaf->llnode, &dpw->rcu_sheaves))
+ irq_work_queue(&dpw->work);
+ } else {
+ call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+ }
+ }
local_unlock(&s->cpu_sheaves->lock);
@@ -6354,31 +6375,21 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
}
}
-struct defer_free {
- struct llist_head objects;
- struct irq_work work;
-};
-
-static void free_deferred_objects(struct irq_work *work);
-
-static DEFINE_PER_CPU(struct defer_free, defer_free_objects) = {
- .objects = LLIST_HEAD_INIT(objects),
- .work = IRQ_WORK_INIT(free_deferred_objects),
-};
-
/*
* In PREEMPT_RT irq_work runs in per-cpu kthread, so it's safe
* to take sleeping spin_locks from __slab_free().
* In !PREEMPT_RT irq_work will run after local_unlock_irqrestore().
*/
-static void free_deferred_objects(struct irq_work *work)
+static void deferred_percpu_work_fn(struct irq_work *work)
{
- struct defer_free *df = container_of(work, struct defer_free, work);
- struct llist_head *objs = &df->objects;
+ struct deferred_percpu_work *dpw;
+ struct llist_head *objs, *rcu_sheaves;
struct llist_node *llnode, *pos, *t;
+ struct slab_sheaf *sheaf, *next;
- if (llist_empty(objs))
- return;
+ dpw = container_of(work, struct deferred_percpu_work, work);
+ rcu_sheaves = &dpw->rcu_sheaves;
+ objs = &dpw->objects;
llnode = llist_del_all(objs);
llist_for_each_safe(pos, t, llnode) {
@@ -6402,27 +6413,31 @@ static void free_deferred_objects(struct irq_work *work)
__slab_free(s, slab, x, x, 1, _THIS_IP_);
stat(s, FREE_SLOWPATH);
}
+
+ llnode = llist_del_all(rcu_sheaves);
+ llist_for_each_entry_safe(sheaf, next, llnode, llnode)
+ call_rcu(&sheaf->rcu_head, rcu_free_sheaf);
}
static void defer_free(struct kmem_cache *s, void *head)
{
- struct defer_free *df;
+ struct deferred_percpu_work *dpw;
guard(preempt)();
head = kasan_reset_tag(head);
- df = this_cpu_ptr(&defer_free_objects);
- if (llist_add(head + s->offset, &df->objects))
- irq_work_queue(&df->work);
+ dpw = this_cpu_ptr(&deferred_percpu_work);
+ if (llist_add(head + s->offset, &dpw->objects))
+ irq_work_queue(&dpw->work);
}
-void defer_free_barrier(void)
+void deferred_work_barrier(void)
{
int cpu;
for_each_possible_cpu(cpu)
- irq_work_sync(&per_cpu_ptr(&defer_free_objects, cpu)->work);
+ irq_work_sync(&per_cpu_ptr(&deferred_percpu_work, cpu)->work);
}
static __fastpath_inline
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
` (3 preceding siblings ...)
2026-07-20 12:44 ` [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
2026-07-20 12:56 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching Harry Yoo (Oracle)
` (2 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
As suggested by Vlastimil Babka [1], kfree_rcu_sheaf() can be used
on PREEMPT_RT if we always assume spinning is not allowed on PREEMPT_RT.
This is because local_trylock and spinlock_t are safe to use with
trylock and unlock as long as the kernel does not spin and the context
is not NMI and not hardirq.
Now that __kfree_rcu_sheaf() knows how to handle SLAB_FREE_NOLOCK,
relax the limitation and try the sheaves path on PREEMPT_RT as well.
Keep the lockdep map on non RT kernels. However, do not use the lockdep
map on PREEMPT_RT to avoid suppressing valid lockdep warnings.
As pointed by Vlastimil Babka [2], on PREEMPT_RT it is unnecessary to
defer call_rcu() under IRQ-disabled section or raw spinlock. However,
let us avoid adding more complexity as the scenario is not supposed
to be common on PREEMPT_RT, with a hope that call_rcu_nolock() will be
soon supported in RCU.
Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be@kernel.org [2]
Suggested-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
mm/slab_common.c | 12 ++++++++++--
mm/slub.c | 17 ++++++++++-------
2 files changed, 20 insertions(+), 9 deletions(-)
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 9c9ae1384f47..8c631bf97cd5 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1595,6 +1595,14 @@ static bool kfree_rcu_sheaf(void *obj)
{
struct kmem_cache *s;
struct slab *slab;
+ unsigned int free_flags = SLAB_FREE_DEFAULT;
+
+ /*
+ * It is not safe to spin on PREEMPT_RT because the kernel might be
+ * holding a raw spinlock and slab acquires sleeping locks.
+ */
+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
+ free_flags = SLAB_FREE_NOLOCK;
if (is_vmalloc_addr(obj))
return false;
@@ -1605,7 +1613,7 @@ static bool kfree_rcu_sheaf(void *obj)
s = slab->slab_cache;
if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
- return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
+ return __kfree_rcu_sheaf(s, obj, free_flags);
return false;
}
@@ -1954,7 +1962,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
if (!head)
might_sleep();
- if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
+ if (kfree_rcu_sheaf(ptr))
return;
// Queue the object but don't yet schedule the batch.
diff --git a/mm/slub.c b/mm/slub.c
index 8afa6b47b1f2..deac315d0f23 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6065,12 +6065,13 @@ static void rcu_free_sheaf(struct rcu_head *head)
* kvfree_call_rcu() can be called while holding a raw_spinlock_t. Since
* __kfree_rcu_sheaf() may acquire a spinlock_t (sleeping lock on PREEMPT_RT),
* this would violate lock nesting rules. Therefore, kvfree_call_rcu() avoids
- * this problem by bypassing the sheaves layer entirely on PREEMPT_RT.
+ * this problem by passing allow_spin = false on PREEMPT_RT.
*
* However, lockdep still complains that it is invalid to acquire spinlock_t
* while holding raw_spinlock_t, even on !PREEMPT_RT where spinlock_t is a
* spinning lock. Tell lockdep that acquiring spinlock_t is valid here
- * by temporarily raising the wait-type to LD_WAIT_CONFIG.
+ * by temporarily raising the wait-type to LD_WAIT_CONFIG. Skip the lockdep map
+ * on PREEMPT_RT to avoid suppressing valid lockdep warnings.
*/
static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG);
@@ -6080,10 +6081,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
struct slab_sheaf *rcu_sheaf;
bool allow_spin = free_flags_allow_spinning(free_flags);
- if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
- return false;
+ VM_WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT) && allow_spin);
- lock_map_acquire_try(&kfree_rcu_sheaf_map);
+ if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+ lock_map_acquire_try(&kfree_rcu_sheaf_map);
if (!local_trylock(&s->cpu_sheaves->lock))
goto fail;
@@ -6181,12 +6182,14 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
local_unlock(&s->cpu_sheaves->lock);
stat(s, FREE_RCU_SHEAF);
- lock_map_release(&kfree_rcu_sheaf_map);
+ if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+ lock_map_release(&kfree_rcu_sheaf_map);
return true;
fail:
stat(s, FREE_RCU_SHEAF_FAIL);
- lock_map_release(&kfree_rcu_sheaf_map);
+ if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+ lock_map_release(&kfree_rcu_sheaf_map);
return false;
}
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
` (4 preceding siblings ...)
2026-07-20 12:44 ` [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
2026-07-20 13:09 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock() Harry Yoo (Oracle)
2026-07-20 12:44 ` [PATCH slab/for-next v4 8/8] slub_kunit: extend the test for kfree_rcu_nolock() Harry Yoo (Oracle)
7 siblings, 1 reply; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
rcu_head is overkill for kvfree_rcu() because the callback
function is always either kfree(), vfree(), or free_large_kmalloc(),
and thus there is no need for a function pointer.
kvfree_rcu batching reuses the field to store the start address
of an object, however, this is not strictly needed because we can
calculate the start address in the slowpath. For the purpose of
kvfree_rcu batching, it is sufficient to implement a linked list using
a single pointer.
Introduce a new struct called kvfree_rcu_head (the name was suggested
by Vlastimil Babka), which is similar to rcu_head but is only a single
pointer to build a linked list, without a function pointer, when
CONFIG_KVFREE_RCU_BATCHED=y.
When kvfree_rcu is not batched, kvfree_rcu_head is the same size
as rcu_head. Note that shrinking struct kvfree_rcu_head on
CONFIG_KVFREE_RCU_BATCHED=n kernels would inevitably require additional
complexity and also some sort of batching (which defeats the purpose of
the config option) because it cannot fall back to call_rcu().
For now there are no user-visible changes to the API. k[v]free_rcu()
simply casts rcu_head to kvfree_rcu_head. While this does not affect
the API, it allows kfree_rcu_nolock() to reuse kvfree_rcu batching
as a fallback when trylock or sheaf allocation fails.
Stop storing the object pointer in rcu_head.func and instead calculate
the object's start address in kvfree_rcu_list(). Factor out the existing
logic to calculate the start address from kvfree_rcu_cb() to
kvmalloc_obj_start_addr().
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
include/linux/rcupdate.h | 10 ++++++----
include/linux/types.h | 8 ++++++++
include/trace/events/rcu.h | 2 +-
mm/slab.h | 28 ++++++++++++++++++++++++++++
mm/slab_common.c | 17 ++++++++---------
mm/slub.c | 40 +++++++++-------------------------------
6 files changed, 60 insertions(+), 45 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 5e95acc33989..ef5bb6981133 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1098,19 +1098,21 @@ static inline void rcu_read_unlock_migrate(void)
/*
* In mm/slab_common.c, no suitable header to include here.
*/
-void kvfree_call_rcu(struct rcu_head *head, void *ptr);
+void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr);
/*
* The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the
* comment of kfree_rcu() for details.
*/
-#define kvfree_rcu_arg_2(ptr, rhf) \
+#define kvfree_rcu_arg_2(ptr, kvrhf) \
do { \
typeof (ptr) ___p = (ptr); \
+ struct kvfree_rcu_head *___head; \
\
if (___p) { \
- BUILD_BUG_ON(offsetof(typeof(*(ptr)), rhf) >= 4096); \
- kvfree_call_rcu(&((___p)->rhf), (void *) (___p)); \
+ BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >= 4096); \
+ ___head = (struct kvfree_rcu_head *) &(___p)->kvrhf; \
+ kvfree_call_rcu(___head, (void *) (___p)); \
} \
} while (0)
diff --git a/include/linux/types.h b/include/linux/types.h
index 93166b0b0617..8b0976669d66 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -255,6 +255,14 @@ struct callback_head {
} __attribute__((aligned(sizeof(void *))));
#define rcu_head callback_head
+#ifdef CONFIG_KVFREE_RCU_BATCHED
+struct kvfree_rcu_head {
+ struct kvfree_rcu_head *next;
+};
+#else
+#define kvfree_rcu_head rcu_head
+#endif
+
typedef void (*rcu_callback_t)(struct rcu_head *head);
typedef void (*call_rcu_func_t)(struct rcu_head *head, rcu_callback_t func);
diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index 5fbdabe3faea..a74027126a2f 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -625,7 +625,7 @@ TRACE_EVENT_RCU(rcu_invoke_callback,
*/
TRACE_EVENT_RCU(rcu_invoke_kvfree_callback,
- TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset),
+ TP_PROTO(const char *rcuname, struct kvfree_rcu_head *rhp, unsigned long offset),
TP_ARGS(rcuname, rhp, offset),
diff --git a/mm/slab.h b/mm/slab.h
index fdd293b3efa5..1efeaceb7db3 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -351,6 +351,33 @@ static inline int objs_per_slab(const struct kmem_cache *cache,
return slab->objects;
}
+/* kvfree_rcu_head offset can be only less than page size */
+static inline void *kvmalloc_obj_start_addr(void *head)
+{
+ void *obj = head;
+
+ if (unlikely(is_vmalloc_addr(obj))) {
+ obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
+ } else {
+ struct page *page = virt_to_page(obj);
+ struct slab *slab = page_slab(page);
+
+ if (!slab) {
+ obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
+ } else if (is_kfence_address(obj)) {
+ obj = kfence_object_start(obj);
+ } else {
+ struct kmem_cache *s = slab->slab_cache;
+ unsigned int idx = __obj_to_index(s, slab_address(slab), obj);
+
+ obj = slab_address(slab) + s->size * idx;
+ obj = fixup_red_left(s, obj);
+ }
+ }
+
+ return obj;
+}
+
/*
* State of the slab allocator.
*
@@ -761,6 +788,7 @@ void __check_heap_object(const void *ptr, unsigned long n,
const struct slab *slab, bool to_user);
void deferred_work_barrier(void);
+void defer_kfree_rcu(struct kvfree_rcu_head *head);
static inline bool slub_debug_orig_size(struct kmem_cache *s)
{
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 8c631bf97cd5..c1c32909fa52 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1346,7 +1346,7 @@ struct kvfree_rcu_bulk_data {
struct kfree_rcu_cpu_work {
struct rcu_work rcu_work;
- struct rcu_head *head_free;
+ struct kvfree_rcu_head *head_free;
struct rcu_gp_oldstate head_free_gp_snap;
struct list_head bulk_head_free[FREE_N_CHANNELS];
struct kfree_rcu_cpu *krcp;
@@ -1382,7 +1382,7 @@ struct kfree_rcu_cpu_work {
struct kfree_rcu_cpu {
// Objects queued on a linked list
// through their rcu_head structures.
- struct rcu_head *head;
+ struct kvfree_rcu_head *head;
unsigned long head_gp_snap;
atomic_t head_count;
@@ -1523,12 +1523,12 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp,
}
static void
-kvfree_rcu_list(struct rcu_head *head)
+kvfree_rcu_list(struct kvfree_rcu_head *head)
{
- struct rcu_head *next;
+ struct kvfree_rcu_head *next;
for (; head; head = next) {
- void *ptr = (void *) head->func;
+ void *ptr = kvmalloc_obj_start_addr(head);
unsigned long offset = (void *) head - ptr;
next = head->next;
@@ -1552,7 +1552,7 @@ static void kfree_rcu_work(struct work_struct *work)
unsigned long flags;
struct kvfree_rcu_bulk_data *bnode, *n;
struct list_head bulk_head[FREE_N_CHANNELS];
- struct rcu_head *head;
+ struct kvfree_rcu_head *head;
struct kfree_rcu_cpu *krcp;
struct kfree_rcu_cpu_work *krwp;
struct rcu_gp_oldstate head_gp_snap;
@@ -1683,7 +1683,7 @@ kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp)
{
struct list_head bulk_ready[FREE_N_CHANNELS];
struct kvfree_rcu_bulk_data *bnode, *n;
- struct rcu_head *head_ready = NULL;
+ struct kvfree_rcu_head *head_ready = NULL;
unsigned long flags;
int i;
@@ -1946,7 +1946,7 @@ void __init kfree_rcu_scheduler_running(void)
* be free'd in workqueue context. This allows us to: batch requests together to
* reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
*/
-void kvfree_call_rcu(struct rcu_head *head, void *ptr)
+void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr)
{
unsigned long flags;
struct kfree_rcu_cpu *krcp;
@@ -1984,7 +1984,6 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
// Inline if kvfree_rcu(one_arg) call.
goto unlock_return;
- head->func = ptr;
head->next = krcp->head;
WRITE_ONCE(krcp->head, head);
atomic_inc(&krcp->head_count);
diff --git a/mm/slub.c b/mm/slub.c
index deac315d0f23..d0976396d703 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6699,43 +6699,21 @@ static void free_large_kmalloc(struct page *page, void *object)
*/
void kvfree_rcu_cb(struct rcu_head *head)
{
- void *obj = head;
- struct page *page;
- struct slab *slab;
- struct kmem_cache *s;
- void *slab_addr;
+ void *obj;
+
+ obj = kvmalloc_obj_start_addr(head);
if (is_vmalloc_addr(obj)) {
- obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
vfree(obj);
- return;
- }
-
- page = virt_to_page(obj);
- slab = page_slab(page);
- if (!slab) {
- /*
- * rcu_head offset can be only less than page size so no need to
- * consider allocation order
- */
- obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
- free_large_kmalloc(page, obj);
- return;
- }
-
- s = slab->slab_cache;
- slab_addr = slab_address(slab);
-
- if (is_kfence_address(obj)) {
- obj = kfence_object_start(obj);
} else {
- unsigned int idx = __obj_to_index(s, slab_addr, obj);
+ struct page *page = virt_to_page(obj);
+ struct slab *slab = page_slab(page);
- obj = slab_addr + s->size * idx;
- obj = fixup_red_left(s, obj);
+ if (slab)
+ slab_free(slab->slab_cache, slab, obj, _RET_IP_);
+ else
+ free_large_kmalloc(page, obj);
}
-
- slab_free(s, slab, obj, _RET_IP_);
}
/**
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock()
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
` (5 preceding siblings ...)
2026-07-20 12:44 ` [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
2026-07-20 13:07 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 8/8] slub_kunit: extend the test for kfree_rcu_nolock() Harry Yoo (Oracle)
7 siblings, 1 reply; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
Currently, k[v]free_rcu() cannot be called in unknown context since
it could lead to a deadlock when called in the middle of k[v]free_rcu().
Make users' lives easier by introducing kfree_rcu_nolock() variant,
now that kfree_rcu_sheaf() is available on PREEMPT_RT and
__kfree_rcu_sheaf() handles unknown context.
kfree_rcu_nolock() falls back to the kvfree_rcu batching when sheaves
path fails. In most cases, the sheaves path is expected to succeed
and it's unnecessary to add complexity to the existing kvfree_rcu
batching by teaching it how to handle unknown context.
Since defer_kfree_rcu() can be called on caches without sheaves, move
deferred_work_barrier() and rcu_barrier() outside the branch in
kvfree_rcu_barrier_on_cache().
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
include/linux/rcupdate.h | 19 +++++++++++++++++++
mm/slab_common.c | 28 ++++++++++++++++++++++++++--
mm/slub.c | 24 +++++++++++++++++++++++-
3 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index ef5bb6981133..ce0fd401352e 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1099,6 +1099,7 @@ static inline void rcu_read_unlock_migrate(void)
* In mm/slab_common.c, no suitable header to include here.
*/
void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr);
+void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr);
/*
* The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the
@@ -1124,6 +1125,24 @@ do { \
kvfree_call_rcu(NULL, (void *) (___p)); \
} while (0)
+/**
+ * kfree_rcu_nolock() - a version of kfree_rcu() that can be called in any context.
+ * @ptr: pointer to kfree for double-argument invocations.
+ * @kvrhf: the name of the struct kvfree_rcu_head within the type of @ptr.
+ *
+ * Objects that are not allocated via kmalloc_nolock() are not supported.
+ * kfree_rcu_nolock() supports 2-arg variant only.
+ */
+#define kfree_rcu_nolock(ptr, kvrhf) \
+do { \
+ typeof (ptr) ___p = (ptr); \
+ \
+ if (___p) { \
+ BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >= 4096); \
+ kfree_call_rcu_nolock(&((___p)->kvrhf), (void *) (___p)); \
+ } \
+} while (0)
+
/*
* Place this after a lock-acquisition primitive to guarantee that
* an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
diff --git a/mm/slab_common.c b/mm/slab_common.c
index c1c32909fa52..09c457645e03 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1263,6 +1263,29 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
EXPORT_TRACEPOINT_SYMBOL(kfree);
EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free);
+void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr)
+{
+ struct slab *slab;
+ struct kmem_cache *s;
+
+ VM_WARN_ON_ONCE(is_vmalloc_addr(ptr) || !virt_to_slab(ptr));
+
+ slab = virt_to_slab(ptr);
+ s = slab->slab_cache;
+
+ if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != numa_mem_id()))
+ goto fallback;
+
+ if (unlikely(!__kfree_rcu_sheaf(s, ptr, SLAB_FREE_NOLOCK)))
+ goto fallback;
+
+ return;
+
+fallback:
+ defer_kfree_rcu(head);
+}
+EXPORT_SYMBOL_GPL(kfree_call_rcu_nolock);
+
#ifndef CONFIG_KVFREE_RCU_BATCHED
void kvfree_call_rcu(struct rcu_head *head, void *ptr)
@@ -2120,10 +2143,11 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
cpus_read_lock();
flush_rcu_sheaves_on_cache(s);
cpus_read_unlock();
- deferred_work_barrier();
- rcu_barrier();
}
+ /* kfree_rcu_nolock() might have deferred frees even without sheaves */
+ deferred_work_barrier();
+ rcu_barrier();
__kvfree_rcu_barrier();
}
EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
diff --git a/mm/slub.c b/mm/slub.c
index d0976396d703..7114451dd5d0 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4081,6 +4081,7 @@ static void flush_all(struct kmem_cache *s)
struct deferred_percpu_work {
struct llist_head objects;
+ struct llist_head objects_by_rcu;
struct llist_head rcu_sheaves;
struct irq_work work;
};
@@ -4089,6 +4090,7 @@ static void deferred_percpu_work_fn(struct irq_work *work);
static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = {
.objects = LLIST_HEAD_INIT(objects),
+ .objects_by_rcu = LLIST_HEAD_INIT(objects_by_rcu),
.rcu_sheaves = LLIST_HEAD_INIT(rcu_sheaves),
.work = IRQ_WORK_INIT(deferred_percpu_work_fn),
};
@@ -6386,13 +6388,14 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
static void deferred_percpu_work_fn(struct irq_work *work)
{
struct deferred_percpu_work *dpw;
- struct llist_head *objs, *rcu_sheaves;
+ struct llist_head *objs, *objs_by_rcu, *rcu_sheaves;
struct llist_node *llnode, *pos, *t;
struct slab_sheaf *sheaf, *next;
dpw = container_of(work, struct deferred_percpu_work, work);
rcu_sheaves = &dpw->rcu_sheaves;
objs = &dpw->objects;
+ objs_by_rcu = &dpw->objects_by_rcu;
llnode = llist_del_all(objs);
llist_for_each_safe(pos, t, llnode) {
@@ -6417,6 +6420,14 @@ static void deferred_percpu_work_fn(struct irq_work *work)
stat(s, FREE_SLOWPATH);
}
+ llnode = llist_del_all(objs_by_rcu);
+ llist_for_each_safe(pos, t, llnode) {
+ void *head = pos;
+ void *objp = kvmalloc_obj_start_addr(head);
+
+ kvfree_call_rcu(head, objp);
+ }
+
llnode = llist_del_all(rcu_sheaves);
llist_for_each_entry_safe(sheaf, next, llnode, llnode)
call_rcu(&sheaf->rcu_head, rcu_free_sheaf);
@@ -6435,6 +6446,17 @@ static void defer_free(struct kmem_cache *s, void *head)
irq_work_queue(&dpw->work);
}
+void defer_kfree_rcu(struct kvfree_rcu_head *head)
+{
+ struct deferred_percpu_work *dpw;
+
+ guard(preempt)();
+
+ dpw = this_cpu_ptr(&deferred_percpu_work);
+ if (llist_add((struct llist_node *)head, &dpw->objects_by_rcu))
+ irq_work_queue(&dpw->work);
+}
+
void deferred_work_barrier(void)
{
int cpu;
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH slab/for-next v4 8/8] slub_kunit: extend the test for kfree_rcu_nolock()
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
` (6 preceding siblings ...)
2026-07-20 12:44 ` [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock() Harry Yoo (Oracle)
@ 2026-07-20 12:44 ` Harry Yoo (Oracle)
7 siblings, 0 replies; 16+ messages in thread
From: Harry Yoo (Oracle) @ 2026-07-20 12:44 UTC (permalink / raw)
To: Vlastimil Babka, Andrew Morton, Hao Li, Christoph Lameter,
David Rientjes, Roman Gushchin, Alexei Starovoitov,
Andrii Nakryiko, Puranjay Mohan, Amery Hung,
Sebastian Andrzej Siewior, Clark Williams, Steven Rostedt,
Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Mathieu Desnoyers, Lai Jiangshan, Zqiang, Pedro Falcato,
Suren Baghdasaryan, Shengming Hu
Cc: linux-mm, linux-kernel, linux-rt-devel, rcu, bpf,
Harry Yoo (Oracle)
When slub_kunit is not built-in, call kfree_rcu() and kfree_rcu_nolock()
to test kfree_rcu_nolock() in slub_kunit.
Rename the test case as the test covers more _nolock() APIs.
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
lib/tests/slub_kunit.c | 45 +++++++++++++++++++++++++++++----------------
1 file changed, 29 insertions(+), 16 deletions(-)
diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
index a391467c1aa6..e3b63f0338d5 100644
--- a/lib/tests/slub_kunit.c
+++ b/lib/tests/slub_kunit.c
@@ -162,7 +162,10 @@ static void test_kmalloc_redzone_access(struct kunit *test)
}
struct test_kfree_rcu_struct {
- struct rcu_head rcu;
+ union {
+ struct rcu_head rcu;
+ struct kvfree_rcu_head kvrcu;
+ };
};
static void test_kfree_rcu(struct kunit *test)
@@ -296,7 +299,7 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)
#if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
#define NR_ITERATIONS 1000
#define NR_OBJECTS 1000
-static void *objects[NR_OBJECTS];
+static struct test_kfree_rcu_struct *objects[NR_OBJECTS];
struct test_nolock_context {
struct kunit *test;
@@ -311,13 +314,14 @@ struct test_nolock_context {
#endif
};
-static void test_kmalloc_kfree(void)
+static void test_kmalloc_and_friends(void)
{
int i, j;
+ bool can_use_kfree_rcu = !IS_BUILTIN(CONFIG_SLUB_KUNIT_TEST);
for (i = 0; i < NR_ITERATIONS; i++) {
for (j = 0; j < NR_OBJECTS; j++) {
- gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
+ gfp_t gfp = (i & 1) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
objects[j] = kmalloc_obj(*objects[j], gfp);
if (!objects[j]) {
@@ -328,26 +332,35 @@ static void test_kmalloc_kfree(void)
}
}
- for (j = 0; j < NR_OBJECTS; j++)
- kfree(objects[j]);
+ for (j = 0; j < NR_OBJECTS; j++) {
+ if (can_use_kfree_rcu && (i & 2))
+ kfree_rcu(objects[j], rcu);
+ else
+ kfree(objects[j]);
+ }
}
}
static void test_nolock(struct test_nolock_context *ctx)
{
- void *objp;
+ struct test_kfree_rcu_struct *objp;
gfp_t gfp;
+ bool can_use_kfree_rcu = !IS_BUILTIN(CONFIG_SLUB_KUNIT_TEST);
/* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */
- gfp = (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT;
- objp = kmalloc_nolock(64, gfp, NUMA_NO_NODE);
+ gfp = (ctx->callback_count & 1) ? 0 : __GFP_ACCOUNT;
+ objp = kmalloc_nolock(sizeof(*objp), gfp, NUMA_NO_NODE);
if (objp)
ctx->alloc_ok++;
else
ctx->alloc_fail++;
- kfree_nolock(objp);
+ if (can_use_kfree_rcu && (ctx->callback_count & 2))
+ kfree_rcu_nolock(objp, kvrcu);
+ else
+ kfree_nolock(objp);
+
ctx->callback_count++;
}
#endif
@@ -397,14 +410,14 @@ static void disable_perf_events(struct test_nolock_context *ctx)
perf_event_release_kernel(ctx->event);
}
-static void test_kmalloc_kfree_nolock_perf(struct kunit *test)
+static void test_kmalloc_nolock_and_friends_perf(struct kunit *test)
{
struct test_nolock_context ctx = { .test = test };
if (!enable_perf_events(&ctx))
kunit_skip(test, "Failed to enable perf event, skipping");
- test_kmalloc_kfree();
+ test_kmalloc_and_friends();
disable_perf_events(&ctx);
KUNIT_EXPECT_EQ(test, 0, slab_errors);
@@ -438,14 +451,14 @@ static void unregister_slab_kprobes(struct test_nolock_context *ctx)
unregister_kprobe(&ctx->kprobe);
}
-static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
+static void test_kmalloc_nolock_and_friends_kprobe(struct kunit *test)
{
struct test_nolock_context ctx = { .test = test };
if (!register_slab_kprobes(&ctx))
kunit_skip(test, "Failed to register kprobe, skipping");
- test_kmalloc_kfree();
+ test_kmalloc_and_friends();
unregister_slab_kprobes(&ctx);
KUNIT_EXPECT_EQ(test, 0, slab_errors);
@@ -477,10 +490,10 @@ static struct kunit_case test_cases[] = {
KUNIT_CASE(test_leak_destroy),
KUNIT_CASE(test_krealloc_redzone_zeroing),
#ifdef CONFIG_PERF_EVENTS
- KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_perf),
+ KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf),
#endif
#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
- KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_kprobe),
+ KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_kprobe),
#endif
{}
};
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
2026-07-20 12:44 ` [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs Harry Yoo (Oracle)
@ 2026-07-20 12:56 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-07-20 12:56 UTC (permalink / raw)
To: Harry Yoo (Oracle); +Cc: linux-rt-devel, linux-kernel, bpf
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The globally registered kprobe pre-handler executes system-wide on all CPUs, causing non-test tasks to inadvertently execute KUnit test logic and concurrently mutate the test thread's stack-allocated context.
- [Medium] The refactoring to `kmalloc_obj(*objects[j], gfp)` incorrectly evaluates the allocation size to 1 byte instead of the intended 64 bytes.
--
--- Patch [1]: [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs ---
commit 632627083aa5c71fb7be8b45f24099b08c94d6a7
Author: Harry Yoo (Oracle) <harry@kernel.org>
mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
This commit registers a kprobe pre-handler in slub_kunit to invoke and
test kmalloc_nolock() and kfree_nolock() from within the slab allocator.
It adds a wrapper for lockdep_assert_held() to attach the kprobe, and
refactors the test allocation loop into a shared helper function.
Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a92d9a4@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b@kernel.org [2]
> diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
> index fa6d31dbca166..a391467c1aa6e 100644
> --- a/lib/tests/slub_kunit.c
> +++ b/lib/tests/slub_kunit.c
[ ... ]
> @@ -302,26 +303,40 @@ struct test_nolock_context {
> int callback_count;
> int alloc_ok;
> int alloc_fail;
> +#ifdef CONFIG_PERF_EVENTS
> struct perf_event *event;
> +#endif
> +#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
> + struct kprobe kprobe;
> +#endif
> };
>
> -static struct perf_event_attr hw_attr = {
> - .type = PERF_TYPE_HARDWARE,
> - .config = PERF_COUNT_HW_CPU_CYCLES,
> - .size = sizeof(struct perf_event_attr),
> - .pinned = 1,
> - .disabled = 1,
> - .freq = 1,
> - .sample_freq = 100000,
> -};
> +static void test_kmalloc_kfree(void)
> +{
> + int i, j;
> +
> + for (i = 0; i < NR_ITERATIONS; i++) {
> + for (j = 0; j < NR_OBJECTS; j++) {
> + gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
>
> -static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
> - struct perf_sample_data *data,
> - struct pt_regs *regs)
> + objects[j] = kmalloc_obj(*objects[j], gfp);
[Severity: Medium]
Does this refactoring unintentionally reduce the allocation size to 1 byte?
Since objects is declared as an array of void *, the dereferenced type
of *objects[j] is void. The kmalloc_obj() macro evaluates
sizeof(typeof(VAR_OR_TYPE)), which resolves to sizeof(void). Under GCC,
sizeof(void) evaluates to 1.
Does this silently reduce the test coverage by targeting kmalloc-8 instead
of the original 64-byte allocations targeted in the previous code?
> + if (!objects[j]) {
> + j--;
> + while (j >= 0)
> + kfree(objects[j--]);
> + return;
> + }
> + }
> +
[ ... ]
> +#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
> +static int slab_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
> +{
> + struct test_nolock_context *ctx;
> +
> + ctx = container_of(p, struct test_nolock_context, kprobe);
> + test_nolock(ctx);
> + return 0;
> +}
[Severity: High]
Does this kprobe pre-handler execute for all CPUs system-wide?
Since this function is invoked via the kprobe attached to
slab_attach_kprobe_locked(), and there is no check ensuring
current == test_task, it appears any non-test task executing SLUB
slowpaths (like add_partial or remove_full) will also trigger this
handler.
Could this allow unrelated tasks to inadvertently execute test_nolock()
from arbitrary contexts and concurrently mutate the test thread's
stack-allocated ctx, causing a data race on ctx->alloc_ok and
ctx->callback_count?
> +
> +static bool register_slab_kprobes(struct test_nolock_context *ctx)
> +{
> + ctx->kprobe.symbol_name = "slab_attach_kprobe_locked";
> + ctx->kprobe.pre_handler = slab_kprobe_pre_handler;
> +
> + if (register_kprobe(&ctx->kprobe))
> + return false;
> + return true;
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org?part=1
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
2026-07-20 12:44 ` [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT Harry Yoo (Oracle)
@ 2026-07-20 12:56 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-07-20 12:56 UTC (permalink / raw)
To: Harry Yoo (Oracle); +Cc: bpf, linux-rt-devel, linux-kernel
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Calling `local_trylock()` inside `kvfree_call_rcu()` on PREEMPT_RT violates lockdep wait-type nesting rules and is unsafe in hardirq/NMI context.
Pre-existing issues:
- [High] Preexisting bug: Using GFP_NOWAIT in __kfree_rcu_sheaf() can cause lock recursion deadlocks when called under scheduler locks.
--
--- Patch [5]: [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT ---
commit 9ccf08cd866ca28e999f59be7998db3bfc32fb7a
Author: Harry Yoo (Oracle) <harry@kernel.org>
Subject: mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
This patch allows kfree_rcu_sheaf() to be used on PREEMPT_RT by setting
SLAB_FREE_NOLOCK when PREEMPT_RT is enabled. It relaxes the limitation
in kvfree_call_rcu() and skips the lockdep wait-type override map on
PREEMPT_RT to avoid suppressing valid warnings.
Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be@kernel.org [2]
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 9c9ae1384f476..8c631bf97cd5b 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
[ ... ]
> @@ -1954,7 +1962,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> if (!head)
> might_sleep();
>
> - if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
> + if (kfree_rcu_sheaf(ptr))
> return;
>
> // Queue the object but don't yet schedule the batch.
> diff --git a/mm/slub.c b/mm/slub.c
> index 8afa6b47b1f2b..deac315d0f23e 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
[ ... ]
> @@ -6080,10 +6081,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
> struct slab_sheaf *rcu_sheaf;
> bool allow_spin = free_flags_allow_spinning(free_flags);
>
> - if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
> - return false;
> + VM_WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT) && allow_spin);
>
> - lock_map_acquire_try(&kfree_rcu_sheaf_map);
> + if (!IS_ENABLED(CONFIG_PREEMPT_RT))
> + lock_map_acquire_try(&kfree_rcu_sheaf_map);
>
> if (!local_trylock(&s->cpu_sheaves->lock))
^^^^^^^^^^^^^
[Severity: High]
Will this cause lockdep wait-type violations and potential crashes on
PREEMPT_RT?
Since this patch removes the !IS_ENABLED(CONFIG_PREEMPT_RT) guard in
kvfree_call_rcu() inside mm/slab_common.c, kvfree_call_rcu() can now reach
this path on PREEMPT_RT.
Because kvfree_call_rcu() can be called while holding a raw spinlock (like
the scheduler's pi_lock) or in hardirq context, it seems this local_trylock()
will attempt to acquire a sleeping lock (rt_mutex on PREEMPT_RT) in an
invalid context.
Since the lockdep override map is intentionally skipped for PREEMPT_RT just
above, will this correctly trigger lockdep "invalid wait context" splats
and potentially cause panics if the trylock is attempted from hardirq/NMI
context?
[Severity: High]
This is a pre-existing issue, but since we are enabling this path for
PREEMPT_RT, does the use of GFP_NOWAIT later in this function cause
lock recursion deadlocks?
When the fast path fails, __kfree_rcu_sheaf() calls:
empty = alloc_empty_sheaf(s, GFP_NOWAIT, alloc_flags);
GFP_NOWAIT includes __GFP_KSWAPD_RECLAIM, which invokes wakeup_kswapd().
The wake-up path enters the scheduler via try_to_wake_up() and attempts to
acquire scheduler locks like pi_lock and the runqueue lock.
If kvfree_call_rcu() was called while the CPU already holds these scheduler
locks, will this wake-up attempt cause a lock recursion self-deadlock?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org?part=5
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled
2026-07-20 12:44 ` [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled Harry Yoo (Oracle)
@ 2026-07-20 13:01 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-07-20 13:01 UTC (permalink / raw)
To: Harry Yoo (Oracle); +Cc: linux-rt-devel, bpf, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Acquiring local_trylock in __kfree_rcu_sheaf() causes an invalid wait context lockdep warning when kvfree_call_rcu() is called under a raw spinlock.
--
--- Patch [3]: [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled ---
commit 48ff7cace4bd8ff5a54e0dc53dea46c9c152b1ad
Author: Harry Yoo (Oracle) <harry@kernel.org>
mm/slab: use call_rcu() in unknown context if irqs are enabled
This patch optimizes the deferred call_rcu() path in __kfree_rcu_sheaf() by
checking if IRQs are disabled. If IRQs are enabled, it avoids falling back
to the deferred path since call_rcu() cannot be interrupted by itself in
this context.
Link: https://lore.kernel.org/linux-mm/CAADnVQKRVD5ZSnEKbZZU7w86gHbGHUug2pvzpgZTngNS+fg4rw@mail.gmail.com
> diff --git a/mm/slub.c b/mm/slub.c
> index 0c350274fbff4..680fe64aff72e 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -6136,8 +6136,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
> if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
> rcu_sheaf = NULL;
> } else {
> - if (unlikely(!allow_spin)) {
> - /* call_rcu() cannot be called in an unknown context */
> + /*
> + * With !allow_spin, we might have interrupted call_rcu()'s
> + * IRQ-disabled critical section. If IRQs are not disabled,
> + * we know that's not the case.
> + */
> + if (unlikely(!allow_spin && irqs_disabled())) {
> rcu_sheaf->size--;
> local_unlock(&s->cpu_sheaves->lock);
[Severity: High]
This is a pre-existing issue, but does using local_unlock() here (and
local_trylock() earlier in __kfree_rcu_sheaf()) violate wait context rules?
The kvfree_call_rcu() path can be called with a raw spinlock held, such as
pi_lock.
Because local_trylock is treated as a sleeping lock under PREEMPT_RT
wait-type checking, acquiring it while holding a raw spinlock triggers a
lockdep "Invalid wait context" warning.
This happens because CONFIG_PROVE_RAW_LOCK_NESTING statically checks
declared wait-types rather than runtime behavior.
Is there a way to defer this freeing without taking local_trylock?
> goto fail;
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org?part=3
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves
2026-07-20 12:44 ` [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves Harry Yoo (Oracle)
@ 2026-07-20 13:03 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-07-20 13:03 UTC (permalink / raw)
To: Harry Yoo (Oracle); +Cc: linux-rt-devel, bpf, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] `__kfree_rcu_sheaf()` acquires a lock using `local_trylock()`, which triggers a lockdep 'Invalid wait context' warning when called via `kvfree_call_rcu()`.
--
--- Patch [4]: [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves ---
commit 136141e9d7cd5370bbb83aa5e26b9b28ea8267ad
Author: Harry Yoo (Oracle) <harry@kernel.org>
mm/slab: extend deferred free mechanism to handle rcu sheaves
This commit extends the deferred free mechanism to handle RCU sheaves when
spinning is not allowed and IRQs are disabled. It queues the RCU sheaf on
an IRQ work to safely invoke call_rcu() later, and updates barrier syncs.
> diff --git a/mm/slub.c b/mm/slub.c
> index 680fe64aff72e..8afa6b47b1f2b 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
[ ... ]
> @@ -6154,8 +6161,22 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
> * we flush before local_unlock to make sure a racing
> * flush_all_rcu_sheaves() doesn't miss this sheaf
> */
> - if (rcu_sheaf)
> - call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
> + if (rcu_sheaf) {
> + /*
> + * With !allow_spin, we might have interrupted call_rcu()'s
> + * IRQ-disabled critical section. If IRQs are not disabled,
> + * we know that's not the case.
> + */
> + if (unlikely(!allow_spin && irqs_disabled())) {
> + struct deferred_percpu_work *dpw;
> +
> + dpw = this_cpu_ptr(&deferred_percpu_work);
> + if (llist_add(&rcu_sheaf->llnode, &dpw->rcu_sheaves))
> + irq_work_queue(&dpw->work);
> + } else {
> + call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
> + }
> + }
[Severity: High]
This isn't a bug introduced by this patch, but since this modifies
__kfree_rcu_sheaf(), I wanted to ask about a preexisting locking issue in
this execution path.
Earlier in __kfree_rcu_sheaf(), the code unconditionally acquires a lock via
local_trylock(&s->cpu_sheaves->lock). However, this function is reachable from
kvfree_call_rcu(), which can be called under a raw_spinlock_t (like pi_lock)
and from hardirq context:
mm/slab_common.c:kvfree_call_rcu() {
...
if (kfree_rcu_sheaf(ptr))
return;
...
}
which routes to:
mm/slub.c:__kfree_rcu_sheaf() {
...
if (!local_trylock(&s->cpu_sheaves->lock))
goto fail;
...
}
Will acquiring a local_trylock within this wait context trigger a lockdep
"Invalid wait context" warning, since it statically violates the raw
spinlock locking rules?
>
> local_unlock(&s->cpu_sheaves->lock);
>
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org?part=4
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock()
2026-07-20 12:44 ` [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock() Harry Yoo (Oracle)
@ 2026-07-20 13:07 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-07-20 13:07 UTC (permalink / raw)
To: Harry Yoo (Oracle); +Cc: linux-kernel, bpf, linux-rt-devel
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] kfree_call_rcu_nolock() crashes with a NULL pointer dereference when given a ZERO_SIZE_PTR or an unsupported vmalloc/large kmalloc pointer.
- [High] Memory leak and cache destruction panic because deferred_work_barrier() is called after flush_rcu_sheaves_on_cache().
- [High] Calling kfree_rcu_nolock() on PREEMPT_RT triggers an 'Invalid wait context' lockdep warning from local_trylock().
--
--- Patch [7]: [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock() ---
commit 278abd26e1c5865b59d3e42ddc961dd9dca8dc17
Author: Harry Yoo (Oracle) <harry@kernel.org>
mm/slab: introduce kfree_rcu_nolock()
This patch introduces the kfree_rcu_nolock() variant to handle
k[v]free_rcu() calls in unknown context. It uses kfree_rcu_sheaf() when
possible and falls back to kvfree_rcu() batching. It also adjusts barrier
placement in kvfree_rcu_barrier_on_cache().
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index c1c32909fa528..09c457645e039 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
[ ... ]
> @@ -1263,6 +1263,29 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
> EXPORT_TRACEPOINT_SYMBOL(kfree);
> EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free);
>
> +void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr)
> +{
> + struct slab *slab;
> + struct kmem_cache *s;
> +
> + VM_WARN_ON_ONCE(is_vmalloc_addr(ptr) || !virt_to_slab(ptr));
> +
> + slab = virt_to_slab(ptr);
> + s = slab->slab_cache;
[Severity: High]
Does this code properly handle unsupported pointers? For example, if a caller
passes a 0-byte allocation (ZERO_SIZE_PTR) or a large kmalloc object,
virt_to_slab() returns NULL.
Since the kfree_rcu_nolock() macro only checks if the pointer evaluates to
true (and ZERO_SIZE_PTR is truthy), this would lead to a NULL pointer
dereference when accessing slab->slab_cache. Should this return early if
virt_to_slab() returns NULL instead of just warning?
> +
> + if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != numa_mem_id()))
> + goto fallback;
> +
> + if (unlikely(!__kfree_rcu_sheaf(s, ptr, SLAB_FREE_NOLOCK)))
[Severity: High]
Could this trigger an 'Invalid wait context' lockdep warning on PREEMPT_RT?
The kfree_rcu_nolock() variant is intended to be safe for any context,
including under raw spinlocks or in hardirq context. However,
__kfree_rcu_sheaf() acquires s->cpu_sheaves->lock via local_trylock():
mm/slub.c:__kfree_rcu_sheaf() {
...
if (!local_trylock(&s->cpu_sheaves->lock))
goto fail;
...
}
On PREEMPT_RT, local_trylock() maps to a sleepable spin_trylock(). Taking
this lock from a raw spinlock or hardirq context would violate wait-type
nesting rules.
> + goto fallback;
> +
> + return;
> +
> +fallback:
> + defer_kfree_rcu(head);
> +}
> +EXPORT_SYMBOL_GPL(kfree_call_rcu_nolock);
> +
[ ... ]
> @@ -2120,10 +2143,11 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
> cpus_read_lock();
> flush_rcu_sheaves_on_cache(s);
> cpus_read_unlock();
> - deferred_work_barrier();
> - rcu_barrier();
> }
>
> + /* kfree_rcu_nolock() might have deferred frees even without sheaves */
> + deferred_work_barrier();
> + rcu_barrier();
[Severity: High]
Can this sequence strand objects in the sheaves and cause a cache
destruction panic?
Since deferred_work_barrier() is now called after
flush_rcu_sheaves_on_cache() completes, the deferred work for objects in
objs_by_rcu will call kvfree_call_rcu().
If the cache has sheaves, kvfree_call_rcu() will succeed at placing those
objects back into the newly emptied sheaves:
mm/slub.c:deferred_percpu_work_fn() {
...
llnode = llist_del_all(objs_by_rcu);
llist_for_each_safe(pos, t, llnode) {
...
kvfree_call_rcu(head, objp);
}
...
}
Since there is no further sheaf flush after this repopulation, could these
objects remain stranded and cause kmem_cache_destroy() to fail with a
'Slab cache still has objects' panic?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org?part=7
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
2026-07-20 12:44 ` [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching Harry Yoo (Oracle)
@ 2026-07-20 13:09 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-07-20 13:09 UTC (permalink / raw)
To: Harry Yoo (Oracle); +Cc: linux-kernel, linux-rt-devel, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] KASAN tag loss in kvmalloc_obj_start_addr() causes debugobjects memory leak, corrupted tracepoint offsets, and KASAN double-free bypass.
--
--- Patch [6]: [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching ---
commit b85878f9ac539d3e206f29f7df390cb30b50c271
Author: Harry Yoo (Oracle) <harry@kernel.org>
Subject: mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
This patch introduces `kvfree_rcu_head` to replace `rcu_head` for
kvfree_rcu batching. It stops storing the object pointer directly and
instead calculates the object's start address dynamically via
`kvmalloc_obj_start_addr()` to reduce structure size overhead.
> diff --git a/mm/slab.h b/mm/slab.h
> index fdd293b3efa56..1efeaceb7db32 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -351,6 +351,33 @@ static inline int objs_per_slab(const struct kmem_cache *cache,
[ ... ]
> +/* kvfree_rcu_head offset can be only less than page size */
> +static inline void *kvmalloc_obj_start_addr(void *head)
> +{
> + void *obj = head;
> +
> + if (unlikely(is_vmalloc_addr(obj))) {
> + obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
> + } else {
> + struct page *page = virt_to_page(obj);
> + struct slab *slab = page_slab(page);
> +
> + if (!slab) {
> + obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
> + } else if (is_kfence_address(obj)) {
> + obj = kfence_object_start(obj);
> + } else {
> + struct kmem_cache *s = slab->slab_cache;
> + unsigned int idx = __obj_to_index(s, slab_address(slab), obj);
> +
> + obj = slab_address(slab) + s->size * idx;
[Severity: High]
Does reconstructing the base address using slab_address(slab) drop the KASAN
tag from the original allocation?
slab_address() returns a tagless linear map address, so the resulting obj
pointer will be untagged.
> + obj = fixup_red_left(s, obj);
> + }
> + }
> +
> + return obj;
> +}
> +
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 8c631bf97cd5b..c1c32909fa528 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
[ ... ]
> static void
> -kvfree_rcu_list(struct rcu_head *head)
> +kvfree_rcu_list(struct kvfree_rcu_head *head)
> {
> - struct rcu_head *next;
> + struct kvfree_rcu_head *next;
>
> for (; head; head = next) {
> - void *ptr = (void *) head->func;
> + void *ptr = kvmalloc_obj_start_addr(head);
> unsigned long offset = (void *) head - ptr;
[Severity: High]
If ptr is untagged but head still has its KASAN tag, could this cause the
offset calculation to yield a massive value due to the tag difference in the
upper bits?
Additionally, does passing the untagged ptr to debug_rcu_head_unqueue()
(called slightly later in this function) cause an exact-match hash failure
and permanently leak the tracking object?
>
> next = head->next;
> diff --git a/mm/slub.c b/mm/slub.c
> index deac315d0f23e..d0976396d703d 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -6699,43 +6699,21 @@ static void free_large_kmalloc(struct page *page, void *object)
> */
> void kvfree_rcu_cb(struct rcu_head *head)
> {
> - void *obj = head;
[ ... ]
> + obj = kvmalloc_obj_start_addr(head);
>
> if (is_vmalloc_addr(obj)) {
> vfree(obj);
> } else {
> - unsigned int idx = __obj_to_index(s, slab_addr, obj);
> + struct page *page = virt_to_page(obj);
> + struct slab *slab = page_slab(page);
>
> - obj = slab_addr + s->size * idx;
> - obj = fixup_red_left(s, obj);
> + if (slab)
> + slab_free(slab->slab_cache, slab, obj, _RET_IP_);
[Severity: High]
Does passing the untagged obj pointer into slab_free() silently bypass
KASAN quarantine and double-free/use-after-free validation checks that
rely on the tag?
> + else
> + free_large_kmalloc(page, obj);
> }
> }
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org?part=6
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
2026-07-20 12:44 ` [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() Harry Yoo (Oracle)
@ 2026-07-20 13:11 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-07-20 13:11 UTC (permalink / raw)
To: Harry Yoo (Oracle); +Cc: linux-kernel, bpf, linux-rt-devel
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] kfree_rcu_sheaf() hardcodes SLAB_FREE_DEFAULT, rendering the new !allow_spin fallback logic unreachable and leaving lock recursion bugs unfixed.
Pre-existing issues:
- [High] Calling kfree_nolock() can trigger RT-mutex deadlocks when executed in NMI context on PREEMPT_RT kernels.
--
--- Patch [2]: [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() ---
commit 412874c34b3d9cc0ed877c03632bf3c3cd9499d6
Author: Harry Yoo (Oracle) <harry@kernel.org>
mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
This commit teaches kfree_rcu_sheaf() how to handle the allow_spin false
case by trying to get an empty sheaf from pcs->spare or the barn even when
spinning is not allowed. It introduces free_flags to describe the context
and converts them to alloc_flags when allocating memory in the free path.
Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5@kernel.org
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index b6426d7ceec92..e07b4e6d6679c 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -1605,7 +1605,7 @@ static bool kfree_rcu_sheaf(void *obj)
>
> s = slab->slab_cache;
> if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
> - return __kfree_rcu_sheaf(s, obj);
> + return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
[Severity: High]
Does this hardcoded SLAB_FREE_DEFAULT prevent the new allow_spin fallback
logic from ever being executed?
Since kfree_rcu_sheaf() is the sole caller of __kfree_rcu_sheaf(), passing
SLAB_FREE_DEFAULT unconditionally appears to make the new context-handling
logic unreachable.
>
> return false;
> }
[ ... ]
> diff --git a/mm/slub.c b/mm/slub.c
> index e32a686775374..0c350274fbff4 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2829,11 +2833,20 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
> mark_obj_codetag_empty(sheaf);
>
> VM_WARN_ON_ONCE(sheaf->size > 0);
> - kfree(sheaf);
> +
> + if (likely(allow_spin))
> + kfree(sheaf);
> + else
> + kfree_nolock(sheaf);
[Severity: High]
This is a pre-existing issue, but can calling kfree_nolock() here trigger
an RT-mutex deadlock when executed in NMI context on PREEMPT_RT kernels?
Looking at kfree_nolock() in mm/slub.c, it calls free_to_pcs() with the
following sequence:
if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, x, false)))
return;
And free_to_pcs() acquires the lock via local_trylock():
if (!local_trylock(&s->cpu_sheaves->lock))
return false;
On PREEMPT_RT, local_trylock() maps to an RT-mutex, which can deadlock if
the NMI interrupted a critical section holding the internal raw spinlock on
the same CPU.
>
> stat(s, SHEAF_FREE);
> }
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org?part=2
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-07-20 13:11 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-20 12:44 [PATCH slab/for-next v4 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage Harry Yoo (Oracle)
2026-07-20 12:44 ` [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs Harry Yoo (Oracle)
2026-07-20 12:56 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() Harry Yoo (Oracle)
2026-07-20 13:11 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled Harry Yoo (Oracle)
2026-07-20 13:01 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves Harry Yoo (Oracle)
2026-07-20 13:03 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT Harry Yoo (Oracle)
2026-07-20 12:56 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching Harry Yoo (Oracle)
2026-07-20 13:09 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock() Harry Yoo (Oracle)
2026-07-20 13:07 ` sashiko-bot
2026-07-20 12:44 ` [PATCH slab/for-next v4 8/8] slub_kunit: extend the test for kfree_rcu_nolock() Harry Yoo (Oracle)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox