[PATCH v5 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage

[PATCH v5 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage

[Harry Yoo (Oracle)]

This series is based on slab/for-next (ffc64805ceba), and is also
available on git.kernel.org [4].

Changes since v4
=================
- Rebased onto slab/for-next (that now includes Vlastimil's
  objext_split work)
- Patch 1: Added Shengming's R-b (thanks!) and fixed nit pointed out
           by sashiko (kmalloc_obj() for void *):

- Patch 2: 
  - do not silently drop __GFP_RECLAIM in alloc_empty_sheaf (Vlastimil)
  - use free_flags instead of allow_spin in __free_empty_sheaf()
    (Vlastimil)
  - Added Vlastimil's R-b, thanks!
- Patch 5:
  - Added Vlastimil's R-b, thanks!
  - Fix comment that mentioned allow_spin instead of SLAB_FREE_NOLOCK
- Patch 6:
  - Keep KASAN tag when calculating the start address of the object
    (Sashiko).
  - Did not add Vlastimil's R-b due to the change above
- Patch 7:
  - Let large kmalloc/vmalloc objects always fall back and adjust
    the kfree_rcu_nolock() kerneldoc comment accordingly
    (Vlastimil)
  - Call deferred_work_barrier() before flush_rcu_sheaves_on_cache()
    for correctness (Sashiko) 
  - Do not use the rcu sheaf path w/o KVFREE_RCU_BATCHED for
    consistency. Not doing this complicates barriers.
  - Did not add Vlastimil's R-b due to changes above
- Patch 8:
  - Convert to kmalloc_obj() in patch 8 instead of patch 1
    (where the type was changed from void * to
     struct test_kfree_rcu_struct *)

Git range-diff is copied-and-pasted at the end of the email.

Actual Cover Letter
===================

This series improves kmalloc_nolock() and kfree_nolock() coverage
in slub_kunit (patch 1) and introduces kfree_rcu_nolock() for
unknown context as suggested by Alexei Starovoitov.

Unknown context means the caller does not know whether spinning on a lock
is safe (e.g., a BPF program attached to an arbitrary kernel function or
in NMI context).

The slab allocator already supports unknown context via kmalloc_nolock()
and kfree_nolock(), but te slab allocator does not support freeing
objects by RCU in unknown context.

It is not ideal to have completely separate batching for unknown context
because the worst scenario where spinning on a lock would lead to
deadlock is very rare, and in most cases, it is safe to use the
existing mechanism (kfree_rcu_sheaf()).

Since most part of the slab allocator already supports unknown context
and sheaves support batching kvfree_rcu() calls for slab objects,
implement kfree_rcu_nolock() with minimal changes by teaching
kfree_rcu_sheaf() how to support unknown context and making
it a little bit harder to allocate an empty sheaf, instead of making
intrusive changes to the existing kvfree_rcu batching logic.

kfree_rcu_nolock() tries to free the object to the rcu sheaf if
trylock succeeds. Once the rcu sheaf becomes full, it is submitted to
RCU via call_rcu() if spinning is allowed or IRQs are enabled (to avoid
calling call_rcu() in the middle of call_rcu()). Otherwise, call_rcu()
is deferred via irq work.

When there is no sheaf available, kfree_rcu_sheaf() falls back to
defer_kfree_rcu(). It submits the object to kvfree_rcu batching
via irq work. To do this, patch 6 converts kvfree_rcu to use
kvfree_rcu_head without visible changes to the API for now.

Unlike kfree_rcu(), only the 2-argument variant is supported.
This is because the last resort of the 1-arg variant is
synchronize_rcu(), which cannot be used in an unknown context.

As suggested by Alexei Starovoitov, kfree_rcu_nolock() can be used with
struct kvfree_rcu_head (8 bytes), which is smaller than struct rcu_head
(16 bytes).

For more background and future plans, please see [5].

[1] RFC v1: https://lore.kernel.org/linux-mm/20260206093410.160622-1-harry.yoo@oracle.com

[2] RFC v2: https://lore.kernel.org/linux-mm/20260416091022.36823-1-harry@kernel.org

[3] v3: https://lore.kernel.org/linux-mm/20260615-kfree_rcu_nolock-v3-0-70a54f3775bb@kernel.org
 
[4] https://git.kernel.org/pub/scm/linux/kernel/git/harry/linux.git/log/?h=kfree_rcu_nolock-v5r1

[5] kmalloc_nolock() follow-ups, including kfree_rcu_nolock(),
    https://lore.kernel.org/linux-mm/esepccfhqg7m6jo76ns2znj2cnuaepx2xvw5zaygtwohq4psma@563ypprp6rr3

[6] v4: https://lore.kernel.org/r/20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org

To: Vlastimil Babka <vbabka@kernel.org>
To: Andrew Morton <akpm@linux-foundation.org>
To: Hao Li <hao.li@linux.dev>
To: Christoph Lameter <cl@gentwo.org>
To: David Rientjes <rientjes@google.com>
To: Roman Gushchin <roman.gushchin@linux.dev>
To: Alexei Starovoitov <ast@kernel.org>
To: Andrii Nakryiko <andrii@kernel.org>
To: Puranjay Mohan <puranjay@kernel.org>
To: Amery Hung <ameryhung@gmail.com>
To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
To: Clark Williams <clrkwllms@kernel.org>
To: Steven Rostedt <rostedt@goodmis.org>
To: Paul E. McKenney <paulmck@kernel.org>
To: Frederic Weisbecker <frederic@kernel.org>
To: Neeraj Upadhyay <neeraj.upadhyay@kernel.org>
To: Joel Fernandes <joelagnelf@nvidia.com>
To: Josh Triplett <josh@joshtriplett.org>
To: Boqun Feng <boqun@kernel.org>
To: Uladzislau Rezki <urezki@gmail.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: Lai Jiangshan <jiangshanlai@gmail.com>
To: Zqiang <qiang.zhang@linux.dev>
To: Pedro Falcato <pfalcato@suse.de>
To: Suren Baghdasaryan <surenb@google.com>
To: Shengming Hu <hu.shengming@zte.com.cn>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-rt-devel@lists.linux.dev
Cc: rcu@vger.kernel.org
Cc: bpf@vger.kernel.org
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>

$ b4 prep --compare-to v4
1:  53f9594e54e6 ! 1:  89762887c7fb mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
    @@ Commit message
    +    Reviewed-by: Shengming Hu <hu.shengming@zte.com.cn>
     
      ## lib/tests/slub_kunit.c ##
    @@ lib/tests/slub_kunit.c: struct test_nolock_context {
     +	for (i = 0; i < NR_ITERATIONS; i++) {
     +		for (j = 0; j < NR_OBJECTS; j++) {
     +			gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
    - 
    --static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
    --						       struct perf_sample_data *data,
    --						       struct pt_regs *regs)
    -+			objects[j] = kmalloc_obj(*objects[j], gfp);
    ++
    ++			objects[j] = kmalloc(64, gfp);
     +			if (!objects[j]) {
     +				j--;
     +				while (j >= 0)
    @@ lib/tests/slub_kunit.c: struct test_nolock_context {
     +			kfree(objects[j]);
     +	}
     +}
    -+
    + 
    +-static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
    +-						       struct perf_sample_data *data,
    +-						       struct pt_regs *regs)
     +static void test_nolock(struct test_nolock_context *ctx)
      {
      	void *objp;
    @@ lib/tests/slub_kunit.c: static void overflow_handler_test_kmalloc_kfree_nolock(s
      	ctx->callback_count++;
      }
     +#endif
    - 
    --static void test_kmalloc_kfree_nolock(struct kunit *test)
    ++
     +#ifdef CONFIG_PERF_EVENTS
     +static struct perf_event_attr hw_attr = {
     +	.type = PERF_TYPE_HARDWARE,
    @@ lib/tests/slub_kunit.c: static void overflow_handler_test_kmalloc_kfree_nolock(s
     +	.freq = 1,
     +	.sample_freq = 100000,
     +};
    -+
    + 
    +-static void test_kmalloc_kfree_nolock(struct kunit *test)
     +static void overflow_handler_test_nolock(struct perf_event *event,
     +					 struct perf_sample_data *data,
     +					 struct pt_regs *regs)
    @@ lib/tests/slub_kunit.c: static void overflow_handler_test_kmalloc_kfree_nolock(s
     +	perf_event_enable(ctx->event);
     +	return true;
     +}
    - 
    --cleanup:
    --	perf_event_disable(ctx.event);
    --	perf_event_release_kernel(ctx.event);
    ++
     +static void disable_perf_events(struct test_nolock_context *ctx)
     +{
     +	kunit_info(ctx->test, "HW perf events: callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
    @@ lib/tests/slub_kunit.c: static void overflow_handler_test_kmalloc_kfree_nolock(s
     +static void test_kmalloc_kfree_nolock_perf(struct kunit *test)
     +{
     +	struct test_nolock_context ctx = { .test = test };
    - 
    --	kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
    --		   ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail);
    ++
     +	if (!enable_perf_events(&ctx))
     +		kunit_skip(test, "Failed to enable perf event, skipping");
    - 
    --	if (alloc_fail)
    --		kunit_skip(test, "Allocation failed");
    ++
     +	test_kmalloc_kfree();
     +
     +	disable_perf_events(&ctx);
    @@ lib/tests/slub_kunit.c: static void overflow_handler_test_kmalloc_kfree_nolock(s
     +static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
     +{
     +	struct test_nolock_context ctx = { .test = test };
    -+
    + 
    +-cleanup:
    +-	perf_event_disable(ctx.event);
    +-	perf_event_release_kernel(ctx.event);
     +	if (!register_slab_kprobes(&ctx))
     +		kunit_skip(test, "Failed to register kprobe, skipping");
    -+
    + 
    +-	kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
    +-		   ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail);
     +	test_kmalloc_kfree();
    -+
    + 
    +-	if (alloc_fail)
    +-		kunit_skip(test, "Allocation failed");
     +	unregister_slab_kprobes(&ctx);
      	KUNIT_EXPECT_EQ(test, 0, slab_errors);
      }
    @@ lib/tests/slub_kunit.c: static struct kunit_case test_cases[] = {
      };
     
      ## mm/slub.c ##
    -@@ mm/slub.c: static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
    +@@ mm/slub.c: static inline void slab_set_obj_exts_in_object(struct slab *slab)
      }
      #endif
      
2:  a1e7b0265e51 ! 2:  025b3ba728a5 mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
    @@ Commit message
     
         Now that slab has internal alloc_flags to describe context, introduce
         free_flags analogously and convert free_flags to alloc_flags when
    -    allocating memory in the free path. alloc_empty_sheaf() now strips
    -    __GFP_RECLAIM when SLAB_ALLOC_NOLOCK is specified.
    +    allocating memory in the free path.
     
         When trylock fails or the kernel observes non-NULL pcs->rcu_free after
         lock acquisition, free the sheaf instead of putting it to the barn.
    @@ Commit message
         kfree_rcu_sheaf() fails when the rcu sheaf becomes full.
     
    +    Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
     
      ## mm/slab.h ##
     @@
    - #define SLAB_ALLOC_NEW_SLAB	0x02 /* a flag for alloc_slab_obj_exts() */
      #define SLAB_ALLOC_NO_RECURSE	0x04 /* prevent kmalloc() recursion */
    + #define SLAB_ALLOC_NO_OBJ_EXT	0x08 /* prevent obj_exts array allocation */
      
     +#define SLAB_FREE_DEFAULT	0x00 /* no flags */
     +#define SLAB_FREE_NOLOCK	0x01 /* spinning not allowed */
    @@ mm/slab.h
      				  unsigned int alloc_flags, int node)
      				  __assume_kmalloc_alignment __alloc_size(1);
     @@ mm/slab.h: static inline bool is_kmalloc_normal(struct kmem_cache *s)
    - 	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
    + 	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT|SLAB_NO_OBJ_EXT));
      }
      
     -bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
    @@ mm/slab_common.c: static bool kfree_rcu_sheaf(void *obj)
     
      ## mm/slub.c ##
     @@ mm/slub.c: static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s,
    - 
    - 	gfp &= ~OBJCGS_CLEAR_MASK;
    - 
    -+	if (alloc_flags & SLAB_ALLOC_NOLOCK)
    -+		gfp &= ~__GFP_RECLAIM;
    -+
      	return __alloc_empty_sheaf(s, gfp, alloc_flags, s->sheaf_capacity);
      }
      
     -static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
     +static void __free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
    -+			       bool allow_spin)
    ++			       unsigned int free_flags)
      {
      	/*
      	 * If the sheaf was created with SLAB_ALLOC_NO_RECURSE flag then its
    @@ mm/slub.c: static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf
      	VM_WARN_ON_ONCE(sheaf->size > 0);
     -	kfree(sheaf);
     +
    -+	if (likely(allow_spin))
    -+		kfree(sheaf);
    -+	else
    ++	if (unlikely(free_flags & SLAB_FREE_NOLOCK))
     +		kfree_nolock(sheaf);
    ++	else
    ++		kfree(sheaf);
      
      	stat(s, SHEAF_FREE);
      }
      
     +static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
     +{
    -+	__free_empty_sheaf(s, sheaf, /* allow_spin = */ true);
    ++	__free_empty_sheaf(s, sheaf, SLAB_FREE_DEFAULT);
     +}
     +
      static unsigned int
    @@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
      		struct slab_sheaf *empty;
      		struct node_barn *barn;
     +		unsigned int alloc_flags = to_alloc_flags(free_flags);
    ++		gfp_t gfp = allow_spin ? GFP_NOWAIT : __GFP_NOWARN;
      
      		/* Bootstrap or debug cache, fall back */
      		if (unlikely(!cache_has_sheaves(s))) {
    @@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
      		local_unlock(&s->cpu_sheaves->lock);
      
     -		empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
    -+		empty = alloc_empty_sheaf(s, GFP_NOWAIT, alloc_flags);
    ++		empty = alloc_empty_sheaf(s, gfp, alloc_flags);
      
      		if (!empty)
      			goto fail;
      
      		if (!local_trylock(&s->cpu_sheaves->lock)) {
     -			barn_put_empty_sheaf(barn, empty);
    -+			__free_empty_sheaf(s, empty, allow_spin);
    ++			__free_empty_sheaf(s, empty, free_flags);
      			goto fail;
      		}
      
    @@ mm/slub.c: bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
      
      		if (unlikely(pcs->rcu_free))
     -			barn_put_empty_sheaf(barn, empty);
    -+			__free_empty_sheaf(s, empty, allow_spin);
    ++			__free_empty_sheaf(s, empty, free_flags);
      		else
      			pcs->rcu_free = empty;
      	}
3:  f63010ae4b65 = 3:  54211dde1020 mm/slab: use call_rcu() in unknown context if irqs are enabled
4:  c7f5ae27c323 = 4:  cb93a68b5183 mm/slab: extend deferred free mechanism to handle rcu sheaves
5:  5e3eceee103b ! 5:  6190f2484d9b mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
    @@ Commit message
    +    Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
     
      ## mm/slab_common.c ##
    @@ mm/slub.c: static void rcu_free_sheaf(struct rcu_head *head)
       * __kfree_rcu_sheaf() may acquire a spinlock_t (sleeping lock on PREEMPT_RT),
       * this would violate lock nesting rules. Therefore, kvfree_call_rcu() avoids
     - * this problem by bypassing the sheaves layer entirely on PREEMPT_RT.
    -+ * this problem by passing allow_spin = false on PREEMPT_RT.
    ++ * this problem by passing SLAB_FREE_NOLOCK on PREEMPT_RT.
       *
       * However, lockdep still complains that it is invalid to acquire spinlock_t
       * while holding raw_spinlock_t, even on !PREEMPT_RT where spinlock_t is a
6:  a15d4da6efbc ! 6:  2e7eefe35a29 mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
    @@ Commit message
         Stop storing the object pointer in rcu_head.func and instead calculate
         the object's start address in kvfree_rcu_list(). Factor out the existing
         logic to calculate the start address from kvfree_rcu_cb() to
    -    kvmalloc_obj_start_addr().
    +    kvmalloc_obj_start_addr(). To avoid losing the KASAN tag, calculate
    +    the offset and subtract it from the address of the kvfree_rcu_head.
     
     
    @@ include/trace/events/rcu.h: TRACE_EVENT_RCU(rcu_invoke_callback,
      
     
      ## mm/slab.h ##
    -@@ mm/slab.h: static inline int objs_per_slab(const struct kmem_cache *cache,
    - 	return slab->objects;
    +@@ mm/slab.h: static inline unsigned int obj_to_index(const struct kmem_cache *cache,
    + 	return __obj_to_index(cache, slab_address(slab), obj);
      }
      
    -+/* kvfree_rcu_head offset can be only less than page size */
    ++/*
    ++ * kvfree_rcu_head offset can be only less than page size.
    ++ * Calculate the start address while preserving the KASAN tag.
    ++ */
     +static inline void *kvmalloc_obj_start_addr(void *head)
     +{
    -+	void *obj = head;
    ++	unsigned long offset;
     +
    -+	if (unlikely(is_vmalloc_addr(obj))) {
    -+		obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
    ++	if (unlikely(is_vmalloc_addr(head))) {
    ++		offset = offset_in_page(head);
     +	} else {
    -+		struct page *page = virt_to_page(obj);
    -+		struct slab *slab = page_slab(page);
    ++		struct slab *slab = virt_to_slab(head);
     +
     +		if (!slab) {
    -+			obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
    -+		} else if (is_kfence_address(obj)) {
    -+			obj = kfence_object_start(obj);
    ++			offset = offset_in_page(head);
    ++		} else if (is_kfence_address(head)) {
    ++			offset = head - kfence_object_start(head);
     +		} else {
     +			struct kmem_cache *s = slab->slab_cache;
    -+			unsigned int idx = __obj_to_index(s, slab_address(slab), obj);
    ++			unsigned int idx = __obj_to_index(s, slab_address(slab), head);
    ++			void *obj = slab_address(slab) + s->size * idx;
     +
    -+			obj = slab_address(slab) + s->size * idx;
     +			obj = fixup_red_left(s, obj);
    ++			obj = kasan_reset_tag(obj);
    ++			offset = kasan_reset_tag(head) - obj;
     +		}
     +	}
     +
    -+	return obj;
    ++	return head - offset;
     +}
     +
      /*
7:  a7e9dbb20f5b ! 7:  c65397db54e8 mm/slab: introduce kfree_rcu_nolock()
    @@ Commit message
         now that kfree_rcu_sheaf() is available on PREEMPT_RT and
         __kfree_rcu_sheaf() handles unknown context.
     
    -    kfree_rcu_nolock() falls back to the kvfree_rcu batching when sheaves
    -    path fails. In most cases, the sheaves path is expected to succeed
    -    and it's unnecessary to add complexity to the existing kvfree_rcu
    -    batching by teaching it how to handle unknown context.
    +    When sheaves path fails, kfree_rcu_nolock() falls back to
    +    defer_kfree_rcu() that uses an irq work to free the object via
    +    kvfree_call_rcu(). In most cases, the sheaves path is expected to
    +    succeed and therefore it's unnecessary to introduce additional
    +    complexity to the existing kvfree_rcu batching by teaching it
    +    how to handle unknown context.
     
         Since defer_kfree_rcu() can be called on caches without sheaves, move
         deferred_work_barrier() and rcu_barrier() outside the branch in
         kvfree_rcu_barrier_on_cache().
     
    +    Now that deferred kvfree_rcu objects are submitted to kvfree_call_rcu()
    +    after deferred_work_barrier() and may end up in RCU sheaves,
    +    deferred_work_barrier() must be invoked before
    +    flush_rcu_sheaves_on_cache().
    +
    +    Since the RCU sheaf path has not been used on !KVFREE_RCU_BATCHED
    +    kernels, always fall back when kvfree_rcu() is not batched, for
    +    consistency. kvfree_rcu_barrier{,_on_cache()}() on
    +    !KVFREE_RCU_BATCHED are moved to mm/slab_common.c to invoke
    +    deferred_work_barrier() before rcu_barrier().
    +
     
      ## include/linux/rcupdate.h ##
    @@ include/linux/rcupdate.h: do {								\
     + * @ptr: pointer to kfree for double-argument invocations.
     + * @kvrhf: the name of the struct kvfree_rcu_head within the type of @ptr.
     + *
    -+ * Objects that are not allocated via kmalloc_nolock() are not supported.
    ++ * With KVFREE_RCU_BATCHED, kfree_rcu_nolock() tries hard to free objects
    ++ * without any deferred processing, but may still defer freeing.
    ++ * Large kmalloc and vmalloc objects are always deferred.
    ++ *
     + * kfree_rcu_nolock() supports 2-arg variant only.
     + */
     +#define kfree_rcu_nolock(ptr, kvrhf)						\
    @@ include/linux/rcupdate.h: do {								\
       * Place this after a lock-acquisition primitive to guarantee that
       * an UNLOCK+LOCK pair acts as a full barrier.  This guarantee applies
     
    + ## include/linux/slab.h ##
    +@@ include/linux/slab.h: extern void kvfree_sensitive(const void *addr, size_t len);
    + unsigned int kmem_cache_size(struct kmem_cache *s);
    + 
    + #ifndef CONFIG_KVFREE_RCU_BATCHED
    +-static inline void kvfree_rcu_barrier(void)
    +-{
    +-	rcu_barrier();
    +-}
    +-
    +-static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
    +-{
    +-	rcu_barrier();
    +-}
    +-
    + static inline void kfree_rcu_scheduler_running(void) { }
    + #else
    ++void kfree_rcu_scheduler_running(void);
    ++#endif
    ++
    + void kvfree_rcu_barrier(void);
    + 
    + void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
    + 
    +-void kfree_rcu_scheduler_running(void);
    +-#endif
    +-
    + /**
    +  * kmalloc_size_roundup - Report allocation bucket size for the given size
    +  *
    +
      ## mm/slab_common.c ##
     @@ mm/slab_common.c: EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
      EXPORT_TRACEPOINT_SYMBOL(kfree);
    @@ mm/slab_common.c: EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
     +void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr)
     +{
     +	struct slab *slab;
    -+	struct kmem_cache *s;
     +
    -+	VM_WARN_ON_ONCE(is_vmalloc_addr(ptr) || !virt_to_slab(ptr));
    ++	if (!IS_ENABLED(CONFIG_KVFREE_RCU_BATCHED))
    ++		goto fallback;
    ++
    ++	if (unlikely(is_vmalloc_addr(ptr)))
    ++		goto fallback;
     +
     +	slab = virt_to_slab(ptr);
    -+	s = slab->slab_cache;
    ++	if (unlikely(!slab))
    ++		goto fallback;
     +
     +	if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != numa_mem_id()))
     +		goto fallback;
     +
    -+	if (unlikely(!__kfree_rcu_sheaf(s, ptr, SLAB_FREE_NOLOCK)))
    ++	if (unlikely(!__kfree_rcu_sheaf(slab->slab_cache, ptr, SLAB_FREE_NOLOCK)))
     +		goto fallback;
     +
     +	return;
    @@ mm/slab_common.c: EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
      #ifndef CONFIG_KVFREE_RCU_BATCHED
      
      void kvfree_call_rcu(struct rcu_head *head, void *ptr)
    -@@ mm/slab_common.c: void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
    +@@ mm/slab_common.c: void kvfree_call_rcu(struct rcu_head *head, void *ptr)
    + }
    + EXPORT_SYMBOL_GPL(kvfree_call_rcu);
    + 
    ++void kvfree_rcu_barrier(void)
    ++{
    ++	deferred_work_barrier();
    ++	rcu_barrier();
    ++}
    ++EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
    ++
    ++void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
    ++{
    ++	deferred_work_barrier();
    ++	rcu_barrier();
    ++}
    ++EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
    ++
    + void __init kvfree_rcu_init(void)
    + {
    + }
    +@@ mm/slab_common.c: EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
    +  */
    + void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
    + {
    ++	/* kfree_rcu_nolock() might have deferred frees even without sheaves */
    ++	deferred_work_barrier();
    ++
    + 	if (cache_has_sheaves(s)) {
      		cpus_read_lock();
      		flush_rcu_sheaves_on_cache(s);
      		cpus_read_unlock();
    @@ mm/slab_common.c: void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
     -		rcu_barrier();
      	}
      
    -+	/* kfree_rcu_nolock() might have deferred frees even without sheaves */
    -+	deferred_work_barrier();
     +	rcu_barrier();
      	__kvfree_rcu_barrier();
      }
    @@ mm/slub.c: static void deferred_percpu_work_fn(struct irq_work *work);
      	.rcu_sheaves = LLIST_HEAD_INIT(rcu_sheaves),
      	.work = IRQ_WORK_INIT(deferred_percpu_work_fn),
      };
    +@@ mm/slub.c: void flush_all_rcu_sheaves(void)
    + {
    + 	struct kmem_cache *s;
    + 
    ++	deferred_work_barrier();
    ++
    + 	cpus_read_lock();
    + 	mutex_lock(&slab_mutex);
    + 
    +@@ mm/slub.c: void flush_all_rcu_sheaves(void)
    + 	mutex_unlock(&slab_mutex);
    + 	cpus_read_unlock();
    + 
    +-	deferred_work_barrier();
    + 	rcu_barrier();
    + }
    + 
     @@ mm/slub.c: static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
      static void deferred_percpu_work_fn(struct irq_work *work)
      {
    @@ mm/slub.c: static void defer_free(struct kmem_cache *s, void *head)
     +		irq_work_queue(&dpw->work);
     +}
     +
    ++/* Must be called before flush_rcu_sheaves_on_cache() */
      void deferred_work_barrier(void)
      {
      	int cpu;
8:  8e69c83a1e52 ! 8:  aeaf2ab60046 slub_kunit: extend the test for kfree_rcu_nolock()
    @@ lib/tests/slub_kunit.c: struct test_nolock_context {
     -			gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
     +			gfp_t gfp = (i & 1) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
      
    - 			objects[j] = kmalloc_obj(*objects[j], gfp);
    +-			objects[j] = kmalloc(64, gfp);
    ++			objects[j] = kmalloc_obj(*objects[j], gfp);
      			if (!objects[j]) {
    + 				j--;
    + 				while (j >= 0)
     @@ lib/tests/slub_kunit.c: static void test_kmalloc_kfree(void)
      			}
      		}

Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a92d9a4@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b@kernel.org  [2]
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5@kernel.org
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be@kernel.org [2]
Suggested-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
Harry Yoo (Oracle) (8):
      mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
      mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
      mm/slab: use call_rcu() in unknown context if irqs are enabled
      mm/slab: extend deferred free mechanism to handle rcu sheaves
      mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
      mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
      mm/slab: introduce kfree_rcu_nolock()
      slub_kunit: extend the test for kfree_rcu_nolock()

 include/linux/rcupdate.h   |  32 ++++++-
 include/linux/slab.h       |  16 +---
 include/linux/types.h      |   8 ++
 include/trace/events/rcu.h |   2 +-
 lib/tests/slub_kunit.c     | 190 ++++++++++++++++++++++++++++-----------
 mm/slab.h                  |  52 ++++++++++-
 mm/slab_common.c           |  81 +++++++++++++----
 mm/slub.c                  | 220 ++++++++++++++++++++++++++++-----------------
 8 files changed, 433 insertions(+), 168 deletions(-)
---
base-commit: ffc64805cebac330b2a60b2f1f144fcc35b8ab46
change-id: 20260615-kfree_rcu_nolock-e5502555992f

Best regards,
-- 
Harry Yoo (Oracle) <harry@kernel.org>

[PATCH v5 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs

[Harry Yoo (Oracle)]

Since kmalloc_nolock() always fails in NMI and hardirq contexts on
PREEMPT_RT, slub_kunit cannot properly test _nolock() APIs.

Register a kprobe pre-handler to invoke kmalloc_nolock() and
kfree_nolock() in the middle of the slab allocator. However, do not
register the handler on UP kernels because that use case is not
well supported [1] in the kernel.

To attach the pre-handler while s->cpu_sheaves->lock or n->list_lock
is held, add a wrapper function for lockdep_assert_held() that calls
a no-op function slab_attach_kprobe_locked() on debug builds. The
function is optimized away when neither CONFIG_PROVE_LOCKING nor
CONFIG_DEBUG_VM is selected and register_kprobe() fails.

The function calls barrier() to prevent the compiler from optimizing
away its callsites. Otherwise, the compiler may consider the function
does not have any side effect and remove callsites.

Compared to using plain kprobe, this has two advantages: 1) it avoids
hardcoding function names in the test, and 2) it can trigger those APIs
in the middle of a function, where the lock is expected to be held as
annotated with lockdep.

While it was proposed [2] to use kunit function redirection to test
this, it is currently infeasible as some lock helpers don't have
symbols.

Factor out the nested loop that calls kmalloc and friends to
test_kmalloc_kfree(), and call them in
test_kmalloc_kfree_nolock_{perf,kprobe}(), each being an independent
test case. During the refactoring, drop alloc_fail handling as it
doesn't provide much benefits.

Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a92d9a4@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b@kernel.org  [2]
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Shengming Hu <hu.shengming@zte.com.cn>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 lib/tests/slub_kunit.c | 167 +++++++++++++++++++++++++++++++++++--------------
 mm/slub.c              |  36 ++++++++---
 2 files changed, 148 insertions(+), 55 deletions(-)

diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
index fa6d31dbca16..8c2b9911471e 100644
--- a/lib/tests/slub_kunit.c
+++ b/lib/tests/slub_kunit.c
@@ -8,6 +8,7 @@
 #include <linux/rcupdate.h>
 #include <linux/delay.h>
 #include <linux/perf_event.h>
+#include <linux/kprobes.h>
 #include "../mm/slab.h"
 
 static struct kunit_resource resource;
@@ -292,7 +293,7 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)
 	kmem_cache_destroy(s);
 }
 
-#ifdef CONFIG_PERF_EVENTS
+#if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
 #define NR_ITERATIONS 1000
 #define NR_OBJECTS 1000
 static void *objects[NR_OBJECTS];
@@ -302,26 +303,40 @@ struct test_nolock_context {
 	int callback_count;
 	int alloc_ok;
 	int alloc_fail;
+#ifdef CONFIG_PERF_EVENTS
 	struct perf_event *event;
+#endif
+#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
+	struct kprobe kprobe;
+#endif
 };
 
-static struct perf_event_attr hw_attr = {
-	.type = PERF_TYPE_HARDWARE,
-	.config = PERF_COUNT_HW_CPU_CYCLES,
-	.size = sizeof(struct perf_event_attr),
-	.pinned = 1,
-	.disabled = 1,
-	.freq = 1,
-	.sample_freq = 100000,
-};
+static void test_kmalloc_kfree(void)
+{
+	int i, j;
+
+	for (i = 0; i < NR_ITERATIONS; i++) {
+		for (j = 0; j < NR_OBJECTS; j++) {
+			gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
+
+			objects[j] = kmalloc(64, gfp);
+			if (!objects[j]) {
+				j--;
+				while (j >= 0)
+					kfree(objects[j--]);
+				return;
+			}
+		}
+
+		for (j = 0; j < NR_OBJECTS; j++)
+			kfree(objects[j]);
+	}
+}
 
-static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
-						       struct perf_sample_data *data,
-						       struct pt_regs *regs)
+static void test_nolock(struct test_nolock_context *ctx)
 {
 	void *objp;
 	gfp_t gfp;
-	struct test_nolock_context *ctx = event->overflow_handler_context;
 
 	/* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */
 	gfp = (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT;
@@ -335,47 +350,104 @@ static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *event,
 	kfree_nolock(objp);
 	ctx->callback_count++;
 }
+#endif
+
+#ifdef CONFIG_PERF_EVENTS
+static struct perf_event_attr hw_attr = {
+	.type = PERF_TYPE_HARDWARE,
+	.config = PERF_COUNT_HW_CPU_CYCLES,
+	.size = sizeof(struct perf_event_attr),
+	.pinned = 1,
+	.disabled = 1,
+	.freq = 1,
+	.sample_freq = 100000,
+};
 
-static void test_kmalloc_kfree_nolock(struct kunit *test)
+static void overflow_handler_test_nolock(struct perf_event *event,
+					 struct perf_sample_data *data,
+					 struct pt_regs *regs)
+{
+	struct test_nolock_context *ctx = event->overflow_handler_context;
+
+	test_nolock(ctx);
+}
+
+static bool enable_perf_events(struct test_nolock_context *ctx)
 {
-	int i, j;
-	struct test_nolock_context ctx = { .test = test };
 	struct perf_event *event;
-	bool alloc_fail = false;
 
 	event = perf_event_create_kernel_counter(&hw_attr, -1, current,
-						 overflow_handler_test_kmalloc_kfree_nolock,
-						 &ctx);
+						 overflow_handler_test_nolock,
+						 ctx);
+
 	if (IS_ERR(event))
-		kunit_skip(test, "Failed to create perf event");
-	ctx.event = event;
-	perf_event_enable(ctx.event);
-	for (i = 0; i < NR_ITERATIONS; i++) {
-		for (j = 0; j < NR_OBJECTS; j++) {
-			gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
+		return false;
 
-			objects[j] = kmalloc(64, gfp);
-			if (!objects[j]) {
-				j--;
-				while (j >= 0)
-					kfree(objects[j--]);
-				alloc_fail = true;
-				goto cleanup;
-			}
-		}
-		for (j = 0; j < NR_OBJECTS; j++)
-			kfree(objects[j]);
-	}
+	ctx->event = event;
+	perf_event_enable(ctx->event);
+	return true;
+}
+
+static void disable_perf_events(struct test_nolock_context *ctx)
+{
+	kunit_info(ctx->test, "HW perf events: callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
+		   ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail);
+
+	perf_event_disable(ctx->event);
+	perf_event_release_kernel(ctx->event);
+}
+
+static void test_kmalloc_kfree_nolock_perf(struct kunit *test)
+{
+	struct test_nolock_context ctx = { .test = test };
+
+	if (!enable_perf_events(&ctx))
+		kunit_skip(test, "Failed to enable perf event, skipping");
+
+	test_kmalloc_kfree();
+
+	disable_perf_events(&ctx);
+	KUNIT_EXPECT_EQ(test, 0, slab_errors);
+}
+#endif
+
+#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
+static int slab_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
+{
+	struct test_nolock_context *ctx;
+
+	ctx = container_of(p, struct test_nolock_context, kprobe);
+	test_nolock(ctx);
+	return 0;
+}
+
+static bool register_slab_kprobes(struct test_nolock_context *ctx)
+{
+	ctx->kprobe.symbol_name = "slab_attach_kprobe_locked";
+	ctx->kprobe.pre_handler = slab_kprobe_pre_handler;
+
+	if (register_kprobe(&ctx->kprobe))
+		return false;
+	return true;
+}
+
+static void unregister_slab_kprobes(struct test_nolock_context *ctx)
+{
+	kunit_info(ctx->test, "kprobes: callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
+		   ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail);
+	unregister_kprobe(&ctx->kprobe);
+}
+
+static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
+{
+	struct test_nolock_context ctx = { .test = test };
 
-cleanup:
-	perf_event_disable(ctx.event);
-	perf_event_release_kernel(ctx.event);
+	if (!register_slab_kprobes(&ctx))
+		kunit_skip(test, "Failed to register kprobe, skipping");
 
-	kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
-		   ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail);
+	test_kmalloc_kfree();
 
-	if (alloc_fail)
-		kunit_skip(test, "Allocation failed");
+	unregister_slab_kprobes(&ctx);
 	KUNIT_EXPECT_EQ(test, 0, slab_errors);
 }
 #endif
@@ -405,7 +477,10 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_leak_destroy),
 	KUNIT_CASE(test_krealloc_redzone_zeroing),
 #ifdef CONFIG_PERF_EVENTS
-	KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock),
+	KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_perf),
+#endif
+#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
+	KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_kprobe),
 #endif
 	{}
 };
diff --git a/mm/slub.c b/mm/slub.c
index 691977e8c09d..d14226ad0982 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -905,6 +905,24 @@ static inline void slab_set_obj_exts_in_object(struct slab *slab)
 }
 #endif
 
+/*
+ * A no-op function used to attach kprobe handlers in slub_kunit tests.
+ * The barrier is needed to prevent the compiler from optimizing out callsites.
+ */
+#if defined(CONFIG_DEBUG_VM) || defined(CONFIG_PROVE_LOCKING)
+static noinline void slab_attach_kprobe_locked(void)
+{
+	barrier();
+}
+#else
+static inline void slab_attach_kprobe_locked(void) { }
+#endif
+
+#define slab_lockdep_assert_held(lock) do {	\
+	lockdep_assert_held(lock);		\
+	slab_attach_kprobe_locked();	\
+} while (0)
+
 #ifdef CONFIG_SLUB_DEBUG
 
 /*
@@ -1662,7 +1680,7 @@ static void add_full(struct kmem_cache *s,
 	if (!(s->flags & SLAB_STORE_USER))
 		return;
 
-	lockdep_assert_held(&n->list_lock);
+	slab_lockdep_assert_held(&n->list_lock);
 	list_add(&slab->slab_list, &n->full);
 }
 
@@ -1671,7 +1689,7 @@ static void remove_full(struct kmem_cache *s, struct kmem_cache_node *n, struct
 	if (!(s->flags & SLAB_STORE_USER))
 		return;
 
-	lockdep_assert_held(&n->list_lock);
+	slab_lockdep_assert_held(&n->list_lock);
 	list_del(&slab->slab_list);
 }
 
@@ -2884,7 +2902,7 @@ static unsigned int __sheaf_flush_main_batch(struct kmem_cache *s)
 	void *objects[PCS_BATCH_MAX];
 	struct slab_sheaf *sheaf;
 
-	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+	slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
 
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 	sheaf = pcs->main;
@@ -3568,7 +3586,7 @@ __add_partial(struct kmem_cache_node *n, struct slab *slab, enum add_mode mode)
 static inline void add_partial(struct kmem_cache_node *n,
 				struct slab *slab, enum add_mode mode)
 {
-	lockdep_assert_held(&n->list_lock);
+	slab_lockdep_assert_held(&n->list_lock);
 	__add_partial(n, slab, mode);
 }
 
@@ -3582,7 +3600,7 @@ static inline void clear_node_partial_state(struct kmem_cache_node *n,
 static inline void remove_partial(struct kmem_cache_node *n,
 					struct slab *slab)
 {
-	lockdep_assert_held(&n->list_lock);
+	slab_lockdep_assert_held(&n->list_lock);
 	list_del(&slab->slab_list);
 	clear_node_partial_state(n, slab);
 }
@@ -3598,7 +3616,7 @@ static void *alloc_single_from_partial(struct kmem_cache *s,
 {
 	void *object;
 
-	lockdep_assert_held(&n->list_lock);
+	slab_lockdep_assert_held(&n->list_lock);
 
 #ifdef CONFIG_SLUB_DEBUG
 	if (s->flags & SLAB_CONSISTENCY_CHECKS) {
@@ -4663,7 +4681,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	struct node_barn *barn;
 	bool allow_spin;
 
-	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+	slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
 
 	/* Bootstrap or debug cache, back off */
 	if (unlikely(!cache_has_sheaves(s))) {
@@ -5788,7 +5806,7 @@ static void __pcs_install_empty_sheaf(struct kmem_cache *s,
 		struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty,
 		struct node_barn *barn)
 {
-	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+	slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
 
 	/* This is what we expect to find if nobody interrupted us. */
 	if (likely(!pcs->spare)) {
@@ -5839,7 +5857,7 @@ __pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	bool put_fail;
 
 restart:
-	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+	slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
 
 	/* Bootstrap or debug cache, back off */
 	if (unlikely(!cache_has_sheaves(s))) {

-- 
2.53.0

[PATCH v5 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()

[Harry Yoo (Oracle)]

Teach kfree_rcu_sheaf() how to handle the !allow_spin case. Try to get
an empty sheaf from pcs->spare or the barn even when spinning is not
allowed. Unlike __pcs_replace_full_main(), try harder to allocate
an empty sheaf because the fallback path will be more expensive than
kfree_nolock().

Now that slab has internal alloc_flags to describe context, introduce
free_flags analogously and convert free_flags to alloc_flags when
allocating memory in the free path.

When trylock fails or the kernel observes non-NULL pcs->rcu_free after
lock acquisition, free the sheaf instead of putting it to the barn.
This is rare and not worth complicating the code.

Since call_rcu() cannot be called in an unknown context,
kfree_rcu_sheaf() fails when the rcu sheaf becomes full.

Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5@kernel.org
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 mm/slab.h        | 18 +++++++++++++++++-
 mm/slab_common.c |  2 +-
 mm/slub.c        | 34 ++++++++++++++++++++++++++--------
 3 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/mm/slab.h b/mm/slab.h
index aec0047ed943..7e7f986b49e9 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -24,11 +24,27 @@
 #define SLAB_ALLOC_NO_RECURSE	0x04 /* prevent kmalloc() recursion */
 #define SLAB_ALLOC_NO_OBJ_EXT	0x08 /* prevent obj_exts array allocation */
 
+#define SLAB_FREE_DEFAULT	0x00 /* no flags */
+#define SLAB_FREE_NOLOCK	0x01 /* spinning not allowed */
+
+static inline unsigned int to_alloc_flags(unsigned int free_flags)
+{
+	if (free_flags & SLAB_FREE_NOLOCK)
+		return SLAB_ALLOC_NOLOCK;
+	else
+		return SLAB_ALLOC_DEFAULT;
+}
+
 static inline bool alloc_flags_allow_spinning(const unsigned int alloc_flags)
 {
 	return !(alloc_flags & SLAB_ALLOC_NOLOCK);
 }
 
+static inline bool free_flags_allow_spinning(const unsigned int free_flags)
+{
+	return !(free_flags & SLAB_FREE_NOLOCK);
+}
+
 void *__kmalloc_flags_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags,
 				  unsigned int alloc_flags, int node)
 				  __assume_kmalloc_alignment __alloc_size(1);
@@ -431,7 +447,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
 	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT|SLAB_NO_OBJ_EXT));
 }
 
-bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags);
 void flush_all_rcu_sheaves(void);
 void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
 
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 1e1d3feec353..42db62f3e801 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1638,7 +1638,7 @@ static bool kfree_rcu_sheaf(void *obj)
 
 	s = slab->slab_cache;
 	if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
-		return __kfree_rcu_sheaf(s, obj);
+		return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
 
 	return false;
 }
diff --git a/mm/slub.c b/mm/slub.c
index d14226ad0982..2c5ba8c326e7 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2833,7 +2833,8 @@ static inline struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s,
 	return __alloc_empty_sheaf(s, gfp, alloc_flags, s->sheaf_capacity);
 }
 
-static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
+static void __free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
+			       unsigned int free_flags)
 {
 	/*
 	 * If the sheaf was created with SLAB_ALLOC_NO_RECURSE flag then its
@@ -2845,11 +2846,20 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
 		mark_obj_codetag_empty(sheaf);
 
 	VM_WARN_ON_ONCE(sheaf->size > 0);
-	kfree(sheaf);
+
+	if (unlikely(free_flags & SLAB_FREE_NOLOCK))
+		kfree_nolock(sheaf);
+	else
+		kfree(sheaf);
 
 	stat(s, SHEAF_FREE);
 }
 
+static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
+{
+	__free_empty_sheaf(s, sheaf, SLAB_FREE_DEFAULT);
+}
+
 static unsigned int
 refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
 	       unsigned int max);
@@ -6067,10 +6077,11 @@ static void rcu_free_sheaf(struct rcu_head *head)
  */
 static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG);
 
-bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 {
 	struct slub_percpu_sheaves *pcs;
 	struct slab_sheaf *rcu_sheaf;
+	bool allow_spin = free_flags_allow_spinning(free_flags);
 
 	if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
 		return false;
@@ -6083,9 +6094,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	if (unlikely(!pcs->rcu_free)) {
-
 		struct slab_sheaf *empty;
 		struct node_barn *barn;
+		unsigned int alloc_flags = to_alloc_flags(free_flags);
+		gfp_t gfp = allow_spin ? GFP_NOWAIT : __GFP_NOWARN;
 
 		/* Bootstrap or debug cache, fall back */
 		if (unlikely(!cache_has_sheaves(s))) {
@@ -6105,7 +6117,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 			goto fail;
 		}
 
-		empty = barn_get_empty_sheaf(barn, true);
+		empty = barn_get_empty_sheaf(barn, allow_spin);
 
 		if (empty) {
 			pcs->rcu_free = empty;
@@ -6114,20 +6126,20 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 
 		local_unlock(&s->cpu_sheaves->lock);
 
-		empty = alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT);
+		empty = alloc_empty_sheaf(s, gfp, alloc_flags);
 
 		if (!empty)
 			goto fail;
 
 		if (!local_trylock(&s->cpu_sheaves->lock)) {
-			barn_put_empty_sheaf(barn, empty);
+			__free_empty_sheaf(s, empty, free_flags);
 			goto fail;
 		}
 
 		pcs = this_cpu_ptr(s->cpu_sheaves);
 
 		if (unlikely(pcs->rcu_free))
-			barn_put_empty_sheaf(barn, empty);
+			__free_empty_sheaf(s, empty, free_flags);
 		else
 			pcs->rcu_free = empty;
 	}
@@ -6145,6 +6157,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 	if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
 		rcu_sheaf = NULL;
 	} else {
+		if (unlikely(!allow_spin)) {
+			/* call_rcu() cannot be called in an unknown context */
+			rcu_sheaf->size--;
+			local_unlock(&s->cpu_sheaves->lock);
+			goto fail;
+		}
 		pcs->rcu_free = NULL;
 		rcu_sheaf->node = numa_node_id();
 	}

-- 
2.53.0

[PATCH v5 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled

[Harry Yoo (Oracle)]

call_rcu() disables IRQs with local_irq_save() to protect its per-cpu
data structures. Therefore, if IRQs are not disabled, they cannot be
corrupted by reentrance into call_rcu(). So fall back to the deferred
path only when !allow_spin && irqs_disabled().

The RCU subsystem does not guarantee this contractually, and this
optimization relies on RCU's implementation details. Ideally, it should
be removed once call_rcu_nolock() is supported by the RCU subsystem.

Link: https://lore.kernel.org/linux-mm/CAADnVQKRVD5ZSnEKbZZU7w86gHbGHUug2pvzpgZTngNS+fg4rw@mail.gmail.com
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 mm/slub.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 2c5ba8c326e7..bb0f2975c95b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6157,8 +6157,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 	if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
 		rcu_sheaf = NULL;
 	} else {
-		if (unlikely(!allow_spin)) {
-			/* call_rcu() cannot be called in an unknown context */
+		/*
+		 * With !allow_spin, we might have interrupted call_rcu()'s
+		 * IRQ-disabled critical section. If IRQs are not disabled,
+		 * we know that's not the case.
+		 */
+		if (unlikely(!allow_spin && irqs_disabled())) {
 			rcu_sheaf->size--;
 			local_unlock(&s->cpu_sheaves->lock);
 			goto fail;

-- 
2.53.0

[PATCH v5 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves

[Harry Yoo (Oracle)]

__kfree_rcu_sheaf() cannot invoke call_rcu() when spinning is not
allowed and IRQs are disabled. To relax the limitation, extend the
deferred free fallback so that a full rcu sheaf can be submitted to
call_rcu() via the existing IRQ work.

Since the deferred mechanism does more than deferred freeing of objects,
rename the struct to deferred_percpu_work and adjust names accordingly.

When a sheaf is queued on an IRQ work, it is detached from
pcs->rcu_free but call_rcu() is not invoked until the irq_work runs.
To keep the kvfree_rcu barrier's promise, call irq_work_sync() on each
CPU before calling rcu_barrier().

In the meantime, remove the TODO item as apparently there is no simple
and effective way to achieve that. This is because, unlike sheaves,
kfree_rcu() batches objects from different caches together.

Suggested-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 mm/slab.h        |  2 +-
 mm/slab_common.c |  7 ++---
 mm/slub.c        | 85 +++++++++++++++++++++++++++++++++-----------------------
 3 files changed, 53 insertions(+), 41 deletions(-)

diff --git a/mm/slab.h b/mm/slab.h
index 7e7f986b49e9..d13a0f6b7a1b 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -909,7 +909,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
 void __check_heap_object(const void *ptr, unsigned long n,
 			 const struct slab *slab, bool to_user);
 
-void defer_free_barrier(void);
+void deferred_work_barrier(void);
 
 static inline bool slub_debug_orig_size(struct kmem_cache *s)
 {
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 42db62f3e801..d81cc2136c68 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -558,7 +558,7 @@ void kmem_cache_destroy(struct kmem_cache *s)
 	}
 
 	/* Wait for deferred work from kmalloc/kfree_nolock() */
-	defer_free_barrier();
+	deferred_work_barrier();
 
 	cpus_read_lock();
 	mutex_lock(&slab_mutex);
@@ -2146,13 +2146,10 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
 		cpus_read_lock();
 		flush_rcu_sheaves_on_cache(s);
 		cpus_read_unlock();
+		deferred_work_barrier();
 		rcu_barrier();
 	}
 
-	/*
-	 * TODO: Introduce a version of __kvfree_rcu_barrier() that works
-	 * on a specific slab cache.
-	 */
 	__kvfree_rcu_barrier();
 }
 EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
diff --git a/mm/slub.c b/mm/slub.c
index bb0f2975c95b..92c99ff34a2c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -423,6 +423,8 @@ struct slab_sheaf {
 	union {
 		struct rcu_head rcu_head;
 		struct list_head barn_list;
+		/* only used to defer call_rcu() in unknown context */
+		struct llist_node llnode;
 		/* only used for prefilled sheafs */
 		struct {
 			unsigned int capacity;
@@ -4095,6 +4097,20 @@ static void flush_all(struct kmem_cache *s)
 	cpus_read_unlock();
 }
 
+struct deferred_percpu_work {
+	struct llist_head objects;
+	struct llist_head rcu_sheaves;
+	struct irq_work work;
+};
+
+static void deferred_percpu_work_fn(struct irq_work *work);
+
+static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = {
+	.objects = LLIST_HEAD_INIT(objects),
+	.rcu_sheaves = LLIST_HEAD_INIT(rcu_sheaves),
+	.work = IRQ_WORK_INIT(deferred_percpu_work_fn),
+};
+
 static void flush_rcu_sheaf(struct work_struct *w)
 {
 	struct slub_percpu_sheaves *pcs;
@@ -4166,6 +4182,7 @@ void flush_all_rcu_sheaves(void)
 	mutex_unlock(&slab_mutex);
 	cpus_read_unlock();
 
+	deferred_work_barrier();
 	rcu_barrier();
 }
 
@@ -6157,16 +6174,6 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 	if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
 		rcu_sheaf = NULL;
 	} else {
-		/*
-		 * With !allow_spin, we might have interrupted call_rcu()'s
-		 * IRQ-disabled critical section. If IRQs are not disabled,
-		 * we know that's not the case.
-		 */
-		if (unlikely(!allow_spin && irqs_disabled())) {
-			rcu_sheaf->size--;
-			local_unlock(&s->cpu_sheaves->lock);
-			goto fail;
-		}
 		pcs->rcu_free = NULL;
 		rcu_sheaf->node = numa_node_id();
 	}
@@ -6175,8 +6182,22 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 	 * we flush before local_unlock to make sure a racing
 	 * flush_all_rcu_sheaves() doesn't miss this sheaf
 	 */
-	if (rcu_sheaf)
-		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+	if (rcu_sheaf) {
+		/*
+		 * With !allow_spin, we might have interrupted call_rcu()'s
+		 * IRQ-disabled critical section. If IRQs are not disabled,
+		 * we know that's not the case.
+		 */
+		if (unlikely(!allow_spin && irqs_disabled())) {
+			struct deferred_percpu_work *dpw;
+
+			dpw = this_cpu_ptr(&deferred_percpu_work);
+			if (llist_add(&rcu_sheaf->llnode, &dpw->rcu_sheaves))
+				irq_work_queue(&dpw->work);
+		} else {
+			call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+		}
+	}
 
 	local_unlock(&s->cpu_sheaves->lock);
 
@@ -6375,31 +6396,21 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 	}
 }
 
-struct defer_free {
-	struct llist_head objects;
-	struct irq_work work;
-};
-
-static void free_deferred_objects(struct irq_work *work);
-
-static DEFINE_PER_CPU(struct defer_free, defer_free_objects) = {
-	.objects = LLIST_HEAD_INIT(objects),
-	.work = IRQ_WORK_INIT(free_deferred_objects),
-};
-
 /*
  * In PREEMPT_RT irq_work runs in per-cpu kthread, so it's safe
  * to take sleeping spin_locks from __slab_free().
  * In !PREEMPT_RT irq_work will run after local_unlock_irqrestore().
  */
-static void free_deferred_objects(struct irq_work *work)
+static void deferred_percpu_work_fn(struct irq_work *work)
 {
-	struct defer_free *df = container_of(work, struct defer_free, work);
-	struct llist_head *objs = &df->objects;
+	struct deferred_percpu_work *dpw;
+	struct llist_head *objs, *rcu_sheaves;
 	struct llist_node *llnode, *pos, *t;
+	struct slab_sheaf *sheaf, *next;
 
-	if (llist_empty(objs))
-		return;
+	dpw = container_of(work, struct deferred_percpu_work, work);
+	rcu_sheaves = &dpw->rcu_sheaves;
+	objs = &dpw->objects;
 
 	llnode = llist_del_all(objs);
 	llist_for_each_safe(pos, t, llnode) {
@@ -6423,27 +6434,31 @@ static void free_deferred_objects(struct irq_work *work)
 		__slab_free(s, slab, x, x, 1, _THIS_IP_);
 		stat(s, FREE_SLOWPATH);
 	}
+
+	llnode = llist_del_all(rcu_sheaves);
+	llist_for_each_entry_safe(sheaf, next, llnode, llnode)
+		call_rcu(&sheaf->rcu_head, rcu_free_sheaf);
 }
 
 static void defer_free(struct kmem_cache *s, void *head)
 {
-	struct defer_free *df;
+	struct deferred_percpu_work *dpw;
 
 	guard(preempt)();
 
 	head = kasan_reset_tag(head);
 
-	df = this_cpu_ptr(&defer_free_objects);
-	if (llist_add(head + s->offset, &df->objects))
-		irq_work_queue(&df->work);
+	dpw = this_cpu_ptr(&deferred_percpu_work);
+	if (llist_add(head + s->offset, &dpw->objects))
+		irq_work_queue(&dpw->work);
 }
 
-void defer_free_barrier(void)
+void deferred_work_barrier(void)
 {
 	int cpu;
 
 	for_each_possible_cpu(cpu)
-		irq_work_sync(&per_cpu_ptr(&defer_free_objects, cpu)->work);
+		irq_work_sync(&per_cpu_ptr(&deferred_percpu_work, cpu)->work);
 }
 
 static __fastpath_inline

-- 
2.53.0

[PATCH v5 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT

[Harry Yoo (Oracle)]

As suggested by Vlastimil Babka [1], kfree_rcu_sheaf() can be used
on PREEMPT_RT if we always assume spinning is not allowed on PREEMPT_RT.
This is because local_trylock and spinlock_t are safe to use with
trylock and unlock as long as the kernel does not spin and the context
is not NMI and not hardirq.

Now that __kfree_rcu_sheaf() knows how to handle SLAB_FREE_NOLOCK,
relax the limitation and try the sheaves path on PREEMPT_RT as well.

Keep the lockdep map on non RT kernels. However, do not use the lockdep
map on PREEMPT_RT to avoid suppressing valid lockdep warnings.

As pointed by Vlastimil Babka [2], on PREEMPT_RT it is unnecessary to
defer call_rcu() under IRQ-disabled section or raw spinlock. However,
let us avoid adding more complexity as the scenario is not supposed
to be common on PREEMPT_RT, with a hope that call_rcu_nolock() will be
soon supported in RCU.

Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be@kernel.org [2]
Suggested-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 mm/slab_common.c | 12 ++++++++++--
 mm/slub.c        | 17 ++++++++++-------
 2 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index d81cc2136c68..9c2cca9add89 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1628,6 +1628,14 @@ static bool kfree_rcu_sheaf(void *obj)
 {
 	struct kmem_cache *s;
 	struct slab *slab;
+	unsigned int free_flags = SLAB_FREE_DEFAULT;
+
+	/*
+	 * It is not safe to spin on PREEMPT_RT because the kernel might be
+	 * holding a raw spinlock and slab acquires sleeping locks.
+	 */
+	if (IS_ENABLED(CONFIG_PREEMPT_RT))
+		free_flags = SLAB_FREE_NOLOCK;
 
 	if (is_vmalloc_addr(obj))
 		return false;
@@ -1638,7 +1646,7 @@ static bool kfree_rcu_sheaf(void *obj)
 
 	s = slab->slab_cache;
 	if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
-		return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
+		return __kfree_rcu_sheaf(s, obj, free_flags);
 
 	return false;
 }
@@ -1987,7 +1995,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	if (!head)
 		might_sleep();
 
-	if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
+	if (kfree_rcu_sheaf(ptr))
 		return;
 
 	// Queue the object but don't yet schedule the batch.
diff --git a/mm/slub.c b/mm/slub.c
index 92c99ff34a2c..6c81722afb18 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6085,12 +6085,13 @@ static void rcu_free_sheaf(struct rcu_head *head)
  * kvfree_call_rcu() can be called while holding a raw_spinlock_t. Since
  * __kfree_rcu_sheaf() may acquire a spinlock_t (sleeping lock on PREEMPT_RT),
  * this would violate lock nesting rules. Therefore, kvfree_call_rcu() avoids
- * this problem by bypassing the sheaves layer entirely on PREEMPT_RT.
+ * this problem by passing SLAB_FREE_NOLOCK on PREEMPT_RT.
  *
  * However, lockdep still complains that it is invalid to acquire spinlock_t
  * while holding raw_spinlock_t, even on !PREEMPT_RT where spinlock_t is a
  * spinning lock. Tell lockdep that acquiring spinlock_t is valid here
- * by temporarily raising the wait-type to LD_WAIT_CONFIG.
+ * by temporarily raising the wait-type to LD_WAIT_CONFIG. Skip the lockdep map
+ * on PREEMPT_RT to avoid suppressing valid lockdep warnings.
  */
 static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG);
 
@@ -6100,10 +6101,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 	struct slab_sheaf *rcu_sheaf;
 	bool allow_spin = free_flags_allow_spinning(free_flags);
 
-	if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
-		return false;
+	VM_WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT) && allow_spin);
 
-	lock_map_acquire_try(&kfree_rcu_sheaf_map);
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		lock_map_acquire_try(&kfree_rcu_sheaf_map);
 
 	if (!local_trylock(&s->cpu_sheaves->lock))
 		goto fail;
@@ -6202,12 +6203,14 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 	local_unlock(&s->cpu_sheaves->lock);
 
 	stat(s, FREE_RCU_SHEAF);
-	lock_map_release(&kfree_rcu_sheaf_map);
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		lock_map_release(&kfree_rcu_sheaf_map);
 	return true;
 
 fail:
 	stat(s, FREE_RCU_SHEAF_FAIL);
-	lock_map_release(&kfree_rcu_sheaf_map);
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		lock_map_release(&kfree_rcu_sheaf_map);
 	return false;
 }
 

-- 
2.53.0

[PATCH v5 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching

[Harry Yoo (Oracle)]

rcu_head is overkill for kvfree_rcu() because the callback
function is always either kfree(), vfree(), or free_large_kmalloc(),
and thus there is no need for a function pointer.

kvfree_rcu batching reuses the field to store the start address
of an object, however, this is not strictly needed because we can
calculate the start address in the slowpath. For the purpose of
kvfree_rcu batching, it is sufficient to implement a linked list using
a single pointer.

Introduce a new struct called kvfree_rcu_head (the name was suggested
by Vlastimil Babka), which is similar to rcu_head but is only a single
pointer to build a linked list, without a function pointer, when
CONFIG_KVFREE_RCU_BATCHED=y.

When kvfree_rcu is not batched, kvfree_rcu_head is the same size
as rcu_head. Note that shrinking struct kvfree_rcu_head on
CONFIG_KVFREE_RCU_BATCHED=n kernels would inevitably require additional
complexity and also some sort of batching (which defeats the purpose of
the config option) because it cannot fall back to call_rcu().

For now there are no user-visible changes to the API. k[v]free_rcu()
simply casts rcu_head to kvfree_rcu_head. While this does not affect
the API, it allows kfree_rcu_nolock() to reuse kvfree_rcu batching
as a fallback when trylock or sheaf allocation fails.

Stop storing the object pointer in rcu_head.func and instead calculate
the object's start address in kvfree_rcu_list(). Factor out the existing
logic to calculate the start address from kvfree_rcu_cb() to
kvmalloc_obj_start_addr(). To avoid losing the KASAN tag, calculate
the offset and subtract it from the address of the kvfree_rcu_head.

Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 include/linux/rcupdate.h   | 10 ++++++----
 include/linux/types.h      |  8 ++++++++
 include/trace/events/rcu.h |  2 +-
 mm/slab.h                  | 32 ++++++++++++++++++++++++++++++++
 mm/slab_common.c           | 17 ++++++++---------
 mm/slub.c                  | 40 +++++++++-------------------------------
 6 files changed, 64 insertions(+), 45 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 5e95acc33989..ef5bb6981133 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1098,19 +1098,21 @@ static inline void rcu_read_unlock_migrate(void)
 /*
  * In mm/slab_common.c, no suitable header to include here.
  */
-void kvfree_call_rcu(struct rcu_head *head, void *ptr);
+void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr);
 
 /*
  * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the
  * comment of kfree_rcu() for details.
  */
-#define kvfree_rcu_arg_2(ptr, rhf)					\
+#define kvfree_rcu_arg_2(ptr, kvrhf)					\
 do {									\
 	typeof (ptr) ___p = (ptr);					\
+	struct kvfree_rcu_head *___head;				\
 									\
 	if (___p) {							\
-		BUILD_BUG_ON(offsetof(typeof(*(ptr)), rhf) >= 4096);	\
-		kvfree_call_rcu(&((___p)->rhf), (void *) (___p));	\
+		BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >= 4096);	\
+		___head = (struct kvfree_rcu_head *) &(___p)->kvrhf;	\
+		kvfree_call_rcu(___head, (void *) (___p));		\
 	}								\
 } while (0)
 
diff --git a/include/linux/types.h b/include/linux/types.h
index 93166b0b0617..8b0976669d66 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -255,6 +255,14 @@ struct callback_head {
 } __attribute__((aligned(sizeof(void *))));
 #define rcu_head callback_head
 
+#ifdef CONFIG_KVFREE_RCU_BATCHED
+struct kvfree_rcu_head {
+	struct kvfree_rcu_head *next;
+};
+#else
+#define kvfree_rcu_head rcu_head
+#endif
+
 typedef void (*rcu_callback_t)(struct rcu_head *head);
 typedef void (*call_rcu_func_t)(struct rcu_head *head, rcu_callback_t func);
 
diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index 5fbdabe3faea..a74027126a2f 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -625,7 +625,7 @@ TRACE_EVENT_RCU(rcu_invoke_callback,
  */
 TRACE_EVENT_RCU(rcu_invoke_kvfree_callback,
 
-	TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset),
+	TP_PROTO(const char *rcuname, struct kvfree_rcu_head *rhp, unsigned long offset),
 
 	TP_ARGS(rcuname, rhp, offset),
 
diff --git a/mm/slab.h b/mm/slab.h
index d13a0f6b7a1b..ec72ed95c21b 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -347,6 +347,37 @@ static inline unsigned int obj_to_index(const struct kmem_cache *cache,
 	return __obj_to_index(cache, slab_address(slab), obj);
 }
 
+/*
+ * kvfree_rcu_head offset can be only less than page size.
+ * Calculate the start address while preserving the KASAN tag.
+ */
+static inline void *kvmalloc_obj_start_addr(void *head)
+{
+	unsigned long offset;
+
+	if (unlikely(is_vmalloc_addr(head))) {
+		offset = offset_in_page(head);
+	} else {
+		struct slab *slab = virt_to_slab(head);
+
+		if (!slab) {
+			offset = offset_in_page(head);
+		} else if (is_kfence_address(head)) {
+			offset = head - kfence_object_start(head);
+		} else {
+			struct kmem_cache *s = slab->slab_cache;
+			unsigned int idx = __obj_to_index(s, slab_address(slab), head);
+			void *obj = slab_address(slab) + s->size * idx;
+
+			obj = fixup_red_left(s, obj);
+			obj = kasan_reset_tag(obj);
+			offset = kasan_reset_tag(head) - obj;
+		}
+	}
+
+	return head - offset;
+}
+
 /*
  * State of the slab allocator.
  *
@@ -910,6 +941,7 @@ void __check_heap_object(const void *ptr, unsigned long n,
 			 const struct slab *slab, bool to_user);
 
 void deferred_work_barrier(void);
+void defer_kfree_rcu(struct kvfree_rcu_head *head);
 
 static inline bool slub_debug_orig_size(struct kmem_cache *s)
 {
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 9c2cca9add89..691ce6db4519 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1379,7 +1379,7 @@ struct kvfree_rcu_bulk_data {
 
 struct kfree_rcu_cpu_work {
 	struct rcu_work rcu_work;
-	struct rcu_head *head_free;
+	struct kvfree_rcu_head *head_free;
 	struct rcu_gp_oldstate head_free_gp_snap;
 	struct list_head bulk_head_free[FREE_N_CHANNELS];
 	struct kfree_rcu_cpu *krcp;
@@ -1415,7 +1415,7 @@ struct kfree_rcu_cpu_work {
 struct kfree_rcu_cpu {
 	// Objects queued on a linked list
 	// through their rcu_head structures.
-	struct rcu_head *head;
+	struct kvfree_rcu_head *head;
 	unsigned long head_gp_snap;
 	atomic_t head_count;
 
@@ -1556,12 +1556,12 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp,
 }
 
 static void
-kvfree_rcu_list(struct rcu_head *head)
+kvfree_rcu_list(struct kvfree_rcu_head *head)
 {
-	struct rcu_head *next;
+	struct kvfree_rcu_head *next;
 
 	for (; head; head = next) {
-		void *ptr = (void *) head->func;
+		void *ptr = kvmalloc_obj_start_addr(head);
 		unsigned long offset = (void *) head - ptr;
 
 		next = head->next;
@@ -1585,7 +1585,7 @@ static void kfree_rcu_work(struct work_struct *work)
 	unsigned long flags;
 	struct kvfree_rcu_bulk_data *bnode, *n;
 	struct list_head bulk_head[FREE_N_CHANNELS];
-	struct rcu_head *head;
+	struct kvfree_rcu_head *head;
 	struct kfree_rcu_cpu *krcp;
 	struct kfree_rcu_cpu_work *krwp;
 	struct rcu_gp_oldstate head_gp_snap;
@@ -1716,7 +1716,7 @@ kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp)
 {
 	struct list_head bulk_ready[FREE_N_CHANNELS];
 	struct kvfree_rcu_bulk_data *bnode, *n;
-	struct rcu_head *head_ready = NULL;
+	struct kvfree_rcu_head *head_ready = NULL;
 	unsigned long flags;
 	int i;
 
@@ -1979,7 +1979,7 @@ void __init kfree_rcu_scheduler_running(void)
  * be free'd in workqueue context. This allows us to: batch requests together to
  * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
  */
-void kvfree_call_rcu(struct rcu_head *head, void *ptr)
+void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr)
 {
 	unsigned long flags;
 	struct kfree_rcu_cpu *krcp;
@@ -2017,7 +2017,6 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 			// Inline if kvfree_rcu(one_arg) call.
 			goto unlock_return;
 
-		head->func = ptr;
 		head->next = krcp->head;
 		WRITE_ONCE(krcp->head, head);
 		atomic_inc(&krcp->head_count);
diff --git a/mm/slub.c b/mm/slub.c
index 6c81722afb18..774104267f3c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6720,43 +6720,21 @@ static void free_large_kmalloc(struct page *page, void *object)
  */
 void kvfree_rcu_cb(struct rcu_head *head)
 {
-	void *obj = head;
-	struct page *page;
-	struct slab *slab;
-	struct kmem_cache *s;
-	void *slab_addr;
+	void *obj;
+
+	obj = kvmalloc_obj_start_addr(head);
 
 	if (is_vmalloc_addr(obj)) {
-		obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
 		vfree(obj);
-		return;
-	}
-
-	page = virt_to_page(obj);
-	slab = page_slab(page);
-	if (!slab) {
-		/*
-		 * rcu_head offset can be only less than page size so no need to
-		 * consider allocation order
-		 */
-		obj = (void *) PAGE_ALIGN_DOWN((unsigned long)obj);
-		free_large_kmalloc(page, obj);
-		return;
-	}
-
-	s = slab->slab_cache;
-	slab_addr = slab_address(slab);
-
-	if (is_kfence_address(obj)) {
-		obj = kfence_object_start(obj);
 	} else {
-		unsigned int idx = __obj_to_index(s, slab_addr, obj);
+		struct page *page = virt_to_page(obj);
+		struct slab *slab = page_slab(page);
 
-		obj = slab_addr + s->size * idx;
-		obj = fixup_red_left(s, obj);
+		if (slab)
+			slab_free(slab->slab_cache, slab, obj, _RET_IP_);
+		else
+			free_large_kmalloc(page, obj);
 	}
-
-	slab_free(s, slab, obj, _RET_IP_);
 }
 
 /**

-- 
2.53.0

[PATCH v5 7/8] mm/slab: introduce kfree_rcu_nolock()

[Harry Yoo (Oracle)]

Currently, k[v]free_rcu() cannot be called in unknown context since
it could lead to a deadlock when called in the middle of k[v]free_rcu().

Make users' lives easier by introducing kfree_rcu_nolock() variant,
now that kfree_rcu_sheaf() is available on PREEMPT_RT and
__kfree_rcu_sheaf() handles unknown context.

When sheaves path fails, kfree_rcu_nolock() falls back to
defer_kfree_rcu() that uses an irq work to free the object via
kvfree_call_rcu(). In most cases, the sheaves path is expected to
succeed and therefore it's unnecessary to introduce additional
complexity to the existing kvfree_rcu batching by teaching it
how to handle unknown context.

Since defer_kfree_rcu() can be called on caches without sheaves, move
deferred_work_barrier() and rcu_barrier() outside the branch in
kvfree_rcu_barrier_on_cache().

Now that deferred kvfree_rcu objects are submitted to kvfree_call_rcu()
after deferred_work_barrier() and may end up in RCU sheaves,
deferred_work_barrier() must be invoked before
flush_rcu_sheaves_on_cache().

Since the RCU sheaf path has not been used on !KVFREE_RCU_BATCHED
kernels, always fall back when kvfree_rcu() is not batched, for
consistency. kvfree_rcu_barrier{,_on_cache()}() on
!KVFREE_RCU_BATCHED are moved to mm/slab_common.c to invoke
deferred_work_barrier() before rcu_barrier().

Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 include/linux/rcupdate.h | 22 ++++++++++++++++++++++
 include/linux/slab.h     | 16 +++-------------
 mm/slab_common.c         | 47 +++++++++++++++++++++++++++++++++++++++++++++--
 mm/slub.c                | 28 ++++++++++++++++++++++++++--
 4 files changed, 96 insertions(+), 17 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index ef5bb6981133..aede77bd0387 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1099,6 +1099,7 @@ static inline void rcu_read_unlock_migrate(void)
  * In mm/slab_common.c, no suitable header to include here.
  */
 void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr);
+void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr);
 
 /*
  * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See the
@@ -1124,6 +1125,27 @@ do {								\
 		kvfree_call_rcu(NULL, (void *) (___p));		\
 } while (0)
 
+/**
+ * kfree_rcu_nolock() - a version of kfree_rcu() that can be called in any context.
+ * @ptr: pointer to kfree for double-argument invocations.
+ * @kvrhf: the name of the struct kvfree_rcu_head within the type of @ptr.
+ *
+ * With KVFREE_RCU_BATCHED, kfree_rcu_nolock() tries hard to free objects
+ * without any deferred processing, but may still defer freeing.
+ * Large kmalloc and vmalloc objects are always deferred.
+ *
+ * kfree_rcu_nolock() supports 2-arg variant only.
+ */
+#define kfree_rcu_nolock(ptr, kvrhf)						\
+do {										\
+	typeof (ptr) ___p = (ptr);						\
+										\
+	if (___p) {								\
+		BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >= 4096);		\
+		kfree_call_rcu_nolock(&((___p)->kvrhf), (void *) (___p));	\
+	}									\
+} while (0)
+
 /*
  * Place this after a lock-acquisition primitive to guarantee that
  * an UNLOCK+LOCK pair acts as a full barrier.  This guarantee applies
diff --git a/include/linux/slab.h b/include/linux/slab.h
index e1915026e030..cda126def67a 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -1430,25 +1430,15 @@ extern void kvfree_sensitive(const void *addr, size_t len);
 unsigned int kmem_cache_size(struct kmem_cache *s);
 
 #ifndef CONFIG_KVFREE_RCU_BATCHED
-static inline void kvfree_rcu_barrier(void)
-{
-	rcu_barrier();
-}
-
-static inline void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
-{
-	rcu_barrier();
-}
-
 static inline void kfree_rcu_scheduler_running(void) { }
 #else
+void kfree_rcu_scheduler_running(void);
+#endif
+
 void kvfree_rcu_barrier(void);
 
 void kvfree_rcu_barrier_on_cache(struct kmem_cache *s);
 
-void kfree_rcu_scheduler_running(void);
-#endif
-
 /**
  * kmalloc_size_roundup - Report allocation bucket size for the given size
  *
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 691ce6db4519..b8f21eabcccc 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1296,6 +1296,33 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
 EXPORT_TRACEPOINT_SYMBOL(kfree);
 EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free);
 
+void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr)
+{
+	struct slab *slab;
+
+	if (!IS_ENABLED(CONFIG_KVFREE_RCU_BATCHED))
+		goto fallback;
+
+	if (unlikely(is_vmalloc_addr(ptr)))
+		goto fallback;
+
+	slab = virt_to_slab(ptr);
+	if (unlikely(!slab))
+		goto fallback;
+
+	if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != numa_mem_id()))
+		goto fallback;
+
+	if (unlikely(!__kfree_rcu_sheaf(slab->slab_cache, ptr, SLAB_FREE_NOLOCK)))
+		goto fallback;
+
+	return;
+
+fallback:
+	defer_kfree_rcu(head);
+}
+EXPORT_SYMBOL_GPL(kfree_call_rcu_nolock);
+
 #ifndef CONFIG_KVFREE_RCU_BATCHED
 
 void kvfree_call_rcu(struct rcu_head *head, void *ptr)
@@ -1313,6 +1340,20 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 }
 EXPORT_SYMBOL_GPL(kvfree_call_rcu);
 
+void kvfree_rcu_barrier(void)
+{
+	deferred_work_barrier();
+	rcu_barrier();
+}
+EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
+
+void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
+{
+	deferred_work_barrier();
+	rcu_barrier();
+}
+EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
+
 void __init kvfree_rcu_init(void)
 {
 }
@@ -2149,14 +2190,16 @@ EXPORT_SYMBOL_GPL(kvfree_rcu_barrier);
  */
 void kvfree_rcu_barrier_on_cache(struct kmem_cache *s)
 {
+	/* kfree_rcu_nolock() might have deferred frees even without sheaves */
+	deferred_work_barrier();
+
 	if (cache_has_sheaves(s)) {
 		cpus_read_lock();
 		flush_rcu_sheaves_on_cache(s);
 		cpus_read_unlock();
-		deferred_work_barrier();
-		rcu_barrier();
 	}
 
+	rcu_barrier();
 	__kvfree_rcu_barrier();
 }
 EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache);
diff --git a/mm/slub.c b/mm/slub.c
index 774104267f3c..6e5aa38e7569 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4099,6 +4099,7 @@ static void flush_all(struct kmem_cache *s)
 
 struct deferred_percpu_work {
 	struct llist_head objects;
+	struct llist_head objects_by_rcu;
 	struct llist_head rcu_sheaves;
 	struct irq_work work;
 };
@@ -4107,6 +4108,7 @@ static void deferred_percpu_work_fn(struct irq_work *work);
 
 static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = {
 	.objects = LLIST_HEAD_INIT(objects),
+	.objects_by_rcu = LLIST_HEAD_INIT(objects_by_rcu),
 	.rcu_sheaves = LLIST_HEAD_INIT(rcu_sheaves),
 	.work = IRQ_WORK_INIT(deferred_percpu_work_fn),
 };
@@ -4170,6 +4172,8 @@ void flush_all_rcu_sheaves(void)
 {
 	struct kmem_cache *s;
 
+	deferred_work_barrier();
+
 	cpus_read_lock();
 	mutex_lock(&slab_mutex);
 
@@ -4182,7 +4186,6 @@ void flush_all_rcu_sheaves(void)
 	mutex_unlock(&slab_mutex);
 	cpus_read_unlock();
 
-	deferred_work_barrier();
 	rcu_barrier();
 }
 
@@ -6407,13 +6410,14 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 static void deferred_percpu_work_fn(struct irq_work *work)
 {
 	struct deferred_percpu_work *dpw;
-	struct llist_head *objs, *rcu_sheaves;
+	struct llist_head *objs, *objs_by_rcu, *rcu_sheaves;
 	struct llist_node *llnode, *pos, *t;
 	struct slab_sheaf *sheaf, *next;
 
 	dpw = container_of(work, struct deferred_percpu_work, work);
 	rcu_sheaves = &dpw->rcu_sheaves;
 	objs = &dpw->objects;
+	objs_by_rcu = &dpw->objects_by_rcu;
 
 	llnode = llist_del_all(objs);
 	llist_for_each_safe(pos, t, llnode) {
@@ -6438,6 +6442,14 @@ static void deferred_percpu_work_fn(struct irq_work *work)
 		stat(s, FREE_SLOWPATH);
 	}
 
+	llnode = llist_del_all(objs_by_rcu);
+	llist_for_each_safe(pos, t, llnode) {
+		void *head = pos;
+		void *objp = kvmalloc_obj_start_addr(head);
+
+		kvfree_call_rcu(head, objp);
+	}
+
 	llnode = llist_del_all(rcu_sheaves);
 	llist_for_each_entry_safe(sheaf, next, llnode, llnode)
 		call_rcu(&sheaf->rcu_head, rcu_free_sheaf);
@@ -6456,6 +6468,18 @@ static void defer_free(struct kmem_cache *s, void *head)
 		irq_work_queue(&dpw->work);
 }
 
+void defer_kfree_rcu(struct kvfree_rcu_head *head)
+{
+	struct deferred_percpu_work *dpw;
+
+	guard(preempt)();
+
+	dpw = this_cpu_ptr(&deferred_percpu_work);
+	if (llist_add((struct llist_node *)head, &dpw->objects_by_rcu))
+		irq_work_queue(&dpw->work);
+}
+
+/* Must be called before flush_rcu_sheaves_on_cache() */
 void deferred_work_barrier(void)
 {
 	int cpu;

-- 
2.53.0

[PATCH v5 8/8] slub_kunit: extend the test for kfree_rcu_nolock()

[Harry Yoo (Oracle)]

When slub_kunit is not built-in, call kfree_rcu() and kfree_rcu_nolock()
to test kfree_rcu_nolock() in slub_kunit.

Rename the test case as the test covers more _nolock() APIs.

Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
---
 lib/tests/slub_kunit.c | 47 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 30 insertions(+), 17 deletions(-)

diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
index 8c2b9911471e..e3b63f0338d5 100644
--- a/lib/tests/slub_kunit.c
+++ b/lib/tests/slub_kunit.c
@@ -162,7 +162,10 @@ static void test_kmalloc_redzone_access(struct kunit *test)
 }
 
 struct test_kfree_rcu_struct {
-	struct rcu_head rcu;
+	union {
+		struct rcu_head rcu;
+		struct kvfree_rcu_head kvrcu;
+	};
 };
 
 static void test_kfree_rcu(struct kunit *test)
@@ -296,7 +299,7 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)
 #if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
 #define NR_ITERATIONS 1000
 #define NR_OBJECTS 1000
-static void *objects[NR_OBJECTS];
+static struct test_kfree_rcu_struct *objects[NR_OBJECTS];
 
 struct test_nolock_context {
 	struct kunit *test;
@@ -311,15 +314,16 @@ struct test_nolock_context {
 #endif
 };
 
-static void test_kmalloc_kfree(void)
+static void test_kmalloc_and_friends(void)
 {
 	int i, j;
+	bool can_use_kfree_rcu = !IS_BUILTIN(CONFIG_SLUB_KUNIT_TEST);
 
 	for (i = 0; i < NR_ITERATIONS; i++) {
 		for (j = 0; j < NR_OBJECTS; j++) {
-			gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
+			gfp_t gfp = (i & 1) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
 
-			objects[j] = kmalloc(64, gfp);
+			objects[j] = kmalloc_obj(*objects[j], gfp);
 			if (!objects[j]) {
 				j--;
 				while (j >= 0)
@@ -328,26 +332,35 @@ static void test_kmalloc_kfree(void)
 			}
 		}
 
-		for (j = 0; j < NR_OBJECTS; j++)
-			kfree(objects[j]);
+		for (j = 0; j < NR_OBJECTS; j++) {
+			if (can_use_kfree_rcu && (i & 2))
+				kfree_rcu(objects[j], rcu);
+			else
+				kfree(objects[j]);
+		}
 	}
 }
 
 static void test_nolock(struct test_nolock_context *ctx)
 {
-	void *objp;
+	struct test_kfree_rcu_struct *objp;
 	gfp_t gfp;
+	bool can_use_kfree_rcu = !IS_BUILTIN(CONFIG_SLUB_KUNIT_TEST);
 
 	/* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */
-	gfp = (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT;
-	objp = kmalloc_nolock(64, gfp, NUMA_NO_NODE);
+	gfp = (ctx->callback_count & 1) ? 0 : __GFP_ACCOUNT;
+	objp = kmalloc_nolock(sizeof(*objp), gfp, NUMA_NO_NODE);
 
 	if (objp)
 		ctx->alloc_ok++;
 	else
 		ctx->alloc_fail++;
 
-	kfree_nolock(objp);
+	if (can_use_kfree_rcu && (ctx->callback_count & 2))
+		kfree_rcu_nolock(objp, kvrcu);
+	else
+		kfree_nolock(objp);
+
 	ctx->callback_count++;
 }
 #endif
@@ -397,14 +410,14 @@ static void disable_perf_events(struct test_nolock_context *ctx)
 	perf_event_release_kernel(ctx->event);
 }
 
-static void test_kmalloc_kfree_nolock_perf(struct kunit *test)
+static void test_kmalloc_nolock_and_friends_perf(struct kunit *test)
 {
 	struct test_nolock_context ctx = { .test = test };
 
 	if (!enable_perf_events(&ctx))
 		kunit_skip(test, "Failed to enable perf event, skipping");
 
-	test_kmalloc_kfree();
+	test_kmalloc_and_friends();
 
 	disable_perf_events(&ctx);
 	KUNIT_EXPECT_EQ(test, 0, slab_errors);
@@ -438,14 +451,14 @@ static void unregister_slab_kprobes(struct test_nolock_context *ctx)
 	unregister_kprobe(&ctx->kprobe);
 }
 
-static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
+static void test_kmalloc_nolock_and_friends_kprobe(struct kunit *test)
 {
 	struct test_nolock_context ctx = { .test = test };
 
 	if (!register_slab_kprobes(&ctx))
 		kunit_skip(test, "Failed to register kprobe, skipping");
 
-	test_kmalloc_kfree();
+	test_kmalloc_and_friends();
 
 	unregister_slab_kprobes(&ctx);
 	KUNIT_EXPECT_EQ(test, 0, slab_errors);
@@ -477,10 +490,10 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_leak_destroy),
 	KUNIT_CASE(test_krealloc_redzone_zeroing),
 #ifdef CONFIG_PERF_EVENTS
-	KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_perf),
+	KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf),
 #endif
 #if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
-	KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_kprobe),
+	KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_kprobe),
 #endif
 	{}
 };

-- 
2.53.0

Re: [PATCH v5 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Manual cleanup of global kprobe leads to stack Use-After-Free if the KUnit test aborts.
- [Medium] Global kprobe registration causes concurrent data races and triggers test allocations from unrelated system processes.
--

--- Patch [1]: [PATCH v5 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs ---
commit a8a477af9c0547e09308b4a325920cd9de52b25e
Author: Harry Yoo (Oracle) <harry@kernel.org>

    mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
    
    Since kmalloc_nolock() always fails in NMI and hardirq contexts on
    PREEMPT_RT, slub_kunit cannot properly test _nolock() APIs.
    
    Register a kprobe pre-handler to invoke kmalloc_nolock() and
    kfree_nolock() in the middle of the slab allocator.
    
    Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a92d9a4@kernel.org [1]
    Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b@kernel.org  [2]

> diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
> index fa6d31dbca166..8c2b9911471eb 100644
> --- a/lib/tests/slub_kunit.c
> +++ b/lib/tests/slub_kunit.c

[ ... ]

> +#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP)
> +static int slab_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs)
> +{
> +	struct test_nolock_context *ctx;
> +
> +	ctx = container_of(p, struct test_nolock_context, kprobe);
> +	test_nolock(ctx);

[Severity: Medium]
Will this handler execute unconditionally for any task that hits the kprobe?

The kprobe is registered globally on slab_attach_kprobe_locked(). Since this
function is inserted into slab_lockdep_assert_held(), which is heavily used
in the slab allocator's paths, any task on any CPU performing an allocation
can trigger the kprobe.

This allows unrelated system processes to execute the test logic concurrently.
Does this code need a check to restrict execution to the test thread, such as
verifying current against the test task?

Additionally, this can cause data races in test_nolock() since unrelated
system processes would perform unsynchronized read-modify-write operations on
the stack-allocated ctx variables:

lib/tests/slub_kunit.c:test_nolock() {
    ...
    if (objp)
        ctx->alloc_ok++;
    else
        ctx->alloc_fail++;
    ...
}

> +	return 0;
> +}
> +
> +static bool register_slab_kprobes(struct test_nolock_context *ctx)
> +{
> +	ctx->kprobe.symbol_name = "slab_attach_kprobe_locked";
> +	ctx->kprobe.pre_handler = slab_kprobe_pre_handler;
> +
> +	if (register_kprobe(&ctx->kprobe))
> +		return false;
> +	return true;
> +}

[ ... ]

> +static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
> +{
> +	struct test_nolock_context ctx = { .test = test };
>  
> -cleanup:
> -	perf_event_disable(ctx.event);
> -	perf_event_release_kernel(ctx.event);
> +	if (!register_slab_kprobes(&ctx))
> +		kunit_skip(test, "Failed to register kprobe, skipping");
>  
> -	kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n",
> -		   ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail);
> +	test_kmalloc_kfree();
>  
> -	if (alloc_fail)
> -		kunit_skip(test, "Allocation failed");
> +	unregister_slab_kprobes(&ctx);

[Severity: High]
Can this manual unregistration lead to a stack use-after-free if the KUnit
test aborts before completing?

If the test aborts (for example, due to a KUnit timeout which is possible
given the system-wide contention caused by the global kprobe), the thread
could exit and its stack memory would be freed, bypassing this manual
cleanup.

If the global kprobe remains active and continues to fire for other tasks,
would the pre-handler then dereference the ctx pointer, pointing to freed
stack memory?

Could kunit_add_action() be used to ensure the cleanup is always executed
even if the test aborts?

>  	KUNIT_EXPECT_EQ(test, 0, slab_errors);
>  }
>  #endif

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org?part=1

Re: [PATCH v5 8/8] slub_kunit: extend the test for kfree_rcu_nolock()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing synchronization for deferred RCU frees causes KUnit test suite contamination and false negatives.
--

--- Patch [8]: [PATCH v5 8/8] slub_kunit: extend the test for kfree_rcu_nolock() ---
commit 975207c355ab3b2db3420df72699cb12e4769b5e
Author: Harry Yoo (Oracle) <harry@kernel.org>

slub_kunit: extend the test for kfree_rcu_nolock()
    
When slub_kunit is not built-in, call kfree_rcu() and kfree_rcu_nolock()
to test kfree_rcu_nolock() in slub_kunit.
    
Rename the test case as the test covers more _nolock() APIs.

> diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c
> index 8c2b9911471eb..e3b63f0338d5f 100644
> --- a/lib/tests/slub_kunit.c
> +++ b/lib/tests/slub_kunit.c
[ ... ]
> @@ -397,14 +410,14 @@ static void disable_perf_events(struct test_nolock_context *ctx)
>  	perf_event_release_kernel(ctx->event);
>  }
>  
> -static void test_kmalloc_kfree_nolock_perf(struct kunit *test)
> +static void test_kmalloc_nolock_and_friends_perf(struct kunit *test)
>  {
>  	struct test_nolock_context ctx = { .test = test };
>  
>  	if (!enable_perf_events(&ctx))
>  		kunit_skip(test, "Failed to enable perf event, skipping");
>  
> -	test_kmalloc_kfree();
> +	test_kmalloc_and_friends();
>  
>  	disable_perf_events(&ctx);
>  	KUNIT_EXPECT_EQ(test, 0, slab_errors);

[Severity: Medium]
Will this evaluate slab_errors before all asynchronous frees complete?

Since test_kmalloc_and_friends() queues objects via kfree_rcu() and
kfree_rcu_nolock(), the callbacks could execute in softirq context later.
If they interrupt a subsequent KUnit test, slab_add_kunit_errors() could
attribute any SLUB errors from these deferred frees to the next test, causing
it to falsely fail while causing this test to falsely pass.

Should we add a synchronization call like kvfree_rcu_barrier() before
checking slab_errors?

[ ... ]
> @@ -438,14 +451,14 @@ static void unregister_slab_kprobes(struct test_nolock_context *ctx)
>  	unregister_kprobe(&ctx->kprobe);
>  }
>  
> -static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test)
> +static void test_kmalloc_nolock_and_friends_kprobe(struct kunit *test)
>  {
>  	struct test_nolock_context ctx = { .test = test };
>  
>  	if (!register_slab_kprobes(&ctx))
>  		kunit_skip(test, "Failed to register kprobe, skipping");
>  
> -	test_kmalloc_kfree();
> +	test_kmalloc_and_friends();
>  
>  	unregister_slab_kprobes(&ctx);
>  	KUNIT_EXPECT_EQ(test, 0, slab_errors);

[Severity: Medium]
Does this test case need synchronization for the same reason?

Without a barrier before evaluating slab_errors, deferred RCU frees from
test_kmalloc_and_friends() might leak into the next KUnit test and cause
false test failures.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org?part=8

Re: [PATCH v5 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Exposing local_trylock() to lockdep wait-type checking on PREEMPT_RT triggers an 'Invalid wait context' splat.
--

--- Patch [6]: [PATCH v5 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching ---
commit f07807d65d1f49531b5a23f93da3e184960e8332
Author: Harry Yoo (Oracle) <harry@kernel.org>

mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching

This commit introduces struct kvfree_rcu_head to optimize kvfree_rcu()
batching. It replaces the use of rcu_head with a single-pointer linked
list when batched, reducing overhead.

> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 9c2cca9add893..691ce6db4519c 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
[ ... ]
> @@ -1979,7 +1979,7 @@ void __init kfree_rcu_scheduler_running(void)
>   * be free'd in workqueue context. This allows us to: batch requests together to
>   * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
>   */
> -void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> +void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr)

[Severity: High]
Does this code path expose local_trylock() to lockdep wait-type checking on
PREEMPT_RT, triggering an 'Invalid wait context' splat?

When kvfree_call_rcu() is called under a raw spinlock or in hardirq context,
it can reach __kfree_rcu_sheaf():

mm/slub.c:__kfree_rcu_sheaf() {
	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
		lock_map_acquire_try(&kfree_rcu_sheaf_map);

	if (!local_trylock(&s->cpu_sheaves->lock))
		goto fail;
	...
}

On PREEMPT_RT, local_lock maps to a sleepable spinlock_t (wait type
LD_WAIT_CONFIG). CONFIG_PROVE_RAW_LOCK_NESTING checks declared wait-types
regardless of runtime behavior or trylock usage.

If the patch series removes the short-circuit in __kfree_rcu_sheaf() that
protected this path, does the subsequent local_trylock() execution violate
raw lock nesting rules?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org?part=6

Re: [PATCH v5 7/8] mm/slab: introduce kfree_rcu_nolock()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] `kfree_rcu_nolock()` and `kvfree_call_rcu()` invoke `__kfree_rcu_sheaf()`, which acquires a `local_trylock()`. This triggers lockdep 'Invalid wait context' warnings and can cause deadlocks on PREEMPT_RT.
- [Medium] Missing `kasan_record_aux_stack()` in the `fallback` path of `kfree_call_rcu_nolock()` loses the original free stack trace.
- [Low] Missing struct pointer cast in the `kfree_rcu_nolock()` macro causes incompatible pointer type compilation errors when `CONFIG_KVFREE_RCU_BATCHED` is enabled.
--

--- Patch [7]: [PATCH v5 7/8] mm/slab: introduce kfree_rcu_nolock() ---
commit 34c7828edcaca9e66aa219ee881130dc05e9102d
Author: Harry Yoo (Oracle) <harry@kernel.org>
Subject: mm/slab: introduce kfree_rcu_nolock()

The commit introduces kfree_rcu_nolock() to handle freeing objects in unknown
contexts without deferred processing when possible. It relies on
__kfree_rcu_sheaf() for the fast path and falls back to defer_kfree_rcu()
which uses an irq_work.

> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -1124,6 +1125,27 @@ do {								\
>  		kvfree_call_rcu(NULL, (void *) (___p));		\
>  } while (0)
>  
> +/**
> + * kfree_rcu_nolock() - a version of kfree_rcu() that can be called in any context.
> + * @ptr: pointer to kfree for double-argument invocations.
> + * @kvrhf: the name of the struct kvfree_rcu_head within the type of @ptr.
> + *
> + * With KVFREE_RCU_BATCHED, kfree_rcu_nolock() tries hard to free objects
> + * without any deferred processing, but may still defer freeing.
> + * Large kmalloc and vmalloc objects are always deferred.
> + *
> + * kfree_rcu_nolock() supports 2-arg variant only.
> + */
> +#define kfree_rcu_nolock(ptr, kvrhf)						\
> +do {										\
> +	typeof (ptr) ___p = (ptr);						\
> +										\
> +	if (___p) {								\
> +		BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >= 4096);		\
> +		kfree_call_rcu_nolock(&((___p)->kvrhf), (void *) (___p));	\

[Severity: Low]
Is a cast to (struct kvfree_rcu_head *) needed here? 

Because the caller typically embeds struct rcu_head, passing the member
address directly to kfree_call_rcu_nolock() results in an incompatible pointer
type compilation error when CONFIG_KVFREE_RCU_BATCHED is enabled.

> diff --git a/mm/slab_common.c b/mm/slab_common.c
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -1296,6 +1296,33 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
>  EXPORT_TRACEPOINT_SYMBOL(kfree);
>  EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free);
>  
> +void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr)
> +{
> +	struct slab *slab;
> +
> +	if (!IS_ENABLED(CONFIG_KVFREE_RCU_BATCHED))
> +		goto fallback;
> +
> +	if (unlikely(is_vmalloc_addr(ptr)))
> +		goto fallback;
> +
> +	slab = virt_to_slab(ptr);
> +	if (unlikely(!slab))
> +		goto fallback;
> +
> +	if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != numa_mem_id()))
> +		goto fallback;
> +
> +	if (unlikely(!__kfree_rcu_sheaf(slab->slab_cache, ptr, SLAB_FREE_NOLOCK)))

[Severity: High]
Could calling __kfree_rcu_sheaf() here violate the unknown context
requirement?

When kfree_rcu_nolock() or kvfree_call_rcu() are called in a hardirq or NMI
context, the fastpath unconditionally executes local_trylock() inside
__kfree_rcu_sheaf():

mm/slub.c:__kfree_rcu_sheaf() {
    ...
	if (!local_trylock(&s->cpu_sheaves->lock))
		goto fail;
    ...
}

Because a local_lock on PREEMPT_RT is a sleeping lock, acquiring it under
a raw spinlock triggers lockdep "Invalid wait context" warnings, and if
called from NMI context on PREEMPT_RT, it can unconditionally deadlock.

> +		goto fallback;
> +
> +	return;
> +
> +fallback:
> +	defer_kfree_rcu(head);

[Severity: Medium]
Will this lose the original free stack trace if KASAN is enabled?

When the fastpath fails and execution jumps to fallback, the object is deferred
to an irq_work without calling kasan_record_aux_stack(). Later, when
kvfree_call_rcu() is finally invoked in the irq_work context, it will
record the worker's stack trace instead of the original caller that
requested the free, making KASAN Use-After-Free reports harder to debug.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org?part=7

Re: [PATCH v5 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Calling kvfree_call_rcu() or kfree_rcu() on PREEMPT_RT under a raw spinlock or hardirq context triggers a lockdep 'Invalid wait context' warning and potential deadlocks due to unconditional local_trylock() acquisition.

Pre-existing issues:
- [High] Allocating memory with GFP_NOWAIT inside __kfree_rcu_sheaf() can deadlock on scheduler locks via kswapd wakeup.
--

--- Patch [5]: [PATCH v5 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT ---
commit 2f2c22c6e4b6eb6bae0d210e9892e5c589edbd06
Author: Harry Yoo (Oracle) <harry@kernel.org>

mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT

This patch enables kfree_rcu_sheaf() on PREEMPT_RT by passing
SLAB_FREE_NOLOCK when on PREEMPT_RT, avoiding spinning. It skips
the lockdep map on PREEMPT_RT to avoid suppressing valid warnings.

Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org [1]
Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be@kernel.org [2]

diff --git a/mm/slab_common.c b/mm/slab_common.c
index d81cc2136c688..9c2cca9add893 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
[ ... ]
@@ -1987,7 +1995,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	if (!head)
 		might_sleep();
 
-	if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
+	if (kfree_rcu_sheaf(ptr))
 		return;
 
 	// Queue the object but don't yet schedule the batch.
diff --git a/mm/slub.c b/mm/slub.c
index 92c99ff34a2c9..6c81722afb18c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
[ ... ]
@@ -6100,10 +6101,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
 	struct slab_sheaf *rcu_sheaf;
 	bool allow_spin = free_flags_allow_spinning(free_flags);
 
-	if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT)))
-		return false;
+	VM_WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT) && allow_spin);
 
-	lock_map_acquire_try(&kfree_rcu_sheaf_map);
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		lock_map_acquire_try(&kfree_rcu_sheaf_map);
 
 	if (!local_trylock(&s->cpu_sheaves->lock))

[Severity: High]
Does this unconditionally acquire a sleepable RT-mutex on PREEMPT_RT?

Since this patch removes the !IS_ENABLED(CONFIG_PREEMPT_RT) safeguard in
kvfree_call_rcu(), kfree_rcu_sheaf() can now reach this local_trylock() on
PREEMPT_RT.

kvfree_call_rcu() is frequently called from hardirq or raw spinlock contexts.
On PREEMPT_RT, local_trylock() maps to an RT-mutex. Acquiring it from these
contexts will trigger a lockdep Invalid wait context warning.

Furthermore, if a hardirq interrupts the CPU while it already holds the
RT-mutex's internal raw wait_lock, can this trylock self-deadlock?

[Severity: High]
This is a pre-existing issue, but can the empty sheaf allocation in this
function deadlock on scheduler locks?

Further down in __kfree_rcu_sheaf(), the code does:

    gfp_t gfp = allow_spin ? GFP_NOWAIT : __GFP_NOWARN;
    ...
    empty = alloc_empty_sheaf(s, gfp, alloc_flags);

When allow_spin is true, this uses GFP_NOWAIT, which includes the
__GFP_KSWAPD_RECLAIM bit.

If kvfree_call_rcu() is called from scheduler code holding pi_lock or
rq_lock, and the allocation falls back to the page allocator, it will invoke
wakeup_kswapd(). This enters the scheduler via try_to_wake_up() and
attempts to acquire pi_lock or rq_lock again, causing a self-deadlock.

Should this use a bare flag like __GFP_NOWARN without reclaim bits?

>  		goto fail;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org?part=5

Re: [PATCH v5 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Kmemleak symmetry violation when a sheaf allocated with SLAB_ALLOC_NOLOCK is freed with standard kfree().

Pre-existing issues:
- [High] Lock recursion deadlock via kswapd wakeup from raw spinlock context due to GFP_NOWAIT usage.
--

--- Patch [2]: [PATCH v5 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() ---
commit 10fcb5282968708fbb08270a6618140198eda999
Author: Harry Yoo (Oracle) <harry@kernel.org>
mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()

This patch teaches kfree_rcu_sheaf() how to handle the !allow_spin case by
introducing free_flags. It attempts to get an empty sheaf even when spinning
is not allowed, translating free_flags to alloc_flags during allocation. It
also ensures kfree_rcu_sheaf() fails when the RCU sheaf becomes full in an
unknown context.

Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5@kernel.org

> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 1e1d3feec3535..42db62f3e801a 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -1638,7 +1638,7 @@ static bool kfree_rcu_sheaf(void *obj)
>  
>  	s = slab->slab_cache;
>  	if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) == numa_mem_id()))
> -		return __kfree_rcu_sheaf(s, obj);
> +		return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);

[Severity: High]
This isn't a bug introduced by this patch, but does this unconditional use of
SLAB_FREE_DEFAULT here cause a lock recursion deadlock?

Because SLAB_FREE_DEFAULT evaluates allow_spin to true, __kfree_rcu_sheaf()
will use GFP_NOWAIT. GFP_NOWAIT includes __GFP_KSWAPD_RECLAIM, which can
trigger wakeup_kswapd().

Since kfree_rcu() (via kvfree_call_rcu()) can be called while holding
scheduler internal raw spinlocks like pi_lock, waking kswapd enters
try_to_wake_up() which attempts to acquire pi_lock again, deadlocking the
system.

>  
>  	return false;
>  }
> diff --git a/mm/slub.c b/mm/slub.c
> index d14226ad09822..2c5ba8c326e73 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
[ ... ]
> -static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
> +static void __free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
> +			       unsigned int free_flags)
>  {
[ ... ]
> +	if (unlikely(free_flags & SLAB_FREE_NOLOCK))
> +		kfree_nolock(sheaf);
> +	else
> +		kfree(sheaf);
>  
>  	stat(s, SHEAF_FREE);
>  }
>  
> +static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
> +{
> +	__free_empty_sheaf(s, sheaf, SLAB_FREE_DEFAULT);
> +}

[Severity: High]
Does this wrapper cause a kmemleak symmetry violation?

When __kfree_rcu_sheaf() is called with SLAB_FREE_NOLOCK, it allocates the
sheaf using SLAB_ALLOC_NOLOCK, which skips kmemleak registration.

However, the allocated sheaf struct does not track its allocation flags.
Later, when this sheaf is flushed or processed for freeing (e.g., via
rcu_free_sheaf()), free_empty_sheaf() is called unconditionally.

Because free_empty_sheaf() hardcodes SLAB_FREE_DEFAULT, it will call
kfree() instead of kfree_nolock() on an unregistered memory block. Could
this lead to kmemleak warnings and false positive memory leaks due to
freeing an unknown object?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org?part=2

Re: [PATCH v5 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The modified fallback condition `!allow_spin && irqs_disabled()` is completely dead code because `allow_spin` evaluates to true unconditionally, rendering the intended NMI protection for `call_rcu()` ineffective.
--

--- Patch [3]: [PATCH v5 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled ---
commit f7ce8e162ca9c0e62e67afe017d8855bcf7b7674
Author: Harry Yoo (Oracle) <harry@kernel.org>

mm/slab: use call_rcu() in unknown context if irqs are enabled

This commit updates the fallback condition in __kfree_rcu_sheaf() to check
if IRQs are disabled, allowing direct use of call_rcu() when IRQs are
enabled. It assumes that if IRQs are enabled, re-entrance into call_rcu()
cannot occur.

Link: https://lore.kernel.org/linux-mm/CAADnVQKRVD5ZSnEKbZZU7w86gHbGHUug2pvzpgZTngNS+fg4rw@mail.gmail.com

> diff --git a/mm/slub.c b/mm/slub.c
> index 2c5ba8c326e73..bb0f2975c95b1 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -6157,8 +6157,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags)
>  	if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
>  		rcu_sheaf = NULL;
>  	} else {
> -		if (unlikely(!allow_spin)) {
> -			/* call_rcu() cannot be called in an unknown context */
> +		/*
> +		 * With !allow_spin, we might have interrupted call_rcu()'s
> +		 * IRQ-disabled critical section. If IRQs are not disabled,
> +		 * we know that's not the case.
> +		 */
> +		if (unlikely(!allow_spin && irqs_disabled())) {

[Severity: High]
Does this make the fallback condition completely dead code?

Since __kfree_rcu_sheaf() is only invoked by kfree_rcu_sheaf() in
mm/slab_common.c with the SLAB_FREE_DEFAULT (0) flag:

mm/slab_common.c:kfree_rcu_sheaf() {
    ...
    return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT);
}

The allow_spin variable will evaluate to true unconditionally. This makes
!allow_spin always false, which prevents this fallback path from executing.

If an eBPF tracing program calls kfree_rcu() from an NMI context exactly when
the CPU's RCU sheaf becomes full, could the code now invoke call_rcu() and
corrupt RCU per-CPU data structures?

>  			rcu_sheaf->size--;
>  			local_unlock(&s->cpu_sheaves->lock);
>  			goto fail;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org?part=3

Re: [PATCH v5 0/8] mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage

[Vlastimil Babka (SUSE)]

On 7/29/26 10:20, Harry Yoo (Oracle) wrote:

...

> 8:  8e69c83a1e52 ! 8:  aeaf2ab60046 slub_kunit: extend the test for kfree_rcu_nolock()
>     @@ lib/tests/slub_kunit.c: struct test_nolock_context {
>      -			gfp_t gfp = (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
>      +			gfp_t gfp = (i & 1) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT;
>       
>     - 			objects[j] = kmalloc_obj(*objects[j], gfp);
>     +-			objects[j] = kmalloc(64, gfp);
>     ++			objects[j] = kmalloc_obj(*objects[j], gfp);
>       			if (!objects[j]) {
>     + 				j--;
>     + 				while (j >= 0)
>      @@ lib/tests/slub_kunit.c: static void test_kmalloc_kfree(void)
>       			}
>       		}
> 
> Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a92d9a4@kernel.org [1]
> Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b@kernel.org  [2]
> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
> Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5@kernel.org
> Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
> Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162@kernel.org [1]
> Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be@kernel.org [2]
> Suggested-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
> Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>
> Signed-off-by: Harry Yoo (Oracle) <harry@kernel.org>

Wonder what produced this weird block of assorted trailers? "b4 prep
--compare-to vX" never did that for me?

> ---
> Harry Yoo (Oracle) (8):
>       mm/slab, slub_kunit: register kprobe to trigger _nolock APIs
>       mm/slab: handle the !allow_spin case in kfree_rcu_sheaf()
>       mm/slab: use call_rcu() in unknown context if irqs are enabled
>       mm/slab: extend deferred free mechanism to handle rcu sheaves
>       mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT
>       mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching
>       mm/slab: introduce kfree_rcu_nolock()
>       slub_kunit: extend the test for kfree_rcu_nolock()
> 
>  include/linux/rcupdate.h   |  32 ++++++-
>  include/linux/slab.h       |  16 +---
>  include/linux/types.h      |   8 ++
>  include/trace/events/rcu.h |   2 +-
>  lib/tests/slub_kunit.c     | 190 ++++++++++++++++++++++++++++-----------
>  mm/slab.h                  |  52 ++++++++++-
>  mm/slab_common.c           |  81 +++++++++++++----
>  mm/slub.c                  | 220 ++++++++++++++++++++++++++++-----------------
>  8 files changed, 433 insertions(+), 168 deletions(-)

Applied to slab/for-7.3/kfree_rcu_nolock and merged go slab/for-next, thanks!
Used slab/for-next-fixes as a base for a better topic branch indepdencence.

> ---
> base-commit: ffc64805cebac330b2a60b2f1f144fcc35b8ab46
> change-id: 20260615-kfree_rcu_nolock-e5502555992f
> 
> Best regards,