[PATCH RFC 0/5] allow kfree_nolock() handle kmalloc() objects

[PATCH RFC 0/5] allow kfree_nolock() handle kmalloc() objects

[Vlastimil Babka (SUSE)]

This is an initial RFC for something that could target 7.4 and
definitely not earlier merge window. Based on
slab/for-7.3/kfree_rcu_nolock

Currently kfree_nolock() can only handle objects from kmalloc_nolock().
Removing this restriction means an easier to use API and might be useful
in some situations. It means dealing properly with kfence, kmemleak and
large kmalloc allocations. Some of the cases may end up being deferred
to the irq_wor() but that's not a big issue.

As a example of usefullness, convert a kfree_rcu() to kfree_nolock() in
the scheduler code, where kfree_rcu() has been used as a workaround for
kfree() being unsafe to call under pi_lock.

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
Vlastimil Babka (SUSE) (5):
      mm/slab: cleanup deferred free handling
      mm/slab, kfence: support kfence objects in kfree_nolock()
      mm/slab, kmemleak: handle kmemleak freeing in kfree_nolock()
      mm/slab: handle large_kmalloc objects in kfree_nolock()
      sched: use kfree_nolock() instead of kfree_rcu()

 include/linux/kfence.h   |   5 ++
 include/linux/kmemleak.h |  17 +++++
 kernel/sched/core.c      |   9 +--
 kernel/sched/sched.h     |   7 +--
 mm/kfence/core.c         |  14 +++++
 mm/kfence/kfence.h       |   5 +-
 mm/kmemleak.c            |  42 +++++++++++++
 mm/slub.c                | 157 +++++++++++++++++++++++++++++++++++++++--------
 8 files changed, 215 insertions(+), 41 deletions(-)
---
base-commit: 648294a02bfcd0eddae51877e3b30f8bbb2d4bb6
change-id: 20260730-kfree_nolock_kmalloc-6067170fd5ad

[PATCH RFC 1/5] mm/slab: cleanup deferred free handling

[Vlastimil Babka (SUSE)]

In deferred_percpu_work_fn() we have a bunch of single-use local
variables for the various llists. Remove them and access the lists
directly.

In defer_free() make it more obvious and documented what we are doing.
Also restrict guard(preempt) to only the necessary part.

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
 mm/slub.c | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index b9aeb02a880f..044db93d64a0 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6371,16 +6371,12 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 static void deferred_percpu_work_fn(struct irq_work *work)
 {
 	struct deferred_percpu_work *dpw;
-	struct llist_head *objs, *objs_by_rcu, *rcu_sheaves;
 	struct llist_node *llnode, *pos, *t;
 	struct slab_sheaf *sheaf, *next;
 
 	dpw = container_of(work, struct deferred_percpu_work, work);
-	rcu_sheaves = &dpw->rcu_sheaves;
-	objs = &dpw->objects;
-	objs_by_rcu = &dpw->objects_by_rcu;
 
-	llnode = llist_del_all(objs);
+	llnode = llist_del_all(&dpw->objects);
 	llist_for_each_safe(pos, t, llnode) {
 		struct kmem_cache *s;
 		struct slab *slab;
@@ -6403,7 +6399,7 @@ static void deferred_percpu_work_fn(struct irq_work *work)
 		stat(s, FREE_SLOWPATH);
 	}
 
-	llnode = llist_del_all(objs_by_rcu);
+	llnode = llist_del_all(&dpw->objects_by_rcu);
 	llist_for_each_safe(pos, t, llnode) {
 		void *head = pos;
 		void *objp = kvmalloc_obj_start_addr(head);
@@ -6411,21 +6407,27 @@ static void deferred_percpu_work_fn(struct irq_work *work)
 		kvfree_call_rcu(head, objp);
 	}
 
-	llnode = llist_del_all(rcu_sheaves);
+	llnode = llist_del_all(&dpw->rcu_sheaves);
 	llist_for_each_entry_safe(sheaf, next, llnode, llnode)
 		call_rcu(&sheaf->rcu_head, rcu_free_sheaf);
 }
 
-static void defer_free(struct kmem_cache *s, void *head)
+static void defer_free(struct kmem_cache *s, void *obj)
 {
 	struct deferred_percpu_work *dpw;
+	struct llist_node *llnode;
 
-	guard(preempt)();
+	/*
+	 * Place the llist node where the freepointer would be if we freed the
+	 * object immediately. That means we can write there safely, only need
+	 * to remove kasan tag first.
+	 */
+	llnode = kasan_reset_tag(obj) + s->offset;
 
-	head = kasan_reset_tag(head);
+	guard(preempt)();
 
 	dpw = this_cpu_ptr(&deferred_percpu_work);
-	if (llist_add(head + s->offset, &dpw->objects))
+	if (llist_add(llnode, &dpw->objects))
 		irq_work_queue(&dpw->work);
 }
 

-- 
2.55.0

[PATCH RFC 2/5] mm/slab, kfence: support kfence objects in kfree_nolock()

[Vlastimil Babka (SUSE)]

KFENCE objects are one of the reasons why kfree_nolock() cannot
currently handle kmalloc() objects. They are however rare so we can
simply defer their freeing to irq_work.

The only complication is where to put the llist node. We cannot use the
freepointer location like in defer_free() because for some caches it may
be outside the object area and KFENCE would detect writes there.

Since KFENCE already solves a similar situation when freeing objects
from SLAB_TYPESAFE_BY_RCU caches with an rcu_head in its internal
metadata, reuse that rcu_head also for the llist node. Introduce
kfence_obj_to_llnode() and kfence_llnode_to_obj() so SLAB can work with
this llist node without being exposed to KFENCE internals.

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
 include/linux/kfence.h |  5 +++++
 mm/kfence/core.c       | 14 ++++++++++++++
 mm/kfence/kfence.h     |  5 ++++-
 mm/slub.c              | 39 ++++++++++++++++++++++++++++++++++++---
 4 files changed, 59 insertions(+), 4 deletions(-)

diff --git a/include/linux/kfence.h b/include/linux/kfence.h
index e5822f6e7f27..00721c85258d 100644
--- a/include/linux/kfence.h
+++ b/include/linux/kfence.h
@@ -188,6 +188,9 @@ static __always_inline __must_check bool kfence_free(void *addr)
 	return true;
 }
 
+struct llist_node *kfence_obj_to_llnode(void *addr);
+void *kfence_llnode_to_obj(struct llist_node *llnode);
+
 /**
  * kfence_handle_page_fault() - perform page fault handling for KFENCE pages
  * @addr: faulting address
@@ -235,6 +238,8 @@ static inline size_t kfence_ksize(const void *addr) { return 0; }
 static inline void *kfence_object_start(const void *addr) { return NULL; }
 static inline void __kfence_free(void *addr) { }
 static inline bool __must_check kfence_free(void *addr) { return false; }
+static inline struct llist_node *kfence_obj_to_llnode(void *addr) { return NULL; }
+static inline void *kfence_llnode_to_obj(struct llist_node *llnode) { return NULL; }
 static inline bool __must_check kfence_handle_page_fault(unsigned long addr, bool is_write,
 							 struct pt_regs *regs)
 {
diff --git a/mm/kfence/core.c b/mm/kfence/core.c
index 6577bd76954e..42519d24687f 100644
--- a/mm/kfence/core.c
+++ b/mm/kfence/core.c
@@ -1271,6 +1271,20 @@ void __kfence_free(void *addr)
 	}
 }
 
+struct llist_node *kfence_obj_to_llnode(void *addr)
+{
+	struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
+
+	return &meta->llnode;
+}
+
+void *kfence_llnode_to_obj(struct llist_node *llnode)
+{
+	struct kfence_metadata *meta = container_of(llnode, struct kfence_metadata, llnode);
+
+	return (void *)meta->addr;
+}
+
 bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs *regs)
 {
 	const int page_index = (addr - (unsigned long)__kfence_pool) / PAGE_SIZE;
diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h
index 1f618f9b0d12..0fca1dc2c794 100644
--- a/mm/kfence/kfence.h
+++ b/mm/kfence/kfence.h
@@ -58,7 +58,10 @@ struct kfence_track {
 /* KFENCE metadata per guarded allocation. */
 struct kfence_metadata {
 	struct list_head list __guarded_by(&kfence_freelist_lock);	/* Freelist node. */
-	struct rcu_head rcu_head;	/* For delayed freeing. */
+	union {
+		struct rcu_head rcu_head;	/* For delayed freeing. */
+		struct llist_node llnode;	/* For kfree_nolock(). */
+	};
 
 	/*
 	 * Lock protecting below data; to ensure consistency of the below data,
diff --git a/mm/slub.c b/mm/slub.c
index 044db93d64a0..2d7648b96bfa 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4050,6 +4050,7 @@ static void flush_all(struct kmem_cache *s)
 
 struct deferred_percpu_work {
 	struct llist_head objects;
+	struct llist_head objects_kfence;
 	struct llist_head objects_by_rcu;
 	struct llist_head rcu_sheaves;
 	struct irq_work work;
@@ -4059,6 +4060,7 @@ static void deferred_percpu_work_fn(struct irq_work *work);
 
 static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = {
 	.objects = LLIST_HEAD_INIT(objects),
+	.objects_kfence = LLIST_HEAD_INIT(objects_kfence),
 	.objects_by_rcu = LLIST_HEAD_INIT(objects_by_rcu),
 	.rcu_sheaves = LLIST_HEAD_INIT(rcu_sheaves),
 	.work = IRQ_WORK_INIT(deferred_percpu_work_fn),
@@ -6399,6 +6401,13 @@ static void deferred_percpu_work_fn(struct irq_work *work)
 		stat(s, FREE_SLOWPATH);
 	}
 
+	llnode = llist_del_all(&dpw->objects_kfence);
+	llist_for_each_safe(pos, t, llnode) {
+		void *obj = kfence_llnode_to_obj(pos);
+
+		__kfence_free(obj);
+	}
+
 	llnode = llist_del_all(&dpw->objects_by_rcu);
 	llist_for_each_safe(pos, t, llnode) {
 		void *head = pos;
@@ -6431,6 +6440,21 @@ static void defer_free(struct kmem_cache *s, void *obj)
 		irq_work_queue(&dpw->work);
 }
 
+static void defer_free_kfence(void *obj)
+{
+	struct deferred_percpu_work *dpw;
+	struct llist_node *llnode;
+
+	/* kasan_reset_tag() is not necessary, kfence objects are not tagged */
+	llnode = kfence_obj_to_llnode(obj);
+
+	guard(preempt)();
+
+	dpw = this_cpu_ptr(&deferred_percpu_work);
+	if (llist_add(llnode, &dpw->objects_kfence))
+		irq_work_queue(&dpw->work);
+}
+
 void defer_kfree_rcu(struct kvfree_rcu_head *head)
 {
 	struct deferred_percpu_work *dpw;
@@ -6758,10 +6782,13 @@ EXPORT_SYMBOL(kfree);
 /*
  * Can be called while holding raw_spinlock_t or from IRQ and NMI,
  * but ONLY for objects allocated by kmalloc_nolock().
- * Debug checks (like kmemleak and kfence) were skipped on allocation,
- * hence
+ *
+ * In case kmemleak is enabled,
+ *
  * obj = kmalloc(); kfree_nolock(obj);
- * will miss kmemleak/kfence book keeping and will cause false positives.
+ *
+ * will miss kmemleak book keeping and will cause false positives.
+ *
  * large_kmalloc is not supported either.
  */
 void kfree_nolock(const void *object)
@@ -6793,6 +6820,12 @@ void kfree_nolock(const void *object)
 	 * since they take spinlocks or not safe from any context.
 	 */
 	kmsan_slab_free(s, x);
+
+	if (is_kfence_address(x)) {
+		defer_free_kfence(x);
+		return;
+	}
+
 	/*
 	 * If KASAN finds a kernel bug it will do kasan_report_invalid_free()
 	 * which will call raw_spin_lock_irqsave() which is technically

-- 
2.55.0

[PATCH RFC 3/5] mm/slab, kmemleak: handle kmemleak freeing in kfree_nolock()

[Vlastimil Babka (SUSE)]

Kmemleak handling is one of the reasons why kfree_nolock() cannot
currently handle kmalloc() objects, because calling kmemleak_free()
would involve spinning on its internal raw spinlocks.

Kmemleak is a debugging mechanism so we could simply defer all
kfree_nolock() to irq_work if it's enabled, and eat the extra cost. But
that would be unnecessary pessimistic. We expect kfree_nolock() will be
still mostly called on objects from kmalloc_nolock() that are not
registered in kmemleak so they still don't need any deferred freeing.

Thus introduce kmemleak_may_need_free() that can check if the object is
registered. This is done using __lookup_object() performed under a
raw_spin_trylock_irqsave(), which is safe to attempt from kfree_nolock()
(except from a NMI on a !CONFIG_SMP system). When that trylock fails or
can't be attempted, we however must assume the object might be
registered, and defer the freeing.

The ordering of kmsan/kasan handling and kmemleak is also different from
what kfree() is doing, but as explained in the comment, it should be OK.

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
 include/linux/kmemleak.h | 17 +++++++++++++++++
 mm/kmemleak.c            | 42 ++++++++++++++++++++++++++++++++++++++++++
 mm/slub.c                | 34 ++++++++++++++++++++++++++--------
 3 files changed, 85 insertions(+), 8 deletions(-)

diff --git a/include/linux/kmemleak.h b/include/linux/kmemleak.h
index fbd424b2abb1..52f75f10a9ce 100644
--- a/include/linux/kmemleak.h
+++ b/include/linux/kmemleak.h
@@ -22,6 +22,7 @@ extern void kmemleak_alloc_percpu(const void __percpu *ptr, size_t size,
 extern void kmemleak_vmalloc(const struct vm_struct *area, size_t size,
 			     gfp_t gfp) __ref;
 extern void kmemleak_free(const void *ptr) __ref;
+bool kmemleak_may_need_free(const void *ptr) __ref;
 extern void kmemleak_free_part(const void *ptr, size_t size) __ref;
 extern void kmemleak_free_percpu(const void __percpu *ptr) __ref;
 extern void kmemleak_update_trace(const void *ptr) __ref;
@@ -50,6 +51,14 @@ static inline void kmemleak_free_recursive(const void *ptr, slab_flags_t flags)
 		kmemleak_free(ptr);
 }
 
+static inline bool kmemleak_may_need_free_recursive(const void *ptr, slab_flags_t flags)
+{
+	if (!(flags & SLAB_NOLEAKTRACE))
+		return kmemleak_may_need_free(ptr);
+
+	return false;
+}
+
 static inline void kmemleak_erase(void **ptr)
 {
 	*ptr = NULL;
@@ -86,6 +95,14 @@ static inline void kmemleak_free_part(const void *ptr, size_t size)
 static inline void kmemleak_free_recursive(const void *ptr, slab_flags_t flags)
 {
 }
+static inline bool kmemleak_may_need_free(const void *ptr)
+{
+	return false;
+}
+static inline bool kmemleak_may_need_free_recursive(const void *ptr, slab_flags_t flags)
+{
+	return false;
+}
 static inline void kmemleak_free_percpu(const void __percpu *ptr)
 {
 }
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 7c7ba17ce7af..e3560ce82632 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -1168,6 +1168,48 @@ void __ref kmemleak_free(const void *ptr)
 }
 EXPORT_SYMBOL_GPL(kmemleak_free);
 
+/**
+ * kmemleak_may_need_free - check if object is registered
+ * @ptr:	pointer to beginning of the object
+ *
+ * This function is called from the kernel allocator when an object should be
+ * freed but the caller context might be unsafe to spin on the internal locks.
+ *
+ * It will therefore only use trylock and thus might return a false positive
+ * if the trylock fails and the status cannot be determined.
+ *
+ * For objects that (might) need free, the allocator has to defer the actual
+ * freeing to a safe context.
+ *
+ * The assumption is that most objects freed from the unsafe context are also
+ * allocated in such context and thus are not registered in kmemleak, so it's
+ * unlikely the defered freeing will be necessary just because kmemleak is
+ * enabled.
+ */
+bool __ref kmemleak_may_need_free(const void *ptr)
+{
+	unsigned long flags;
+	struct kmemleak_object *object;
+
+	pr_debug("%s(0x%px)\n", __func__, ptr);
+
+	if (!kmemleak_free_enabled || !ptr || IS_ERR(ptr))
+		return false;
+
+	/* On UP, raw_spin_trylock() always succeeds even when it is locked */
+	if (!IS_ENABLED(CONFIG_SMP) && in_nmi())
+		return true;
+
+	if (!raw_spin_trylock_irqsave(&kmemleak_lock, flags))
+		return true;
+
+	object = __lookup_object((unsigned long)ptr, 0, 0);
+
+	raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
+
+	return !!object;
+}
+
 /**
  * kmemleak_free_part - partially unregister a previously registered object
  * @ptr:	pointer to the beginning or inside the object. This also
diff --git a/mm/slub.c b/mm/slub.c
index 2d7648b96bfa..423b5bdb910b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -6390,6 +6390,8 @@ static void deferred_percpu_work_fn(struct irq_work *work)
 		/* Point 'x' back to the beginning of allocated object */
 		x -= s->offset;
 
+		kmemleak_free_recursive(x, s->flags);
+
 		/*
 		 * We used freepointer in 'x' to link 'x' into df->objects.
 		 * Clear it to NULL to avoid false positive detection
@@ -6403,8 +6405,15 @@ static void deferred_percpu_work_fn(struct irq_work *work)
 
 	llnode = llist_del_all(&dpw->objects_kfence);
 	llist_for_each_safe(pos, t, llnode) {
+		struct kmem_cache *s;
+		struct slab *slab;
 		void *obj = kfence_llnode_to_obj(pos);
 
+		slab = virt_to_slab(obj);
+		s = slab->slab_cache;
+
+		kmemleak_free_recursive(obj, s->flags);
+
 		__kfence_free(obj);
 	}
 
@@ -6781,15 +6790,10 @@ EXPORT_SYMBOL(kfree);
 
 /*
  * Can be called while holding raw_spinlock_t or from IRQ and NMI,
- * but ONLY for objects allocated by kmalloc_nolock().
- *
- * In case kmemleak is enabled,
+ * but may defer freeing to irq_work() in some cases.
  *
- * obj = kmalloc(); kfree_nolock(obj);
- *
- * will miss kmemleak book keeping and will cause false positives.
- *
- * large_kmalloc is not supported either.
+ * Intended mainly for objects allocated from kmalloc_nolock(), but can handle
+ * also kmem_cache_alloc() and kmalloc() objects, except large_kmalloc.
  */
 void kfree_nolock(const void *object)
 {
@@ -6844,9 +6848,23 @@ void kfree_nolock(const void *object)
 	 */
 	kasan_slab_free(s, x, false, false, /* skip quarantine */true);
 
+	/*
+	 * with kfree() the kmemleak handling happens much sooner, but for
+	 * defering we need to write llnode to the object's freepointer so
+	 * we should have it in the state when it's no longer treated as
+	 * allocated by kasan etc.
+	 *
+	 * defer_free will also reset the pointer tag, but it's ok to do a
+	 * deferred kmemleak_free() using the untagged pointer, because
+	 * __lookup_object() resets the tag anyway
+	 */
+	if (unlikely(kmemleak_may_need_free_recursive(x, s->flags)))
+		goto defer;
+
 	if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, x, false)))
 		return;
 
+defer:
 	/*
 	 * __slab_free() can locklessly cmpxchg16 into a slab, but then it might
 	 * need to take spin_lock for further processing.

-- 
2.55.0

[PATCH RFC 4/5] mm/slab: handle large_kmalloc objects in kfree_nolock()

[Vlastimil Babka (SUSE)]

Large kmalloc objects is the only remaining case that kfree_nolock()
cannot handle from kmalloc() allocations. Note kmalloc_nolock() does not
return large kmalloc objects.

Supporting them is however mostly straigtforward. free_large_kmalloc()
calls kmsan and kasan hooks that should be safe and similar to those
called in kfree_nolock(). Freeing the pages can be handled by
free_frozen_pages_nolock().

The only obstacle is kmemleak_free(), which we can solve by deferring
when necessary, the same way as done for small kmalloc objects.

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
 mm/slub.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 62 insertions(+), 12 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 423b5bdb910b..3be98faa9f0d 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4051,6 +4051,7 @@ static void flush_all(struct kmem_cache *s)
 struct deferred_percpu_work {
 	struct llist_head objects;
 	struct llist_head objects_kfence;
+	struct llist_head objects_large_kmalloc;
 	struct llist_head objects_by_rcu;
 	struct llist_head rcu_sheaves;
 	struct irq_work work;
@@ -4061,6 +4062,7 @@ static void deferred_percpu_work_fn(struct irq_work *work);
 static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = {
 	.objects = LLIST_HEAD_INIT(objects),
 	.objects_kfence = LLIST_HEAD_INIT(objects_kfence),
+	.objects_large_kmalloc = LLIST_HEAD_INIT(objects_large_kmalloc),
 	.objects_by_rcu = LLIST_HEAD_INIT(objects_by_rcu),
 	.rcu_sheaves = LLIST_HEAD_INIT(rcu_sheaves),
 	.work = IRQ_WORK_INIT(deferred_percpu_work_fn),
@@ -6365,6 +6367,21 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 	}
 }
 
+static inline void
+__free_large_kmalloc_page(struct page *page, unsigned int free_flags)
+{
+	unsigned int order = compound_order(page);
+
+	mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B,
+			      -(PAGE_SIZE << order));
+	__ClearPageLargeKmalloc(page);
+
+	if (free_flags & SLAB_FREE_NOLOCK)
+		free_frozen_pages_nolock(page, order);
+	else
+		free_frozen_pages(page, order);
+}
+
 /*
  * In PREEMPT_RT irq_work runs in per-cpu kthread, so it's safe
  * to take sleeping spin_locks from __slab_free().
@@ -6417,6 +6434,15 @@ static void deferred_percpu_work_fn(struct irq_work *work)
 		__kfence_free(obj);
 	}
 
+	llnode = llist_del_all(&dpw->objects_large_kmalloc);
+	llist_for_each_safe(pos, t, llnode) {
+		struct page *page = virt_to_page(pos);
+
+		kmemleak_free(pos);
+
+		__free_large_kmalloc_page(page, SLAB_FREE_DEFAULT);
+	}
+
 	llnode = llist_del_all(&dpw->objects_by_rcu);
 	llist_for_each_safe(pos, t, llnode) {
 		void *head = pos;
@@ -6464,6 +6490,24 @@ static void defer_free_kfence(void *obj)
 		irq_work_queue(&dpw->work);
 }
 
+static void defer_free_large_kmalloc(void *obj)
+{
+	struct deferred_percpu_work *dpw;
+	struct llist_node *llnode;
+
+	/*
+	 * we can simply use the first word of the large kmalloc object
+	 * for the llnode, as there's no ctor or TYPESAFE_BY_RCU
+	 */
+	llnode = kasan_reset_tag(obj);
+
+	guard(preempt)();
+
+	dpw = this_cpu_ptr(&deferred_percpu_work);
+	if (llist_add(llnode, &dpw->objects_large_kmalloc))
+		irq_work_queue(&dpw->work);
+}
+
 void defer_kfree_rcu(struct kvfree_rcu_head *head)
 {
 	struct deferred_percpu_work *dpw;
@@ -6712,9 +6756,11 @@ size_t ksize(const void *objp)
 }
 EXPORT_SYMBOL(ksize);
 
-static void free_large_kmalloc(struct page *page, void *object)
+static void free_large_kmalloc(struct page *page, void *object,
+			       unsigned int free_flags)
 {
 	unsigned int order = compound_order(page);
+	bool nolock = free_flags & SLAB_FREE_NOLOCK;
 
 	if (WARN_ON_ONCE(!PageLargeKmalloc(page))) {
 		dump_page(page, "Not a kmalloc allocation");
@@ -6724,14 +6770,16 @@ static void free_large_kmalloc(struct page *page, void *object)
 	if (WARN_ON_ONCE(order == 0))
 		pr_warn_once("object pointer: 0x%p\n", object);
 
-	kmemleak_free(object);
+	if (!nolock)
+		kmemleak_free(object);
+
 	kasan_kfree_large(object);
 	kmsan_kfree_large(object);
 
-	mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B,
-			      -(PAGE_SIZE << order));
-	__ClearPageLargeKmalloc(page);
-	free_frozen_pages(page, order);
+	if (unlikely(nolock && kmemleak_may_need_free(object)))
+		defer_free_large_kmalloc(object);
+	else
+		__free_large_kmalloc_page(page, free_flags);
 }
 
 /*
@@ -6753,7 +6801,7 @@ void kvfree_rcu_cb(struct rcu_head *head)
 		if (slab)
 			slab_free(slab->slab_cache, slab, obj, _RET_IP_);
 		else
-			free_large_kmalloc(page, obj);
+			free_large_kmalloc(page, obj, SLAB_FREE_DEFAULT);
 	}
 }
 
@@ -6779,7 +6827,7 @@ void kfree(const void *object)
 	slab = page_slab(page);
 	if (!slab) {
 		/* kmalloc_nolock() doesn't support large kmalloc */
-		free_large_kmalloc(page, (void *)object);
+		free_large_kmalloc(page, (void *)object, SLAB_FREE_DEFAULT);
 		return;
 	}
 
@@ -6793,10 +6841,11 @@ EXPORT_SYMBOL(kfree);
  * but may defer freeing to irq_work() in some cases.
  *
  * Intended mainly for objects allocated from kmalloc_nolock(), but can handle
- * also kmem_cache_alloc() and kmalloc() objects, except large_kmalloc.
+ * also kmem_cache_alloc() and kmalloc() objects, including large_kmalloc.
  */
 void kfree_nolock(const void *object)
 {
+	struct page *page;
 	struct slab *slab;
 	struct kmem_cache *s;
 	void *x = (void *)object;
@@ -6804,9 +6853,10 @@ void kfree_nolock(const void *object)
 	if (unlikely(ZERO_OR_NULL_PTR(object)))
 		return;
 
-	slab = virt_to_slab(object);
+	page = virt_to_page(object);
+	slab = page_slab(page);
 	if (unlikely(!slab)) {
-		WARN_ONCE(1, "large_kmalloc is not supported by kfree_nolock()");
+		free_large_kmalloc(page, (void *)object, SLAB_FREE_NOLOCK);
 		return;
 	}
 
@@ -7167,7 +7217,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,
 	if (!s) {
 		/* Handle kalloc'ed objects */
 		if (!slab) {
-			free_large_kmalloc(page, object);
+			free_large_kmalloc(page, object, SLAB_FREE_DEFAULT);
 			df->slab = NULL;
 			return size;
 		}

-- 
2.55.0

[PATCH RFC 5/5] sched: use kfree_nolock() instead of kfree_rcu()

[Vlastimil Babka (SUSE)]

In set_cpus_allowed_force() we use kfree_rcu() because kfree() is unsafe
under p->pi_lock.

With kfree_nolock() now being able to free arbitrary kmalloc() objects,
we can switch to kfree_nolock() and avoid the unnecessary rcu grace
period delay. Only in some cases the freeing might be deferred to
irq_work().

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
 kernel/sched/core.c  | 9 ++-------
 kernel/sched/sched.h | 7 +------
 2 files changed, 3 insertions(+), 13 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 96226707c2f6..d2929e4e23f1 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2807,20 +2807,15 @@ void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *new_mas
 		.user_mask = NULL,
 		.flags     = SCA_USER,	/* clear the user requested mask */
 	};
-	union cpumask_rcuhead {
-		cpumask_t cpumask;
-		struct rcu_head rcu;
-	};
 
 	scoped_guard (__task_rq_lock, p)
 		do_set_cpus_allowed(p, &ac);
 
 	/*
 	 * Because this is called with p->pi_lock held, it is not possible
-	 * to use kfree() here (when PREEMPT_RT=y), therefore punt to using
-	 * kfree_rcu().
+	 * to use kfree() here (when PREEMPT_RT=y), thus use kfree_nolock()
 	 */
-	kfree_rcu((union cpumask_rcuhead *)ac.user_mask, rcu);
+	kfree_nolock(ac.user_mask);
 }
 
 int dup_user_cpus_ptr(struct task_struct *dst, struct task_struct *src,
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 56acf502ba26..6a8d0578e963 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2886,12 +2886,7 @@ static inline bool task_allowed_on_cpu(struct task_struct *p, int cpu)
 
 static inline cpumask_t *alloc_user_cpus_ptr(int node)
 {
-	/*
-	 * See set_cpus_allowed_force() above for the rcu_head usage.
-	 */
-	int size = max_t(int, cpumask_size(), sizeof(struct rcu_head));
-
-	return kmalloc_node(size, GFP_KERNEL, node);
+	return kmalloc_node(cpumask_size(), GFP_KERNEL, node);
 }
 
 static inline struct task_struct *get_push_task(struct rq *rq)

-- 
2.55.0

Re: [PATCH RFC 2/5] mm/slab, kfence: support kfence objects in kfree_nolock()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Double-freeing a KFENCE object via kfree_nolock() causes critical list corruption (llist cycle or rcu_head overwrite), turning a detectable bug into an unrecoverable crash.
- [Medium] kfree_nolock() silently bypasses KFENCE's invalid free detection for offset pointers, stripping the invalid offsets during deferral.

Pre-existing issues:
- [High] KFENCE fails to safely catch double-frees for SLAB_TYPESAFE_BY_RCU caches, corrupting RCU and freelists.
--

--- Patch [2]: [PATCH RFC 2/5] mm/slab, kfence: support kfence objects in kfree_nolock() ---
commit c37e10ff24a4e2a898c52d16f3e89a13a643b486
Author: Vlastimil Babka (SUSE) <vbabka@kernel.org>

mm/slab, kfence: support kfence objects in kfree_nolock()

This commit adds support for freeing KFENCE objects via kfree_nolock(). It
defers the freeing to an irq_work context using an llist node, reusing the
rcu_head space in kfence_metadata.

> diff --git a/mm/kfence/core.c b/mm/kfence/core.c
> index 6577bd76954eb..42519d24687ff 100644
> --- a/mm/kfence/core.c
> +++ b/mm/kfence/core.c
> @@ -1271,6 +1271,20 @@ void __kfence_free(void *addr)

[Severity: High]
This is a pre-existing issue, but does __kfence_free() safely handle double
frees for SLAB_TYPESAFE_BY_RCU caches?

It appears that if an already-freed object is passed, it blindly calls
metadata_update_state() and call_rcu(&meta->rcu_head) without checking if
the object is already freed.

If the object is already in the RCU queue, this executes call_rcu() twice on
the same rcu_head, corrupting the RCU list. If it was already fully freed,
the state is forced back to RCU_FREEING, causing the subsequent callback to
bypass the double-free check and corrupt the freelist.

>  	}
>  }
>  
> +struct llist_node *kfence_obj_to_llnode(void *addr)
> +{
> +	struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr);
> +
> +	return &meta->llnode;
> +}
> +
> +void *kfence_llnode_to_obj(struct llist_node *llnode)
> +{
> +	struct kfence_metadata *meta = container_of(llnode, struct kfence_metadata, llnode);
> +
> +	return (void *)meta->addr;
> +}

[Severity: Medium]
Does this sequence bypass KFENCE's invalid free detection?

When kfree_nolock(obj) receives an invalid KFENCE address, such as an offset
pointer or redzone address, defer_free_kfence() calls kfence_obj_to_llnode()
which drops the original invalid offset.

Later, kfence_llnode_to_obj() reconstructs the pointer strictly as meta->addr,
which is the valid base address.

This passes the reconstructed valid address to __kfence_free(), masking the
original invalid pointer and preventing KFENCE from reporting the bug.

[ ... ]

> diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h
> index 1f618f9b0d12d..0fca1dc2c7941 100644
> --- a/mm/kfence/kfence.h
> +++ b/mm/kfence/kfence.h
> @@ -58,7 +58,10 @@ struct kfence_track {
>  /* KFENCE metadata per guarded allocation. */
>  struct kfence_metadata {
>  	struct list_head list __guarded_by(&kfence_freelist_lock);	/* Freelist node. */
> -	struct rcu_head rcu_head;	/* For delayed freeing. */
> +	union {
> +		struct rcu_head rcu_head;	/* For delayed freeing. */
> +		struct llist_node llnode;	/* For kfree_nolock(). */
> +	};

[Severity: High]
Can aliasing rcu_head and llnode risk list corruption on double frees?

If a KFENCE object is double-freed via kfree_nolock() before the pending
irq_work runs, llist_add() is called twice on the same llnode. This would
create a circular linked list and an infinite loop in IRQ context.

Additionally, for SLAB_TYPESAFE_BY_RCU caches, a double free via
kfree_nolock() during the RCU grace period will locklessly overwrite
meta->rcu_head.next via llist_add(), corrupting the active RCU callback list.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260807-kfree_nolock_kmalloc-v1-0-ba993cbf7a60@kernel.org?part=2

Re: [PATCH RFC 4/5] mm/slab: handle large_kmalloc objects in kfree_nolock()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Synchronous execution of __free_large_kmalloc_page in kfree_nolock() violates NMI wait-context and corrupts vmstat counters
- [High] Deferred clearing of PageLargeKmalloc bypasses double-free protection, causing infinite loops
--

--- Patch [4]: [PATCH RFC 4/5] mm/slab: handle large_kmalloc objects in kfree_nolock() ---
commit 9af1d6506b418b2d9de0922f293e12ee55babdd3
Author: Vlastimil Babka (SUSE) <vbabka@kernel.org>

mm/slab: handle large_kmalloc objects in kfree_nolock()
    
Large kmalloc objects is the only remaining case that kfree_nolock()
cannot handle from kmalloc() allocations. Note kmalloc_nolock() does not
return large kmalloc objects.

> diff --git a/mm/slub.c b/mm/slub.c
> index 423b5bdb910bd..3be98faa9f0de 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c

[ ... ]

> @@ -6365,6 +6367,21 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
>  	}
>  }
>  
> +static inline void
> +__free_large_kmalloc_page(struct page *page, unsigned int free_flags)
> +{
> +	unsigned int order = compound_order(page);
> +
> +	mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B,
> +			      -(PAGE_SIZE << order));
> +	__ClearPageLargeKmalloc(page);
> +
> +	if (free_flags & SLAB_FREE_NOLOCK)
> +		free_frozen_pages_nolock(page, order);

[Severity: High]
Is it safe to execute this synchronously in NMI context on PREEMPT_RT kernels?

When kmemleak is disabled, kmemleak_may_need_free() returns false, which
causes free_large_kmalloc() to call __free_large_kmalloc_page() synchronously
from kfree_nolock(), even in NMI context.

This results in free_frozen_pages_nolock() attempting to acquire the zone
lock, which is an rt_mutex (sleeping lock) on PREEMPT_RT, leading to a panic
when called from NMI.

Additionally, mod_lruvec_page_state() uses a non-atomic read-modify-write
protected by local_irq_save() on architectures without
CONFIG_HAVE_CMPXCHG_LOCAL. This fails to mask NMIs and can silently corrupt
vmstat counters.

Would it be better to always defer freeing large kmalloc pages in
kfree_nolock() to avoid these issues?

> +	else
> +		free_frozen_pages(page, order);
> +}
> +

[ ... ]

> @@ -6712,9 +6756,11 @@ size_t ksize(const void *objp)
>  }
>  EXPORT_SYMBOL(ksize);
>  
> -static void free_large_kmalloc(struct page *page, void *object)
> +static void free_large_kmalloc(struct page *page, void *object,
> +			       unsigned int free_flags)
>  {
>  	unsigned int order = compound_order(page);
> +	bool nolock = free_flags & SLAB_FREE_NOLOCK;
>  
>  	if (WARN_ON_ONCE(!PageLargeKmalloc(page))) {
>  		dump_page(page, "Not a kmalloc allocation");
> @@ -6724,14 +6770,16 @@ static void free_large_kmalloc(struct page *page, void *object)
>  	if (WARN_ON_ONCE(order == 0))
>  		pr_warn_once("object pointer: 0x%p\n", object);
>  
> -	kmemleak_free(object);
> +	if (!nolock)
> +		kmemleak_free(object);
> +
>  	kasan_kfree_large(object);
>  	kmsan_kfree_large(object);
>  
> -	mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B,
> -			      -(PAGE_SIZE << order));
> -	__ClearPageLargeKmalloc(page);
> -	free_frozen_pages(page, order);
> +	if (unlikely(nolock && kmemleak_may_need_free(object)))
> +		defer_free_large_kmalloc(object);

[Severity: High]
Could moving the __ClearPageLargeKmalloc() call to the asynchronous
__free_large_kmalloc_page() bypass double-free protection?

When a free is deferred via defer_free_large_kmalloc(), the PageLargeKmalloc
flag remains set until the IRQ work executes. If the caller has a double-free
bug and kfree_nolock() is called again on the same object during this window,
the WARN_ON_ONCE(!PageLargeKmalloc(page)) check at the start of
free_large_kmalloc() will pass.

This allows the second free to proceed and call defer_free_large_kmalloc()
again. Since this uses llist_add() with the same llnode, it creates a circular
linked list (llnode->next = llnode). When deferred_percpu_work_fn() later
processes this list with llist_for_each_safe(), it will loop infinitely and
cause a hard lockup.

> +	else
> +		__free_large_kmalloc_page(page, free_flags);
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260807-kfree_nolock_kmalloc-v1-0-ba993cbf7a60@kernel.org?part=4

Re: [PATCH RFC 5/5] sched: use kfree_nolock() instead of kfree_rcu()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Replacing `kfree_rcu()` with `kfree_nolock()` introduces an 'Invalid wait context' lockdep warning and RT deadlock risk.

Pre-existing issues:
- [Critical] Pre-existing Use-After-Free (UAF) due to lockless access of `user_cpus_ptr` in `relax_compatible_cpus_allowed_ptr()` racing with concurrent `sched_setaffinity()`.
--

--- Patch [5]: [PATCH RFC 5/5] sched: use kfree_nolock() instead of kfree_rcu() ---
commit 59f1ba4e68974041e778f52147c67a01b2854ec6
Author: Vlastimil Babka (SUSE) <vbabka@kernel.org>

sched: use kfree_nolock() instead of kfree_rcu()

This commit replaces kfree_rcu() with kfree_nolock() in set_cpus_allowed_force()
to avoid unnecessary RCU grace period delays. It also updates
alloc_user_cpus_ptr() to no longer allocate extra space for the RCU head.

> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 96226707c2f61..d2929e4e23f13 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -2807,20 +2807,15 @@ void set_cpus_allowed_force(struct task_struct *p, const struct cpumask *new_mas
>  		.user_mask = NULL,
>  		.flags     = SCA_USER,	/* clear the user requested mask */
>  	};
[ ... ]
>  	/*
>  	 * Because this is called with p->pi_lock held, it is not possible
> -	 * to use kfree() here (when PREEMPT_RT=y), therefore punt to using
> -	 * kfree_rcu().
> +	 * to use kfree() here (when PREEMPT_RT=y), thus use kfree_nolock()
>  	 */
> -	kfree_rcu((union cpumask_rcuhead *)ac.user_mask, rcu);
> +	kfree_nolock(ac.user_mask);

[Severity: High]
Does this change trigger an "Invalid wait context" lockdep warning and RT
deadlock risk?

set_cpus_allowed_force() is called with the target task's pi_lock held, which
is a raw_spinlock_t.

kfree_nolock() forwards the call to free_to_pcs():

mm/slub.c:kfree_nolock() {
    ...
	if (likely(can_free_to_pcs(slab)) && likely(free_to_pcs(s, x, false)))
    ...
}

free_to_pcs() then unconditionally acquires a local_trylock_t:

mm/slub.c:free_to_pcs() {
    ...
	if (!local_trylock(&s->cpu_sheaves->lock))
    ...
}

On PREEMPT_RT, local_trylock_t maps to a sleeping spinlock. Acquiring a
sleeping lock (even via a trylock) while holding a raw spinlock violates the
wait-context rules (WAIT_SLEEP under WAIT_RAW).

Could this cause a lockdep splat since CONFIG_PROVE_RAW_LOCK_NESTING checks
declared wait-types rather than runtime behavior?


[Severity: Critical]
This is a pre-existing issue, but can the lockless access of user_cpus_ptr in
relax_compatible_cpus_allowed_ptr() race with a concurrent sched_setaffinity()?

relax_compatible_cpus_allowed_ptr() performs a lockless read of
p->user_cpus_ptr and passes it as ctx->new_mask:

kernel/sched/core.c:relax_compatible_cpus_allowed_ptr() {
    ...
	.new_mask  = task_user_cpus(p),
    ...
}

__sched_setaffinity() then allocates temporary masks with GFP_KERNEL:

kernel/sched/syscalls.c:__sched_setaffinity() {
    ...
	if (!alloc_cpumask_var(&cpus_allowed, GFP_KERNEL))
    ...
}

Since GFP_KERNEL can sleep or be preempted, a concurrent thread calling the
sched_setaffinity() syscall on the same task can update user_cpus_ptr and free
the old mask.

When __sched_setaffinity() resumes, it dereferences the potentially freed
memory:

kernel/sched/syscalls.c:__sched_setaffinity() {
    ...
	cpumask_and(new_mask, ctx->new_mask, cpus_allowed);
    ...
}

Can this use-after-free lead to memory corruption or a kernel crash if
triggered by unprivileged userspace executing a compat binary?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260807-kfree_nolock_kmalloc-v1-0-ba993cbf7a60@kernel.org?part=5