[PATCH RFC bpf-next 0/3] bpf: Fix LRU NMI/tracepoint re-entry deadlock

[PATCH RFC bpf-next 0/3] bpf: Fix LRU NMI/tracepoint re-entry deadlock

[Mykyta Yatsenko]

This series fixes AA-deadlocks where NMI and tracepoint BPF programs
re-enter the per-CPU or global LRU lock already held on the same CPU
(syzbot c69a0a2c816716f1e0d5, 18b26edb69b2e19f3b33). Three prior
attempts have been nacked since 2024.

Patch 1 converts every LRU lock site (six of them) to rqspinlock_t
and adds explicit recovery for some failures so no node leaks:
per-CPU lockless free_llist for stolen-node recovery, plus a
pending_free flag honored by flush, shrink, and the steal path.
rqspinlock blocks like raw_spin_lock for normal contention and only
fails on deadlock or a timeout, so steady-state contention does
not degrade reliability.

Patch 2 refreshes Documentation/bpf/map_lru_hash_update.dot to show
the new rqspinlock failure exits and recovery routes.

Patch 3 replaces the original single-CPU NMI reproducer with a
parameterized stress harness across three map flavors (common LRU,
NO_COMMON_LRU per-CPU lists, per-CPU LRU map): perf-event NMI BPF on
every online CPU plus one pinned userspace hammer per CPU, both
mixing update and delete. Post-stress drain-and-refill asserts every
slot is re-allocatable - a stranded node from a failed push_free
recovery would surface as -ENOMEM that never clears.

Tested on a 4-CPU vng VM with PROVE_LOCKING: lru_lock_nmi (3
subtests) and the broader test_maps run pass with no lockdep splats.

Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
---
Mykyta Yatsenko (3):
      bpf: Fix NMI/tracepoint re-entry deadlock on lru locks
      Documentation/bpf: Refresh map_lru_hash_update.dot for rqspinlock
      selftests/bpf: Stress LRU rqspinlock recovery paths

 Documentation/bpf/map_lru_hash_update.dot          |  44 ++++-
 kernel/bpf/bpf_lru_list.c                          | 163 ++++++++++-------
 kernel/bpf/bpf_lru_list.h                          |  25 ++-
 .../selftests/bpf/prog_tests/lru_lock_nmi.c        | 202 +++++++++++++++++++++
 tools/testing/selftests/bpf/progs/lru_lock_nmi.c   |  33 ++++
 5 files changed, 394 insertions(+), 73 deletions(-)
---
base-commit: d34771f2f3db4ea0dd532cdd579002fc95dc483a
change-id: 20260519-lru_map_spin-8e20dabbf460

Best regards,
--  
Mykyta Yatsenko <yatsenko@meta.com>

[PATCH RFC bpf-next 1/3] bpf: Fix NMI/tracepoint re-entry deadlock on lru locks

[Mykyta Yatsenko]

From: Mykyta Yatsenko <yatsenko@meta.com>

NMI and tracepoint BPF programs can re-enter the per-CPU or global
LRU lock that bpf_lru_pop_free()/push_free() already hold on the
same CPU, AA-deadlocking. Lockdep reports "inconsistent
{INITIAL USE} -> {IN-NMI}" on &l->lock (syzbot c69a0a2c816716f1e0d5)
and "possible recursive locking detected" on &loc_l->lock (syzbot
18b26edb69b2e19f3b33).

Prior trylock-based fixes were nacked because trylock fails on any
contention, returning -ENOMEM even when no deadlock exists - that
sacrifices LRU map reliability under normal load. rqspinlock is
different: it blocks like raw_spin_lock for benign contention and
only fails on AA self-deadlock (the bug being fixed) or contention
timeout (an actual cycle). The normal blocking-acquire path is
preserved, so steady-state contention does not produce spurious
-ENOMEM.

This patch converts every LRU lock site to rqspinlock_t and adds a
recovery path for each failure window so no node leaks.

Failure recovery:

 - *_pop_free top-level: return NULL; prealloc_lru_pop() already
   treats that as no-free-element (-ENOMEM).

 - Cross-CPU steal: skip the victim's locked loc_l, try next CPU.

 - Post-steal local lock fail: publish stolen node to lockless
   per-CPU free_llist; next pop on this CPU picks it up.

 - push_free fail: mark node pending_free=1. __local_list_flush(),
   __bpf_lru_list_shrink_inactive() and the steal path
   (__local_list_pop_pending) all honor the flag and reclaim
   regardless of del_from_htab(). For lru->percpu maps the strand
   is reclaimed on the owning CPU's subsequent pop_free calls (up
   to nr_scans=4 marked nodes per call); cross-CPU recovery is not
   available because per-CPU LRU lists are owned by their CPU.

Fixes: 3a08c2fd7634 ("bpf: LRU List")
Reported-by: syzbot+c69a0a2c816716f1e0d5@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=c69a0a2c816716f1e0d5
Reported-by: syzbot+18b26edb69b2e19f3b33@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=18b26edb69b2e19f3b33
Link: https://lore.kernel.org/bpf/CAPPBnEYO4R+m+SpVc2gNj_x31R6fo1uJvj2bK2YS1P09GWT6kQ@mail.gmail.com/
Link: https://lore.kernel.org/bpf/CAPPBnEZmFA3ab8Uc=PEm0bdojZy=7T_F5_+eyZSHyZR3MBG4Vw@mail.gmail.com/
Link: https://lore.kernel.org/bpf/20251030030010.95352-1-dongml2@chinatelecom.cn/
Link: https://lore.kernel.org/bpf/20260119142120.28170-1-leon.hwang@linux.dev/
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
---
 kernel/bpf/bpf_lru_list.c | 163 ++++++++++++++++++++++++++++------------------
 kernel/bpf/bpf_lru_list.h |  25 +++++--
 2 files changed, 117 insertions(+), 71 deletions(-)

diff --git a/kernel/bpf/bpf_lru_list.c b/kernel/bpf/bpf_lru_list.c
index e7a2fc60523f..a5faf4464d88 100644
--- a/kernel/bpf/bpf_lru_list.c
+++ b/kernel/bpf/bpf_lru_list.c
@@ -13,23 +13,8 @@
 #define PERCPU_FREE_TARGET		(4)
 #define PERCPU_NR_SCANS			PERCPU_FREE_TARGET
 
-/* Helpers to get the local list index */
-#define LOCAL_LIST_IDX(t)	((t) - BPF_LOCAL_LIST_T_OFFSET)
-#define LOCAL_FREE_LIST_IDX	LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE)
-#define LOCAL_PENDING_LIST_IDX	LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_PENDING)
 #define IS_LOCAL_LIST_TYPE(t)	((t) >= BPF_LOCAL_LIST_T_OFFSET)
 
-/* Local list helpers */
-static struct list_head *local_free_list(struct bpf_lru_locallist *loc_l)
-{
-	return &loc_l->lists[LOCAL_FREE_LIST_IDX];
-}
-
-static struct list_head *local_pending_list(struct bpf_lru_locallist *loc_l)
-{
-	return &loc_l->lists[LOCAL_PENDING_LIST_IDX];
-}
-
 /* bpf_lru_node helpers */
 static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node)
 {
@@ -72,6 +57,7 @@ static void __bpf_lru_node_move_to_free(struct bpf_lru_list *l,
 	bpf_lru_list_count_dec(l, node->type);
 
 	node->type = tgt_free_type;
+	WRITE_ONCE(node->pending_free, 0);
 	list_move(&node->list, free_list);
 }
 
@@ -87,6 +73,7 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l,
 	bpf_lru_list_count_inc(l, tgt_type);
 	node->type = tgt_type;
 	bpf_lru_node_clear_ref(node);
+	WRITE_ONCE(node->pending_free, 0);
 	list_move(&node->list, &l->lists[tgt_type]);
 }
 
@@ -212,9 +199,11 @@ __bpf_lru_list_shrink_inactive(struct bpf_lru *lru,
 	unsigned int i = 0;
 
 	list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) {
-		if (bpf_lru_node_is_ref(node)) {
+		if (bpf_lru_node_is_ref(node) &&
+		    !READ_ONCE(node->pending_free)) {
 			__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);
-		} else if (lru->del_from_htab(lru->del_arg, node)) {
+		} else if (READ_ONCE(node->pending_free) ||
+			   lru->del_from_htab(lru->del_arg, node)) {
 			__bpf_lru_node_move_to_free(l, node, free_list,
 						    tgt_free_type);
 			if (++nshrinked == tgt_nshrink)
@@ -273,7 +262,8 @@ static unsigned int __bpf_lru_list_shrink(struct bpf_lru *lru,
 
 	list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list,
 					 list) {
-		if (lru->del_from_htab(lru->del_arg, node)) {
+		if (READ_ONCE(node->pending_free) ||
+		    lru->del_from_htab(lru->del_arg, node)) {
 			__bpf_lru_node_move_to_free(l, node, free_list,
 						    tgt_free_type);
 			return 1;
@@ -290,8 +280,10 @@ static void __local_list_flush(struct bpf_lru_list *l,
 	struct bpf_lru_node *node, *tmp_node;
 
 	list_for_each_entry_safe_reverse(node, tmp_node,
-					 local_pending_list(loc_l), list) {
-		if (bpf_lru_node_is_ref(node))
+					 &loc_l->pending_list, list) {
+		if (READ_ONCE(node->pending_free))
+			__bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_FREE);
+		else if (bpf_lru_node_is_ref(node))
 			__bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE);
 		else
 			__bpf_lru_node_move_in(l, node,
@@ -307,9 +299,12 @@ static void bpf_lru_list_push_free(struct bpf_lru_list *l,
 	if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))
 		return;
 
-	raw_spin_lock_irqsave(&l->lock, flags);
+	if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
+		WRITE_ONCE(node->pending_free, 1);
+		return;
+	}
 	__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
-	raw_spin_unlock_irqrestore(&l->lock, flags);
+	raw_res_spin_unlock_irqrestore(&l->lock, flags);
 }
 
 static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
@@ -318,8 +313,10 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
 	struct bpf_lru_list *l = &lru->common_lru.lru_list;
 	struct bpf_lru_node *node, *tmp_node;
 	unsigned int nfree = 0;
+	LIST_HEAD(tmp_free);
 
-	raw_spin_lock(&l->lock);
+	if (raw_res_spin_lock(&l->lock))
+		return;
 
 	__local_list_flush(l, loc_l);
 
@@ -327,7 +324,7 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
 
 	list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE],
 				 list) {
-		__bpf_lru_node_move_to_free(l, node, local_free_list(loc_l),
+		__bpf_lru_node_move_to_free(l, node, &tmp_free,
 					    BPF_LRU_LOCAL_LIST_T_FREE);
 		if (++nfree == lru->target_free)
 			break;
@@ -335,10 +332,19 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
 
 	if (nfree < lru->target_free)
 		__bpf_lru_list_shrink(lru, l, lru->target_free - nfree,
-				      local_free_list(loc_l),
+				      &tmp_free,
 				      BPF_LRU_LOCAL_LIST_T_FREE);
 
-	raw_spin_unlock(&l->lock);
+	raw_res_spin_unlock(&l->lock);
+
+	/*
+	 * Transfer the harvested nodes from the temporary list_head into
+	 * the lockless per-CPU free llist.
+	 */
+	list_for_each_entry_safe(node, tmp_node, &tmp_free, list) {
+		list_del(&node->list);
+		llist_add(&node->llist, &loc_l->free_llist);
+	}
 }
 
 static void __local_list_add_pending(struct bpf_lru *lru,
@@ -350,22 +356,21 @@ static void __local_list_add_pending(struct bpf_lru *lru,
 	*(u32 *)((void *)node + lru->hash_offset) = hash;
 	node->cpu = cpu;
 	node->type = BPF_LRU_LOCAL_LIST_T_PENDING;
+	WRITE_ONCE(node->pending_free, 0);
 	bpf_lru_node_clear_ref(node);
-	list_add(&node->list, local_pending_list(loc_l));
+	list_add(&node->list, &loc_l->pending_list);
 }
 
 static struct bpf_lru_node *
 __local_list_pop_free(struct bpf_lru_locallist *loc_l)
 {
-	struct bpf_lru_node *node;
+	struct llist_node *llnode;
 
-	node = list_first_entry_or_null(local_free_list(loc_l),
-					struct bpf_lru_node,
-					list);
-	if (node)
-		list_del(&node->list);
+	llnode = llist_del_first(&loc_l->free_llist);
+	if (!llnode)
+		return NULL;
 
-	return node;
+	return container_of(llnode, struct bpf_lru_node, llist);
 }
 
 static struct bpf_lru_node *
@@ -376,10 +381,10 @@ __local_list_pop_pending(struct bpf_lru *lru, struct bpf_lru_locallist *loc_l)
 
 ignore_ref:
 	/* Get from the tail (i.e. older element) of the pending list. */
-	list_for_each_entry_reverse(node, local_pending_list(loc_l),
-				    list) {
+	list_for_each_entry_reverse(node, &loc_l->pending_list, list) {
 		if ((!bpf_lru_node_is_ref(node) || force) &&
-		    lru->del_from_htab(lru->del_arg, node)) {
+		    (READ_ONCE(node->pending_free) ||
+		     lru->del_from_htab(lru->del_arg, node))) {
 			list_del(&node->list);
 			return node;
 		}
@@ -404,7 +409,8 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru,
 
 	l = per_cpu_ptr(lru->percpu_lru, cpu);
 
-	raw_spin_lock_irqsave(&l->lock, flags);
+	if (raw_res_spin_lock_irqsave(&l->lock, flags))
+		return NULL;
 
 	__bpf_lru_list_rotate(lru, l);
 
@@ -420,7 +426,7 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru,
 		__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE);
 	}
 
-	raw_spin_unlock_irqrestore(&l->lock, flags);
+	raw_res_spin_unlock_irqrestore(&l->lock, flags);
 
 	return node;
 }
@@ -437,7 +443,8 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
 
 	loc_l = per_cpu_ptr(clru->local_list, cpu);
 
-	raw_spin_lock_irqsave(&loc_l->lock, flags);
+	if (raw_res_spin_lock_irqsave(&loc_l->lock, flags))
+		return NULL;
 
 	node = __local_list_pop_free(loc_l);
 	if (!node) {
@@ -448,17 +455,22 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
 	if (node)
 		__local_list_add_pending(lru, loc_l, cpu, node, hash);
 
-	raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+	raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
 
 	if (node)
 		return node;
 
-	/* No free nodes found from the local free list and
+	/*
+	 * No free nodes found from the local free list and
 	 * the global LRU list.
 	 *
 	 * Steal from the local free/pending list of the
 	 * current CPU and remote CPU in RR.  It starts
 	 * with the loc_l->next_steal CPU.
+	 *
+	 * Acquire the victim's lock before touching either list. On
+	 * acquisition failure (rqspinlock AA or timeout) skip the victim
+	 * and try the next CPU.
 	 */
 
 	first_steal = loc_l->next_steal;
@@ -466,24 +478,36 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
 	do {
 		steal_loc_l = per_cpu_ptr(clru->local_list, steal);
 
-		raw_spin_lock_irqsave(&steal_loc_l->lock, flags);
-
-		node = __local_list_pop_free(steal_loc_l);
-		if (!node)
-			node = __local_list_pop_pending(lru, steal_loc_l);
-
-		raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
+		if (!raw_res_spin_lock_irqsave(&steal_loc_l->lock, flags)) {
+			node = __local_list_pop_free(steal_loc_l);
+			if (!node)
+				node = __local_list_pop_pending(lru, steal_loc_l);
+			raw_res_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
+		}
 
 		steal = cpumask_next_wrap(steal, cpu_possible_mask);
 	} while (!node && steal != first_steal);
 
 	loc_l->next_steal = steal;
 
-	if (node) {
-		raw_spin_lock_irqsave(&loc_l->lock, flags);
-		__local_list_add_pending(lru, loc_l, cpu, node, hash);
-		raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+	if (!node)
+		return NULL;
+
+	if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
+		/*
+		 * The local pending lock can't be acquired (rqspinlock AA
+		 * or timeout). Return the stolen node to the per-CPU
+		 * free_llist instead of orphaning it; the next pop_free on
+		 * this CPU will pick it up.
+		 */
+		node->type = BPF_LRU_LOCAL_LIST_T_FREE;
+		bpf_lru_node_clear_ref(node);
+		WRITE_ONCE(node->pending_free, 0);
+		llist_add(&node->llist, &loc_l->free_llist);
+		return NULL;
 	}
+	__local_list_add_pending(lru, loc_l, cpu, node, hash);
+	raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
 
 	return node;
 }
@@ -511,18 +535,24 @@ static void bpf_common_lru_push_free(struct bpf_lru *lru,
 
 		loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu);
 
-		raw_spin_lock_irqsave(&loc_l->lock, flags);
+		if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
+			WRITE_ONCE(node->pending_free, 1);
+			return;
+		}
 
 		if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) {
-			raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+			raw_res_spin_unlock_irqrestore(&loc_l->lock,
+						       flags);
 			goto check_lru_list;
 		}
 
 		node->type = BPF_LRU_LOCAL_LIST_T_FREE;
 		bpf_lru_node_clear_ref(node);
-		list_move(&node->list, local_free_list(loc_l));
+		list_del(&node->list);
+
+		raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
 
-		raw_spin_unlock_irqrestore(&loc_l->lock, flags);
+		llist_add(&node->llist, &loc_l->free_llist);
 		return;
 	}
 
@@ -538,11 +568,14 @@ static void bpf_percpu_lru_push_free(struct bpf_lru *lru,
 
 	l = per_cpu_ptr(lru->percpu_lru, node->cpu);
 
-	raw_spin_lock_irqsave(&l->lock, flags);
+	if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
+		WRITE_ONCE(node->pending_free, 1);
+		return;
+	}
 
 	__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
 
-	raw_spin_unlock_irqrestore(&l->lock, flags);
+	raw_res_spin_unlock_irqrestore(&l->lock, flags);
 }
 
 void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
@@ -565,6 +598,7 @@ static void bpf_common_lru_populate(struct bpf_lru *lru, void *buf,
 
 		node = (struct bpf_lru_node *)(buf + node_offset);
 		node->type = BPF_LRU_LIST_T_FREE;
+		node->pending_free = 0;
 		bpf_lru_node_clear_ref(node);
 		list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
 		buf += elem_size;
@@ -594,6 +628,7 @@ static void bpf_percpu_lru_populate(struct bpf_lru *lru, void *buf,
 		node = (struct bpf_lru_node *)(buf + node_offset);
 		node->cpu = cpu;
 		node->type = BPF_LRU_LIST_T_FREE;
+		node->pending_free = 0;
 		bpf_lru_node_clear_ref(node);
 		list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
 		i++;
@@ -618,14 +653,12 @@ void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
 
 static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu)
 {
-	int i;
-
-	for (i = 0; i < NR_BPF_LRU_LOCAL_LIST_T; i++)
-		INIT_LIST_HEAD(&loc_l->lists[i]);
+	INIT_LIST_HEAD(&loc_l->pending_list);
+	init_llist_head(&loc_l->free_llist);
 
 	loc_l->next_steal = cpu;
 
-	raw_spin_lock_init(&loc_l->lock);
+	raw_res_spin_lock_init(&loc_l->lock);
 }
 
 static void bpf_lru_list_init(struct bpf_lru_list *l)
@@ -640,7 +673,7 @@ static void bpf_lru_list_init(struct bpf_lru_list *l)
 
 	l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE];
 
-	raw_spin_lock_init(&l->lock);
+	raw_res_spin_lock_init(&l->lock);
 }
 
 int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset,
diff --git a/kernel/bpf/bpf_lru_list.h b/kernel/bpf/bpf_lru_list.h
index fe2661a58ea9..8d0ee61622af 100644
--- a/kernel/bpf/bpf_lru_list.h
+++ b/kernel/bpf/bpf_lru_list.h
@@ -6,11 +6,11 @@
 
 #include <linux/cache.h>
 #include <linux/list.h>
-#include <linux/spinlock_types.h>
+#include <linux/llist.h>
+#include <asm/rqspinlock.h>
 
 #define NR_BPF_LRU_LIST_T	(3)
 #define NR_BPF_LRU_LIST_COUNT	(2)
-#define NR_BPF_LRU_LOCAL_LIST_T (2)
 #define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T
 
 enum bpf_lru_list_type {
@@ -22,10 +22,22 @@ enum bpf_lru_list_type {
 };
 
 struct bpf_lru_node {
-	struct list_head list;
+	/*
+	 * A node is in at most one list at a time. The free path on the
+	 * per-CPU locallist uses an llist, so share storage via a union.
+	 */
+	union {
+		struct list_head list;
+		struct llist_node llist;
+	};
 	u16 cpu;
 	u8 type;
 	u8 ref;
+	/*
+	 * Marks nodes whose *_push_free() lock acquire failed; reclaimed
+	 * by flush/shrink which honor the flag instead of del_from_htab().
+	 */
+	u8 pending_free;
 };
 
 struct bpf_lru_list {
@@ -34,13 +46,14 @@ struct bpf_lru_list {
 	/* The next inactive list rotation starts from here */
 	struct list_head *next_inactive_rotation;
 
-	raw_spinlock_t lock ____cacheline_aligned_in_smp;
+	rqspinlock_t lock ____cacheline_aligned_in_smp;
 };
 
 struct bpf_lru_locallist {
-	struct list_head lists[NR_BPF_LRU_LOCAL_LIST_T];
+	struct list_head pending_list;
+	struct llist_head free_llist;
 	u16 next_steal;
-	raw_spinlock_t lock;
+	rqspinlock_t lock;
 };
 
 struct bpf_common_lru {

-- 
2.53.0-Meta

[PATCH RFC bpf-next 2/3] Documentation/bpf: Refresh map_lru_hash_update.dot for rqspinlock

[Mykyta Yatsenko]

From: Mykyta Yatsenko <yatsenko@meta.com>

Reflect the rqspinlock conversion and orphan-recovery paths added in
the previous commit:

 - All LRU locks are rqspinlock_t; any acquire can fail (AA or
   timeout). A shared "rqspinlock acquire failed" terminal collapses
   to the existing -ENOMEM exit. Dashed arrows from each acquire site
   mark the failure paths.

 - The per-CPU local freelist is now lockless (free_llist).

 - Post-steal: re-acquiring loc_l->lock to insert the stolen node
   into the local pending list can fail; on failure the node is
   published to free_llist instead of being orphaned, and the call
   returns -ENOMEM.

 - Steal-loop victim lock failure is silent: skip the victim and try
   the next CPU.

Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
---
 Documentation/bpf/map_lru_hash_update.dot | 44 +++++++++++++++++++++++++++++--
 1 file changed, 42 insertions(+), 2 deletions(-)

diff --git a/Documentation/bpf/map_lru_hash_update.dot b/Documentation/bpf/map_lru_hash_update.dot
index ab10058f5b79..412bc8b3b57e 100644
--- a/Documentation/bpf/map_lru_hash_update.dot
+++ b/Documentation/bpf/map_lru_hash_update.dot
@@ -21,10 +21,18 @@ digraph {
   // names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c.
   // Number suffixes and errno suffixes handle subsections of the corresponding
   // logic in the function as of the writing of this dot.
+  //
+  // All LRU locks are rqspinlock_t. Every acquire can fail (AA self-deadlock
+  // or contention timeout); on failure the corresponding helper returns NULL
+  // and the caller propagates -ENOMEM. The "rqspinlock acquire failed"
+  // terminal below is reached via the dashed arrows from each acquire site.
+
+  rqspinlock_failed [shape=rectangle,
+    label="Any LRU rqspinlock\nacquire fails\n(AA or timeout)"]
 
   // cf. __local_list_pop_free() / bpf_percpu_lru_pop_free()
   local_freelist_check [shape=diamond,fillcolor=1,
-    label="Local freelist\nnode available?"];
+    label="Local freelist\nnode available?\n(lockless free_llist)"];
   use_local_node [shape=rectangle,
     label="Use node owned\nby this CPU"]
 
@@ -82,6 +90,15 @@ digraph {
     // fn__local_list_pop_pending()
   }
 
+  // Post-steal: re-acquire local loc_l->lock to insert the stolen node into
+  // the local pending list. If the acquire fails, the stolen node is published
+  // to the lockless local free_llist so the next pop on this CPU picks it up
+  // instead of orphaning it.
+  post_steal_lock [shape=diamond,fillcolor=1,
+    label="Acquire local\nloc_l->lock\nto add pending"]
+  post_steal_to_free_llist [shape=rectangle,
+    label="Publish stolen node to\nlocal free_llist (lockless)"]
+
   fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle,
     label="Use node that was\nnot recently referenced"]
   local_freelist_check4 [shape=rectangle,
@@ -97,10 +114,19 @@ digraph {
   fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"]
 
   begin -> local_freelist_check
+  // The initial per-CPU lock (loc_l->lock for common, l->lock for percpu) is
+  // acquired before the local freelist check; rqspinlock failure here exits
+  // directly to -ENOMEM (no recovery needed: nothing was removed yet).
+  local_freelist_check -> rqspinlock_failed [style=dashed,
+    xlabel="acquire fails"]
   local_freelist_check -> use_local_node [xlabel="Y"]
   local_freelist_check -> common_lru_check [xlabel="N"]
   common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"]
   common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"]
+  // Global lru_list lock acquire failure in pop_free_to_local: skip refill,
+  // fall through to the steal path. Not ENOMEM by itself.
+  fn_bpf_lru_list_pop_free_to_local -> common_lru_check2 [style=dashed,
+    xlabel="global lru_lock\nacquire fails"]
   fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free
   fn___bpf_lru_node_move_to_free ->
     fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"]
@@ -120,13 +146,27 @@ digraph {
   local_freelist_check6 -> local_freelist_check7
   local_freelist_check7 -> fn_htab_lru_map_update_elem
 
-  fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem3 [xlabel = "Y"]
+  // Steal-loop victim lock failure is silent: treat as "no node found here"
+  // and continue to next CPU; same edge as the existing "N" path.
+  local_freelist_check5 -> fn_htab_lru_map_update_elem2 [style=dashed,
+    xlabel="victim's lock\nfails: skip"]
+  // After a successful steal, re-acquire the local loc_l->lock. On failure
+  // the stolen node is published to free_llist (recovered, not orphaned)
+  // and the update returns -ENOMEM.
+  fn_htab_lru_map_update_elem -> post_steal_lock [xlabel = "Y"]
+  post_steal_lock -> fn_htab_lru_map_update_elem3 [xlabel = "OK"]
+  post_steal_lock -> post_steal_to_free_llist [style=dashed,
+    xlabel="loc_l->lock\nacquire fails"]
+  post_steal_to_free_llist -> fn_htab_lru_map_update_elem_ENOMEM
   fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2  [xlabel = "N"]
   fn_htab_lru_map_update_elem2 ->
     fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"]
   fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"]
   fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4
 
+  // Shared rqspinlock-failure terminal collapses to the same -ENOMEM exit.
+  rqspinlock_failed -> fn_htab_lru_map_update_elem_ENOMEM
+
   use_local_node -> fn_htab_lru_map_update_elem4
   fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4
   local_freelist_check4 -> fn_htab_lru_map_update_elem4

-- 
2.53.0-Meta

[PATCH RFC bpf-next 3/3] selftests/bpf: Stress LRU rqspinlock recovery paths

[Mykyta Yatsenko]

From: Mykyta Yatsenko <yatsenko@meta.com>

Replaces the single-CPU NMI re-entry reproducer with a parameterized
stress harness that exercises every lock-failure and orphan-recovery
branch added by the LRU rqspinlock conversion: per-CPU loc_l->lock
NMI re-entry, cross-CPU steal contention, post-steal local lock fail
with free_llist recovery, and pending_free reclaim via flush,
shrink_inactive, and the new __local_list_pop_pending path.

Runs three subtests: common LRU, per-CPU LRU lists (BPF_F_NO_COMMON_LRU),
and per-CPU LRU map. Each pins one userspace hammer per CPU and attaches
the perf_event NMI BPF prog (update+delete mix) on every online CPU.
Pre-fix, lockdep fires the "INITIAL USE -> IN-NMI" splat during stress.

Post-stress, the test drains all keys and refills max_entries fresh
slots with bounded retries on -ENOMEM. A permanently stranded node
from a failed push_free recovery would surface as -ENOMEM that never
clears.

Marked serial_ because per-CPU pinning and high-rate HW perf events
would perturb parallel tests.

Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
---
 .../selftests/bpf/prog_tests/lru_lock_nmi.c        | 202 +++++++++++++++++++++
 tools/testing/selftests/bpf/progs/lru_lock_nmi.c   |  33 ++++
 2 files changed, 235 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c b/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c
new file mode 100644
index 000000000000..15b6606fef4d
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c
@@ -0,0 +1,202 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Stress every LRU lock-failure and orphan-recovery branch added by the
+ * rqspinlock conversion. perf_event NMI BPF on every online CPU does
+ * update+delete on a small LRU map; userspace threads on every CPU do
+ * the same from syscall context. Exercises:
+ *
+ *   - per-CPU loc_l->lock NMI re-entry (the original syzbot deadlock)
+ *   - cross-CPU steal-loop victim lock contention
+ *   - post-steal local lock contention and free_llist recovery
+ *   - push_free lock failure and pending_free recovery via flush,
+ *     shrink_inactive, and the new __local_list_pop_pending path
+ *
+ * Runs against three map flavors:
+ *   - BPF_MAP_TYPE_LRU_HASH (common LRU)
+ *   - BPF_MAP_TYPE_LRU_HASH | BPF_F_NO_COMMON_LRU (per-CPU LRU lists,
+ *     common values)
+ *   - BPF_MAP_TYPE_LRU_PERCPU_HASH (per-CPU LRU + per-CPU values)
+ *
+ * Post-stress: every map slot must be re-allocatable, proving no node
+ * was permanently stranded by a failed push_free recovery. Transient
+ * -ENOMEM during the drain is allowed and retried.
+ *
+ * With PROVE_LOCKING enabled, the pre-fix kernel fires a lockdep splat
+ * during stress; the framework's dmesg capture surfaces it.
+ */
+#define _GNU_SOURCE
+#include <pthread.h>
+#include <sched.h>
+#include <sys/syscall.h>
+#include <linux/perf_event.h>
+#include <test_progs.h>
+#include "testing_helpers.h"
+#include "lru_lock_nmi.skel.h"
+
+#define MAP_ENTRIES	64
+#define KEY_RANGE	(MAP_ENTRIES * 2)	/* wider than capacity to force eviction */
+#define STRESS_NS	(500 * 1000 * 1000ULL)
+
+struct hammer_arg {
+	int map_fd;
+	int cpu;
+	int nr_cpus;
+	__u64 deadline_ns;
+};
+
+static void *hammer_thread(void *p)
+{
+	struct hammer_arg *a = p;
+	__u64 val[a->nr_cpus];
+	cpu_set_t cs;
+	__u32 key;
+
+	memset(val, 0, sizeof(val));
+	CPU_ZERO(&cs);
+	CPU_SET(a->cpu, &cs);
+	pthread_setaffinity_np(pthread_self(), sizeof(cs), &cs);
+
+	while (get_time_ns() < a->deadline_ns) {
+		bool do_update = rand() & 1;
+
+		key = rand() % KEY_RANGE;
+		if (do_update)
+			bpf_map_update_elem(a->map_fd, &key, val, BPF_ANY);
+		else
+			bpf_map_delete_elem(a->map_fd, &key);
+	}
+	return NULL;
+}
+
+/*
+ * After stress, the map must accept MAP_ENTRIES fresh allocations.
+ * A node permanently stranded by a failed push_free recovery would
+ * surface as -ENOMEM from update_elem: shrink_inactive cannot find
+ * either a pending_free=1 marker or a del_from_htab=true elem.
+ * No retry needed - by the time drain runs, NMI is detached and no
+ * userspace contention is present, so every update either pulls
+ * from FREE or evicts a previously-added entry on the same CPU.
+ */
+static int drain_and_refill(int map_fd, int nr_cpus)
+{
+	__u64 val[nr_cpus];
+	__u32 key;
+
+	memset(val, 0, sizeof(val));
+	for (key = 0; key < KEY_RANGE; key++)
+		bpf_map_delete_elem(map_fd, &key);
+
+	for (key = 0; key < MAP_ENTRIES; key++)
+		if (bpf_map_update_elem(map_fd, &key, val, BPF_ANY))
+			return -ENOMEM;
+	return 0;
+}
+
+static void run_variant(enum bpf_map_type type, __u32 map_flags,
+			const char *name)
+{
+	struct perf_event_attr attr = {
+		.size = sizeof(attr),
+		.type = PERF_TYPE_HARDWARE,
+		.config = PERF_COUNT_HW_CPU_CYCLES,
+		.freq = 1,
+	};
+	int nr_cpus = libbpf_num_possible_cpus();
+	int pmu_fds[nr_cpus];
+	struct bpf_link *links[nr_cpus];
+	pthread_t threads[nr_cpus];
+	struct hammer_arg args[nr_cpus];
+	struct lru_lock_nmi *skel = NULL;
+	int map_fd, i, err, nr_threads = 0;
+	__u64 deadline;
+
+	if (!test__start_subtest(name))
+		return;
+
+	memset(pmu_fds, 0xff, sizeof(pmu_fds));	/* -1 sentinel */
+	memset(links, 0, sizeof(links));
+
+	skel = lru_lock_nmi__open();
+	if (!ASSERT_OK_PTR(skel, "skel_open"))
+		goto cleanup;
+
+	err = bpf_map__set_type(skel->maps.lru_map, type);
+	if (!ASSERT_OK(err, "set_type"))
+		goto cleanup;
+	err = bpf_map__set_map_flags(skel->maps.lru_map, map_flags);
+	if (!ASSERT_OK(err, "set_flags"))
+		goto cleanup;
+	err = bpf_map__set_max_entries(skel->maps.lru_map, MAP_ENTRIES);
+	if (!ASSERT_OK(err, "set_max_entries"))
+		goto cleanup;
+
+	err = lru_lock_nmi__load(skel);
+	if (!ASSERT_OK(err, "skel_load"))
+		goto cleanup;
+
+	skel->bss->hits = 0;
+	map_fd = bpf_map__fd(skel->maps.lru_map);
+	attr.sample_freq = read_perf_max_sample_freq();
+
+	for (i = 0; i < nr_cpus; i++) {
+		pmu_fds[i] = syscall(__NR_perf_event_open, &attr, -1, i,
+				     -1, 0);
+		if (pmu_fds[i] < 0) {
+			if (i == 0 &&
+			    (errno == ENOENT || errno == EOPNOTSUPP)) {
+				test__skip();
+				goto cleanup;
+			}
+			continue;
+		}
+		links[i] = bpf_program__attach_perf_event(skel->progs.oncpu,
+							  pmu_fds[i]);
+		if (!links[i]) {
+			close(pmu_fds[i]);
+			pmu_fds[i] = -1;
+		}
+	}
+
+	deadline = get_time_ns() + STRESS_NS;
+	for (i = 0; i < nr_cpus; i++) {
+		args[i].map_fd = map_fd;
+		args[i].cpu = i;
+		args[i].nr_cpus = nr_cpus;
+		args[i].deadline_ns = deadline;
+		if (pthread_create(&threads[nr_threads], NULL, hammer_thread,
+				   &args[i]) == 0)
+			nr_threads++;
+	}
+	for (i = 0; i < nr_threads; i++)
+		pthread_join(threads[i], NULL);
+
+	for (i = 0; i < nr_cpus; i++) {
+		if (links[i]) {
+			bpf_link__destroy(links[i]);
+			links[i] = NULL;
+		}
+		if (pmu_fds[i] >= 0) {
+			close(pmu_fds[i]);
+			pmu_fds[i] = -1;
+		}
+	}
+
+	ASSERT_GT(skel->bss->hits, 0, "nmi_bpf_ran");
+	ASSERT_OK(drain_and_refill(map_fd, nr_cpus), "drain_and_refill");
+
+cleanup:
+	for (i = 0; i < nr_cpus; i++) {
+		if (links[i])
+			bpf_link__destroy(links[i]);
+		if (pmu_fds[i] >= 0)
+			close(pmu_fds[i]);
+	}
+	lru_lock_nmi__destroy(skel);
+}
+
+void serial_test_lru_lock_nmi(void)
+{
+	run_variant(BPF_MAP_TYPE_LRU_HASH, 0, "common_lru");
+	run_variant(BPF_MAP_TYPE_LRU_HASH, BPF_F_NO_COMMON_LRU, "no_common_lru");
+	run_variant(BPF_MAP_TYPE_LRU_PERCPU_HASH, 0, "percpu_lru");
+}
diff --git a/tools/testing/selftests/bpf/progs/lru_lock_nmi.c b/tools/testing/selftests/bpf/progs/lru_lock_nmi.c
new file mode 100644
index 000000000000..c0692cd54237
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/lru_lock_nmi.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+struct {
+	__uint(type, BPF_MAP_TYPE_LRU_HASH);
+	__uint(max_entries, 64);
+	__type(key, __u32);
+	__type(value, __u64);
+} lru_map SEC(".maps");
+
+int hits;
+
+SEC("perf_event")
+int oncpu(void *ctx)
+{
+	/*
+	 * Key range deliberately wider than max_entries to force LRU
+	 * eviction on every other update.
+	 */
+	__u32 key = bpf_get_prandom_u32() % 128;
+	bool do_update = bpf_get_prandom_u32() & 1;
+	__u64 val = 1;
+
+	if (do_update)
+		bpf_map_update_elem(&lru_map, &key, &val, BPF_ANY);
+	else
+		bpf_map_delete_elem(&lru_map, &key);
+	__sync_fetch_and_add(&hits, 1);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";

-- 
2.53.0-Meta

Re: [PATCH RFC bpf-next 1/3] bpf: Fix NMI/tracepoint re-entry deadlock on lru locks

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] A race condition between `__local_list_flush` and an NMI calling `bpf_common_lru_push_free` can cause a deleted LRU node to be permanently leaked, because `__bpf_lru_node_move_in` unconditionally clears the `pending_free` flag.
--

commit c004ee038ff11ff6f1bdd2bb613d6dc9eddf7165
Author: Mykyta Yatsenko <yatsenko@meta.com>
bpf: Fix NMI/tracepoint re-entry deadlock on lru locks

This commit converts LRU lock sites to rqspinlock_t to prevent AA-deadlocks
when NMI and tracepoint BPF programs re-enter the per-CPU or global LRU
lock. It also adds a recovery path for each failure window to ensure nodes
are not leaked on contention or deadlock.

Link: https://lore.kernel.org/bpf/CAPPBnEYO4R+m+SpVc2gNj_x31R6fo1uJvj2bK2YS1P09GWT6kQ@mail.gmail.com/
Link: https://lore.kernel.org/bpf/CAPPBnEZmFA3ab8Uc=PEm0bdojZy=7T_F5_+eyZSHyZR3MBG4Vw@mail.gmail.com/
Link: https://lore.kernel.org/bpf/20251030030010.95352-1-dongml2@chinatelecom.cn/
Link: https://lore.kernel.org/bpf/20260119142120.28170-1-leon.hwang@linux.dev/

> diff --git a/kernel/bpf/bpf_lru_list.c b/kernel/bpf/bpf_lru_list.c
> index e7a2fc60523f6..a5faf4464d889 100644
> --- a/kernel/bpf/bpf_lru_list.c
> +++ b/kernel/bpf/bpf_lru_list.c

[ ... ]

> @@ -87,6 +73,7 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l,
>  	bpf_lru_list_count_inc(l, tgt_type);
>  	node->type = tgt_type;
>  	bpf_lru_node_clear_ref(node);
> +	WRITE_ONCE(node->pending_free, 0);

[Severity: High]
Can a race condition here cause an LRU node to be leaked if an NMI
interrupts __local_list_flush()?

If an NMI occurs exactly after __local_list_flush() reads
node->pending_free as 0:

__local_list_flush()
    if (READ_ONCE(node->pending_free))
        // skipped
    else
        // interrupted here by NMI

And the NMI's BPF program deletes this element from the hash table, calling
bpf_common_lru_push_free(). It fails to acquire the lock because
__local_list_flush()'s caller already holds it:

NMI: bpf_common_lru_push_free()
    if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
        WRITE_ONCE(node->pending_free, 1);
        return;
    }

When the NMI completes and __local_list_flush() resumes, it will call
__bpf_lru_node_move_in() which unconditionally clears the flag:

__bpf_lru_node_move_in()
    WRITE_ONCE(node->pending_free, 0);

Will this wipe the pending_free state set by the NMI? Since the node
is already deleted from the hash table, future shrink scans would see
del_from_htab() return false and skip the node, potentially permanently
leaking it and degrading the LRU map over time.

>  	list_move(&node->list, &l->lists[tgt_type]);
>  }

[ ... ]

> @@ -290,8 +280,10 @@ static void __local_list_flush(struct bpf_lru_list *l,
>  	struct bpf_lru_node *node, *tmp_node;
>  
>  	list_for_each_entry_safe_reverse(node, tmp_node,
> -					 local_pending_list(loc_l), list) {
> -		if (bpf_lru_node_is_ref(node))
> +					 &loc_l->pending_list, list) {
> +		if (READ_ONCE(node->pending_free))
> +			__bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_FREE);
> +		else if (bpf_lru_node_is_ref(node))
>  			__bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE);
>  		else
>  			__bpf_lru_node_move_in(l, node,

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260528-lru_map_spin-v1-0-4f52223170cf@meta.com?part=1

Re: [PATCH RFC bpf-next 1/3] bpf: Fix NMI/tracepoint re-entry deadlock on lru locks

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]

> diff --git a/kernel/bpf/bpf_lru_list.c b/kernel/bpf/bpf_lru_list.c

[ ... ]

> @@ -466,24 +478,36 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
>  	do {
>  		steal_loc_l = per_cpu_ptr(clru->local_list, steal);
>
> -		raw_spin_lock_irqsave(&steal_loc_l->lock, flags);
> -
> -		node = __local_list_pop_free(steal_loc_l);
> -		if (!node)
> -			node = __local_list_pop_pending(lru, steal_loc_l);
> -
> -		raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
> +		if (!raw_res_spin_lock_irqsave(&steal_loc_l->lock, flags)) {
> +			node = __local_list_pop_free(steal_loc_l);
> +			if (!node)
> +				node = __local_list_pop_pending(lru, steal_loc_l);
> +			raw_res_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
> +		}
>
>  		steal = cpumask_next_wrap(steal, cpu_possible_mask);
>  	} while (!node && steal != first_steal);
>
>  	loc_l->next_steal = steal;
>
> -	if (node) {
> -		raw_spin_lock_irqsave(&loc_l->lock, flags);
> -		__local_list_add_pending(lru, loc_l, cpu, node, hash);
> -		raw_spin_unlock_irqrestore(&loc_l->lock, flags);
> +	if (!node)
> +		return NULL;
> +
> +	if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
> +		/*
> +		 * The local pending lock can't be acquired (rqspinlock AA
> +		 * or timeout). Return the stolen node to the per-CPU
> +		 * free_llist instead of orphaning it; the next pop_free on
> +		 * this CPU will pick it up.
> +		 */
> +		node->type = BPF_LRU_LOCAL_LIST_T_FREE;
> +		bpf_lru_node_clear_ref(node);
> +		WRITE_ONCE(node->pending_free, 0);
> +		llist_add(&node->llist, &loc_l->free_llist);
> +		return NULL;
>  	}
> +	__local_list_add_pending(lru, loc_l, cpu, node, hash);
> +	raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);

This implementation adds the free_llist recovery mechanism to handle
post-steal lock failures. The stolen node is published to the lockless
per-CPU free_llist when the local pending lock cannot be acquired, which
prevents the node leak that plagued Menglong's version.

Can you confirm that this recovery path covers all the node leak scenarios
identified in the October 2025 review? The commit message describes the
recovery mechanism, and the code shows nodes being added to free_llist on
post-steal lock failure, which should be picked up by subsequent
__local_list_pop_free calls.

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26594280936

Re: [PATCH RFC bpf-next 3/3] selftests/bpf: Stress LRU rqspinlock recovery paths

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [High] The `drain_and_refill` function fails to detect partial node leaks because LRU map eviction transparently hides lost capacity.
- [High] Stack buffer overflow when using `CPU_SET` on systems with a large number of CPUs.
- [Medium] Using glibc's `rand()` in concurrent userspace threads causes severe lock contention, serializing the stress threads and defeating the test's purpose.
- [Medium] Unchecked return value from `libbpf_num_possible_cpus()` is used to allocate Variable Length Arrays (VLAs).
- [Medium] Double close of the PMU perf_event file descriptors due to missing ownership transfer accounting.
--

commit de08c9fe543d53ce17c858bad1ef0cdae6fb41af
Author: Mykyta Yatsenko <yatsenko@meta.com>

selftests/bpf: Stress LRU rqspinlock recovery paths

This commit replaces the single-CPU NMI re-entry reproducer with a
parameterized stress harness to exercise LRU rqspinlock recovery paths.
It runs subtests against common LRU, per-CPU LRU lists, and per-CPU LRU
maps using multiple userspace threads and perf_event NMI BPF programs.

> diff --git a/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c b/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c
> new file mode 100644
> index 0000000000000..15b6606fef4d4
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/lru_lock_nmi.c

[ ... ]

> +static void *hammer_thread(void *p)
> +{
> +	struct hammer_arg *a = p;
> +	__u64 val[a->nr_cpus];
> +	cpu_set_t cs;
> +	__u32 key;
> +
> +	memset(val, 0, sizeof(val));
> +	CPU_ZERO(&cs);
> +	CPU_SET(a->cpu, &cs);

[Severity: High]
Could this CPU_SET call write out of bounds on the stack?

The cpu_set_t structure is statically sized and often defaults to supporting
1024 CPUs. On machines with 1024 or more CPUs, setting a bit beyond
CPU_SETSIZE might corrupt memory. Would it be safer to use dynamically
allocated CPU sets via CPU_ALLOC and CPU_SET_S here?

> +	pthread_setaffinity_np(pthread_self(), sizeof(cs), &cs);
> +
> +	while (get_time_ns() < a->deadline_ns) {
> +		bool do_update = rand() & 1;
> +
> +		key = rand() % KEY_RANGE;

[Severity: Medium]
Will the use of glibc rand serialize these userspace threads?

Because rand uses a global internal lock to protect its PRNG state, calling
it in a tight loop across multiple threads pinned to different CPUs could
cause massive userspace contention. This might serialize the threads before
they enter the kernel and defeat the test's goal of stressing cross-CPU LRU
lock contention. Could a thread-local PRNG like rand_r be used instead?

> +		if (do_update)
> +			bpf_map_update_elem(a->map_fd, &key, val, BPF_ANY);
> +		else
> +			bpf_map_delete_elem(a->map_fd, &key);
> +	}
> +	return NULL;
> +}

[ ... ]

> +static int drain_and_refill(int map_fd, int nr_cpus)
> +{
> +	__u64 val[nr_cpus];
> +	__u32 key;
> +
> +	memset(val, 0, sizeof(val));
> +	for (key = 0; key < KEY_RANGE; key++)
> +		bpf_map_delete_elem(map_fd, &key);
> +
> +	for (key = 0; key < MAP_ENTRIES; key++)
> +		if (bpf_map_update_elem(map_fd, &key, val, BPF_ANY))
> +			return -ENOMEM;

[Severity: High]
Does this check reliably detect partial leaks in an LRU map?

Since this is an LRU map, if the usable capacity shrinks (for example, to 63
due to a leaked node), the 64th insertion will likely not fail with -ENOMEM.
Instead, it will evict the first inserted element and succeed.

Should the test perform lookups for all inserted keys after this loop finishes
to ensure none were prematurely evicted, in order to accurately verify that no
capacity was lost?

> +	return 0;
> +}
> +
> +static void run_variant(enum bpf_map_type type, __u32 map_flags,
> +			const char *name)
> +{
> +	struct perf_event_attr attr = {
> +		.size = sizeof(attr),
> +		.type = PERF_TYPE_HARDWARE,
> +		.config = PERF_COUNT_HW_CPU_CYCLES,
> +		.freq = 1,
> +	};
> +	int nr_cpus = libbpf_num_possible_cpus();
> +	int pmu_fds[nr_cpus];
> +	struct bpf_link *links[nr_cpus];

[Severity: Medium]
What happens if libbpf_num_possible_cpus returns a negative error code?

Passing a negative size to these variable length arrays would result in
undefined behavior and immediate stack pointer corruption. Should we check
if nr_cpus is positive before using it to declare these arrays?

[ ... ]

> +	for (i = 0; i < nr_cpus; i++) {
> +		pmu_fds[i] = syscall(__NR_perf_event_open, &attr, -1, i,
> +				     -1, 0);
> +		if (pmu_fds[i] < 0) {
> +			if (i == 0 &&
> +			    (errno == ENOENT || errno == EOPNOTSUPP)) {
> +				test__skip();
> +				goto cleanup;
> +			}
> +			continue;
> +		}
> +		links[i] = bpf_program__attach_perf_event(skel->progs.oncpu,
> +							  pmu_fds[i]);
> +		if (!links[i]) {
> +			close(pmu_fds[i]);
> +			pmu_fds[i] = -1;
> +		}
> +	}

[Severity: Medium]
Could this result in a double close of the PMU file descriptors?

When bpf_program__attach_perf_event successfully attaches, the returned
bpf_link takes ownership of the perf event file descriptor and will close it
automatically during bpf_link__destroy.

Since pmu_fds[i] isn't set to -1 on a successful attachment, the cleanup loop
later in the function will call close on pmu_fds[i] after bpf_link__destroy
has already closed it.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260528-lru_map_spin-v1-0-4f52223170cf@meta.com?part=3