[PATCH 0/3] bpf: Make sleepable arena paths use sleepable alloc_pages

[PATCH 0/3] bpf: Make sleepable arena paths use sleepable alloc_pages

[Emil Tsalapatis]

The arena_alloc_pages() call takes a sleepable argument based on whether
its caller is a sleepable BPF function. This flag, along with the context
the kfunc is called in, decides whether the call will try to fulfill the
allocation using the regular or the _nolock variant of the alloc_pages
API, by means of bpf_map_alloc_pages().

However, the arena_alloc_pages() call currently only makes allocations
inside an IRQ-disabled critical section. This forces all allocations to
use the _nolock() API, which may eagerly fail where the regular variant
would eventually succeed. There have been reports of this happening for
sched-ext schedulers.

Restructure arena_alloc_pages() to use the _nolock() page allocation API
only when necessary. This requires moving allocations outside of the 
spinlock critical section for sleepable calls, which in turn requires
slightly different logic in the allocation path. Make a separate
sleepable code path that implements this logic.

Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>

Emil Tsalapatis (3):
  bpf: Factor out nonsleepable arena allocation logic
  bpf: Add sleepable arena page allocation path
  selftests/bpf: Test large allocations for both sleepable/nonsleepable
    arena users

 include/linux/bpf.h                           |   6 +
 kernel/bpf/arena.c                            | 188 +++++++++++++-----
 kernel/bpf/syscall.c                          |   8 +-
 .../bpf/progs/verifier_arena_large.c          |  27 ++-
 4 files changed, 175 insertions(+), 54 deletions(-)

-- 
2.54.0

[PATCH 1/3] bpf: Factor out nonsleepable arena allocation logic

[Emil Tsalapatis]

The bpf_arena_alloc_pages() is supposed to transparently choose whether
to use _nolock() allocation variants or not depending on the caller's
context. This is implemented internally by testing the context a
__bpf_alloc_page call is made in. However, the implementation of
bpf_arena_alloc_pages() forces the use of the _nolock() variants by
calling __bpf_alloc_page after disabling IRQs by taking a spinlock.
This in turn causes spurious allocation failures for some workloads
like scx schedulers due to ZONE_NORMAL falling below the minimum
watermark, even though the caller is sleepable could wait until
there is memory available.

To fix this we need to call __bpf_alloc_page outside of the spinlock
critical section when the allocation is called from a sleepable BPF
function. As a first step, refactor the existing logic to simplify
adding the path in the next commit.

No functional changes in this patch. The followup will add a proper
sleepable path.

Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
---
 kernel/bpf/arena.c | 126 +++++++++++++++++++++++++++------------------
 1 file changed, 77 insertions(+), 49 deletions(-)

diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index 7b6847200b43..da356989786a 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -661,68 +661,57 @@ static u64 clear_lo32(u64 val)
 	return val & ~(u64)~0U;
 }
 
-/*
- * Allocate pages and vmap them into kernel vmalloc area.
- * Later the pages will be mmaped into user space vma.
- */
-static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt, int node_id,
-			      bool sleepable)
+
+static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, long *pgoff)
+{
+	int ret;
+
+	/* Special case where user is requesting specific range. */
+	if (uaddr) {
+		ret = is_range_tree_set(&arena->rt, *pgoff, page_cnt);
+		if (ret)
+			return ret;
+		return range_tree_clear(&arena->rt, *pgoff, page_cnt);
+	}
+
+	ret = range_tree_find(&arena->rt, page_cnt);
+	if (ret < 0)
+		return ret;
+
+	*pgoff = ret;
+
+	return range_tree_clear(&arena->rt, *pgoff, page_cnt);
+}
+
+static long arena_alloc_pages_internal(struct bpf_arena *arena, long page_cnt,
+		long uaddr, long pgoff, int node_id, bool sleepable)
 {
-	/* user_vm_end/start are fixed before bpf prog runs */
-	long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT;
 	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
-	struct mem_cgroup *new_memcg, *old_memcg;
 	struct apply_range_data data;
 	struct page **pages = NULL;
 	long remaining, mapped = 0;
 	long alloc_pages;
 	unsigned long flags;
-	long pgoff = 0;
 	u32 uaddr32;
 	int ret, i;
 
-	if (node_id != NUMA_NO_NODE &&
-	    ((unsigned int)node_id >= nr_node_ids || !node_online(node_id)))
-		return 0;
-
-	if (page_cnt > page_cnt_max)
-		return 0;
-
-	if (uaddr) {
-		if (uaddr & ~PAGE_MASK)
-			return 0;
-		pgoff = compute_pgoff(arena, uaddr);
-		if (pgoff > page_cnt_max - page_cnt)
-			/* requested address will be outside of user VMA */
-			return 0;
-	}
-
-	bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
 	/* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. */
 	alloc_pages = min(page_cnt, KMALLOC_MAX_CACHE_SIZE / sizeof(struct page *));
 	pages = kmalloc_nolock(alloc_pages * sizeof(struct page *), __GFP_ACCOUNT, NUMA_NO_NODE);
-	if (!pages) {
-		bpf_map_memcg_exit(old_memcg, new_memcg);
+	if (!pages)
 		return 0;
-	}
+
 	data.arena = arena;
 	data.pages = pages;
 
 	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
 		goto out_free_pages;
 
-	if (uaddr) {
-		ret = is_range_tree_set(&arena->rt, pgoff, page_cnt);
-		if (ret)
-			goto out_unlock_free_pages;
-		ret = range_tree_clear(&arena->rt, pgoff, page_cnt);
-	} else {
-		ret = pgoff = range_tree_find(&arena->rt, page_cnt);
-		if (pgoff >= 0)
-			ret = range_tree_clear(&arena->rt, pgoff, page_cnt);
+	ret = arena_adjust_tree(arena, uaddr, page_cnt, &pgoff);
+	if (ret) {
+		raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+		goto out_free_pages;
 	}
-	if (ret)
-		goto out_unlock_free_pages;
 
 	remaining = page_cnt;
 	uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);
@@ -735,7 +724,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 
 		ret = bpf_map_alloc_pages(&arena->map, node_id, this_batch, pages);
 		if (ret)
-			goto out;
+			goto out_unmap;
 
 		/*
 		 * Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1
@@ -754,31 +743,70 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 			mapped += data.i;
 			for (i = data.i; i < this_batch; i++)
 				free_pages_nolock(pages[i], 0);
-			goto out;
+			goto out_unmap;
 		}
 
 		mapped += this_batch;
 		remaining -= this_batch;
 	}
+
 	flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
 	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+
 	kfree_nolock(pages);
-	bpf_map_memcg_exit(old_memcg, new_memcg);
+
 	return clear_lo32(arena->user_vm_start) + uaddr32;
-out:
+
+out_unmap:
 	range_tree_set(&arena->rt, pgoff + mapped, page_cnt - mapped);
 	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
 	if (mapped) {
 		flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
 		arena_free_pages(arena, uaddr32, mapped, sleepable);
 	}
-	goto out_free_pages;
-out_unlock_free_pages:
-	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+
 out_free_pages:
 	kfree_nolock(pages);
-	bpf_map_memcg_exit(old_memcg, new_memcg);
 	return 0;
+
+}
+
+/*
+ * Allocate pages and vmap them into kernel vmalloc area.
+ * Later the pages will be mmaped into user space vma.
+ */
+static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt, int node_id,
+			      bool sleepable)
+{
+	/* user_vm_end/start are fixed before bpf prog runs */
+	long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT;
+	struct mem_cgroup *new_memcg, *old_memcg;
+	long addr;
+	long pgoff = 0;
+
+	if (node_id != NUMA_NO_NODE &&
+	    ((unsigned int)node_id >= nr_node_ids || !node_online(node_id)))
+		return 0;
+
+	if (page_cnt > page_cnt_max)
+		return 0;
+
+	if (uaddr) {
+		if (uaddr & ~PAGE_MASK)
+			return 0;
+		pgoff = compute_pgoff(arena, uaddr);
+		if (pgoff > page_cnt_max - page_cnt)
+			/* requested address will be outside of user VMA */
+			return 0;
+	}
+
+	bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
+
+	addr = arena_alloc_pages_internal(arena, page_cnt, uaddr, pgoff, node_id, sleepable);
+
+	bpf_map_memcg_exit(old_memcg, new_memcg);
+
+	return addr;
 }
 
 /*
-- 
2.54.0

Re: [PATCH 1/3] bpf: Factor out nonsleepable arena allocation logic

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2840 bytes --]

> commit 434fc482ce1027a7af536690a218094b882f2ad2
> Author: Emil Tsalapatis <emil@etsalapatis.com>
> Date:   Mon Aug 24 04:25:28 2026 -0400
>
>     bpf: Factor out nonsleepable arena allocation logic
>
>     The bpf_arena_alloc_pages() is supposed to transparently choose whether
>     to use _nolock() allocation variants or not depending on the caller's
>     context. This is implemented internally by testing the context a
>     __bpf_alloc_page call is made in. However, the implementation of
>     bpf_arena_alloc_pages() forces the use of the _nolock() variants by
>     calling __bpf_alloc_page after disabling IRQs by taking a spinlock.
>     This in turn causes spurious allocation failures for some workloads
>     like scx schedulers due to ZONE_NORMAL falling below the minimum
>     watermark, even though the caller is sleepable could wait until
>     there is memory available.

This isn't a bug, but could this sentence be reworded (e.g. "even though a
sleepable caller could wait until memory becomes available") so the
motivation reads cleanly?

> diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
> index 7b6847200b43..da356989786a 100644
> --- a/kernel/bpf/arena.c
> +++ b/kernel/bpf/arena.c
> @@ -661,68 +661,57 @@ static u64 clear_lo32(u64 val)
>  	return val & ~(u64)~0U;
>  }
>
> -/*
> - * Allocate pages and vmap them into kernel vmalloc area.
> - * Later the pages will be mmaped into user space vma.
> - */
> -static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt, int node_id,
> -			      bool sleepable)
> +
> +static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, long *pgoff)
> +{
> +	int ret;
> +
> +	/* Special case where user is requesting specific range. */
> +	if (uaddr) {
> +		ret = is_range_tree_set(&arena->rt, *pgoff, page_cnt);
> +		if (ret)
> +			return ret;
> +		return range_tree_clear(&arena->rt, *pgoff, page_cnt);
> +	}
> +
> +	ret = range_tree_find(&arena->rt, page_cnt);
> +	if (ret < 0)
> +		return ret;
> +
> +	*pgoff = ret;
> +
> +	return range_tree_clear(&arena->rt, *pgoff, page_cnt);
> +}
> +
> +static long arena_alloc_pages_internal(struct bpf_arena *arena, long page_cnt,
> +		long uaddr, long pgoff, int node_id, bool sleepable)

This isn't a bug, but would it read better to keep the existing
(uaddr, page_cnt) argument order in arena_alloc_pages_internal() so all
three functions agree, given both parameters are plain long?

arena_adjust_tree() uses (uaddr, page_cnt) and arena_alloc_pages() also
uses (uaddr, page_cnt), but arena_alloc_pages_internal() uses
(page_cnt, uaddr).


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32709208315

[PATCH 2/3] bpf: Add sleepable arena page allocation path

[Emil Tsalapatis]

The bpf_arena_alloc_pages() function currently only allocates pages
inside a spinlock critical section with IRQs off. This forces the use
of alloc_pages_nolock() in the BPF allocator, even when the caller is
a sleepable BPF function. This in turn causes allocation failures even
in cases where falling into the allocator slow path and possibly
sleeping would eventually succeed. This can be triggered consistently
by heavy BPF arena users like scx.

Add a separate arena page allocation path just for sleepable callers.
The path preallocates the arena memory to be added to the tree before
taking the critical section.

Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
---
 include/linux/bpf.h  |  6 ++++
 kernel/bpf/arena.c   | 76 +++++++++++++++++++++++++++++++++++++++++---
 kernel/bpf/syscall.c |  8 +----
 3 files changed, 79 insertions(+), 11 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index ffa5626411ac..d15ed7a3879b 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);
 int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
 				   struct bpf_dynptr *ptr__uninit);
 
+static inline bool is_bpf_alloc_nonsleepable(void)
+{
+	return preempt_count() > 0 || irqs_disabled() ||
+		IS_ENABLED(CONFIG_PREEMPT_RT);
+}
+
 #if defined(CONFIG_MMU) && defined(CONFIG_64BIT)
 void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
 					  u64 flags);
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index da356989786a..126ddae763f9 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -683,8 +683,73 @@ static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt,
 	return range_tree_clear(&arena->rt, *pgoff, page_cnt);
 }
 
-static long arena_alloc_pages_internal(struct bpf_arena *arena, long page_cnt,
-		long uaddr, long pgoff, int node_id, bool sleepable)
+static long arena_alloc_pages_sleepable(struct bpf_arena *arena, long page_cnt,
+		long uaddr, long pgoff, int node_id)
+{
+	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
+	struct apply_range_data data;
+	struct page **pages = NULL;
+	unsigned long flags;
+	u32 uaddr32;
+	int ret, i;
+
+	pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL_ACCOUNT);
+	if (!pages)
+		return 0;
+
+	ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages);
+	if (ret) {
+		kvfree(pages);
+		return 0;
+	}
+
+	data.i = 0;
+	data.pages = pages;
+	data.arena = arena;
+
+	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
+		goto out_free_pages;
+
+	ret = arena_adjust_tree(arena, uaddr, page_cnt, &pgoff);
+	if (ret) {
+		raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+		goto out_free_pages;
+	}
+
+	uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);
+
+	ret = apply_to_page_range(&init_mm, kern_vm_start + uaddr32,
+			page_cnt << PAGE_SHIFT, apply_range_set_cb, &data);
+
+	if (ret)
+		goto out_unmap;
+
+	flush_vmap_cache(kern_vm_start + uaddr32, page_cnt << PAGE_SHIFT);
+	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+
+	kvfree(pages);
+
+	return clear_lo32(arena->user_vm_start) + uaddr32;
+
+out_unmap:
+	/* data.i pages were mapped, undo only those who failed to map */
+	flush_vmap_cache(kern_vm_start + uaddr32, data.i << PAGE_SHIFT);
+	range_tree_set(&arena->rt, pgoff + data.i, page_cnt - data.i);
+
+	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
+
+	arena_free_pages(arena, uaddr32, data.i, true);
+
+out_free_pages:
+	for (i = data.i; i < page_cnt; i++)
+		__free_page(pages[i]);
+
+	kvfree(pages);
+	return 0;
+}
+
+static long arena_alloc_pages_non_sleepable(struct bpf_arena *arena, long page_cnt,
+		long uaddr, long pgoff, int node_id)
 {
 	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
 	struct apply_range_data data;
@@ -762,7 +827,7 @@ static long arena_alloc_pages_internal(struct bpf_arena *arena, long page_cnt,
 	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
 	if (mapped) {
 		flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
-		arena_free_pages(arena, uaddr32, mapped, sleepable);
+		arena_free_pages(arena, uaddr32, mapped, false);
 	}
 
 out_free_pages:
@@ -802,7 +867,10 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
 
 	bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
 
-	addr = arena_alloc_pages_internal(arena, page_cnt, uaddr, pgoff, node_id, sleepable);
+	if (sleepable && !is_bpf_alloc_nonsleepable())
+		addr = arena_alloc_pages_sleepable(arena, page_cnt, uaddr, pgoff, node_id);
+	else
+		addr = arena_alloc_pages_non_sleepable(arena, page_cnt, uaddr, pgoff, node_id);
 
 	bpf_map_memcg_exit(old_memcg, new_memcg);
 
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 6874ba1424af..700ed7a99579 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -596,15 +596,9 @@ static void bpf_map_release_memcg(struct bpf_map *map)
 }
 #endif
 
-static bool can_alloc_pages(void)
-{
-	return preempt_count() == 0 && !irqs_disabled() &&
-		!IS_ENABLED(CONFIG_PREEMPT_RT);
-}
-
 static struct page *__bpf_alloc_page(int nid)
 {
-	if (!can_alloc_pages())
+	if (is_bpf_alloc_nonsleepable())
 		return alloc_pages_nolock(__GFP_ACCOUNT, nid, 0);
 
 	return alloc_pages_node(nid,
-- 
2.54.0

Re: [PATCH 2/3] bpf: Add sleepable arena page allocation path

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Hard lockup via integer overflow: `data.i << PAGE_SHIFT` passes an astronomically large negative size to `flush_vmap_cache` on mapping failure.
--

--- Patch [2]: [PATCH 2/3] bpf: Add sleepable arena page allocation path ---
commit a93c0f542ad1ecf8f00cde8cf86699ce952035a7
Author: Emil Tsalapatis <emil@etsalapatis.com>

bpf: Add sleepable arena page allocation path

This commit adds a separate sleepable allocation path for BPF arena memory.
It preallocates memory before taking the critical section to prevent
frequent allocation failures from sleepable BPF contexts.

> diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
> index da356989786a0..126ddae763f9a 100644
> --- a/kernel/bpf/arena.c
> +++ b/kernel/bpf/arena.c

[ ... ]

> +static long arena_alloc_pages_sleepable(struct bpf_arena *arena, long page_cnt,
> +		long uaddr, long pgoff, int node_id)
> +{

[ ... ]

> +out_unmap:
> +	/* data.i pages were mapped, undo only those who failed to map */
> +	flush_vmap_cache(kern_vm_start + uaddr32, data.i << PAGE_SHIFT);

[Severity: High]
Can this shift operation overflow?

If apply_to_page_range() maps a large number of pages (e.g. over 524,287)
before failing, data.i << PAGE_SHIFT will overflow since data.i is a
signed 32-bit int.

Because flush_vmap_cache() expects an unsigned long size, a negative value
here would sign-extend to a massive 64-bit size, potentially causing a CPU
stall or lockup in the flush routine.

Would casting data.i to unsigned long before the shift prevent this?

> +	range_tree_set(&arena->rt, pgoff + data.i, page_cnt - data.i);
> +
> +	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260824082530.47553-1-emil@etsalapatis.com?part=2

Re: [PATCH 2/3] bpf: Add sleepable arena page allocation path

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 8818 bytes --]

> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index b3cd28d9e3b5..fcb21c29aef2 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);
>  int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
>  				   struct bpf_dynptr *ptr__uninit);
>
> +static inline bool is_bpf_alloc_nonsleepable(void)
> +{
> +	return preempt_count() > 0 || irqs_disabled() ||
> +		IS_ENABLED(CONFIG_PREEMPT_RT);
> +}
> +
>  #if defined(CONFIG_MMU) && defined(CONFIG_64BIT)
>  void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
>  					  u64 flags);
> diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
> index da356989786a..126ddae763f9 100644
> --- a/kernel/bpf/arena.c
> +++ b/kernel/bpf/arena.c
> @@ -683,8 +683,73 @@ static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt,
>  	return range_tree_clear(&arena->rt, *pgoff, page_cnt);
>  }
>
> -static long arena_alloc_pages_internal(struct bpf_arena *arena, long page_cnt,
> -		long uaddr, long pgoff, int node_id, bool sleepable)
> +static long arena_alloc_pages_sleepable(struct bpf_arena *arena, long page_cnt,
> +		long uaddr, long pgoff, int node_id)
> +{
> +	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
> +	struct apply_range_data data;
> +	struct page **pages = NULL;
> +	unsigned long flags;
> +	u32 uaddr32;
> +	int ret, i;
> +
> +	pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL_ACCOUNT);
> +	if (!pages)
> +		return 0;
> +
> +	ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages);
> +	if (ret) {
> +		kvfree(pages);
> +		return 0;
> +	}
> +
> +	data.i = 0;
> +	data.pages = pages;
> +	data.arena = arena;
> +
> +	if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
> +		goto out_free_pages;
> +
> +	ret = arena_adjust_tree(arena, uaddr, page_cnt, &pgoff);
> +	if (ret) {
> +		raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
> +		goto out_free_pages;
> +	}
> +
> +	uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);
> +
> +	ret = apply_to_page_range(&init_mm, kern_vm_start + uaddr32,
> +			page_cnt << PAGE_SHIFT, apply_range_set_cb, &data);
> +
> +	if (ret)
> +		goto out_unmap;
> +
> +	flush_vmap_cache(kern_vm_start + uaddr32, page_cnt << PAGE_SHIFT);
> +	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
> +
> +	kvfree(pages);
> +
> +	return clear_lo32(arena->user_vm_start) + uaddr32;
> +
> +out_unmap:
> +	/* data.i pages were mapped, undo only those who failed to map */
> +	flush_vmap_cache(kern_vm_start + uaddr32, data.i << PAGE_SHIFT);
> +	range_tree_set(&arena->rt, pgoff + data.i, page_cnt - data.i);
> +
> +	raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
> +
> +	arena_free_pages(arena, uaddr32, data.i, true);

This isn't a bug, but does the comment describe the right thing?

The comment says "undo only those who failed to map", but the code below
unmaps the pages that succeeded:

  arena_free_pages(arena, uaddr32, data.i, true);

This unmaps exactly data.i pages starting at uaddr32, which are the ones
that apply_range_set_cb() installed (data.i is incremented once per
successful PTE). The pages that failed to map get freed in the
out_free_pages loop below:

  for (i = data.i; i < page_cnt; i++)
          __free_page(pages[i]);

Should the comment say something like "data.i pages were mapped, undo
those" instead?

> +
> +out_free_pages:
> +	for (i = data.i; i < page_cnt; i++)
> +		__free_page(pages[i]);
> +
> +	kvfree(pages);
> +	return 0;
> +}
> +
> +static long arena_alloc_pages_non_sleepable(struct bpf_arena *arena, long page_cnt,
> +		long uaddr, long pgoff, int node_id)
>  {
>  	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
>  	struct apply_range_data data;

[ ... ]

> @@ -802,7 +867,10 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
>
>  	bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
>
> -	addr = arena_alloc_pages_internal(arena, page_cnt, uaddr, pgoff, node_id, sleepable);
> +	if (sleepable && !is_bpf_alloc_nonsleepable())
> +		addr = arena_alloc_pages_sleepable(arena, page_cnt, uaddr, pgoff, node_id);
> +	else
> +		addr = arena_alloc_pages_non_sleepable(arena, page_cnt, uaddr, pgoff, node_id);

Can this dispatch block while in an RCU read-side critical section?

The sleepable parameter is hardcoded to true by the kfunc entry point
in kernel/bpf/arena.c:

  void *bpf_arena_alloc_pages(struct bpf_map *map, void *addr__ign,
                              u32 page_cnt, int node_id) {
    return (void *)arena_alloc_pages(arena, (long)addr__ign,
                                     page_cnt, node_id, true);
  }

and that kfunc is registered without KF_SLEEPABLE:

  BTF_ID_FLAGS(func, bpf_arena_alloc_pages,
               KF_ARENA_RET | KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)

The verifier checks in kernel/bpf/verifier.c specifically prevent
non-sleepable programs from calling sleepable kfuncs:

  sleepable = bpf_is_kfunc_sleepable(&meta);
  if (sleepable && !in_sleepable(env)) {
          verbose(env, "program must be sleepable to call sleepable "
                  "kfunc %s\n", func_name);
          return -EACCES;
  }

but that check never triggers because bpf_arena_alloc_pages() is not
tagged KF_SLEEPABLE. So non-sleepable programs can call it.

is_bpf_alloc_nonsleepable() checks for:

  preempt_count() > 0 || irqs_disabled() || IS_ENABLED(CONFIG_PREEMPT_RT)

That detects hardirq/softirq/NMI/preempt-disabled/IRQ-disabled/RT
contexts, but it does not detect the RCU read-side critical section
that every non-sleepable BPF program runs inside.

Non-sleepable trampoline programs (fentry/fexit/fmod_ret/raw_tp/
struct_ops) enter via __bpf_prog_enter() in kernel/bpf/trampoline.c,
which calls rcu_read_lock_dont_migrate():

  rcu_read_lock_dont_migrate() {
    if (IS_ENABLED(CONFIG_PREEMPT_RCU))
            migrate_disable();
    rcu_read_lock();
  }

On CONFIG_PREEMPT_RCU=y kernels (the default for CONFIG_PREEMPT_DYNAMIC
distro kernels), rcu_read_lock() is __rcu_read_lock() which only
increments current->rcu_read_lock_nesting, and migrate_disable() uses
a scoped guard(preempt) so it does not leave preempt_count() elevated.

So with CONFIG_PREEMPT_RT=n, is_bpf_alloc_nonsleepable() returns false
and arena_alloc_pages_sleepable() is selected.

arena_alloc_pages_sleepable() then calls:

  kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL_ACCOUNT)
  bpf_map_alloc_pages() -> alloc_pages_node(..., GFP_KERNEL, ...)
  arena_free_pages() -> mutex_lock() and flush_tlb_kernel_range()

All of these can sleep. Documentation/RCU/whatisRCU.rst states "It is
illegal to block while in an RCU read-side critical section" even under
CONFIG_PREEMPT_RCU. Being preempted is permitted, but voluntarily
blocking is not.

The blocked reader is queued on rnp->blkd_tasks, so every concurrent
synchronize_rcu() on the system waits for the duration of the direct
reclaim. Under memory pressure (which the commit message says triggers
this path for "heavy BPF arena users like scx"), this yields unbounded
RCU grace-period extension, RCU CPU stall warnings, and delayed freeing
kernel-wide. The task also blocks with current->migration_disabled set
and rq->nr_pinned elevated, which stalls CPU hotplug for the same
duration.

The in-tree selftests confirm that non-sleepable programs are accepted:
tools/testing/selftests/bpf/progs/verifier_arena.c has several
SEC("socket") __success programs calling bpf_arena_alloc_pages(), and
tools/testing/selftests/bpf/progs/struct_ops_arena.c shows non-sleepable
SEC("struct_ops/...") callbacks bound to an arena map.

Should the sleepability decision be based on the verifier-known program
context rather than this runtime heuristic? For example, tagging
bpf_arena_alloc_pages() with KF_SLEEPABLE and adding a separate
non-sleepable kfunc, or plumbing the prog's sleepable flag down to
arena_alloc_pages()?

> diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> index 6874ba1424af..700ed7a99579 100644
> --- a/kernel/bpf/syscall.c
> +++ b/kernel/bpf/syscall.c
> @@ -596,15 +596,9 @@ static void bpf_map_release_memcg(struct bpf_map *map)
>  }
>  #endif
>
> -static bool can_alloc_pages(void)
> -{
> -	return preempt_count() == 0 && !irqs_disabled() &&
> -		!IS_ENABLED(CONFIG_PREEMPT_RT);
> -}
> -
>  static struct page *__bpf_alloc_page(int nid)
>  {
> -	if (!can_alloc_pages())
> +	if (is_bpf_alloc_nonsleepable())
>  		return alloc_pages_nolock(__GFP_ACCOUNT, nid, 0);
>
>  	return alloc_pages_node(nid,


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32709208315

Re: [PATCH 2/3] bpf: Add sleepable arena page allocation path

[Alexei Starovoitov]

On Mon Aug 24, 2026 at 1:25 AM PDT, Emil Tsalapatis wrote:
> The bpf_arena_alloc_pages() function currently only allocates pages
> inside a spinlock critical section with IRQs off. This forces the use
> of alloc_pages_nolock() in the BPF allocator, even when the caller is
> a sleepable BPF function. This in turn causes allocation failures even
> in cases where falling into the allocator slow path and possibly
> sleeping would eventually succeed. This can be triggered consistently
> by heavy BPF arena users like scx.
>
> Add a separate arena page allocation path just for sleepable callers.
> The path preallocates the arena memory to be added to the tree before
> taking the critical section.
>
> Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
> ---
>  include/linux/bpf.h  |  6 ++++
>  kernel/bpf/arena.c   | 76 +++++++++++++++++++++++++++++++++++++++++---
>  kernel/bpf/syscall.c |  8 +----
>  3 files changed, 79 insertions(+), 11 deletions(-)
>
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index ffa5626411ac..d15ed7a3879b 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -710,6 +710,12 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);
>  int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
>  				   struct bpf_dynptr *ptr__uninit);
>  
> +static inline bool is_bpf_alloc_nonsleepable(void)
> +{
> +	return preempt_count() > 0 || irqs_disabled() ||
> +		IS_ENABLED(CONFIG_PREEMPT_RT);
> +}
> +
>  #if defined(CONFIG_MMU) && defined(CONFIG_64BIT)
>  void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
>  					  u64 flags);
> diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
> index da356989786a..126ddae763f9 100644
> --- a/kernel/bpf/arena.c
> +++ b/kernel/bpf/arena.c
> @@ -683,8 +683,73 @@ static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt,
>  	return range_tree_clear(&arena->rt, *pgoff, page_cnt);
>  }
>  
> -static long arena_alloc_pages_internal(struct bpf_arena *arena, long page_cnt,
> -		long uaddr, long pgoff, int node_id, bool sleepable)
> +static long arena_alloc_pages_sleepable(struct bpf_arena *arena, long page_cnt,
> +		long uaddr, long pgoff, int node_id)
> +{
> +	u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
> +	struct apply_range_data data;
> +	struct page **pages = NULL;
> +	unsigned long flags;
> +	u32 uaddr32;
> +	int ret, i;
> +
> +	pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL_ACCOUNT);
> +	if (!pages)
> +		return 0;
> +
> +	ret = bpf_map_alloc_pages(&arena->map, node_id, page_cnt, pages);
> +	if (ret) {
> +		kvfree(pages);
> +		return 0;
> +	}
> +
> +	data.i = 0;
> +	data.pages = pages;
> +	data.arena = arena;
> +

Why have this split?
Always use an approach of allocating pages outside of the lock?

Considering severity of the issue bpf tree is probably appropriate.

pw-bot: cr

[PATCH 3/3] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users

[Emil Tsalapatis]

We now use different code paths in the internal allocator when
allocating arena memory, depending on whether the caller is sleepable
or not. These paths mostly differ functionally for large allocations,
so add extra testing for that case.

Signed-off-by: Emil Tsalapatis <emil@etsalapatis.com>
---
 .../bpf/progs/verifier_arena_large.c          | 27 ++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_large.c b/tools/testing/selftests/bpf/progs/verifier_arena_large.c
index 6ab8730d4878..e388a67d9129 100644
--- a/tools/testing/selftests/bpf/progs/verifier_arena_large.c
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_large.c
@@ -284,6 +284,7 @@ int big_alloc2(void *ctx)
 	return 0;
 }
 
+/* Nonsleepable because it binds to a socket program. */
 SEC("socket")
 __success __retval(0)
 int big_alloc3(void *ctx)
@@ -300,7 +301,31 @@ int big_alloc3(void *ctx)
 	 */
 	pages = bpf_arena_alloc_pages(&arena, NULL, 2051, NUMA_NO_NODE, 0);
 	if (!pages)
-		return 0;
+		return 1;
+
+	bpf_for(i, 0, 2051)
+			pages[i * PAGE_SIZE] = 123;
+	bpf_for(i, 0, 2051)
+			if (pages[i * PAGE_SIZE] != 123)
+				return i;
+
+	bpf_arena_free_pages(&arena, pages, 2051);
+#endif
+	return 0;
+}
+
+/* SYSCALL programs are always sleepable. */
+SEC("syscall")
+__success __retval(0)
+int big_alloc4(void *ctx)
+{
+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
+	char __arena *pages;
+	u64 i;
+
+	pages = bpf_arena_alloc_pages(&arena, NULL, 2051, NUMA_NO_NODE, 0);
+	if (!pages)
+		return 1;
 
 	bpf_for(i, 0, 2051)
 			pages[i * PAGE_SIZE] = 123;
-- 
2.54.0

Re: [PATCH 3/3] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Medium] The validation loop's error path incorrectly returns the loop index `i`, meaning a memory validation failure on the first page (`i == 0`) will falsely report as a successful test.

Pre-existing issues:
- [High] Hard lockup (watchdog timeout) in `arena_alloc_pages_non_sleepable` due to a massive page allocation loop executing with local interrupts disabled.
--

--- Patch [3]: [PATCH 3/3] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users ---
commit 97b45503d1d3323d8eb083b751d71f1bec8430fe
Author: Emil Tsalapatis <emil@etsalapatis.com>

selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users

We now use different code paths in the internal allocator when
allocating arena memory, depending on whether the caller is sleepable
or not. These paths mostly differ functionally for large allocations,
so add extra testing for that case.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_large.c b/tools/testing/selftests/bpf/progs/verifier_arena_large.c
> --- a/tools/testing/selftests/bpf/progs/verifier_arena_large.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_arena_large.c

[ ... ]

> @@ -300,7 +301,31 @@ int big_alloc3(void *ctx)
>  	 */
>  	pages = bpf_arena_alloc_pages(&arena, NULL, 2051, NUMA_NO_NODE, 0);

[Severity: High]
This is a pre-existing issue, but does this allocation of 2051 pages trigger
a hard lockup in the underlying allocator?

Looking at arena_alloc_pages_non_sleepable() in kernel/bpf/arena.c, the code
disables interrupts and runs a batch allocation loop:

kernel/bpf/arena.c:arena_alloc_pages_non_sleepable() {
    ...
    while (remaining) {
        long this_batch = min(remaining, alloc_pages);
        memset(pages, 0, this_batch * sizeof(struct page *));
        ret = bpf_map_alloc_pages(&arena->map, node_id, this_batch, pages);
        ...
        ret = apply_to_page_range(&init_mm, ...);
    }
    ...
}

Since this path is executed with a raw spinlock held and local interrupts
disabled, could a large page_cnt like 2051 cause a watchdog timeout?

>  	if (!pages)
> -		return 0;
> +		return 1;
> +
> +	bpf_for(i, 0, 2051)
> +			pages[i * PAGE_SIZE] = 123;
> +	bpf_for(i, 0, 2051)
> +			if (pages[i * PAGE_SIZE] != 123)
> +				return i;

[Severity: Medium]
Does this error path mask validation failures on the first page?

Since the program expects 0 for success (__success __retval(0)), if the 
validation fails on the first page (i == 0), the loop returns 0. This
perfectly aliases the expected success code and completely masks the failure.

Additionally, failing on the second page (i == 1) returns 1, which aliases
the allocation failure code above. 

Could this return a distinct error code instead of the loop index?

> +
> +	bpf_arena_free_pages(&arena, pages, 2051);
> +#endif
> +	return 0;
> +}
> +
> +/* SYSCALL programs are always sleepable. */
> +SEC("syscall")
> +__success __retval(0)
> +int big_alloc4(void *ctx)
> +{
> +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
> +	char __arena *pages;
> +	u64 i;
> +
> +	pages = bpf_arena_alloc_pages(&arena, NULL, 2051, NUMA_NO_NODE, 0);
> +	if (!pages)
> +		return 1;
>  
>  	bpf_for(i, 0, 2051)
>  			pages[i * PAGE_SIZE] = 123;
> +	bpf_for(i, 0, 2051)
> +			if (pages[i * PAGE_SIZE] != 123)
> +				return i;

[Severity: Medium]
Similarly, does returning the loop index here falsely report success if the
first page fails validation?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260824082530.47553-1-emil@etsalapatis.com?part=3

Re: [PATCH 3/3] selftests/bpf: Test large allocations for both sleepable/nonsleepable arena users

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2593 bytes --]

> diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_large.c b/tools/testing/selftests/bpf/progs/verifier_arena_large.c
> index 6ab8730d4878..e388a67d9129 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_arena_large.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_arena_large.c

[ ... ]

> @@ -284,6 +284,7 @@ int big_alloc2(void *ctx)
>  	return 0;
>  }
>
> +/* Nonsleepable because it binds to a socket program. */
>  SEC("socket")
>  __success __retval(0)
>  int big_alloc3(void *ctx)
> @@ -300,7 +301,31 @@ int big_alloc3(void *ctx)
>  	 */
>  	pages = bpf_arena_alloc_pages(&arena, NULL, 2051, NUMA_NO_NODE, 0);
>  	if (!pages)
> -		return 0;
> +		return 1;

Is it safe to make big_alloc3 fail the test when allocation fails?

The commit message says "add extra testing for that case," but the pre-existing
big_alloc3 deliberately tolerated allocation failure with 'return 0'. This
matters because big_alloc3 is SEC("socket"), and BPF_PROG_TEST_RUN executes
socket programs with local_bh_disable():

net/bpf/test_run.c:bpf_prog_test_run_skb()
    local_bh_disable();
    ...
    ret = bpf_test_run(skb, repeat, &retval, &duration, false);
    ...
    local_bh_enable();

That makes preempt_count() > 0, so is_bpf_alloc_nonsleepable() returns true
and bpf_arena_alloc_pages() selects arena_alloc_pages_non_sleepable() (as
intended, matching the commit message "different code paths ... depending on
whether the caller is sleepable or not").

The non-sleepable path can legitimately return NULL for reasons that aren't
kernel bugs. From d1f68a079731 in this same series:

    "This forces the use of alloc_pages_nolock() in the BPF allocator ...
     This in turn causes allocation failures even in cases where falling
     into the allocator slow path and possibly sleeping would eventually
     succeed."

So a 2051-page nolock allocation can fail when ZONE_NORMAL is below the min
watermark, or memory is fragmented, or the test runs on a busy CI VM. With
'return 1', big_alloc3 would report a false regression in those cases, even
though the allocator is working as designed.

Would it make sense to keep the tolerant 'return 0' for big_alloc3 (the
non-sleepable variant that uses alloc_pages_nolock), and reserve the strict
'return 1' for the new big_alloc4 (the sleepable variant that can use the
slow path)?

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/32709208315