* [PATCH bpf-next v3 01/14] libbpf: Make arena 4 Gbyte when max_entries is not set
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 02/14] selftests/bpf: Test the default size of arena Alexei Starovoitov
` (14 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
max_entries of an arena is its size in pages. The kernel rejects 0. Thus
each prog must select a number, and the same number is a different size on
a kernel with a different page size.
A small arena is not cheaper than a large one. For each arena the kernel
reserves 4 Gbyte of kernel addresses and allocates the page tables for all
of them: 8 Mbyte with pages of 4 Kbyte. max_entries changes neither. After
that only allocated pages use memory.
When max_entries is not set, make the arena 4 Gbyte, which is the maximum.
libbpf calculates max_entries from the page size.
The default is in libbpf. There it also works on the kernels that reject 0,
and it is easier to change. For example, a smaller default can be better
when the kernel allocates page tables only for the pages that an arena can
have. A kernel that accepts 0 is a separate change. libbpf needs the
default with it too.
libbpf maps the full arena into the process that loads the object. Thus
VmSize of this process grows by 4 Gbyte. If the process called
mlockall(MCL_FUTURE), the kernel populates all these pages.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/libbpf.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 02e75b183ed0..76d22a739e00 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -552,6 +552,12 @@ struct bpf_struct_ops {
#define STRUCT_OPS_SEC ".struct_ops"
#define STRUCT_OPS_LINK_SEC ".struct_ops.link"
#define ARENA_SEC ".addr_space.1"
+/*
+ * The size of the arena map when max_entries is not set: the largest one.
+ * It's in bytes: max_entries of arena is in pages and the size of a page
+ * depends on the kernel. What an empty arena costs doesn't depend on its size.
+ */
+#define ARENA_DEFAULT_SZ (1ULL << 32)
enum libbpf_map_type {
LIBBPF_MAP_UNSPEC,
@@ -3165,6 +3171,8 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
return -EINVAL;
}
obj->arena_map_idx = i;
+ if (!map->def.max_entries)
+ map->def.max_entries = ARENA_DEFAULT_SZ / sysconf(_SC_PAGE_SIZE);
if (obj->efile.arena_data) {
err = init_arena_map_data(obj, map, obj->efile.arena_data->d_buf,
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 02/14] selftests/bpf: Test the default size of arena
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 01/14] libbpf: Make arena 4 Gbyte when max_entries is not set Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 03/14] bpf: Treat addr_space_cast of a pointer that is not in arena as a move Alexei Starovoitov
` (13 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The test declares an arena without max_entries. The kernel rejects such a
map with -EINVAL, thus libbpf must set the size. See
"libbpf: Make arena 4 Gbyte when max_entries is not set".
The test makes sure that:
- After open, max_entries is 4 Gbyte divided by the page size.
- The object loads.
- The prog can allocate pages in the arena.
The test is serial, as the other tests in prog_tests/arena_*.c. See commit
a14f9ba080c1 ("selftests/bpf: Run arena tests serially").
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/prog_tests/arena_default_size.c | 22 +++++++++++++++++++
.../selftests/bpf/progs/arena_default_size.c | 21 ++++++++++++++++++
2 files changed, 43 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_default_size.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_default_size.c
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_default_size.c b/tools/testing/selftests/bpf/prog_tests/arena_default_size.c
new file mode 100644
index 000000000000..a65b3f8baece
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/arena_default_size.c
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include <linux/sizes.h>
+#include "arena_default_size.skel.h"
+
+void serial_test_arena_default_size(void)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+ struct arena_default_size *skel;
+
+ skel = arena_default_size__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+ ASSERT_EQ(bpf_map__max_entries(skel->maps.arena), SZ_4G / getpagesize(), "max_entries");
+ if (!ASSERT_OK(arena_default_size__load(skel), "load"))
+ goto out;
+ ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.alloc), &opts), "run");
+ ASSERT_OK(opts.retval, "retval");
+out:
+ arena_default_size__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/arena_default_size.c b/tools/testing/selftests/bpf/progs/arena_default_size.c
new file mode 100644
index 000000000000..1fae38126094
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_default_size.c
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+
+/* max_entries is not set */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+} arena SEC(".maps");
+
+SEC("syscall")
+int alloc(void *ctx)
+{
+ return bpf_arena_alloc_pages(&arena, NULL, 2, NUMA_NO_NODE, 0) ? 0 : 1;
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 03/14] bpf: Treat addr_space_cast of a pointer that is not in arena as a move
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 01/14] libbpf: Make arena 4 Gbyte when max_entries is not set Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 02/14] selftests/bpf: Test the default size of arena Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 2:12 ` bot+bpf-ci
2026-11-04 1:23 ` [PATCH bpf-next v3 04/14] selftests/bpf: Test addr_space_cast of pointers that are not in arena Alexei Starovoitov
` (12 subsequent siblings)
15 siblings, 1 reply; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
'rX = addr_space_cast(rY, 0, 1)' makes a pointer to arena from an address
in arena. The verifier gives rX the type PTR_TO_ARENA and replaces the insn
with 'wX = wY', which keeps the low 32 bits. The JIT adds the base of the
arena in each access. Thus an access through PTR_TO_ARENA cannot leave the
arena, and the verifier does not check it.
In C the __arena attribute marks the pointers to arena, and clang casts
only these pointers. Rust has no address spaces. The compiler follows each
pointer back to its origin, and it does not cast a pointer that comes from
the kernel. But it cannot always find the origin. For example, two progs of
an object call the same function. One prog passes a ring buffer record, and
the other prog passes an object in arena:
fn fill(e: &mut Event)
r6 = addr_space_cast(r1, 0, 1) ; a ring buffer record, or a number
*(u64 *)(r6 + 0) = r2
When the compiler does not know the origin, it casts the pointer. The
verifier does not examine the type of r1. r6 becomes PTR_TO_ARENA in the
two progs. The first prog loads, and it writes to arena instead of the
record.
The verifier knows the type of rY. Let the type select what the cast does:
rY the cast becomes rX
----------------------------- -------------------- ------------
number (scalar), PTR_TO_ARENA wX = wY, as before PTR_TO_ARENA
all other pointers rX = rY, or nothing a copy of rY
when X == Y
All other pointers are kernel pointers for the verifier, trusted or not.
A kernel pointer is in address space 0 already. Thus a move is the correct
cast. rX is a copy of rY. The verifier applies the same checks to the two
registers: bounds, NULL, references.
The verifier replaces the cast one time, for all paths. Thus it rejects a
prog when it finds a number at a cast on one path and a kernel pointer at
the same cast on a different path:
same addr_space_cast insn cannot be used with a number and with a pointer
NULL is a number too.
The analysis of live stack slots follows a pointer to the stack through a
move. Make arg_track_xfer() follow it through the cast too. Otherwise the
analysis does not see a read of the stack through rX, and the verifier
forgets a slot that the prog reads. This analysis runs before the verifier
knows the type of rY, and it can take a number for a pointer to the stack.
Then rX points to arena. Thus rX is "the slot of rY, or not a pointer" for
the analysis: an access through rX can read or write that slot, and it is
never a certain write.
The cast continues to be one insn at most. A prog in C gets different code
only where the verifier finds a kernel pointer at a cast.
The kernel marks the pointers to arena in its structs with
btf_type_tag("arena"). gcc does not emit the tag. Then such a member is
a kernel pointer for the verifier, and the cast is a move now. The members
'u64 __arena *' and 'void __arena *' of an object from bpf_obj_new() are
rdonly_untrusted_mem for the verifier, and the cast is a move too.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 3 ++
kernel/bpf/fixups.c | 20 +++++++++++++
kernel/bpf/liveness.c | 15 ++++++++--
kernel/bpf/verifier.c | 54 ++++++++++++++++++++++++++++++++++--
4 files changed, 86 insertions(+), 6 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 4a514f27f239..9ae6e109d211 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -795,6 +795,8 @@ struct bpf_insn_aux_data {
u64 non_sleepable:1; /* helper/kfunc may be called from non-sleepable context */
u64 is_iter_next:1; /* bpf_iter_<type>_next() kfunc call */
u64 call_with_percpu_alloc_ptr:1; /* {this,per}_cpu_ptr() with prog percpu alloc */
+ u64 cast_arena:1; /* addr_space_cast of an address in arena */
+ u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
u64 bb_end:1; /* last instruction of a basic block */
u64 need_scev:1;
u64 alu_state:5; /* used in combination with alu_limit */
@@ -2089,6 +2091,7 @@ int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);
void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env);
int bpf_opt_remove_dead_code(struct bpf_verifier_env *env);
int bpf_opt_remove_nops(struct bpf_verifier_env *env);
+void bpf_lower_cast_mov(struct bpf_verifier_env *env);
int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, const union bpf_attr *attr);
int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);
int bpf_jit_subprogs(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 6b719a28be68..eb5756cbc7e5 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -709,6 +709,26 @@ static bool insn_is_nop(struct bpf_verifier_env *env, int i, int next)
!memcmp(&insn, &MAY_GOTO_0, sizeof(MAY_GOTO_0));
}
+/* addr_space_cast of a pointer that is not in arena is a move, see is_cast_mov() */
+void bpf_lower_cast_mov(struct bpf_verifier_env *env)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ struct bpf_insn *insn = env->prog->insnsi;
+ int i;
+
+ for (i = 0; i < env->prog->len; i++, insn++) {
+ if (!aux[i].cast_mov)
+ continue;
+ /* an address in arena has 32 bits, this pointer has 64 */
+ aux[i].zext_dst = false;
+ if (insn->dst_reg == insn->src_reg)
+ /* bpf_opt_remove_nops() removes it */
+ *insn = NOP;
+ else
+ *insn = BPF_MOV64_REG(insn->dst_reg, insn->src_reg);
+ }
+}
+
int bpf_opt_remove_nops(struct bpf_verifier_env *env)
{
int insn_cnt = env->prog->len;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index f51445ead384..1a4cfb575077 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1456,8 +1456,16 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (code == BPF_MOV) {
if (insn->off == 0) {
*dst = *src;
+ } else if (insn->off == BPF_ADDR_SPACE_CAST && insn->imm == 1) {
+ /*
+ * The cast of a pointer to stack to as(0) is a move. But src
+ * can be a stale spill, see below. Then it's a number for the
+ * verifier, and dst points to arena. A store through dst must
+ * not count as a certain store to a slot of the stack.
+ */
+ *dst = __arg_track_join(*src, none);
} else {
- /* addr_space_cast destroys a pointer */
+ /* the cast to as(1) and sign extension destroy a pointer */
*dst = none;
}
} else {
@@ -1474,8 +1482,9 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
* Helpers, kfuncs and global subprogs can scratch stack slots. The verifier reads
* a scalar from a scratched slot, but at_stack_out[slot] is not cleared, and
* fill_from_stack() returns the stale spill: an FP-derived arg_track for a scalar.
- * A scalar can't be dereferenced, so the stale value can only set liveness bits
- * for stack accesses that don't happen. That is conservative.
+ * A scalar can't be dereferenced before addr_space_cast, see above. So the stale
+ * value can only set liveness bits for stack accesses that don't happen. That is
+ * conservative.
*
* Static subprogs and callbacks can't store an FP-derived pointer into the frame of
* the caller, but they can store any other pointer there. The caller can fill it
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 62f26149c3ae..488ee53561be 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17894,6 +17894,51 @@ int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
return reg_bounds_sanity_check(env, acc, "bpf_reg_union");
}
+/*
+ * There are no address spaces in Rust. The compiler doesn't cast a pointer that
+ * comes from the kernel. But it can't always tell, and then it casts the pointer
+ * to as(0) before the access, like an address in arena.
+ * A pointer that the verifier knows is in as(0) already, so the cast is a move.
+ * arg_track_xfer() does the same for pointers to stack.
+ */
+static bool is_cast_mov(const struct bpf_insn *insn, const struct bpf_reg_state *src_reg)
+{
+ /* cast from as(1) to as(0) */
+ if (insn->imm != 1)
+ return false;
+
+ switch (base_type(src_reg->type)) {
+ case SCALAR_VALUE:
+ case PTR_TO_ARENA:
+ return false;
+ default:
+ return true;
+ }
+}
+
+static int check_addr_space_cast(struct bpf_verifier_env *env, struct bpf_insn *insn,
+ struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg)
+{
+ struct bpf_insn_aux_data *aux = cur_aux(env);
+
+ if (is_cast_mov(insn, src_reg)) {
+ aux->cast_mov = true;
+ *dst_reg = *src_reg;
+ } else {
+ aux->cast_arena = true;
+ mark_reg_unknown(env, cur_regs(env), insn->dst_reg);
+ if (insn->imm == 1) /* cast from as(1) to as(0) */
+ dst_reg->type = PTR_TO_ARENA;
+ }
+
+ /* it's either wX = wY or rX = rY, see bpf_lower_cast_mov() */
+ if (aux->cast_mov && aux->cast_arena) {
+ verbose(env, "same addr_space_cast insn cannot be used with a number and with a pointer\n");
+ return -EINVAL;
+ }
+ return 0;
+}
+
/* check validity of 32-bit and 64-bit arithmetic operations */
static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
{
@@ -17955,9 +18000,9 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
if (BPF_CLASS(insn->code) == BPF_ALU64) {
if (insn->imm) {
/* off == BPF_ADDR_SPACE_CAST */
- mark_reg_unknown(env, regs, insn->dst_reg);
- if (insn->imm == 1) /* cast from as(1) to as(0) */
- dst_reg->type = PTR_TO_ARENA;
+ err = check_addr_space_cast(env, insn, dst_reg, src_reg);
+ if (err)
+ return err;
} else if (insn->off == 0) {
/* case: R1 = R2
* copy register state to dest reg
@@ -24231,6 +24276,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0)
ret = bpf_optimize_bpf_loop(env);
+ if (ret == 0)
+ bpf_lower_cast_mov(env);
+
if (is_priv) {
if (ret == 0)
bpf_opt_hard_wire_dead_code_branches(env);
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 03/14] bpf: Treat addr_space_cast of a pointer that is not in arena as a move
2026-11-04 1:23 ` [PATCH bpf-next v3 03/14] bpf: Treat addr_space_cast of a pointer that is not in arena as a move Alexei Starovoitov
@ 2026-11-04 2:12 ` bot+bpf-ci
2026-11-04 2:32 ` Alexei Starovoitov
0 siblings, 1 reply; 32+ messages in thread
From: bot+bpf-ci @ 2026-11-04 2:12 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3259 bytes --]
> bpf: Treat addr_space_cast of a pointer that is not in arena as a move
TL;DR: is_cast_mov() may treat arena pointers loaded from bpf_obj_new()
objects (untrusted mem) as as(0); previously working programs may now fail to
load on stores or silently read zeros on loads. Commit message doesn't mention
this.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> +static bool is_cast_mov(const struct bpf_insn *insn, const struct bpf_reg_state *src_reg)
> +{
> + /* cast from as(1) to as(0) */
> + if (insn->imm != 1)
> + return false;
> +
> + switch (base_type(src_reg->type)) {
> + case SCALAR_VALUE:
> + case PTR_TO_ARENA:
> + return false;
> + default:
> + return true;
> + }
> +}
Is it correct for is_cast_mov() to treat every pointer type other than
PTR_TO_ARENA as already being in as(0)?
A load of a non-struct pointer member (for example 'u64 __arena *data')
from an object described by program BTF, such as one from bpf_obj_new(),
bpf_percpu_obj_new() or a local kptr, goes through btf_struct_walk() which
returns WALK_PTR_UNTRUSTED. Unlike WALK_PTR, btf_struct_access() has no
type_is_alloc() exception for it:
case WALK_PTR_UNTRUSTED:
*flag = MEM_RDONLY | PTR_UNTRUSTED;
ret = PTR_TO_MEM;
So after something like 'e->data = bpf_arena_alloc_pages(...)', loading
e->data gives PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED, but the value in
the register is an arena user address, since arena_alloc_pages() returns
clear_lo32(arena->user_vm_start) + uaddr32. When the program dereferences
it, clang emits 'rX = addr_space_cast(rY, 0, 1)'.
Before this patch rX became PTR_TO_ARENA, the access was rewritten to
BPF_PROBE_MEM32, and it reached the arena page. After this patch the cast
is lowered to 'rX = rY' and rX stays rdonly_untrusted_mem:
- A store through rX is now rejected at load time ("cannot write into
rdonly_untrusted_mem").
- A load through rX becomes a BPF_PROBE_MEM load from the 64-bit user VA.
The x86 JIT emits its "src_reg + off > TASK_SIZE_MAX + PAGE_SIZE" check
for that load, so the program reads zeros instead of the arena contents,
and nothing reports it.
The commit message says these members are rdonly_untrusted_mem and "the
cast is a move too", but it does not say that these registers hold as(1)
addresses rather than kernel addresses, or that programs which loaded and
worked before now either fail to load or silently read 0.
Could btf_struct_access() return SCALAR_VALUE for WALK_PTR_UNTRUSTED when
type_is_alloc(reg->type), as it already does for WALK_PTR, so that such a
member takes the cast_arena path? Alternatively, should is_cast_mov() not
treat untrusted memory derived from program BTF as as(0)?
The same applies to the case in the commit message of an untagged
(gcc-built) kernel __arena member: there, writes would stop loading
instead of reaching the arena.
No later commit in the series changes is_cast_mov() or the
WALK_PTR_UNTRUSTED handling.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/41755299448
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 03/14] bpf: Treat addr_space_cast of a pointer that is not in arena as a move
2026-11-04 2:12 ` bot+bpf-ci
@ 2026-11-04 2:32 ` Alexei Starovoitov
0 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 2:32 UTC (permalink / raw)
To: bot+bpf-ci, bpf
Cc: daniel, andrii, eddyz87, memxor, martin.lau, yonghong.song, mason,
ihor.solodrai
On Tue, Nov 03, 2026 at 06:12 PM bot+bpf-ci@kernel.org <bot+bpf-ci@kernel.org> wrote:
> The commit message says these members are rdonly_untrusted_mem and "the
> cast is a move too", but it does not say that these registers hold as(1)
> addresses rather than kernel addresses, or that programs which loaded and
> worked before now either fail to load or silently read 0.
It's already explained in the commit log.
> Could btf_struct_access() return SCALAR_VALUE for WALK_PTR_UNTRUSTED when
> type_is_alloc(reg->type), as it already does for WALK_PTR, so that such a
> member takes the cast_arena path? Alternatively, should is_cast_mov() not
> treat untrusted memory derived from program BTF as as(0)?
No.
> The same applies to the case in the commit message of an untagged
> (gcc-built) kernel __arena member: there, writes would stop loading
> instead of reaching the arena.
No.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH bpf-next v3 04/14] selftests/bpf: Test addr_space_cast of pointers that are not in arena
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (2 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 03/14] bpf: Treat addr_space_cast of a pointer that is not in arena as a move Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:40 ` sashiko-bot
2026-11-04 1:23 ` [PATCH bpf-next v3 05/14] libbpf: Support .bss.addr_space.1 section Alexei Starovoitov
` (11 subsequent siblings)
15 siblings, 1 reply; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier does not check an access through a pointer to arena. Thus a
kernel pointer must keep its type after the cast to address space 0. See
"bpf: Treat addr_space_cast of a pointer that is not in arena as a move".
test what the prog does or shows result
--------------------------- ------------------------------------ ------
map_value the cast is 'r6 = r0' loads
map_value_same_reg the cast of r0 to r0 is removed loads
map_value_bounds access beyond the map value reject
map_value_or_null access without a NULL check reject
map_value_null_check_of_src NULL check of r0 applies to r6 loads
ctx r6 is ctx loads
btf_id r6 is trusted_ptr_task_struct loads
untrusted_btf_id r6 is untrusted_ptr_task_struct loads
untrusted_mem r6 is rdonly_untrusted_mem loads
ref_released releases a reference through r1 loads
ref_leaked does not release a reference reject
stack r6 is fp-8 loads
spill_through_cast spills ctx through r6, fills it loads
callee_writes callee writes the caller's stack loads
callee_reads callee reads the caller's stack loads
number r6 is arena, the cast is 'w6 = w6' loads
number_and_pointer one cast gets a number and ctx reject
pointer_and_number the same, the paths are exchanged reject
cast_user cast of ctx to as(1) gives a scalar loads
The progs that return the content of a map value or of a stack slot also
run. The result shows that they read the object and not arena. callee_reads
needs the change in arg_track_xfer().
The analysis of live stack slots can take a number for a pointer to the
stack: a slot held such a pointer, and a helper or a short store overwrote
it. The cast of this number gives a pointer to arena. A store through it
must not count as a write to the stack:
test what the prog does or shows result
------------------------------- -------------------------------- ------
stale_spill_store the store hides a write to the reject
counter of a loop
stale_spill_store_r10 the same, the stale value is r10 reject
stale_spill_live_slot a slot is read after the store loads
stale_spill_live_slot_of_caller the same, a slot of the caller loads
store_over_spilled_r10 a pointer to a map value is reject
stored over r10 through the cast
r10_stored_to_arena r10 is stored to arena reject
callee_writes_in_loop the loop stays counted when the loads
callee writes through the cast
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_arena_cast.c | 800 ++++++++++++++++++
2 files changed, 802 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_cast.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 99f0551c8b59..3d0eda3b4da4 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -12,6 +12,7 @@
#include "verifier_align.skel.h"
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
+#include "verifier_arena_cast.skel.h"
#include "verifier_arena_large.skel.h"
#include "verifier_arena_globals1.skel.h"
#include "verifier_arena_globals2.skel.h"
@@ -207,6 +208,7 @@ void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret
void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }
+void test_verifier_arena_cast(void) { RUN(verifier_arena_cast); }
void test_verifier_arena_large(void) { RUN(verifier_arena_large); }
void test_verifier_arena_globals1(void) { RUN(verifier_arena_globals1); }
void test_verifier_arena_globals2(void) { RUN(verifier_arena_globals2); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_cast.c b/tools/testing/selftests/bpf/progs/verifier_arena_cast.c
new file mode 100644
index 000000000000..14a199318d0d
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_cast.c
@@ -0,0 +1,800 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+struct task_struct {
+ struct task_struct *group_leader;
+ void *stack;
+} __attribute__((preserve_access_index));
+struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym;
+void bpf_task_release(struct task_struct *p) __ksym;
+
+#ifdef __TARGET_ARCH_arm64
+#define ARENA_VM_START (1ull << 32)
+#else
+#define ARENA_VM_START (1ull << 44)
+#endif
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 4);
+ __ulong(map_extra, ARENA_VM_START);
+} arena SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, long long);
+} hash SEC(".maps");
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_task_release(bpf_task_acquire(0));
+}
+
+/* dst = addr_space_cast(src, 0, 1) */
+#define CAST_KERN(dst, src) \
+ BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, src, BPF_ADDR_SPACE_CAST, 1)
+/* dst = addr_space_cast(src, 1, 0) */
+#define CAST_USER(dst, src) \
+ BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, src, BPF_ADDR_SPACE_CAST, 1 << 16)
+
+/* r0 = the element of the hash map or NULL. There is one after the update. */
+#define UPDATE_AND_LOOKUP \
+ "r1 = 0;" \
+ "*(u32 *)(r10 - 4) = r1;" \
+ "r1 = 7;" \
+ "*(u64 *)(r10 - 16) = r1;" \
+ "r1 = %[hash] ll;" \
+ "r2 = r10;" \
+ "r2 += -4;" \
+ "r3 = r10;" \
+ "r3 += -16;" \
+ "r4 = 0;" \
+ "call %[bpf_map_update_elem];" \
+ "r1 = %[hash] ll;" \
+ "r2 = r10;" \
+ "r2 += -4;" \
+ "call %[bpf_map_lookup_elem];"
+
+SEC("syscall")
+__description("arena cast: a pointer to map value, it's a move")
+__success __retval(7)
+__xlated("r6 = r0")
+__xlated("r0 = *(u64 *)(r6 +0)")
+__naked void map_value(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u64 *)(r6 + 0);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer to map value into the same register, it's removed")
+__success __retval(7)
+__xlated("r7 = r0")
+__xlated("r0 = *(u64 *)(r0 +0)")
+__xlated("exit")
+__naked void map_value_same_reg(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 1f;"
+ "r7 = r0;"
+ ".8byte %[cast_r0_r0];"
+ "r0 = *(u64 *)(r0 + 0);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r0_r0, CAST_KERN(BPF_REG_0, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: bounds of map value are checked after the cast")
+__failure __msg("invalid access to map value, value_size=8 off=8 size=8")
+__naked void map_value_bounds(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u64 *)(r6 + 8);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer that may be NULL has to be checked after the cast")
+__failure __msg("R6 invalid mem access 'map_value_or_null'")
+__naked void map_value_or_null(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u64 *)(r6 + 0);"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: NULL check of the pointer is for its copy too")
+__success __retval(7)
+__naked void map_value_null_check_of_src(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ ".8byte %[cast_r6_r0];"
+ "if r0 == 0 goto 1f;"
+ "r0 = *(u64 *)(r6 + 0);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: a pointer to ctx")
+__success __log_level(2)
+__msg("R6=ctx()")
+__naked void ctx(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ ".8byte %[cast_r6_r1];"
+ "r0 = *(u32 *)(r6 + %[len]);"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_insn(cast_r6_r1, CAST_KERN(BPF_REG_6, BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer to task_struct")
+__success __log_level(2)
+__msg("R6=trusted_ptr_task_struct()")
+__naked void btf_id(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "call %[bpf_get_current_task_btf];"
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u32 *)(r6 + 0);"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_current_task_btf),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+/* r6 = addr_space_cast(p, 0, 1) */
+#define CAST_R6(p) \
+ asm volatile ( \
+ "r1 = %[ptr];" \
+ ".8byte %[cast_r6_r1];" \
+ "r1 = %[arena] ll;" \
+ : \
+ : [ptr]"r"(p), \
+ __imm_addr(arena), \
+ __imm_insn(cast_r6_r1, CAST_KERN(BPF_REG_6, BPF_REG_1)) \
+ : "r1", "r6")
+
+/* the prog can sleep: what it reads from task_struct outside of an RCU section is untrusted */
+SEC("syscall")
+__description("arena cast: an untrusted pointer to task_struct, it's a move")
+__success __log_level(2)
+__msg("R6=untrusted_ptr_task_struct()")
+__xlated("r6 = r1")
+int untrusted_btf_id(void *ctx)
+{
+ CAST_R6(bpf_get_current_task_btf()->group_leader);
+ return 0;
+}
+
+SEC("syscall")
+__description("arena cast: an untrusted pointer to kernel memory, it's a move")
+__success __log_level(2)
+__msg("R6=rdonly_untrusted_mem(sz=0)")
+__xlated("r6 = r1")
+int untrusted_mem(void *ctx)
+{
+ CAST_R6(bpf_get_current_task_btf()->stack);
+ return 0;
+}
+
+SEC("syscall")
+__description("arena cast: a reference is released through the copy")
+__success __retval(0)
+__naked void ref_released(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "call %[bpf_get_current_task_btf];"
+ "r1 = r0;"
+ "call %[bpf_task_acquire];"
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r1_r0];"
+ "call %[bpf_task_release];"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_current_task_btf),
+ __imm(bpf_task_acquire),
+ __imm(bpf_task_release),
+ __imm_insn(cast_r1_r0, CAST_KERN(BPF_REG_1, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a reference doesn't go away")
+__failure __msg("Unreleased reference")
+__naked void ref_leaked(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "call %[bpf_get_current_task_btf];"
+ "r1 = r0;"
+ "call %[bpf_task_acquire];"
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r0_r0];"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_current_task_btf),
+ __imm(bpf_task_acquire),
+ __imm_insn(cast_r0_r0, CAST_KERN(BPF_REG_0, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a number")
+__success __log_level(2)
+__msg("R6=arena")
+__xlated("w6 = w6")
+__naked void number(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r6 = %[addr] ll;"
+ ".8byte %[cast_r6_r6];"
+ "r0 = *(u64 *)(r6 + 0);"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_const(addr, ARENA_VM_START),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+/* static analysis of the stack follows the pointer through the cast */
+SEC("syscall")
+__description("arena cast: a pointer to stack")
+__success __retval(7) __log_level(2)
+__msg("R6=fp-8")
+__naked void stack(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r6 = r10;"
+ "r6 += -8;"
+ ".8byte %[cast_r6_r6];"
+ "r0 = 7;"
+ "*(u64 *)(r6 + 0) = r0;"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer is spilled through the cast of a pointer to stack")
+__success
+__naked void spill_through_cast(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ "r6 = r10;"
+ "r6 += -8;"
+ ".8byte %[cast_r6_r6];"
+ "*(u64 *)(r6 + 0) = r1;"
+ "r2 = *(u64 *)(r10 - 8);"
+ /* it's ctx */
+ "r0 = *(u32 *)(r2 + 0);"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long writes_through_cast(long p)
+{
+ asm volatile (
+ ".8byte %[cast_r1_r1];"
+ "r0 = 7;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(cast_r1_r1, CAST_KERN(BPF_REG_1, BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: the callee writes to the stack of the caller")
+__success __retval(7)
+__naked void callee_writes(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r0 = 0;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call writes_through_cast;"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ :
+ : __imm_addr(arena)
+ : __clobber_all);
+}
+
+static __naked __noinline __used long reads_through_cast(long p)
+{
+ asm volatile (
+ "r6 = r1;"
+ /* a place where what is not live is forgotten */
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto +0;"
+ ".8byte %[cast_r6_r6];"
+ "r0 = *(u64 *)(r6 + 0);"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: the callee reads the stack of the caller")
+__success __retval(5)
+__naked void callee_reads(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r0 = 5;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call reads_through_cast;"
+ "exit;"
+ :
+ : __imm_addr(arena)
+ : __clobber_all);
+}
+
+/*
+ * r7 = a number that static analysis of the stack takes for fp + off: fp-16 holds fp + off,
+ * a helper overwrites fp-16, and the analysis doesn't follow what helpers write.
+ */
+#define STALE_SPILL_R7(off) \
+ "r1 = r10;" \
+ "r1 += " #off ";" \
+ "*(u64 *)(r10 - 16) = r1;" \
+ "r1 = r10;" \
+ "r1 += -16;" \
+ "r2 = 8;" \
+ "call %[bpf_get_current_comm];" \
+ "r7 = *(u64 *)(r10 - 16);"
+
+#define SPILL_FP24_TO_FP8 \
+ "r1 = r10;" \
+ "r1 += -24;" \
+ "*(u64 *)(r10 - 8) = r1;"
+
+/*
+ * fp-24 counts 0, 1, ... 7 and it's the offset in the map value in r9. fp-8 holds &fp[-24],
+ * and the store through r8 changes the offset. When the analysis loses the pointer in fp-8,
+ * it doesn't see that store, and the verifier takes 0 ... 7 for all values of fp-24.
+ */
+#define LOOP_WRITES_FP24_THROUGH_FP8 \
+ "r8 = *(u64 *)(r10 - 8);" \
+ "r1 = 0;" \
+ "*(u64 *)(r10 - 24) = r1;" \
+ "r6 = 0;" \
+"1:" \
+ "r1 = *(u64 *)(r10 - 24);" \
+ "r2 = r9;" \
+ "r2 += r1;" \
+ "r0 = *(u8 *)(r2 + 0);" \
+ "r1 += 1;" \
+ "*(u64 *)(r10 - 24) = r1;" \
+ "r3 = 4096;" \
+ "*(u64 *)(r8 + 0) = r3;" \
+ "r6 += 1;" \
+ "if r6 < 8 goto 1b;"
+
+SEC("syscall")
+__description("arena cast: a store to arena doesn't replace what is spilled to the stack")
+__failure __msg("invalid access to map value, value_size=8 off=4096 size=1")
+__naked void stale_spill_store(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 2f;"
+ "r9 = r0;"
+ STALE_SPILL_R7(-8)
+ ".8byte %[cast_r7_r7];"
+ SPILL_FP24_TO_FP8
+ /* to arena, not to fp-8 */
+ "r0 = 0;"
+ "*(u64 *)(r7 + 0) = r0;"
+ LOOP_WRITES_FP24_THROUGH_FP8
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_get_current_comm),
+ __imm_insn(cast_r7_r7, CAST_KERN(BPF_REG_7, BPF_REG_7))
+ : __clobber_all);
+}
+
+/* fp+0 is what the analysis adds to fp-N to say "or not a pointer to stack" */
+SEC("syscall")
+__description("arena cast: a store to arena doesn't replace what is spilled, stale r10")
+__failure __msg("invalid access to map value, value_size=8 off=4096 size=1")
+__naked void stale_spill_store_r10(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 2f;"
+ "r9 = r0;"
+ STALE_SPILL_R7(0)
+ ".8byte %[cast_r7_r7];"
+ SPILL_FP24_TO_FP8
+ "r0 = 0;"
+ "*(u64 *)(r7 - 8) = r0;"
+ LOOP_WRITES_FP24_THROUGH_FP8
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_get_current_comm),
+ __imm_insn(cast_r7_r7, CAST_KERN(BPF_REG_7, BPF_REG_7))
+ : __clobber_all);
+}
+
+/* compilers use a slot for two variables: a pointer to a local, then the buffer of a helper */
+SEC("syscall")
+__description("arena cast: a stack slot is live over a store to arena")
+__success
+__naked void stale_spill_live_slot(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 3f;"
+ "r9 = r0;"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0;"
+ STALE_SPILL_R7(-8)
+ ".8byte %[cast_r7_r7];"
+ "r1 = 0;"
+ "if r6 == 0 goto 1f;"
+ "r1 = 7;"
+"1:"
+ "*(u64 *)(r10 - 8) = r1;"
+ /* a place where what is not live is forgotten */
+ "if r6 == 1 goto +0;"
+ "r1 = 0;"
+ "*(u64 *)(r7 + 0) = r1;"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r9 += r1;"
+ "r0 = *(u8 *)(r9 + 0);"
+"3:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_get_current_comm),
+ __imm(bpf_get_prandom_u32),
+ __imm_insn(cast_r7_r7, CAST_KERN(BPF_REG_7, BPF_REG_7))
+ : __clobber_all);
+}
+
+/* r1 is a number, r2 is &fp[-8] of the caller */
+static __naked __noinline __used long writes_arena_reads_stack(long a, long *p)
+{
+ asm volatile (
+ ".8byte %[cast_r1_r1];"
+ "r0 = 0;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r0 = *(u64 *)(r2 + 0);"
+ "exit;"
+ :
+ : __imm_insn(cast_r1_r1, CAST_KERN(BPF_REG_1, BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a stack slot of the caller is live over a store to arena in the callee")
+__success
+__naked void stale_spill_live_slot_of_caller(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 3f;"
+ "r9 = r0;"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0;"
+ STALE_SPILL_R7(-8)
+ "r1 = 0;"
+ "if r6 == 0 goto 1f;"
+ "r1 = 7;"
+"1:"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = r7;"
+ "r2 = r10;"
+ "r2 += -8;"
+ "call writes_arena_reads_stack;"
+ "r9 += r0;"
+ "r0 = *(u8 *)(r9 + 0);"
+"3:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_get_current_comm),
+ __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/*
+ * A store through the result of the cast can miss the stack: the slot keeps what it had, or
+ * it has the new value. Static analysis must not make "it's fp+0" from
+ * "it's fp+0, or it's not a pointer to stack".
+ */
+SEC("syscall")
+__description("arena cast: a pointer to map value is stored over r10 through the cast")
+__failure __msg("invalid access to map value, value_size=8 off=4096 size=1")
+__naked void store_over_spilled_r10(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 2f;"
+ "r9 = r0;"
+ "*(u64 *)(r10 - 16) = r10;"
+ "r7 = r10;"
+ "r7 += -16;"
+ ".8byte %[cast_r7_r7];"
+ "r1 = r9;"
+ "r1 += 8;"
+ "*(u64 *)(r7 + 0) = r1;"
+ /* it's the map value, not r10 */
+ "r7 = *(u64 *)(r10 - 16);"
+ SPILL_FP24_TO_FP8
+ "r0 = 0;"
+ "*(u64 *)(r7 - 8) = r0;"
+ LOOP_WRITES_FP24_THROUGH_FP8
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r7_r7, CAST_KERN(BPF_REG_7, BPF_REG_7))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: r10 is stored to arena, a pointer to map value is in the stack")
+__failure __msg("invalid access to map value, value_size=8 off=4096 size=1")
+__naked void r10_stored_to_arena(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 2f;"
+ "r9 = r0;"
+ STALE_SPILL_R7(-8)
+ ".8byte %[cast_r7_r7];"
+ "r1 = r9;"
+ "r1 += 8;"
+ "*(u64 *)(r10 - 32) = r1;"
+ /* to arena, not to fp-32 */
+ "*(u64 *)(r7 + 0) = r10;"
+ "r7 = *(u64 *)(r10 - 32);"
+ SPILL_FP24_TO_FP8
+ "r0 = 0;"
+ "*(u64 *)(r7 - 8) = r0;"
+ LOOP_WRITES_FP24_THROUGH_FP8
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_get_current_comm),
+ __imm_insn(cast_r7_r7, CAST_KERN(BPF_REG_7, BPF_REG_7))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long fills_through_cast(long *p, long v)
+{
+ asm volatile (
+ ".8byte %[cast_r1_r1];"
+ "*(u64 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(cast_r1_r1, CAST_KERN(BPF_REG_1, BPF_REG_1))
+ : __clobber_all);
+}
+
+/* the callee writes to fp-16 only, the verifier still can count the loop */
+SEC("syscall")
+__description("arena cast: the callee writes to a local, the counter of the loop is in the stack")
+__success
+__naked void callee_writes_in_loop(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+"1:"
+ "r1 = r10;"
+ "r1 += -16;"
+ "r2 = *(u64 *)(r10 - 8);"
+ "call fills_through_cast;"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r1 += 1;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "if r1 < 1000000 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena)
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: a number and a pointer at the same insn")
+__failure __msg("same addr_space_cast insn cannot be used with a number and with a pointer")
+__naked void number_and_pointer(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ "r6 = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r6 = 4096;"
+"1:"
+ ".8byte %[cast_r6_r6];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_prandom_u32),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: a pointer and a number at the same insn")
+__failure __msg("same addr_space_cast insn cannot be used with a number and with a pointer")
+__naked void pointer_and_number(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ "r6 = 4096;"
+ "r7 = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r6 = r7;"
+"1:"
+ ".8byte %[cast_r6_r6];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_prandom_u32),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: the cast to as(1) of a pointer is a number")
+__success __log_level(2)
+__msg("R6=scalar()")
+__naked void cast_user(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ ".8byte %[cast_r6_r1];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_insn(cast_r6_r1, CAST_USER(BPF_REG_6, BPF_REG_1))
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 04/14] selftests/bpf: Test addr_space_cast of pointers that are not in arena
2026-11-04 1:23 ` [PATCH bpf-next v3 04/14] selftests/bpf: Test addr_space_cast of pointers that are not in arena Alexei Starovoitov
@ 2026-11-04 1:40 ` sashiko-bot
0 siblings, 0 replies; 32+ messages in thread
From: sashiko-bot @ 2026-11-04 1:40 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] selftests/bpf: wrong stack offset in r10_stored_to_arena()
--
commit 54fbcf6815ff6f19ca4963d449d29e574fbc1941
Author: Alexei Starovoitov <ast@kernel.org>
selftests/bpf: Test addr_space_cast of pointers that are not in arena
This commit adds BPF verifier selftests in verifier_arena_cast.c to test
addr_space_cast behavior on non-arena pointers, numbers, and stack slot
liveness tracking across casts.
> diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_cast.c b/tools/testing/selftests/bpf/progs/verifier_arena_cast.c
> new file mode 100644
> index 0000000000000..14a199318d0d5
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/verifier_arena_cast.c
> @@ -0,0 +1,800 @@
[ ... ]
> +SEC("syscall")
> +__description("arena cast: r10 is stored to arena, a pointer to map value is in the stack")
> +__failure __msg("invalid access to map value, value_size=8 off=4096 size=1")
> +__naked void r10_stored_to_arena(void)
> +{
> + asm volatile (
> + "r1 = %[arena] ll;"
> + UPDATE_AND_LOOKUP
> + "if r0 == 0 goto 2f;"
> + "r9 = r0;"
> + STALE_SPILL_R7(-8)
> + ".8byte %[cast_r7_r7];"
> + "r1 = r9;"
> + "r1 += 8;"
> + "*(u64 *)(r10 - 32) = r1;"
> + /* to arena, not to fp-32 */
> + "*(u64 *)(r7 + 0) = r10;"
> + "r7 = *(u64 *)(r10 - 32);"
[Severity: Medium]
In r10_stored_to_arena() in
tools/testing/selftests/bpf/progs/verifier_arena_cast.c, should
STALE_SPILL_R7(-8) be STALE_SPILL_R7(-32) instead?
As written, STALE_SPILL_R7(-8) gives r7 a stale stack offset of fp-8 rather
than fp-32. When *(u64 *)(r7 + 0) = r10 executes, static stack analysis
tracks the store against slot fp-8 (which is immediately overwritten by
SPILL_FP24_TO_FP8) and never touches slot fp-32, so the test does not
exercise joining the map value pointer in fp-32 (ARG_NONE) with r10 (fp+0).
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261104012401.1354861-1-alexei.starovoitov@gmail.com?part=4
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH bpf-next v3 05/14] libbpf: Support .bss.addr_space.1 section
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (3 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 04/14] selftests/bpf: Test addr_space_cast of pointers that are not in arena Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:59 ` bot+bpf-ci
2026-11-04 1:23 ` [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena Alexei Starovoitov
` (10 subsequent siblings)
15 siblings, 1 reply; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Global variables in arena are in the section .addr_space.1. Its type is
SHT_PROGBITS. Thus the variables that are zero use space in the object file
too.
Support .bss.addr_space.1 as the .bss of arena. The name starts with .bss.
Thus compilers make the section SHT_NOBITS, and it has no bytes in the
file:
long __arena big[8192] SEC(".bss.addr_space.1");
An object with 8 Mbyte of zeros shrinks from 8410296 to 21696 bytes.
A declaration in a different file needs the same section:
extern long __arena big[8192] SEC(".bss.addr_space.1");
Without SEC() the compiler puts the declaration in .addr_space.1. The
static linker took the index of the variable in one section for an index in
the other section, and it crashed or wrote incorrect BTF. Make it report
a section mismatch, as it does when the definition comes first.
libbpf puts the new section after .addr_space.1 in arena and fills it with
zeros. It adds the offset of the section to the relocations that refer to
it.
Only the file becomes smaller. At load the new variables get pages in
arena, as the variables of .addr_space.1 do. The layout of skel->arena does
not change, and the new variables are not in it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/libbpf.c | 41 ++++++++++++++++++++++++++++++++---------
tools/lib/bpf/linker.c | 11 +++++++++++
2 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 76d22a739e00..a7c53dc5b91d 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -552,6 +552,11 @@ struct bpf_struct_ops {
#define STRUCT_OPS_SEC ".struct_ops"
#define STRUCT_OPS_LINK_SEC ".struct_ops.link"
#define ARENA_SEC ".addr_space.1"
+/*
+ * Variables in arena that are zero. The name starts with .bss, so compilers make
+ * it SHT_NOBITS and it takes no space in the file. It's placed after ARENA_SEC.
+ */
+#define ARENA_BSS_SEC ".bss.addr_space.1"
/*
* The size of the arena map when max_entries is not set: the largest one.
* It's in bytes: max_entries of arena is in pages and the size of a page
@@ -697,6 +702,8 @@ struct elf_state {
Elf64_Ehdr *ehdr;
Elf_Data *symbols;
Elf_Data *arena_data;
+ Elf_Data *arena_bss;
+ size_t arena_bss_align;
size_t shstrndx; /* section index for section name strings */
size_t strtabidx;
struct elf_sec_desc *secs;
@@ -707,6 +714,7 @@ struct elf_state {
int symbols_shndx;
bool has_st_ops;
int arena_data_shndx;
+ int arena_bss_shndx;
int jumptables_data_shndx;
};
@@ -788,6 +796,7 @@ struct bpf_object {
void *arena_data;
size_t arena_data_sz;
size_t arena_data_off;
+ size_t arena_bss_off; /* from the start of arena_data */
void *jumptables_data;
size_t jumptables_data_sz;
@@ -1629,6 +1638,7 @@ static struct bpf_object *bpf_object__new(const char *path,
obj->efile.obj_buf = obj_buf;
obj->efile.obj_buf_sz = obj_buf_sz;
obj->efile.btf_maps_shndx = -1;
+ obj->efile.arena_bss_shndx = -1;
obj->kconfig_map_idx = -1;
obj->arena_map_idx = -1;
@@ -1648,6 +1658,7 @@ static void bpf_object__elf_finish(struct bpf_object *obj)
obj->efile.ehdr = NULL;
obj->efile.symbols = NULL;
obj->efile.arena_data = NULL;
+ obj->efile.arena_bss = NULL;
zfree(&obj->efile.secs);
obj->efile.sec_cnt = 0;
@@ -3095,13 +3106,20 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj,
return 0;
}
-static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,
- void *data, size_t data_sz)
+static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map)
{
- obj->arena_data = malloc(data_sz);
+ Elf_Data *data = obj->efile.arena_data, *bss = obj->efile.arena_bss;
+ size_t data_sz = data ? data->d_size : 0;
+
+ if (bss) {
+ obj->arena_bss_off = roundup(data_sz, obj->efile.arena_bss_align);
+ data_sz = obj->arena_bss_off + bss->d_size;
+ }
+ obj->arena_data = calloc(1, data_sz);
if (!obj->arena_data)
return -ENOMEM;
- memcpy(obj->arena_data, data, data_sz);
+ if (data)
+ memcpy(obj->arena_data, data->d_buf, data->d_size);
obj->arena_data_sz = data_sz;
/* make bpf_map__init_value() work for ARENA maps */
@@ -3174,14 +3192,13 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
if (!map->def.max_entries)
map->def.max_entries = ARENA_DEFAULT_SZ / sysconf(_SC_PAGE_SIZE);
- if (obj->efile.arena_data) {
- err = init_arena_map_data(obj, map, obj->efile.arena_data->d_buf,
- obj->efile.arena_data->d_size);
+ if (obj->efile.arena_data || obj->efile.arena_bss) {
+ err = init_arena_map_data(obj, map);
if (err)
return err;
}
}
- if (obj->efile.arena_data && obj->arena_map_idx < 0) {
+ if ((obj->efile.arena_data || obj->efile.arena_bss) && obj->arena_map_idx < 0) {
pr_warn("elf: sec '%s': to use global __arena variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
ARENA_SEC);
return -ENOENT;
@@ -4176,6 +4193,10 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
sec_desc->sec_type = SEC_RELO;
sec_desc->shdr = sh;
sec_desc->data = data;
+ } else if (sh->sh_type == SHT_NOBITS && strcmp(name, ARENA_BSS_SEC) == 0) {
+ obj->efile.arena_bss = data;
+ obj->efile.arena_bss_align = sh->sh_addralign ?: 1;
+ obj->efile.arena_bss_shndx = idx;
} else if (sh->sh_type == SHT_NOBITS && (strcmp(name, BSS_SEC) == 0 ||
str_has_pfx(name, BSS_SEC "."))) {
sec_desc->sec_type = SEC_BSS;
@@ -4826,7 +4847,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
sym_sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, shdr_idx));
/* arena data relocation */
- if (shdr_idx == obj->efile.arena_data_shndx) {
+ if (shdr_idx == obj->efile.arena_data_shndx || shdr_idx == obj->efile.arena_bss_shndx) {
if (obj->arena_map_idx < 0) {
pr_warn("prog '%s': bad arena data relocation at insn %u, no arena maps defined\n",
prog->name, insn_idx);
@@ -4836,6 +4857,8 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
reloc_desc->insn_idx = insn_idx;
reloc_desc->map_idx = obj->arena_map_idx;
reloc_desc->sym_off = sym->st_value;
+ if (shdr_idx == obj->efile.arena_bss_shndx)
+ reloc_desc->sym_off += obj->arena_bss_off;
map = &obj->maps[obj->arena_map_idx];
pr_debug("prog '%s': found arena map %d (%s, sec %d, off %zu) for insn %u\n",
diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
index f3f71c452f00..dba88f4fd6cd 100644
--- a/tools/lib/bpf/linker.c
+++ b/tools/lib/bpf/linker.c
@@ -2123,6 +2123,17 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj,
if (!glob_sym->is_extern && sym_bind == STB_WEAK)
return 0;
+ /*
+ * The symbol has an entry in the DATASEC of a different section.
+ * var_idx is an index in that section. linker_append_btf() reports
+ * the same when the definition is first.
+ */
+ if (glob_sym->var_idx >= 0 && glob_sym->sec_id != dst_sec->id) {
+ pr_warn("global '%s': section mismatch %d vs %d\n",
+ sym_name, glob_sym->sec_id, dst_sec->id);
+ return -EINVAL;
+ }
+
/* At this point, new symbol is strong non-extern,
* so overwrite glob_sym with new symbol information.
* Preserve binding and visibility.
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 05/14] libbpf: Support .bss.addr_space.1 section
2026-11-04 1:23 ` [PATCH bpf-next v3 05/14] libbpf: Support .bss.addr_space.1 section Alexei Starovoitov
@ 2026-11-04 1:59 ` bot+bpf-ci
0 siblings, 0 replies; 32+ messages in thread
From: bot+bpf-ci @ 2026-11-04 1:59 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2746 bytes --]
> libbpf: Support .bss.addr_space.1 section
TL;DR: In init_arena_map_data(), unvalidated ELF sh_addralign/sh_size in the
new .bss.addr_space.1 handling may wrap data_sz, so a malformed ELF could
cause a heap buffer overflow in memcpy.
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
[ ... ]
> @@ -3095,13 +3106,20 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj,
> return 0;
> }
>
> -static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,
> - void *data, size_t data_sz)
> +static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map)
> {
> - obj->arena_data = malloc(data_sz);
> + Elf_Data *data = obj->efile.arena_data, *bss = obj->efile.arena_bss;
> + size_t data_sz = data ? data->d_size : 0;
> +
> + if (bss) {
> + obj->arena_bss_off = roundup(data_sz, obj->efile.arena_bss_align);
> + data_sz = obj->arena_bss_off + bss->d_size;
> + }
> + obj->arena_data = calloc(1, data_sz);
> if (!obj->arena_data)
> return -ENOMEM;
> - memcpy(obj->arena_data, data, data_sz);
> + if (data)
> + memcpy(obj->arena_data, data->d_buf, data->d_size);
> obj->arena_data_sz = data_sz;
Can data_sz wrap here? Both arena_bss_align (from sh_addralign) and
bss->d_size (sh_size of a SHT_NOBITS section) come straight from the ELF
section header and are not validated.
For example, with data->d_size = 4096 and sh_addralign = 0xffffffffffffffff,
roundup() computes (4096 + (__y - 1)) / __y * __y, which wraps to
arena_bss_off = 0. With sh_size = 8, data_sz = 8.
Similarly, with a sane alignment (arena_bss_off = 4096) and
sh_size = 2^64 - 4096 + 8, data_sz also wraps to 8.
In both cases calloc(1, 8) succeeds and the following
memcpy(obj->arena_data, data->d_buf, data->d_size) copies 4096 bytes into an
8-byte buffer. The path is:
bpf_object__open_mem()
bpf_object_open()
bpf_object__elf_collect()
bpf_object__init_maps()
bpf_object__init_user_btf_maps()
init_arena_map_data()
The old code was malloc(d_size) followed by memcpy of d_size, so it could
not overflow. libbpf has fixed other malformed-ELF arena crashes before,
for example 42be23e8f2dc ("libbpf: Verify that arena map exists when adding
arena relocations").
Should this reject a sh_addralign that is not a power of two, and fail when
arena_bss_off < data_sz or bss->d_size > SIZE_MAX - arena_bss_off? Capping
the total at the 4 GB arena maximum would also work.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/41755299448
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (4 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 05/14] libbpf: Support .bss.addr_space.1 section Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:44 ` sashiko-bot
2026-11-04 1:59 ` bot+bpf-ci
2026-11-04 1:23 ` [PATCH bpf-next v3 07/14] selftests/bpf: Test pointers in arena and .bss.addr_space.1 Alexei Starovoitov
` (9 subsequent siblings)
15 siblings, 2 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A variable in arena that points to a different variable in arena gets an
incorrect value:
int __arena x = 42;
int __arena * __arena px = &x;
The compiler emits an R_BPF_64_ABS64 relocation for px. libbpf ignores the
relocations of .addr_space.1. The object loads without an error, and px is
0.
The address of x is not known until libbpf maps the arena. Thus libbpf does
the work in two steps:
- At open it records where each such pointer is and which variable it
points to.
- At load it maps the arena, copies the variables, and adds the address of
the variable to each pointer.
The pointer gets the address in user space. The prog can use the same
value, because an access to arena from a prog uses only the low 32 bits.
A pointer in arena to a variable that is not in arena has no correct value.
Fail the open.
For a pointer to a static variable the relocation refers to the section,
and the offset of the variable is stored in place. The static linker
rejects such a relocation in a data section. Make it add the new offset of
the section, as it does for pointers to static functions. All pointers in
the arena data of Rust objects have this form.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/libbpf.c | 83 +++++++++++++++++++++++++++++++++++++++++-
tools/lib/bpf/linker.c | 10 +++--
2 files changed, 87 insertions(+), 6 deletions(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index a7c53dc5b91d..1172fe7992bf 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -797,6 +797,12 @@ struct bpf_object {
size_t arena_data_sz;
size_t arena_data_off;
size_t arena_bss_off; /* from the start of arena_data */
+ /* pointers to variables in arena that are stored in variables in arena */
+ struct arena_ptr {
+ size_t off; /* where the pointer is, from the start of arena_data */
+ size_t targ_off; /* the symbol it points to, from the start of arena_data */
+ } *arena_ptrs;
+ size_t arena_ptr_cnt;
void *jumptables_data;
size_t jumptables_data_sz;
@@ -4171,10 +4177,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
/*
* Only do relo for section with exec instructions,
- * struct_ops, maps, and read-only data that might
- * have pointers to functions.
+ * struct_ops, maps, read-only data that might
+ * have pointers to functions, and arena.
*/
if (!section_have_execinstr(obj, targ_sec_idx) &&
+ strcmp(name, ".rel" ARENA_SEC) &&
strcmp(name, ".rel" RODATA_SEC) &&
!str_has_pfx(name, ".rel" RODATA_SEC ".") &&
strcmp(name, ".rel" DATA_REL_RO_SEC) &&
@@ -5430,6 +5437,21 @@ bpf_object__reuse_map(struct bpf_map *map)
return 0;
}
+/* The arena is mapped. The address of a variable is the same for the program and for user space. */
+static void relocate_arena_ptrs(struct bpf_object *obj, const struct bpf_map *arena)
+{
+ void *data = arena->mmaped + obj->arena_data_off;
+ struct arena_ptr *p = obj->arena_ptrs;
+ __u64 val;
+ size_t i;
+
+ for (i = 0; i < obj->arena_ptr_cnt; i++, p++) {
+ memcpy(&val, data + p->off, sizeof(val));
+ val += (__u64)(unsigned long)data + p->targ_off;
+ memcpy(data + p->off, &val, sizeof(val));
+ }
+}
+
static int
bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
{
@@ -5856,6 +5878,7 @@ bpf_object__create_maps(struct bpf_object *obj)
memcpy(map->mmaped + obj->arena_data_off, obj->arena_data,
obj->arena_data_sz);
zfree(&obj->arena_data);
+ relocate_arena_ptrs(obj, map);
}
}
if (map->init_slots_sz && map->def.type != BPF_MAP_TYPE_PROG_ARRAY) {
@@ -8223,6 +8246,59 @@ static int bpf_object__collect_rodata_relos(struct bpf_object *obj,
return 0;
}
+/*
+ * Collect pointers that are stored in variables in arena, like
+ * int __arena * __arena p = &var;
+ * They are R_BPF_64_ABS64 relocations. The addend is stored in place.
+ */
+static int bpf_object__collect_arena_relos(struct bpf_object *obj,
+ Elf64_Shdr *shdr, Elf_Data *data)
+{
+ int i, nrels = shdr->sh_size / shdr->sh_entsize;
+ struct arena_ptr *ptrs;
+ const char *sym_name;
+ size_t sym_idx;
+ Elf64_Sym *sym;
+ Elf64_Rel *rel;
+
+ ptrs = libbpf_reallocarray(NULL, nrels, sizeof(*ptrs));
+ if (!ptrs)
+ return -ENOMEM;
+ obj->arena_ptrs = ptrs;
+
+ for (i = 0; i < nrels; i++, ptrs++) {
+ rel = elf_rel_by_idx(data, i);
+ if (!rel) {
+ pr_warn("sec '.rel%s': failed to get relo #%d\n", ARENA_SEC, i);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ sym_idx = ELF64_R_SYM(rel->r_info);
+ sym = elf_sym_by_idx(obj, sym_idx);
+ if (!sym) {
+ pr_warn("sec '.rel%s': symbol #%zu not found for relo #%d\n",
+ ARENA_SEC, sym_idx, i);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ if (ELF64_R_TYPE(rel->r_info) != R_BPF_64_ABS64 ||
+ (sym->st_shndx != obj->efile.arena_data_shndx &&
+ sym->st_shndx != obj->efile.arena_bss_shndx)) {
+ sym_name = elf_sym_str(obj, sym->st_name) ?: "<?>";
+ pr_warn("sec '%s': pointer to '%s' at offset %zu: it's not in arena\n",
+ ARENA_SEC, sym_name, (size_t)rel->r_offset);
+ return -LIBBPF_ERRNO__RELOC;
+ }
+
+ ptrs->off = rel->r_offset;
+ ptrs->targ_off = sym->st_value;
+ if (sym->st_shndx == obj->efile.arena_bss_shndx)
+ ptrs->targ_off += obj->arena_bss_off;
+ obj->arena_ptr_cnt++;
+ }
+ return 0;
+}
+
static int cmp_func_ptrs(const void *_a, const void *_b)
{
const struct func_ptr *a = _a;
@@ -8263,6 +8339,8 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
err = bpf_object__collect_st_ops_relos(obj, shdr, data);
else if (idx == obj->efile.btf_maps_shndx)
err = bpf_object__collect_map_relos(obj, shdr, data);
+ else if (idx == obj->efile.arena_data_shndx)
+ err = bpf_object__collect_arena_relos(obj, shdr, data);
else
err = bpf_object__collect_prog_relos(obj, shdr, data);
if (err)
@@ -10384,6 +10462,7 @@ void bpf_object__close(struct bpf_object *obj)
close(obj->token_fd);
zfree(&obj->arena_data);
+ zfree(&obj->arena_ptrs);
zfree(&obj->jumptables_data);
obj->jumptables_data_sz = 0;
diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
index dba88f4fd6cd..d537388d495c 100644
--- a/tools/lib/bpf/linker.c
+++ b/tools/lib/bpf/linker.c
@@ -2291,12 +2291,14 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
else
insn->imm += sec->dst_off;
} else if (sym_type == R_BPF_64_ABS64 &&
- (sec->shdr->sh_flags & SHF_EXECINSTR)) {
+ ((sec->shdr->sh_flags & SHF_EXECINSTR) ||
+ !strcmp(src_linked_sec->sec_name, ".addr_space.1"))) {
/*
* A pointer to a static function in a data section,
- * which is stored in place as an offset of the
- * function in its section. Data sections are kept
- * in the byte order of the object.
+ * or a pointer in arena to a static variable. It's
+ * stored in place as an offset of the function or
+ * the variable in its section. Data sections are
+ * kept in the byte order of the object.
*/
void *ptr = dst_linked_sec->raw_data + dst_rel->r_offset;
__u64 off;
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena
2026-11-04 1:23 ` [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena Alexei Starovoitov
@ 2026-11-04 1:44 ` sashiko-bot
2026-11-04 2:02 ` Alexei Starovoitov
2026-11-04 1:59 ` bot+bpf-ci
1 sibling, 1 reply; 32+ messages in thread
From: sashiko-bot @ 2026-11-04 1:44 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] libbpf: memory leak and OOB access in bpf_object__collect_arena_relos()
--
commit 146cf79dea60b56b79e9a35a046106f0fa3e6d27
Author: Alexei Starovoitov <ast@kernel.org>
libbpf: Relocate pointers that are stored in arena
This commit teaches libbpf and the BPF static linker to relocate R_BPF_64_ABS64
pointers stored in arena data sections that reference other arena variables. At
open time, libbpf records the relocation offsets, and at load time, it adds the
mapped arena base address and target offsets in place.
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index a7c53dc5b91df..1172fe7992bf5 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
[ ... ]
> @@ -8223,6 +8246,59 @@ static int bpf_object__collect_rodata_relos(struct bpf_object *obj,
> return 0;
> }
>
> +/*
> + * Collect pointers that are stored in variables in arena, like
> + * int __arena * __arena p = &var;
> + * They are R_BPF_64_ABS64 relocations. The addend is stored in place.
> + */
> +static int bpf_object__collect_arena_relos(struct bpf_object *obj,
> + Elf64_Shdr *shdr, Elf_Data *data)
> +{
> + int i, nrels = shdr->sh_size / shdr->sh_entsize;
> + struct arena_ptr *ptrs;
> + const char *sym_name;
> + size_t sym_idx;
> + Elf64_Sym *sym;
> + Elf64_Rel *rel;
> +
> + ptrs = libbpf_reallocarray(NULL, nrels, sizeof(*ptrs));
> + if (!ptrs)
> + return -ENOMEM;
> + obj->arena_ptrs = ptrs;
> +
> + for (i = 0; i < nrels; i++, ptrs++) {
> + rel = elf_rel_by_idx(data, i);
> + if (!rel) {
> + pr_warn("sec '.rel%s': failed to get relo #%d\n", ARENA_SEC, i);
> + return -LIBBPF_ERRNO__FORMAT;
> + }
> +
> + sym_idx = ELF64_R_SYM(rel->r_info);
> + sym = elf_sym_by_idx(obj, sym_idx);
> + if (!sym) {
> + pr_warn("sec '.rel%s': symbol #%zu not found for relo #%d\n",
> + ARENA_SEC, sym_idx, i);
> + return -LIBBPF_ERRNO__FORMAT;
> + }
> +
> + if (ELF64_R_TYPE(rel->r_info) != R_BPF_64_ABS64 ||
> + (sym->st_shndx != obj->efile.arena_data_shndx &&
> + sym->st_shndx != obj->efile.arena_bss_shndx)) {
> + sym_name = elf_sym_str(obj, sym->st_name) ?: "<?>";
> + pr_warn("sec '%s': pointer to '%s' at offset %zu: it's not in arena\n",
> + ARENA_SEC, sym_name, (size_t)rel->r_offset);
> + return -LIBBPF_ERRNO__RELOC;
> + }
> +
> + ptrs->off = rel->r_offset;
> + ptrs->targ_off = sym->st_value;
> + if (sym->st_shndx == obj->efile.arena_bss_shndx)
> + ptrs->targ_off += obj->arena_bss_off;
> + obj->arena_ptr_cnt++;
> + }
> + return 0;
> +}
[Severity: Medium]
Can bpf_object__collect_arena_relos() leak obj->arena_ptrs and overflow it
during relocate_arena_ptrs(), or fail with -ENOMEM when nrels is 0?
In bpf_object__elf_collect(), every SHT_REL section matching .rel.addr_space.1
is marked as SEC_RELO without checking sh_size > 0 or deduplicating target
sections. bpf_object__collect_relos() then calls
bpf_object__collect_arena_relos() for each SEC_RELO section whose sh_info
matches obj->efile.arena_data_shndx.
Because libbpf_reallocarray(NULL, nrels, sizeof(*ptrs)) passes NULL instead of
reallocating obj->arena_ptrs for obj->arena_ptr_cnt + nrels, while the loop
still increments obj->arena_ptr_cnt++ cumulatively, two issues can occur:
1. If multiple SEC_RELO sections target arena_data_shndx, the second call
overwrites obj->arena_ptrs (leaking the prior allocation) and allocates only
nrels entries, while obj->arena_ptr_cnt holds the sum across both calls. When
bpf_object__create_maps() -> relocate_arena_ptrs() later runs:
for (i = 0; i < obj->arena_ptr_cnt; i++, p++) {
memcpy(&val, data + p->off, sizeof(val));
val += (__u64)(unsigned long)data + p->targ_off;
memcpy(data + p->off, &val, sizeof(val));
}
it reads past the end of obj->arena_ptrs and corrupts memory at data + p->off.
2. If an empty .rel.addr_space.1 section has sh_size == 0 (nrels == 0),
libbpf_reallocarray(NULL, 0, sizeof(*ptrs)) calls realloc(NULL, 0), which may
return NULL and cause bpf_object__collect_arena_relos() to return -ENOMEM.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261104012401.1354861-1-alexei.starovoitov@gmail.com?part=6
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena
2026-11-04 1:44 ` sashiko-bot
@ 2026-11-04 2:02 ` Alexei Starovoitov
2026-11-04 2:32 ` Roman Gushchin
0 siblings, 1 reply; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 2:02 UTC (permalink / raw)
To: sashiko-reviews, bpf, roman.gushchin
On Tue, Nov 03, 2026 at 05:44 PM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
>> + ptrs = libbpf_reallocarray(NULL, nrels, sizeof(*ptrs));
>> + if (!ptrs)
>> + return -ENOMEM;
>> + obj->arena_ptrs = ptrs;
[...]
> 1. If multiple SEC_RELO sections target arena_data_shndx, the second call
> overwrites obj->arena_ptrs (leaking the prior allocation) and allocates only
> nrels entries, while obj->arena_ptr_cnt holds the sum across both calls. When
> bpf_object__create_maps() -> relocate_arena_ptrs() later runs:
Not a bug. Only a hand-made ELF has two SHT_REL sections for one section.
clang emits one for .addr_space.1, and the static linker merges them by
name in linker_append_elf_relos().
> 2. If an empty .rel.addr_space.1 section has sh_size == 0 (nrels == 0),
> libbpf_reallocarray(NULL, 0, sizeof(*ptrs)) calls realloc(NULL, 0), which may
> return NULL and cause bpf_object__collect_arena_relos() to return -ENOMEM.
No. Nothing emits an empty SHT_REL section.
glibc returns a pointer for realloc(NULL, 0). Where it's NULL the open of
such a file fails, and that's fine.
I'm replying to a bot hoping that Roman will teach it to learn from maintainers
and stop reporting such bugs in libbpf where input is trusted.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena
2026-11-04 2:02 ` Alexei Starovoitov
@ 2026-11-04 2:32 ` Roman Gushchin
2026-11-04 2:37 ` Alexei Starovoitov
0 siblings, 1 reply; 32+ messages in thread
From: Roman Gushchin @ 2026-11-04 2:32 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: sashiko-reviews, bpf
> On Nov 3, 2026, at 6:02 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
> On Tue, Nov 03, 2026 at 05:44 PM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
>>> + ptrs = libbpf_reallocarray(NULL, nrels, sizeof(*ptrs));
>>> + if (!ptrs)
>>> + return -ENOMEM;
>>> + obj->arena_ptrs = ptrs;
>
> [...]
>
>> 1. If multiple SEC_RELO sections target arena_data_shndx, the second call
>> overwrites obj->arena_ptrs (leaking the prior allocation) and allocates only
>> nrels entries, while obj->arena_ptr_cnt holds the sum across both calls. When
>> bpf_object__create_maps() -> relocate_arena_ptrs() later runs:
>
> Not a bug. Only a hand-made ELF has two SHT_REL sections for one section.
> clang emits one for .addr_space.1, and the static linker merges them by
> name in linker_append_elf_relos().
>
>> 2. If an empty .rel.addr_space.1 section has sh_size == 0 (nrels == 0),
>> libbpf_reallocarray(NULL, 0, sizeof(*ptrs)) calls realloc(NULL, 0), which may
>> return NULL and cause bpf_object__collect_arena_relos() to return -ENOMEM.
>
> No. Nothing emits an empty SHT_REL section.
> glibc returns a pointer for realloc(NULL, 0). Where it's NULL the open of
> such a file fails, and that's fine.
>
> I'm replying to a bot hoping that Roman will teach it to learn from maintainers
> and stop reporting such bugs in libbpf where input is trusted.
Thanks, well appreciated!
It’s going into that direction, now at least it should not raise it again in next versions.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena
2026-11-04 2:32 ` Roman Gushchin
@ 2026-11-04 2:37 ` Alexei Starovoitov
0 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 2:37 UTC (permalink / raw)
To: Roman Gushchin; +Cc: sashiko-reviews, bpf
On Wed Nov 4, 2026 at 2:32 AM UTC, Roman Gushchin wrote:
>> On Nov 3, 2026, at 6:02 PM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>>
>> On Tue, Nov 03, 2026 at 05:44 PM sashiko-bot@kernel.org <sashiko-bot@kernel.org> wrote:
>>>> + ptrs = libbpf_reallocarray(NULL, nrels, sizeof(*ptrs));
>>>> + if (!ptrs)
>>>> + return -ENOMEM;
>>>> + obj->arena_ptrs = ptrs;
>>
>> [...]
>>
>>> 1. If multiple SEC_RELO sections target arena_data_shndx, the second call
>>> overwrites obj->arena_ptrs (leaking the prior allocation) and allocates only
>>> nrels entries, while obj->arena_ptr_cnt holds the sum across both calls. When
>>> bpf_object__create_maps() -> relocate_arena_ptrs() later runs:
>>
>> Not a bug. Only a hand-made ELF has two SHT_REL sections for one section.
>> clang emits one for .addr_space.1, and the static linker merges them by
>> name in linker_append_elf_relos().
>>
>>> 2. If an empty .rel.addr_space.1 section has sh_size == 0 (nrels == 0),
>>> libbpf_reallocarray(NULL, 0, sizeof(*ptrs)) calls realloc(NULL, 0), which may
>>> return NULL and cause bpf_object__collect_arena_relos() to return -ENOMEM.
>>
>> No. Nothing emits an empty SHT_REL section.
>> glibc returns a pointer for realloc(NULL, 0). Where it's NULL the open of
>> such a file fails, and that's fine.
>>
>> I'm replying to a bot hoping that Roman will teach it to learn from maintainers
>> and stop reporting such bugs in libbpf where input is trusted.
>
> Thanks, well appreciated!
> It’s going into that direction, now at least it should not raise it again in next versions.
Thank you!
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena
2026-11-04 1:23 ` [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena Alexei Starovoitov
2026-11-04 1:44 ` sashiko-bot
@ 2026-11-04 1:59 ` bot+bpf-ci
2026-11-04 2:30 ` Alexei Starovoitov
1 sibling, 1 reply; 32+ messages in thread
From: bot+bpf-ci @ 2026-11-04 1:59 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3161 bytes --]
> libbpf: Relocate pointers that are stored in arena
TL;DR:
- libbpf arena relocation may not bounds-check r_offset, so a crafted object
could crash or corrupt memory at load
- a second arena relocation section may leak the array and let the count
overrun it
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
[ ... ]
> +/* The arena is mapped. The address of a variable is the same for the program and for user space. */
> +static void relocate_arena_ptrs(struct bpf_object *obj, const struct bpf_map *arena)
> +{
> + void *data = arena->mmaped + obj->arena_data_off;
> + struct arena_ptr *p = obj->arena_ptrs;
> + __u64 val;
> + size_t i;
> +
> + for (i = 0; i < obj->arena_ptr_cnt; i++, p++) {
> + memcpy(&val, data + p->off, sizeof(val));
> + val += (__u64)(unsigned long)data + p->targ_off;
> + memcpy(data + p->off, &val, sizeof(val));
> + }
> +}
[ ... ]
> +static int bpf_object__collect_arena_relos(struct bpf_object *obj,
> + Elf64_Shdr *shdr, Elf_Data *data)
> +{
[ ... ]
> + ptrs = libbpf_reallocarray(NULL, nrels, sizeof(*ptrs));
> + if (!ptrs)
> + return -ENOMEM;
> + obj->arena_ptrs = ptrs;
[ ... ]
> + ptrs->off = rel->r_offset;
> + ptrs->targ_off = sym->st_value;
> + if (sym->st_shndx == obj->efile.arena_bss_shndx)
> + ptrs->targ_off += obj->arena_bss_off;
> + obj->arena_ptr_cnt++;
> + }
> + return 0;
> +}
Can rel->r_offset point past the end of the arena data here? It is stored
in ptrs->off without being compared against the size of the .addr_space.1
data.
At load, relocate_arena_ptrs() reads and writes 8 bytes at data + p->off:
memcpy(&val, data + p->off, sizeof(val));
...
memcpy(data + p->off, &val, sizeof(val));
When FEAT_LDIMM64_FULL_RANGE_OFF is supported, arena_data_off is
mmap_sz - roundup(arena_data_sz, page), so an r_offset larger than
roundup(arena_data_sz, page) - 8 lands beyond the arena mapping. A crafted
object would open fine and then crash or corrupt memory in
bpf_object__load().
The sibling collectors reject such offsets, for example
bpf_object__collect_rodata_relos():
rel->r_offset >= scn_data->d_size ||
scn_data->d_size - rel->r_offset < sizeof(__u64)
Should this function do a similar check and return -LIBBPF_ERRNO__FORMAT?
Also, obj->arena_ptrs is overwritten on every call while obj->arena_ptr_cnt
keeps accumulating. If a malformed object has two SHT_REL sections whose
sh_info refers to the arena section, the first array is leaked and
arena_ptr_cnt can exceed the size of the second array:
obj->arena_ptrs = ptrs;
...
obj->arena_ptr_cnt++;
relocate_arena_ptrs() would then read past the heap allocation and use
garbage off values as write targets. Should a second call be rejected when
obj->arena_ptrs is already set?
No later commit in the series touches tools/lib/bpf, so neither case is
addressed later.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/41755299448
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena
2026-11-04 1:59 ` bot+bpf-ci
@ 2026-11-04 2:30 ` Alexei Starovoitov
0 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 2:30 UTC (permalink / raw)
To: bot+bpf-ci, bpf
Cc: daniel, andrii, eddyz87, memxor, martin.lau, yonghong.song, mason,
ihor.solodrai
On Tue, Nov 03, 2026 at 05:59 PM bot+bpf-ci@kernel.org <bot+bpf-ci@kernel.org> wrote:
>> + ptrs->off = rel->r_offset;
>> + ptrs->targ_off = sym->st_value;
>
> Can rel->r_offset point past the end of the arena data here? It is stored
> in ptrs->off without being compared against the size of the .addr_space.1
> data.
[...]
> Should this function do a similar check and return -LIBBPF_ERRNO__FORMAT?
Not a bug. Only a hand-made ELF has such r_offset.
The input of libbpf is trusted.
I thought bpf CI knows this better than sashiko :(
> relocate_arena_ptrs() would then read past the heap allocation and use
> garbage off values as write targets. Should a second call be rejected when
> obj->arena_ptrs is already set?
No. See my reply to sashiko.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH bpf-next v3 07/14] selftests/bpf: Test pointers in arena and .bss.addr_space.1
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (5 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 06/14] libbpf: Relocate pointers that are stored in arena Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 08/14] bpf: Introduce bpf register BPF_REG_ARENA_SP Alexei Starovoitov
` (8 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The initial values of variables in arena point to other variables in arena:
pointer points to
------------ -------------------------------------------
px an int
parr the third element of an array
head, next the elements of a list
pbig an element of an array in .bss.addr_space.1
psx a static int
psbig an element of a static array in .bss.addr_space.1
The prog and user space must see the same pointers:
- The prog follows each pointer and examines the value that it finds.
- User space compares px, parr, head and next with the addresses in the
skeleton.
- User space reads through pbig, psx and psbig the values that the prog
stored there. Static variables are not in the skeleton.
The selftests link each object with the static linker. Thus the object with
psx and psbig does not build without the change in linker.c.
The variables in .bss.addr_space.1 must be zero and aligned, and the object
file must be smaller than these variables. The size of .addr_space.1 is not
a multiple of the alignment, thus libbpf must align the second section.
A second object has a pointer in arena to a variable that is not in arena.
The open of this object must fail.
The test is serial, as the other tests in prog_tests/arena_*.c. See commit
a14f9ba080c1 ("selftests/bpf: Run arena tests serially").
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile.skel | 3 +-
.../bpf/prog_tests/arena_data_ptrs.c | 70 +++++++++
.../selftests/bpf/progs/arena_data_ptrs.c | 138 ++++++++++++++++++
.../bpf/progs/arena_data_ptrs_fail.c | 25 ++++
4 files changed, 235 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_data_ptrs.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c
diff --git a/tools/testing/selftests/bpf/Makefile.skel b/tools/testing/selftests/bpf/Makefile.skel
index e8699903feef..b8eaecc7f429 100644
--- a/tools/testing/selftests/bpf/Makefile.skel
+++ b/tools/testing/selftests/bpf/Makefile.skel
@@ -20,7 +20,8 @@ ifneq ($(BPF_CC),)
BPF_SRCS := $(notdir $(wildcard progs/*.c))
BPF_OBJS := $(patsubst %.c,$(RDIR)/%.bpf.o,$(BPF_SRCS))
-SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
+SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c \
+ arena_data_ptrs_fail.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h linked_arena.skel.h \
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c b/tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c
new file mode 100644
index 000000000000..a01c712ff3d0
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+
+struct elem {
+ struct elem *next;
+ int val;
+};
+
+#include "arena_data_ptrs.skel.h"
+
+static int run_prog(struct bpf_program *prog)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+
+ if (!ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(prog), &opts), "run"))
+ return -1;
+ return opts.retval;
+}
+
+static void test_ptrs_and_bss(void)
+{
+ struct arena_data_ptrs *skel;
+ size_t elf_sz;
+
+ skel = arena_data_ptrs__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ ASSERT_OK(run_prog(skel->progs.use_ptrs), "use_ptrs");
+ if (skel->bss->skip) {
+ printf("%s:SKIP:compiler doesn't support arena_cast\n", __func__);
+ test__skip();
+ goto out;
+ }
+ ASSERT_OK(run_prog(skel->progs.use_bss), "use_bss");
+
+ /* addresses are the same in user space */
+ ASSERT_EQ((void *)skel->arena->px, (void *)&skel->arena->x, "px");
+ ASSERT_EQ((void *)skel->arena->parr, (void *)&skel->arena->arr[2], "parr");
+ ASSERT_EQ((void *)skel->arena->head, (void *)&skel->arena->n1, "head");
+ ASSERT_EQ((void *)skel->arena->n1.next, (void *)&skel->arena->n2, "n1.next");
+ ASSERT_EQ(*(long *)skel->arena->pbig, 7, "*pbig");
+ /* static variables are not in the skeleton */
+ ASSERT_EQ(*(int *)skel->arena->psx, 45, "*psx");
+ ASSERT_EQ(*(long *)skel->arena->psbig, 9, "*psbig");
+
+ /* 'big' is not in the file */
+ arena_data_ptrs__elf_bytes(&elf_sz);
+ ASSERT_LT(elf_sz, 8192 * sizeof(long), "elf_sz");
+out:
+ arena_data_ptrs__destroy(skel);
+}
+
+static void test_ptr_to_not_arena(void)
+{
+ struct bpf_object *obj;
+
+ obj = bpf_object__open_file("./arena_data_ptrs_fail.bpf.o", NULL);
+ if (!ASSERT_ERR_PTR(obj, "open"))
+ bpf_object__close(obj);
+}
+
+void serial_test_arena_data_ptrs(void)
+{
+ if (test__start_subtest("ptrs_and_bss"))
+ test_ptrs_and_bss();
+ if (test__start_subtest("ptr_to_not_arena"))
+ test_ptr_to_not_arena();
+}
diff --git a/tools/testing/selftests/bpf/progs/arena_data_ptrs.c b/tools/testing/selftests/bpf/progs/arena_data_ptrs.c
new file mode 100644
index 000000000000..bf3048ab2d7a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_data_ptrs.c
@@ -0,0 +1,138 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 64);
+} arena SEC(".maps");
+
+bool skip;
+
+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
+
+struct elem {
+ struct elem __arena *next;
+ int val;
+};
+
+int __arena x = 42;
+int __arena arr[4] = { 1, 2, 3, 4 };
+struct elem __arena n3 = { NULL, 3 };
+struct elem __arena n2 = { &n3, 20 };
+struct elem __arena n1 = { &n2, 100 };
+
+/* variables that are zero and take no space in the file */
+#define __arena_bss __arena SEC(".bss.addr_space.1")
+char __arena_bss one;
+long __arena_bss big[8192];
+char __arena_bss aligned64[64] __attribute__((aligned(64)));
+
+/* a pointer to a static variable is relocated against the section, not against a symbol */
+static int __arena sx = 44;
+static long __arena_bss sbig[16];
+
+/* pointers to variables in arena that are stored in variables in arena */
+int __arena * __arena px = &x;
+int __arena * __arena parr = &arr[2];
+struct elem __arena * __arena head = &n1;
+long __arena * __arena pbig = &big[8000];
+int __arena * __arena psx = &sx;
+long __arena * __arena psbig = &sbig[3];
+
+/* the size of .addr_space.1 is not a multiple of the alignment of .bss.addr_space.1 */
+char __arena odd = 1;
+
+SEC("syscall")
+int use_ptrs(void *ctx)
+{
+ struct elem __arena *n;
+ int i, sum = 0;
+
+ if (*px != 42)
+ return 1;
+ *px = 43;
+ if (x != 43)
+ return 2;
+ if (parr[0] != 3 || parr[1] != 4)
+ return 3;
+ for (n = head, i = 0; n && i < 10; n = n->next, i++)
+ sum += n->val;
+ if (sum != 123)
+ return 4;
+ if (*psx != 44)
+ return 5;
+ *psx = 45;
+ if (sx != 45)
+ return 6;
+ return 0;
+}
+
+SEC("syscall")
+int use_bss(void *ctx)
+{
+ unsigned long addr = (unsigned long)&aligned64;
+
+ if (one || big[0] || big[8000] || big[8191] || aligned64[63])
+ return 1;
+ *pbig = 7;
+ if (big[8000] != 7 || big[7999] || big[8001])
+ return 2;
+ big[8191] = 8;
+ one = 1;
+ /* variables that are not zero are where they were */
+ if (arr[3] != 4)
+ return 3;
+ /* hide the address from the compiler, it knows that '& 63' is 0 */
+ asm volatile ("" : "+r"(addr));
+ if (addr & 63)
+ return 4;
+ if (sbig[2] || sbig[3] || sbig[4])
+ return 5;
+ *psbig = 9;
+ if (sbig[3] != 9 || sbig[2] || sbig[4])
+ return 6;
+ return 0;
+}
+
+#else
+
+/* The skeleton has to have what user space looks at */
+struct elem {
+ struct elem *next;
+ int val;
+};
+
+int __arena_global x;
+int __arena_global arr[4];
+struct elem __arena_global n2;
+struct elem __arena_global n1;
+int * __arena_global px;
+int * __arena_global parr;
+struct elem * __arena_global head;
+long * __arena_global pbig;
+int * __arena_global psx;
+long * __arena_global psbig;
+
+SEC("syscall")
+int use_ptrs(void *ctx)
+{
+ skip = true;
+ return 0;
+}
+
+SEC("syscall")
+int use_bss(void *ctx)
+{
+ skip = true;
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c b/tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c
new file mode 100644
index 000000000000..93533dea5f71
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+/* the pointer is in arena, what it points to is not */
+int y = 1;
+int *py SEC(".addr_space.1") = &y;
+
+SEC("syscall")
+int prog(void *ctx)
+{
+ return y;
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 08/14] bpf: Introduce bpf register BPF_REG_ARENA_SP
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (6 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 07/14] selftests/bpf: Test pointers in arena and .bss.addr_space.1 Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 09/14] bpf: Add stacks for programs to arena Alexei Starovoitov
` (7 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A Rust function that gets &T does not know where T is. T can be on the
heap, a static, or a local variable of the caller. The same insn accesses
all three. The heap and the statics of a Rust prog are in arena. Thus a
local variable whose address is taken must be in arena too, and the prog
needs a second stack there.
Each function must find the end of this stack. The places for the stack
pointer that were tried:
stack pointer in what is wrong with it
----------------------- ----------------------------------------------
a per-cpu variable A second run on the same cpu moves it, and the
first run can continue before the second ends.
bpf_run_ctx Load, add and store in each function.
a hidden argument It takes an argument register and a callee
saved register. BTF does not match the code.
r9, reserved The verifier changes too, and the prog has one
register less.
a new register It costs verifier code and a register in each
JIT. This series.
Introduce BPF_REG_ARENA_SP, r12, the stack pointer of the stack in arena.
BPF_REG_AX stays the last register and becomes r13:
register before after
---------------- ------ -----
BPF_REG_PARAMS r11 r11
BPF_REG_ARENA_SP r12
BPF_REG_AX r12 r13
This patch only reserves the number. The verifier continues to reject insns
with r12.
BPF_REG_AX is visible in xlated insns. Adjust the selftests that match it
there.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/filter.h | 5 +-
kernel/bpf/core.c | 3 +-
.../selftests/bpf/prog_tests/ctx_rewrite.c | 14 ++--
.../bpf/progs/verifier_bpf_fastcall.c | 24 +++----
.../selftests/bpf/progs/verifier_may_goto_1.c | 12 ++--
.../selftests/bpf/progs/verifier_patch_list.c | 6 +-
.../selftests/bpf/progs/verifier_sdiv.c | 64 +++++++++----------
7 files changed, 65 insertions(+), 63 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index e9f619deb2a8..abc759ad14a6 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -61,8 +61,9 @@ struct ctl_table_header;
/* Kernel hidden auxiliary/helper register. */
#define BPF_REG_PARAMS MAX_BPF_REG
-#define BPF_REG_AX (MAX_BPF_REG + 1)
-#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)
+#define BPF_REG_ARENA_SP (MAX_BPF_REG + 1)
+#define BPF_REG_AX (MAX_BPF_REG + 2)
+#define MAX_BPF_EXT_REG (MAX_BPF_REG + 3)
#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG
/* unused opcode to mark special call to bpf_tail_call() helper */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index dcaf9cf13ce9..06d6d45b1c0e 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1578,7 +1578,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
u32 imm_rnd = get_random_u32();
int off;
- BUILD_BUG_ON(BPF_REG_PARAMS + 2 != MAX_BPF_JIT_REG);
+ BUILD_BUG_ON(BPF_REG_PARAMS + 3 != MAX_BPF_JIT_REG);
+ BUILD_BUG_ON(BPF_REG_ARENA_SP + 2 != MAX_BPF_JIT_REG);
BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
/* Constraints on AX register:
diff --git a/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c b/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
index 2c3124092b73..2d0bb771626d 100644
--- a/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
+++ b/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
@@ -69,19 +69,19 @@ static struct test_case test_cases[] = {
#if defined(__x86_64__) || defined(__aarch64__)
{
N(SCHED_CLS, struct __sk_buff, tstamp),
- .read = "r12 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
- "if w12 & 0x4 goto pc+1;"
+ .read = "r13 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
+ "if w13 & 0x4 goto pc+1;"
"goto pc+4;"
- "if w12 & 0x3 goto pc+1;"
+ "if w13 & 0x3 goto pc+1;"
"goto pc+2;"
"$dst = 0;"
"goto pc+1;"
"$dst = *(u64 *)($ctx + sk_buff::tstamp);",
- .write = "r12 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
- "if w12 & 0x4 goto pc+1;"
+ .write = "r13 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
+ "if w13 & 0x4 goto pc+1;"
"goto pc+2;"
- "w12 &= -4;"
- "*(u8 *)($ctx + sk_buff::__mono_tc_offset) = r12;"
+ "w13 &= -4;"
+ "*(u8 *)($ctx + sk_buff::__mono_tc_offset) = r13;"
"*(u64 *)($ctx + sk_buff::tstamp) = $src;",
},
#endif
diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
index d908298b9fc7..314652a0006d 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
@@ -915,13 +915,13 @@ __xlated("...")
/* may_goto expansion starts, r0 and r1 are live */
__xlated("10: r2 = ")
__xlated("12: r2 = &(void __percpu *)(r2)")
-__xlated("13: r12 = *(u64 *)(r2 +0)")
-__xlated("14: if r12 == 0x0 goto pc+6")
-__xlated("15: r12 -= 1")
-__xlated("16: if r12 != 0x0 goto pc+2")
-__xlated("17: r12 = r2")
+__xlated("13: r13 = *(u64 *)(r2 +0)")
+__xlated("14: if r13 == 0x0 goto pc+6")
+__xlated("15: r13 -= 1")
+__xlated("16: if r13 != 0x0 goto pc+2")
+__xlated("17: r13 = r2")
__xlated("18: call unknown")
-__xlated("19: *(u64 *)(r2 +0) = r12")
+__xlated("19: *(u64 *)(r2 +0) = r13")
/* may_goto expansion ends */
__xlated("20: *(u64 *)(r10 -8) = r1")
__xlated("21: exit")
@@ -958,13 +958,13 @@ __xlated("7: call bpf_get_smp_processor_id")
/* may_goto expansion starts, r0 and r1 are live */
__xlated("8: r2 = ")
__xlated("10: r2 = &(void __percpu *)(r2)")
-__xlated("11: r12 = *(u64 *)(r2 +0)")
-__xlated("12: if r12 == 0x0 goto pc+6")
-__xlated("13: r12 -= 1")
-__xlated("14: if r12 != 0x0 goto pc+2")
-__xlated("15: r12 = r2")
+__xlated("11: r13 = *(u64 *)(r2 +0)")
+__xlated("12: if r13 == 0x0 goto pc+6")
+__xlated("13: r13 -= 1")
+__xlated("14: if r13 != 0x0 goto pc+2")
+__xlated("15: r13 = r2")
__xlated("16: call unknown")
-__xlated("17: *(u64 *)(r2 +0) = r12")
+__xlated("17: *(u64 *)(r2 +0) = r13")
/* may_goto expansion ends */
__xlated("18: *(u64 *)(r10 -8) = r1")
__xlated("19: exit")
diff --git a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
index 20152a15ca36..8d231fb2a317 100644
--- a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
+++ b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
@@ -97,13 +97,13 @@ __xlated("5: *(u64 *)(r0 +16) = 0")
/* may_goto, r0 is dead */
__xlated("6: r0 = ")
__xlated("8: r0 = &(void __percpu *)(r0)")
-__xlated("9: r12 = *(u64 *)(r0 +0)")
-__xlated("10: if r12 == 0x0 goto pc+6")
-__xlated("11: r12 -= 1")
-__xlated("12: if r12 != 0x0 goto pc+2")
-__xlated("13: r12 = r0")
+__xlated("9: r13 = *(u64 *)(r0 +0)")
+__xlated("10: if r13 == 0x0 goto pc+6")
+__xlated("11: r13 -= 1")
+__xlated("12: if r13 != 0x0 goto pc+2")
+__xlated("13: r13 = r0")
__xlated("14: call unknown")
-__xlated("15: *(u64 *)(r0 +0) = r12")
+__xlated("15: *(u64 *)(r0 +0) = r13")
__xlated("16: r0 = 1")
__xlated("17: r0 = 2")
__xlated("18: exit")
diff --git a/tools/testing/selftests/bpf/progs/verifier_patch_list.c b/tools/testing/selftests/bpf/progs/verifier_patch_list.c
index 581b75b7b5c5..5e8166bb79cc 100644
--- a/tools/testing/selftests/bpf/progs/verifier_patch_list.c
+++ b/tools/testing/selftests/bpf/progs/verifier_patch_list.c
@@ -103,8 +103,8 @@ __xlated("10: goto pc+1")
__xlated("11: r0 /= r1")
__xlated("12: r2 = ")
__xlated("14: r2 = &(void __percpu *)(r2)")
-__xlated("15: r12 = *(u64 *)(r2 +0)")
-__xlated("16: if r12 == 0x0 goto pc+9")
+__xlated("15: r13 = *(u64 *)(r2 +0)")
+__xlated("16: if r13 == 0x0 goto pc+9")
__xlated("...")
__xlated("22: if r1 != 0x0 goto pc+2")
__xlated("23: w0 ^= w0")
@@ -220,7 +220,7 @@ __arch_arm64
__success
__xlated("12: if r6 != 0x0 goto pc-3")
__xlated("...")
-__xlated("32783: if r12 != 0x0 goto pc+1")
+__xlated("32783: if r13 != 0x0 goto pc+1")
__xlated("32784: gotol pc-32773")
__naked void patch_list_may_goto_gotol(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_sdiv.c b/tools/testing/selftests/bpf/progs/verifier_sdiv.c
index 4ac92335da91..02e2065c9669 100644
--- a/tools/testing/selftests/bpf/progs/verifier_sdiv.c
+++ b/tools/testing/selftests/bpf/progs/verifier_sdiv.c
@@ -779,10 +779,10 @@ __arch_loongarch
__xlated("0: r2 = 0x8000000000000000")
__xlated("2: r3 = -1")
__xlated("3: r4 = r2")
-__xlated("4: r12 = r3")
-__xlated("5: r12 += 1")
-__xlated("6: if r12 > 0x1 goto pc+4")
-__xlated("7: if r12 == 0x0 goto pc+1")
+__xlated("4: r13 = r3")
+__xlated("5: r13 += 1")
+__xlated("6: if r13 > 0x1 goto pc+4")
+__xlated("7: if r13 == 0x0 goto pc+1")
__xlated("8: r2 = 0")
__xlated("9: r2 = -r2")
__xlated("10: goto pc+1")
@@ -814,10 +814,10 @@ __arch_x86_64
__arch_loongarch
__xlated("0: r2 = 5")
__xlated("1: r3 = -1")
-__xlated("2: r12 = r3")
-__xlated("3: r12 += 1")
-__xlated("4: if r12 > 0x1 goto pc+4")
-__xlated("5: if r12 == 0x0 goto pc+1")
+__xlated("2: r13 = r3")
+__xlated("3: r13 += 1")
+__xlated("4: if r13 > 0x1 goto pc+4")
+__xlated("5: if r13 == 0x0 goto pc+1")
__xlated("6: r2 = 0")
__xlated("7: r2 = -r2")
__xlated("8: goto pc+1")
@@ -895,10 +895,10 @@ __arch_loongarch
__xlated("0: w2 = -2147483648")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+4")
-__xlated("6: if w12 == 0x0 goto pc+1")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+4")
+__xlated("6: if w13 == 0x0 goto pc+1")
__xlated("7: w2 = 0")
__xlated("8: w2 = -w2")
__xlated("9: goto pc+1")
@@ -931,10 +931,10 @@ __arch_loongarch
__xlated("0: w2 = -5")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+4")
-__xlated("6: if w12 == 0x0 goto pc+1")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+4")
+__xlated("6: if w13 == 0x0 goto pc+1")
__xlated("7: w2 = 0")
__xlated("8: w2 = -w2")
__xlated("9: goto pc+1")
@@ -1013,10 +1013,10 @@ __arch_loongarch
__xlated("0: r2 = 0x8000000000000000")
__xlated("2: r3 = -1")
__xlated("3: r4 = r2")
-__xlated("4: r12 = r3")
-__xlated("5: r12 += 1")
-__xlated("6: if r12 > 0x1 goto pc+3")
-__xlated("7: if r12 == 0x1 goto pc+3")
+__xlated("4: r13 = r3")
+__xlated("5: r13 += 1")
+__xlated("6: if r13 > 0x1 goto pc+3")
+__xlated("7: if r13 == 0x1 goto pc+3")
__xlated("8: w2 = 0")
__xlated("9: goto pc+1")
__xlated("10: r2 s%= r3")
@@ -1044,10 +1044,10 @@ __arch_loongarch
__xlated("0: r2 = 5")
__xlated("1: r3 = -1")
__xlated("2: r4 = r2")
-__xlated("3: r12 = r3")
-__xlated("4: r12 += 1")
-__xlated("5: if r12 > 0x1 goto pc+3")
-__xlated("6: if r12 == 0x1 goto pc+3")
+__xlated("3: r13 = r3")
+__xlated("4: r13 += 1")
+__xlated("5: if r13 > 0x1 goto pc+3")
+__xlated("6: if r13 == 0x1 goto pc+3")
__xlated("7: w2 = 0")
__xlated("8: goto pc+1")
__xlated("9: r2 s%= r3")
@@ -1121,10 +1121,10 @@ __arch_loongarch
__xlated("0: w2 = -2147483648")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+3")
-__xlated("6: if w12 == 0x1 goto pc+4")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+3")
+__xlated("6: if w13 == 0x1 goto pc+4")
__xlated("7: w2 = 0")
__xlated("8: goto pc+1")
__xlated("9: w2 s%= w3")
@@ -1154,10 +1154,10 @@ __arch_loongarch
__xlated("0: w2 = -5")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+3")
-__xlated("6: if w12 == 0x1 goto pc+4")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+3")
+__xlated("6: if w13 == 0x1 goto pc+4")
__xlated("7: w2 = 0")
__xlated("8: goto pc+1")
__xlated("9: w2 s%= w3")
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 09/14] bpf: Add stacks for programs to arena
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (7 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 08/14] bpf: Introduce bpf register BPF_REG_ARENA_SP Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 10/14] bpf, x86: Add BPF_ATOMIC_LOCAL Alexei Starovoitov
` (6 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog that keeps local variables in arena needs a stack in arena while it
runs. r12 (BPF_REG_ARENA_SP) is the stack pointer of this stack.
One stack for each cpu is not sufficient. A run of a prog is interrupted,
is preempted or sleeps, and a second run starts on the same cpu. The second
run cannot put its frames after the frames of the first run, because the
first run can continue before the second run ends:
cpu 0 run 1: f() run 2: h() run 1: f() calls g()
stack | f | | f | h | | f | g |
^ g overwrites h
Thus each run takes a stack when it starts and gives it back when it exits.
Usually there is only one run on a cpu. Make this case a few insns without
a call:
stack that a run gets condition nsec per run
------------------------ ----------------------------- ------------
the per-cpu stack no other run is on this cpu 2.2
one of 16 more per cpu the per-cpu stack is taken 9.1 more
new pages for this run all 17 stacks are taken 1300 to 1600
The times are for x86-64 (Zen 5), in a KVM guest. The first time is the
difference to a prog that has the same variables in the r10 stack. The
third time is for a stack of 1 page.
Each run increments the per-cpu counter 'busy'. The run that finds 0 has
the per-cpu stack, and it stores 0 when it exits. The other runs call
bpf_arena_stack_get() and bpf_arena_stack_put(). One bit of r12,
BPF_ARENA_STACK_SLOW, shows at the exit which of the two ways is necessary.
The other choices:
in this patch what was rejected, and why
----------------------------- ----------------------------------------
The kernel supplies the The compiler or libbpf supplies them:
stacks, one set for each the code that takes a stack is in each
arena. object file, for ever.
A stack is contiguous. It 4 fixed stacks per cpu: 512 Mbyte of
has the size that the prog addresses. Segments: each function can
needs: 32 Kbyte of frames at fail. Pages on a fault: a prog can fail
most. It gets pages at the in the middle of a run.
first use.
The pages are regular pages Pages that user space does not see: no
of the arena. gain. User space can corrupt all other
data of the prog.
A run takes a stack: a Preemption off: no sleepable progs. Two
counter, then more stacks, calls, get and put: 4 nsec. One way for
then new pages. sleepable progs and one for the others:
two mechanisms. Skip the second run: not
possible for a scheduler.
The third row is for a burst. The kernel frees pages of arena later, in
a work item. It takes 1700 to 2000 nsec more for each such run, on the same
cpu, and that is slower than the allocation. While all 17 stacks of
a cpu stay taken, runs that follow each other without a pause take pages
faster than the work item frees them. When the arena is full, runs are
skipped and allocations of the prog fail until the work item catches up.
This patch adds the data and the two functions.
"bpf: Support BPF_REG_ARENA_SP in the verifier" emits the insns that use
them. Progs without r12 get the same code.
Memory:
- A stack gets pages of the arena at its first use, in
bpf_arena_stack_get(), and keeps them until a prog needs a larger stack.
- The bookkeeping has 152 bytes per cpu. It is in kernel memory, because
the prog and user space can write to arena.
- struct bpf_arena has one more pointer and no longer fits in 512 bytes.
Each arena, with or without stacks, takes 512 bytes more.
- When there is no memory for a stack, the prog does not run. The kernel
counts a miss and writes a message to the stderr of the prog. A callback
that gets no stack does not run either: it returns 0 to the helper, and
the prog continues.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 30 ++++++++++
kernel/bpf/arena.c | 142 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 172 insertions(+)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5413fa5d2b2b..5c059a05f89d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -673,6 +673,31 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock);
u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
+/*
+ * The stack in arena of a cpu. The invocation of a program that finds it free
+ * takes it. The others, that interrupt or preempt that one, take a stack from
+ * bpf_arena_stack_get(). bpf_insert_arena_stack() emits the code.
+ */
+struct bpf_arena_cpu_stack {
+ /*
+ * Every invocation increments it. The one that finds 0 has the stack
+ * and sets it to 0 when it exits.
+ */
+ u64 busy;
+ u32 base; /* r12 when the main program starts, the low 32 bits of the address */
+ u32 size;
+};
+
+/*
+ * r12 is a multiple of 32 when the main program starts in the stack of the cpu.
+ * It has this bit set in all other stacks. They go back with bpf_arena_stack_put().
+ */
+#define BPF_ARENA_STACK_SLOW 16
+
+int bpf_arena_stack_init(struct bpf_arena *arena);
+void __percpu *bpf_arena_stack_percpu(struct bpf_arena *arena);
+u64 bpf_arena_stack_get(struct bpf_prog_aux *aux, u64 busy);
+void bpf_arena_stack_put(struct bpf_prog_aux *aux, u32 base);
u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
@@ -1919,6 +1944,11 @@ struct bpf_prog_aux {
u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
struct mutex ext_mutex; /* mutex for freplace_link_cnt and prog_array_member_cnt */
struct bpf_arena *arena;
+ /*
+ * The size of the stack in arena that the program needs, a multiple of
+ * the page size. 0 when r12 is not used.
+ */
+ u32 arena_stack_size;
void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index ec2bda1a6ebf..366c4c2b6a41 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -68,6 +68,33 @@ struct bpf_arena {
struct irq_work free_irq;
struct work_struct free_work;
struct llist_head free_spans;
+ struct arena_stacks __percpu *stacks;
+};
+
+/*
+ * The stack in arena, see BPF_REG_ARENA_SP. An invocation of a program takes one
+ * when it starts and gives it back when it exits.
+ *
+ * Nothing here is in arena: the program and user space can write there.
+ *
+ * Invocations on a cpu don't nest: a program that is preempted or sleeps can exit
+ * before the one that started later. Hence a bit per stack and not a counter.
+ * Programs don't migrate, so the stack is returned on the cpu it was taken on.
+ *
+ * Most of the time there is one invocation on a cpu. It takes the stack of
+ * the cpu, see struct bpf_arena_cpu_stack, and doesn't need these.
+ */
+#define ARENA_STACK_SLOTS 16
+
+struct arena_stack {
+ u32 base; /* low 32 bits of r12 when the main program starts */
+ u32 size; /* 0 until the first use */
+};
+
+struct arena_stacks {
+ struct bpf_arena_cpu_stack cpu;
+ unsigned long busy;
+ struct arena_stack slot[ARENA_STACK_SLOTS];
};
static void arena_free_worker(struct work_struct *work);
@@ -401,6 +428,7 @@ static void arena_map_free(struct bpf_map *map)
free_vm_area(arena->kern_vm);
range_tree_destroy(&arena->rt);
__free_page(arena->scratch_page);
+ free_percpu(arena->stacks);
bpf_map_area_free(arena);
}
@@ -1040,6 +1068,120 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
irq_work_queue(&arena->free_irq);
}
+/* Called when a program that uses BPF_REG_ARENA_SP is loaded */
+int bpf_arena_stack_init(struct bpf_arena *arena)
+{
+ struct arena_stacks __percpu *stacks;
+
+ if (READ_ONCE(arena->stacks))
+ return 0;
+ stacks = bpf_map_alloc_percpu(&arena->map, sizeof(*stacks), __alignof__(*stacks),
+ GFP_USER | __GFP_NOWARN);
+ if (!stacks)
+ return -ENOMEM;
+ if (cmpxchg(&arena->stacks, NULL, stacks))
+ free_percpu(stacks);
+ return 0;
+}
+
+void __percpu *bpf_arena_stack_percpu(struct bpf_arena *arena)
+{
+ return arena->stacks;
+}
+
+/*
+ * Called at the start of the main program or of a callback, in any context. When
+ * @busy is 0 the program has taken the stack of the cpu and it's too small.
+ * Returns the low 32 bits of r12 with bit 32 set, or 0 when there is no memory.
+ * The program doesn't run then.
+ */
+u64 bpf_arena_stack_get(struct bpf_prog_aux *aux, u64 busy)
+{
+ struct arena_stacks *stacks = this_cpu_ptr(aux->arena->stacks);
+ struct bpf_arena_cpu_stack *cpu = &stacks->cpu;
+ struct bpf_arena *arena = aux->arena;
+ u32 size = aux->arena_stack_size;
+ struct arena_stack *st = NULL, old;
+ struct bpf_stream_stage ss;
+ long addr;
+ int i;
+
+ if (!busy) {
+ /* It's not there yet, or programs that used it needed less */
+ addr = arena_alloc_pages(arena, 0, size >> PAGE_SHIFT, NUMA_NO_NODE, false);
+ if (!addr) {
+ WRITE_ONCE(cpu->busy, 0);
+ goto err;
+ }
+ old = (struct arena_stack){ cpu->base, cpu->size };
+ /* r12 is not 0: neither at the start of the stack nor at the end */
+ cpu->base = addr + 32;
+ cpu->size = size - 64;
+ if (old.size)
+ arena_free_pages(arena, old.base, (old.size + 64) >> PAGE_SHIFT, false);
+ return BIT_ULL(32) | cpu->base;
+ }
+
+ while ((i = ffz(READ_ONCE(stacks->busy))) < ARENA_STACK_SLOTS) {
+ if (test_and_set_bit(i, &stacks->busy))
+ continue;
+ st = &stacks->slot[i];
+ if (st->size >= size)
+ return BIT_ULL(32) | st->base;
+ break;
+ }
+
+ /*
+ * It's the first use of the slot, or programs that used it needed less,
+ * or all slots are busy and bpf_arena_stack_put() is going to free the pages.
+ */
+ addr = arena_alloc_pages(arena, 0, size >> PAGE_SHIFT, NUMA_NO_NODE, false);
+ if (!addr) {
+ if (st)
+ clear_bit(i, &stacks->busy);
+ goto err;
+ }
+ if (!st)
+ return BIT_ULL(32) | (u32)(addr + BPF_ARENA_STACK_SLOW);
+
+ /*
+ * bpf_arena_stack_put() finds a slot by its base. No slot may name pages that
+ * another run can get.
+ */
+ old = *st;
+ st->base = addr + BPF_ARENA_STACK_SLOW;
+ st->size = size;
+ if (old.size)
+ arena_free_pages(arena, old.base, old.size >> PAGE_SHIFT, false);
+ return BIT_ULL(32) | st->base;
+
+err:
+ bpf_prog_inc_misses_counter(aux->prog);
+ bpf_stream_stage(ss, aux->prog, BPF_STDERR, ({
+ bpf_stream_printk(ss, "ERROR: No memory for the stack in arena. The program didn't run\n");
+ }));
+ return 0;
+}
+
+/*
+ * Called before the main program or a callback exits with a stack that is not
+ * the stack of the cpu: BPF_ARENA_STACK_SLOW is set in @base.
+ */
+void bpf_arena_stack_put(struct bpf_prog_aux *aux, u32 base)
+{
+ struct arena_stacks *stacks = this_cpu_ptr(aux->arena->stacks);
+ u32 size = aux->arena_stack_size;
+ int i;
+
+ for (i = 0; i < ARENA_STACK_SLOTS; i++) {
+ if (stacks->slot[i].base != base || !stacks->slot[i].size)
+ continue;
+ clear_bit(i, &stacks->busy);
+ return;
+ }
+ arena_free_pages(aux->arena, base, size >> PAGE_SHIFT, false);
+}
+
/*
* Reserve an arena virtual address range without populating it. This call stops
* bpf_arena_alloc_pages from adding pages to this range.
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 10/14] bpf, x86: Add BPF_ATOMIC_LOCAL
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (8 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 09/14] bpf: Add stacks for programs to arena Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 11/14] bpf: Support BPF_REG_ARENA_SP in the verifier Alexei Starovoitov
` (5 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog that has a stack in arena takes the per-cpu stack each time it
starts. It does this with fetch_add on a per-cpu word. Only this cpu
accesses the word. Thus the insn must be atomic only for an interrupt or
for a different task on the same cpu. But the fetch_add of BPF is atomic
for all cpus. On x86 it has the lock prefix, which waits for the stores in
the store buffer.
Add BPF_ATOMIC_LOCAL, a flag in imm of fetch_add. It means that the insn is
atomic only for code that runs on this cpu. One x86 insn is atomic for such
code. Thus the x86 JIT does not emit the lock prefix:
BPF_ADD | BPF_FETCH lock xadd
BPF_ADD | BPF_FETCH | BPF_ATOMIC_LOCAL xadd
The flag is for per-cpu memory only. It is correct there because progs do
not migrate.
Only the verifier emits the flag, and only when the JIT reports
bpf_jit_supports_local_atomics(). The other JITs get the usual fetch_add.
The first user is "bpf: Support BPF_REG_ARENA_SP in the verifier". The
verifier continues to reject the flag in the insns of a prog. The atomics
of progs get the same code.
The time that the flag saves depends on the stores that are in the store
buffer when the prog starts. On x86-64 (Zen 5), in nsec per run:
hot caches 0.44
after a store to 16 Mbyte of data (L3) 1.6
after a store that misses all caches 47
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 10 +++++++++-
include/linux/filter.h | 8 ++++++++
kernel/bpf/core.c | 6 ++++++
kernel/bpf/disasm.c | 7 +++++++
4 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 8ade87bd56e3..711d774b40e1 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1625,7 +1625,10 @@ static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,
{
u8 *prog = *pprog;
- if (atomic_op != BPF_XCHG)
+ /* one insn is atomic for what runs on this cpu */
+ if (atomic_op & BPF_ATOMIC_LOCAL)
+ atomic_op &= ~BPF_ATOMIC_LOCAL;
+ else if (atomic_op != BPF_XCHG)
EMIT1(0xF0); /* lock prefix */
maybe_emit_mod(&prog, dst_reg, src_reg, bpf_size == BPF_DW);
@@ -5187,6 +5190,11 @@ bool bpf_jit_supports_private_stack(void)
return true;
}
+bool bpf_jit_supports_local_atomics(void)
+{
+ return true;
+}
+
void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
{
#if defined(CONFIG_UNWINDER_ORC)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index abc759ad14a6..7b730b6488ec 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -81,6 +81,13 @@ struct ctl_table_header;
/* unused opcode to mark special atomic instruction */
#define BPF_PROBE_ATOMIC 0xe0
+/*
+ * Flag in imm of BPF_ATOMIC fetch_add: atomic for what runs on this cpu only,
+ * an interrupt or another task. The memory is per-cpu and the program
+ * doesn't migrate. It's cheaper where one insn without a lock does it.
+ */
+#define BPF_ATOMIC_LOCAL 0x1000
+
/* unused opcode to mark special ldsx instruction. Same as BPF_NOSPEC */
#define BPF_PROBE_MEM32SX 0xc0
@@ -1330,6 +1337,7 @@ bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
+bool bpf_jit_supports_local_atomics(void);
bool bpf_jit_supports_large_stack(void);
bool bpf_jit_supports_timed_may_goto(void);
bool bpf_jit_supports_fsession(void);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 06d6d45b1c0e..1466a85c2b1a 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3874,6 +3874,12 @@ bool __weak bpf_jit_supports_private_stack(void)
return false;
}
+/* Whether JIT supports BPF_ATOMIC_LOCAL */
+bool __weak bpf_jit_supports_local_atomics(void)
+{
+ return false;
+}
+
/*
* Return TRUE if the JIT lays out frames of up to MAX_BPF_STACK_JIT bytes.
* Its prologue, epilogue and tail call sequences must encode such frame
diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c
index 0b1bbb27e9e4..2e377a7ea262 100644
--- a/kernel/bpf/disasm.c
+++ b/kernel/bpf/disasm.c
@@ -9,6 +9,7 @@
/* Only defined by the non-UAPI linux/filter.h, which this file cannot use. */
#define BPF_PROBE_ATOMIC 0xe0
+#define BPF_ATOMIC_LOCAL 0x1000
#define __BPF_FUNC_STR_FN(x) [BPF_FUNC_ ## x] = __stringify(bpf_ ## x)
static const char * const func_id_str[] = {
@@ -257,6 +258,12 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->dst_reg,
insn->off, insn->src_reg);
+ else if (atomic && insn->imm == (BPF_ADD | BPF_FETCH | BPF_ATOMIC_LOCAL))
+ verbose(cbs->private_data, "(%02x) r%d = local%s_fetch_add((%s *)(r%d %+d), r%d)",
+ insn->code, insn->src_reg,
+ BPF_SIZE(insn->code) == BPF_DW ? "64" : "",
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->dst_reg, insn->off, insn->src_reg);
else if (atomic &&
(insn->imm == BPF_ADD || insn->imm == BPF_AND ||
insn->imm == BPF_OR || insn->imm == BPF_XOR)) {
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 11/14] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (9 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 10/14] bpf, x86: Add BPF_ATOMIC_LOCAL Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 2:12 ` bot+bpf-ci
2026-11-04 1:23 ` [PATCH bpf-next v3 12/14] bpf, x86: Support BPF_REG_ARENA_SP Alexei Starovoitov
` (4 subsequent siblings)
15 siblings, 1 reply; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
In Rust &local, &heap and &static are the same type of pointer, and the
same insn dereferences all of them. The heap and the statics of a Rust prog
are in arena. Thus a local variable whose address is taken must be in arena
too. Spills and all other local variables stay in the r10 stack.
Add a second stack, in arena. r12 (BPF_REG_ARENA_SP) is its stack pointer.
The stack grows up, and r12 points to the end of the frame of the current
function:
f:
r12 += 32 ; the frame of f, 32 bytes in arena
*(u64 *)(r12 - 8) = r1 ; a local variable
r1 = r12
r1 += -8 ; its address, a number (scalar)
call g ; g accesses the variable through r1
exit ; the verifier adds 'r12 += -32' here
The verifier tracks neither r12 nor the content of this stack. It does not
track the rest of arena either. r12 holds the low 32 bits of the address,
and the JIT adds the base of the arena. Thus an access through r12 stays in
arena. But the code at the exit of the prog needs r12 to find the stack
that it must give back. Thus accept r12 in 4 insns only:
r12 += N first insn of a function, N % 16 == 0
rX = r12 number (scalar): user space address
rX = *(size *)(r12 - off) off is not checked against N
*(size *)(r12 - off) = rX or imm
Only the compiler knows N, the size of the frame. The verifier cannot
calculate it, because callees access the frame through numbers. The frames
of a call chain have 32 Kbyte at most.
The reasons for these rules:
in this patch what was rejected, and why
----------------------------- ----------------------------------------
One 'r12 += N', the first 'r12 -= rX': the depth has no bound.
insn of a function. More than one 'r12 += imm': r12 becomes
verifier state.
The verifier tracks nothing Tracked slots: each store through a
in this stack. number invalidates them.
'rX = r12' gives a number. PTR_TO_ARENA: after a store to memory
and a load it is a number. Then one insn
sees two types.
The stack grows up, and the Down, with r12 + off: harder to read.
offsets are negative, as with
r10.
Each run takes a stack, see "bpf: Add stacks for programs to arena". The
verifier adds this code to the main prog. busy, size and base are per-cpu:
entry before each exit
old = busy++ if (r12 & BPF_ARENA_STACK_SLOW)
if (old == 0 && size is sufficient) bpf_arena_stack_put()
r12 = base else
else busy = 0
r12 = bpf_arena_stack_get()
no memory: return 0
The verifier adds this code before gen_prologue() and gen_epilogue() add
theirs. Thus the prologue of the prog type runs before the prog takes
a stack, and the epilogue runs after each exit, also after 'return 0'.
The usual path has 8 insns at the entry and 4 insns at the exit. The kernel
does not pass r12 to a callback. Thus a callback that uses r12 gets the
same code and takes a stack too. To reject r12 in callbacks is not
possible: real schedulers need it.
Progs without r12 get the same code. A prog that uses r12 in one of its
functions pays in each run: up to 22 more x86 insns, no call, no lock
prefix. The cost on x86-64 (Zen 5), in nsec more than the same prog with
r10:
each run, hot caches 0.28 to 2.30
each run, cold caches and TLB 45 to 70
each run that finds the per-cpu stack taken 8.7
iteration of bpf_loop(), the callback uses r12 17.5 (1.13 -> 18.6)
call of a function with a frame, load, store below 0.23
A callback of bpf_loop() always finds the per-cpu stack taken, because the
main prog has it. The verifier also does not inline such a loop.
Limitations:
- The JIT must report bpf_jit_supports_arena_stack(). No JIT does in this
patch.
- The verifier rejects tail calls, because the prog would not give the
stack back.
- It rejects extension progs. The code at the entry uses r2-r5, and an
extension prog has arguments there.
- It rejects BPF_LD_ABS and BPF_LD_IND. They end the function when the
offset is not in the packet, and the prog would not give the stack back.
- It rejects a jump and a gotox to 'r12 += N', because the insn would run
again.
- It rejects a struct_ops prog that has a __ref argument. A run without
a stack is skipped, and nothing would release the argument.
- It rejects a call of the main prog, and a call and a callx of a callback
that takes a stack, because the caller would get the r12 of the callee.
- Nothing can attach to a subprog, because the x86 trampoline does not
keep r12.
- The private stack is off, because the x86 JIT has one register for the
two.
- bpf_throw() does not give the stack back. The stack stays taken until
the arena is freed, for all progs of the arena on that cpu. After 17
throws on a cpu each run there allocates a stack, and each more throw
loses the pages of one stack. Rust progs unwind and do not throw.
may_goto sets is_called in struct bpf_subprog_info for a function that BPF
code calls, and for a callback of a helper other than bpf_loop() and
bpf_for_each_map_elem(). Such a callback can take a stack. A function that
BPF code calls cannot. Give these callbacks a bit of their own,
is_other_cb. may_goto examines the two bits.
The test "invalid src register in LDX" of test_verifier used r12. Use r13.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 2 +
include/linux/bpf_verifier.h | 14 +-
include/linux/filter.h | 59 +++++
kernel/bpf/backtrack.c | 2 +-
kernel/bpf/const_fold.c | 7 +
kernel/bpf/core.c | 25 +++
kernel/bpf/fixups.c | 202 +++++++++++++++++-
kernel/bpf/liveness.c | 8 +
kernel/bpf/scev.c | 9 +-
kernel/bpf/verifier.c | 194 ++++++++++++++++-
.../selftests/bpf/verifier/basic_stx_ldx.c | 4 +-
11 files changed, 516 insertions(+), 10 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5c059a05f89d..acc8e0050cde 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1949,6 +1949,8 @@ struct bpf_prog_aux {
* the page size. 0 when r12 is not used.
*/
u32 arena_stack_size;
+ /* a callback that takes a stack in arena, like the main program does */
+ bool arena_stack_entry;
void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 9ae6e109d211..f5a4fe8a58fe 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -944,6 +944,9 @@ enum priv_stack_mode {
PRIV_STACK_ADAPTIVE,
};
+/* The most that the frames of a call chain take in the stack in arena */
+#define BPF_ARENA_STACK_MAX (32 << 10)
+
struct bpf_subprog_info {
const char *name; /* name extracted from BTF */
u32 start; /* insn idx of function entry point */
@@ -957,6 +960,7 @@ struct bpf_subprog_info {
struct bpf_iarray *jt; /* jump table shared by all gotox of this subprogram */
u16 stack_depth; /* max. stack depth used by this function */
u16 stack_extra;
+ u16 arena_stack_depth; /* size of the frame in the stack in arena */
u32 insns_total;
u32 insns_self;
/* offsets in range [stack_depth .. fastcall_stack_off)
@@ -973,7 +977,9 @@ struct bpf_subprog_info {
bool is_exception_cb: 1;
/* bpf_loop() or bpf_for_each_map_elem() call it in a loop */
bool is_loop_cb: 1;
- /* BPF code, another helper or a kfunc call it */
+ /* another helper or a kfunc calls it */
+ bool is_other_cb: 1;
+ /* a call insn or callx calls it */
bool is_called: 1;
bool args_cached: 1;
/* true if the return value is passed in the R0:R2 register pair */
@@ -982,6 +988,9 @@ struct bpf_subprog_info {
bool keep_fastcall_stack: 1;
bool changes_pkt_data: 1;
bool might_sleep: 1;
+ bool uses_arena_sp: 1;
+ /* a callback that takes a stack in arena, like the main program does */
+ bool arena_stack_entry: 1;
u8 arg_slot_cnt:4;
enum priv_stack_mode priv_stack_mode;
@@ -1145,6 +1154,8 @@ struct bpf_verifier_env {
bool strict_alignment; /* perform strict pointer alignment checks */
bool test_state_freq; /* test verifier with different pruning frequency */
bool test_reg_invariants; /* fail verification on register invariants violations */
+ bool uses_arena_sp; /* the program has insns with BPF_REG_ARENA_SP */
+ u32 max_arena_stack_depth; /* of all call chains */
struct bpf_verifier_state *cur_state; /* current verifier state */
/* Search pruning optimization, array of list_heads for
* lists of struct bpf_verifier_state_list.
@@ -2092,6 +2103,7 @@ void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env);
int bpf_opt_remove_dead_code(struct bpf_verifier_env *env);
int bpf_opt_remove_nops(struct bpf_verifier_env *env);
void bpf_lower_cast_mov(struct bpf_verifier_env *env);
+int bpf_insert_arena_stack(struct bpf_verifier_env *env);
int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, const union bpf_attr *attr);
int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);
int bpf_jit_subprogs(struct bpf_verifier_env *env);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 7b730b6488ec..a58ad2318412 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -599,6 +599,21 @@ static inline int bpf_atomic_load_reg(const struct bpf_insn *insn)
.off = 0, \
.imm = BPF_CALL_IMM(FUNC) })
+/*
+ * The same in code that the verifier emits before bpf_do_misc_fixups(). That pass
+ * takes imm of a call for the id of a helper. This value in off, which is 0 in the
+ * call of a helper, says that imm is final. The pass clears it.
+ */
+#define BPF_KERNEL_CALL 1
+
+#define BPF_EMIT_KERNEL_CALL(FUNC) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_CALL, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = BPF_KERNEL_CALL, \
+ .imm = BPF_CALL_IMM(FUNC) })
+
/* Kfunc call */
#define BPF_CALL_KFUNC(OFF, IMM) \
@@ -901,6 +916,49 @@ static inline bool is_stack_arg_stx(const struct bpf_insn *insn)
insn->off < 0 && insn->off % 8 == 0;
}
+/*
+ * BPF_REG_ARENA_SP, r12, points to the end of the frame of the function in
+ * the stack in arena, like r10 does in the stack of the program. The stack
+ * in arena grows up. r12 is used in four insns:
+ * r12 += N the first insn of the function
+ * N is a multiple of 16, so is r12
+ * rX = r12
+ * rX = *(size *)(r12 - off)
+ * *(size *)(r12 - off) = rX or imm
+ */
+static inline bool is_arena_sp_adj(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_ALU64 | BPF_ADD | BPF_K) &&
+ insn->dst_reg == BPF_REG_ARENA_SP;
+}
+
+static inline bool is_arena_sp_mov(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) &&
+ insn->src_reg == BPF_REG_ARENA_SP && !insn->off && !insn->imm;
+}
+
+static inline bool is_arena_sp_ldx(const struct bpf_insn *insn)
+{
+ return BPF_CLASS(insn->code) == BPF_LDX &&
+ (BPF_MODE(insn->code) == BPF_MEM || BPF_MODE(insn->code) == BPF_MEMSX) &&
+ insn->src_reg == BPF_REG_ARENA_SP;
+}
+
+/* st and stx */
+static inline bool is_arena_sp_st(const struct bpf_insn *insn)
+{
+ return (BPF_CLASS(insn->code) == BPF_ST || BPF_CLASS(insn->code) == BPF_STX) &&
+ BPF_MODE(insn->code) == BPF_MEM &&
+ insn->dst_reg == BPF_REG_ARENA_SP;
+}
+
+static inline bool is_arena_sp_insn(const struct bpf_insn *insn)
+{
+ return is_arena_sp_adj(insn) || is_arena_sp_mov(insn) ||
+ is_arena_sp_ldx(insn) || is_arena_sp_st(insn);
+}
+
#define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN
struct bpf_skb_data_end {
@@ -1337,6 +1395,7 @@ bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
+bool bpf_jit_supports_arena_stack(void);
bool bpf_jit_supports_local_atomics(void);
bool bpf_jit_supports_large_stack(void);
bool bpf_jit_supports_timed_may_goto(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 2d6535f271a6..b9653965e525 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -325,7 +325,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* sreg needs precision before this insn
*/
bt_clear_reg(bt, dreg);
- if (sreg != BPF_REG_FP)
+ if (sreg != BPF_REG_FP && sreg != BPF_REG_ARENA_SP)
bt_set_reg(bt, sreg);
} else {
/* dreg = K
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index f9edfde965dc..0e0cad2b4b8e 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -65,6 +65,13 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
ci_out[insn->dst_reg] = unknown;
return;
}
+ /* r12 is outside of it too */
+ if (is_arena_sp_adj(insn) || is_arena_sp_st(insn))
+ return;
+ if (is_arena_sp_mov(insn) || is_arena_sp_ldx(insn)) {
+ ci_out[insn->dst_reg] = unknown;
+ return;
+ }
switch (class) {
case BPF_ALU:
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 1466a85c2b1a..8df24e6f7c1f 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3880,6 +3880,12 @@ bool __weak bpf_jit_supports_local_atomics(void)
return false;
}
+/* Whether JIT supports BPF_REG_ARENA_SP */
+bool __weak bpf_jit_supports_arena_stack(void)
+{
+ return false;
+}
+
/*
* Return TRUE if the JIT lays out frames of up to MAX_BPF_STACK_JIT bytes.
* Its prologue, epilogue and tail call sequences must encode such frame
@@ -3984,6 +3990,25 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena)
return 0;
}
+__weak int bpf_arena_stack_init(struct bpf_arena *arena)
+{
+ return -EOPNOTSUPP;
+}
+
+__weak void __percpu *bpf_arena_stack_percpu(struct bpf_arena *arena)
+{
+ return NULL;
+}
+
+__weak u64 bpf_arena_stack_get(struct bpf_prog_aux *aux, u64 busy)
+{
+ return 0;
+}
+
+__weak void bpf_arena_stack_put(struct bpf_prog_aux *aux, u32 base)
+{
+}
+
#ifdef CONFIG_BPF_SYSCALL
__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
unsigned long fault_ip)
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index eb5756cbc7e5..fe8899c65240 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -729,6 +729,195 @@ void bpf_lower_cast_mov(struct bpf_verifier_env *env)
}
}
+/*
+ * Make BPF_REG_ARENA_SP work. It holds the low 32 bits of the address, which is
+ * what JITs add to the base of arena.
+ * - the main program takes a stack when it starts and returns it when it exits:
+ * the stack of the cpu when it's free, without a call.
+ * See struct bpf_arena_cpu_stack. So does a callback.
+ * bpf_throw() doesn't return it: the stack stays taken until the arena is freed.
+ * - a function that has a frame drops it before exit.
+ * - 'rX = r12' gives the address like user space sees it, as other addresses
+ * in arena are.
+ * It's done before bpf_convert_ctx_accesses(), so that this code is a part of the
+ * program for gen_prologue() and gen_epilogue(). The prologue runs before the
+ * program takes a stack. The epilogue runs after each exit, also when there is
+ * no stack and the program is skipped.
+ */
+
+#define CPU_STACK_OFF(field) offsetof(struct bpf_arena_cpu_stack, field)
+
+/* reg = this_cpu_ptr(stacks) */
+static struct bpf_insn *emit_arena_stack_percpu(struct bpf_verifier_env *env,
+ struct bpf_insn *patch, int reg)
+{
+ struct bpf_insn ld_stacks[2] = {
+ BPF_LD_IMM64(reg, (__force long)bpf_arena_stack_percpu(env->prog->aux->arena))
+ };
+
+ *patch++ = ld_stacks[0];
+ *patch++ = ld_stacks[1];
+ *patch++ = BPF_MOV64_PERCPU_REG(reg, reg);
+ return patch;
+}
+
+/*
+ * Take a stack at the start of the main program or of a callback.
+ *
+ * cnt = busy++;
+ * if (!cnt)
+ * goto free;
+ * slow:
+ * r12 = bpf_arena_stack_get(aux, cnt);
+ * if (r12)
+ * goto done;
+ * return 0;
+ * free:
+ * if (size < what the program needs)
+ * goto slow;
+ * r12 = base;
+ * done:
+ *
+ * r0 and r6-r9 are not initialized when a function starts. Neither are r2-r5 in
+ * the main program. A callback has arguments there.
+ */
+static int insert_arena_stack_get(struct bpf_verifier_env *env, int s)
+{
+ struct bpf_insn ld_aux[2] = { BPF_LD_IMM64(BPF_REG_1, (long)env->prog->aux) };
+ int local = bpf_jit_supports_local_atomics() ? BPF_ATOMIC_LOCAL : 0;
+ int ptr = s ? BPF_REG_6 : BPF_REG_2, cnt = s ? BPF_REG_7 : BPF_REG_3;
+ int size = s ? BPF_REG_8 : BPF_REG_4;
+ struct bpf_subprog_info *sub = &env->subprog_info[s];
+ struct bpf_insn *buf = env->insn_buf, *patch;
+ int jfree, jdone, jslow, slow, free, done, i, spill = 0;
+
+ if (s) {
+ spill = round_up(sub->stack_depth, BPF_REG_SIZE);
+ sub->stack_depth = spill + MAX_BPF_FUNC_REG_ARGS * BPF_REG_SIZE;
+ if (sub->stack_depth > env->stack_limit) {
+ verbose(env, "no room in the stack of %s() for its arguments\n",
+ bpf_subprog_name(env, s));
+ return -EACCES;
+ }
+ }
+
+ patch = emit_arena_stack_percpu(env, buf, ptr);
+ *patch++ = BPF_MOV32_IMM(cnt, 1);
+ *patch++ = BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH | local, ptr, cnt,
+ CPU_STACK_OFF(busy));
+ jfree = patch++ - buf;
+ slow = patch - buf;
+ if (!s)
+ *patch++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
+ for (i = BPF_REG_1; s && i <= BPF_REG_5; i++)
+ *patch++ = BPF_STX_MEM(BPF_DW, BPF_REG_FP, i, -(spill + i * BPF_REG_SIZE));
+ *patch++ = ld_aux[0];
+ *patch++ = ld_aux[1];
+ *patch++ = BPF_MOV64_REG(BPF_REG_2, cnt);
+ *patch++ = BPF_EMIT_KERNEL_CALL(bpf_arena_stack_get);
+ *patch++ = BPF_MOV32_REG(BPF_REG_ARENA_SP, BPF_REG_0);
+ if (!s)
+ *patch++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6);
+ for (i = BPF_REG_1; s && i <= BPF_REG_5; i++)
+ *patch++ = BPF_LDX_MEM(BPF_DW, i, BPF_REG_FP, -(spill + i * BPF_REG_SIZE));
+ jdone = patch++ - buf;
+ *patch++ = BPF_MOV64_IMM(BPF_REG_0, 0);
+ *patch++ = BPF_EXIT_INSN();
+ free = patch - buf;
+ *patch++ = BPF_LDX_MEM(BPF_W, size, ptr, CPU_STACK_OFF(size));
+ jslow = patch++ - buf;
+ *patch++ = BPF_LDX_MEM(BPF_W, BPF_REG_ARENA_SP, ptr, CPU_STACK_OFF(base));
+ done = patch - buf;
+
+ buf[jfree] = BPF_JMP_IMM(BPF_JEQ, cnt, 0, free - jfree - 1);
+ buf[jdone] = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, done - jdone - 1);
+ /* size is 0 until the stack is allocated */
+ buf[jslow] = BPF_JMP32_IMM(BPF_JLT, size, max(env->max_arena_stack_depth, 1),
+ slow - jslow - 1);
+
+ /* a jump to the first insn doesn't take the stack again, neither does gotox */
+ return bpf_patch_list_insert(env, sub->start, buf, done);
+}
+
+int bpf_insert_arena_stack(struct bpf_verifier_env *env)
+{
+ struct bpf_map *arena = (struct bpf_map *)env->prog->aux->arena;
+ struct bpf_subprog_info *subprogs = env->subprog_info;
+ struct bpf_insn *insn_buf = env->insn_buf, *patch, *insn;
+ struct bpf_prog_aux *aux = env->prog->aux;
+ struct bpf_insn ld_aux[2] = { BPF_LD_IMM64(BPF_REG_1, (long)aux) };
+ int i, cur = 0, err;
+
+ if (!env->uses_arena_sp)
+ return 0;
+
+ /* the caller would get r12 of the callee back */
+ for (i = 0; i < env->subprog_cnt; i++) {
+ if ((i && !subprogs[i].arena_stack_entry) || !subprogs[i].is_called)
+ continue;
+ verbose(env, "%s() uses R%d and is both called and a callback\n",
+ bpf_subprog_name(env, i), BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+
+ /*
+ * With spare bytes r12 is neither the start nor the end of the stack,
+ * which may be at 0, so 'rX = r12' is not NULL.
+ */
+ aux->arena_stack_size = round_up(env->max_arena_stack_depth + 64, PAGE_SIZE);
+
+ for (i = 0; i < env->prog->len; i++) {
+ insn = env->prog->insnsi + i;
+ if (subprogs[cur + 1].start == i)
+ cur++;
+ patch = insn_buf;
+
+ if (is_arena_sp_adj(insn)) {
+ insn->code = BPF_ALU | BPF_ADD | BPF_K;
+ env->insn_aux_data[i].zext_dst = true;
+ } else if (is_arena_sp_mov(insn)) {
+ /* rX = addr_space_cast(r12, 1, 0) */
+ if (!(arena->map_flags & BPF_F_NO_USER_CONV)) {
+ insn->off = BPF_ADDR_SPACE_CAST;
+ insn->imm = 1 << 16;
+ }
+ } else if (insn->code == (BPF_JMP | BPF_EXIT)) {
+ if (subprogs[cur].arena_stack_depth)
+ *patch++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_ARENA_SP,
+ -(int)subprogs[cur].arena_stack_depth);
+ if (!cur || subprogs[cur].arena_stack_entry) {
+ /* r1-r5 are free */
+ *patch++ = BPF_JMP32_IMM(BPF_JSET, BPF_REG_ARENA_SP,
+ BPF_ARENA_STACK_SLOW, 5);
+ patch = emit_arena_stack_percpu(env, patch, BPF_REG_2);
+ *patch++ = BPF_ST_MEM(BPF_DW, BPF_REG_2, CPU_STACK_OFF(busy), 0);
+ *patch++ = *insn;
+ *patch++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_0);
+ *patch++ = ld_aux[0];
+ *patch++ = ld_aux[1];
+ *patch++ = BPF_MOV64_REG(BPF_REG_2, BPF_REG_ARENA_SP);
+ *patch++ = BPF_EMIT_KERNEL_CALL(bpf_arena_stack_put);
+ *patch++ = BPF_MOV64_REG(BPF_REG_0, BPF_REG_6);
+ }
+ if (patch != insn_buf)
+ *patch++ = *insn;
+ }
+
+ if (patch != insn_buf) {
+ insn = bpf_patch_list_add(env, i, insn_buf, patch - insn_buf);
+ if (IS_ERR(insn))
+ return PTR_ERR(insn);
+ }
+ /* Not before the insn is rewritten: the list takes a copy of it. */
+ if (subprogs[cur].start == i && (!cur || subprogs[cur].arena_stack_entry)) {
+ err = insert_arena_stack_get(env, cur);
+ if (err)
+ return err;
+ }
+ }
+ return bpf_patch_list_commit(env);
+}
+
int bpf_opt_remove_nops(struct bpf_verifier_env *env)
{
int insn_cnt = env->prog->len;
@@ -1423,6 +1612,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->jited_linfo = prog->aux->jited_linfo;
func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx;
func[i]->aux->arena = prog->aux->arena;
+ func[i]->aux->arena_stack_size = prog->aux->arena_stack_size;
+ func[i]->aux->arena_stack_entry = env->subprog_info[i].arena_stack_entry;
func[i]->aux->uses_arena_insns = bpf_prog_has_arena_insns(func[i]);
func[i]->aux->used_maps = env->used_maps;
func[i]->aux->used_map_cnt = env->used_map_cnt;
@@ -2201,6 +2392,11 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
goto next_insn;
if (insn->src_reg == BPF_PSEUDO_CALL)
goto next_insn;
+ /* imm is not the id of a helper */
+ if (!insn->src_reg && insn->off == BPF_KERNEL_CALL) {
+ insn->off = 0;
+ goto next_insn;
+ }
if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
ret = bpf_fixup_kfunc_call(env, insn, insn_buf, i, &cnt);
if (ret)
@@ -2703,7 +2899,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
bool main_prog = tmg && !i;
/* only a helper calls it, in a loop that ends when it returns 1 */
bool loop_cb = tmg && i && subprogs[i].is_loop_cb && !subprogs[i].is_called &&
- !subprogs[i].is_async_cb;
+ !subprogs[i].is_other_cb && !subprogs[i].is_async_cb;
if (timed_may_goto ? !main_prog && !loop_cb : !stack_slots)
continue;
@@ -2879,7 +3075,9 @@ int bpf_optimize_bpf_loop(struct bpf_verifier_env *env)
struct bpf_loop_inline_state *inline_state =
&env->insn_aux_data[i].loop_inline_state;
- if (is_bpf_loop_call(insn) && inline_state->fit_for_inline) {
+ /* the callback that takes a stack in arena is not for a call from the program */
+ if (is_bpf_loop_call(insn) && inline_state->fit_for_inline &&
+ !subprogs[inline_state->callback_subprogno].arena_stack_entry) {
stack_depth_extra = BPF_REG_SIZE * 3 + stack_depth_roundup;
err = inline_bpf_loop(env, i,
-(stack_depth + stack_depth_extra),
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 1a4cfb575077..f9e80e39c4f0 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1440,6 +1440,11 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
} else if (is_stack_arg_ldx(insn)) {
slot = stack_arg_off_to_slot(insn->off);
at_out[insn->dst_reg] = (slot >= 0) ? at_stack_arg_entry[slot] : none;
+ } else if (is_arena_sp_adj(insn) || is_arena_sp_st(insn)) {
+ /* at_out[12] is a stack arg slot, not r12 */
+ } else if (is_arena_sp_mov(insn) || is_arena_sp_ldx(insn)) {
+ /* an address in arena, or what is loaded from arena */
+ *dst = none;
} else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {
if (code == BPF_MOV) {
*dst = none;
@@ -1673,6 +1678,9 @@ static int record_load_store_access(struct bpf_verifier_env *env,
*/
if (is_stack_arg_stx(insn) || is_stack_arg_st(insn) || is_stack_arg_ldx(insn))
return 0;
+ /* Likewise at[12] is not r12, and the access is to arena. */
+ if (is_arena_sp_st(insn) || is_arena_sp_ldx(insn))
+ return 0;
switch (class) {
case BPF_LDX:
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 06dee7e421ea..093dd6fdcfe5 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -670,6 +670,12 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
int i, id;
s64 imm;
+ /* reg2expr[12] is a stack slot, not r12 */
+ if (is_arena_sp_adj(insn) || is_arena_sp_st(insn))
+ return 0;
+ if (is_arena_sp_mov(insn) || is_arena_sp_ldx(insn))
+ goto mark_dst_unknown;
+
switch (class) {
case BPF_ALU:
case BPF_ALU64:
@@ -1156,7 +1162,8 @@ static void collect_store_base_regs(struct bpf_verifier_env *env,
u16 base_regs = 0;
u32 r;
- if (size == BPF_W || size == BPF_DW) {
+ if ((size == BPF_W || size == BPF_DW) &&
+ !is_arena_sp_st(insn) && !is_arena_sp_ldx(insn)) {
if ((class == BPF_STX || class == BPF_ST) && insn->dst_reg != BPF_REG_FP)
base_regs |= BIT(insn->dst_reg);
else if (class == BPF_LDX && insn->src_reg != BPF_REG_FP)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 488ee53561be..d834575f2628 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -5792,6 +5792,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
struct bpf_insn *insn = env->prog->insnsi;
int depth = 0, frame = 0, i, subprog_end, subprog_depth;
bool tail_call_reachable = false;
+ int arena_depth = 0;
int total;
int tmp;
@@ -5832,6 +5833,24 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
* tailcall will unwind the current stack frame but it will not get rid
* of caller's stack as shown on the example above.
*/
+ /* The kernel calls a callback. It knows nothing about r12. */
+ if (subprog[idx].uses_arena_sp) {
+ for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
+ if (!subprog[tmp].is_cb)
+ continue;
+ subprog[tmp].arena_stack_entry = true;
+ break;
+ }
+ }
+ arena_depth += subprog[idx].arena_stack_depth;
+ if (arena_depth > env->max_arena_stack_depth)
+ env->max_arena_stack_depth = arena_depth;
+ if (arena_depth > BPF_ARENA_STACK_MAX) {
+ verbose(env, "combined size of the frames in the stack in arena is %d. Too large\n",
+ arena_depth);
+ return -EACCES;
+ }
+
if (idx && subprog[idx].has_tail_call && depth >= 256) {
verbose(env,
"tail_calls are not allowed when call stack of previous frames is %d bytes. Too large\n",
@@ -6020,6 +6039,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
return 0;
if (subprog[idx].priv_stack_mode != PRIV_STACK_ADAPTIVE)
depth -= round_up_stack_depth(env, subprog[idx].stack_depth);
+ arena_depth -= subprog[idx].arena_stack_depth;
/* pop caller idx from callee */
idx = dinfo[idx].caller;
@@ -6048,10 +6068,20 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
for (int i = 0; i < env->subprog_cnt; i++) {
if (si[i].has_tail_call) {
+ /* the stack in arena wouldn't be returned */
+ if (env->uses_arena_sp) {
+ verbose(env, "tail_calls are not allowed in programs that use R%d\n",
+ BPF_REG_ARENA_SP);
+ kvfree(dinfo);
+ return -EINVAL;
+ }
priv_stack_mode = NO_PRIV_STACK;
break;
}
}
+ /* x86 JIT has one register for both */
+ if (env->uses_arena_sp)
+ priv_stack_mode = NO_PRIV_STACK;
if (priv_stack_mode == PRIV_STACK_UNKNOWN)
priv_stack_mode = bpf_enable_priv_stack(env->prog);
@@ -11476,7 +11506,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
(insn->imm == BPF_FUNC_loop || insn->imm == BPF_FUNC_for_each_map_elem))
env->subprog_info[subprog].is_loop_cb = true;
else
- env->subprog_info[subprog].is_called = true;
+ env->subprog_info[subprog].is_other_cb = true;
/* for callback functions enqueue entry to callback and
* proceed with next instruction within current frame.
@@ -20366,6 +20396,11 @@ static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *in
if (env->gotox_tmp_buf->items[0] < subprog_start)
return reject_gotox_out_of_subprog(env, env->gotox_tmp_buf->items[0],
subprog_start, subprog_end);
+ /* it's the first insn of a function, it would be executed again */
+ if (is_arena_sp_adj(&env->prog->insnsi[env->gotox_tmp_buf->items[0]])) {
+ verbose(env, "jump to the insn that changes R%d\n", BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
if (env->gotox_tmp_buf->items[n - 1] >= subprog_end)
return reject_gotox_out_of_subprog(env, env->gotox_tmp_buf->items[n - 1],
subprog_start, subprog_end);
@@ -20382,12 +20417,64 @@ static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *in
return INSN_IDX_UPDATED;
}
+/*
+ * r12 is not in regs[]. Only the kernel and 'r12 += N' change it, so it points
+ * to the stack in arena. What is there is not tracked, like the rest of arena.
+ */
+static int check_arena_sp_insn(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ struct bpf_reg_state arena = { .type = PTR_TO_ARENA };
+ argno_t argno = argno_from_reg(BPF_REG_ARENA_SP);
+ struct bpf_reg_state *regs = cur_regs(env);
+ int size = BPF_SIZE(insn->code);
+ int err, value_regno = -1;
+
+ if (is_arena_sp_adj(insn))
+ return 0;
+
+ /* the address of the frame is a number */
+ if (is_arena_sp_mov(insn)) {
+ bpf_diag_mod_begin(env, ®s[insn->dst_reg], NULL, BPF_DIAG_MOD_WRITE);
+ err = check_reg_arg(env, insn->dst_reg, DST_OP);
+ if (!err)
+ bpf_diag_mod_end(env);
+ return err;
+ }
+
+ if (is_arena_sp_ldx(insn)) {
+ err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
+ if (err)
+ return err;
+ bpf_diag_mod_begin(env, ®s[insn->dst_reg], NULL, BPF_DIAG_MOD_WRITE);
+ err = check_mem_access(env, env->insn_idx, &arena, argno, insn->off, size,
+ BPF_READ, insn->dst_reg, false,
+ BPF_MODE(insn->code) == BPF_MEMSX);
+ err = err ?: save_aux_ptr_type(env, PTR_TO_ARENA, false);
+ if (!err)
+ bpf_diag_mod_end(env);
+ return err;
+ }
+
+ if (BPF_CLASS(insn->code) == BPF_STX) {
+ err = check_reg_arg(env, insn->src_reg, SRC_OP);
+ if (err)
+ return err;
+ value_regno = insn->src_reg;
+ }
+ err = check_mem_access(env, env->insn_idx, &arena, argno, insn->off, size,
+ BPF_WRITE, value_regno, false, false);
+ return err ?: save_aux_ptr_type(env, PTR_TO_ARENA, false);
+}
+
static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
{
int err;
struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
u8 class = BPF_CLASS(insn->code);
+ if (is_arena_sp_insn(insn))
+ return check_arena_sp_insn(env, insn);
+
switch (class) {
case BPF_ALU:
case BPF_ALU64:
@@ -21582,6 +21669,86 @@ static int check_insn_fields(struct bpf_verifier_env *env, struct bpf_insn *insn
}
}
+static bool jumps_to_arena_sp_adj(struct bpf_insn *insn)
+{
+ u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+ if (class != BPF_JMP && class != BPF_JMP32)
+ return false;
+ if (op == BPF_CALL || op == BPF_EXIT || insn->code == (BPF_JMP | BPF_JA | BPF_X))
+ return false;
+ return is_arena_sp_adj(insn + bpf_jmp_offset(insn) + 1);
+}
+
+/*
+ * Find the insns with BPF_REG_ARENA_SP. The sizes of the frames are in the text
+ * of the program.
+ */
+static int check_arena_stack(struct bpf_verifier_env *env)
+{
+ struct bpf_subprog_info *sub = env->subprog_info;
+ struct bpf_insn *insn = env->prog->insnsi;
+ int i;
+
+ for (i = 0; i < env->prog->len; i++, insn++) {
+ if (i == (sub + 1)->start)
+ sub++;
+ /* it would be executed again */
+ if (jumps_to_arena_sp_adj(insn)) {
+ verbose(env, "jump to the insn that changes R%d\n", BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ if (!is_arena_sp_insn(insn))
+ continue;
+ sub->uses_arena_sp = true;
+ env->uses_arena_sp = true;
+ if (!is_arena_sp_adj(insn))
+ continue;
+ if (i != sub->start) {
+ verbose(env, "R%d can be changed by the first insn of a function only\n",
+ BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ if (insn->imm <= 0 || insn->imm % 16 || insn->imm > BPF_ARENA_STACK_MAX) {
+ verbose(env, "invalid size of the frame in the stack in arena %d\n",
+ insn->imm);
+ return -EINVAL;
+ }
+ sub->arena_stack_depth = insn->imm;
+ }
+ if (!env->uses_arena_sp)
+ return 0;
+
+ if (!env->prog->aux->arena) {
+ verbose(env, "R%d can only be used in a program that has an associated arena\n",
+ BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ if (!bpf_jit_supports_arena_stack() || !bpf_jit_supports_percpu_insn() ||
+ bpf_prog_is_offloaded(env->prog->aux)) {
+ verbose(env, "JIT doesn't support R%d\n", BPF_REG_ARENA_SP);
+ return -EOPNOTSUPP;
+ }
+ /* bpf_insert_arena_stack() relies on r2-r5 being free when the program starts */
+ if (env->prog->type == BPF_PROG_TYPE_EXT) {
+ verbose(env, "R%d is not supported in extension programs\n", BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ /*
+ * It ends the function when the offset is out of the packet. bpf_do_misc_fixups()
+ * emits that exit, after bpf_insert_arena_stack(): the frame wouldn't be dropped
+ * and the stack wouldn't be returned.
+ */
+ for (i = 0; i < env->subprog_cnt; i++) {
+ if (!env->subprog_info[i].has_ld_abs)
+ continue;
+ verbose(env, "BPF_LD_[ABS|IND] is not allowed in programs that use R%d\n",
+ BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ return bpf_arena_stack_init(env->prog->aux->arena);
+}
+
/*
* Check that insns are sane and rewrite pseudo imm in ld_imm64 instructions:
*
@@ -21602,11 +21769,13 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
for (i = 0; i < insn_cnt; i++, insn++) {
if (insn->dst_reg >= MAX_BPF_REG &&
- !is_stack_arg_st(insn) && !is_stack_arg_stx(insn)) {
+ !is_stack_arg_st(insn) && !is_stack_arg_stx(insn) &&
+ !is_arena_sp_adj(insn) && !is_arena_sp_st(insn)) {
verbose(env, "R%d is invalid\n", insn->dst_reg);
return -EINVAL;
}
- if (insn->src_reg >= MAX_BPF_REG && !is_stack_arg_ldx(insn)) {
+ if (insn->src_reg >= MAX_BPF_REG && !is_stack_arg_ldx(insn) &&
+ !is_arena_sp_mov(insn) && !is_arena_sp_ldx(insn)) {
verbose(env, "R%d is invalid\n", insn->src_reg);
return -EINVAL;
}
@@ -22355,6 +22524,12 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
}
}
+ /* A run that gets no stack in arena is skipped. Nobody would release the argument. */
+ if (has_refcounted_arg && env->uses_arena_sp) {
+ verbose(env, "program with __ref argument cannot use R%d\n", BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+
prog->aux->st_ops = st_ops;
prog->aux->attach_st_ops_member_off = member_off;
@@ -22770,6 +22945,12 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
bpf_log(log, "Cannot attach to a target with arena context arguments\n");
return -EOPNOTSUPP;
}
+ /* x86: the trampoline and the extension don't preserve BPF_REG_ARENA_SP */
+ if (subprog && aux->arena_stack_size) {
+ bpf_log(log, "Cannot attach to a subprog of a program that uses R%d\n",
+ BPF_REG_ARENA_SP);
+ return -EOPNOTSUPP;
+ }
if (aux->func && aux->func[subprog]->aux->exception_cb) {
bpf_log(log,
"%s programs cannot attach to exception callback\n",
@@ -24162,6 +24343,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ ret = check_arena_stack(env);
+ if (ret < 0)
+ goto skip_full_check;
+
/* Find pointers to functions in the read-only maps of the program. */
ret = resolve_func_ptrs(env);
if (ret < 0)
@@ -24291,6 +24476,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
sanitize_dead_code(env);
}
+ if (ret == 0)
+ ret = bpf_insert_arena_stack(env);
+
if (ret == 0)
/* program is valid, convert *(u32*)(ctx + off) accesses */
ret = bpf_convert_ctx_accesses(env);
diff --git a/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c b/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c
index 7a0aab3f2cd2..66fde558df33 100644
--- a/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c
+++ b/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c
@@ -28,10 +28,10 @@
{
"invalid src register in LDX",
.insns = {
- BPF_LDX_MEM(BPF_B, BPF_REG_0, 12, 0),
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, 13, 0),
BPF_EXIT_INSN(),
},
- .errstr = "R12 is invalid",
+ .errstr = "R13 is invalid",
.result = REJECT,
},
{
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 11/14] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-11-04 1:23 ` [PATCH bpf-next v3 11/14] bpf: Support BPF_REG_ARENA_SP in the verifier Alexei Starovoitov
@ 2026-11-04 2:12 ` bot+bpf-ci
0 siblings, 0 replies; 32+ messages in thread
From: bot+bpf-ci @ 2026-11-04 2:12 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]
> bpf: Support BPF_REG_ARENA_SP in the verifier
TL;DR: Nit only: the new arena stack block in check_max_stack_depth_subprog()
separates the tail call comment from the check it documents; consider moving
the block above the comment or after the check.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -5832,6 +5833,24 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
> * tailcall will unwind the current stack frame but it will not get rid
> * of caller's stack as shown on the example above.
> */
> + /* The kernel calls a callback. It knows nothing about r12. */
> + if (subprog[idx].uses_arena_sp) {
> + for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
> + if (!subprog[tmp].is_cb)
> + continue;
> + subprog[tmp].arena_stack_entry = true;
> + break;
> + }
> + }
> + arena_depth += subprog[idx].arena_stack_depth;
> + if (arena_depth > env->max_arena_stack_depth)
> + env->max_arena_stack_depth = arena_depth;
> + if (arena_depth > BPF_ARENA_STACK_MAX) {
> + verbose(env, "combined size of the frames in the stack in arena is %d. Too large\n",
> + arena_depth);
> + return -EACCES;
> + }
> +
> if (idx && subprog[idx].has_tail_call && depth >= 256) {
This isn't a bug, but the new arena stack block now sits between the
existing comment about limiting the caller's stack depth to 256 bytes when
bpf2bpf calls are combined with tail calls, and the check it documents:
if (idx && subprog[idx].has_tail_call && depth >= 256) {
The tail call comment now ends directly above "if (subprog[idx].uses_arena_sp)".
Could the new block be moved above that comment, or after the tail call
check, so the comment stays next to the code it describes?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/41755299448
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH bpf-next v3 12/14] bpf, x86: Support BPF_REG_ARENA_SP
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (10 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 11/14] bpf: Support BPF_REG_ARENA_SP in the verifier Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:23 ` [PATCH bpf-next v3 13/14] bpf, arm64: " Alexei Starovoitov
` (3 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
BPF r12 (BPF_REG_ARENA_SP) is the stack pointer of the stack in arena. It
needs an x86 register for the full run of the prog. R9 is the only x86
register that has no fixed function. Map r12 to R9.
R9 has other uses too. In a prog that uses r12 the JIT changes them:
R9 is also in a prog with r12
-------------------------------- -------------------------------------
clobbered by helpers and kfuncs store R9 in the frame before the call,
load it after the call
arg 6 of a BPF-to-BPF call arg 6 is on the stack, before arg 7
arg 6 of a kfunc load arg 6 from the stack into R9,
move rsp to arg 7 for the call. Only
for a kfunc with more than 5 args
frame pointer of private stack the private stack is off
To reject 6 arguments in such progs is not possible, because Rust
functions have them. r12 in a slot of the frame costs a load for each
access.
Only JITed code sees the arguments of a BPF-to-BPF call, because no
trampoline attaches to a subprog of a prog with r12. Thus the JIT can
select the location of arg 6.
Progs without r12 get the same JITed code. The cost in a prog with r12, in
a hot loop on x86-64 (Zen 5), in nsec:
call of a helper or a kfunc store and load of R9 0.012
function that has such a call push r9, add rsp, 8 < 0.01
BPF-to-BPF call with 6 or 7 args arg 6 is on the stack < 0.007
call of a kfunc with 10 args load of R9, moves of rsp 0.13 to 0.27
The last time depends on the address of the kfunc.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 140 +++++++++++++++++++++++++++++++-----
1 file changed, 124 insertions(+), 16 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 711d774b40e1..320cf9b526b3 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -155,8 +155,9 @@ static int bpf_size_to_x86_bytes(int bpf_size)
* register in load/store instructions, it always needs an
* extra byte of encoding and is callee saved.
*
- * x86-64 register R9 is not used by BPF programs, but can be used by BPF
- * trampoline. x86-64 register R10 is used for blinding (if enabled).
+ * x86-64 register R9 is the 6th argument, the frame pointer of the private
+ * stack, or BPF_REG_ARENA_SP. It can be used by BPF trampoline.
+ * x86-64 register R10 is used for blinding (if enabled).
*/
static const int reg2hex[] = {
[BPF_REG_0] = 0, /* RAX */
@@ -170,6 +171,7 @@ static const int reg2hex[] = {
[BPF_REG_8] = 6, /* R14 callee saved */
[BPF_REG_9] = 7, /* R15 callee saved */
[BPF_REG_FP] = 5, /* RBP readonly */
+ [BPF_REG_ARENA_SP] = 1, /* R9 */
[BPF_REG_AX] = 2, /* R10 temp register */
[AUX_REG] = 3, /* R11 temp register */
[X86_REG_R9] = 1, /* R9 register, 6th function argument */
@@ -187,6 +189,7 @@ static const int reg2pt_regs[] = {
[BPF_REG_7] = offsetof(struct pt_regs, r13),
[BPF_REG_8] = offsetof(struct pt_regs, r14),
[BPF_REG_9] = offsetof(struct pt_regs, r15),
+ [BPF_REG_ARENA_SP] = offsetof(struct pt_regs, r9),
[BPF_REG_AX] = offsetof(struct pt_regs, r10),
};
@@ -211,6 +214,7 @@ static bool is_ereg(u32 reg)
BIT(BPF_REG_9) |
BIT(X86_REG_R9) |
BIT(X86_REG_R12) |
+ BIT(BPF_REG_ARENA_SP) |
BIT(BPF_REG_AX));
}
@@ -2326,6 +2330,37 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
return 0;
}
+/*
+ * Helpers and kfuncs clobber R9, which is BPF_REG_ARENA_SP. It's set after
+ * bpf_arena_stack_get() and is not used after bpf_arena_stack_put().
+ */
+static bool call_clobbers_arena_sp(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg != BPF_PSEUDO_CALL &&
+ insn->imm != BPF_CALL_IMM(bpf_arena_stack_get) &&
+ insn->imm != BPF_CALL_IMM(bpf_arena_stack_put);
+}
+
+/* Does the kfunc take more arguments than R1-R5 hold? */
+static bool kfunc_has_stack_args(const struct btf_func_model *fm)
+{
+ u32 i, nslots = 0;
+
+ for (i = 0; i < fm->nr_args; i++)
+ nslots += btf_func_model_arg_slots(fm, i);
+ return nslots > MAX_BPF_FUNC_REG_ARGS;
+}
+
+static bool calls_clobber_arena_sp(const struct bpf_prog *prog)
+{
+ int i;
+
+ for (i = 0; i < prog->len; i++)
+ if (call_clobbers_arena_sp(&prog->insnsi[i]))
+ return true;
+ return false;
+}
+
static const struct bpf_jit_arg_abi x86_arg_abi = {
.nr_arg_regs = 6,
.backfill_after_stack = true,
@@ -2533,16 +2568,23 @@ static u64 bpf_unwind_target(u64 ret_addr)
return e ? ret_addr + e->fixup : ret_addr;
}
-/* unwind_exit: fix the return address of this frame. r0 and r2 are returned. */
-static int emit_unwind_ret_addr(u8 **pprog, u8 *ip)
+/*
+ * unwind_exit: fix the return address of this frame. r0 and r2 are returned.
+ * So is BPF_REG_ARENA_SP, when it's used.
+ */
+static int emit_unwind_ret_addr(u8 **pprog, u8 *ip, bool arena_sp)
{
u8 *prog = *pprog, *start = prog;
EMIT2(0x50, 0x56); /* push rax; push rsi */
+ if (arena_sp)
+ push_r9(&prog);
EMIT4(0x48, 0x8B, 0x7D, 0x08); /* mov rdi, [rbp + 8] */
if (emit_rsb_call(&prog, bpf_unwind_target, ip + (prog - start)))
return -EINVAL;
EMIT4(0x48, 0x89, 0x45, 0x08); /* mov [rbp + 8], rax */
+ if (arena_sp)
+ pop_r9(&prog);
EMIT2(0x5E, 0x58); /* pop rsi; pop rax */
*pprog = prog;
return 0;
@@ -2567,6 +2609,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
u32 stack_depth;
int callee_saved_size;
s32 outgoing_arg_base;
+ /* the program uses BPF_REG_ARENA_SP, which is R9 */
+ bool arena_sp = bpf_prog->aux->arena_stack_size;
+ /* it has to be saved around calls of helpers and kfuncs */
+ bool arena_sp_save = arena_sp && calls_clobber_arena_sp(bpf_prog);
+ s32 arena_sp_off = 0;
int err;
stack_depth = bpf_prog->aux->stack_depth;
@@ -2591,12 +2638,21 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* is disabled by the JIT when stack args exist, so R9 is always
* available.
*
+ * Unless the program uses BPF_REG_ARENA_SP. Then for BPF-to-BPF calls
+ * arg 6 is on the stack too: incoming args 6+ are at [rbp + 16], ...
+ * and outgoing args 6+ are at [rsp], ... No trampoline is attached to
+ * subprogs of such program. For a kfunc call arg 6 is moved to R9 and
+ * rsp is moved to arg 7.
+ * Helpers and kfuncs clobber R9. BPF_REG_ARENA_SP is stored in the frame
+ * before the call and is loaded from there after it.
+ *
* Stack layout (high to low):
* [rbp + 16 + ...] incoming stack args 7+ (from caller)
* [rbp + 8] return address
* [rbp] saved rbp
* [rbp - prog_stack] program stack
* [below] callee-saved regs
+ * [below] BPF_REG_ARENA_SP, when it's used and there are calls
* [below] outgoing args 7+ (= rsp)
*/
if (bpf_prog_uses_arena_insns(bpf_prog)) {
@@ -2655,19 +2711,29 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* stack args exist, so tcc pushes need not be accounted for.
*/
outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
+ if (arena_sp_save) {
+ /* make the slot */
+ push_r9(&prog);
+ outgoing_arg_base -= 8;
+ arena_sp_off = outgoing_arg_base;
+ }
/*
* Allocate outgoing stack arg area for args 7+ only.
* Arg 6 goes into r9 register, not on stack.
*/
outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+ if (arena_sp)
+ outgoing_rsp = out_stack_arg_cnt * 8;
/*
* bpf_throw() sees rsp of this frame as it is at a call. The callee-saved
* regs are above the outgoing args, or above r9, which is pushed around
- * every call when the stack is private.
+ * every call when the stack is private. With BPF_REG_ARENA_SP the slot
+ * that keeps r9 over calls is between the two.
*/
if (bpf_prog->aux->exception_boundary)
- bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp + (priv_frame_ptr ? 8 : 0);
+ bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp +
+ (priv_frame_ptr || arena_sp_save ? 8 : 0);
emit_sub_rsp(&prog, outgoing_rsp);
if (arena_vm_start)
@@ -3089,7 +3155,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_ST | BPF_MEM | BPF_H:
case BPF_ST | BPF_MEM | BPF_W:
case BPF_ST | BPF_MEM | BPF_DW:
- if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
+ if (dst_reg == BPF_REG_PARAMS && insn->off == -8 && !arena_sp) {
/* Arg 6: store immediate in r9 register */
emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31,
imm32);
@@ -3103,7 +3169,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ...
*/
insn_off = outgoing_arg_base - outgoing_rsp -
- insn_off - 16;
+ insn_off - (arena_sp ? 8 : 16);
dst_reg = BPF_REG_FP;
}
if (!accesses_stack_only) {
@@ -3123,14 +3189,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_STX | BPF_MEM | BPF_H:
case BPF_STX | BPF_MEM | BPF_W:
case BPF_STX | BPF_MEM | BPF_DW:
- if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
+ if (dst_reg == BPF_REG_PARAMS && insn->off == -8 && !arena_sp) {
/* Arg 6: store register value in r9 */
EMIT_mov(X86_REG_R9, src_reg);
break;
}
insn_off = insn->off;
if (dst_reg == BPF_REG_PARAMS) {
- insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
+ insn_off = outgoing_arg_base - outgoing_rsp - insn_off -
+ (arena_sp ? 8 : 16);
dst_reg = BPF_REG_FP;
}
if (!accesses_stack_only) {
@@ -3239,11 +3306,13 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
insn_off = insn->off;
if (src_reg == BPF_REG_PARAMS) {
- if (insn_off == 8) {
+ if (insn_off == 8 && !arena_sp) {
/* Incoming arg 6: read from r9 */
EMIT_mov(dst_reg, X86_REG_R9);
break;
}
+ if (arena_sp)
+ insn_off += 8;
src_reg = BPF_REG_FP;
/*
* Incoming args 7+: native_off == bpf_off
@@ -3471,6 +3540,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
/* call */
case BPF_JMP | BPF_CALL: {
const struct btf_func_model *fm = NULL;
+ bool arena_sp_args = false;
func = (u8 *) __bpf_call_base + imm32;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
@@ -3481,6 +3551,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
if (!imm32)
return -EINVAL;
+ if (arena_sp && call_clobbers_arena_sp(insn)) {
+ u8 *start = prog;
+
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_ARENA_SP, arena_sp_off);
+ ip += prog - start;
+ }
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
if (!fm)
@@ -3489,8 +3565,31 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
if (err < 0)
return err;
ip += err;
- ip += emit_kfunc_arg_moves(fm, outgoing_arg_base -
- outgoing_rsp, &prog);
+ /*
+ * Other kfuncs leave rsp alone. bpf_throw() is one of
+ * them, see stack_arg_sp_adjust.
+ */
+ arena_sp_args = arena_sp && kfunc_has_stack_args(fm);
+ if (arena_sp_args) {
+ u8 *start = prog;
+
+ /*
+ * Args 6+ are at [rsp]. Put arg 6 where
+ * emit_kfunc_arg_moves() expects it.
+ */
+ emit_ldx(&prog, BPF_DW, X86_REG_R9, BPF_REG_FP,
+ outgoing_arg_base - outgoing_rsp);
+ ip += prog - start;
+ }
+ ip += emit_kfunc_arg_moves(fm, outgoing_arg_base - outgoing_rsp +
+ (arena_sp ? 8 : 0), &prog);
+ if (arena_sp_args) {
+ u8 *start = prog;
+
+ /* arg 7 is at [rsp] */
+ emit_add_rsp(&prog, 8);
+ ip += prog - start;
+ }
}
if (priv_frame_ptr) {
push_r9(&prog);
@@ -3501,6 +3600,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
return -EINVAL;
if (priv_frame_ptr)
pop_r9(&prog);
+ if (arena_sp_args)
+ emit_sub_rsp(&prog, 8);
+ if (arena_sp && call_clobbers_arena_sp(insn))
+ emit_ldx(&prog, BPF_DW, BPF_REG_ARENA_SP, BPF_REG_FP, arena_sp_off);
/*
* A kfunc returning more than 8 bytes hands the second
* half back in RDX (the native ABI's second return reg),
@@ -3827,7 +3930,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_JMP | BPF_EXIT:
if (src_reg == BPF_UNWIND_EXIT && bpf_prog_can_unwind(bpf_prog) &&
- emit_unwind_ret_addr(&prog, ip))
+ emit_unwind_ret_addr(&prog, ip, arena_sp))
return -EINVAL;
if (seen_exit) {
jmp_offset = ctx->cleanup_addr - addrs[i] + moved;
@@ -3841,8 +3944,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
return -EINVAL;
}
- /* Deallocate outgoing args 7+ area. */
- emit_add_rsp(&prog, outgoing_rsp);
+ /* Deallocate outgoing args 7+ area and the slot of BPF_REG_ARENA_SP. */
+ emit_add_rsp(&prog, outgoing_rsp + (arena_sp_save ? 8 : 0));
if (bpf_prog->aux->exception_boundary) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
@@ -5190,6 +5293,11 @@ bool bpf_jit_supports_private_stack(void)
return true;
}
+bool bpf_jit_supports_arena_stack(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_local_atomics(void)
{
return true;
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 13/14] bpf, arm64: Support BPF_REG_ARENA_SP
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (11 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 12/14] bpf, x86: Support BPF_REG_ARENA_SP Alexei Starovoitov
@ 2026-11-04 1:23 ` Alexei Starovoitov
2026-11-04 1:24 ` [PATCH bpf-next v3 14/14] selftests/bpf: Add tests for BPF_REG_ARENA_SP Alexei Starovoitov
` (2 subsequent siblings)
15 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:23 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
r12 (BPF_REG_ARENA_SP) is the stack pointer of the second stack of a prog.
This stack is in arena. The verifier replaces the insns that use r12 with
insns that the JIT already knows: a 32-bit add, addr_space_cast, and loads
and stores in arena. Thus the JIT only has to supply a register.
Map r12 to x23. The JIT does not use x23 for other data. x23 is callee
saved, thus helpers and kfuncs keep its value. It is not an argument
register, thus calls do not change.
The kernel calls the main prog and callbacks, and it does not know about
r12. The main prog, and a callback that uses r12, take a stack at entry and
set r12. Thus they save x23 at entry and restore it at exit. Other subprogs
get x23 from the caller.
Progs without r12 get the same code. The cost in progs with r12:
save and restore of x23 0 or 2 insns (stp, ldp). The JIT saves
registers in pairs.
load or store through r12 3 insns: add of the base of arena, mov of
the negative offset, ldr or str.
load or store through r10 1 insn, for reference.
Tested in QEMU with the selftests of this series and bpf_testmod: with LSE
atomics and without, at EL1 and at EL2 (VHE), with 2 and 4 cpus, with
bpf_jit_harden 0 and 2, with pages of 4 Kbyte and of 64 Kbyte, with KASAN
and lockdep.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/arm64/net/bpf_jit_comp.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index e99f105b7f61..e6d6df96fa99 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -61,6 +61,8 @@ static const int bpf2a64[] = {
[BPF_REG_9] = A64_R(22),
/* read-only frame pointer to access stack */
[BPF_REG_FP] = A64_R(25),
+ /* callee saved register for the pointer to the stack in arena */
+ [BPF_REG_ARENA_SP] = A64_R(23),
/* temporary registers for BPF JIT */
[TMP_REG_1] = A64_R(10),
[TMP_REG_2] = A64_R(11),
@@ -430,6 +432,11 @@ static void find_used_callee_regs(struct jit_ctx *ctx)
if (ctx->arena_vm_start)
ctx->used_callee_reg[i++] = bpf2a64[ARENA_VM_START];
+ /* subprogs get it from the caller and leave it as it was, unless the kernel calls them */
+ if (prog->aux->arena_stack_size &&
+ (!bpf_is_subprog(prog) || prog->aux->arena_stack_entry))
+ ctx->used_callee_reg[i++] = bpf2a64[BPF_REG_ARENA_SP];
+
ctx->nr_used_callee_reg = i;
}
@@ -2548,6 +2555,11 @@ bool bpf_jit_supports_private_stack(void)
return true;
}
+bool bpf_jit_supports_arena_stack(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_kfunc_call(void)
{
return true;
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH bpf-next v3 14/14] selftests/bpf: Add tests for BPF_REG_ARENA_SP
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (12 preceding siblings ...)
2026-11-04 1:23 ` [PATCH bpf-next v3 13/14] bpf, arm64: " Alexei Starovoitov
@ 2026-11-04 1:24 ` Alexei Starovoitov
2026-11-04 2:12 ` bot+bpf-ci
2026-11-04 3:31 ` [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
2026-11-04 20:10 ` patchwork-bot+netdevbpf
15 siblings, 1 reply; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 1:24 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
r12 (BPF_REG_ARENA_SP) is the stack pointer of the second stack of a prog.
This stack is in arena. Each run of a prog takes a stack at entry and gives
it back at exit. Runs that preempt or interrupt each other on one cpu must
not get the same stack.
arena_stack runs progs:
subtest what it does
------------------ ---------------------------------------------------
preemption 8 threads on one cpu run a prog that fills its
frames, spins, and then examines the frames.
nesting The same with preemption disabled. A prog in the
timer interrupt writes to its own frames.
timer A timer callback takes a stack of its own.
throw bpf_throw() does not give the stack back. After many
throws other progs continue to run correctly.
no_memory The arena has no free page. The prog does not run,
returns 0, and its stderr has a message.
attach_to_subprog fentry to the main prog loads. fentry to a subprog
is rejected.
jump_to_first_insn A jump and a gotox to the first insn do not take
a stack again.
slot_grows A run needs a larger stack than the slot has. The
next run on the cpu does not get the same stack.
Skipped when the stack does not grow (64 Kbyte
pages) or the compiler inlined arena_free_pages().
throw_stack_args The main prog has stack arguments and calls
bpf_throw(). The kernel gets its registers back.
The first two subtests fail if all runs on a cpu get the same stack. For
the first one the kernel must preempt a thread in the middle of the prog,
as it does with CONFIG_PREEMPT=y (config.x86_64). With lazy preemption and
without HRTICK, the defaults of arm64, it switches the threads after the
prog exits.
arena_stack is serial, as the other tests in prog_tests/arena_*.c. See
commit a14f9ba080c1 ("selftests/bpf: Run arena tests serially").
verifier_arena_stack has the tests of the verifier:
- Progs with the four insns that can use r12 load and return correct
values.
- All other insns with r12 are rejected. So are a frame of an incorrect
size, call chains with more than 32 Kbyte of frames, tail calls,
BPF_LD_ABS, a gotox to 'r12 += N', and a call and a callx of a callback
that takes a stack.
- A callback that starts with a call, or with the address of a function,
keeps the target when the verifier adds code in front of this insn.
- The prologue of a prog that writes to the packet is in front of the code
that takes a stack.
- r12 and the frame stay correct after a helper, a subprog with or without
a frame, a callback and unwind_exit of a callee.
- On x86 r12 is in the register of the 6th argument. Thus a subprog with 7
arguments and a kfunc with 10 arguments are tested too.
- may_goto: r12 stays correct after arch_bpf_timed_may_goto(). When the
time is up a callback of bpf_loop() returns before it takes a stack.
- The callback of bpf_user_ringbuf_drain() can use r12.
pro_epilogue_arena_stack uses the struct_ops of bpf_testmod, which has
gen_prologue() and gen_epilogue(). The two run also when the arena has no
page for a stack and the prog is skipped.
struct_ops_refcounted_fail__arena_stack has a __ref argument and uses r12.
It is rejected.
arena_noarena gets a prog that uses r12 and has no arena. It is rejected.
Compilers do not emit r12. Thus the tests encode these insns with .8byte.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/arena_stack.c | 323 +++++
.../selftests/bpf/prog_tests/pro_epilogue.c | 2 +
.../prog_tests/test_struct_ops_refcounted.c | 2 +
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/arena_noarena.c | 14 +
.../testing/selftests/bpf/progs/arena_stack.c | 334 +++++
.../selftests/bpf/progs/arena_stack_fentry.c | 13 +
.../selftests/bpf/progs/arena_stack_grow.c | 151 +++
.../selftests/bpf/progs/arena_stack_loop.c | 77 ++
.../selftests/bpf/progs/arena_stack_nomem.c | 41 +
.../selftests/bpf/progs/arena_stack_throw.c | 110 ++
.../bpf/progs/pro_epilogue_arena_stack.c | 199 +++
.../struct_ops_refcounted_fail__arena_stack.c | 61 +
.../bpf/progs/verifier_arena_stack.c | 1169 +++++++++++++++++
14 files changed, 2498 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_stack.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_fentry.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_grow.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_loop.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_nomem.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_throw.c
create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_arena_stack.c
create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_refcounted_fail__arena_stack.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_stack.c
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_stack.c b/tools/testing/selftests/bpf/prog_tests/arena_stack.c
new file mode 100644
index 000000000000..20950a8df3d8
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/arena_stack.c
@@ -0,0 +1,323 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define _GNU_SOURCE
+#include <test_progs.h>
+#include <pthread.h>
+#include <sched.h>
+#include <linux/perf_event.h>
+#include <sys/syscall.h>
+#include "arena_stack.skel.h"
+#include "arena_stack_nomem.skel.h"
+#include "arena_stack_fentry.skel.h"
+#include "arena_stack_loop.skel.h"
+#include "arena_stack_grow.skel.h"
+#include "arena_stack_throw.skel.h"
+
+static int run_prog(int fd)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+
+ if (bpf_prog_test_run_opts(fd, &opts))
+ return -1;
+ return opts.retval;
+}
+
+static int run_tc_prog(int fd)
+{
+ char pkt[64] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts, .data_in = pkt, .data_size_in = sizeof(pkt));
+
+ if (bpf_prog_test_run_opts(fd, &opts))
+ return -1;
+ return opts.retval;
+}
+
+static int pin_to_cpu0(cpu_set_t *old)
+{
+ cpu_set_t set;
+
+ if (sched_getaffinity(0, sizeof(*old), old))
+ return -1;
+ CPU_ZERO(&set);
+ CPU_SET(0, &set);
+ return sched_setaffinity(0, sizeof(set), &set);
+}
+
+#define NR_THREADS 8
+#define NR_RUNS 20
+
+static int preempted_fd;
+
+static void *preempted_thread(void *arg)
+{
+ long bad = 0;
+ int i;
+
+ for (i = 0; i < NR_RUNS; i++)
+ bad += run_prog(preempted_fd) != 0;
+ return (void *)bad;
+}
+
+/* Programs on one cpu preempt each other. They don't exit in the order they started. */
+static void test_preemption(struct arena_stack *skel)
+{
+ pthread_t thread[NR_THREADS];
+ long bad = 0;
+ cpu_set_t old;
+ void *ret;
+ int i, n;
+
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ return;
+ preempted_fd = bpf_program__fd(skel->progs.preempted);
+ /* threads inherit the affinity */
+ for (n = 0; n < NR_THREADS; n++)
+ if (!ASSERT_OK(pthread_create(&thread[n], NULL, preempted_thread, NULL), "thread"))
+ break;
+ for (i = 0; i < n; i++) {
+ pthread_join(thread[i], &ret);
+ bad += (long)ret;
+ }
+ ASSERT_EQ(bad, 0, "runs with a corrupted frame");
+ sched_setaffinity(0, sizeof(old), &old);
+}
+
+/*
+ * A program that runs with preemption disabled is interrupted by a program in
+ * the timer interrupt
+ */
+static void test_nesting(struct arena_stack *skel)
+{
+ struct perf_event_attr attr = {
+ .size = sizeof(attr),
+ .type = PERF_TYPE_SOFTWARE,
+ .config = PERF_COUNT_SW_CPU_CLOCK,
+ .sample_period = 50000, /* nsec */
+ };
+ struct bpf_link *link = NULL;
+ int i, pfd, bad = 0;
+ cpu_set_t old;
+
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ return;
+ pfd = syscall(__NR_perf_event_open, &attr, -1, 0, -1, 0);
+ if (!ASSERT_GE(pfd, 0, "perf_event_open"))
+ goto out;
+ link = bpf_program__attach_perf_event(skel->progs.in_interrupt, pfd);
+ if (!ASSERT_OK_PTR(link, "attach_perf_event")) {
+ close(pfd);
+ goto out;
+ }
+ for (i = 0; i < 200; i++)
+ bad += run_tc_prog(bpf_program__fd(skel->progs.interrupted)) != 0;
+ ASSERT_EQ(bad, 0, "runs with a corrupted frame");
+ /* the program was interrupted */
+ ASSERT_GT(skel->bss->interrupts, 10, "interrupts");
+ bpf_link__destroy(link);
+out:
+ sched_setaffinity(0, sizeof(old), &old);
+}
+
+/* bpf_throw() doesn't return the stack. Programs run when all stacks of the cpu are taken. */
+static void test_throw(struct arena_stack *skel)
+{
+ cpu_set_t old;
+ int i;
+
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ return;
+ for (i = 0; i < 40; i++)
+ run_prog(bpf_program__fd(skel->progs.throws));
+ for (i = 0; i < 40; i++)
+ if (!ASSERT_EQ(run_prog(bpf_program__fd(skel->progs.prog_returns_7)), 7, "returns_7"))
+ break;
+ /* the same with preemption disabled */
+ for (i = 0; i < 200; i++)
+ run_tc_prog(bpf_program__fd(skel->progs.throws_nested));
+ for (i = 0; i < 40; i++)
+ if (!ASSERT_EQ(run_tc_prog(bpf_program__fd(skel->progs.returns_8_nested)), 8,
+ "returns_8_nested"))
+ break;
+ sched_setaffinity(0, sizeof(old), &old);
+}
+
+/* A callback that the kernel calls takes a stack of its own */
+static void test_timer(struct arena_stack *skel)
+{
+ int i, j;
+
+ for (i = 0; i < 100; i++) {
+ if (!ASSERT_OK(run_prog(bpf_program__fd(skel->progs.start_timer)), "start_timer"))
+ return;
+ for (j = 0; j < 1000 && skel->bss->timer_runs <= i; j++)
+ usleep(100);
+ }
+ ASSERT_EQ(skel->bss->timer_runs, 100, "runs of the callback");
+ ASSERT_EQ(skel->bss->timer_bad, 0, "runs with a wrong frame");
+}
+
+static void test_attach_to_subprog(struct arena_stack *skel)
+{
+ struct arena_stack_fentry *fentry;
+ int tgt_fd = bpf_program__fd(skel->progs.preempted);
+
+ /* the main program is fine */
+ fentry = arena_stack_fentry__open();
+ if (!ASSERT_OK_PTR(fentry, "open"))
+ return;
+ ASSERT_OK(bpf_program__set_attach_target(fentry->progs.on_entry, tgt_fd, "preempted"),
+ "set_attach_target");
+ ASSERT_OK(arena_stack_fentry__load(fentry), "load for the main program");
+ arena_stack_fentry__destroy(fentry);
+
+ fentry = arena_stack_fentry__open();
+ if (!ASSERT_OK_PTR(fentry, "open"))
+ return;
+ ASSERT_OK(bpf_program__set_attach_target(fentry->progs.on_entry, tgt_fd, "spin"),
+ "set_attach_target");
+ ASSERT_ERR(arena_stack_fentry__load(fentry), "load for a subprog");
+ arena_stack_fentry__destroy(fentry);
+}
+
+/* The insns that take the stack are in front of the first insn. A jump to it doesn't run them. */
+static void test_jump_to_first_insn(void)
+{
+ struct arena_stack_loop *skel;
+ cpu_set_t old;
+ int i;
+
+ skel = arena_stack_loop__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ goto out;
+ for (i = 0; i < 1000; i++) {
+ if (!ASSERT_EQ(run_tc_prog(bpf_program__fd(skel->progs.loop)), 7, "loop"))
+ break;
+ if (!ASSERT_EQ(run_tc_prog(bpf_program__fd(skel->progs.nop_loop)), 7, "nop_loop"))
+ break;
+ if (!ASSERT_EQ(run_tc_prog(bpf_program__fd(skel->progs.gotox_loop)), 7,
+ "gotox_loop"))
+ break;
+ }
+ sched_setaffinity(0, sizeof(old), &old);
+out:
+ arena_stack_loop__destroy(skel);
+}
+
+/*
+ * The stack that a callback finds is too small. It gets a new one and the pages of the old
+ * one are freed. Before the callback starts another program gets these pages as its stack
+ * and gives it back. The stack of the callback has to stay taken.
+ */
+static void test_slot_grows(void)
+{
+ struct arena_stack_grow *skel;
+ cpu_set_t old;
+
+ /* the compiler may inline it */
+ if (libbpf_find_vmlinux_btf_id("arena_free_pages", BPF_TRACE_FEXIT) <= 0) {
+ test__skip();
+ return;
+ }
+ skel = arena_stack_grow__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ goto out;
+ /* the stack of the cpu is large enough for both programs */
+ ASSERT_OK(run_prog(bpf_program__fd(skel->progs.big)), "big");
+ ASSERT_OK(run_prog(bpf_program__fd(skel->progs.small)), "small");
+ ASSERT_OK(arena_stack_grow__attach(skel), "attach");
+ skel->bss->nest = 1;
+ ASSERT_OK(run_prog(bpf_program__fd(skel->progs.big)), "big with callbacks");
+ ASSERT_EQ(skel->bss->frame, 1, "frame of outer_cb");
+ /* the stack did not grow, or arena_free_pages() is inlined where it grows */
+ if (!skel->bss->waited)
+ test__skip();
+ sched_setaffinity(0, sizeof(old), &old);
+out:
+ arena_stack_grow__destroy(skel);
+}
+
+static void test_no_memory(void)
+{
+ struct bpf_prog_info info = {};
+ __u32 info_len = sizeof(info);
+ struct arena_stack_nomem *skel;
+ char buf[256] = {};
+ int fd;
+
+ skel = arena_stack_nomem__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ ASSERT_EQ(run_prog(bpf_program__fd(skel->progs.take_page)), 0, "take_page");
+ fd = bpf_program__fd(skel->progs.returns_1);
+ ASSERT_EQ(run_prog(fd), 0, "retval");
+ ASSERT_EQ(skel->bss->runs, 0, "runs");
+ ASSERT_OK(bpf_prog_get_info_by_fd(fd, &info, &info_len), "info");
+ ASSERT_EQ(info.recursion_misses, 1, "misses");
+ ASSERT_GT(bpf_prog_stream_read(fd, BPF_STREAM_STDERR, buf, sizeof(buf) - 1, NULL), 0,
+ "stream_read");
+ ASSERT_HAS_SUBSTR(buf, "No memory for the stack in arena", "stderr");
+ arena_stack_nomem__destroy(skel);
+}
+
+/*
+ * bpf_throw() in a main program that has stack arguments. The exception callback restores the
+ * registers of the caller.
+ */
+static void test_throw_stack_args(void)
+{
+ struct arena_stack_throw *skel;
+ int i;
+
+ skel = arena_stack_throw__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ if (!skel->rodata->has_stack_arg) {
+ test__skip();
+ goto out;
+ }
+ if (!ASSERT_OK(arena_stack_throw__attach(skel), "attach"))
+ goto out;
+ ASSERT_OK(run_prog(bpf_program__fd(skel->progs.calls_kfunc)), "calls_kfunc");
+ ASSERT_GE(skel->bss->throws, 1, "throws");
+ for (i = 0; i < 4; i++)
+ ASSERT_EQ(skel->bss->regs[i], 6 + i, "register");
+out:
+ arena_stack_throw__destroy(skel);
+}
+
+void serial_test_arena_stack(void)
+{
+ struct arena_stack *skel;
+
+#if !defined(__x86_64__) && !defined(__aarch64__)
+ /* other JITs don't support r12 */
+ test__skip();
+ return;
+#endif
+ skel = arena_stack__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ if (test__start_subtest("preemption"))
+ test_preemption(skel);
+ if (test__start_subtest("nesting"))
+ test_nesting(skel);
+ if (test__start_subtest("attach_to_subprog"))
+ test_attach_to_subprog(skel);
+ if (test__start_subtest("timer"))
+ test_timer(skel);
+ if (test__start_subtest("throw"))
+ test_throw(skel);
+ arena_stack__destroy(skel);
+ if (test__start_subtest("throw_stack_args"))
+ test_throw_stack_args();
+ if (test__start_subtest("jump_to_first_insn"))
+ test_jump_to_first_insn();
+ if (test__start_subtest("slot_grows"))
+ test_slot_grows();
+ if (test__start_subtest("no_memory"))
+ test_no_memory();
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
index 5d3c00a08a88..403aaaf12d1c 100644
--- a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
+++ b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
@@ -7,6 +7,7 @@
#include "pro_epilogue_goto_start.skel.h"
#include "epilogue_exit.skel.h"
#include "pro_epilogue_with_kfunc.skel.h"
+#include "pro_epilogue_arena_stack.skel.h"
struct st_ops_args {
__u64 a;
@@ -57,6 +58,7 @@ void test_pro_epilogue(void)
RUN_TESTS(pro_epilogue_goto_start);
RUN_TESTS(epilogue_exit);
RUN_TESTS(pro_epilogue_with_kfunc);
+ RUN_TESTS(pro_epilogue_arena_stack);
if (test__start_subtest("tailcall"))
test_tailcall();
}
diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
index da60c715fc59..db79ac9b98d3 100644
--- a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
+++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_refcounted.c
@@ -4,6 +4,7 @@
#include "struct_ops_refcounted_fail__ref_leak.skel.h"
#include "struct_ops_refcounted_fail__global_subprog.skel.h"
#include "struct_ops_refcounted_fail__tail_call.skel.h"
+#include "struct_ops_refcounted_fail__arena_stack.skel.h"
void test_struct_ops_refcounted(void)
{
@@ -11,4 +12,5 @@ void test_struct_ops_refcounted(void)
RUN_TESTS(struct_ops_refcounted_fail__ref_leak);
RUN_TESTS(struct_ops_refcounted_fail__global_subprog);
RUN_TESTS(struct_ops_refcounted_fail__tail_call);
+ RUN_TESTS(struct_ops_refcounted_fail__arena_stack);
}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 3d0eda3b4da4..01286be73335 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -13,6 +13,7 @@
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
#include "verifier_arena_cast.skel.h"
+#include "verifier_arena_stack.skel.h"
#include "verifier_arena_large.skel.h"
#include "verifier_arena_globals1.skel.h"
#include "verifier_arena_globals2.skel.h"
@@ -209,6 +210,7 @@ void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }
void test_verifier_arena_cast(void) { RUN(verifier_arena_cast); }
+void test_verifier_arena_stack(void) { RUN_TESTS(verifier_arena_stack); }
void test_verifier_arena_large(void) { RUN(verifier_arena_large); }
void test_verifier_arena_globals1(void) { RUN(verifier_arena_globals1); }
void test_verifier_arena_globals2(void) { RUN(verifier_arena_globals2); }
diff --git a/tools/testing/selftests/bpf/progs/arena_noarena.c b/tools/testing/selftests/bpf/progs/arena_noarena.c
index 1f46e5aa403b..8a7ce3e36703 100644
--- a/tools/testing/selftests/bpf/progs/arena_noarena.c
+++ b/tools/testing/selftests/bpf/progs/arena_noarena.c
@@ -21,4 +21,18 @@ int arena_arg_no_arena(void *ctx)
return 0;
}
+SEC("syscall")
+__arch_x86_64
+__arch_arm64
+__description("arena stack: no arena")
+__failure __msg("R12 can only be used in a program that has an associated arena")
+__naked void r12_no_arena(void)
+{
+ asm volatile (
+ ".8byte 0xc1bf;" /* r1 = r12 */
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack.c b/tools/testing/selftests/bpf/progs/arena_stack.c
new file mode 100644
index 000000000000..8c93e4cea71c
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack.c
@@ -0,0 +1,334 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+extern void bpf_throw(__u64 cookie) __ksym;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 4096);
+} arena SEC(".maps");
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_throw(0);
+}
+
+/* Compilers don't know r12 yet */
+#define R12 12
+#define ADJ(n) BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, R12, 0, 0, n)
+#define LDX(dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEM | BPF_DW, dst, R12, off, 0)
+#define STX(off, src) BPF_RAW_INSN(BPF_STX | BPF_MEM | BPF_DW, R12, src, off, 0)
+#define ST(off, imm) BPF_RAW_INSN(BPF_ST | BPF_MEM | BPF_DW, R12, 0, off, imm)
+
+/*
+ * Fill the frame, spin long enough to be preempted, check the frame.
+ * Returns the number of words that have changed.
+ */
+static __naked __noinline __used long spin(long seed)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[stx32];"
+ "r1 += 1;"
+ ".8byte %[stx24];"
+ "r1 += 1;"
+ ".8byte %[stx16];"
+ "r1 += 1;"
+ ".8byte %[stx8];"
+ "r2 = 3000000;"
+"1:"
+ "r2 += -1;"
+ "if r2 != 0 goto 1b;"
+ "r0 = 0;"
+ ".8byte %[ldx8];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "r1 -= 1;"
+ ".8byte %[ldx16];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "r1 -= 1;"
+ ".8byte %[ldx24];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "r1 -= 1;"
+ ".8byte %[ldx32];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(32)),
+ __imm_insn(stx32, STX(-32, BPF_REG_1)),
+ __imm_insn(stx24, STX(-24, BPF_REG_1)),
+ __imm_insn(stx16, STX(-16, BPF_REG_1)),
+ __imm_insn(stx8, STX(-8, BPF_REG_1)),
+ __imm_insn(ldx32, LDX(BPF_REG_2, -32)),
+ __imm_insn(ldx24, LDX(BPF_REG_2, -24)),
+ __imm_insn(ldx16, LDX(BPF_REG_2, -16)),
+ __imm_insn(ldx8, LDX(BPF_REG_2, -8))
+ : __clobber_all);
+}
+
+/* the same, one frame deeper */
+static __naked __noinline __used long spin_deeper(long seed)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[stx];"
+ "r6 = r1;"
+ "r1 += 100;"
+ "call spin;"
+ ".8byte %[ldx];"
+ "if r1 == r6 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(128)),
+ __imm_insn(stx, STX(-64, BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_REG_1, -64))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int preempted(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0;"
+ ".8byte %[stx];"
+ "r1 = r6;"
+ "if r6 & 1 goto 1f;"
+ "call spin;"
+ "goto 2f;"
+"1:"
+ "call spin_deeper;"
+"2:"
+ ".8byte %[ldx];"
+ "if r1 == r6 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(-16, BPF_REG_6)),
+ __imm_insn(ldx, LDX(BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+static long thrower(void);
+
+/* The same with preemption disabled. Programs in interrupts nest. */
+SEC("tc")
+__naked int interrupted(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0;"
+ ".8byte %[stx];"
+ "r1 = r6;"
+ "call spin_deeper;"
+ ".8byte %[ldx];"
+ "if r1 == r6 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(-16, BPF_REG_6)),
+ __imm_insn(ldx, LDX(BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+long interrupts;
+
+static __naked __noinline __used long scribble(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st128];"
+ ".8byte %[st64];"
+ ".8byte %[st8];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(128)),
+ __imm_insn(st128, ST(-128, -1)),
+ __imm_insn(st64, ST(-64, -1)),
+ __imm_insn(st8, ST(-8, -1))
+ : __clobber_all);
+}
+
+SEC("perf_event")
+__naked int in_interrupt(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st32];"
+ ".8byte %[st8];"
+ "call scribble;"
+ "r1 = %[interrupts] ll;"
+ "r2 = 1;"
+ "lock *(u64 *)(r1 + 0) += r2;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(interrupts),
+ __imm_insn(adj, ADJ(32)),
+ __imm_insn(st32, ST(-32, -1)),
+ __imm_insn(st8, ST(-8, -1))
+ : __clobber_all);
+}
+
+SEC("tc")
+__naked int throws_nested(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call thrower;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+SEC("tc")
+__naked int returns_8_nested(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(-16, 8)),
+ __imm_insn(ldx, LDX(BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long thrower(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 5;"
+ "call %[bpf_throw];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_throw),
+ __imm_insn(adj, ADJ(64))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int throws(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call thrower;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long returns_7(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(-16, 7)),
+ __imm_insn(ldx, LDX(BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int prog_returns_7(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(-16, 7)),
+ __imm_insn(ldx, LDX(BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+struct elem {
+ struct bpf_timer t;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct elem);
+} timers SEC(".maps");
+
+long timer_runs, timer_bad;
+
+/* The kernel calls it. Fill the frame, call a function that has a frame, check the frame. */
+static __naked __noinline __used int timer_cb(void *map, int *key, struct elem *val)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st32];"
+ ".8byte %[st8];"
+ "call returns_7;"
+ ".8byte %[ldx32];"
+ ".8byte %[ldx8];"
+ "r1 += r2;"
+ "r1 += r0;"
+ "r2 = %[timer_runs] ll;"
+ "r3 = 1;"
+ "lock *(u64 *)(r2 + 0) += r3;"
+ "if r1 == 49 goto 1f;"
+ "r2 = %[timer_bad] ll;"
+ "lock *(u64 *)(r2 + 0) += r3;"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(timer_runs),
+ __imm_addr(timer_bad),
+ __imm_insn(adj, ADJ(32)),
+ __imm_insn(st32, ST(-32, 11)),
+ __imm_insn(st8, ST(-8, 31)),
+ __imm_insn(ldx32, LDX(BPF_REG_1, -32)),
+ __imm_insn(ldx8, LDX(BPF_REG_2, -8))
+ : __clobber_all);
+}
+
+SEC("syscall")
+int start_timer(void *ctx)
+{
+ struct elem *e;
+ int key = 0;
+
+ /* the program has to have an arena */
+ asm volatile ("" :: "r"(&arena));
+ e = bpf_map_lookup_elem(&timers, &key);
+ if (!e)
+ return 2;
+ bpf_timer_init(&e->t, &timers, 1 /* CLOCK_MONOTONIC */);
+ if (bpf_timer_set_callback(&e->t, timer_cb))
+ return 3;
+ return bpf_timer_start(&e->t, 0, 0) ? 4 : 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_fentry.c b/tools/testing/selftests/bpf/progs/arena_stack_fentry.c
new file mode 100644
index 000000000000..4dd870c51166
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_fentry.c
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+SEC("fentry")
+int BPF_PROG(on_entry)
+{
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_grow.c b/tools/testing/selftests/bpf/progs/arena_stack_grow.c
new file mode 100644
index 000000000000..be64296431c3
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_grow.c
@@ -0,0 +1,151 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 4096);
+} arena SEC(".maps");
+
+/* Compilers don't know r12 yet */
+#define R12 12
+#define ADJ(n) BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, R12, 0, 0, n)
+#define LDX(dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEM | BPF_DW, dst, R12, off, 0)
+#define ST(off, imm) BPF_RAW_INSN(BPF_ST | BPF_MEM | BPF_DW, R12, 0, off, imm)
+
+/* With 4K pages a program with such a frame needs a stack of two pages, the others need one */
+#define BIG 4096
+
+int nest, freed, waited;
+long frame;
+
+static __naked __noinline __used long small_cb(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+/* The callback takes a stack of its own, the first one after the stack of the cpu */
+SEC("syscall")
+__naked int small(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 1;"
+ "r2 = small_cb ll;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long inner_cb(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(-16, 2))
+ : __clobber_all);
+}
+
+/* Write to the start of the stack, let inner_cb() do the same in its stack, read it back */
+static __naked __noinline __used long outer_cb(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "r1 = 1;"
+ "r2 = inner_cb ll;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ ".8byte %[ldx];"
+ "r2 = %[frame] ll;"
+ "*(u64 *)(r2 + 0) = r1;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_addr(frame),
+ __imm_insn(adj, ADJ(BIG)),
+ __imm_insn(st, ST(-BIG, 1)),
+ __imm_insn(ldx, LDX(BPF_REG_1, -BIG))
+ : __clobber_all);
+}
+
+/* The stack that small_cb() has used is too small for outer_cb() */
+SEC("syscall")
+__naked int big(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = %[nest] ll;"
+ "r1 = *(u32 *)(r1 + 0);"
+ "if r1 == 0 goto 1f;"
+ "r1 = 1;"
+ "r2 = outer_cb ll;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_addr(nest),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+/* The pages of the old stack are free. This program gets them. */
+SEC("fexit/arena_free_worker")
+__naked int after_free(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = %[freed] ll;"
+ "r2 = 1;"
+ "*(u32 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(freed),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static long is_freed(__u64 i, void *ctx)
+{
+ return *(volatile int *)&freed != 0;
+}
+
+/* The old pages are queued. Keep outer_cb() here until the worker has preempted it. */
+SEC("fexit/arena_free_pages")
+int wait_for_worker(void *ctx)
+{
+ if (nest && !bpf_get_smp_processor_id()) {
+ waited = 1;
+ bpf_loop(1 << 23, is_freed, NULL, 0);
+ }
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_loop.c b/tools/testing/selftests/bpf/progs/arena_stack_loop.c
new file mode 100644
index 000000000000..21e7e784b267
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_loop.c
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/* A stack that is not returned is noticed soon */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 8);
+} arena SEC(".maps");
+
+static __naked __noinline __used long returns_7(void)
+{
+ asm volatile (
+ ".8byte %[adj];" /* r12 += 16 */
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 7 */
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 8) */
+ "exit;"
+ :
+ : __imm_insn(adj, BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, 12, 0, 0, 16)),
+ __imm_insn(st, BPF_RAW_INSN(BPF_ST | BPF_MEM | BPF_DW, 12, 0, -8, 7)),
+ __imm_insn(ldx, BPF_RAW_INSN(BPF_LDX | BPF_MEM | BPF_DW, 0, 12, -8, 0))
+ : __clobber_all);
+}
+
+/* r1 is ctx, then 0: one jump to the first insn of the program */
+#define LOOP \
+"1:" \
+ "if r1 == 0 goto 2f;" \
+ "r1 = 0;" \
+ "goto 1b;" \
+"2:" \
+ "call returns_7;" \
+ "exit;"
+
+SEC("tc")
+__naked int loop(void)
+{
+ asm volatile (LOOP ::: __clobber_all);
+}
+
+/* the same after the nop is removed */
+SEC("tc")
+__naked int nop_loop(void)
+{
+ asm volatile ("goto +0;" LOOP ::: __clobber_all);
+}
+
+/* the same through a jump table */
+SEC("tc")
+__naked int gotox_loop(void)
+{
+ asm volatile (
+ ".pushsection .jumptables,\"\",@progbits;"
+"jt_%=:"
+ ".quad first_%= - tc;"
+ ".size jt_%=, 8;"
+ ".global jt_%=;"
+ ".popsection;"
+"first_%=:"
+ "if r1 == 0 goto 2f;"
+ "r1 = 0;"
+ "r2 = jt_%= ll;"
+ "r2 = *(u64 *)(r2 + 0);"
+ ".8byte %[gotox];" /* gotox r2 */
+"2:"
+ "call returns_7;"
+ "exit;"
+ :
+ : __imm_insn(gotox, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_2, 0, 0, 0))
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_nomem.c b/tools/testing/selftests/bpf/progs/arena_stack_nomem.c
new file mode 100644
index 000000000000..9519249478a9
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_nomem.c
@@ -0,0 +1,41 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/* The arena has one page. take_page() takes it, so there is no page for a stack. */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+void *bpf_arena_alloc_pages(void *map, void *addr, __u32 page_cnt, int node_id,
+ __u64 flags) __ksym;
+
+int runs;
+
+SEC("syscall")
+int take_page(void *ctx)
+{
+ return bpf_arena_alloc_pages(&arena, NULL, 1, -1 /* NUMA_NO_NODE */, 0) ? 0 : 1;
+}
+
+SEC("syscall")
+__naked int returns_1(void)
+{
+ asm volatile (
+ ".8byte %[adj];" /* r12 += 16 */
+ "r1 = %[runs] ll;"
+ "r0 = 1;"
+ "*(u32 *)(r1 + 0) = r0;"
+ "exit;"
+ :
+ : __imm_addr(runs),
+ __imm_insn(adj, BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, 12, 0, 0, 16))
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_throw.c b/tools/testing/selftests/bpf/progs/arena_stack_throw.c
new file mode 100644
index 000000000000..ac14139fb901
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_throw.c
@@ -0,0 +1,110 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <stdbool.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+extern void bpf_throw(__u64 cookie) __ksym;
+extern void bpf_kfunc_common_test(void) __ksym;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 64);
+} arena SEC(".maps");
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_throw(0);
+ bpf_kfunc_common_test();
+}
+
+long never, throws, regs[4];
+
+/*
+ * r6-r9 are in registers that a callee keeps. What the kfunc returns with is stored in regs[].
+ * A program that can throw saves all such registers, so its own caller is not affected.
+ */
+SEC("syscall")
+__naked int calls_kfunc(void)
+{
+ asm volatile (
+ "r1 = %[never] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 == 0 goto +1;"
+ "call %[bpf_throw];"
+ "r6 = 6;"
+ "r7 = 7;"
+ "r8 = 8;"
+ "r9 = 9;"
+ "call %[bpf_kfunc_common_test];"
+ "r1 = %[regs] ll;"
+ "*(u64 *)(r1 + 0) = r6;"
+ "*(u64 *)(r1 + 8) = r7;"
+ "*(u64 *)(r1 + 16) = r8;"
+ "*(u64 *)(r1 + 24) = r9;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_throw),
+ __imm(bpf_kfunc_common_test),
+ __imm_addr(never),
+ __imm_addr(regs)
+ : __clobber_all);
+}
+
+#ifdef __BPF_FEATURE_STACK_ARGUMENT
+
+const volatile bool has_stack_arg = true;
+
+static __naked __noinline __used
+long callee_6args(long a, long b, long c, long d, long e, long f)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r11 + 8);"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* The main program uses r12, has a stack argument and throws */
+SEC("fentry/bpf_kfunc_common_test")
+__naked int on_kfunc(void)
+{
+ asm volatile (
+ ".8byte %[adj];" /* r12 += 16 */
+ "r1 = %[throws] ll;"
+ "r2 = 1;"
+ "lock *(u64 *)(r1 + 0) += r2;"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call callee_6args;"
+ "r1 = 0;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm(bpf_throw),
+ __imm_addr(throws),
+ __imm_insn(adj, BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, 12, 0, 0, 16))
+ : __clobber_all);
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("fentry/bpf_kfunc_common_test")
+int on_kfunc(void *ctx)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/pro_epilogue_arena_stack.c b/tools/testing/selftests/bpf/progs/pro_epilogue_arena_stack.c
new file mode 100644
index 000000000000..40fbcc004608
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/pro_epilogue_arena_stack.c
@@ -0,0 +1,199 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+char _license[] SEC("license") = "GPL";
+
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+
+/* One page: the stack of the cpu, or what take_page() takes so that there is no stack */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+/* Compilers don't know r12 yet, vmlinux.h has no BPF_ALU64 and the like */
+#define INSN(code, dst, src, off, imm) \
+ ((__u64)(code) | (__u64)(dst) << 8 | (__u64)(src) << 12 | \
+ (__u64)(__u16)(off) << 16 | (__u64)(__u32)(imm) << 32)
+/* r12 += n */
+#define ADJ(n) INSN(0x07, 12, 0, 0, n)
+/* *(u64 *)(r12 + off) = imm */
+#define ST(off, imm) INSN(0x7a, 12, 0, off, imm)
+/* dst = *(u64 *)(r12 + off) */
+#define LDX(dst, off) INSN(0x79, dst, 12, off, 0)
+
+/* args->a += 1, through the stack in arena */
+static __naked __noinline __used int subprog(struct st_ops_args *args)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 1 */
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 8) */
+ "r2 = *(u64 *)(r1 +0);"
+ "r0 += r2;"
+ "*(u64 *)(r1 +0) = r0;"
+ "exit;"
+ :
+ : __imm_const(adj, ADJ(48)),
+ __imm_const(st, ST(-8, 1)),
+ __imm_const(ldx, LDX(BPF_REG_0, -8))
+ : __clobber_all);
+}
+
+/*
+ * The prologue and the store of ctx are in front of the code that takes a stack.
+ * Every exit goes to the epilogue: the one without a stack is the first of them.
+ */
+__success
+/* prologue */
+__xlated("0: r6 = *(u64 *)(r1 +0)")
+__xlated("1: r7 = *(u64 *)(r6 +0)")
+__xlated("2: r7 += 1000")
+__xlated("3: *(u64 *)(r6 +0) = r7")
+/* save __u64 *ctx to stack */
+__xlated("4: *(u64 *)(r10 -8) = r1")
+/* take a stack */
+__xlated("5: r2 = ")
+__xlated("7: r2 = &(void __percpu *)(r2)")
+__xlated("...")
+__xlated("w12 = w0")
+__xlated("r1 = r6")
+__xlated("if r0 != 0x0 goto pc+")
+/* no stack */
+__xlated("r0 = 0")
+/* epilogue */
+__xlated("r1 = *(u64 *)(r10 -8)")
+__xlated("r1 = *(u64 *)(r1 +0)")
+__xlated("r6 = *(u64 *)(r1 +0)")
+__xlated("r6 += 10000")
+__xlated("*(u64 *)(r1 +0) = r6")
+__xlated("r0 = r6")
+__xlated("r0 *= 2")
+__xlated("exit")
+__xlated("...")
+/* main prog */
+__xlated("w12 += 16")
+__xlated("*(u64 *)(r12 -8) = 10")
+__xlated("...")
+__xlated("call pc+")
+/* give the stack back, then the epilogue */
+__xlated("w12 += -16")
+__xlated("if w12 & 0x10 goto pc+5")
+__xlated("...")
+__xlated("*(u64 *)(r2 +0) = 0")
+__xlated("gotol pc-")
+__xlated("...")
+__xlated("r0 = r6")
+__xlated("gotol pc-")
+SEC("struct_ops/test_pro_epilogue")
+__naked int test_pro_epilogue(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 10 */
+ "r1 = *(u64 *)(r1 +0);"
+ "r2 = *(u64 *)(r1 +0);"
+ ".8byte %[ldx];" /* r3 = *(u64 *)(r12 - 8) */
+ "r2 += r3;"
+ "*(u64 *)(r1 +0) = r2;"
+ "call subprog;"
+ "exit;"
+ :
+ : __imm_const(adj, ADJ(16)),
+ __imm_const(st, ST(-8, 10)),
+ __imm_const(ldx, LDX(BPF_REG_3, -8))
+ : __clobber_all);
+}
+
+/* No frame in the main prog, r12 is used by the function that it calls */
+__success
+SEC("struct_ops/test_prologue")
+__naked int test_prologue(void)
+{
+ asm volatile (
+ "r1 = *(u64 *)(r1 +0);"
+ "call subprog;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+__success
+SEC("struct_ops/test_epilogue")
+__naked int test_epilogue(void)
+{
+ asm volatile (
+ "r1 = *(u64 *)(r1 +0);"
+ "call subprog;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("syscall")
+__retval(1001) /* PROLOGUE_A [1000] + SUBPROG_A [1] */
+int syscall_prologue(void *ctx)
+{
+ struct st_ops_args args = {};
+
+ return bpf_kfunc_st_ops_test_prologue(&args);
+}
+
+SEC("syscall")
+__retval(20002) /* (SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */
+int syscall_epilogue(void *ctx)
+{
+ struct st_ops_args args = {};
+
+ return bpf_kfunc_st_ops_test_epilogue(&args);
+}
+
+SEC("syscall")
+__retval(22022) /* (PROLOGUE_A [1000] + MAIN_A [10] + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */
+int syscall_pro_epilogue(void *ctx)
+{
+ struct st_ops_args args = {};
+
+ return bpf_kfunc_st_ops_test_pro_epilogue(&args);
+}
+
+static bool take_page(void)
+{
+ return bpf_arena_alloc_pages(&arena, NULL, 1, -1 /* NUMA_NO_NODE */, 0);
+}
+
+/* A prog that gets no stack is skipped. Its prologue and its epilogue are not. */
+SEC("syscall")
+__retval(1000) /* PROLOGUE_A [1000], the prog returns 0 */
+int syscall_prologue_no_stack(void *ctx)
+{
+ struct st_ops_args args = {};
+
+ if (!take_page() || bpf_kfunc_st_ops_test_prologue(&args))
+ return -1;
+ return args.a;
+}
+
+SEC("syscall")
+__retval(20000) /* EPILOGUE_A [10000] * 2 */
+int syscall_epilogue_no_stack(void *ctx)
+{
+ struct st_ops_args args = {};
+
+ if (!take_page())
+ return -1;
+ return bpf_kfunc_st_ops_test_epilogue(&args);
+}
+
+SEC(".struct_ops.link")
+struct bpf_testmod_st_ops pro_epilogue_arena_stack = {
+ .test_prologue = (void *)test_prologue,
+ .test_epilogue = (void *)test_epilogue,
+ .test_pro_epilogue = (void *)test_pro_epilogue,
+};
+
+#endif
diff --git a/tools/testing/selftests/bpf/progs/struct_ops_refcounted_fail__arena_stack.c b/tools/testing/selftests/bpf/progs/struct_ops_refcounted_fail__arena_stack.c
new file mode 100644
index 000000000000..35ed2fc28a94
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/struct_ops_refcounted_fail__arena_stack.c
@@ -0,0 +1,61 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include "../test_kmods/bpf_testmod.h"
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_task_release(NULL);
+}
+
+/* r12 += 16, compilers don't know r12 yet */
+#define ADJ_16 (0x07 | 12 << 8 | 16ULL << 32)
+
+static __naked __noinline __used void uses_arena_stack(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "exit;"
+ :
+ : __imm_const(adj, ADJ_16)
+ : __clobber_all);
+}
+
+/*
+ * A program that gets no stack in arena is skipped: nobody would release the
+ * task. It doesn't matter which function of the program uses r12.
+ */
+SEC("struct_ops/test_refcounted")
+__failure __msg("program with __ref argument cannot use R12")
+__naked int refcounted_fail__arena_stack(void)
+{
+ asm volatile (
+ "r1 = *(u64 *)(r1 + 8);"
+ "call %[bpf_task_release];"
+ "call uses_arena_stack;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_task_release)
+ : __clobber_all);
+}
+
+SEC(".struct_ops.link")
+struct bpf_testmod_ops testmod_ref_acquire = {
+ .test_refcounted = (void *)refcounted_fail__arena_stack,
+};
+
+#endif
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
new file mode 100644
index 000000000000..a23ab1ed2f96
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
@@ -0,0 +1,1169 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 64);
+} arena SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(int));
+} jmp_table SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_USER_RINGBUF);
+ __uint(max_entries, 4096);
+} user_ringbuf SEC(".maps");
+
+/* Compilers don't know r12 yet */
+#define R12 12
+/* r12 += n */
+#define ADJ(n) BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, R12, 0, 0, n)
+/* dst = r12 */
+#define MOV(dst) BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, R12, 0, 0)
+/* dst = *(sz *)(r12 + off) */
+#define LDX(sz, dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEM | sz, dst, R12, off, 0)
+#define LDSX(sz, dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEMSX | sz, dst, R12, off, 0)
+/* *(sz *)(r12 + off) = src */
+#define STX(sz, off, src) BPF_RAW_INSN(BPF_STX | BPF_MEM | sz, R12, src, off, 0)
+/* *(sz *)(r12 + off) = imm */
+#define ST(sz, off, imm) BPF_RAW_INSN(BPF_ST | BPF_MEM | sz, R12, 0, off, imm)
+/* dst = addr_space_cast(dst, 0, 1) */
+#define CAST_KERN(dst) BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, dst, BPF_ADDR_SPACE_CAST, 1)
+/* gotox reg */
+#define GOTOX(reg) BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, reg, 0, 0, 0)
+
+SEC("syscall")
+__description("arena stack: load and store")
+__success __retval(12)
+__xlated("r0 = *(u64 *)(r12 -16)")
+__xlated("r2 = *(u64 *)(r12 -8)")
+__naked void load_store(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 7 */
+ "r1 = 5;"
+ ".8byte %[stx];" /* *(u64 *)(r12 - 8) = r1 */
+ ".8byte %[ldx0];" /* r0 = *(u64 *)(r12 - 16) */
+ ".8byte %[ldx1];" /* r2 = *(u64 *)(r12 - 8) */
+ "r0 += r2;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 7)),
+ __imm_insn(stx, STX(BPF_DW, -8, BPF_REG_1)),
+ __imm_insn(ldx0, LDX(BPF_DW, BPF_REG_0, -16)),
+ __imm_insn(ldx1, LDX(BPF_DW, BPF_REG_2, -8))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: every size, sign extension")
+__success __retval(0)
+__naked void sizes(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 0x8899aabb ll;"
+ ".8byte %[stx];" /* *(u64 *)(r12 - 16) = r1 */
+ "r0 = 1;"
+ ".8byte %[ldx_b];"
+ "if r2 != 0xbb goto 1f;"
+ "r0 = 2;"
+ ".8byte %[ldx_h];"
+ "if r2 != 0xaabb goto 1f;"
+ "r0 = 3;"
+ ".8byte %[ldx_w];"
+ "r3 = 0x8899aabb ll;"
+ "if r2 != r3 goto 1f;"
+ "r0 = 4;"
+ ".8byte %[ldsx_w];"
+ "if r2 s>= 0 goto 1f;"
+ "r0 = 5;"
+ ".8byte %[st_b];" /* *(u8 *)(r12 - 15) = 0 */
+ ".8byte %[ldx_h];"
+ "if r2 != 0xbb goto 1f;"
+ "r0 = 0;"
+"1:"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(BPF_DW, -16, BPF_REG_1)),
+ __imm_insn(ldx_b, LDX(BPF_B, BPF_REG_2, -16)),
+ __imm_insn(ldx_h, LDX(BPF_H, BPF_REG_2, -16)),
+ __imm_insn(ldx_w, LDX(BPF_W, BPF_REG_2, -16)),
+ __imm_insn(ldsx_w, LDSX(BPF_W, BPF_REG_2, -16)),
+ __imm_insn(st_b, ST(BPF_B, -15, 0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: rX = r12 is the address of the frame")
+__success __retval(9)
+__naked void address(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[mov];" /* r1 = r12 */
+ "r1 += -8;"
+ ".8byte %[cast];"
+ "r2 = 9;"
+ "*(u64 *)(r1 + 0) = r2;"
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 8) */
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(mov, MOV(BPF_REG_1)),
+ __imm_insn(cast, CAST_KERN(BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -8))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long callee_with_frame(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st0];"
+ ".8byte %[st1];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st0, ST(BPF_DW, -16, 10)),
+ __imm_insn(st1, ST(BPF_DW, -8, 10)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: the frame of the callee is above")
+__success __retval(21)
+__naked void call(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 1 */
+ "call callee_with_frame;"
+ "r6 = r0;"
+ "call callee_with_frame;"
+ "r0 += r6;"
+ ".8byte %[ldx];" /* r1 = *(u64 *)(r12 - 16) */
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+/* no frame of its own, r12 goes through */
+static __naked __noinline __used long callee_in_the_middle(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "call callee_with_frame;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: a function without a frame in the middle")
+__success __retval(11)
+__naked void call_through(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "call callee_in_the_middle;"
+ ".8byte %[ldx];"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: helpers don't change r12")
+__success __retval(3)
+__naked void helper(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "call %[bpf_get_prandom_u32];"
+ "call %[bpf_ktime_get_ns];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm(bpf_ktime_get_ns),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 3)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long store_through(long addr)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[cast];"
+ "r2 = 5;"
+ "*(u64 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(64)),
+ __imm_insn(st, ST(BPF_DW, -64, 77)),
+ __imm_insn(cast, CAST_KERN(BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: the callee writes to the frame of the caller")
+__success __retval(5)
+__naked void pass_address(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[mov];"
+ "r1 += -16;"
+ "call store_through;"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(mov, MOV(BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long unwinds(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "r0 = 0;"
+ ".8byte %[unwind_exit];"
+ :
+ : __imm_insn(adj, ADJ(32)),
+ __imm_insn(st, ST(BPF_DW, -32, -1)),
+ __imm_insn(unwind_exit, BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+ : __clobber_all);
+}
+
+/* its pad only carries on */
+static __naked __noinline __used long unwinds_through(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call unwinds;"
+ ".8byte %[unwind_goto];"
+ "r0 = 0;"
+ "exit;"
+ ".8byte %[unwind_exit];"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(unwind_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, 2, 0)),
+ __imm_insn(unwind_exit, BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: r12 in the landing pad")
+__success __retval(5)
+__naked void unwind(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 5 */
+ "call unwinds_through;"
+ ".8byte %[unwind_goto];"
+ "r0 = 1;"
+ "exit;"
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 16) */
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 5)),
+ __imm_insn(unwind_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, 2, 0)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+#ifdef __BPF_FEATURE_STACK_ARGUMENT
+
+extern __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f,
+ __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
+
+struct prog_test_pair_arg {
+ __u64 lo;
+ __u64 hi;
+};
+
+extern __u64 bpf_kfunc_call_test_pair_arg_nofit(__u64 a, __u64 b, __u64 c, __u64 d,
+ struct prog_test_pair_arg s) __ksym;
+extern void bpf_throw(__u64 cookie) __ksym;
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ struct prog_test_pair_arg s = {};
+
+ bpf_kfunc_call_stack_arg(0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
+ bpf_kfunc_call_test_pair_arg_nofit(0, 0, 0, 0, s);
+ bpf_throw(0);
+}
+
+static __naked __noinline __used
+long callee_7args(long a, long b, long c, long d, long e, long f, long g)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r0 = *(u64 *)(r11 + 8);"
+ ".8byte %[stx0];" /* *(u64 *)(r12 - 16) = f */
+ "r0 = *(u64 *)(r11 + 16);"
+ ".8byte %[stx8];" /* *(u64 *)(r12 - 8) = g */
+ "r6 = r5;"
+ "call %[bpf_get_prandom_u32];"
+ ".8byte %[ldx0];"
+ ".8byte %[ldx8];"
+ "r0 *= 10;"
+ "r0 += r1;"
+ "r0 *= 10;"
+ "r0 += r6;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx0, STX(BPF_DW, -16, BPF_REG_0)),
+ __imm_insn(stx8, STX(BPF_DW, -8, BPF_REG_0)),
+ __imm_insn(ldx0, LDX(BPF_DW, BPF_REG_0, -16)),
+ __imm_insn(ldx8, LDX(BPF_DW, BPF_REG_1, -8))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: a function with 7 arguments")
+__success __retval(1675)
+__naked void stack_args(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 1000 */
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call callee_7args;"
+ ".8byte %[ldx];"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1000)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena stack: a kfunc with 10 arguments")
+__success __retval(1055)
+__naked void kfunc_stack_args(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 1000 */
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 40) = 10;"
+ "*(u64 *)(r11 - 32) = 9;"
+ "*(u64 *)(r11 - 24) = 8;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call %[bpf_kfunc_call_stack_arg];"
+ ".8byte %[ldx];"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_stack_arg),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1000)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+/*
+ * bpf_throw() finds the registers that the main program saved from its stack pointer at the
+ * call. x86 moves rsp for a kfunc that has stack arguments. bpf_throw() is not one of them.
+ */
+SEC("syscall")
+__description("arena stack: no move of the stack pointer in front of bpf_throw()")
+__success
+__arch_x86_64
+__jited(" movq %r9, {{.*}}(%rbp)")
+__jited(" callq {{.*}}")
+__naked void throw_stack_args_jit(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call callee_7args;"
+ "r1 = r0;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm(bpf_throw),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+/* The caller of the program goes on with the registers that the exception callback restores */
+SEC("syscall")
+__description("arena stack: the main program has stack arguments and throws")
+__success __retval(675)
+__naked void throw_stack_args(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call callee_7args;"
+ "r1 = r0;"
+ "call %[bpf_throw];"
+ "exit;"
+ :
+ : __imm(bpf_throw),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+/* The second half of the struct is argument 6 for the JIT: slots count, not arguments */
+SEC("tc")
+__description("arena stack: a kfunc with 5 arguments in 6 slots")
+__success __retval(1091)
+__naked void kfunc_5args_6slots(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 1000 */
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call %[bpf_kfunc_call_test_pair_arg_nofit];"
+ ".8byte %[ldx];"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_pair_arg_nofit),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1000)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+#endif /* __BPF_FEATURE_STACK_ARGUMENT */
+
+SEC("syscall")
+__description("arena stack: what is loaded is a number")
+__failure __msg("R1 invalid mem access 'scalar'")
+/* the report names the load as the origin of the number */
+__msg("Causal path:")
+__msg(">>> 3 | (79) r1 = *(u64 *)(r12 -16)")
+__naked void loaded_is_number(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = r10;"
+ ".8byte %[stx];"
+ ".8byte %[ldx];"
+ "r0 = *(u64 *)(r1 - 8);"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(BPF_DW, -16, BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: rX = r12 is a number")
+__failure __msg("R1 invalid mem access 'scalar'")
+__msg("Causal path:")
+__msg(">>> 1 | (bf) r1 = r12")
+__naked void address_is_number(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[mov];"
+ "r0 = *(u64 *)(r1 + 0);"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(mov, MOV(BPF_REG_1))
+ : __clobber_all);
+}
+
+#define BAD_INSN(name, desc, msg, first, second) \
+SEC("syscall") \
+__description("arena stack: " desc) \
+__failure __msg(msg) \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ ".8byte %[first_insn];" \
+ ".8byte %[second_insn];" \
+ "r0 = 0;" \
+ "exit;" \
+ : \
+ : __imm_insn(first_insn, first), \
+ __imm_insn(second_insn, second) \
+ : __clobber_all); \
+}
+
+#define NOP BPF_MOV64_IMM(BPF_REG_1, 0)
+
+BAD_INSN(adj_not_first, "r12 += 16 is not the first insn",
+ "R12 can be changed by the first insn of a function only", NOP, ADJ(16))
+BAD_INSN(adj_twice, "r12 += 16 twice",
+ "R12 can be changed by the first insn of a function only", ADJ(16), ADJ(16))
+BAD_INSN(adj_negative, "r12 += -16", "invalid size of the frame", ADJ(-16), NOP)
+BAD_INSN(adj_zero, "r12 += 0", "invalid size of the frame", ADJ(0), NOP)
+BAD_INSN(adj_unaligned, "r12 += 8", "invalid size of the frame", ADJ(8), NOP)
+BAD_INSN(adj_large, "r12 += 32784", "invalid size of the frame", ADJ(32784), NOP)
+BAD_INSN(mov_to, "r12 = r1", "R12 is invalid", NOP, BPF_MOV64_REG(R12, BPF_REG_1))
+BAD_INSN(mov_imm, "r12 = 0", "R12 is invalid", NOP, BPF_MOV64_IMM(R12, 0))
+BAD_INSN(add_reg, "r12 += r1", "R12 is invalid", NOP, BPF_ALU64_REG(BPF_ADD, R12, BPF_REG_1))
+BAD_INSN(adj32, "w12 += 16", "R12 is invalid", BPF_ALU32_IMM(BPF_ADD, R12, 16), NOP)
+BAD_INSN(sub_imm, "r12 -= 16", "R12 is invalid", BPF_ALU64_IMM(BPF_SUB, R12, 16), NOP)
+BAD_INSN(add_from, "r1 += r12", "R12 is invalid", NOP, BPF_ALU64_REG(BPF_ADD, BPF_REG_1, R12))
+BAD_INSN(mov32_from, "w1 = w12", "R12 is invalid", NOP, BPF_MOV32_REG(BPF_REG_1, R12))
+BAD_INSN(load_to, "r12 = *(u64 *)(r10 - 8)", "R12 is invalid", NOP,
+ BPF_LDX_MEM(BPF_DW, R12, BPF_REG_10, -8))
+BAD_INSN(store_from, "*(u64 *)(r10 - 8) = r12", "R12 is invalid", NOP,
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, R12, -8))
+BAD_INSN(store_itself, "*(u64 *)(r12 - 8) = r12", "R12 is invalid", NOP,
+ BPF_STX_MEM(BPF_DW, R12, R12, -8))
+BAD_INSN(atomic, "lock *(u64 *)(r12 - 8) += r1", "R12 is invalid", NOP,
+ BPF_ATOMIC_OP(BPF_DW, BPF_ADD, R12, BPF_REG_1, -8))
+BAD_INSN(jump, "if r12 == 0 goto", "R12 is invalid", NOP, BPF_JMP_IMM(BPF_JEQ, R12, 0, 0))
+BAD_INSN(r13, "r1 = r13", "R13 is invalid", NOP, BPF_MOV64_REG(BPF_REG_1, 13))
+BAD_INSN(local_atomic, "atomic that is local to the cpu", "BPF_ATOMIC uses invalid atomic opcode",
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH | 0x1000, BPF_REG_10, BPF_REG_1, -8))
+
+SEC("syscall")
+__description("arena stack: jump to r12 += 16")
+__failure __msg("jump to the insn that changes R12")
+__naked void jump_to_adj(void)
+{
+ asm volatile (
+"1:"
+ ".8byte %[adj];"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 != 0 goto 1b;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+/* adds the index and what the context points to, through its frame */
+static __naked __noinline __used long loop_cb(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[stx16];" /* *(u64 *)(r12 - 16) = r1 */
+ "r4 = *(u64 *)(r2 + 0);"
+ ".8byte %[ldx16];" /* r5 = *(u64 *)(r12 - 16) */
+ "r4 += r5;"
+ "*(u64 *)(r2 + 0) = r4;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx16, STX(BPF_DW, -16, BPF_REG_1)),
+ __imm_insn(ldx16, LDX(BPF_DW, BPF_REG_5, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: r12 in a callback")
+__success __retval(145)
+__naked void callback(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 100 */
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 10;"
+ "r2 = loop_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ /* 0 + 1 + ... + 9 and what is in the frame of the main program */
+ "r0 = *(u64 *)(r10 - 8);"
+ ".8byte %[ldx];" /* r1 = *(u64 *)(r12 - 8) */
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -8, 100)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -8))
+ : __clobber_all);
+}
+
+/* has no frame: its first insn is the call, not r12 += N */
+static __naked __noinline __used long calls_loop_cb(void)
+{
+ asm volatile (
+ "call loop_cb;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * The main program names loop_cb() first, so libbpf puts it in front of the callback.
+ * The code that takes a stack moves the first insn of the callback down.
+ * The call has to find loop_cb() from there.
+ */
+SEC("syscall")
+__description("arena stack: a callback starts with a call of a function in front of it")
+__success __retval(145)
+__xlated("call pc-{{[0-9]+}}")
+__naked void callback_calls_backward(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 100;"
+ "r2 = r10;"
+ "r2 += -8;"
+ "call loop_cb;"
+ "r1 = 10;"
+ "r2 = calls_loop_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ /* 100 + 0 + 1 + ... + 9 */
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ :
+ : __imm(bpf_loop)
+ : __clobber_all);
+}
+
+/* runs loop_cb() twice. It takes a stack for callee_with_frame() */
+static __naked __noinline __used long loops_loop_cb(void)
+{
+ asm volatile (
+ "r6 = loop_cb ll;"
+ "r7 = r2;"
+ "call callee_with_frame;"
+ "r1 = 2;"
+ "r2 = r6;"
+ "r3 = r7;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_loop)
+ : __clobber_all);
+}
+
+/* the same for the address of a function */
+SEC("syscall")
+__description("arena stack: a callback starts with the address of a function in front of it")
+__success __retval(13)
+__naked void callback_names_backward(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ /* 0 + 1 + 2 + 3 + 4 */
+ "r1 = 5;"
+ "r2 = loop_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ /* 3 * (0 + 1) */
+ "r1 = 3;"
+ "r2 = loops_loop_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ :
+ : __imm(bpf_loop)
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: call of a callback that uses r12")
+__failure __msg("loop_cb() uses R12 and is both called and a callback")
+__naked void call_of_callback(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 1;"
+ "r2 = loop_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r1 = 1;"
+ "r2 = r10;"
+ "r2 += -8;"
+ "call loop_cb;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+#ifdef CAN_USE_CALLX
+SEC("syscall")
+__description("arena stack: callx of a callback that uses r12")
+__failure __msg("loop_cb() uses R12 and is both called and a callback")
+__naked void callx_of_callback(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 1;"
+ "r2 = loop_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r1 = 1;"
+ "r2 = r10;"
+ "r2 += -8;"
+ "r6 = loop_cb ll;"
+ "callx r6;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+#endif
+
+static __naked __noinline __used long drain_cb(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+/* Only a helper calls it, like the callback of bpf_loop(): it can take a stack. */
+SEC("syscall")
+__description("arena stack: r12 in a callback of bpf_user_ringbuf_drain()")
+__success __retval(0)
+__naked void callback_of_drain(void)
+{
+ asm volatile (
+ "r1 = %[user_ringbuf] ll;"
+ "r2 = drain_cb ll;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_user_ringbuf_drain];"
+ "exit;"
+ :
+ : __imm(bpf_user_ringbuf_drain),
+ __imm_addr(user_ringbuf)
+ : __clobber_all);
+}
+
+/* 200000 iterations: may_goto calls arch_bpf_timed_may_goto() three times */
+SEC("syscall")
+__description("arena stack: may_goto doesn't change r12")
+__success __retval(42)
+__naked void may_goto_keeps_r12(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 42 */
+ ".8byte %[mov6];" /* r6 = r12 */
+ "r7 = 0;"
+"1:"
+ ".8byte %[may_goto];"
+ "r7 += 1;"
+ "if r7 < 200000 goto 1b;"
+ "r0 = 1;"
+ "if r7 != 200000 goto 2f;"
+ "r0 = 2;"
+ ".8byte %[mov1];" /* r1 = r12 */
+ "if r1 != r6 goto 2f;"
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 8) */
+"2:"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(32)),
+ __imm_insn(st, ST(BPF_DW, -8, 42)),
+ __imm_insn(mov6, MOV(BPF_REG_6)),
+ __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 2, 0)),
+ __imm_insn(mov1, MOV(BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -8))
+ : __clobber_all);
+}
+
+/* counts the calls and spins until the time of may_goto is up */
+static __naked __noinline __used long spin_cb(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r3 = *(u64 *)(r2 + 0);"
+ "r3 += 1;"
+ "*(u64 *)(r2 + 0) = r3;"
+"1:"
+ ".8byte %[may_goto];"
+ "goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: bpf_loop() ends when the time of may_goto is up")
+__success __retval(121)
+/* the second call of the callback returns 1 before it takes a stack */
+__xlated("if r0 != 0x0 goto pc+2")
+__xlated("r0 = 1")
+__xlated("exit")
+__xlated("r6 = ")
+__xlated("r6 = &(void __percpu *)(r6)")
+__xlated("...")
+__xlated("w12 += 16")
+__naked void loop_ends_early(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 100 */
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = %[nr_loops];"
+ "r2 = spin_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ /* 2 calls, 1 of them ran */
+ "r0 *= 10;"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r0 += r1;"
+ ".8byte %[ldx];" /* r1 = *(u64 *)(r12 - 8) */
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_const(nr_loops, 1 << 23),
+ __imm_insn(adj, ADJ(32)),
+ __imm_insn(st, ST(BPF_DW, -8, 100)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -8))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena stack: tail call")
+__failure __msg("tail_calls are not allowed in programs that use R12")
+__naked void tail_call(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r2 = %[jmp_table] ll;"
+ "r3 = 0;"
+ "call %[bpf_tail_call];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena stack: ld_abs")
+__failure __msg("BPF_LD_[ABS|IND] is not allowed in programs that use R12")
+__naked void ld_abs(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r6 = r1;"
+ "r0 = *(u8 *)skb[0];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long frame_16(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long frame_16k_calls_16(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call frame_16;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long frame_16k(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384)),
+ __imm_insn(st, ST(BPF_DW, -16384, 4)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16384))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: 32K")
+__success __retval(4)
+__naked void depth_max(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call frame_16k;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: more than 32K")
+__failure __msg("combined size of the frames in the stack in arena is 32784. Too large")
+__naked void depth_too_large(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call frame_16k_calls_16;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: gotox to r12 += 16")
+__failure __msg("jump to the insn that changes R12")
+__naked void gotox_to_adj(void)
+{
+ asm volatile (
+ ".pushsection .jumptables,\"\",@progbits;"
+"jt_%=:"
+ ".quad out_%= - syscall;"
+ ".quad first_%= - syscall;"
+ ".size jt_%=, 16;"
+ ".global jt_%=;"
+ ".popsection;"
+"first_%=:"
+ ".8byte %[adj];"
+ /* r1 is ctx, then 0: the loop ends */
+ "if r1 == 0 goto out_%=;"
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 1;"
+ "r0 <<= 3;"
+ "r2 = jt_%= ll;"
+ "r2 += r0;"
+ "r2 = *(u64 *)(r2 + 0);"
+ "r1 = 0;"
+ ".8byte %[gotox];" /* gotox r2 */
+"out_%=:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(gotox, GOTOX(BPF_REG_2))
+ : __clobber_all);
+}
+
+/*
+ * The flag in the stack of the caller is 0, then 1: one jump to the first insn. The insns that
+ * take the stack are in front of it and use r6-r8.
+ */
+static __naked __noinline __used long gotox_first_insn_cb(void)
+{
+ asm volatile (
+ ".pushsection .jumptables,\"\",@progbits;"
+"jt_%=:"
+ ".quad first_%= - .text;"
+ ".size jt_%=, 8;"
+ ".global jt_%=;"
+ ".popsection;"
+"first_%=:"
+ "r3 = *(u64 *)(r2 + 0);"
+ "if r3 != 0 goto second_%=;"
+ "r3 = 1;"
+ "*(u64 *)(r2 + 0) = r3;"
+ "r6 = r2;"
+ "r7 = 10;"
+ "r8 = 100;"
+ ".8byte %[mov9];" /* r9 = r12 */
+ "r3 = jt_%= ll;"
+ "r3 = *(u64 *)(r3 + 0);"
+ ".8byte %[gotox];" /* gotox r3 */
+"second_%=:"
+ "r0 = *(u64 *)(r6 + 0);"
+ "r0 += r7;"
+ "r0 += r8;"
+ ".8byte %[mov3];" /* r3 = r12 */
+ "r3 -= r9;"
+ "r0 += r3;"
+ "*(u64 *)(r2 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(mov9, MOV(BPF_REG_9)),
+ __imm_insn(mov3, MOV(BPF_REG_3)),
+ __imm_insn(gotox, GOTOX(BPF_REG_3))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: gotox to the first insn of a callback")
+__success __retval(111)
+__naked void gotox_first_insn_of_callback(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 1;"
+ "r2 = gotox_first_insn_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ /* the flag through r6, r7, r8 and no change of r12: 1 + 10 + 100 + 0 */
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ :
+ : __imm(bpf_loop)
+ : __clobber_all);
+}
+
+/*
+ * The kernel's prologue is in front of the code that takes a stack. The exit in
+ * the prologue has no frame to drop and no stack to give back.
+ */
+SEC("tc")
+__description("arena stack: prologue of a prog that writes to the packet")
+__success __retval(7)
+__xlated("5: call unknown") /* bpf_skb_pull_data() */
+__xlated("6: if r0 == 0x0 goto pc+2")
+__xlated("7: w0 = 2")
+__xlated("8: exit")
+__xlated("9: r1 = r6")
+/* take a stack */
+__xlated("10: r2 = 0x")
+__xlated("12: r2 = &(void __percpu *)(r2)")
+__xlated("...")
+__xlated("r12 = *(u32 *)(r2 +8)")
+/* main prog */
+__xlated("w12 += 48")
+__xlated("r2 = *(u64 *)(r1 +")
+__naked void packet_write_prologue(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r2 = *(u32 *)(r1 + %[data]);"
+ "r3 = *(u32 *)(r1 + %[data_end]);"
+ "r4 = r2;"
+ "r4 += 1;"
+ "r0 = 1;"
+ "if r4 > r3 goto 1f;"
+ "r0 = 7;"
+ "*(u8 *)(r2 + 0) = r0;"
+ ".8byte %[stx];" /* *(u64 *)(r12 - 8) = r0 */
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 8) */
+"1:"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(48)),
+ __imm_insn(stx, STX(BPF_DW, -8, BPF_REG_0)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -8)),
+ __imm_const(data, offsetof(struct __sk_buff, data)),
+ __imm_const(data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 14/14] selftests/bpf: Add tests for BPF_REG_ARENA_SP
2026-11-04 1:24 ` [PATCH bpf-next v3 14/14] selftests/bpf: Add tests for BPF_REG_ARENA_SP Alexei Starovoitov
@ 2026-11-04 2:12 ` bot+bpf-ci
2026-11-04 2:29 ` Alexei Starovoitov
0 siblings, 1 reply; 32+ messages in thread
From: bot+bpf-ci @ 2026-11-04 2:12 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]
> selftests/bpf: Add tests for BPF_REG_ARENA_SP
TL;DR: Nit in selftests: the r12 instruction macros (ADJ, LDX, ST) are
duplicated across several arena_stack test files, some hand-encoded. Suggests
moving them to one shared header.
> diff --git a/tools/testing/selftests/bpf/progs/arena_stack_grow.c b/tools/testing/selftests/bpf/progs/arena_stack_grow.c
> new file mode 100644
> index 000000000000..be64296431c3
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/arena_stack_grow.c
[ ... ]
> +/* Compilers don't know r12 yet */
> +#define R12 12
> +#define ADJ(n) BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, R12, 0, 0, n)
> +#define LDX(dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEM | BPF_DW, dst, R12, off, 0)
> +#define ST(off, imm) BPF_RAW_INSN(BPF_ST | BPF_MEM | BPF_DW, R12, 0, off, imm)
This isn't a bug, but this block of r12 insn macros (R12, ADJ, LDX, ST) is
the same as the one in arena_stack.c, while arena_stack_loop.c and
arena_stack_nomem.c write the same encodings out by hand, e.g.
__imm_insn(adj, BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, 12, 0, 0, 16))
with a "r12 += 16" comment. verifier_arena_stack.c defines a third,
size-taking variant.
Could these macros live in one shared place, for example bpf_misc.h or a
small arena-stack header, so that arena_stack_loop.c and arena_stack_nomem.c
can use ADJ(16) like their neighbours, and the encodings can be dropped in
one spot once compilers emit r12?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/41755299448
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH bpf-next v3 14/14] selftests/bpf: Add tests for BPF_REG_ARENA_SP
2026-11-04 2:12 ` bot+bpf-ci
@ 2026-11-04 2:29 ` Alexei Starovoitov
0 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 2:29 UTC (permalink / raw)
To: bot+bpf-ci, bpf
Cc: daniel, andrii, eddyz87, memxor, martin.lau, yonghong.song, mason,
ihor.solodrai
On Tue, Nov 03, 2026 at 06:12 PM bot+bpf-ci@kernel.org <bot+bpf-ci@kernel.org> wrote:
> TL;DR: Nit in selftests: the r12 instruction macros (ADJ, LDX, ST) are
> duplicated across several arena_stack test files, some hand-encoded. Suggests
> moving them to one shared header.
Not worth a respin.
The macros are temporary. They go away when compilers know r12.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH bpf-next v3 00/14] bpf: A stack in arena
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (13 preceding siblings ...)
2026-11-04 1:24 ` [PATCH bpf-next v3 14/14] selftests/bpf: Add tests for BPF_REG_ARENA_SP Alexei Starovoitov
@ 2026-11-04 3:31 ` Alexei Starovoitov
2026-11-04 7:35 ` Kumar Kartikeya Dwivedi
2026-11-04 20:10 ` patchwork-bot+netdevbpf
15 siblings, 1 reply; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 3:31 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
On Tue, Nov 03, 2026 at 05:24 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> No released compiler emits this code. The change to the BPF backend of LLVM
> is in review: https://github.com/llvm/llvm-project/pull/235577
The flags are made for Rust. I tried them on C.
C without __arena works. The kernel side of this series is enough for it.
The LLVM side needs more work.
-- libarena without __arena --
'#define __arena' is empty. The compiler moves the globals, the heap and
the locals to arena.
63 libarena progs out of 63 pass the verifier.
The price against libarena with __arena: two times more casts, JITed code
+18%, arena_malloc() +7%.
So probably not a recommended approach, but works for "lazy" C developers.
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [PATCH bpf-next v3 00/14] bpf: A stack in arena
2026-11-04 3:31 ` [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
@ 2026-11-04 7:35 ` Kumar Kartikeya Dwivedi
2026-11-04 18:02 ` Alexei Starovoitov
0 siblings, 1 reply; 32+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-11-04 7:35 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87
On Wed Nov 4, 2026 at 4:31 AM CET, Alexei Starovoitov wrote:
> On Tue, Nov 03, 2026 at 05:24 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>> No released compiler emits this code. The change to the BPF backend of LLVM
>> is in review: https://github.com/llvm/llvm-project/pull/235577
>
> The flags are made for Rust. I tried them on C.
> C without __arena works. The kernel side of this series is enough for it.
> The LLVM side needs more work.
>
> -- libarena without __arena --
>
> '#define __arena' is empty. The compiler moves the globals, the heap and
> the locals to arena.
>
> 63 libarena progs out of 63 pass the verifier.
>
> The price against libarena with __arena: two times more casts, JITed code
> +18%, arena_malloc() +7%.
>
> So probably not a recommended approach, but works for "lazy" C developers.
Could you elaborate on this? Are the extra casts due to the fact that globals,
heaps, and locals on the stack are moving to arena memory? My understanding is
that 1-1, the compiler approach is still more efficient in terms of the casts
emitted (and 32-bit narrowing) vs BPF_F_ARENA_SCALAR.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH bpf-next v3 00/14] bpf: A stack in arena
2026-11-04 7:35 ` Kumar Kartikeya Dwivedi
@ 2026-11-04 18:02 ` Alexei Starovoitov
0 siblings, 0 replies; 32+ messages in thread
From: Alexei Starovoitov @ 2026-11-04 18:02 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, bpf; +Cc: daniel, andrii, eddyz87
On Tue, Nov 03, 2026 at 11:35 PM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
>> The price against libarena with __arena: two times more casts, JITed code
>> +18%, arena_malloc() +7%.
>
> Could you elaborate on this? Are the extra casts due to the fact that globals,
> heaps, and locals on the stack are moving to arena memory?
exactly.
> My understanding is
> that 1-1, the compiler approach is still more efficient in terms of the casts
> emitted (and 32-bit narrowing) vs BPF_F_ARENA_SCALAR.
Right. llvm adding casts is a clear win over BPF_F_ARENA_SCALAR.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH bpf-next v3 00/14] bpf: A stack in arena
2026-11-04 1:23 [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
` (14 preceding siblings ...)
2026-11-04 3:31 ` [PATCH bpf-next v3 00/14] bpf: A stack in arena Alexei Starovoitov
@ 2026-11-04 20:10 ` patchwork-bot+netdevbpf
15 siblings, 0 replies; 32+ messages in thread
From: patchwork-bot+netdevbpf @ 2026-11-04 20:10 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87, memxor
Hello:
This series was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <ast@kernel.org>:
On Wed, 4 Nov 2026 01:23:46 +0000 you wrote:
> From: Alexei Starovoitov <ast@kernel.org>
>
> The goal is to load Rust programs as rustc and LLVM compile them. The test
> programs are in https://github.com/4ast/rust-bpf
>
> This is part 2 of "bpf: Stack in arena and a verifier that joins states"
> [1], as a series of its own. Part 1 is merged. This series does not depend
> on part 3, and it does not change the main pass of the verifier.
>
> [...]
Here is the summary with links:
- [bpf-next,v3,01/14] libbpf: Make arena 4 Gbyte when max_entries is not set
https://git.kernel.org/bpf/bpf-next/c/a3405322d2f2
- [bpf-next,v3,02/14] selftests/bpf: Test the default size of arena
https://git.kernel.org/bpf/bpf-next/c/e7ef59089c74
- [bpf-next,v3,03/14] bpf: Treat addr_space_cast of a pointer that is not in arena as a move
https://git.kernel.org/bpf/bpf-next/c/8adaa3791632
- [bpf-next,v3,04/14] selftests/bpf: Test addr_space_cast of pointers that are not in arena
https://git.kernel.org/bpf/bpf-next/c/6a52d60c0932
- [bpf-next,v3,05/14] libbpf: Support .bss.addr_space.1 section
https://git.kernel.org/bpf/bpf-next/c/f9420eb3626e
- [bpf-next,v3,06/14] libbpf: Relocate pointers that are stored in arena
https://git.kernel.org/bpf/bpf-next/c/32d56ccb1d30
- [bpf-next,v3,07/14] selftests/bpf: Test pointers in arena and .bss.addr_space.1
https://git.kernel.org/bpf/bpf-next/c/66c4a3b9bf82
- [bpf-next,v3,08/14] bpf: Introduce bpf register BPF_REG_ARENA_SP
https://git.kernel.org/bpf/bpf-next/c/51a641232390
- [bpf-next,v3,09/14] bpf: Add stacks for programs to arena
https://git.kernel.org/bpf/bpf-next/c/df664dbd6e98
- [bpf-next,v3,10/14] bpf, x86: Add BPF_ATOMIC_LOCAL
https://git.kernel.org/bpf/bpf-next/c/f02597ebe9dc
- [bpf-next,v3,11/14] bpf: Support BPF_REG_ARENA_SP in the verifier
https://git.kernel.org/bpf/bpf-next/c/c3c27a5b489b
- [bpf-next,v3,12/14] bpf, x86: Support BPF_REG_ARENA_SP
https://git.kernel.org/bpf/bpf-next/c/1298d494a71d
- [bpf-next,v3,13/14] bpf, arm64: Support BPF_REG_ARENA_SP
https://git.kernel.org/bpf/bpf-next/c/9eb00cb851d2
- [bpf-next,v3,14/14] selftests/bpf: Add tests for BPF_REG_ARENA_SP
https://git.kernel.org/bpf/bpf-next/c/cf8e78c1073e
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 32+ messages in thread