* [PATCH bpf-next v2 01/11] bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
@ 2026-04-05 16:53 ` Yonghong Song
2026-04-05 16:53 ` [PATCH bpf-next v2 02/11] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments Yonghong Song
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:53 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
The newly-added register BPF_REG_STACK_ARG_BASE corresponds to bpf
register R12 added by [1]. R12 is used as the base for stack arguments
so it won't mess out R10 based stacks.
[1] https://github.com/llvm/llvm-project/pull/189060
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
include/linux/filter.h | 3 ++-
kernel/bpf/core.c | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index e40d4071a345..68f018dd4b9c 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -59,7 +59,8 @@ struct ctl_table_header;
/* Kernel hidden auxiliary/helper register. */
#define BPF_REG_AX MAX_BPF_REG
-#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1)
+#define BPF_REG_STACK_ARG_BASE (MAX_BPF_REG + 1)
+#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)
#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG
/* unused opcode to mark special call to bpf_tail_call() helper */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 1af5fb3f21d9..3520337a1c0e 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
u32 imm_rnd = get_random_u32();
s16 off;
- BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
- BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG);
+ BUILD_BUG_ON(BPF_REG_AX + 2 != MAX_BPF_JIT_REG);
+ BUILD_BUG_ON(BPF_REG_STACK_ARG_BASE + 1 != MAX_BPF_JIT_REG);
/* Constraints on AX register:
*
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 02/11] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
2026-04-05 16:53 ` [PATCH bpf-next v2 01/11] bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE Yonghong Song
@ 2026-04-05 16:53 ` Yonghong Song
2026-04-05 16:53 ` [PATCH bpf-next v2 03/11] bpf: Support stack arguments for bpf functions Yonghong Song
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:53 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Currently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where
the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.
Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments
for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS
is 12 which should be sufficient for majority of bpf functions and
kfunc's.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
include/linux/bpf.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 35b1e25bd104..99701c4cc34e 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
+ *
+ * Also reuse this macro for maximum number of arguments a BPF function
+ * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
+ * stack arg slots.
*/
#define MAX_BPF_FUNC_ARGS 12
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 03/11] bpf: Support stack arguments for bpf functions
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
2026-04-05 16:53 ` [PATCH bpf-next v2 01/11] bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE Yonghong Song
2026-04-05 16:53 ` [PATCH bpf-next v2 02/11] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments Yonghong Song
@ 2026-04-05 16:53 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 04/11] bpf: Refactor process_iter_arg() to have proper argument index Yonghong Song
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:53 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Currently BPF functions (subprogs) are limited to 5 register arguments.
With [1], the compiler can emit code that passes additional arguments
via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.
The following is an example to show how stack arguments are saved
and transferred between caller and callee:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar(a1, a2, a3, a4, a5, a6, a7, a8);
...
}
The following is a illustration of stack allocation:
Caller (foo) Callee (bar)
============ ============
r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
The caller writes outgoing args past its own incoming area.
At the call site, the verifier transfers the caller's outgoing
slots into the callee's incoming slots.
The verifier tracks stack arg slots separately from the regular r10
stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack
slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated
'stack_arg_slots' array in bpf_func_state. This separation keeps the
stack arg area from interfering with the normal stack and frame pointer
(r10) bookkeeping. Similar to stacksafe(), introduce stack_arg_safe()
to do pruning check.
If the bpf function has more than one calls, e.g.,
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
bar1(a1, a2, a3, a4, a5, a6, a7, a8);
...
bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}
The following is an illustration:
Caller (foo) Callee (bar1)
============ =============
r12-relative stack arg area: r12-relative stack arg area:
r12-8: [incoming arg 6] +--> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
r12-16: [incoming arg 7] |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
---- incoming/outgoing boundary ||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+|| ...
r12-32: [outgoing arg 7 to callee]-+|
r12-40: [outgoing arg 8 to callee]--+
...
Back from bar1
... Callee (bar2)
=== =============
+---> r12-8: [incoming arg 6] (from caller's outgoing r12-24)
|+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
|||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
---- incoming/outgoing boundary |||| ---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]+||| ...
r12-32: [outgoing arg 7 to callee]-+||
r12-40: [outgoing arg 8 to callee]--+|
r12-48: [outgoing arg 9 to callee]---+
Global subprogs with >5 args are not yet supported.
[1] https://github.com/llvm/llvm-project/pull/189060
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
include/linux/bpf.h | 2 +
include/linux/bpf_verifier.h | 31 +++-
kernel/bpf/btf.c | 14 +-
kernel/bpf/verifier.c | 311 +++++++++++++++++++++++++++++++++--
4 files changed, 334 insertions(+), 24 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 99701c4cc34e..72f990054d07 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
u32 max_pkt_offset;
u32 max_tp_access;
u32 stack_depth;
+ u16 incoming_stack_arg_depth;
+ u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
u32 id;
u32 func_cnt; /* used by non-func prog as the number of func progs */
u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 36bfd96d4563..75fe67baa833 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -268,6 +268,11 @@ struct bpf_retval_range {
bool return_32bit;
};
+struct bpf_stack_arg_state {
+ struct bpf_reg_state spilled_ptr; /* for spilled scalar/pointer semantics */
+ u8 slot_type[BPF_REG_SIZE];
+};
+
/* state of the program:
* type of all registers and stack info
*/
@@ -319,6 +324,10 @@ struct bpf_func_state {
* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
*/
int allocated_stack;
+
+ u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+ u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+ struct bpf_stack_arg_state *stack_arg_slots;
};
#define MAX_CALL_FRAMES 8
@@ -456,6 +465,17 @@ struct bpf_verifier_state {
iter < frame->allocated_stack / BPF_REG_SIZE; \
iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
+#define bpf_get_spilled_stack_arg(slot, frame, mask) \
+ (((slot < frame->stack_arg_depth / BPF_REG_SIZE) && \
+ ((1 << frame->stack_arg_slots[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \
+ ? &frame->stack_arg_slots[slot].spilled_ptr : NULL)
+
+/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
+#define bpf_for_each_spilled_stack_arg(iter, frame, reg, mask) \
+ for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame, mask); \
+ iter < frame->stack_arg_depth / BPF_REG_SIZE; \
+ iter++, reg = bpf_get_spilled_stack_arg(iter, frame, mask))
+
#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr) \
({ \
struct bpf_verifier_state *___vstate = __vst; \
@@ -473,6 +493,11 @@ struct bpf_verifier_state {
continue; \
(void)(__expr); \
} \
+ bpf_for_each_spilled_stack_arg(___j, __state, __reg, __mask) { \
+ if (!__reg) \
+ continue; \
+ (void)(__expr); \
+ } \
} \
})
@@ -686,10 +711,12 @@ struct bpf_subprog_info {
bool keep_fastcall_stack: 1;
bool changes_pkt_data: 1;
bool might_sleep: 1;
- u8 arg_cnt:3;
+ u8 arg_cnt:4;
enum priv_stack_mode priv_stack_mode;
- struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
+ struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+ u16 incoming_stack_arg_depth;
+ u16 outgoing_stack_arg_depth;
};
struct bpf_verifier_env;
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index a62d78581207..c5f3aa05d5a3 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
}
args = (const struct btf_param *)(t + 1);
nargs = btf_type_vlen(t);
- if (nargs > MAX_BPF_FUNC_REG_ARGS) {
- if (!is_global)
- return -EINVAL;
- bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
+ if (nargs > MAX_BPF_FUNC_ARGS) {
+ bpf_log(log, "Function %s() with %d > %d args not supported.\n",
+ tname, nargs, MAX_BPF_FUNC_ARGS);
+ return -EINVAL;
+ }
+ if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
+ bpf_log(log, "Global function %s() with %d > %d args not supported.\n",
tname, nargs, MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
}
+ if (nargs > MAX_BPF_FUNC_REG_ARGS)
+ sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
/* check that function is void or returns int, exception cb also requires this */
t = btf_type_by_id(btf, t->type);
while (btf_type_is_modifier(t))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 84699a428077..ebf164036849 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1477,6 +1477,19 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st
return -ENOMEM;
dst->allocated_stack = src->allocated_stack;
+
+ /* copy stack_arg_slots state */
+ n = src->stack_arg_depth / BPF_REG_SIZE;
+ if (n) {
+ dst->stack_arg_slots = copy_array(dst->stack_arg_slots, src->stack_arg_slots, n,
+ sizeof(struct bpf_stack_arg_state),
+ GFP_KERNEL_ACCOUNT);
+ if (!dst->stack_arg_slots)
+ return -ENOMEM;
+
+ dst->stack_arg_depth = src->stack_arg_depth;
+ dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth;
+ }
return 0;
}
@@ -1518,6 +1531,25 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state
return 0;
}
+static int grow_stack_arg_slots(struct bpf_verifier_env *env,
+ struct bpf_func_state *state, int size)
+{
+ size_t old_n = state->stack_arg_depth / BPF_REG_SIZE, n;
+
+ size = round_up(size, BPF_REG_SIZE);
+ n = size / BPF_REG_SIZE;
+ if (old_n >= n)
+ return 0;
+
+ state->stack_arg_slots = realloc_array(state->stack_arg_slots, old_n, n,
+ sizeof(struct bpf_stack_arg_state));
+ if (!state->stack_arg_slots)
+ return -ENOMEM;
+
+ state->stack_arg_depth = size;
+ return 0;
+}
+
/* Acquire a pointer id from the env and update the state->refs to include
* this new pointer reference.
* On success, returns a valid pointer id to associate with the register
@@ -1688,6 +1720,7 @@ static void free_func_state(struct bpf_func_state *state)
{
if (!state)
return;
+ kfree(state->stack_arg_slots);
kfree(state->stack);
kfree(state);
}
@@ -5934,6 +5967,107 @@ static int check_stack_write(struct bpf_verifier_env *env,
return err;
}
+/* Validate that a stack arg access is 8-byte sized and aligned. */
+static int check_stack_arg_access(struct bpf_verifier_env *env,
+ struct bpf_insn *insn, const char *op)
+{
+ int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
+
+ if (size != BPF_REG_SIZE) {
+ verbose(env, "stack arg %s must be %d bytes, got %d\n",
+ op, BPF_REG_SIZE, size);
+ return -EINVAL;
+ }
+ if (insn->off % BPF_REG_SIZE) {
+ verbose(env, "stack arg %s offset %d not aligned to %d\n",
+ op, insn->off, BPF_REG_SIZE);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+/* Check that a stack arg slot has been properly initialized. */
+static bool is_stack_arg_slot_initialized(struct bpf_func_state *state, int spi)
+{
+ u8 type;
+
+ if (spi >= (int)(state->stack_arg_depth / BPF_REG_SIZE))
+ return false;
+ type = state->stack_arg_slots[spi].slot_type[BPF_REG_SIZE - 1];
+ return type == STACK_SPILL || type == STACK_MISC;
+}
+
+/*
+ * Write a value to the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
+ int off, int value_regno)
+{
+ int spi = (-off - 1) / BPF_REG_SIZE;
+ struct bpf_subprog_info *subprog;
+ struct bpf_func_state *cur;
+ struct bpf_reg_state *reg;
+ int i, err;
+ u8 type;
+
+ err = grow_stack_arg_slots(env, state, -off);
+ if (err)
+ return err;
+
+ /* Ensure the JIT allocates space for the stack arg area. */
+ subprog = &env->subprog_info[state->subprogno];
+ if (-off > subprog->outgoing_stack_arg_depth)
+ subprog->outgoing_stack_arg_depth = -off;
+
+ cur = env->cur_state->frame[env->cur_state->curframe];
+ if (value_regno >= 0) {
+ reg = &cur->regs[value_regno];
+ state->stack_arg_slots[spi].spilled_ptr = *reg;
+ type = is_spillable_regtype(reg->type) ? STACK_SPILL : STACK_MISC;
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ state->stack_arg_slots[spi].slot_type[i] = type;
+ } else {
+ /* BPF_ST: store immediate, treat as scalar */
+ reg = &state->stack_arg_slots[spi].spilled_ptr;
+ reg->type = SCALAR_VALUE;
+ __mark_reg_known(reg, (u32)env->prog->insnsi[env->insn_idx].imm);
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ state->stack_arg_slots[spi].slot_type[i] = STACK_MISC;
+ }
+ return 0;
+}
+
+/*
+ * Read a value from the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,
+ int off, int dst_regno)
+{
+ int spi = (-off - 1) / BPF_REG_SIZE;
+ struct bpf_func_state *cur;
+ u8 *stype;
+
+ if (-off > state->stack_arg_depth) {
+ verbose(env, "invalid read from stack arg off %d depth %d\n",
+ off, state->stack_arg_depth);
+ return -EACCES;
+ }
+
+ stype = state->stack_arg_slots[spi].slot_type;
+ cur = env->cur_state->frame[env->cur_state->curframe];
+
+ if (stype[BPF_REG_SIZE - 1] == STACK_SPILL)
+ copy_register_state(&cur->regs[dst_regno],
+ &state->stack_arg_slots[spi].spilled_ptr);
+ else
+ mark_reg_unknown(env, cur->regs, dst_regno);
+ return 0;
+}
+
static int check_map_access_type(struct bpf_verifier_env *env, u32 regno,
int off, int size, enum bpf_access_type type)
{
@@ -8109,10 +8243,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
bool strict_alignment_once, bool is_ldsx,
bool allow_trust_mismatch, const char *ctx)
{
+ struct bpf_verifier_state *vstate = env->cur_state;
+ struct bpf_func_state *state = vstate->frame[vstate->curframe];
struct bpf_reg_state *regs = cur_regs(env);
enum bpf_reg_type src_reg_type;
int err;
+ /* Handle stack arg access */
+ if (insn->src_reg == BPF_REG_STACK_ARG_BASE) {
+ err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
+ if (err)
+ return err;
+ err = check_stack_arg_access(env, insn, "read");
+ if (err)
+ return err;
+ return check_stack_arg_read(env, state, insn->off, insn->dst_reg);
+ }
+
/* check src operand */
err = check_reg_arg(env, insn->src_reg, SRC_OP);
if (err)
@@ -8141,10 +8288,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
bool strict_alignment_once)
{
+ struct bpf_verifier_state *vstate = env->cur_state;
+ struct bpf_func_state *state = vstate->frame[vstate->curframe];
struct bpf_reg_state *regs = cur_regs(env);
enum bpf_reg_type dst_reg_type;
int err;
+ /* Handle stack arg write */
+ if (insn->dst_reg == BPF_REG_STACK_ARG_BASE) {
+ err = check_reg_arg(env, insn->src_reg, SRC_OP);
+ if (err)
+ return err;
+ err = check_stack_arg_access(env, insn, "write");
+ if (err)
+ return err;
+ return check_stack_arg_write(env, state, insn->off, insn->src_reg);
+ }
+
/* check src1 operand */
err = check_reg_arg(env, insn->src_reg, SRC_OP);
if (err)
@@ -11027,8 +11187,10 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
int *insn_idx)
{
struct bpf_verifier_state *state = env->cur_state;
+ struct bpf_subprog_info *caller_info;
struct bpf_func_state *caller;
int err, subprog, target_insn;
+ u16 callee_incoming;
target_insn = *insn_idx + insn->imm + 1;
subprog = bpf_find_subprog(env, target_insn);
@@ -11080,6 +11242,15 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
return 0;
}
+ /*
+ * Track caller's outgoing stack arg depth (max across all callees).
+ * This is needed so the JIT knows how much stack arg space to allocate.
+ */
+ caller_info = &env->subprog_info[caller->subprogno];
+ callee_incoming = env->subprog_info[subprog].incoming_stack_arg_depth;
+ if (callee_incoming > caller_info->outgoing_stack_arg_depth)
+ caller_info->outgoing_stack_arg_depth = callee_incoming;
+
/* for regular function entry setup new frame and continue
* from that frame.
*/
@@ -11135,13 +11306,41 @@ static int set_callee_state(struct bpf_verifier_env *env,
struct bpf_func_state *caller,
struct bpf_func_state *callee, int insn_idx)
{
- int i;
+ struct bpf_subprog_info *callee_info;
+ int i, err;
/* copy r1 - r5 args that callee can access. The copy includes parent
* pointers, which connects us up to the liveness chain
*/
for (i = BPF_REG_1; i <= BPF_REG_5; i++)
callee->regs[i] = caller->regs[i];
+
+ /*
+ * Transfer stack args from caller's outgoing area to callee's incoming area.
+ * Caller wrote outgoing args at offsets '-(incoming + 8)', '-(incoming + 16)', ...
+ * These outgoing args will go to callee's incoming area.
+ */
+ callee_info = &env->subprog_info[callee->subprogno];
+ if (callee_info->incoming_stack_arg_depth) {
+ int caller_incoming_slots = caller->incoming_stack_arg_depth / BPF_REG_SIZE;
+ int callee_incoming_slots = callee_info->incoming_stack_arg_depth / BPF_REG_SIZE;
+
+ callee->incoming_stack_arg_depth = callee_info->incoming_stack_arg_depth;
+ err = grow_stack_arg_slots(env, callee, callee_info->incoming_stack_arg_depth);
+ if (err)
+ return err;
+
+ for (i = 0; i < callee_incoming_slots; i++) {
+ int caller_spi = i + caller_incoming_slots;
+
+ if (!is_stack_arg_slot_initialized(caller, caller_spi)) {
+ verbose(env, "stack arg#%d not properly initialized\n",
+ i + MAX_BPF_FUNC_REG_ARGS);
+ return -EINVAL;
+ }
+ callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi];
+ }
+ }
return 0;
}
@@ -20535,6 +20734,56 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
return true;
}
+/*
+ * Compare stack arg slots between old and current states.
+ * Only incoming stack args need comparison — outgoing slots are transient
+ * (written before each call, consumed at the call site) so they don't carry
+ * meaningful state across pruning points.
+ */
+static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
+ struct bpf_func_state *cur, struct bpf_idmap *idmap,
+ enum exact_level exact)
+{
+ int i, spi;
+
+ if (old->incoming_stack_arg_depth != cur->incoming_stack_arg_depth)
+ return false;
+
+ for (i = 0; i < old->incoming_stack_arg_depth; i++) {
+ spi = i / BPF_REG_SIZE;
+
+ if (exact == EXACT &&
+ old->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE] !=
+ cur->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE])
+ return false;
+
+ if (old->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID)
+ continue;
+
+ if (old->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE] !=
+ cur->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE])
+ return false;
+
+ if (i % BPF_REG_SIZE != BPF_REG_SIZE - 1)
+ continue;
+
+ switch (old->stack_arg_slots[spi].slot_type[BPF_REG_SIZE - 1]) {
+ case STACK_SPILL:
+ if (!regsafe(env, &old->stack_arg_slots[spi].spilled_ptr,
+ &cur->stack_arg_slots[spi].spilled_ptr, idmap, exact))
+ return false;
+ break;
+ case STACK_MISC:
+ case STACK_ZERO:
+ case STACK_INVALID:
+ continue;
+ default:
+ return false;
+ }
+ }
+ return true;
+}
+
static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,
struct bpf_idmap *idmap)
{
@@ -20626,6 +20875,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
return false;
+ if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
+ return false;
+
return true;
}
@@ -21534,23 +21786,37 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
verbose(env, "BPF_ST uses reserved fields\n");
return -EINVAL;
}
- /* check src operand */
- err = check_reg_arg(env, insn->dst_reg, SRC_OP);
- if (err)
- return err;
- dst_reg_type = cur_regs(env)[insn->dst_reg].type;
+ /* Handle stack arg write (store immediate) */
+ if (insn->dst_reg == BPF_REG_STACK_ARG_BASE) {
+ struct bpf_verifier_state *vstate = env->cur_state;
+ struct bpf_func_state *state = vstate->frame[vstate->curframe];
- /* check that memory (dst_reg + off) is writeable */
- err = check_mem_access(env, env->insn_idx, insn->dst_reg,
- insn->off, BPF_SIZE(insn->code),
- BPF_WRITE, -1, false, false);
- if (err)
- return err;
+ err = check_stack_arg_access(env, insn, "write");
+ if (err)
+ return err;
+ err = check_stack_arg_write(env, state, insn->off, -1);
+ if (err)
+ return err;
+ } else {
+ /* check src operand */
+ err = check_reg_arg(env, insn->dst_reg, SRC_OP);
+ if (err)
+ return err;
- err = save_aux_ptr_type(env, dst_reg_type, false);
- if (err)
- return err;
+ dst_reg_type = cur_regs(env)[insn->dst_reg].type;
+
+ /* check that memory (dst_reg + off) is writeable */
+ err = check_mem_access(env, env->insn_idx, insn->dst_reg,
+ insn->off, BPF_SIZE(insn->code),
+ BPF_WRITE, -1, false, false);
+ if (err)
+ return err;
+
+ err = save_aux_ptr_type(env, dst_reg_type, false);
+ if (err)
+ return err;
+ }
} else if (class == BPF_JMP || class == BPF_JMP32) {
u8 opcode = BPF_OP(insn->code);
@@ -22246,11 +22512,11 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)
return err;
for (i = 0; i < insn_cnt; i++, insn++) {
- if (insn->dst_reg >= MAX_BPF_REG) {
+ if (insn->dst_reg >= MAX_BPF_REG && insn->dst_reg != BPF_REG_STACK_ARG_BASE) {
verbose(env, "R%d is invalid\n", insn->dst_reg);
return -EINVAL;
}
- if (insn->src_reg >= MAX_BPF_REG) {
+ if (insn->src_reg >= MAX_BPF_REG && insn->src_reg != BPF_REG_STACK_ARG_BASE) {
verbose(env, "R%d is invalid\n", insn->src_reg);
return -EINVAL;
}
@@ -23275,8 +23541,14 @@ static int jit_subprogs(struct bpf_verifier_env *env)
int err, num_exentries;
int old_len, subprog_start_adjustment = 0;
- if (env->subprog_cnt <= 1)
+ if (env->subprog_cnt <= 1) {
+ /*
+ * Even without subprogs, kfunc calls with >5 args need stack arg space
+ * allocated by the root program.
+ */
+ prog->aux->stack_arg_depth = env->subprog_info[0].outgoing_stack_arg_depth;
return 0;
+ }
for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
@@ -23366,6 +23638,9 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->name[0] = 'F';
func[i]->aux->stack_depth = env->subprog_info[i].stack_depth;
+ func[i]->aux->incoming_stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth;
+ func[i]->aux->stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth +
+ env->subprog_info[i].outgoing_stack_arg_depth;
if (env->subprog_info[i].priv_stack_mode == PRIV_STACK_ADAPTIVE)
func[i]->aux->jits_use_priv_stack = true;
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 04/11] bpf: Refactor process_iter_arg() to have proper argument index
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (2 preceding siblings ...)
2026-04-05 16:53 ` [PATCH bpf-next v2 03/11] bpf: Support stack arguments for bpf functions Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 05/11] bpf: Support stack arguments for kfunc calls Yonghong Song
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
In the next patch for kfunc stack arguments, a faked register is
used to do proper verification checking. For process_iter_arg(),
the regno is passed in and the iterator assumes 'regno - 1' as
the argument index. This is wrong as regno is fake. So refactor
process_iter_arg() by adding actual argument index which is used
inside the function.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/verifier.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ebf164036849..fd13309d9006 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -9259,14 +9259,14 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx,
}
static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idx,
- struct bpf_kfunc_call_arg_meta *meta)
+ int argno, struct bpf_kfunc_call_arg_meta *meta)
{
struct bpf_reg_state *reg = reg_state(env, regno);
const struct btf_type *t;
int spi, err, i, nr_slots, btf_id;
if (reg->type != PTR_TO_STACK) {
- verbose(env, "arg#%d expected pointer to an iterator on stack\n", regno - 1);
+ verbose(env, "arg#%d expected pointer to an iterator on stack\n", argno);
return -EINVAL;
}
@@ -9276,9 +9276,9 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
* to any kfunc, if arg has "__iter" suffix, we need to be a bit more
* conservative here.
*/
- btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, regno - 1);
+ btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, argno);
if (btf_id < 0) {
- verbose(env, "expected valid iter pointer as arg #%d\n", regno - 1);
+ verbose(env, "expected valid iter pointer as arg #%d\n", argno);
return -EINVAL;
}
t = btf_type_by_id(meta->btf, btf_id);
@@ -9288,7 +9288,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
/* bpf_iter_<type>_new() expects pointer to uninit iter state */
if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) {
verbose(env, "expected uninitialized iter_%s as arg #%d\n",
- iter_type_str(meta->btf, btf_id), regno - 1);
+ iter_type_str(meta->btf, btf_id), argno);
return -EINVAL;
}
@@ -9312,7 +9312,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
break;
case -EINVAL:
verbose(env, "expected an initialized iter_%s as arg #%d\n",
- iter_type_str(meta->btf, btf_id), regno - 1);
+ iter_type_str(meta->btf, btf_id), argno);
return err;
case -EPROTO:
verbose(env, "expected an RCU CS when using %s\n", meta->func_name);
@@ -14063,7 +14063,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
return -EINVAL;
}
}
- ret = process_iter_arg(env, regno, insn_idx, meta);
+ ret = process_iter_arg(env, regno, insn_idx, i, meta);
if (ret < 0)
return ret;
break;
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 05/11] bpf: Support stack arguments for kfunc calls
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (3 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 04/11] bpf: Refactor process_iter_arg() to have proper argument index Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 06/11] bpf: Reject stack arguments in non-JITed programs Yonghong Song
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Extend the stack argument mechanism to kfunc calls, allowing kfuncs
with more than 5 parameters to receive additional arguments via the
r12-based stack arg area.
For kfuncs, the caller is a BPF program and the callee is a kernel
function. The BPF program writes outgoing args at r12-relative offsets
past its own incoming area.
The following is an example to show how stack arguments are saved:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
...
kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}
The following is an illustration:
Caller (foo)
============
r12-relative stack arg area:
r12-8: [incoming arg 6]
r12-16: [incoming arg 7]
---- incoming/outgoing boundary (kfunc1)
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
...
Back from kfunc1
...
---- incoming/outgoing boundary
r12-24: [outgoing arg 6 to callee]
r12-32: [outgoing arg 7 to callee]
r12-40: [outgoing arg 8 to callee]
r12-48: [outgoing arg 9 to callee]
Later JIT will marshal outgoing arguments to the native calling convention
for kfunc1() and kfunc2().
In check_kfunc_args(), for args beyond the 5th, retrieve the spilled
register state from the caller's stack arg slots. Temporarily copy
it into regs[BPF_REG_1] to reuse the existing type checking
infrastructure, then restore after checking. The following is one
of examples based on a later selftest:
13: (85) call bpf_kfunc_call_stack_arg_mem#152105
Use reg 1 for stack arg#5
Use reg 2 to represent mem_size
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2
mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2
End of using reg 1 for stack arg#6
14: R0=scalar()
14: (95) exit
The above example is for KF_ARG_PTR_TO_MEM_SIZE case.
Registers 1 and 2 are used as an temporary register for argument
checking. The verifier log will identify when the temporary register
is used and when the temporary register is not used any more.
There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno'
is also keep regno for later fixup. Since regno is a faked one
such three cases are rejected for now if they are in stack arguments.
If possible, new kfuncs could keep them in first 5 registers so
there are no issues at all.
If faked register approach is not favored, the following is an
alternative approach with the following data structure:
struct reg_or_arg_t {
struct bpf_reg_state *state;
union {
int regno;
int argno;
};
bool is_reg; /* distinguish between reg and arg */
};
Such a struct can replace existing regno/argno/reg_state parameters
to make it easy to distinguish reg vs. arg.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/verifier.c | 143 ++++++++++++++++++++++++++++++++++++------
1 file changed, 124 insertions(+), 19 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fd13309d9006..f19398f628ba 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3491,7 +3491,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
struct bpf_kfunc_meta kfunc;
struct bpf_kfunc_desc *desc;
unsigned long addr;
- int err;
+ int i, err;
prog_aux = env->prog->aux;
tab = prog_aux->kfunc_tab;
@@ -3567,6 +3567,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
if (err)
return err;
+ for (i = MAX_BPF_FUNC_REG_ARGS; i < func_model.nr_args; i++) {
+ if (func_model.arg_size[i] > sizeof(u64)) {
+ verbose(env, "kfunc %s arg#%d size %d > %zu not supported for stack args\n",
+ kfunc.name, i, func_model.arg_size[i], sizeof(u64));
+ return -EINVAL;
+ }
+ }
+
desc = &tab->descs[tab->nr_descs++];
desc->func_id = func_id;
desc->offset = offset;
@@ -13083,6 +13091,19 @@ static bool is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)
return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data];
}
+static struct bpf_reg_state *get_kfunc_arg_reg(struct bpf_verifier_env *env, int argno)
+{
+ struct bpf_func_state *caller;
+ int spi;
+
+ if (argno < MAX_BPF_FUNC_REG_ARGS)
+ return &cur_regs(env)[argno + 1];
+
+ caller = cur_func(env);
+ spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE + (argno - MAX_BPF_FUNC_REG_ARGS);
+ return &caller->stack_arg_slots[spi].spilled_ptr;
+}
+
static enum kfunc_ptr_arg_type
get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
struct bpf_kfunc_call_arg_meta *meta,
@@ -13101,8 +13122,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
return KF_ARG_PTR_TO_CTX;
if (argno + 1 < nargs &&
- (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]) ||
- is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1])))
+ (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) ||
+ is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1))))
arg_mem_size = true;
/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -13770,9 +13791,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
args = (const struct btf_param *)(meta->func_proto + 1);
nargs = btf_type_vlen(meta->func_proto);
- if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+ if (nargs > MAX_BPF_FUNC_ARGS) {
verbose(env, "Function %s has %d > %d args\n", func_name, nargs,
- MAX_BPF_FUNC_REG_ARGS);
+ MAX_BPF_FUNC_ARGS);
return -EINVAL;
}
@@ -13780,26 +13801,59 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
* verifier sees.
*/
for (i = 0; i < nargs; i++) {
- struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[i + 1];
+ struct bpf_reg_state *regs = cur_regs(env), *reg;
+ struct bpf_reg_state saved_reg;
const struct btf_type *t, *ref_t, *resolve_ret;
enum bpf_arg_type arg_type = ARG_DONTCARE;
u32 regno = i + 1, ref_id, type_size;
bool is_ret_buf_sz = false;
+ bool is_stack_arg = false;
int kf_arg_type;
+ if (i < MAX_BPF_FUNC_REG_ARGS) {
+ reg = ®s[i + 1];
+ } else {
+ /*
+ * Retrieve the spilled reg state from the stack arg slot.
+ * Reuse the existing type checking infrastructure which
+ * reads from cur_regs(env)[regno], temporarily copy the
+ * stack arg reg state into regs[BPF_REG_1] and restore
+ * it after checking.
+ */
+ struct bpf_func_state *caller = cur_func(env);
+ int spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE +
+ (i - MAX_BPF_FUNC_REG_ARGS);
+
+ if (!is_stack_arg_slot_initialized(caller, spi)) {
+ verbose(env, "stack arg#%d not properly initialized\n", i);
+ return -EINVAL;
+ }
+
+ is_stack_arg = true;
+ regno = BPF_REG_1;
+ saved_reg = regs[BPF_REG_1];
+ regs[BPF_REG_1] = caller->stack_arg_slots[spi].spilled_ptr;
+ reg = ®s[BPF_REG_1];
+ verbose(env, "Use reg %d for stack arg#%d\n", regno, i);
+ }
+
if (is_kfunc_arg_prog_aux(btf, &args[i])) {
/* Reject repeated use bpf_prog_aux */
if (meta->arg_prog) {
verifier_bug(env, "Only 1 prog->aux argument supported per-kfunc");
return -EFAULT;
}
+ if (is_stack_arg) {
+ verbose(env, "arg#%d prog->aux cannot be a stack argument\n", i);
+ return -EINVAL;
+ }
meta->arg_prog = true;
cur_aux(env)->arg_prog = regno;
- continue;
+ goto next_arg;
}
if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
- continue;
+ goto next_arg;
t = btf_type_skip_modifiers(btf, args[i].type, NULL);
@@ -13818,9 +13872,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
verbose(env, "R%d must be a known constant\n", regno);
return -EINVAL;
}
- ret = mark_chain_precision(env, regno);
- if (ret < 0)
- return ret;
+ if (!is_stack_arg) {
+ ret = mark_chain_precision(env, regno);
+ if (ret < 0)
+ return ret;
+ }
meta->arg_constant.found = true;
meta->arg_constant.value = reg->var_off.value;
} else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) {
@@ -13842,11 +13898,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
}
meta->r0_size = reg->var_off.value;
- ret = mark_chain_precision(env, regno);
- if (ret)
- return ret;
+ if (!is_stack_arg) {
+ ret = mark_chain_precision(env, regno);
+ if (ret)
+ return ret;
+ }
}
- continue;
+ goto next_arg;
}
if (!btf_type_is_ptr(t)) {
@@ -13868,8 +13926,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
return -EFAULT;
}
meta->ref_obj_id = reg->ref_obj_id;
- if (is_kfunc_release(meta))
+ if (is_kfunc_release(meta)) {
+ if (is_stack_arg) {
+ verbose(env, "arg#%d release arg cannot be a stack argument\n", i);
+ return -EINVAL;
+ }
meta->release_regno = regno;
+ }
}
ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
@@ -13881,7 +13944,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
switch (kf_arg_type) {
case KF_ARG_PTR_TO_NULL:
- continue;
+ goto next_arg;
case KF_ARG_PTR_TO_MAP:
if (!reg->map_ptr) {
verbose(env, "pointer in R%d isn't map pointer\n", regno);
@@ -14020,6 +14083,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
dynptr_arg_type |= DYNPTR_TYPE_FILE;
} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) {
dynptr_arg_type |= DYNPTR_TYPE_FILE;
+ if (is_stack_arg) {
+ verbose(env, "arg#%d release arg cannot be a stack argument\n", i);
+ return -EINVAL;
+ }
meta->release_regno = regno;
} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] &&
(dynptr_arg_type & MEM_UNINIT)) {
@@ -14169,8 +14236,18 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
{
struct bpf_reg_state *buff_reg = ®s[regno];
const struct btf_param *buff_arg = &args[i];
- struct bpf_reg_state *size_reg = ®s[regno + 1];
+ struct bpf_reg_state *size_reg;
const struct btf_param *size_arg = &args[i + 1];
+ struct bpf_reg_state saved_size_reg = {};
+ bool size_is_stack_arg = false;
+
+ if (i >= MAX_BPF_FUNC_REG_ARGS) {
+ size_is_stack_arg = true;
+ saved_size_reg = regs[regno + 1];
+ regs[regno + 1] = *get_kfunc_arg_reg(env, i + 1);
+ verbose(env, "Use reg %d to represent mem_size\n", regno + 1);
+ }
+ size_reg = ®s[regno + 1];
if (!register_is_null(buff_reg) || !is_kfunc_arg_nullable(meta->btf, buff_arg)) {
ret = check_kfunc_mem_size_reg(env, size_reg, regno + 1);
@@ -14193,6 +14270,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
meta->arg_constant.value = size_reg->var_off.value;
}
+ if (size_is_stack_arg)
+ regs[regno + 1] = saved_size_reg;
+
/* Skip next '__sz' or '__szk' argument */
i++;
break;
@@ -14294,6 +14374,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
break;
}
}
+next_arg:
+ if (is_stack_arg) {
+ regs[regno] = saved_reg;
+ verbose(env, "End of using reg %d for stack arg#%d\n", regno, i);
+ }
}
if (is_kfunc_release(meta) && !meta->release_regno) {
@@ -15059,7 +15144,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
nargs = btf_type_vlen(meta.func_proto);
args = (const struct btf_param *)(meta.func_proto + 1);
- for (i = 0; i < nargs; i++) {
+ for (i = 0; i < nargs && i < MAX_BPF_FUNC_REG_ARGS; i++) {
u32 regno = i + 1;
t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL);
@@ -15070,6 +15155,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
mark_btf_func_reg_size(env, regno, t->size);
}
+ /* Track outgoing stack arg depth for kfuncs with >5 args */
+ if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+ struct bpf_func_state *caller = cur_func(env);
+ struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
+ u16 kfunc_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
+ if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth)
+ caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth;
+ }
+
if (is_iter_next_kfunc(&meta)) {
err = process_iter_next_call(env, insn_idx, &meta);
if (err)
@@ -23975,6 +24070,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (!bpf_jit_supports_far_kfunc_call())
insn->imm = BPF_CALL_IMM(desc->addr);
+ /*
+ * After resolving the kfunc address, insn->off is no longer needed
+ * for BTF fd index. Repurpose it to store the number of stack args
+ * so the JIT can marshal them.
+ */
+ if (desc->func_model.nr_args > MAX_BPF_FUNC_REG_ARGS)
+ insn->off = desc->func_model.nr_args - MAX_BPF_FUNC_REG_ARGS;
+ else
+ insn->off = 0;
+
if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;
struct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) };
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 06/11] bpf: Reject stack arguments in non-JITed programs
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (4 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 05/11] bpf: Support stack arguments for kfunc calls Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 07/11] bpf: Enable stack argument support for x86_64 Yonghong Song
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
The interpreter does not understand the bpf register r12
(BPF_REG_STACK_ARG_BASE) used for stack argument addressing. So
reject interpreter usage if stack arguments are used either
in the main program or any subprogram.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/core.c | 3 ++-
kernel/bpf/verifier.c | 6 ++++++
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 3520337a1c0e..a04b31eb4c49 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2553,7 +2553,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
goto finalize;
if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
- bpf_prog_has_kfunc_call(fp))
+ bpf_prog_has_kfunc_call(fp) ||
+ fp->aux->stack_arg_depth)
jit_needed = true;
if (!bpf_prog_select_interpreter(fp))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f19398f628ba..010ef7fc6c72 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -23944,6 +23944,12 @@ static int fixup_call_args(struct bpf_verifier_env *env)
verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
return -EINVAL;
}
+ for (i = 0; i < env->subprog_cnt; i++) {
+ if (env->subprog_info[i].incoming_stack_arg_depth) {
+ verbose(env, "stack args are not supported in non-JITed programs\n");
+ return -EINVAL;
+ }
+ }
if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
/* When JIT fails the progs with bpf2bpf calls and tail_calls
* have to be rejected, since interpreter doesn't support them yet.
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 07/11] bpf: Enable stack argument support for x86_64
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (5 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 06/11] bpf: Reject stack arguments in non-JITed programs Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 08/11] bpf,x86: Implement JIT support for stack arguments Yonghong Song
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Add stack argument support for x86_64.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
arch/x86/net/bpf_jit_comp.c | 5 +++++
include/linux/filter.h | 1 +
kernel/bpf/btf.c | 9 ++++++++-
kernel/bpf/core.c | 5 +++++
4 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index e9b78040d703..32864dbc2c4e 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
+bool bpf_jit_supports_stack_args(void)
+{
+ return true;
+}
+
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
{
if (text_poke_copy(dst, src, len) == NULL)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 68f018dd4b9c..a5035fb80a6b 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1160,6 +1160,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
bool bpf_jit_supports_ptr_xchg(void);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index c5f3aa05d5a3..1cbe0f2b0e41 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -20,6 +20,7 @@
#include <linux/btf.h>
#include <linux/btf_ids.h>
#include <linux/bpf.h>
+#include <linux/filter.h>
#include <linux/bpf_lsm.h>
#include <linux/skmsg.h>
#include <linux/perf_event.h>
@@ -7897,8 +7898,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
tname, nargs, MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
}
- if (nargs > MAX_BPF_FUNC_REG_ARGS)
+ if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+ if (!bpf_jit_supports_stack_args()) {
+ bpf_log(log, "JIT does not support function %s() with %d args\n",
+ tname, nargs);
+ return -ENOTSUPP;
+ }
sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+ }
/* check that function is void or returns int, exception cb also requires this */
t = btf_type_by_id(btf, t->type);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index a04b31eb4c49..01de4f7f3a82 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3156,6 +3156,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
return false;
}
+bool __weak bpf_jit_supports_stack_args(void)
+{
+ return false;
+}
+
bool __weak bpf_jit_supports_far_kfunc_call(void)
{
return false;
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 08/11] bpf,x86: Implement JIT support for stack arguments
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (6 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 07/11] bpf: Enable stack argument support for x86_64 Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 09/11] selftests/bpf: Add tests for BPF function " Yonghong Song
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Add x86_64 JIT support for BPF functions and kfuncs with more than
5 arguments. The extra arguments are passed through a stack area
addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode,
which the JIT translates to RBP-relative accesses in native code.
The JIT follows the native x86_64 calling convention for stack
argument placement. Incoming stack args from the caller sit above
the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly
where x86_64 expects them after CALL + PUSH RBP. Only the outgoing
stack arg area is allocated below the program stack in the prologue.
The native x86_64 stack layout for a function with incoming and
outgoing stack args:
high address
┌─────────────────────────┐
│ incoming stack arg N │ [rbp + 16 + (N-1)*8] (from caller)
│ ... │
│ incoming stack arg 1 │ [rbp + 16]
├─────────────────────────┤
│ return address │ [rbp + 8]
│ saved rbp │ [rbp]
├─────────────────────────┤
│ BPF program stack │ (stack_depth bytes)
├─────────────────────────┤
│ outgoing stack arg 1 │ [rbp - prog_stack_depth - outgoing_depth]
│ ... │ (written via r12-relative STX/ST)
│ outgoing stack arg M │ [rbp - prog_stack_depth - 8]
├─────────────────────────┤
│ callee-saved regs ... │ (pushed after sub rsp)
└─────────────────────────┘ rsp
low address
BPF r12-relative offsets are translated to native RBP-relative
offsets with two formulas:
- Incoming args (load: -off <= incoming_depth):
native_off = 8 - bpf_off → [rbp + 16 + ...]
- Outgoing args (store: -off > incoming_depth):
native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
Since callee-saved registers are pushed below the outgoing area,
outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF
calls and kfunc calls, outgoing args are explicitly pushed from the
outgoing area onto the stack before CALL and rsp is restored after return.
For kfunc calls specifically, arg 6 is loaded into R9 and args 7+
are pushed onto the native stack, per the x86_64 calling convention.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
1 file changed, 129 insertions(+), 6 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 32864dbc2c4e..206f342a0ca0 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
*pprog = prog;
}
+/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
+static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
+{
+ u8 *prog = *pprog;
+ int k, bytes = 0;
+ s32 off;
+
+ for (k = from; k >= to; k--) {
+ off = outgoing_base + (k - 1) * 8;
+ /* push qword [rbp + off] */
+ if (is_imm8(off)) {
+ EMIT3(0xFF, 0x75, off);
+ bytes += 3;
+ } else {
+ EMIT2_off32(0xFF, 0xB5, off);
+ bytes += 6;
+ }
+ }
+ *pprog = prog;
+ return bytes;
+}
+
static void emit_nops(u8 **pprog, int len)
{
u8 *prog = *pprog;
@@ -1664,16 +1686,33 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
int i, excnt = 0;
int ilen, proglen = 0;
u8 *prog = temp;
- u32 stack_depth;
+ u16 stack_arg_depth, incoming_stack_arg_depth, outgoing_stack_arg_depth;
+ u32 prog_stack_depth, stack_depth;
+ bool has_stack_args;
int err;
stack_depth = bpf_prog->aux->stack_depth;
+ stack_arg_depth = bpf_prog->aux->stack_arg_depth;
+ incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth;
+ outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth;
priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
if (priv_stack_ptr) {
priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
stack_depth = 0;
}
+ /*
+ * Save program stack depth before adding outgoing stack arg space.
+ * Incoming stack args are read directly from [rbp + 16 + ...].
+ * Only the outgoing stack arg area is allocated below the
+ * program stack. Outgoing args written here become the callee's
+ * incoming args.
+ */
+ prog_stack_depth = round_up(stack_depth, 8);
+ if (outgoing_stack_arg_depth)
+ stack_depth += outgoing_stack_arg_depth;
+ has_stack_args = stack_arg_depth > 0;
+
arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
@@ -1715,13 +1754,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
prog = temp;
for (i = 1; i <= insn_cnt; i++, insn++) {
+ bool adjust_stack_arg_off = false;
const s32 imm32 = insn->imm;
u32 dst_reg = insn->dst_reg;
u32 src_reg = insn->src_reg;
u8 b2 = 0, b3 = 0;
u8 *start_of_ldx;
s64 jmp_offset;
- s16 insn_off;
+ s32 insn_off;
u8 jmp_cond;
u8 *func;
int nops;
@@ -1734,6 +1774,37 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
dst_reg = X86_REG_R9;
}
+ if (has_stack_args) {
+ u8 class = BPF_CLASS(insn->code);
+
+ if (class == BPF_LDX &&
+ src_reg == BPF_REG_STACK_ARG_BASE) {
+ src_reg = BPF_REG_FP;
+ adjust_stack_arg_off = true;
+ }
+ if ((class == BPF_STX || class == BPF_ST) &&
+ dst_reg == BPF_REG_STACK_ARG_BASE) {
+ dst_reg = BPF_REG_FP;
+ adjust_stack_arg_off = true;
+ }
+ }
+
+ /*
+ * Translate BPF r12-relative offset to native RBP-relative:
+ *
+ * Incoming args (load: offset >= -incoming_depth):
+ * BPF: r12 + bpf_off = r12 - k * 8 (k = 1,2,...) for incoming arg k
+ * Native: [rbp + 8 + k * 8]
+ * Formula: native_off = 8 + k * 8 = 8 - bpf_off
+ *
+ * Outgoing args (store: offset < -incoming_depth):
+ * BPF: r12 + bpf_off = r12 - (incoming + k * 8) for outgoing arg k
+ * Native: [rbp - prog_stack_depth - outgoing + (k - 1) * 8]
+ * Formula: native_off = -(prog_stack_depth + outgoing) + (k - 1) * 8
+ * = -(prog_stack_depth + outgoing + incoming + 8) - bpf_off
+ * = -(prog_stack_depth + stack_arg_depth + 8) - bpf_off
+ */
+
switch (insn->code) {
/* ALU */
case BPF_ALU | BPF_ADD | BPF_X:
@@ -2131,10 +2202,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
case BPF_ST | BPF_MEM | BPF_DW:
EMIT2(add_1mod(0x48, dst_reg), 0xC7);
-st: if (is_imm8(insn->off))
- EMIT2(add_1reg(0x40, dst_reg), insn->off);
+st: insn_off = insn->off;
+ if (adjust_stack_arg_off)
+ insn_off = -(prog_stack_depth + stack_arg_depth + 8) - insn_off;
+ if (is_imm8(insn_off))
+ EMIT2(add_1reg(0x40, dst_reg), insn_off);
else
- EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);
+ EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
break;
@@ -2144,7 +2218,10 @@ st: if (is_imm8(insn->off))
case BPF_STX | BPF_MEM | BPF_H:
case BPF_STX | BPF_MEM | BPF_W:
case BPF_STX | BPF_MEM | BPF_DW:
- emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+ insn_off = insn->off;
+ if (adjust_stack_arg_off)
+ insn_off = -(prog_stack_depth + stack_arg_depth + 8) - insn_off;
+ emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
break;
case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
@@ -2243,6 +2320,8 @@ st: if (is_imm8(insn->off))
case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
insn_off = insn->off;
+ if (adjust_stack_arg_off)
+ insn_off = 8 - insn_off;
if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2441,6 +2520,7 @@ st: if (is_imm8(insn->off))
/* call */
case BPF_JMP | BPF_CALL: {
u8 *ip = image + addrs[i - 1];
+ int stack_args = 0;
func = (u8 *) __bpf_call_base + imm32;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
@@ -2449,6 +2529,41 @@ st: if (is_imm8(insn->off))
}
if (!imm32)
return -EINVAL;
+
+ if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
+ /*
+ * BPF-to-BPF calls: push outgoing stack args from
+ * the outgoing area onto the stack before CALL.
+ * The outgoing area is at [rbp - prog_stack - outgoing],
+ * but rsp is below that due to callee-saved reg pushes,
+ * so we must explicitly push args for the callee.
+ */
+ s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
+ int n_args = outgoing_stack_arg_depth / 8;
+
+ ip += push_stack_args(&prog, outgoing_base, n_args, 1);
+ }
+
+ if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
+ /* Kfunc calls: arg 6 → R9, args 7+ → push. */
+ s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
+ int kfunc_stack_args = insn->off;
+
+ stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
+
+ /* Push args 7+ in reverse order */
+ if (stack_args > 0)
+ ip += push_stack_args(&prog, outgoing_base, kfunc_stack_args, 2);
+
+ /* mov r9, [rbp + outgoing_base] (arg 6) */
+ if (is_imm8(outgoing_base)) {
+ EMIT4(0x4C, 0x8B, 0x4D, outgoing_base);
+ ip += 4;
+ } else {
+ EMIT3_off32(0x4C, 0x8B, 0x8D, outgoing_base);
+ ip += 7;
+ }
+ }
if (priv_frame_ptr) {
push_r9(&prog);
ip += 2;
@@ -2458,6 +2573,14 @@ st: if (is_imm8(insn->off))
return -EINVAL;
if (priv_frame_ptr)
pop_r9(&prog);
+ if (stack_args > 0) {
+ /* add rsp, stack_args * 8 */
+ EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
+ }
+ if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
+ /* add rsp, outgoing_stack_arg_depth */
+ EMIT4(0x48, 0x83, 0xC4, outgoing_stack_arg_depth);
+ }
break;
}
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 09/11] selftests/bpf: Add tests for BPF function stack arguments
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (7 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 08/11] bpf,x86: Implement JIT support for stack arguments Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 10/11] selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument Yonghong Song
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Add selftests covering stack argument passing for both BPF-to-BPF
subprog calls and kfunc calls with more than 5 arguments. All tests
are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
.../selftests/bpf/prog_tests/stack_arg.c | 132 +++++++++++
tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++++++++++++
.../selftests/bpf/progs/stack_arg_kfunc.c | 164 ++++++++++++++
.../selftests/bpf/test_kmods/bpf_testmod.c | 65 ++++++
.../bpf/test_kmods/bpf_testmod_kfunc.h | 20 +-
5 files changed, 592 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
diff --git a/tools/testing/selftests/bpf/prog_tests/stack_arg.c b/tools/testing/selftests/bpf/prog_tests/stack_arg.c
new file mode 100644
index 000000000000..1af5e5c91e0d
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/stack_arg.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "stack_arg.skel.h"
+#include "stack_arg_kfunc.skel.h"
+
+static void run_subtest(struct bpf_program *prog, int expected)
+{
+ int err, prog_fd;
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .data_in = &pkt_v4,
+ .data_size_in = sizeof(pkt_v4),
+ .repeat = 1,
+ );
+
+ prog_fd = bpf_program__fd(prog);
+ err = bpf_prog_test_run_opts(prog_fd, &topts);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(topts.retval, expected, "retval");
+}
+
+static void test_global_many(void)
+{
+ struct stack_arg *skel;
+
+ skel = stack_arg__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+
+ if (!skel->rodata->has_stack_arg) {
+ test__skip();
+ goto out;
+ }
+
+ if (!ASSERT_OK(stack_arg__load(skel), "load"))
+ goto out;
+
+ run_subtest(skel->progs.test_global_many_args, 36);
+
+out:
+ stack_arg__destroy(skel);
+}
+
+static void test_async_cb_many(void)
+{
+ struct stack_arg *skel;
+
+ skel = stack_arg__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+
+ if (!skel->rodata->has_stack_arg) {
+ test__skip();
+ goto out;
+ }
+
+ if (!ASSERT_OK(stack_arg__load(skel), "load"))
+ goto out;
+
+ run_subtest(skel->progs.test_async_cb_many_args, 0);
+
+out:
+ stack_arg__destroy(skel);
+}
+
+static void test_bpf2bpf(void)
+{
+ struct stack_arg *skel;
+
+ skel = stack_arg__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+
+ if (!skel->rodata->has_stack_arg) {
+ test__skip();
+ goto out;
+ }
+
+ if (!ASSERT_OK(stack_arg__load(skel), "load"))
+ goto out;
+
+ run_subtest(skel->progs.test_bpf2bpf_ptr_stack_arg, 45);
+ run_subtest(skel->progs.test_bpf2bpf_mix_stack_args, 51);
+ run_subtest(skel->progs.test_bpf2bpf_nesting_stack_arg, 50);
+ run_subtest(skel->progs.test_bpf2bpf_dynptr_stack_arg, 69);
+
+out:
+ stack_arg__destroy(skel);
+}
+
+static void test_kfunc(void)
+{
+ struct stack_arg_kfunc *skel;
+
+ skel = stack_arg_kfunc__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+
+ if (!skel->rodata->has_stack_arg) {
+ test__skip();
+ goto out;
+ }
+
+ if (!ASSERT_OK(stack_arg_kfunc__load(skel), "load"))
+ goto out;
+
+ run_subtest(skel->progs.test_stack_arg_scalar, 36);
+ run_subtest(skel->progs.test_stack_arg_ptr, 45);
+ run_subtest(skel->progs.test_stack_arg_mix, 51);
+ run_subtest(skel->progs.test_stack_arg_dynptr, 69);
+ run_subtest(skel->progs.test_stack_arg_mem, 151);
+ run_subtest(skel->progs.test_stack_arg_iter, 115);
+ run_subtest(skel->progs.test_stack_arg_const_str, 15);
+ run_subtest(skel->progs.test_stack_arg_timer, 15);
+
+out:
+ stack_arg_kfunc__destroy(skel);
+}
+
+void test_stack_arg(void)
+{
+ if (test__start_subtest("global_many_args"))
+ test_global_many();
+ if (test__start_subtest("async_cb_many_args"))
+ test_async_cb_many();
+ if (test__start_subtest("bpf2bpf"))
+ test_bpf2bpf();
+ if (test__start_subtest("kfunc"))
+ test_kfunc();
+}
diff --git a/tools/testing/selftests/bpf/progs/stack_arg.c b/tools/testing/selftests/bpf/progs/stack_arg.c
new file mode 100644
index 000000000000..c6bf89c3a0db
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/stack_arg.c
@@ -0,0 +1,212 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <stdbool.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_kfuncs.h"
+
+#define CLOCK_MONOTONIC 1
+
+long a, b, c, d, e, f, g, i;
+
+struct timer_elem {
+ struct bpf_timer timer;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct timer_elem);
+} timer_map SEC(".maps");
+
+int timer_result;
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+__noinline static int static_func_many_args(int a, int b, int c, int d,
+ int e, int f, int g, int h)
+{
+ return a + b + c + d + e + f + g + h;
+}
+
+__noinline int global_calls_many_args(int a, int b, int c)
+{
+ return static_func_many_args(a, b, c, 4, 5, 6, 7, 8);
+}
+
+SEC("tc")
+int test_global_many_args(void)
+{
+ return global_calls_many_args(1, 2, 3);
+}
+
+struct test_data {
+ long x;
+ long y;
+};
+
+/* 1 + 2 + 3 + 4 + 5 + 10 + 20 = 45 */
+__noinline static long func_with_ptr_stack_arg(long a, long b, long c, long d,
+ long e, struct test_data *p)
+{
+ return a + b + c + d + e + p->x + p->y;
+}
+
+__noinline long global_ptr_stack_arg(long a, long b, long c, long d, long e)
+{
+ struct test_data data = { .x = 10, .y = 20 };
+
+ return func_with_ptr_stack_arg(a, b, c, d, e, &data);
+}
+
+SEC("tc")
+int test_bpf2bpf_ptr_stack_arg(void)
+{
+ return global_ptr_stack_arg(1, 2, 3, 4, 5);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + 10 + 6 + 20 = 51 */
+__noinline static long func_with_mix_stack_args(long a, long b, long c, long d,
+ long e, struct test_data *p,
+ long f, struct test_data *q)
+{
+ return a + b + c + d + e + p->x + f + q->y;
+}
+
+__noinline long global_mix_stack_args(long a, long b, long c, long d, long e)
+{
+ struct test_data p = { .x = 10 };
+ struct test_data q = { .y = 20 };
+
+ return func_with_mix_stack_args(a, b, c, d, e, &p, e + 1, &q);
+}
+
+SEC("tc")
+int test_bpf2bpf_mix_stack_args(void)
+{
+ return global_mix_stack_args(1, 2, 3, 4, 5);
+}
+
+/*
+ * Nesting test: func_outer calls func_inner, both with struct pointer
+ * as stack arg.
+ *
+ * func_inner: (a+1) + (b+1) + (c+1) + (d+1) + (e+1) + p->x + p->y
+ * = 2 + 3 + 4 + 5 + 6 + 10 + 20 = 50
+ */
+__noinline static long func_inner_ptr(long a, long b, long c, long d,
+ long e, struct test_data *p)
+{
+ return a + b + c + d + e + p->x + p->y;
+}
+
+__noinline static long func_outer_ptr(long a, long b, long c, long d,
+ long e, struct test_data *p)
+{
+ return func_inner_ptr(a + 1, b + 1, c + 1, d + 1, e + 1, p);
+}
+
+__noinline long global_nesting_ptr(long a, long b, long c, long d, long e)
+{
+ struct test_data data = { .x = 10, .y = 20 };
+
+ return func_outer_ptr(a, b, c, d, e, &data);
+}
+
+SEC("tc")
+int test_bpf2bpf_nesting_stack_arg(void)
+{
+ return global_nesting_ptr(1, 2, 3, 4, 5);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + sizeof(pkt_v4) = 15 + 54 = 69 */
+__noinline static long func_with_dynptr(long a, long b, long c, long d,
+ long e, struct bpf_dynptr *ptr)
+{
+ return a + b + c + d + e + bpf_dynptr_size(ptr);
+}
+
+__noinline long global_dynptr_stack_arg(void *ctx __arg_ctx, long a, long b,
+ long c, long d)
+{
+ struct bpf_dynptr ptr;
+
+ bpf_dynptr_from_skb(ctx, 0, &ptr);
+ return func_with_dynptr(a, b, c, d, d + 1, &ptr);
+}
+
+SEC("tc")
+int test_bpf2bpf_dynptr_stack_arg(struct __sk_buff *skb)
+{
+ return global_dynptr_stack_arg(skb, 1, 2, 3, 4);
+}
+
+static int timer_cb_many_args(void *map, int *key, struct bpf_timer *timer)
+{
+ timer_result = static_func_many_args(10, 20, 30, 40, 50, 60, 70, 80);
+ return 0;
+}
+
+SEC("tc")
+int test_async_cb_many_args(void)
+{
+ struct timer_elem *elem;
+ int key = 0;
+
+ elem = bpf_map_lookup_elem(&timer_map, &key);
+ if (!elem)
+ return -1;
+
+ bpf_timer_init(&elem->timer, &timer_map, CLOCK_MONOTONIC);
+ bpf_timer_set_callback(&elem->timer, timer_cb_many_args);
+ bpf_timer_start(&elem->timer, 1, 0);
+ return 0;
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_global_many_args(void)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_ptr_stack_arg(void)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_mix_stack_args(void)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_nesting_stack_arg(void)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_dynptr_stack_arg(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_async_cb_many_args(void)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/stack_arg_kfunc.c b/tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
new file mode 100644
index 000000000000..6cc404d57863
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
@@ -0,0 +1,164 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_kfuncs.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+struct bpf_iter_testmod_seq {
+ u64 :64;
+ u64 :64;
+};
+
+extern int bpf_iter_testmod_seq_new(struct bpf_iter_testmod_seq *it, s64 value, int cnt) __ksym;
+extern int *bpf_iter_testmod_seq_next(struct bpf_iter_testmod_seq *it) __ksym;
+extern void bpf_iter_testmod_seq_destroy(struct bpf_iter_testmod_seq *it) __ksym;
+
+struct timer_map_value {
+ struct bpf_timer timer;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct timer_map_value);
+} kfunc_timer_map SEC(".maps");
+
+SEC("tc")
+int test_stack_arg_scalar(struct __sk_buff *skb)
+{
+ return bpf_kfunc_call_stack_arg(1, 2, 3, 4, 5, 6, 7, 8);
+}
+
+SEC("tc")
+int test_stack_arg_ptr(struct __sk_buff *skb)
+{
+ struct prog_test_pass1 p = { .x0 = 10, .x1 = 20 };
+
+ return bpf_kfunc_call_stack_arg_ptr(1, 2, 3, 4, 5, &p);
+}
+
+SEC("tc")
+int test_stack_arg_mix(struct __sk_buff *skb)
+{
+ struct prog_test_pass1 p = { .x0 = 10 };
+ struct prog_test_pass1 q = { .x1 = 20 };
+
+ return bpf_kfunc_call_stack_arg_mix(1, 2, 3, 4, 5, &p, 6, &q);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + sizeof(pkt_v4) = 15 + 54 = 69 */
+SEC("tc")
+int test_stack_arg_dynptr(struct __sk_buff *skb)
+{
+ struct bpf_dynptr ptr;
+
+ bpf_dynptr_from_skb(skb, 0, &ptr);
+ return bpf_kfunc_call_stack_arg_dynptr(1, 2, 3, 4, 5, &ptr);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + (1 + 2 + ... + 16) = 15 + 136 = 151 */
+SEC("tc")
+int test_stack_arg_mem(struct __sk_buff *skb)
+{
+ char buf[16] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16};
+
+ return bpf_kfunc_call_stack_arg_mem(1, 2, 3, 4, 5, buf, sizeof(buf));
+}
+
+/* 1 + 2 + 3 + 4 + 5 + 100 = 115 */
+SEC("tc")
+int test_stack_arg_iter(struct __sk_buff *skb)
+{
+ struct bpf_iter_testmod_seq it;
+ u64 ret;
+
+ bpf_iter_testmod_seq_new(&it, 100, 10);
+ ret = bpf_kfunc_call_stack_arg_iter(1, 2, 3, 4, 5, &it);
+ bpf_iter_testmod_seq_destroy(&it);
+ return ret;
+}
+
+const char cstr[] = "hello";
+
+/* 1 + 2 + 3 + 4 + 5 = 15 */
+SEC("tc")
+int test_stack_arg_const_str(struct __sk_buff *skb)
+{
+ return bpf_kfunc_call_stack_arg_const_str(1, 2, 3, 4, 5, cstr);
+}
+
+/* 1 + 2 + 3 + 4 + 5 = 15 */
+SEC("tc")
+int test_stack_arg_timer(struct __sk_buff *skb)
+{
+ struct timer_map_value *val;
+ int key = 0;
+
+ val = bpf_map_lookup_elem(&kfunc_timer_map, &key);
+ if (!val)
+ return 0;
+ return bpf_kfunc_call_stack_arg_timer(1, 2, 3, 4, 5, &val->timer);
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_stack_arg_scalar(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_stack_arg_ptr(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_stack_arg_mix(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_stack_arg_dynptr(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_stack_arg_mem(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_stack_arg_iter(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_stack_arg_const_str(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+SEC("tc")
+int test_stack_arg_timer(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 061356f10093..4fcdb13e4a9a 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -824,6 +824,63 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
return 0;
}
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
+ u64 e, u64 f, u64 g, u64 h)
+{
+ return a + b + c + d + e + f + g + h;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_ptr(u64 a, u64 b, u64 c, u64 d, u64 e,
+ struct prog_test_pass1 *p)
+{
+ return a + b + c + d + e + p->x0 + p->x1;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_mix(u64 a, u64 b, u64 c, u64 d, u64 e,
+ struct prog_test_pass1 *p, u64 f,
+ struct prog_test_pass1 *q)
+{
+ return a + b + c + d + e + p->x0 + f + q->x1;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_dynptr(u64 a, u64 b, u64 c, u64 d, u64 e,
+ struct bpf_dynptr *ptr)
+{
+ const struct bpf_dynptr_kern *kern_ptr = (void *)ptr;
+
+ return a + b + c + d + e + (kern_ptr->size & 0xFFFFFF);
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_mem(u64 a, u64 b, u64 c, u64 d, u64 e,
+ void *mem, int mem__sz)
+{
+ const unsigned char *p = mem;
+ u64 sum = a + b + c + d + e;
+ int i;
+
+ for (i = 0; i < mem__sz; i++)
+ sum += p[i];
+ return sum;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_iter(u64 a, u64 b, u64 c, u64 d, u64 e,
+ struct bpf_iter_testmod_seq *it__iter)
+{
+ return a + b + c + d + e + it__iter->value;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_const_str(u64 a, u64 b, u64 c, u64 d, u64 e,
+ const char *str__str)
+{
+ return a + b + c + d + e;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 e,
+ struct bpf_timer *timer)
+{
+ return a + b + c + d + e;
+}
+
static struct prog_test_ref_kfunc prog_test_struct = {
.a = 42,
.b = 108,
@@ -1287,6 +1344,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_dynptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index aa0b8d41e71b..2c1cb118f886 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -26,6 +26,8 @@ struct prog_test_ref_kfunc {
};
#endif
+struct bpf_iter_testmod_seq;
+
struct prog_test_pass1 {
int x0;
struct {
@@ -111,7 +113,23 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
-
+__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
+ __u64 e, __u64 f, __u64 g, __u64 h) __ksym;
+__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ struct prog_test_pass1 *p) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ struct prog_test_pass1 *p, __u64 f,
+ struct prog_test_pass1 *q) __ksym;
+__u64 bpf_kfunc_call_stack_arg_dynptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ struct bpf_dynptr *ptr) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mem(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ void *mem, int mem__sz) __ksym;
+__u64 bpf_kfunc_call_stack_arg_iter(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ struct bpf_iter_testmod_seq *it__iter) __ksym;
+__u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ const char *str__str) __ksym;
+__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ struct bpf_timer *timer) __ksym;
void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 10/11] selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (8 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 09/11] selftests/bpf: Add tests for BPF function " Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 16:54 ` [PATCH bpf-next v2 11/11] selftests/bpf: Add verifier tests for stack argument validation Yonghong Song
2026-04-05 17:08 ` [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Add a test that the verifier rejects kfunc calls where a stack argument
exceeds 8 bytes (the register-sized slot limit).
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
.../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++
.../selftests/bpf/progs/stack_arg_fail.c | 32 +++++++++++++++++++
.../selftests/bpf/test_kmods/bpf_testmod.c | 7 ++++
.../bpf/test_kmods/bpf_testmod_kfunc.h | 8 +++++
4 files changed, 71 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
diff --git a/tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c b/tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
new file mode 100644
index 000000000000..328a79edee45
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include "stack_arg_fail.skel.h"
+
+void test_stack_arg_fail(void)
+{
+ struct stack_arg_fail *skel;
+
+ skel = stack_arg_fail__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+
+ if (!skel->rodata->has_stack_arg) {
+ test__skip();
+ goto out;
+ }
+
+ ASSERT_ERR(stack_arg_fail__load(skel), "load_should_fail");
+
+out:
+ stack_arg_fail__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/stack_arg_fail.c b/tools/testing/selftests/bpf/progs/stack_arg_fail.c
new file mode 100644
index 000000000000..caa63b6f6a80
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/stack_arg_fail.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+ struct prog_test_big_arg s = { .a = 1, .b = 2 };
+
+ return bpf_kfunc_call_stack_arg_big(1, 2, 3, 4, 5, s);
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 4fcdb13e4a9a..861c564cc330 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -881,6 +881,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 e
return a + b + c + d + e;
}
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e,
+ struct prog_test_big_arg s)
+{
+ return a + b + c + d + e + s.a + s.b;
+}
+
static struct prog_test_ref_kfunc prog_test_struct = {
.a = 42,
.b = 108,
@@ -1352,6 +1358,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 2c1cb118f886..2a40f80b074a 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -50,6 +50,11 @@ struct prog_test_pass2 {
} x;
};
+struct prog_test_big_arg {
+ long a;
+ long b;
+};
+
struct prog_test_fail1 {
void *p;
int x;
@@ -130,6 +135,9 @@ __u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u
const char *str__str) __ksym;
__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
struct bpf_timer *timer) __ksym;
+__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+ struct prog_test_big_arg s) __ksym;
+
void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH bpf-next v2 11/11] selftests/bpf: Add verifier tests for stack argument validation
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (9 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 10/11] selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument Yonghong Song
@ 2026-04-05 16:54 ` Yonghong Song
2026-04-05 17:08 ` [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 16:54 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
Add inline-asm-based verifier tests that exercise the stack argument
validation logic directly.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_stack_arg.c | 302 ++++++++++++++++++
2 files changed, 304 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 1ac366fd4dae..f39ed1f28a7b 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -90,6 +90,7 @@
#include "verifier_sockmap_mutate.skel.h"
#include "verifier_spill_fill.skel.h"
#include "verifier_spin_lock.skel.h"
+#include "verifier_stack_arg.skel.h"
#include "verifier_stack_ptr.skel.h"
#include "verifier_store_release.skel.h"
#include "verifier_subprog_precision.skel.h"
@@ -236,6 +237,7 @@ void test_verifier_sock_addr(void) { RUN(verifier_sock_addr); }
void test_verifier_sockmap_mutate(void) { RUN(verifier_sockmap_mutate); }
void test_verifier_spill_fill(void) { RUN(verifier_spill_fill); }
void test_verifier_spin_lock(void) { RUN(verifier_spin_lock); }
+void test_verifier_stack_arg(void) { RUN(verifier_stack_arg); }
void test_verifier_stack_ptr(void) { RUN(verifier_stack_ptr); }
void test_verifier_store_release(void) { RUN(verifier_store_release); }
void test_verifier_subprog_precision(void) { RUN(verifier_subprog_precision); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
new file mode 100644
index 000000000000..921a9b3bee7f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -0,0 +1,302 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 1);
+ __type(key, long long);
+ __type(value, long long);
+} map_hash_8b SEC(".maps");
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+__noinline __used
+static int subprog_6args(int a, int b, int c, int d, int e, int f)
+{
+ return a + b + c + d + e + f;
+}
+
+__noinline __used
+static int subprog_7args(int a, int b, int c, int d, int e, int f, int g)
+{
+ return a + b + c + d + e + f + g;
+}
+
+SEC("tc")
+__description("stack_arg: subprog with 6 args")
+__success
+__arch_x86_64
+__naked void stack_arg_6args(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r12 - 8) = 6;"
+ "call subprog_6args;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: two subprogs with >5 args")
+__success
+__arch_x86_64
+__naked void stack_arg_two_subprogs(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r12 - 8) = 10;"
+ "call subprog_6args;"
+ "r6 = r0;"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r12 - 16) = 30;"
+ "*(u64 *)(r12 - 8) = 20;"
+ "call subprog_7args;"
+ "r0 += r6;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: read from uninitialized stack arg slot")
+__failure
+__arch_x86_64
+__msg("invalid read from stack arg")
+__naked void stack_arg_read_uninitialized(void)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r12 - 8);"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: gap at offset -8, only wrote -16")
+__failure
+__arch_x86_64
+__msg("stack arg#6 not properly initialized")
+__naked void stack_arg_gap_at_minus8(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r12 - 16) = 30;"
+ "call subprog_7args;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: incorrect size of stack arg write")
+__failure
+__arch_x86_64
+__msg("stack arg write must be 8 bytes, got 4")
+__naked void stack_arg_not_written(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u32 *)(r12 - 8) = 30;"
+ "call subprog_6args;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+__noinline __used
+static long subprog_stack_arg_pruning_deref(int a, int b, int c, int d, int e, int f)
+{
+ long local = 0, *p, *ptr;
+
+ if (a <= 3) {
+ /* Overwrite the stack arg slot with a pointer to local */
+ p = &local;
+ asm volatile ("*(u64 *)(r12 - 8) = %[p];" :: [p] "r"(p) : "memory");
+ }
+
+ /* Read back the (possibly overwritten) stack arg slot */
+ asm volatile ("%[ptr] = *(u64 *)(r12 - 8);" : [ptr] "=r"(ptr) :: "memory");
+
+ /* Deref: safe for PTR_TO_STACK, unsafe for scalar */
+ return *ptr;
+}
+
+SEC("tc")
+__description("stack_arg: pruning with different stack arg types")
+__failure
+__flag(BPF_F_TEST_STATE_FREQ)
+__arch_x86_64
+__msg("invalid mem access 'scalar'")
+__naked void stack_arg_pruning_type_mismatch(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r1 = r0;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r12 - 8) = 42;"
+ "call subprog_stack_arg_pruning_deref;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: release_reference invalidates stack arg slot")
+__failure
+__arch_x86_64
+__msg("R0 invalid mem access 'scalar'")
+__naked void stack_arg_release_ref(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ /* struct bpf_sock_tuple tuple = {} */
+ "r2 = 0;"
+ "*(u32 *)(r10 - 8) = r2;"
+ "*(u64 *)(r10 - 16) = r2;"
+ "*(u64 *)(r10 - 24) = r2;"
+ "*(u64 *)(r10 - 32) = r2;"
+ "*(u64 *)(r10 - 40) = r2;"
+ "*(u64 *)(r10 - 48) = r2;"
+ /* sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof(tuple), 0, 0) */
+ "r1 = r6;"
+ "r2 = r10;"
+ "r2 += -48;"
+ "r3 = %[sizeof_bpf_sock_tuple];"
+ "r4 = 0;"
+ "r5 = 0;"
+ "call %[bpf_sk_lookup_tcp];"
+ /* r0 = sk (PTR_TO_SOCK_OR_NULL) */
+ "if r0 == 0 goto l0_%=;"
+ /* spill sk to stack arg slot */
+ "*(u64 *)(r12 - 8) = r0;"
+ /* release the reference */
+ "r1 = r0;"
+ "call %[bpf_sk_release];"
+ /* r0 and stack arg slot (r12 - 8) should be invalid now. */
+ "r0 = *(u64 *)(r12 - 8);"
+ "r0 = *(u8 *)(r0 + 0);"
+ "l0_%=:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_sk_lookup_tcp),
+ __imm(bpf_sk_release),
+ __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: pkt pointer in stack arg slot invalidated after pull_data")
+__failure
+__arch_x86_64
+__msg("R0 invalid mem access 'scalar'")
+__naked void stack_arg_stale_pkt_ptr(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "r7 = *(u32 *)(r6 + %[__sk_buff_data]);"
+ "r8 = *(u32 *)(r6 + %[__sk_buff_data_end]);"
+ /* check pkt has at least 1 byte */
+ "r0 = r7;"
+ "r0 += 1;"
+ "if r0 > r8 goto l0_%=;"
+ /* spill valid pkt pointer to stack arg slot */
+ "*(u64 *)(r12 - 8) = r7;"
+ /* bpf_skb_pull_data(skb, 0) — invalidates all pkt pointers */
+ "r1 = r6;"
+ "r2 = 0;"
+ "call %[bpf_skb_pull_data];"
+ /* read back the stale pkt pointer from stack arg slot */
+ "r0 = *(u64 *)(r12 - 8);"
+ /* dereferencing stale pkt pointer should fail */
+ "r0 = *(u8 *)(r0 + 0);"
+ "l0_%=:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_skb_pull_data),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: null propagation rejects deref on null branch")
+__failure
+__arch_x86_64
+__msg("R1 invalid mem access 'scalar'")
+__naked void stack_arg_null_propagation_fail(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ /* r0 = bpf_map_lookup_elem(&map_hash_8b, &key) */
+ "r2 = r10;"
+ "r2 += -8;"
+ "r1 = %[map_hash_8b] ll;"
+ "call %[bpf_map_lookup_elem];"
+ /* spill PTR_TO_MAP_VALUE_OR_NULL to stack arg slot */
+ "*(u64 *)(r12 - 8) = r0;"
+ /* null check on r0 */
+ "if r0 != 0 goto l0_%=;"
+ /*
+ * on null branch, stack arg slot should be null (scalar 0).
+ * read it back and dereference should fail.
+ */
+ "r1 = *(u64 *)(r12 - 8);"
+ "r0 = *(u64 *)(r1 + 0);"
+ "l0_%=:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all
+ );
+}
+
+#else
+
+SEC("socket")
+__description("stack_arg is not supported by compiler or jit, use a dummy test")
+__success
+int dummy_test(void)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
--
2.52.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs
2026-04-05 16:53 [PATCH bpf-next v2 00/11] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
` (10 preceding siblings ...)
2026-04-05 16:54 ` [PATCH bpf-next v2 11/11] selftests/bpf: Add verifier tests for stack argument validation Yonghong Song
@ 2026-04-05 17:08 ` Yonghong Song
11 siblings, 0 replies; 13+ messages in thread
From: Yonghong Song @ 2026-04-05 17:08 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Jose E . Marchesi, kernel-team, Martin KaFai Lau
On 4/5/26 9:53 AM, Yonghong Song wrote:
> Currently, bpf function calls and kfunc's are limited by 5 reg-level
> parameters. For function calls with more than 5 parameters,
> developers can use always inlining or pass a struct pointer
> after packing more parameters in that struct. But there is
> no workaround for kfunc if more than 5 parameters is needed.
>
> This patch set lifts the 5-argument limit by introducing stack-based
> argument passing for BPF functions and kfunc's, coordinated with
> compiler support in LLVM [1]. The compiler emits loads/stores through
> a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond
> the 5th, keeping the stack arg area separate from the r10-based program
> stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS
> (12), which is sufficient for the vast majority of use cases.
>
> The x86_64 JIT translates r12-relative accesses to RBP-relative
> native instructions. Each function's stack allocation is extended
> by 'max_outgoing' bytes to hold the outgoing arg area below the
> program stack. This makes implementation easier as the r10 can be
> reused for stack argument access. At BPF-to-BPF call sites, outgoing
> args are pushed onto the native stack before CALL. The incoming
> parameters can directly get the value from pushed native stack from
> caller. For kfunc calls, args are marshaled per the x86_64 C calling
> convention (arg 6 in R9, args 7+ on the native stack).
>
> Global subprogs with >5 args are not yet supported. Only x86_64
> is supported for now.
>
> For the rest of patches, patches 1-6 added verifier support of
> stack arguments for bpf-to-bpf functions and kfunc's. Patch 7
> enables x86_64 for stack arguments. Patch 8 implemented JIT for
> x86_64. Patches 9-11 are some selftests.
>
> [1] https://github.com/llvm/llvm-project/pull/189060
>
> Changelogs:
> v1 -> v2:
> - v1: https://lore.kernel.org/bpf/20260402012727.3916819-1-yonghong.song@linux.dev/
> - Add stack_arg_safe() to do pruning for stack arguments.
> - Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is
> used, added verification log to indicate the start and end of such
> faked register usage.
> - For x86_64 JIT, copying incoming parameter values directly from caller's stack.
> - Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.
>
> Yonghong Song (11):
> bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
> bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
> bpf: Support stack arguments for bpf functions
> bpf: Refactor process_iter_arg() to have proper argument index
> bpf: Support stack arguments for kfunc calls
> bpf: Reject stack arguments in non-JITed programs
> bpf: Enable stack argument support for x86_64
> bpf,x86: Implement JIT support for stack arguments
> selftests/bpf: Add tests for BPF function stack arguments
> selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
> argument
> selftests/bpf: Add verifier tests for stack argument validation
>
> arch/x86/net/bpf_jit_comp.c | 140 +++++-
> include/linux/bpf.h | 6 +
> include/linux/bpf_verifier.h | 31 +-
> include/linux/filter.h | 4 +-
> kernel/bpf/btf.c | 21 +-
> kernel/bpf/core.c | 12 +-
> kernel/bpf/verifier.c | 474 ++++++++++++++++--
> .../selftests/bpf/prog_tests/stack_arg.c | 132 +++++
> .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 +
> .../selftests/bpf/prog_tests/verifier.c | 2 +
> tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++
> .../selftests/bpf/progs/stack_arg_fail.c | 32 ++
> .../selftests/bpf/progs/stack_arg_kfunc.c | 164 ++++++
> .../selftests/bpf/progs/verifier_stack_arg.c | 302 +++++++++++
> .../selftests/bpf/test_kmods/bpf_testmod.c | 72 +++
> .../bpf/test_kmods/bpf_testmod_kfunc.h | 26 +
> 16 files changed, 1594 insertions(+), 60 deletions(-)
> create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
> create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
> create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
> create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
> create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
> create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
>
Please ignore this patch set. Just discovered that there is a selftest failure
and a couple of 'UTF-8' issues. I will fix these issues and send the next revision
shortly.
^ permalink raw reply [flat|nested] 13+ messages in thread