[PATCH bpf-next v3 00/11] bpf: Support stack arguments for BPF functions and kfuncs

[PATCH bpf-next v3 00/11] bpf: Support stack arguments for BPF functions and kfuncs

[Yonghong Song]

Currently, bpf function calls and kfunc's are limited by 5 reg-level
parameters. For function calls with more than 5 parameters,
developers can use always inlining or pass a struct pointer
after packing more parameters in that struct. But there is
no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based
argument passing for BPF functions and kfunc's, coordinated with
compiler support in LLVM [1]. The compiler emits loads/stores through
a new bpf register r12 (BPF_REG_STACK_ARG_BASE) to pass arguments beyond
the 5th, keeping the stack arg area separate from the r10-based program
stack. The maximum number of arguments is capped at MAX_BPF_FUNC_ARGS
(12), which is sufficient for the vast majority of use cases.

The x86_64 JIT translates r12-relative accesses to RBP-relative
native instructions. Each function's stack allocation is extended
by 'max_outgoing' bytes to hold the outgoing arg area below the
program stack. This makes implementation easier as the r10 can be
reused for stack argument access. At BPF-to-BPF call sites, outgoing
args are pushed onto the native stack before CALL. The incoming
parameters can directly get the value from pushed native stack from
caller. For kfunc calls, args are marshaled per the x86_64 C calling
convention (arg 6 in R9, args 7+ on the native stack).

Global subprogs with >5 args are not yet supported. Only x86_64
is supported for now.

For the rest of patches, patches 1-6 added verifier support of
stack arguments for bpf-to-bpf functions and kfunc's. Patch 7
enables x86_64 for stack arguments. Patch 8 implemented JIT for
x86_64. Patches 9-11 are some selftests.

  [1] https://github.com/llvm/llvm-project/pull/189060

Changelogs:
  v2 -> v3:
    - v2: https://lore.kernel.org/bpf/20260405165300.826241-1-yonghong.song@linux.dev/
    - Fix selftest stack_arg_gap_at_minus8().
    - Fix a few 'UTF-8' issues.
  v1 -> v2:
    - v1: https://lore.kernel.org/bpf/20260402012727.3916819-1-yonghong.song@linux.dev/
    - Add stack_arg_safe() to do pruning for stack arguments.
    - Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is
      used, added verification log to indicate the start and end of such
      faked register usage.
    - For x86_64 JIT, copying incoming parameter values directly from caller's stack.
    - Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.

Yonghong Song (11):
  bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE
  bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
  bpf: Support stack arguments for bpf functions
  bpf: Refactor process_iter_arg() to have proper argument index
  bpf: Support stack arguments for kfunc calls
  bpf: Reject stack arguments in non-JITed programs
  bpf: Enable stack argument support for x86_64
  bpf,x86: Implement JIT support for stack arguments
  selftests/bpf: Add tests for BPF function stack arguments
  selftests/bpf: Add negative test for greater-than-8-byte kfunc stack
    argument
  selftests/bpf: Add verifier tests for stack argument validation

 arch/x86/net/bpf_jit_comp.c                   | 140 +++++-
 include/linux/bpf.h                           |   6 +
 include/linux/bpf_verifier.h                  |  31 +-
 include/linux/filter.h                        |   4 +-
 kernel/bpf/btf.c                              |  21 +-
 kernel/bpf/core.c                             |  12 +-
 kernel/bpf/verifier.c                         | 474 ++++++++++++++++--
 .../selftests/bpf/prog_tests/stack_arg.c      | 132 +++++
 .../selftests/bpf/prog_tests/stack_arg_fail.c |  24 +
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++
 .../selftests/bpf/progs/stack_arg_fail.c      |  32 ++
 .../selftests/bpf/progs/stack_arg_kfunc.c     | 164 ++++++
 .../selftests/bpf/progs/verifier_stack_arg.c  | 302 +++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  72 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  26 +
 16 files changed, 1594 insertions(+), 60 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c

-- 
2.52.0

[PATCH bpf-next v3 01/11] bpf: Introduce bpf register BPF_REG_STACK_ARG_BASE

[Yonghong Song]

The newly-added register BPF_REG_STACK_ARG_BASE corresponds to bpf
register R12 added by [1]. R12 is used as the base for stack arguments
so it won't mess out R10 based stacks.

  [1] https://github.com/llvm/llvm-project/pull/189060

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/filter.h | 3 ++-
 kernel/bpf/core.c      | 4 ++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/include/linux/filter.h b/include/linux/filter.h
index e40d4071a345..68f018dd4b9c 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -59,7 +59,8 @@ struct ctl_table_header;
 
 /* Kernel hidden auxiliary/helper register. */
 #define BPF_REG_AX		MAX_BPF_REG
-#define MAX_BPF_EXT_REG		(MAX_BPF_REG + 1)
+#define BPF_REG_STACK_ARG_BASE	(MAX_BPF_REG + 1)
+#define MAX_BPF_EXT_REG		(MAX_BPF_REG + 2)
 #define MAX_BPF_JIT_REG		MAX_BPF_EXT_REG
 
 /* unused opcode to mark special call to bpf_tail_call() helper */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 1af5fb3f21d9..3520337a1c0e 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
 	u32 imm_rnd = get_random_u32();
 	s16 off;
 
-	BUILD_BUG_ON(BPF_REG_AX  + 1 != MAX_BPF_JIT_REG);
-	BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG);
+	BUILD_BUG_ON(BPF_REG_AX + 2 != MAX_BPF_JIT_REG);
+	BUILD_BUG_ON(BPF_REG_STACK_ARG_BASE + 1 != MAX_BPF_JIT_REG);
 
 	/* Constraints on AX register:
 	 *
-- 
2.52.0

[PATCH bpf-next v3 02/11] bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments

[Yonghong Song]

Currently, MAX_BPF_FUNC_ARGS is used for tracepoint related progs where
the number of parameters cannot exceed MAX_BPF_FUNC_ARGS.

Here, MAX_BPF_FUNC_ARGS is reused to set a limit of the number of arguments
for bpf functions and kfunc's. The current value for MAX_BPF_FUNC_ARGS
is 12 which should be sufficient for majority of bpf functions and
kfunc's.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 35b1e25bd104..99701c4cc34e 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1151,6 +1151,10 @@ struct bpf_prog_offload {
 
 /* The longest tracepoint has 12 args.
  * See include/trace/bpf_probe.h
+ *
+ * Also reuse this macro for maximum number of arguments a BPF function
+ * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
+ * stack arg slots.
  */
 #define MAX_BPF_FUNC_ARGS 12
 
-- 
2.52.0

[PATCH bpf-next v3 03/11] bpf: Support stack arguments for bpf functions

[Yonghong Song]

Currently BPF functions (subprogs) are limited to 5 register arguments.
With [1], the compiler can emit code that passes additional arguments
via a dedicated stack area through bpf register
BPF_REG_STACK_ARG_BASE (r12), introduced in the previous patch.

The following is an example to show how stack arguments are saved
and transferred between caller and callee:

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
  }

The following is a illustration of stack allocation:

   Caller (foo)                           Callee (bar)
   ============                           ============
   r12-relative stack arg area:           r12-relative stack arg area:

   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+||   ...
   r12-32: [outgoing arg 7 to callee]-+|
   r12-40: [outgoing arg 8 to callee]--+

  The caller writes outgoing args past its own incoming area.
  At the call site, the verifier transfers the caller's outgoing
  slots into the callee's incoming slots.

The verifier tracks stack arg slots separately from the regular r10
stack. A new 'bpf_stack_arg_state' structure mirrors the existing stack
slot tracking (spilled_ptr + slot_type[]) but lives in a dedicated
'stack_arg_slots' array in bpf_func_state. This separation keeps the
stack arg area from interfering with the normal stack and frame pointer
(r10) bookkeeping. Similar to stacksafe(), introduce stack_arg_safe()
to do pruning check.

If the bpf function has more than one calls, e.g.,

  int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
    ...
    bar1(a1, a2, a3, a4, a5, a6, a7, a8);
    ...
    bar2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
    ...
  }

The following is an illustration:

   Caller (foo)                           Callee (bar1)
   ============                           =============
   r12-relative stack arg area:           r12-relative stack arg area:

   r12-8:  [incoming arg 6]          +--> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
   r12-16: [incoming arg 7]          |+-> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
   ---- incoming/outgoing boundary   |||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+||  ...
   r12-32: [outgoing arg 7 to callee]-+|
   r12-40: [outgoing arg 8 to callee]--+
   ...
   Back from bar1
   ...                                     Callee (bar2)
   ===                                     =============
                                     +---> r12-8:  [incoming arg 6] (from caller's outgoing r12-24)
                                     |+--> r12-16: [incoming arg 7] (from caller's outgoing r12-32)
                                     ||+-> r12-24: [incoming arg 8] (from caller's outgoing r12-40)
                                     |||+> r12-32: [incoming arg 9] (from caller's outgoing r12-48)
   ---- incoming/outgoing boundary   ||||  ---- incoming/outgoing boundary
   r12-24: [outgoing arg 6 to callee]+|||  ...
   r12-32: [outgoing arg 7 to callee]-+||
   r12-40: [outgoing arg 8 to callee]--+|
   r12-48: [outgoing arg 9 to callee]---+

Global subprogs with >5 args are not yet supported.

  [1] https://github.com/llvm/llvm-project/pull/189060

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf.h          |   2 +
 include/linux/bpf_verifier.h |  31 +++-
 kernel/bpf/btf.c             |  14 +-
 kernel/bpf/verifier.c        | 311 +++++++++++++++++++++++++++++++++--
 4 files changed, 334 insertions(+), 24 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 99701c4cc34e..72f990054d07 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1666,6 +1666,8 @@ struct bpf_prog_aux {
 	u32 max_pkt_offset;
 	u32 max_tp_access;
 	u32 stack_depth;
+	u16 incoming_stack_arg_depth;
+	u16 stack_arg_depth; /* both incoming and max outgoing of stack arguments */
 	u32 id;
 	u32 func_cnt; /* used by non-func prog as the number of func progs */
 	u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 36bfd96d4563..75fe67baa833 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -268,6 +268,11 @@ struct bpf_retval_range {
 	bool return_32bit;
 };
 
+struct bpf_stack_arg_state {
+	struct bpf_reg_state spilled_ptr; /* for spilled scalar/pointer semantics */
+	u8 slot_type[BPF_REG_SIZE];
+};
+
 /* state of the program:
  * type of all registers and stack info
  */
@@ -319,6 +324,10 @@ struct bpf_func_state {
 	 * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
 	 */
 	int allocated_stack;
+
+	u16 stack_arg_depth; /* Size of incoming + max outgoing stack args in bytes. */
+	u16 incoming_stack_arg_depth; /* Size of incoming stack args in bytes. */
+	struct bpf_stack_arg_state *stack_arg_slots;
 };
 
 #define MAX_CALL_FRAMES 8
@@ -456,6 +465,17 @@ struct bpf_verifier_state {
 	     iter < frame->allocated_stack / BPF_REG_SIZE;		\
 	     iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
 
+#define bpf_get_spilled_stack_arg(slot, frame, mask)			\
+	(((slot < frame->stack_arg_depth / BPF_REG_SIZE) &&		\
+	  ((1 << frame->stack_arg_slots[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \
+	 ? &frame->stack_arg_slots[slot].spilled_ptr : NULL)
+
+/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
+#define bpf_for_each_spilled_stack_arg(iter, frame, reg, mask)		\
+	for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame, mask); \
+	     iter < frame->stack_arg_depth / BPF_REG_SIZE;		\
+	     iter++, reg = bpf_get_spilled_stack_arg(iter, frame, mask))
+
 #define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr)   \
 	({                                                               \
 		struct bpf_verifier_state *___vstate = __vst;            \
@@ -473,6 +493,11 @@ struct bpf_verifier_state {
 					continue;                        \
 				(void)(__expr);                          \
 			}                                                \
+			bpf_for_each_spilled_stack_arg(___j, __state, __reg, __mask) { \
+				if (!__reg)                              \
+					continue;                        \
+				(void)(__expr);                          \
+			}                                                \
 		}                                                        \
 	})
 
@@ -686,10 +711,12 @@ struct bpf_subprog_info {
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
 	bool might_sleep: 1;
-	u8 arg_cnt:3;
+	u8 arg_cnt:4;
 
 	enum priv_stack_mode priv_stack_mode;
-	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
+	struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+	u16 incoming_stack_arg_depth;
+	u16 outgoing_stack_arg_depth;
 };
 
 struct bpf_verifier_env;
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index a62d78581207..c5f3aa05d5a3 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7887,13 +7887,19 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	}
 	args = (const struct btf_param *)(t + 1);
 	nargs = btf_type_vlen(t);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
-		if (!is_global)
-			return -EINVAL;
-		bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
+	if (nargs > MAX_BPF_FUNC_ARGS) {
+		bpf_log(log, "Function %s() with %d > %d args not supported.\n",
+			tname, nargs, MAX_BPF_FUNC_ARGS);
+		return -EINVAL;
+	}
+	if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
+		bpf_log(log, "Global function %s() with %d > %d args not supported.\n",
 			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
 		return -EINVAL;
 	}
+	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+		sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
 	/* check that function is void or returns int, exception cb also requires this */
 	t = btf_type_by_id(btf, t->type);
 	while (btf_type_is_modifier(t))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 84699a428077..52a61021613b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1477,6 +1477,19 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st
 		return -ENOMEM;
 
 	dst->allocated_stack = src->allocated_stack;
+
+	/* copy stack_arg_slots state */
+	n = src->stack_arg_depth / BPF_REG_SIZE;
+	if (n) {
+		dst->stack_arg_slots = copy_array(dst->stack_arg_slots, src->stack_arg_slots, n,
+						  sizeof(struct bpf_stack_arg_state),
+						  GFP_KERNEL_ACCOUNT);
+		if (!dst->stack_arg_slots)
+			return -ENOMEM;
+
+		dst->stack_arg_depth = src->stack_arg_depth;
+		dst->incoming_stack_arg_depth = src->incoming_stack_arg_depth;
+	}
 	return 0;
 }
 
@@ -1518,6 +1531,25 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state
 	return 0;
 }
 
+static int grow_stack_arg_slots(struct bpf_verifier_env *env,
+				struct bpf_func_state *state, int size)
+{
+	size_t old_n = state->stack_arg_depth / BPF_REG_SIZE, n;
+
+	size = round_up(size, BPF_REG_SIZE);
+	n = size / BPF_REG_SIZE;
+	if (old_n >= n)
+		return 0;
+
+	state->stack_arg_slots = realloc_array(state->stack_arg_slots, old_n, n,
+					       sizeof(struct bpf_stack_arg_state));
+	if (!state->stack_arg_slots)
+		return -ENOMEM;
+
+	state->stack_arg_depth = size;
+	return 0;
+}
+
 /* Acquire a pointer id from the env and update the state->refs to include
  * this new pointer reference.
  * On success, returns a valid pointer id to associate with the register
@@ -1688,6 +1720,7 @@ static void free_func_state(struct bpf_func_state *state)
 {
 	if (!state)
 		return;
+	kfree(state->stack_arg_slots);
 	kfree(state->stack);
 	kfree(state);
 }
@@ -5934,6 +5967,107 @@ static int check_stack_write(struct bpf_verifier_env *env,
 	return err;
 }
 
+/* Validate that a stack arg access is 8-byte sized and aligned. */
+static int check_stack_arg_access(struct bpf_verifier_env *env,
+				  struct bpf_insn *insn, const char *op)
+{
+	int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
+
+	if (size != BPF_REG_SIZE) {
+		verbose(env, "stack arg %s must be %d bytes, got %d\n",
+			op, BPF_REG_SIZE, size);
+		return -EINVAL;
+	}
+	if (insn->off % BPF_REG_SIZE) {
+		verbose(env, "stack arg %s offset %d not aligned to %d\n",
+			op, insn->off, BPF_REG_SIZE);
+		return -EINVAL;
+	}
+	return 0;
+}
+
+/* Check that a stack arg slot has been properly initialized. */
+static bool is_stack_arg_slot_initialized(struct bpf_func_state *state, int spi)
+{
+	u8 type;
+
+	if (spi >= (int)(state->stack_arg_depth / BPF_REG_SIZE))
+		return false;
+	type = state->stack_arg_slots[spi].slot_type[BPF_REG_SIZE - 1];
+	return type == STACK_SPILL || type == STACK_MISC;
+}
+
+/*
+ * Write a value to the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
+				 int off, int value_regno)
+{
+	int spi = (-off - 1) / BPF_REG_SIZE;
+	struct bpf_subprog_info *subprog;
+	struct bpf_func_state *cur;
+	struct bpf_reg_state *reg;
+	int i, err;
+	u8 type;
+
+	err = grow_stack_arg_slots(env, state, -off);
+	if (err)
+		return err;
+
+	/* Ensure the JIT allocates space for the stack arg area. */
+	subprog = &env->subprog_info[state->subprogno];
+	if (-off > subprog->outgoing_stack_arg_depth)
+		subprog->outgoing_stack_arg_depth = -off;
+
+	cur = env->cur_state->frame[env->cur_state->curframe];
+	if (value_regno >= 0) {
+		reg = &cur->regs[value_regno];
+		state->stack_arg_slots[spi].spilled_ptr = *reg;
+		type = is_spillable_regtype(reg->type) ? STACK_SPILL : STACK_MISC;
+		for (i = 0; i < BPF_REG_SIZE; i++)
+			state->stack_arg_slots[spi].slot_type[i] = type;
+	} else {
+		/* BPF_ST: store immediate, treat as scalar */
+		reg = &state->stack_arg_slots[spi].spilled_ptr;
+		reg->type = SCALAR_VALUE;
+		__mark_reg_known(reg, (u32)env->prog->insnsi[env->insn_idx].imm);
+		for (i = 0; i < BPF_REG_SIZE; i++)
+			state->stack_arg_slots[spi].slot_type[i] = STACK_MISC;
+	}
+	return 0;
+}
+
+/*
+ * Read a value from the stack arg area.
+ * off is the negative offset from the stack arg frame pointer.
+ * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
+ */
+static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,
+				int off, int dst_regno)
+{
+	int spi = (-off - 1) / BPF_REG_SIZE;
+	struct bpf_func_state *cur;
+	u8 *stype;
+
+	if (-off > state->stack_arg_depth) {
+		verbose(env, "invalid read from stack arg off %d depth %d\n",
+			off, state->stack_arg_depth);
+		return -EACCES;
+	}
+
+	stype = state->stack_arg_slots[spi].slot_type;
+	cur = env->cur_state->frame[env->cur_state->curframe];
+
+	if (stype[BPF_REG_SIZE - 1] == STACK_SPILL)
+		copy_register_state(&cur->regs[dst_regno],
+				    &state->stack_arg_slots[spi].spilled_ptr);
+	else
+		mark_reg_unknown(env, cur->regs, dst_regno);
+	return 0;
+}
+
 static int check_map_access_type(struct bpf_verifier_env *env, u32 regno,
 				 int off, int size, enum bpf_access_type type)
 {
@@ -8109,10 +8243,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			  bool strict_alignment_once, bool is_ldsx,
 			  bool allow_trust_mismatch, const char *ctx)
 {
+	struct bpf_verifier_state *vstate = env->cur_state;
+	struct bpf_func_state *state = vstate->frame[vstate->curframe];
 	struct bpf_reg_state *regs = cur_regs(env);
 	enum bpf_reg_type src_reg_type;
 	int err;
 
+	/* Handle stack arg access */
+	if (insn->src_reg == BPF_REG_STACK_ARG_BASE) {
+		err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
+		if (err)
+			return err;
+		err = check_stack_arg_access(env, insn, "read");
+		if (err)
+			return err;
+		return check_stack_arg_read(env, state, insn->off, insn->dst_reg);
+	}
+
 	/* check src operand */
 	err = check_reg_arg(env, insn->src_reg, SRC_OP);
 	if (err)
@@ -8141,10 +8288,23 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
 static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			   bool strict_alignment_once)
 {
+	struct bpf_verifier_state *vstate = env->cur_state;
+	struct bpf_func_state *state = vstate->frame[vstate->curframe];
 	struct bpf_reg_state *regs = cur_regs(env);
 	enum bpf_reg_type dst_reg_type;
 	int err;
 
+	/* Handle stack arg write */
+	if (insn->dst_reg == BPF_REG_STACK_ARG_BASE) {
+		err = check_reg_arg(env, insn->src_reg, SRC_OP);
+		if (err)
+			return err;
+		err = check_stack_arg_access(env, insn, "write");
+		if (err)
+			return err;
+		return check_stack_arg_write(env, state, insn->off, insn->src_reg);
+	}
+
 	/* check src1 operand */
 	err = check_reg_arg(env, insn->src_reg, SRC_OP);
 	if (err)
@@ -11027,8 +11187,10 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			   int *insn_idx)
 {
 	struct bpf_verifier_state *state = env->cur_state;
+	struct bpf_subprog_info *caller_info;
 	struct bpf_func_state *caller;
 	int err, subprog, target_insn;
+	u16 callee_incoming;
 
 	target_insn = *insn_idx + insn->imm + 1;
 	subprog = bpf_find_subprog(env, target_insn);
@@ -11080,6 +11242,15 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		return 0;
 	}
 
+	/*
+	 * Track caller's outgoing stack arg depth (max across all callees).
+	 * This is needed so the JIT knows how much stack arg space to allocate.
+	 */
+	caller_info = &env->subprog_info[caller->subprogno];
+	callee_incoming = env->subprog_info[subprog].incoming_stack_arg_depth;
+	if (callee_incoming > caller_info->outgoing_stack_arg_depth)
+		caller_info->outgoing_stack_arg_depth = callee_incoming;
+
 	/* for regular function entry setup new frame and continue
 	 * from that frame.
 	 */
@@ -11135,13 +11306,41 @@ static int set_callee_state(struct bpf_verifier_env *env,
 			    struct bpf_func_state *caller,
 			    struct bpf_func_state *callee, int insn_idx)
 {
-	int i;
+	struct bpf_subprog_info *callee_info;
+	int i, err;
 
 	/* copy r1 - r5 args that callee can access.  The copy includes parent
 	 * pointers, which connects us up to the liveness chain
 	 */
 	for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 		callee->regs[i] = caller->regs[i];
+
+	/*
+	 * Transfer stack args from caller's outgoing area to callee's incoming area.
+	 * Caller wrote outgoing args at offsets '-(incoming + 8)', '-(incoming + 16)', ...
+	 * These outgoing args will go to callee's incoming area.
+	 */
+	callee_info = &env->subprog_info[callee->subprogno];
+	if (callee_info->incoming_stack_arg_depth) {
+		int caller_incoming_slots = caller->incoming_stack_arg_depth / BPF_REG_SIZE;
+		int callee_incoming_slots = callee_info->incoming_stack_arg_depth / BPF_REG_SIZE;
+
+		callee->incoming_stack_arg_depth = callee_info->incoming_stack_arg_depth;
+		err = grow_stack_arg_slots(env, callee, callee_info->incoming_stack_arg_depth);
+		if (err)
+			return err;
+
+		for (i = 0; i < callee_incoming_slots; i++) {
+			int caller_spi = i + caller_incoming_slots;
+
+			if (!is_stack_arg_slot_initialized(caller, caller_spi)) {
+				verbose(env, "stack arg#%d not properly initialized\n",
+					i + MAX_BPF_FUNC_REG_ARGS);
+				return -EINVAL;
+			}
+			callee->stack_arg_slots[i] = caller->stack_arg_slots[caller_spi];
+		}
+	}
 	return 0;
 }
 
@@ -20535,6 +20734,56 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 	return true;
 }
 
+/*
+ * Compare stack arg slots between old and current states.
+ * Only incoming stack args need comparison -— outgoing slots are transient
+ * (written before each call, consumed at the call site) so they don't carry
+ * meaningful state across pruning points.
+ */
+static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
+			   struct bpf_func_state *cur, struct bpf_idmap *idmap,
+			   enum exact_level exact)
+{
+	int i, spi;
+
+	if (old->incoming_stack_arg_depth != cur->incoming_stack_arg_depth)
+		return false;
+
+	for (i = 0; i < old->incoming_stack_arg_depth; i++) {
+		spi = i / BPF_REG_SIZE;
+
+		if (exact == EXACT &&
+		    old->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE] !=
+		    cur->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE])
+			return false;
+
+		if (old->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID)
+			continue;
+
+		if (old->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE] !=
+		    cur->stack_arg_slots[spi].slot_type[i % BPF_REG_SIZE])
+			return false;
+
+		if (i % BPF_REG_SIZE != BPF_REG_SIZE - 1)
+			continue;
+
+		switch (old->stack_arg_slots[spi].slot_type[BPF_REG_SIZE - 1]) {
+		case STACK_SPILL:
+			if (!regsafe(env, &old->stack_arg_slots[spi].spilled_ptr,
+				     &cur->stack_arg_slots[spi].spilled_ptr, idmap, exact))
+				return false;
+			break;
+		case STACK_MISC:
+		case STACK_ZERO:
+		case STACK_INVALID:
+			continue;
+		default:
+			return false;
+		}
+	}
+	return true;
+}
+
 static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,
 		    struct bpf_idmap *idmap)
 {
@@ -20626,6 +20875,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 	if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
 		return false;
 
+	if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
+		return false;
+
 	return true;
 }
 
@@ -21534,23 +21786,37 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 			verbose(env, "BPF_ST uses reserved fields\n");
 			return -EINVAL;
 		}
-		/* check src operand */
-		err = check_reg_arg(env, insn->dst_reg, SRC_OP);
-		if (err)
-			return err;
 
-		dst_reg_type = cur_regs(env)[insn->dst_reg].type;
+		/* Handle stack arg write (store immediate) */
+		if (insn->dst_reg == BPF_REG_STACK_ARG_BASE) {
+			struct bpf_verifier_state *vstate = env->cur_state;
+			struct bpf_func_state *state = vstate->frame[vstate->curframe];
 
-		/* check that memory (dst_reg + off) is writeable */
-		err = check_mem_access(env, env->insn_idx, insn->dst_reg,
-				       insn->off, BPF_SIZE(insn->code),
-				       BPF_WRITE, -1, false, false);
-		if (err)
-			return err;
+			err = check_stack_arg_access(env, insn, "write");
+			if (err)
+				return err;
+			err = check_stack_arg_write(env, state, insn->off, -1);
+			if (err)
+				return err;
+		} else {
+			/* check src operand */
+			err = check_reg_arg(env, insn->dst_reg, SRC_OP);
+			if (err)
+				return err;
 
-		err = save_aux_ptr_type(env, dst_reg_type, false);
-		if (err)
-			return err;
+			dst_reg_type = cur_regs(env)[insn->dst_reg].type;
+
+			/* check that memory (dst_reg + off) is writeable */
+			err = check_mem_access(env, env->insn_idx, insn->dst_reg,
+					       insn->off, BPF_SIZE(insn->code),
+					       BPF_WRITE, -1, false, false);
+			if (err)
+				return err;
+
+			err = save_aux_ptr_type(env, dst_reg_type, false);
+			if (err)
+				return err;
+		}
 	} else if (class == BPF_JMP || class == BPF_JMP32) {
 		u8 opcode = BPF_OP(insn->code);
 
@@ -22246,11 +22512,11 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)
 		return err;
 
 	for (i = 0; i < insn_cnt; i++, insn++) {
-		if (insn->dst_reg >= MAX_BPF_REG) {
+		if (insn->dst_reg >= MAX_BPF_REG && insn->dst_reg != BPF_REG_STACK_ARG_BASE) {
 			verbose(env, "R%d is invalid\n", insn->dst_reg);
 			return -EINVAL;
 		}
-		if (insn->src_reg >= MAX_BPF_REG) {
+		if (insn->src_reg >= MAX_BPF_REG && insn->src_reg != BPF_REG_STACK_ARG_BASE) {
 			verbose(env, "R%d is invalid\n", insn->src_reg);
 			return -EINVAL;
 		}
@@ -23275,8 +23541,14 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 	int err, num_exentries;
 	int old_len, subprog_start_adjustment = 0;
 
-	if (env->subprog_cnt <= 1)
+	if (env->subprog_cnt <= 1) {
+		/*
+		 * Even without subprogs, kfunc calls with >5 args need stack arg space
+		 * allocated by the root program.
+		 */
+		prog->aux->stack_arg_depth = env->subprog_info[0].outgoing_stack_arg_depth;
 		return 0;
+	}
 
 	for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
 		if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
@@ -23366,6 +23638,9 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 
 		func[i]->aux->name[0] = 'F';
 		func[i]->aux->stack_depth = env->subprog_info[i].stack_depth;
+		func[i]->aux->incoming_stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth;
+		func[i]->aux->stack_arg_depth = env->subprog_info[i].incoming_stack_arg_depth +
+						env->subprog_info[i].outgoing_stack_arg_depth;
 		if (env->subprog_info[i].priv_stack_mode == PRIV_STACK_ADAPTIVE)
 			func[i]->aux->jits_use_priv_stack = true;
 
-- 
2.52.0

[PATCH bpf-next v3 04/11] bpf: Refactor process_iter_arg() to have proper argument index

[Yonghong Song]

In the next patch for kfunc stack arguments, a faked register is
used to do proper verification checking. For process_iter_arg(),
the regno is passed in and the iterator assumes 'regno - 1' as
the argument index. This is wrong as regno is fake. So refactor
process_iter_arg() by adding actual argument index which is used
inside the function.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 52a61021613b..183a4108fd4d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -9259,14 +9259,14 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx,
 }
 
 static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idx,
-			    struct bpf_kfunc_call_arg_meta *meta)
+			    int argno, struct bpf_kfunc_call_arg_meta *meta)
 {
 	struct bpf_reg_state *reg = reg_state(env, regno);
 	const struct btf_type *t;
 	int spi, err, i, nr_slots, btf_id;
 
 	if (reg->type != PTR_TO_STACK) {
-		verbose(env, "arg#%d expected pointer to an iterator on stack\n", regno - 1);
+		verbose(env, "arg#%d expected pointer to an iterator on stack\n", argno);
 		return -EINVAL;
 	}
 
@@ -9276,9 +9276,9 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
 	 * to any kfunc, if arg has "__iter" suffix, we need to be a bit more
 	 * conservative here.
 	 */
-	btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, regno - 1);
+	btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, argno);
 	if (btf_id < 0) {
-		verbose(env, "expected valid iter pointer as arg #%d\n", regno - 1);
+		verbose(env, "expected valid iter pointer as arg #%d\n", argno);
 		return -EINVAL;
 	}
 	t = btf_type_by_id(meta->btf, btf_id);
@@ -9288,7 +9288,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
 		/* bpf_iter_<type>_new() expects pointer to uninit iter state */
 		if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) {
 			verbose(env, "expected uninitialized iter_%s as arg #%d\n",
-				iter_type_str(meta->btf, btf_id), regno - 1);
+				iter_type_str(meta->btf, btf_id), argno);
 			return -EINVAL;
 		}
 
@@ -9312,7 +9312,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id
 			break;
 		case -EINVAL:
 			verbose(env, "expected an initialized iter_%s as arg #%d\n",
-				iter_type_str(meta->btf, btf_id), regno - 1);
+				iter_type_str(meta->btf, btf_id), argno);
 			return err;
 		case -EPROTO:
 			verbose(env, "expected an RCU CS when using %s\n", meta->func_name);
@@ -14063,7 +14063,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 					return -EINVAL;
 				}
 			}
-			ret = process_iter_arg(env, regno, insn_idx, meta);
+			ret = process_iter_arg(env, regno, insn_idx, i, meta);
 			if (ret < 0)
 				return ret;
 			break;
-- 
2.52.0

[PATCH bpf-next v3 05/11] bpf: Support stack arguments for kfunc calls

[Yonghong Song]

Extend the stack argument mechanism to kfunc calls, allowing kfuncs
with more than 5 parameters to receive additional arguments via the
r12-based stack arg area.

For kfuncs, the caller is a BPF program and the callee is a kernel
function. The BPF program writes outgoing args at r12-relative offsets
past its own incoming area.

The following is an example to show how stack arguments are saved:

   int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
     ...
     kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
     ...
     kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
     ...
   }

The following is an illustration:

   Caller (foo)
   ============
       r12-relative stack arg area:

       r12-8:  [incoming arg 6]
       r12-16: [incoming arg 7]

       ---- incoming/outgoing boundary (kfunc1)
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       ...
       Back from kfunc1
       ...

       ---- incoming/outgoing boundary
       r12-24: [outgoing arg 6 to callee]
       r12-32: [outgoing arg 7 to callee]
       r12-40: [outgoing arg 8 to callee]
       r12-48: [outgoing arg 9 to callee]

Later JIT will marshal outgoing arguments to the native calling convention
for kfunc1() and kfunc2().

In check_kfunc_args(), for args beyond the 5th, retrieve the spilled
register state from the caller's stack arg slots. Temporarily copy
it into regs[BPF_REG_1] to reuse the existing type checking
infrastructure, then restore after checking. The following is one
of examples based on a later selftest:

  13: (85) call bpf_kfunc_call_stack_arg_mem#152105
  Use reg 1 for stack arg#5
  Use reg 2 to represent mem_size
  mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
  mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
  mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
  mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
  mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2
  mark_precise: frame0: last_idx 13 first_idx 0 subseq_idx -1
  mark_precise: frame0: regs=r2 stack= before 12: (b7) r5 = 5
  mark_precise: frame0: regs=r2 stack= before 11: (b7) r4 = 4
  mark_precise: frame0: regs=r2 stack= before 10: (b7) r3 = 3
  mark_precise: frame0: regs=r2 stack= before 9: (b7) r2 = 2
  End of using reg 1 for stack arg#6
  14: R0=scalar()
  14: (95) exit

The above example is for KF_ARG_PTR_TO_MEM_SIZE case.
Registers 1 and 2 are used as an temporary register for argument
checking. The verifier log will identify when the temporary register
is used and when the temporary register is not used any more.

There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno'
is also keep regno for later fixup. Since regno is a faked one
such three cases are rejected for now if they are in stack arguments.
If possible, new kfuncs could keep them in first 5 registers so
there are no issues at all.

If faked register approach is not favored, the following is an
alternative approach with the following data structure:

   struct reg_or_arg_t {
     struct bpf_reg_state *state;
     union {
       int regno;
       int argno;
     };
     bool is_reg; /* distinguish between reg and arg */
   };

Such a struct can replace existing regno/argno/reg_state parameters
to make it easy to distinguish reg vs. arg.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 143 ++++++++++++++++++++++++++++++++++++------
 1 file changed, 124 insertions(+), 19 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 183a4108fd4d..dba889cece1c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3491,7 +3491,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
 	struct bpf_kfunc_meta kfunc;
 	struct bpf_kfunc_desc *desc;
 	unsigned long addr;
-	int err;
+	int i, err;
 
 	prog_aux = env->prog->aux;
 	tab = prog_aux->kfunc_tab;
@@ -3567,6 +3567,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset)
 	if (err)
 		return err;
 
+	for (i = MAX_BPF_FUNC_REG_ARGS; i < func_model.nr_args; i++) {
+		if (func_model.arg_size[i] > sizeof(u64)) {
+			verbose(env, "kfunc %s arg#%d size %d > %zu not supported for stack args\n",
+				kfunc.name, i, func_model.arg_size[i], sizeof(u64));
+			return -EINVAL;
+		}
+	}
+
 	desc = &tab->descs[tab->nr_descs++];
 	desc->func_id = func_id;
 	desc->offset = offset;
@@ -13083,6 +13091,19 @@ static bool is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)
 	return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data];
 }
 
+static struct bpf_reg_state *get_kfunc_arg_reg(struct bpf_verifier_env *env, int argno)
+{
+	struct bpf_func_state *caller;
+	int spi;
+
+	if (argno < MAX_BPF_FUNC_REG_ARGS)
+		return &cur_regs(env)[argno + 1];
+
+	caller = cur_func(env);
+	spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE + (argno - MAX_BPF_FUNC_REG_ARGS);
+	return &caller->stack_arg_slots[spi].spilled_ptr;
+}
+
 static enum kfunc_ptr_arg_type
 get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
 		       struct bpf_kfunc_call_arg_meta *meta,
@@ -13101,8 +13122,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
 		return KF_ARG_PTR_TO_CTX;
 
 	if (argno + 1 < nargs &&
-	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1]) ||
-	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1])))
+	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) ||
+	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1))))
 		arg_mem_size = true;
 
 	/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -13770,9 +13791,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 
 	args = (const struct btf_param *)(meta->func_proto + 1);
 	nargs = btf_type_vlen(meta->func_proto);
-	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+	if (nargs > MAX_BPF_FUNC_ARGS) {
 		verbose(env, "Function %s has %d > %d args\n", func_name, nargs,
-			MAX_BPF_FUNC_REG_ARGS);
+			MAX_BPF_FUNC_ARGS);
 		return -EINVAL;
 	}
 
@@ -13780,26 +13801,59 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 	 * verifier sees.
 	 */
 	for (i = 0; i < nargs; i++) {
-		struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[i + 1];
+		struct bpf_reg_state *regs = cur_regs(env), *reg;
+		struct bpf_reg_state saved_reg;
 		const struct btf_type *t, *ref_t, *resolve_ret;
 		enum bpf_arg_type arg_type = ARG_DONTCARE;
 		u32 regno = i + 1, ref_id, type_size;
 		bool is_ret_buf_sz = false;
+		bool is_stack_arg = false;
 		int kf_arg_type;
 
+		if (i < MAX_BPF_FUNC_REG_ARGS) {
+			reg = &regs[i + 1];
+		} else {
+			/*
+			 * Retrieve the spilled reg state from the stack arg slot.
+			 * Reuse the existing type checking infrastructure which
+			 * reads from cur_regs(env)[regno], temporarily copy the
+			 * stack arg reg state into regs[BPF_REG_1] and restore
+			 * it after checking.
+			 */
+			struct bpf_func_state *caller = cur_func(env);
+			int spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE +
+				  (i - MAX_BPF_FUNC_REG_ARGS);
+
+			if (!is_stack_arg_slot_initialized(caller, spi)) {
+				verbose(env, "stack arg#%d not properly initialized\n", i);
+				return -EINVAL;
+			}
+
+			is_stack_arg = true;
+			regno = BPF_REG_1;
+			saved_reg = regs[BPF_REG_1];
+			regs[BPF_REG_1] = caller->stack_arg_slots[spi].spilled_ptr;
+			reg = &regs[BPF_REG_1];
+			verbose(env, "Use reg %d for stack arg#%d\n", regno, i);
+		}
+
 		if (is_kfunc_arg_prog_aux(btf, &args[i])) {
 			/* Reject repeated use bpf_prog_aux */
 			if (meta->arg_prog) {
 				verifier_bug(env, "Only 1 prog->aux argument supported per-kfunc");
 				return -EFAULT;
 			}
+			if (is_stack_arg) {
+				verbose(env, "arg#%d prog->aux cannot be a stack argument\n", i);
+				return -EINVAL;
+			}
 			meta->arg_prog = true;
 			cur_aux(env)->arg_prog = regno;
-			continue;
+			goto next_arg;
 		}
 
 		if (is_kfunc_arg_ignore(btf, &args[i]) || is_kfunc_arg_implicit(meta, i))
-			continue;
+			goto next_arg;
 
 		t = btf_type_skip_modifiers(btf, args[i].type, NULL);
 
@@ -13818,9 +13872,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 					verbose(env, "R%d must be a known constant\n", regno);
 					return -EINVAL;
 				}
-				ret = mark_chain_precision(env, regno);
-				if (ret < 0)
-					return ret;
+				if (!is_stack_arg) {
+					ret = mark_chain_precision(env, regno);
+					if (ret < 0)
+						return ret;
+				}
 				meta->arg_constant.found = true;
 				meta->arg_constant.value = reg->var_off.value;
 			} else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) {
@@ -13842,11 +13898,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				}
 
 				meta->r0_size = reg->var_off.value;
-				ret = mark_chain_precision(env, regno);
-				if (ret)
-					return ret;
+				if (!is_stack_arg) {
+					ret = mark_chain_precision(env, regno);
+					if (ret)
+						return ret;
+				}
 			}
-			continue;
+			goto next_arg;
 		}
 
 		if (!btf_type_is_ptr(t)) {
@@ -13868,8 +13926,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				return -EFAULT;
 			}
 			meta->ref_obj_id = reg->ref_obj_id;
-			if (is_kfunc_release(meta))
+			if (is_kfunc_release(meta)) {
+				if (is_stack_arg) {
+					verbose(env, "arg#%d release arg cannot be a stack argument\n", i);
+					return -EINVAL;
+				}
 				meta->release_regno = regno;
+			}
 		}
 
 		ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
@@ -13881,7 +13944,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 
 		switch (kf_arg_type) {
 		case KF_ARG_PTR_TO_NULL:
-			continue;
+			goto next_arg;
 		case KF_ARG_PTR_TO_MAP:
 			if (!reg->map_ptr) {
 				verbose(env, "pointer in R%d isn't map pointer\n", regno);
@@ -14020,6 +14083,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				dynptr_arg_type |= DYNPTR_TYPE_FILE;
 			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) {
 				dynptr_arg_type |= DYNPTR_TYPE_FILE;
+				if (is_stack_arg) {
+					verbose(env, "arg#%d release arg cannot be a stack argument\n", i);
+					return -EINVAL;
+				}
 				meta->release_regno = regno;
 			} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] &&
 				   (dynptr_arg_type & MEM_UNINIT)) {
@@ -14169,8 +14236,18 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 		{
 			struct bpf_reg_state *buff_reg = &regs[regno];
 			const struct btf_param *buff_arg = &args[i];
-			struct bpf_reg_state *size_reg = &regs[regno + 1];
+			struct bpf_reg_state *size_reg;
 			const struct btf_param *size_arg = &args[i + 1];
+			struct bpf_reg_state saved_size_reg = {};
+			bool size_is_stack_arg = false;
+
+			if (i >= MAX_BPF_FUNC_REG_ARGS) {
+				size_is_stack_arg = true;
+				saved_size_reg = regs[regno + 1];
+				regs[regno + 1] = *get_kfunc_arg_reg(env, i + 1);
+				verbose(env, "Use reg %d to represent mem_size\n", regno + 1);
+			}
+			size_reg = &regs[regno + 1];
 
 			if (!register_is_null(buff_reg) || !is_kfunc_arg_nullable(meta->btf, buff_arg)) {
 				ret = check_kfunc_mem_size_reg(env, size_reg, regno + 1);
@@ -14193,6 +14270,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 				meta->arg_constant.value = size_reg->var_off.value;
 			}
 
+			if (size_is_stack_arg)
+				regs[regno + 1] = saved_size_reg;
+
 			/* Skip next '__sz' or '__szk' argument */
 			i++;
 			break;
@@ -14294,6 +14374,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
 			break;
 		}
 		}
+next_arg:
+		if (is_stack_arg) {
+			regs[regno] = saved_reg;
+			verbose(env, "End of using reg %d for stack arg#%d\n", regno, i);
+		}
 	}
 
 	if (is_kfunc_release(meta) && !meta->release_regno) {
@@ -15059,7 +15144,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
 	nargs = btf_type_vlen(meta.func_proto);
 	args = (const struct btf_param *)(meta.func_proto + 1);
-	for (i = 0; i < nargs; i++) {
+	for (i = 0; i < nargs && i < MAX_BPF_FUNC_REG_ARGS; i++) {
 		u32 regno = i + 1;
 
 		t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL);
@@ -15070,6 +15155,16 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 			mark_btf_func_reg_size(env, regno, t->size);
 	}
 
+	/* Track outgoing stack arg depth for kfuncs with >5 args */
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		struct bpf_func_state *caller = cur_func(env);
+		struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
+		u16 kfunc_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+
+		if (kfunc_stack_arg_depth > caller_info->outgoing_stack_arg_depth)
+			caller_info->outgoing_stack_arg_depth = kfunc_stack_arg_depth;
+	}
+
 	if (is_iter_next_kfunc(&meta)) {
 		err = process_iter_next_call(env, insn_idx, &meta);
 		if (err)
@@ -23975,6 +24070,16 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	if (!bpf_jit_supports_far_kfunc_call())
 		insn->imm = BPF_CALL_IMM(desc->addr);
 
+	/*
+	 * After resolving the kfunc address, insn->off is no longer needed
+	 * for BTF fd index. Repurpose it to store the number of stack args
+	 * so the JIT can marshal them.
+	 */
+	if (desc->func_model.nr_args > MAX_BPF_FUNC_REG_ARGS)
+		insn->off = desc->func_model.nr_args - MAX_BPF_FUNC_REG_ARGS;
+	else
+		insn->off = 0;
+
 	if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
 		struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;
 		struct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) };
-- 
2.52.0

[PATCH bpf-next v3 06/11] bpf: Reject stack arguments in non-JITed programs

[Yonghong Song]

The interpreter does not understand the bpf register r12
(BPF_REG_STACK_ARG_BASE) used for stack argument addressing. So
reject interpreter usage if stack arguments are used either
in the main program or any subprogram.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/core.c     | 3 ++-
 kernel/bpf/verifier.c | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 3520337a1c0e..a04b31eb4c49 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2553,7 +2553,8 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) ||
+	    fp->aux->stack_arg_depth)
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index dba889cece1c..019840010b43 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -23944,6 +23944,12 @@ static int fixup_call_args(struct bpf_verifier_env *env)
 		verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
 		return -EINVAL;
 	}
+	for (i = 0; i < env->subprog_cnt; i++) {
+		if (env->subprog_info[i].incoming_stack_arg_depth) {
+			verbose(env, "stack args are not supported in non-JITed programs\n");
+			return -EINVAL;
+		}
+	}
 	if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
 		/* When JIT fails the progs with bpf2bpf calls and tail_calls
 		 * have to be rejected, since interpreter doesn't support them yet.
-- 
2.52.0

[PATCH bpf-next v3 07/11] bpf: Enable stack argument support for x86_64

[Yonghong Song]

Add stack argument support for x86_64.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 arch/x86/net/bpf_jit_comp.c | 5 +++++
 include/linux/filter.h      | 1 +
 kernel/bpf/btf.c            | 9 ++++++++-
 kernel/bpf/core.c           | 5 +++++
 4 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index e9b78040d703..32864dbc2c4e 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3937,6 +3937,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_stack_args(void)
+{
+	return true;
+}
+
 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 {
 	if (text_poke_copy(dst, src, len) == NULL)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 68f018dd4b9c..a5035fb80a6b 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1160,6 +1160,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
 bool bpf_jit_supports_ptr_xchg(void);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index c5f3aa05d5a3..1cbe0f2b0e41 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -20,6 +20,7 @@
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
 #include <linux/bpf.h>
+#include <linux/filter.h>
 #include <linux/bpf_lsm.h>
 #include <linux/skmsg.h>
 #include <linux/perf_event.h>
@@ -7897,8 +7898,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 			tname, nargs, MAX_BPF_FUNC_REG_ARGS);
 		return -EINVAL;
 	}
-	if (nargs > MAX_BPF_FUNC_REG_ARGS)
+	if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+		if (!bpf_jit_supports_stack_args()) {
+			bpf_log(log, "JIT does not support function %s() with %d args\n",
+				tname, nargs);
+			return -ENOTSUPP;
+		}
 		sub->incoming_stack_arg_depth = (nargs - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE;
+	}
 
 	/* check that function is void or returns int, exception cb also requires this */
 	t = btf_type_by_id(btf, t->type);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index a04b31eb4c49..01de4f7f3a82 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3156,6 +3156,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_stack_args(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_far_kfunc_call(void)
 {
 	return false;
-- 
2.52.0

[PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Yonghong Song]

Add x86_64 JIT support for BPF functions and kfuncs with more than
5 arguments. The extra arguments are passed through a stack area
addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode,
which the JIT translates to RBP-relative accesses in native code.

The JIT follows the native x86_64 calling convention for stack
argument placement. Incoming stack args from the caller sit above
the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly
where x86_64 expects them after CALL + PUSH RBP. Only the outgoing
stack arg area is allocated below the program stack in the prologue.

The native x86_64 stack layout for a function with incoming and
outgoing stack args:

  high address
  ┌─────────────────────────┐
  │ incoming stack arg N    │  [rbp + 16 + (N-1)*8]  (from caller)
  │ ...                     │
  │ incoming stack arg 1    │  [rbp + 16]
  ├─────────────────────────┤
  │ return address          │  [rbp + 8]
  │ saved rbp               │  [rbp]
  ├─────────────────────────┤
  │ BPF program stack       │  (stack_depth bytes)
  ├─────────────────────────┤
  │ outgoing stack arg 1    │  [rbp - prog_stack_depth - outgoing_depth]
  │ ...                     │   (written via r12-relative STX/ST)
  │ outgoing stack arg M    │  [rbp - prog_stack_depth - 8]
  ├─────────────────────────┤
  │ callee-saved regs ...   │  (pushed after sub rsp)
  └─────────────────────────┘  rsp
  low address

BPF r12-relative offsets are translated to native RBP-relative
offsets with two formulas:
  - Incoming args (load: -off <= incoming_depth):
      native_off = 8 - bpf_off  →  [rbp + 16 + ...]
  - Outgoing args (store: -off > incoming_depth):
      native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off

Since callee-saved registers are pushed below the outgoing area,
outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF
calls and kfunc calls, outgoing args are explicitly pushed from the
outgoing area onto the stack before CALL and rsp is restored after return.

For kfunc calls specifically, arg 6 is loaded into R9 and args 7+
are pushed onto the native stack, per the x86_64 calling convention.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
 1 file changed, 129 insertions(+), 6 deletions(-)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 32864dbc2c4e..206f342a0ca0 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
 	*pprog = prog;
 }
 
+/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
+static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
+{
+	u8 *prog = *pprog;
+	int k, bytes = 0;
+	s32 off;
+
+	for (k = from; k >= to; k--) {
+		off = outgoing_base + (k - 1) * 8;
+		/* push qword [rbp + off] */
+		if (is_imm8(off)) {
+			EMIT3(0xFF, 0x75, off);
+			bytes += 3;
+		} else {
+			EMIT2_off32(0xFF, 0xB5, off);
+			bytes += 6;
+		}
+	}
+	*pprog = prog;
+	return bytes;
+}
+
 static void emit_nops(u8 **pprog, int len)
 {
 	u8 *prog = *pprog;
@@ -1664,16 +1686,33 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	int i, excnt = 0;
 	int ilen, proglen = 0;
 	u8 *prog = temp;
-	u32 stack_depth;
+	u16 stack_arg_depth, incoming_stack_arg_depth, outgoing_stack_arg_depth;
+	u32 prog_stack_depth, stack_depth;
+	bool has_stack_args;
 	int err;
 
 	stack_depth = bpf_prog->aux->stack_depth;
+	stack_arg_depth = bpf_prog->aux->stack_arg_depth;
+	incoming_stack_arg_depth = bpf_prog->aux->incoming_stack_arg_depth;
+	outgoing_stack_arg_depth = stack_arg_depth - incoming_stack_arg_depth;
 	priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
 	if (priv_stack_ptr) {
 		priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
 		stack_depth = 0;
 	}
 
+	/*
+	 * Save program stack depth before adding outgoing stack arg space.
+	 * Incoming stack args are read directly from [rbp + 16 + ...].
+	 * Only the outgoing stack arg area is allocated below the
+	 * program stack. Outgoing args written here become the callee's
+	 * incoming args.
+	 */
+	prog_stack_depth = round_up(stack_depth, 8);
+	if (outgoing_stack_arg_depth)
+		stack_depth += outgoing_stack_arg_depth;
+	has_stack_args = stack_arg_depth > 0;
+
 	arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
 	user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
 
@@ -1715,13 +1754,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	prog = temp;
 
 	for (i = 1; i <= insn_cnt; i++, insn++) {
+		bool adjust_stack_arg_off = false;
 		const s32 imm32 = insn->imm;
 		u32 dst_reg = insn->dst_reg;
 		u32 src_reg = insn->src_reg;
 		u8 b2 = 0, b3 = 0;
 		u8 *start_of_ldx;
 		s64 jmp_offset;
-		s16 insn_off;
+		s32 insn_off;
 		u8 jmp_cond;
 		u8 *func;
 		int nops;
@@ -1734,6 +1774,37 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 				dst_reg = X86_REG_R9;
 		}
 
+		if (has_stack_args) {
+			u8 class = BPF_CLASS(insn->code);
+
+			if (class == BPF_LDX &&
+			    src_reg == BPF_REG_STACK_ARG_BASE) {
+				src_reg = BPF_REG_FP;
+				adjust_stack_arg_off = true;
+			}
+			if ((class == BPF_STX || class == BPF_ST) &&
+			    dst_reg == BPF_REG_STACK_ARG_BASE) {
+				dst_reg = BPF_REG_FP;
+				adjust_stack_arg_off = true;
+			}
+		}
+
+		/*
+		 * Translate BPF r12-relative offset to native RBP-relative:
+		 *
+		 * Incoming args (load: offset >= -incoming_depth):
+		 *   BPF: r12 + bpf_off = r12 - k * 8 (k = 1,2,...) for incoming arg k
+		 *   Native: [rbp + 8 + k * 8]
+		 *   Formula: native_off = 8 + k * 8 = 8 - bpf_off
+		 *
+		 * Outgoing args (store: offset < -incoming_depth):
+		 *   BPF: r12 + bpf_off = r12 - (incoming + k * 8) for outgoing arg k
+		 *   Native: [rbp - prog_stack_depth - outgoing + (k - 1) * 8]
+		 *   Formula: native_off = -(prog_stack_depth + outgoing) + (k - 1) * 8
+		 *          = -(prog_stack_depth + outgoing + incoming + 8) - bpf_off
+		 *          = -(prog_stack_depth + stack_arg_depth + 8) - bpf_off
+		 */
+
 		switch (insn->code) {
 			/* ALU */
 		case BPF_ALU | BPF_ADD | BPF_X:
@@ -2131,10 +2202,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 		case BPF_ST | BPF_MEM | BPF_DW:
 			EMIT2(add_1mod(0x48, dst_reg), 0xC7);
 
-st:			if (is_imm8(insn->off))
-				EMIT2(add_1reg(0x40, dst_reg), insn->off);
+st:			insn_off = insn->off;
+			if (adjust_stack_arg_off)
+				insn_off = -(prog_stack_depth + stack_arg_depth + 8) - insn_off;
+			if (is_imm8(insn_off))
+				EMIT2(add_1reg(0x40, dst_reg), insn_off);
 			else
-				EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);
+				EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
 
 			EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
 			break;
@@ -2144,7 +2218,10 @@ st:			if (is_imm8(insn->off))
 		case BPF_STX | BPF_MEM | BPF_H:
 		case BPF_STX | BPF_MEM | BPF_W:
 		case BPF_STX | BPF_MEM | BPF_DW:
-			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+			insn_off = insn->off;
+			if (adjust_stack_arg_off)
+				insn_off = -(prog_stack_depth + stack_arg_depth + 8) - insn_off;
+			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			break;
 
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
@@ -2243,6 +2320,8 @@ st:			if (is_imm8(insn->off))
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
 		case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
 			insn_off = insn->off;
+			if (adjust_stack_arg_off)
+				insn_off = 8 - insn_off;
 
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
 			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2441,6 +2520,7 @@ st:			if (is_imm8(insn->off))
 			/* call */
 		case BPF_JMP | BPF_CALL: {
 			u8 *ip = image + addrs[i - 1];
+			int stack_args = 0;
 
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
@@ -2449,6 +2529,41 @@ st:			if (is_imm8(insn->off))
 			}
 			if (!imm32)
 				return -EINVAL;
+
+			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
+				/*
+				 * BPF-to-BPF calls: push outgoing stack args from
+				 * the outgoing area onto the stack before CALL.
+				 * The outgoing area is at [rbp - prog_stack - outgoing],
+				 * but rsp is below that due to callee-saved reg pushes,
+				 * so we must explicitly push args for the callee.
+				*/
+				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
+				int n_args = outgoing_stack_arg_depth / 8;
+
+				ip += push_stack_args(&prog, outgoing_base, n_args, 1);
+			}
+
+			if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
+				/* Kfunc calls: arg 6 → R9, args 7+ → push. */
+				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
+				int kfunc_stack_args = insn->off;
+
+				stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
+
+				/* Push args 7+ in reverse order */
+				if (stack_args > 0)
+					ip += push_stack_args(&prog, outgoing_base, kfunc_stack_args, 2);
+
+				/* mov r9, [rbp + outgoing_base] (arg 6) */
+				if (is_imm8(outgoing_base)) {
+					EMIT4(0x4C, 0x8B, 0x4D, outgoing_base);
+					ip += 4;
+				} else {
+					EMIT3_off32(0x4C, 0x8B, 0x8D, outgoing_base);
+					ip += 7;
+				}
+			}
 			if (priv_frame_ptr) {
 				push_r9(&prog);
 				ip += 2;
@@ -2458,6 +2573,14 @@ st:			if (is_imm8(insn->off))
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			if (stack_args > 0) {
+				/* add rsp, stack_args * 8 */
+				EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
+			}
+			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
+				/* add rsp, outgoing_stack_arg_depth */
+				EMIT4(0x48, 0x83, 0xC4, outgoing_stack_arg_depth);
+			}
 			break;
 		}
 
-- 
2.52.0

[PATCH bpf-next v3 09/11] selftests/bpf: Add tests for BPF function stack arguments

[Yonghong Song]

Add selftests covering stack argument passing for both BPF-to-BPF
subprog calls and kfunc calls with more than 5 arguments. All tests
are guarded by __BPF_FEATURE_STACK_ARGUMENT and __TARGET_ARCH_x86.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/stack_arg.c      | 132 +++++++++++
 tools/testing/selftests/bpf/progs/stack_arg.c | 212 ++++++++++++++++++
 .../selftests/bpf/progs/stack_arg_kfunc.c     | 164 ++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  65 ++++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  20 +-
 5 files changed, 592 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c

diff --git a/tools/testing/selftests/bpf/prog_tests/stack_arg.c b/tools/testing/selftests/bpf/prog_tests/stack_arg.c
new file mode 100644
index 000000000000..1af5e5c91e0d
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/stack_arg.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "stack_arg.skel.h"
+#include "stack_arg_kfunc.skel.h"
+
+static void run_subtest(struct bpf_program *prog, int expected)
+{
+	int err, prog_fd;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = &pkt_v4,
+		.data_size_in = sizeof(pkt_v4),
+		.repeat = 1,
+	);
+
+	prog_fd = bpf_program__fd(prog);
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	ASSERT_OK(err, "test_run");
+	ASSERT_EQ(topts.retval, expected, "retval");
+}
+
+static void test_global_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_global_many_args, 36);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_async_cb_many(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_async_cb_many_args, 0);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_bpf2bpf(void)
+{
+	struct stack_arg *skel;
+
+	skel = stack_arg__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_bpf2bpf_ptr_stack_arg, 45);
+	run_subtest(skel->progs.test_bpf2bpf_mix_stack_args, 51);
+	run_subtest(skel->progs.test_bpf2bpf_nesting_stack_arg, 50);
+	run_subtest(skel->progs.test_bpf2bpf_dynptr_stack_arg, 69);
+
+out:
+	stack_arg__destroy(skel);
+}
+
+static void test_kfunc(void)
+{
+	struct stack_arg_kfunc *skel;
+
+	skel = stack_arg_kfunc__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	if (!ASSERT_OK(stack_arg_kfunc__load(skel), "load"))
+		goto out;
+
+	run_subtest(skel->progs.test_stack_arg_scalar, 36);
+	run_subtest(skel->progs.test_stack_arg_ptr, 45);
+	run_subtest(skel->progs.test_stack_arg_mix, 51);
+	run_subtest(skel->progs.test_stack_arg_dynptr, 69);
+	run_subtest(skel->progs.test_stack_arg_mem, 151);
+	run_subtest(skel->progs.test_stack_arg_iter, 115);
+	run_subtest(skel->progs.test_stack_arg_const_str, 15);
+	run_subtest(skel->progs.test_stack_arg_timer, 15);
+
+out:
+	stack_arg_kfunc__destroy(skel);
+}
+
+void test_stack_arg(void)
+{
+	if (test__start_subtest("global_many_args"))
+		test_global_many();
+	if (test__start_subtest("async_cb_many_args"))
+		test_async_cb_many();
+	if (test__start_subtest("bpf2bpf"))
+		test_bpf2bpf();
+	if (test__start_subtest("kfunc"))
+		test_kfunc();
+}
diff --git a/tools/testing/selftests/bpf/progs/stack_arg.c b/tools/testing/selftests/bpf/progs/stack_arg.c
new file mode 100644
index 000000000000..c6bf89c3a0db
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/stack_arg.c
@@ -0,0 +1,212 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <stdbool.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_kfuncs.h"
+
+#define CLOCK_MONOTONIC 1
+
+long a, b, c, d, e, f, g, i;
+
+struct timer_elem {
+	struct bpf_timer timer;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer_elem);
+} timer_map SEC(".maps");
+
+int timer_result;
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+__noinline static int static_func_many_args(int a, int b, int c, int d,
+					    int e, int f, int g, int h)
+{
+	return a + b + c + d + e + f + g + h;
+}
+
+__noinline int global_calls_many_args(int a, int b, int c)
+{
+	return static_func_many_args(a, b, c, 4, 5, 6, 7, 8);
+}
+
+SEC("tc")
+int test_global_many_args(void)
+{
+	return global_calls_many_args(1, 2, 3);
+}
+
+struct test_data {
+	long x;
+	long y;
+};
+
+/* 1 + 2 + 3 + 4 + 5 + 10 + 20 = 45 */
+__noinline static long func_with_ptr_stack_arg(long a, long b, long c, long d,
+					       long e, struct test_data *p)
+{
+	return a + b + c + d + e + p->x + p->y;
+}
+
+__noinline long global_ptr_stack_arg(long a, long b, long c, long d, long e)
+{
+	struct test_data data = { .x = 10, .y = 20 };
+
+	return func_with_ptr_stack_arg(a, b, c, d, e, &data);
+}
+
+SEC("tc")
+int test_bpf2bpf_ptr_stack_arg(void)
+{
+	return global_ptr_stack_arg(1, 2, 3, 4, 5);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + 10 + 6 + 20 = 51 */
+__noinline static long func_with_mix_stack_args(long a, long b, long c, long d,
+						long e, struct test_data *p,
+						long f, struct test_data *q)
+{
+	return a + b + c + d + e + p->x + f + q->y;
+}
+
+__noinline long global_mix_stack_args(long a, long b, long c, long d, long e)
+{
+	struct test_data p = { .x = 10 };
+	struct test_data q = { .y = 20 };
+
+	return func_with_mix_stack_args(a, b, c, d, e, &p, e + 1, &q);
+}
+
+SEC("tc")
+int test_bpf2bpf_mix_stack_args(void)
+{
+	return global_mix_stack_args(1, 2, 3, 4, 5);
+}
+
+/*
+ * Nesting test: func_outer calls func_inner, both with struct pointer
+ * as stack arg.
+ *
+ * func_inner: (a+1) + (b+1) + (c+1) + (d+1) + (e+1) + p->x + p->y
+ *           = 2 + 3 + 4 + 5 + 6 + 10 + 20 = 50
+ */
+__noinline static long func_inner_ptr(long a, long b, long c, long d,
+				      long e, struct test_data *p)
+{
+	return a + b + c + d + e + p->x + p->y;
+}
+
+__noinline static long func_outer_ptr(long a, long b, long c, long d,
+				      long e, struct test_data *p)
+{
+	return func_inner_ptr(a + 1, b + 1, c + 1, d + 1, e + 1, p);
+}
+
+__noinline long global_nesting_ptr(long a, long b, long c, long d, long e)
+{
+	struct test_data data = { .x = 10, .y = 20 };
+
+	return func_outer_ptr(a, b, c, d, e, &data);
+}
+
+SEC("tc")
+int test_bpf2bpf_nesting_stack_arg(void)
+{
+	return global_nesting_ptr(1, 2, 3, 4, 5);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + sizeof(pkt_v4) = 15 + 54 = 69 */
+__noinline static long func_with_dynptr(long a, long b, long c, long d,
+					long e, struct bpf_dynptr *ptr)
+{
+	return a + b + c + d + e + bpf_dynptr_size(ptr);
+}
+
+__noinline long global_dynptr_stack_arg(void *ctx __arg_ctx, long a, long b,
+					long c, long d)
+{
+	struct bpf_dynptr ptr;
+
+	bpf_dynptr_from_skb(ctx, 0, &ptr);
+	return func_with_dynptr(a, b, c, d, d + 1, &ptr);
+}
+
+SEC("tc")
+int test_bpf2bpf_dynptr_stack_arg(struct __sk_buff *skb)
+{
+	return global_dynptr_stack_arg(skb, 1, 2, 3, 4);
+}
+
+static int timer_cb_many_args(void *map, int *key, struct bpf_timer *timer)
+{
+	timer_result = static_func_many_args(10, 20, 30, 40, 50, 60, 70, 80);
+	return 0;
+}
+
+SEC("tc")
+int test_async_cb_many_args(void)
+{
+	struct timer_elem *elem;
+	int key = 0;
+
+	elem = bpf_map_lookup_elem(&timer_map, &key);
+	if (!elem)
+		return -1;
+
+	bpf_timer_init(&elem->timer, &timer_map, CLOCK_MONOTONIC);
+	bpf_timer_set_callback(&elem->timer, timer_cb_many_args);
+	bpf_timer_start(&elem->timer, 1, 0);
+	return 0;
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_global_many_args(void)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_ptr_stack_arg(void)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_mix_stack_args(void)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_nesting_stack_arg(void)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_bpf2bpf_dynptr_stack_arg(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_async_cb_many_args(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/stack_arg_kfunc.c b/tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
new file mode 100644
index 000000000000..6cc404d57863
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
@@ -0,0 +1,164 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_kfuncs.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+struct bpf_iter_testmod_seq {
+	u64 :64;
+	u64 :64;
+};
+
+extern int bpf_iter_testmod_seq_new(struct bpf_iter_testmod_seq *it, s64 value, int cnt) __ksym;
+extern int *bpf_iter_testmod_seq_next(struct bpf_iter_testmod_seq *it) __ksym;
+extern void bpf_iter_testmod_seq_destroy(struct bpf_iter_testmod_seq *it) __ksym;
+
+struct timer_map_value {
+	struct bpf_timer timer;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct timer_map_value);
+} kfunc_timer_map SEC(".maps");
+
+SEC("tc")
+int test_stack_arg_scalar(struct __sk_buff *skb)
+{
+	return bpf_kfunc_call_stack_arg(1, 2, 3, 4, 5, 6, 7, 8);
+}
+
+SEC("tc")
+int test_stack_arg_ptr(struct __sk_buff *skb)
+{
+	struct prog_test_pass1 p = { .x0 = 10, .x1 = 20 };
+
+	return bpf_kfunc_call_stack_arg_ptr(1, 2, 3, 4, 5, &p);
+}
+
+SEC("tc")
+int test_stack_arg_mix(struct __sk_buff *skb)
+{
+	struct prog_test_pass1 p = { .x0 = 10 };
+	struct prog_test_pass1 q = { .x1 = 20 };
+
+	return bpf_kfunc_call_stack_arg_mix(1, 2, 3, 4, 5, &p, 6, &q);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + sizeof(pkt_v4) = 15 + 54 = 69 */
+SEC("tc")
+int test_stack_arg_dynptr(struct __sk_buff *skb)
+{
+	struct bpf_dynptr ptr;
+
+	bpf_dynptr_from_skb(skb, 0, &ptr);
+	return bpf_kfunc_call_stack_arg_dynptr(1, 2, 3, 4, 5, &ptr);
+}
+
+/* 1 + 2 + 3 + 4 + 5 + (1 + 2 + ... + 16) = 15 + 136 = 151 */
+SEC("tc")
+int test_stack_arg_mem(struct __sk_buff *skb)
+{
+	char buf[16] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16};
+
+	return bpf_kfunc_call_stack_arg_mem(1, 2, 3, 4, 5, buf, sizeof(buf));
+}
+
+/* 1 + 2 + 3 + 4 + 5 + 100 = 115 */
+SEC("tc")
+int test_stack_arg_iter(struct __sk_buff *skb)
+{
+	struct bpf_iter_testmod_seq it;
+	u64 ret;
+
+	bpf_iter_testmod_seq_new(&it, 100, 10);
+	ret = bpf_kfunc_call_stack_arg_iter(1, 2, 3, 4, 5, &it);
+	bpf_iter_testmod_seq_destroy(&it);
+	return ret;
+}
+
+const char cstr[] = "hello";
+
+/* 1 + 2 + 3 + 4 + 5 = 15 */
+SEC("tc")
+int test_stack_arg_const_str(struct __sk_buff *skb)
+{
+	return bpf_kfunc_call_stack_arg_const_str(1, 2, 3, 4, 5, cstr);
+}
+
+/* 1 + 2 + 3 + 4 + 5 = 15 */
+SEC("tc")
+int test_stack_arg_timer(struct __sk_buff *skb)
+{
+	struct timer_map_value *val;
+	int key = 0;
+
+	val = bpf_map_lookup_elem(&kfunc_timer_map, &key);
+	if (!val)
+		return 0;
+	return bpf_kfunc_call_stack_arg_timer(1, 2, 3, 4, 5, &val->timer);
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_stack_arg_scalar(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_stack_arg_ptr(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_stack_arg_mix(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_stack_arg_dynptr(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_stack_arg_mem(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_stack_arg_iter(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_stack_arg_const_str(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int test_stack_arg_timer(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 061356f10093..4fcdb13e4a9a 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -824,6 +824,63 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
+					 u64 e, u64 f, u64 g, u64 h)
+{
+	return a + b + c + d + e + f + g + h;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_ptr(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_pass1 *p)
+{
+	return a + b + c + d + e + p->x0 + p->x1;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_mix(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_pass1 *p, u64 f,
+					     struct prog_test_pass1 *q)
+{
+	return a + b + c + d + e + p->x0 + f + q->x1;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_dynptr(u64 a, u64 b, u64 c, u64 d, u64 e,
+					       struct bpf_dynptr *ptr)
+{
+	const struct bpf_dynptr_kern *kern_ptr = (void *)ptr;
+
+	return a + b + c + d + e + (kern_ptr->size & 0xFFFFFF);
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_mem(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     void *mem, int mem__sz)
+{
+	const unsigned char *p = mem;
+	u64 sum = a + b + c + d + e;
+	int i;
+
+	for (i = 0; i < mem__sz; i++)
+		sum += p[i];
+	return sum;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_iter(u64 a, u64 b, u64 c, u64 d, u64 e,
+					      struct bpf_iter_testmod_seq *it__iter)
+{
+	return a + b + c + d + e + it__iter->value;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_const_str(u64 a, u64 b, u64 c, u64 d, u64 e,
+						   const char *str__str)
+{
+	return a + b + c + d + e;
+}
+
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 e,
+					       struct bpf_timer *timer)
+{
+	return a + b + c + d + e;
+}
+
 static struct prog_test_ref_kfunc prog_test_struct = {
 	.a = 42,
 	.b = 108,
@@ -1287,6 +1344,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_dynptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index aa0b8d41e71b..2c1cb118f886 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -26,6 +26,8 @@ struct prog_test_ref_kfunc {
 };
 #endif
 
+struct bpf_iter_testmod_seq;
+
 struct prog_test_pass1 {
 	int x0;
 	struct {
@@ -111,7 +113,23 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
-
+__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
+			       __u64 e, __u64 f, __u64 g, __u64 h) __ksym;
+__u64 bpf_kfunc_call_stack_arg_ptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mix(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_pass1 *p, __u64 f,
+				   struct prog_test_pass1 *q) __ksym;
+__u64 bpf_kfunc_call_stack_arg_dynptr(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				      struct bpf_dynptr *ptr) __ksym;
+__u64 bpf_kfunc_call_stack_arg_mem(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   void *mem, int mem__sz) __ksym;
+__u64 bpf_kfunc_call_stack_arg_iter(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				    struct bpf_iter_testmod_seq *it__iter) __ksym;
+__u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+					 const char *str__str) __ksym;
+__u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				     struct bpf_timer *timer) __ksym;
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
 void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
-- 
2.52.0

[PATCH bpf-next v3 10/11] selftests/bpf: Add negative test for greater-than-8-byte kfunc stack argument

[Yonghong Song]

Add a test that the verifier rejects kfunc calls where a stack argument
exceeds 8 bytes (the register-sized slot limit).

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/stack_arg_fail.c | 24 ++++++++++++++
 .../selftests/bpf/progs/stack_arg_fail.c      | 32 +++++++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  7 ++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  8 +++++
 4 files changed, 71 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c

diff --git a/tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c b/tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
new file mode 100644
index 000000000000..328a79edee45
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <test_progs.h>
+#include "stack_arg_fail.skel.h"
+
+void test_stack_arg_fail(void)
+{
+	struct stack_arg_fail *skel;
+
+	skel = stack_arg_fail__open();
+	if (!ASSERT_OK_PTR(skel, "open"))
+		return;
+
+	if (!skel->rodata->has_stack_arg) {
+		test__skip();
+		goto out;
+	}
+
+	ASSERT_ERR(stack_arg_fail__load(skel), "load_should_fail");
+
+out:
+	stack_arg_fail__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/stack_arg_fail.c b/tools/testing/selftests/bpf/progs/stack_arg_fail.c
new file mode 100644
index 000000000000..caa63b6f6a80
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/stack_arg_fail.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+const volatile bool has_stack_arg = true;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+	struct prog_test_big_arg s = { .a = 1, .b = 2 };
+
+	return bpf_kfunc_call_stack_arg_big(1, 2, 3, 4, 5, s);
+}
+
+#else
+
+const volatile bool has_stack_arg = false;
+
+SEC("tc")
+int test_stack_arg_big(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 4fcdb13e4a9a..861c564cc330 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -881,6 +881,12 @@ __bpf_kfunc u64 bpf_kfunc_call_stack_arg_timer(u64 a, u64 b, u64 c, u64 d, u64 e
 	return a + b + c + d + e;
 }
 
+__bpf_kfunc u64 bpf_kfunc_call_stack_arg_big(u64 a, u64 b, u64 c, u64 d, u64 e,
+					     struct prog_test_big_arg s)
+{
+	return a + b + c + d + e + s.a + s.b;
+}
+
 static struct prog_test_ref_kfunc prog_test_struct = {
 	.a = 42,
 	.b = 108,
@@ -1352,6 +1358,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mem)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_iter)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_const_str)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_timer)
+BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail1)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_fail2)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 2c1cb118f886..2a40f80b074a 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -50,6 +50,11 @@ struct prog_test_pass2 {
 	} x;
 };
 
+struct prog_test_big_arg {
+	long a;
+	long b;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -130,6 +135,9 @@ __u64 bpf_kfunc_call_stack_arg_const_str(__u64 a, __u64 b, __u64 c, __u64 d, __u
 					 const char *str__str) __ksym;
 __u64 bpf_kfunc_call_stack_arg_timer(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
 				     struct bpf_timer *timer) __ksym;
+__u64 bpf_kfunc_call_stack_arg_big(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
+				   struct prog_test_big_arg s) __ksym;
+
 void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) __ksym;
 void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) __ksym;
 void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym;
-- 
2.52.0

[PATCH bpf-next v3 11/11] selftests/bpf: Add verifier tests for stack argument validation

[Yonghong Song]

Add inline-asm-based verifier tests that exercise the stack argument
validation logic directly.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/verifier_stack_arg.c  | 302 ++++++++++++++++++
 2 files changed, 304 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 1ac366fd4dae..f39ed1f28a7b 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -90,6 +90,7 @@
 #include "verifier_sockmap_mutate.skel.h"
 #include "verifier_spill_fill.skel.h"
 #include "verifier_spin_lock.skel.h"
+#include "verifier_stack_arg.skel.h"
 #include "verifier_stack_ptr.skel.h"
 #include "verifier_store_release.skel.h"
 #include "verifier_subprog_precision.skel.h"
@@ -236,6 +237,7 @@ void test_verifier_sock_addr(void)            { RUN(verifier_sock_addr); }
 void test_verifier_sockmap_mutate(void)       { RUN(verifier_sockmap_mutate); }
 void test_verifier_spill_fill(void)           { RUN(verifier_spill_fill); }
 void test_verifier_spin_lock(void)            { RUN(verifier_spin_lock); }
+void test_verifier_stack_arg(void)            { RUN(verifier_stack_arg); }
 void test_verifier_stack_ptr(void)            { RUN(verifier_stack_ptr); }
 void test_verifier_store_release(void)        { RUN(verifier_store_release); }
 void test_verifier_subprog_precision(void)    { RUN(verifier_subprog_precision); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
new file mode 100644
index 000000000000..43a0636d995d
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -0,0 +1,302 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, long long);
+	__type(value, long long);
+} map_hash_8b SEC(".maps");
+
+#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+__noinline __used
+static int subprog_6args(int a, int b, int c, int d, int e, int f)
+{
+	return a + b + c + d + e + f;
+}
+
+__noinline __used
+static int subprog_7args(int a, int b, int c, int d, int e, int f, int g)
+{
+	return a + b + c + d + e + f + g;
+}
+
+SEC("tc")
+__description("stack_arg: subprog with 6 args")
+__success
+__arch_x86_64
+__naked void stack_arg_6args(void)
+{
+	asm volatile (
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"*(u64 *)(r12 - 8) = 6;"
+		"call subprog_6args;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("tc")
+__description("stack_arg: two subprogs with >5 args")
+__success
+__arch_x86_64
+__naked void stack_arg_two_subprogs(void)
+{
+	asm volatile (
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"*(u64 *)(r12 - 8) = 10;"
+		"call subprog_6args;"
+		"r6 = r0;"
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"*(u64 *)(r12 - 16) = 30;"
+		"*(u64 *)(r12 - 8) = 20;"
+		"call subprog_7args;"
+		"r0 += r6;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("tc")
+__description("stack_arg: read from uninitialized stack arg slot")
+__failure
+__arch_x86_64
+__msg("invalid read from stack arg")
+__naked void stack_arg_read_uninitialized(void)
+{
+	asm volatile (
+		"r0 = *(u64 *)(r12 - 8);"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("tc")
+__description("stack_arg: gap at offset -8, only wrote -16")
+__failure
+__arch_x86_64
+__msg("stack arg#5 not properly initialized")
+__naked void stack_arg_gap_at_minus8(void)
+{
+	asm volatile (
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"*(u64 *)(r12 - 16) = 30;"
+		"call subprog_7args;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("tc")
+__description("stack_arg: incorrect size of stack arg write")
+__failure
+__arch_x86_64
+__msg("stack arg write must be 8 bytes, got 4")
+__naked void stack_arg_not_written(void)
+{
+	asm volatile (
+		"r1 = 1;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"*(u32 *)(r12 - 8) = 30;"
+		"call subprog_6args;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+__noinline __used
+static long subprog_stack_arg_pruning_deref(int a, int b, int c, int d, int e, int f)
+{
+	long local = 0, *p, *ptr;
+
+	if (a <= 3) {
+		/* Overwrite the stack arg slot with a pointer to local */
+		p = &local;
+		asm volatile ("*(u64 *)(r12 - 8) = %[p];" :: [p] "r"(p) : "memory");
+	}
+
+	/* Read back the (possibly overwritten) stack arg slot */
+	asm volatile ("%[ptr] = *(u64 *)(r12 - 8);" : [ptr] "=r"(ptr) :: "memory");
+
+	/* Deref: safe for PTR_TO_STACK, unsafe for scalar */
+	return *ptr;
+}
+
+SEC("tc")
+__description("stack_arg: pruning with different stack arg types")
+__failure
+__flag(BPF_F_TEST_STATE_FREQ)
+__arch_x86_64
+__msg("invalid mem access 'scalar'")
+__naked void stack_arg_pruning_type_mismatch(void)
+{
+	asm volatile (
+		"call %[bpf_get_prandom_u32];"
+		"r1 = r0;"
+		"r2 = 2;"
+		"r3 = 3;"
+		"r4 = 4;"
+		"r5 = 5;"
+		"*(u64 *)(r12 - 8) = 42;"
+		"call subprog_stack_arg_pruning_deref;"
+		"exit;"
+		:: __imm(bpf_get_prandom_u32)
+		: __clobber_all
+	);
+}
+
+SEC("tc")
+__description("stack_arg: release_reference invalidates stack arg slot")
+__failure
+__arch_x86_64
+__msg("R0 invalid mem access 'scalar'")
+__naked void stack_arg_release_ref(void)
+{
+	asm volatile (
+		"r6 = r1;"
+		/* struct bpf_sock_tuple tuple = {} */
+		"r2 = 0;"
+		"*(u32 *)(r10 - 8) = r2;"
+		"*(u64 *)(r10 - 16) = r2;"
+		"*(u64 *)(r10 - 24) = r2;"
+		"*(u64 *)(r10 - 32) = r2;"
+		"*(u64 *)(r10 - 40) = r2;"
+		"*(u64 *)(r10 - 48) = r2;"
+		/* sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof(tuple), 0, 0) */
+		"r1 = r6;"
+		"r2 = r10;"
+		"r2 += -48;"
+		"r3 = %[sizeof_bpf_sock_tuple];"
+		"r4 = 0;"
+		"r5 = 0;"
+		"call %[bpf_sk_lookup_tcp];"
+		/* r0 = sk (PTR_TO_SOCK_OR_NULL) */
+		"if r0 == 0 goto l0_%=;"
+		/* spill sk to stack arg slot */
+		"*(u64 *)(r12 - 8) = r0;"
+		/* release the reference */
+		"r1 = r0;"
+		"call %[bpf_sk_release];"
+		/* r0 and stack arg slot (r12 - 8) should be invalid now. */
+		"r0 = *(u64 *)(r12 - 8);"
+		"r0 = *(u8 *)(r0 + 0);"
+	"l0_%=:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_sk_lookup_tcp),
+		  __imm(bpf_sk_release),
+		  __imm_const(sizeof_bpf_sock_tuple, sizeof(struct bpf_sock_tuple))
+		: __clobber_all
+	);
+}
+
+SEC("tc")
+__description("stack_arg: pkt pointer in stack arg slot invalidated after pull_data")
+__failure
+__arch_x86_64
+__msg("R0 invalid mem access 'scalar'")
+__naked void stack_arg_stale_pkt_ptr(void)
+{
+	asm volatile (
+		"r6 = r1;"
+		"r7 = *(u32 *)(r6 + %[__sk_buff_data]);"
+		"r8 = *(u32 *)(r6 + %[__sk_buff_data_end]);"
+		/* check pkt has at least 1 byte */
+		"r0 = r7;"
+		"r0 += 1;"
+		"if r0 > r8 goto l0_%=;"
+		/* spill valid pkt pointer to stack arg slot */
+		"*(u64 *)(r12 - 8) = r7;"
+		/* bpf_skb_pull_data(skb, 0) -- invalidates all pkt pointers */
+		"r1 = r6;"
+		"r2 = 0;"
+		"call %[bpf_skb_pull_data];"
+		/* read back the stale pkt pointer from stack arg slot */
+		"r0 = *(u64 *)(r12 - 8);"
+		/* dereferencing stale pkt pointer should fail */
+		"r0 = *(u8 *)(r0 + 0);"
+	"l0_%=:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_skb_pull_data),
+		  __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+		  __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+		: __clobber_all
+	);
+}
+
+SEC("tc")
+__description("stack_arg: null propagation rejects deref on null branch")
+__failure
+__arch_x86_64
+__msg("R1 invalid mem access 'scalar'")
+__naked void stack_arg_null_propagation_fail(void)
+{
+	asm volatile (
+		"r1 = 0;"
+		"*(u64 *)(r10 - 8) = r1;"
+		/* r0 = bpf_map_lookup_elem(&map_hash_8b, &key) */
+		"r2 = r10;"
+		"r2 += -8;"
+		"r1 = %[map_hash_8b] ll;"
+		"call %[bpf_map_lookup_elem];"
+		/* spill PTR_TO_MAP_VALUE_OR_NULL to stack arg slot */
+		"*(u64 *)(r12 - 8) = r0;"
+		/* null check on r0 */
+		"if r0 != 0 goto l0_%=;"
+		/*
+		 * on null branch, stack arg slot should be null (scalar 0).
+		 * read it back and dereference should fail.
+		 */
+		"r1 = *(u64 *)(r12 - 8);"
+		"r0 = *(u64 *)(r1 + 0);"
+	"l0_%=:"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_map_lookup_elem),
+		  __imm_addr(map_hash_8b)
+		: __clobber_all
+	);
+}
+
+#else
+
+SEC("socket")
+__description("stack_arg is not supported by compiler or jit, use a dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
-- 
2.52.0

Re: [PATCH bpf-next v3 03/11] bpf: Support stack arguments for bpf functions

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3567 bytes --]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 84699a428077..52a61021613b 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> +/* Validate that a stack arg access is 8-byte sized and aligned. */
> +static int check_stack_arg_access(struct bpf_verifier_env *env,
> +				  struct bpf_insn *insn, const char *op)
> +{
> +	int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
> +
> +	if (size != BPF_REG_SIZE) {
> +		verbose(env, "stack arg %s must be %d bytes, got %d\n",
> +			op, BPF_REG_SIZE, size);
> +		return -EINVAL;
> +	}
> +	if (insn->off % BPF_REG_SIZE) {
> +		verbose(env, "stack arg %s offset %d not aligned to %d\n",
> +			op, insn->off, BPF_REG_SIZE);
> +		return -EINVAL;
> +	}
> +	return 0;
> +}

Should this also reject non-negative offsets? The commit message and
check_stack_arg_write() comment both state that off is a negative
offset from the stack arg frame pointer, and the diagram shows only
negative offsets (r12-8, r12-16, etc.).

If a crafted BPF program uses a positive offset (e.g., off=+8), this
function accepts it because +8 is 8-byte aligned. Then in
check_stack_arg_write():

    int spi = (-off - 1) / BPF_REG_SIZE;

With off=+8, spi becomes (-8 - 1) / 8 = -1. The grow_stack_arg_slots()
call receives a negative size, which round_up() turns to 0, so no
allocation happens. The subsequent access to stack_arg_slots[-1] is
an out-of-bounds write (heap underflow if previously allocated, or a
crash if stack_arg_slots is NULL).

The same issue applies to the read path in check_stack_arg_read()
where -off becomes negative, passing the depth check, and spi
again computes to -1.

resolve_pseudo_ldimm64() explicitly allows BPF_REG_STACK_ARG_BASE
in both src_reg and dst_reg positions, so a BPF program can encode
these instructions.

Something like the following may be sufficient:

    if (insn->off >= 0) {
        verbose(env, "stack arg %s offset must be negative\n", op);
        return -EINVAL;
    }

> +
> +/*
> + * Write a value to the stack arg area.
> + * off is the negative offset from the stack arg frame pointer.
> + * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
> + */
> +static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
> +				 int off, int value_regno)
> +{
> +	int spi = (-off - 1) / BPF_REG_SIZE;

[ ... ]

> +	} else {
> +		/* BPF_ST: store immediate, treat as scalar */
> +		reg = &state->stack_arg_slots[spi].spilled_ptr;
> +		reg->type = SCALAR_VALUE;
> +		__mark_reg_known(reg, (u32)env->prog->insnsi[env->insn_idx].imm);
> +		for (i = 0; i < BPF_REG_SIZE; i++)
> +			state->stack_arg_slots[spi].slot_type[i] = STACK_MISC;
> +	}
> +	return 0;
> +}

Is the (u32) cast on the imm field intentional here? The regular
stack path in check_stack_write_fixed_off() passes insn->imm
directly to __mark_reg_known() without a cast:

    __mark_reg_known(tmp_reg, insn->imm);

Since insn->imm is __s32 and __mark_reg_known() takes u64, the
regular path sign-extends (e.g., imm=-1 becomes 0xffffffffffffffff).
The (u32) cast here zero-extends instead (imm=-1 becomes
0x00000000ffffffff).

BPF_ST with BPF_DW stores the 32-bit immediate sign-extended to
64 bits at runtime, so the verifier's tracked value would not match
the actual runtime value for negative immediates.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024

Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3398 bytes --]

> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 32864dbc2c4e..206f342a0ca0 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c

[ ... ]

> @@ -2449,6 +2529,41 @@ st:			if (is_imm8(insn->off))
>  			}
>  			if (!imm32)
>  				return -EINVAL;
> +
> +			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
> +				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
> +				int n_args = outgoing_stack_arg_depth / 8;
> +
> +				ip += push_stack_args(&prog, outgoing_base, n_args, 1);
> +			}
> +
> +			if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
> +				/* Kfunc calls: arg 6 -> R9, args 7+ -> push. */
> +				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
> +				int kfunc_stack_args = insn->off;
> +
> +				stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
> +
> +				/* Push args 7+ in reverse order */
> +				if (stack_args > 0)
> +					ip += push_stack_args(&prog, outgoing_base, kfunc_stack_args, 2);
> +
> +				/* mov r9, [rbp + outgoing_base] (arg 6) */
> +				if (is_imm8(outgoing_base)) {
> +					EMIT4(0x4C, 0x8B, 0x4D, outgoing_base);
> +					ip += 4;
> +				} else {
> +					EMIT3_off32(0x4C, 0x8B, 0x8D, outgoing_base);
> +					ip += 7;
> +				}
> +			}
>  			if (priv_frame_ptr) {
>  				push_r9(&prog);
>  				ip += 2;

When priv_frame_ptr is active (private stack in use), R9 holds the
private frame pointer throughout the program. The existing push_r9 /
pop_r9 around calls preserves R9 across the function call.

With this new code, two issues arise when stack args and
priv_frame_ptr are both active.

For kfunc calls with stack args: the MOV R9, [rbp + outgoing_base]
overwrites the priv_frame_ptr in R9 with arg6 before push_r9 saves
it. After the call, pop_r9 restores arg6 into R9 instead of the
priv_frame_ptr, so subsequent BPF_REG_FP accesses (which map to R9
when priv_frame_ptr is set) would reference the wrong memory.

For both kfunc and BPF-to-BPF calls: push_r9 inserts an extra 8
bytes onto the stack between the pushed args and the CALL
instruction. The callee then sees the wrong values for its stack
arguments. For example, with a BPF-to-BPF call the callee's
incoming arg formula expects [rbp + 16] to be the first stack arg,
but instead it finds the caller's saved R9 value:

    push_stack_args:  [RSP] = arg1, [RSP+8] = arg2
    push_r9:          [RSP] = R9,   [RSP+8] = arg1, [RSP+16] = arg2
    CALL:             [RSP] = ret,  [RSP+8] = R9 ...
    callee prologue:  [rbp+16] = R9  (expected arg1)

Both priv_stack_mode and stack_arg_depth are set independently per
subprog in jit_subprogs(), so can this combination actually be
reached?

> @@ -2458,6 +2573,14 @@ st:			if (is_imm8(insn->off))
>  				return -EINVAL;
>  			if (priv_frame_ptr)
>  				pop_r9(&prog);
> +			if (stack_args > 0) {
> +				/* add rsp, stack_args * 8 */
> +				EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
> +			}
> +			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
> +				/* add rsp, outgoing_stack_arg_depth */
> +				EMIT4(0x48, 0x83, 0xC4, outgoing_stack_arg_depth);
> +			}
>  			break;
>  		}


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024

Re: [PATCH bpf-next v3 05/11] bpf: Support stack arguments for kfunc calls

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3720 bytes --]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> +static struct bpf_reg_state *get_kfunc_arg_reg(struct bpf_verifier_env *env, int argno)
> +{
> +	struct bpf_func_state *caller;
> +	int spi;
> +
> +	if (argno < MAX_BPF_FUNC_REG_ARGS)
> +		return &cur_regs(env)[argno + 1];
> +
> +	caller = cur_func(env);
> +	spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE + (argno - MAX_BPF_FUNC_REG_ARGS);
> +	return &caller->stack_arg_slots[spi].spilled_ptr;
> +}
> +
>  static enum kfunc_ptr_arg_type
>  get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
>  		       struct bpf_kfunc_call_arg_meta *meta,

[ ... ]

>  {
>  	u32 regno = argno + 1;
>  	struct bpf_reg_state *regs = cur_regs(env);
>  	struct bpf_reg_state *reg = &regs[regno];

When check_kfunc_args() calls this function for a stack arg (argno >= 5),
the stack arg state was placed into regs[BPF_REG_1], but this local
reg reads regs[argno + 1] instead, which is a callee-saved register
(BPF_REG_6 through BPF_REG_9) or the frame pointer (BPF_REG_10).

For argno >= 10 (MAX_BPF_FUNC_ARGS is 12), this reads past
regs[MAX_BPF_REG - 1] into adjacent struct fields, since
bpf_func_state.regs[] only has 11 entries (BPF_REG_0..BPF_REG_10).

This reg is used later for register_is_null() and base_type(reg->type),
which would cause wrong argument type classification for pointer-typed
stack args.  The is_kfunc_arg_mem_size() lookahead a few lines below
was correctly updated to use get_kfunc_arg_reg() -- should this local
reg use the same helper?

> 	if (argno + 1 < nargs &&
> -	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1]) ||
> -	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1])))
> +	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) ||
> +	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1))))
>  		arg_mem_size = true;

[ ... ]

> 		case KF_ARG_PTR_TO_MEM_SIZE:
>  		{
>  			struct bpf_reg_state *buff_reg = &regs[regno];
>  			const struct btf_param *buff_arg = &args[i];
> -			struct bpf_reg_state *size_reg = &regs[regno + 1];
> +			struct bpf_reg_state *size_reg;
>  			const struct btf_param *size_arg = &args[i + 1];
> +			struct bpf_reg_state saved_size_reg = {};
> +			bool size_is_stack_arg = false;
> +
> +			if (i >= MAX_BPF_FUNC_REG_ARGS) {
> +				size_is_stack_arg = true;
> +				saved_size_reg = regs[regno + 1];
> +				regs[regno + 1] = *get_kfunc_arg_reg(env, i + 1);
> +				verbose(env, "Use reg %d to represent mem_size\n", regno + 1);
> +			}
> +			size_reg = &regs[regno + 1];

Is there a missing case when the pointer arg is the last register arg
(i == MAX_BPF_FUNC_REG_ARGS - 1, i.e. i == 4) and the size arg is
the first stack arg (i + 1 == 5)?

In that case i < MAX_BPF_FUNC_REG_ARGS, so is_stack_arg is false and
regno is 5 (BPF_REG_5).  The condition "i >= MAX_BPF_FUNC_REG_ARGS"
is false, so size_is_stack_arg stays false.  Then size_reg becomes
&regs[6], which is BPF_REG_6 (a callee-saved register) instead of
the stack arg slot for arg 5.

The get_kfunc_ptr_arg_type() lookahead correctly handles this boundary
with get_kfunc_arg_reg(env, argno + 1), so the arg gets classified as
KF_ARG_PTR_TO_MEM_SIZE, but then the actual size check reads from the
wrong register.

Would something like "if (i + 1 >= MAX_BPF_FUNC_REG_ARGS)" be the
right condition here?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024

Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Alexei Starovoitov]

On Sun, Apr 5, 2026 at 10:26 AM Yonghong Song <yonghong.song@linux.dev> wrote:
>
> Add x86_64 JIT support for BPF functions and kfuncs with more than
> 5 arguments. The extra arguments are passed through a stack area
> addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode,
> which the JIT translates to RBP-relative accesses in native code.
>
> The JIT follows the native x86_64 calling convention for stack
> argument placement. Incoming stack args from the caller sit above
> the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly
> where x86_64 expects them after CALL + PUSH RBP. Only the outgoing
> stack arg area is allocated below the program stack in the prologue.
>
> The native x86_64 stack layout for a function with incoming and
> outgoing stack args:
>
>   high address
>   ┌─────────────────────────┐
>   │ incoming stack arg N    │  [rbp + 16 + (N-1)*8]  (from caller)
>   │ ...                     │
>   │ incoming stack arg 1    │  [rbp + 16]
>   ├─────────────────────────┤
>   │ return address          │  [rbp + 8]
>   │ saved rbp               │  [rbp]
>   ├─────────────────────────┤
>   │ BPF program stack       │  (stack_depth bytes)
>   ├─────────────────────────┤
>   │ outgoing stack arg 1    │  [rbp - prog_stack_depth - outgoing_depth]
>   │ ...                     │   (written via r12-relative STX/ST)
>   │ outgoing stack arg M    │  [rbp - prog_stack_depth - 8]
>   ├─────────────────────────┤
>   │ callee-saved regs ...   │  (pushed after sub rsp)
>   └─────────────────────────┘  rsp
>   low address
>
> BPF r12-relative offsets are translated to native RBP-relative
> offsets with two formulas:
>   - Incoming args (load: -off <= incoming_depth):
>       native_off = 8 - bpf_off  →  [rbp + 16 + ...]
>   - Outgoing args (store: -off > incoming_depth):
>       native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
>
> Since callee-saved registers are pushed below the outgoing area,
> outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF
> calls and kfunc calls, outgoing args are explicitly pushed from the
> outgoing area onto the stack before CALL and rsp is restored after return.
>
> For kfunc calls specifically, arg 6 is loaded into R9 and args 7+
> are pushed onto the native stack, per the x86_64 calling convention.
>
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---
>  arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
>  1 file changed, 129 insertions(+), 6 deletions(-)
>
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 32864dbc2c4e..206f342a0ca0 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
>         *pprog = prog;
>  }
>
> +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
> +static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
> +{
> +       u8 *prog = *pprog;
> +       int k, bytes = 0;
> +       s32 off;
> +
> +       for (k = from; k >= to; k--) {
> +               off = outgoing_base + (k - 1) * 8;
> +               /* push qword [rbp + off] */
> +               if (is_imm8(off)) {
> +                       EMIT3(0xFF, 0x75, off);
> +                       bytes += 3;
> +               } else {
> +                       EMIT2_off32(0xFF, 0xB5, off);
> +                       bytes += 6;
> +               }
> +       }

This is not any better than v1.
It is still a copy.
As I said earlier:
https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
It has to be zero overhead. Copy pasting:

"
bpf calling convention for 6+ args needs to match x86.
With an exception of 6th arg.
All bpf insn need to remain as-is when calling another bpf prog
or kfunc. There should be no additional moves.
JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg
into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86.
The rest of STX needs to be jitted pretty much as-is
with a twist that bpf's r12 becomes %rbp on x86.
And similar things in the callee.
Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9.
Other LDX from [r12+M] will remain as-is, but r12->%rbp.
On arm64 more of the STX/LDX insns become native 'mov'-s
because arm64 has more registers for arguments.
"

Remapping in earlier patches is unnecessary.
These STX [r12-N], src_reg emitted by LLVM will be JITed as-is into
store of src_reg into %rbp-M slot.
Only shift by 8 bytes is necessary for N to become M.
where STX of 6th argument becomes 'mov' from one register to x86's r9.

The feature has to be zero overhead to pass these args from bpf to native
and from struct_ops hooks into bpf progs.
All verifier considerations are secondary.

pw-bot: cr

Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Yonghong Song]

On 4/5/26 1:36 PM, Alexei Starovoitov wrote:
> On Sun, Apr 5, 2026 at 10:26 AM Yonghong Song <yonghong.song@linux.dev> wrote:
>> Add x86_64 JIT support for BPF functions and kfuncs with more than
>> 5 arguments. The extra arguments are passed through a stack area
>> addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode,
>> which the JIT translates to RBP-relative accesses in native code.
>>
>> The JIT follows the native x86_64 calling convention for stack
>> argument placement. Incoming stack args from the caller sit above
>> the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly
>> where x86_64 expects them after CALL + PUSH RBP. Only the outgoing
>> stack arg area is allocated below the program stack in the prologue.
>>
>> The native x86_64 stack layout for a function with incoming and
>> outgoing stack args:
>>
>>    high address
>>    ┌─────────────────────────┐
>>    │ incoming stack arg N    │  [rbp + 16 + (N-1)*8]  (from caller)
>>    │ ...                     │
>>    │ incoming stack arg 1    │  [rbp + 16]
>>    ├─────────────────────────┤
>>    │ return address          │  [rbp + 8]
>>    │ saved rbp               │  [rbp]
>>    ├─────────────────────────┤
>>    │ BPF program stack       │  (stack_depth bytes)
>>    ├─────────────────────────┤
>>    │ outgoing stack arg 1    │  [rbp - prog_stack_depth - outgoing_depth]
>>    │ ...                     │   (written via r12-relative STX/ST)
>>    │ outgoing stack arg M    │  [rbp - prog_stack_depth - 8]
>>    ├─────────────────────────┤
>>    │ callee-saved regs ...   │  (pushed after sub rsp)
>>    └─────────────────────────┘  rsp
>>    low address
>>
>> BPF r12-relative offsets are translated to native RBP-relative
>> offsets with two formulas:
>>    - Incoming args (load: -off <= incoming_depth):
>>        native_off = 8 - bpf_off  →  [rbp + 16 + ...]
>>    - Outgoing args (store: -off > incoming_depth):
>>        native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
>>
>> Since callee-saved registers are pushed below the outgoing area,
>> outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF
>> calls and kfunc calls, outgoing args are explicitly pushed from the
>> outgoing area onto the stack before CALL and rsp is restored after return.
>>
>> For kfunc calls specifically, arg 6 is loaded into R9 and args 7+
>> are pushed onto the native stack, per the x86_64 calling convention.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
>>   arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
>>   1 file changed, 129 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
>> index 32864dbc2c4e..206f342a0ca0 100644
>> --- a/arch/x86/net/bpf_jit_comp.c
>> +++ b/arch/x86/net/bpf_jit_comp.c
>> @@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
>>          *pprog = prog;
>>   }
>>
>> +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
>> +static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
>> +{
>> +       u8 *prog = *pprog;
>> +       int k, bytes = 0;
>> +       s32 off;
>> +
>> +       for (k = from; k >= to; k--) {
>> +               off = outgoing_base + (k - 1) * 8;
>> +               /* push qword [rbp + off] */
>> +               if (is_imm8(off)) {
>> +                       EMIT3(0xFF, 0x75, off);
>> +                       bytes += 3;
>> +               } else {
>> +                       EMIT2_off32(0xFF, 0xB5, off);
>> +                       bytes += 6;
>> +               }
>> +       }
> This is not any better than v1.
> It is still a copy.
> As I said earlier:
> https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
> It has to be zero overhead. Copy pasting:
>
> "
> bpf calling convention for 6+ args needs to match x86.
> With an exception of 6th arg.
> All bpf insn need to remain as-is when calling another bpf prog
> or kfunc. There should be no additional moves.
> JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg
> into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86.
> The rest of STX needs to be jitted pretty much as-is
> with a twist that bpf's r12 becomes %rbp on x86.
> And similar things in the callee.
> Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9.
> Other LDX from [r12+M] will remain as-is, but r12->%rbp.
> On arm64 more of the STX/LDX insns become native 'mov'-s
> because arm64 has more registers for arguments.
> "
>
> Remapping in earlier patches is unnecessary.
> These STX [r12-N], src_reg emitted by LLVM will be JITed as-is into
> store of src_reg into %rbp-M slot.
> Only shift by 8 bytes is necessary for N to become M.
> where STX of 6th argument becomes 'mov' from one register to x86's r9.

Okay, I will do the following jit stack layout:

   incoming stack arg N -> 1
   return adderss
   saved rbp
   BPF program stack
   tail call cnt <== if tail call reachable
   callee-saved regs
   r9 <== if priv_frame_ptr is not null
   outgoing stack arg M -> 1
   call ...
   undo stack of outgoing stack arg + r9

In this case, insn *(u64 *)(r12 - off) = val can directly write 'val'
into proper outgoing stack arg locations.

if the call is a kfunc, outgoing stack should remove the bottom one
for the 6th argument before the call.

>
> The feature has to be zero overhead to pass these args from bpf to native
> and from struct_ops hooks into bpf progs.
> All verifier considerations are secondary.
>
> pw-bot: cr

Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Alexei Starovoitov]

On Sun, Apr 5, 2026 at 9:14 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>
>
>
> On 4/5/26 1:36 PM, Alexei Starovoitov wrote:
> > On Sun, Apr 5, 2026 at 10:26 AM Yonghong Song <yonghong.song@linux.dev> wrote:
> >> Add x86_64 JIT support for BPF functions and kfuncs with more than
> >> 5 arguments. The extra arguments are passed through a stack area
> >> addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode,
> >> which the JIT translates to RBP-relative accesses in native code.
> >>
> >> The JIT follows the native x86_64 calling convention for stack
> >> argument placement. Incoming stack args from the caller sit above
> >> the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly
> >> where x86_64 expects them after CALL + PUSH RBP. Only the outgoing
> >> stack arg area is allocated below the program stack in the prologue.
> >>
> >> The native x86_64 stack layout for a function with incoming and
> >> outgoing stack args:
> >>
> >>    high address
> >>    ┌─────────────────────────┐
> >>    │ incoming stack arg N    │  [rbp + 16 + (N-1)*8]  (from caller)
> >>    │ ...                     │
> >>    │ incoming stack arg 1    │  [rbp + 16]
> >>    ├─────────────────────────┤
> >>    │ return address          │  [rbp + 8]
> >>    │ saved rbp               │  [rbp]
> >>    ├─────────────────────────┤
> >>    │ BPF program stack       │  (stack_depth bytes)
> >>    ├─────────────────────────┤
> >>    │ outgoing stack arg 1    │  [rbp - prog_stack_depth - outgoing_depth]
> >>    │ ...                     │   (written via r12-relative STX/ST)
> >>    │ outgoing stack arg M    │  [rbp - prog_stack_depth - 8]
> >>    ├─────────────────────────┤
> >>    │ callee-saved regs ...   │  (pushed after sub rsp)
> >>    └─────────────────────────┘  rsp
> >>    low address
> >>
> >> BPF r12-relative offsets are translated to native RBP-relative
> >> offsets with two formulas:
> >>    - Incoming args (load: -off <= incoming_depth):
> >>        native_off = 8 - bpf_off  →  [rbp + 16 + ...]
> >>    - Outgoing args (store: -off > incoming_depth):
> >>        native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
> >>
> >> Since callee-saved registers are pushed below the outgoing area,
> >> outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF
> >> calls and kfunc calls, outgoing args are explicitly pushed from the
> >> outgoing area onto the stack before CALL and rsp is restored after return.
> >>
> >> For kfunc calls specifically, arg 6 is loaded into R9 and args 7+
> >> are pushed onto the native stack, per the x86_64 calling convention.
> >>
> >> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> >> ---
> >>   arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
> >>   1 file changed, 129 insertions(+), 6 deletions(-)
> >>
> >> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> >> index 32864dbc2c4e..206f342a0ca0 100644
> >> --- a/arch/x86/net/bpf_jit_comp.c
> >> +++ b/arch/x86/net/bpf_jit_comp.c
> >> @@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
> >>          *pprog = prog;
> >>   }
> >>
> >> +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
> >> +static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
> >> +{
> >> +       u8 *prog = *pprog;
> >> +       int k, bytes = 0;
> >> +       s32 off;
> >> +
> >> +       for (k = from; k >= to; k--) {
> >> +               off = outgoing_base + (k - 1) * 8;
> >> +               /* push qword [rbp + off] */
> >> +               if (is_imm8(off)) {
> >> +                       EMIT3(0xFF, 0x75, off);
> >> +                       bytes += 3;
> >> +               } else {
> >> +                       EMIT2_off32(0xFF, 0xB5, off);
> >> +                       bytes += 6;
> >> +               }
> >> +       }
> > This is not any better than v1.
> > It is still a copy.
> > As I said earlier:
> > https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
> > It has to be zero overhead. Copy pasting:
> >
> > "
> > bpf calling convention for 6+ args needs to match x86.
> > With an exception of 6th arg.
> > All bpf insn need to remain as-is when calling another bpf prog
> > or kfunc. There should be no additional moves.
> > JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg
> > into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86.
> > The rest of STX needs to be jitted pretty much as-is
> > with a twist that bpf's r12 becomes %rbp on x86.
> > And similar things in the callee.
> > Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9.
> > Other LDX from [r12+M] will remain as-is, but r12->%rbp.
> > On arm64 more of the STX/LDX insns become native 'mov'-s
> > because arm64 has more registers for arguments.
> > "
> >
> > Remapping in earlier patches is unnecessary.
> > These STX [r12-N], src_reg emitted by LLVM will be JITed as-is into
> > store of src_reg into %rbp-M slot.
> > Only shift by 8 bytes is necessary for N to become M.
> > where STX of 6th argument becomes 'mov' from one register to x86's r9.
>
> Okay, I will do the following jit stack layout:
>
>    incoming stack arg N -> 1
>    return adderss
>    saved rbp
>    BPF program stack
>    tail call cnt <== if tail call reachable
>    callee-saved regs
>    r9 <== if priv_frame_ptr is not null
>    outgoing stack arg M -> 1
>    call ...
>    undo stack of outgoing stack arg + r9

It looks like you're trying to preserve r9 as an auxiliary register.
If it's in the way, rewrite JIT handling. The size of the diff
doesn't matter.
r9 should be the 6th argument.

Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Yonghong Song]

On 4/5/26 9:54 PM, Alexei Starovoitov wrote:
> On Sun, Apr 5, 2026 at 9:14 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>>
>>
>> On 4/5/26 1:36 PM, Alexei Starovoitov wrote:
>>> On Sun, Apr 5, 2026 at 10:26 AM Yonghong Song <yonghong.song@linux.dev> wrote:
>>>> Add x86_64 JIT support for BPF functions and kfuncs with more than
>>>> 5 arguments. The extra arguments are passed through a stack area
>>>> addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode,
>>>> which the JIT translates to RBP-relative accesses in native code.
>>>>
>>>> The JIT follows the native x86_64 calling convention for stack
>>>> argument placement. Incoming stack args from the caller sit above
>>>> the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly
>>>> where x86_64 expects them after CALL + PUSH RBP. Only the outgoing
>>>> stack arg area is allocated below the program stack in the prologue.
>>>>
>>>> The native x86_64 stack layout for a function with incoming and
>>>> outgoing stack args:
>>>>
>>>>     high address
>>>>     ┌─────────────────────────┐
>>>>     │ incoming stack arg N    │  [rbp + 16 + (N-1)*8]  (from caller)
>>>>     │ ...                     │
>>>>     │ incoming stack arg 1    │  [rbp + 16]
>>>>     ├─────────────────────────┤
>>>>     │ return address          │  [rbp + 8]
>>>>     │ saved rbp               │  [rbp]
>>>>     ├─────────────────────────┤
>>>>     │ BPF program stack       │  (stack_depth bytes)
>>>>     ├─────────────────────────┤
>>>>     │ outgoing stack arg 1    │  [rbp - prog_stack_depth - outgoing_depth]
>>>>     │ ...                     │   (written via r12-relative STX/ST)
>>>>     │ outgoing stack arg M    │  [rbp - prog_stack_depth - 8]
>>>>     ├─────────────────────────┤
>>>>     │ callee-saved regs ...   │  (pushed after sub rsp)
>>>>     └─────────────────────────┘  rsp
>>>>     low address
>>>>
>>>> BPF r12-relative offsets are translated to native RBP-relative
>>>> offsets with two formulas:
>>>>     - Incoming args (load: -off <= incoming_depth):
>>>>         native_off = 8 - bpf_off  →  [rbp + 16 + ...]
>>>>     - Outgoing args (store: -off > incoming_depth):
>>>>         native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
>>>>
>>>> Since callee-saved registers are pushed below the outgoing area,
>>>> outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF
>>>> calls and kfunc calls, outgoing args are explicitly pushed from the
>>>> outgoing area onto the stack before CALL and rsp is restored after return.
>>>>
>>>> For kfunc calls specifically, arg 6 is loaded into R9 and args 7+
>>>> are pushed onto the native stack, per the x86_64 calling convention.
>>>>
>>>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>>>> ---
>>>>    arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
>>>>    1 file changed, 129 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
>>>> index 32864dbc2c4e..206f342a0ca0 100644
>>>> --- a/arch/x86/net/bpf_jit_comp.c
>>>> +++ b/arch/x86/net/bpf_jit_comp.c
>>>> @@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
>>>>           *pprog = prog;
>>>>    }
>>>>
>>>> +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
>>>> +static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
>>>> +{
>>>> +       u8 *prog = *pprog;
>>>> +       int k, bytes = 0;
>>>> +       s32 off;
>>>> +
>>>> +       for (k = from; k >= to; k--) {
>>>> +               off = outgoing_base + (k - 1) * 8;
>>>> +               /* push qword [rbp + off] */
>>>> +               if (is_imm8(off)) {
>>>> +                       EMIT3(0xFF, 0x75, off);
>>>> +                       bytes += 3;
>>>> +               } else {
>>>> +                       EMIT2_off32(0xFF, 0xB5, off);
>>>> +                       bytes += 6;
>>>> +               }
>>>> +       }
>>> This is not any better than v1.
>>> It is still a copy.
>>> As I said earlier:
>>> https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
>>> It has to be zero overhead. Copy pasting:
>>>
>>> "
>>> bpf calling convention for 6+ args needs to match x86.
>>> With an exception of 6th arg.
>>> All bpf insn need to remain as-is when calling another bpf prog
>>> or kfunc. There should be no additional moves.
>>> JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg
>>> into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86.
>>> The rest of STX needs to be jitted pretty much as-is
>>> with a twist that bpf's r12 becomes %rbp on x86.
>>> And similar things in the callee.
>>> Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9.
>>> Other LDX from [r12+M] will remain as-is, but r12->%rbp.
>>> On arm64 more of the STX/LDX insns become native 'mov'-s
>>> because arm64 has more registers for arguments.
>>> "
>>>
>>> Remapping in earlier patches is unnecessary.
>>> These STX [r12-N], src_reg emitted by LLVM will be JITed as-is into
>>> store of src_reg into %rbp-M slot.
>>> Only shift by 8 bytes is necessary for N to become M.
>>> where STX of 6th argument becomes 'mov' from one register to x86's r9.
>> Okay, I will do the following jit stack layout:
>>
>>     incoming stack arg N -> 1
>>     return adderss
>>     saved rbp
>>     BPF program stack
>>     tail call cnt <== if tail call reachable
>>     callee-saved regs
>>     r9 <== if priv_frame_ptr is not null
>>     outgoing stack arg M -> 1
>>     call ...
>>     undo stack of outgoing stack arg + r9
> It looks like you're trying to preserve r9 as an auxiliary register.
> If it's in the way, rewrite JIT handling. The size of the diff
> doesn't matter.

I actually will put r9 (priv_frame_ptr) into the stack. The following
is stack layout:

    incoming stack arg N -> 1
    return adderss
    saved rbp
    BPF program stack
    tail call cnt <== if tail call reachable
    callee-saved regs
    r9 <== if priv_frame_ptr is not null
    outgoing stack arg M -> 1

> r9 should be the 6th argument.

Re: [PATCH bpf-next v3 03/11] bpf: Support stack arguments for bpf functions

[Yonghong Song]

On 4/5/26 11:20 AM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index 84699a428077..52a61021613b 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
> [ ... ]
>
>> +/* Validate that a stack arg access is 8-byte sized and aligned. */
>> +static int check_stack_arg_access(struct bpf_verifier_env *env,
>> +				  struct bpf_insn *insn, const char *op)
>> +{
>> +	int size = bpf_size_to_bytes(BPF_SIZE(insn->code));
>> +
>> +	if (size != BPF_REG_SIZE) {
>> +		verbose(env, "stack arg %s must be %d bytes, got %d\n",
>> +			op, BPF_REG_SIZE, size);
>> +		return -EINVAL;
>> +	}
>> +	if (insn->off % BPF_REG_SIZE) {
>> +		verbose(env, "stack arg %s offset %d not aligned to %d\n",
>> +			op, insn->off, BPF_REG_SIZE);
>> +		return -EINVAL;
>> +	}
>> +	return 0;
>> +}
> Should this also reject non-negative offsets? The commit message and
> check_stack_arg_write() comment both state that off is a negative
> offset from the stack arg frame pointer, and the diagram shows only
> negative offsets (r12-8, r12-16, etc.).

We should. I will add more checking about positive or negative offset
(as applied) in the next revision.

>
> If a crafted BPF program uses a positive offset (e.g., off=+8), this
> function accepts it because +8 is 8-byte aligned. Then in
> check_stack_arg_write():
>
>      int spi = (-off - 1) / BPF_REG_SIZE;
>
> With off=+8, spi becomes (-8 - 1) / 8 = -1. The grow_stack_arg_slots()
> call receives a negative size, which round_up() turns to 0, so no
> allocation happens. The subsequent access to stack_arg_slots[-1] is
> an out-of-bounds write (heap underflow if previously allocated, or a
> crash if stack_arg_slots is NULL).
>
> The same issue applies to the read path in check_stack_arg_read()
> where -off becomes negative, passing the depth check, and spi
> again computes to -1.
>
> resolve_pseudo_ldimm64() explicitly allows BPF_REG_STACK_ARG_BASE
> in both src_reg and dst_reg positions, so a BPF program can encode
> these instructions.
>
> Something like the following may be sufficient:
>
>      if (insn->off >= 0) {
>          verbose(env, "stack arg %s offset must be negative\n", op);
>          return -EINVAL;
>      }
>
>> +
>> +/*
>> + * Write a value to the stack arg area.
>> + * off is the negative offset from the stack arg frame pointer.
>> + * Callers ensures off is 8-byte aligned and size is BPF_REG_SIZE.
>> + */
>> +static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
>> +				 int off, int value_regno)
>> +{
>> +	int spi = (-off - 1) / BPF_REG_SIZE;
> [ ... ]
>
>> +	} else {
>> +		/* BPF_ST: store immediate, treat as scalar */
>> +		reg = &state->stack_arg_slots[spi].spilled_ptr;
>> +		reg->type = SCALAR_VALUE;
>> +		__mark_reg_known(reg, (u32)env->prog->insnsi[env->insn_idx].imm);
>> +		for (i = 0; i < BPF_REG_SIZE; i++)
>> +			state->stack_arg_slots[spi].slot_type[i] = STACK_MISC;
>> +	}
>> +	return 0;
>> +}
> Is the (u32) cast on the imm field intentional here? The regular
> stack path in check_stack_write_fixed_off() passes insn->imm

Yes, good point. Will remove (u32).

> directly to __mark_reg_known() without a cast:
>
>      __mark_reg_known(tmp_reg, insn->imm);
>
> Since insn->imm is __s32 and __mark_reg_known() takes u64, the
> regular path sign-extends (e.g., imm=-1 becomes 0xffffffffffffffff).
> The (u32) cast here zero-extends instead (imm=-1 becomes
> 0x00000000ffffffff).
>
> BPF_ST with BPF_DW stores the 32-bit immediate sign-extended to
> 64 bits at runtime, so the verifier's tracked value would not match
> the actual runtime value for negative immediates.
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024

Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Yonghong Song]

On 4/5/26 11:20 AM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
>> index 32864dbc2c4e..206f342a0ca0 100644
>> --- a/arch/x86/net/bpf_jit_comp.c
>> +++ b/arch/x86/net/bpf_jit_comp.c
> [ ... ]
>
>> @@ -2449,6 +2529,41 @@ st:			if (is_imm8(insn->off))
>>   			}
>>   			if (!imm32)
>>   				return -EINVAL;
>> +
>> +			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
>> +				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
>> +				int n_args = outgoing_stack_arg_depth / 8;
>> +
>> +				ip += push_stack_args(&prog, outgoing_base, n_args, 1);
>> +			}
>> +
>> +			if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
>> +				/* Kfunc calls: arg 6 -> R9, args 7+ -> push. */
>> +				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
>> +				int kfunc_stack_args = insn->off;
>> +
>> +				stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
>> +
>> +				/* Push args 7+ in reverse order */
>> +				if (stack_args > 0)
>> +					ip += push_stack_args(&prog, outgoing_base, kfunc_stack_args, 2);
>> +
>> +				/* mov r9, [rbp + outgoing_base] (arg 6) */
>> +				if (is_imm8(outgoing_base)) {
>> +					EMIT4(0x4C, 0x8B, 0x4D, outgoing_base);
>> +					ip += 4;
>> +				} else {
>> +					EMIT3_off32(0x4C, 0x8B, 0x8D, outgoing_base);
>> +					ip += 7;
>> +				}
>> +			}
>>   			if (priv_frame_ptr) {
>>   				push_r9(&prog);
>>   				ip += 2;
> When priv_frame_ptr is active (private stack in use), R9 holds the
> private frame pointer throughout the program. The existing push_r9 /
> pop_r9 around calls preserves R9 across the function call.
>
> With this new code, two issues arise when stack args and
> priv_frame_ptr are both active.
>
> For kfunc calls with stack args: the MOV R9, [rbp + outgoing_base]
> overwrites the priv_frame_ptr in R9 with arg6 before push_r9 saves
> it. After the call, pop_r9 restores arg6 into R9 instead of the
> priv_frame_ptr, so subsequent BPF_REG_FP accesses (which map to R9
> when priv_frame_ptr is set) would reference the wrong memory.
>
> For both kfunc and BPF-to-BPF calls: push_r9 inserts an extra 8
> bytes onto the stack between the pushed args and the CALL
> instruction. The callee then sees the wrong values for its stack
> arguments. For example, with a BPF-to-BPF call the callee's
> incoming arg formula expects [rbp + 16] to be the first stack arg,
> but instead it finds the caller's saved R9 value:
>
>      push_stack_args:  [RSP] = arg1, [RSP+8] = arg2
>      push_r9:          [RSP] = R9,   [RSP+8] = arg1, [RSP+16] = arg2
>      CALL:             [RSP] = ret,  [RSP+8] = R9 ...
>      callee prologue:  [rbp+16] = R9  (expected arg1)
>
> Both priv_stack_mode and stack_arg_depth are set independently per
> subprog in jit_subprogs(), so can this combination actually be
> reached?

Yes, it could. So in next revision, we will disable private stack
if outgoing stack arguments is not empty.

>
>> @@ -2458,6 +2573,14 @@ st:			if (is_imm8(insn->off))
>>   				return -EINVAL;
>>   			if (priv_frame_ptr)
>>   				pop_r9(&prog);
>> +			if (stack_args > 0) {
>> +				/* add rsp, stack_args * 8 */
>> +				EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
>> +			}
>> +			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
>> +				/* add rsp, outgoing_stack_arg_depth */
>> +				EMIT4(0x48, 0x83, 0xC4, outgoing_stack_arg_depth);
>> +			}
>>   			break;
>>   		}
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024

Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Yonghong Song]

On 4/5/26 11:20 AM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
>> index 32864dbc2c4e..206f342a0ca0 100644
>> --- a/arch/x86/net/bpf_jit_comp.c
>> +++ b/arch/x86/net/bpf_jit_comp.c
> [ ... ]
>
>> @@ -2449,6 +2529,41 @@ st:			if (is_imm8(insn->off))
>>   			}
>>   			if (!imm32)
>>   				return -EINVAL;
>> +
>> +			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
>> +				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
>> +				int n_args = outgoing_stack_arg_depth / 8;
>> +
>> +				ip += push_stack_args(&prog, outgoing_base, n_args, 1);
>> +			}
>> +
>> +			if (src_reg != BPF_PSEUDO_CALL && insn->off > 0) {
>> +				/* Kfunc calls: arg 6 -> R9, args 7+ -> push. */
>> +				s32 outgoing_base = -(prog_stack_depth + outgoing_stack_arg_depth);
>> +				int kfunc_stack_args = insn->off;
>> +
>> +				stack_args = kfunc_stack_args > 1 ? kfunc_stack_args - 1 : 0;
>> +
>> +				/* Push args 7+ in reverse order */
>> +				if (stack_args > 0)
>> +					ip += push_stack_args(&prog, outgoing_base, kfunc_stack_args, 2);
>> +
>> +				/* mov r9, [rbp + outgoing_base] (arg 6) */
>> +				if (is_imm8(outgoing_base)) {
>> +					EMIT4(0x4C, 0x8B, 0x4D, outgoing_base);
>> +					ip += 4;
>> +				} else {
>> +					EMIT3_off32(0x4C, 0x8B, 0x8D, outgoing_base);
>> +					ip += 7;
>> +				}
>> +			}
>>   			if (priv_frame_ptr) {
>>   				push_r9(&prog);
>>   				ip += 2;
> When priv_frame_ptr is active (private stack in use), R9 holds the
> private frame pointer throughout the program. The existing push_r9 /
> pop_r9 around calls preserves R9 across the function call.
>
> With this new code, two issues arise when stack args and
> priv_frame_ptr are both active.
>
> For kfunc calls with stack args: the MOV R9, [rbp + outgoing_base]
> overwrites the priv_frame_ptr in R9 with arg6 before push_r9 saves
> it. After the call, pop_r9 restores arg6 into R9 instead of the
> priv_frame_ptr, so subsequent BPF_REG_FP accesses (which map to R9
> when priv_frame_ptr is set) would reference the wrong memory.
>
> For both kfunc and BPF-to-BPF calls: push_r9 inserts an extra 8
> bytes onto the stack between the pushed args and the CALL
> instruction. The callee then sees the wrong values for its stack
> arguments. For example, with a BPF-to-BPF call the callee's
> incoming arg formula expects [rbp + 16] to be the first stack arg,
> but instead it finds the caller's saved R9 value:
>
>      push_stack_args:  [RSP] = arg1, [RSP+8] = arg2
>      push_r9:          [RSP] = R9,   [RSP+8] = arg1, [RSP+16] = arg2
>      CALL:             [RSP] = ret,  [RSP+8] = R9 ...
>      callee prologue:  [rbp+16] = R9  (expected arg1)
>
> Both priv_stack_mode and stack_arg_depth are set independently per
> subprog in jit_subprogs(), so can this combination actually be
> reached?

You are right. Similar to my previous comments, private stack will
be disabled if outgoing stack arguments are not empty.

>
>> @@ -2458,6 +2573,14 @@ st:			if (is_imm8(insn->off))
>>   				return -EINVAL;
>>   			if (priv_frame_ptr)
>>   				pop_r9(&prog);
>> +			if (stack_args > 0) {
>> +				/* add rsp, stack_args * 8 */
>> +				EMIT4(0x48, 0x83, 0xC4, stack_args * 8);
>> +			}
>> +			if (src_reg == BPF_PSEUDO_CALL && outgoing_stack_arg_depth > 0) {
>> +				/* add rsp, outgoing_stack_arg_depth */
>> +				EMIT4(0x48, 0x83, 0xC4, outgoing_stack_arg_depth);
>> +			}
>>   			break;
>>   		}
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024

Re: [PATCH bpf-next v3 05/11] bpf: Support stack arguments for kfunc calls

[Yonghong Song]

On 4/5/26 11:20 AM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
> [ ... ]
>
>> +static struct bpf_reg_state *get_kfunc_arg_reg(struct bpf_verifier_env *env, int argno)
>> +{
>> +	struct bpf_func_state *caller;
>> +	int spi;
>> +
>> +	if (argno < MAX_BPF_FUNC_REG_ARGS)
>> +		return &cur_regs(env)[argno + 1];
>> +
>> +	caller = cur_func(env);
>> +	spi = caller->incoming_stack_arg_depth / BPF_REG_SIZE + (argno - MAX_BPF_FUNC_REG_ARGS);
>> +	return &caller->stack_arg_slots[spi].spilled_ptr;
>> +}
>> +
>>   static enum kfunc_ptr_arg_type
>>   get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
>>   		       struct bpf_kfunc_call_arg_meta *meta,
> [ ... ]
>
>>   {
>>   	u32 regno = argno + 1;
>>   	struct bpf_reg_state *regs = cur_regs(env);
>>   	struct bpf_reg_state *reg = &regs[regno];
> When check_kfunc_args() calls this function for a stack arg (argno >= 5),
> the stack arg state was placed into regs[BPF_REG_1], but this local
> reg reads regs[argno + 1] instead, which is a callee-saved register
> (BPF_REG_6 through BPF_REG_9) or the frame pointer (BPF_REG_10).
>
> For argno >= 10 (MAX_BPF_FUNC_ARGS is 12), this reads past
> regs[MAX_BPF_REG - 1] into adjacent struct fields, since
> bpf_func_state.regs[] only has 11 entries (BPF_REG_0..BPF_REG_10).
>
> This reg is used later for register_is_null() and base_type(reg->type),
> which would cause wrong argument type classification for pointer-typed
> stack args.  The is_kfunc_arg_mem_size() lookahead a few lines below
> was correctly updated to use get_kfunc_arg_reg() -- should this local
> reg use the same helper?

To use BPF_REG_1 to represent the stack argument is a hack. It may confuse
people and it may have some subtle issues due to this.

In the next revision, I will discard this patch and implement it properly.
In most cases, for related functions w.r.t. stack argument validation,
I will ensure functions has parameters like (reg_state, regno, argno).
If the argument is a stack argument, regno will be -1 (not a register).
In cases where argno is not really available, argno will be -1. But in
any case, at least one of them is positive. The verification log will
print out message properly. If regno is non-negative, log 'regno' is needed.
Otherwise, log 'argno'.

>
>> 	if (argno + 1 < nargs &&
>> -	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1]) ||
>> -	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1])))
>> +	    (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1)) ||
>> +	     is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], get_kfunc_arg_reg(env, argno + 1))))
>>   		arg_mem_size = true;
> [ ... ]
>
>> 		case KF_ARG_PTR_TO_MEM_SIZE:
>>   		{
>>   			struct bpf_reg_state *buff_reg = &regs[regno];
>>   			const struct btf_param *buff_arg = &args[i];
>> -			struct bpf_reg_state *size_reg = &regs[regno + 1];
>> +			struct bpf_reg_state *size_reg;
>>   			const struct btf_param *size_arg = &args[i + 1];
>> +			struct bpf_reg_state saved_size_reg = {};
>> +			bool size_is_stack_arg = false;
>> +
>> +			if (i >= MAX_BPF_FUNC_REG_ARGS) {
>> +				size_is_stack_arg = true;
>> +				saved_size_reg = regs[regno + 1];
>> +				regs[regno + 1] = *get_kfunc_arg_reg(env, i + 1);
>> +				verbose(env, "Use reg %d to represent mem_size\n", regno + 1);
>> +			}
>> +			size_reg = &regs[regno + 1];
> Is there a missing case when the pointer arg is the last register arg
> (i == MAX_BPF_FUNC_REG_ARGS - 1, i.e. i == 4) and the size arg is
> the first stack arg (i + 1 == 5)?

Yes. This is something I intend to support. arg4 (starting from arg0)
is a memory pointer at reg5, and arg5 represents the memory size at
the first stack argument.

>
> In that case i < MAX_BPF_FUNC_REG_ARGS, so is_stack_arg is false and
> regno is 5 (BPF_REG_5).  The condition "i >= MAX_BPF_FUNC_REG_ARGS"
> is false, so size_is_stack_arg stays false.  Then size_reg becomes
> &regs[6], which is BPF_REG_6 (a callee-saved register) instead of
> the stack arg slot for arg 5.

In this case, BPF_REG_6 will be saved first, and then BPF_REG_6 will
have the first stack argument for future verification.

But in any case, this patch set is a hack. I will have proper
implementation in the next revision.

>
> The get_kfunc_ptr_arg_type() lookahead correctly handles this boundary
> with get_kfunc_arg_reg(env, argno + 1), so the arg gets classified as
> KF_ARG_PTR_TO_MEM_SIZE, but then the actual size check reads from the
> wrong register.
>
> Would something like "if (i + 1 >= MAX_BPF_FUNC_REG_ARGS)" be the
> right condition here?
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24006922024