[PATCH bpf-next v1 0/3] Fix fastcall rewrite with indirect stack accesses

[PATCH bpf-next v1 0/3] Fix fastcall rewrite with indirect stack accesses

[Kumar Kartikeya Dwivedi]

When a bpf_fastcall call is inlined, the verifier removes the spill/fill
pairs around it and shrinks the stack frame to exclude their slots. Two
kinds of stack access skip the check that must disable this rewrite, so
the program can then access kernel stack outside its frame.

Patch 1 checks helper and kfunc stack buffers, which BPF CI reported
after commit 5da4a9f26fca ("bpf: Preserve stack initialization for
generic output buffers"). Patch 2 makes a callee's load from its
caller's stack check the caller's frame. Patch 3 adds tests.

Kumar Kartikeya Dwivedi (3):
  bpf: Check fastcall contract for helper and kfunc stack buffers
  bpf: Check fastcall contract of the frame a stack read targets
  selftests/bpf: Test fastcall rewrite with indirect stack accesses

 kernel/bpf/verifier.c                         |  10 +-
 .../bpf/progs/verifier_bpf_fastcall.c         | 161 ++++++++++++++++++
 2 files changed, 169 insertions(+), 2 deletions(-)


base-commit: 91f8613d95ad8cd99d8baf094806d1ef98bc6380
-- 
2.53.0

[PATCH bpf-next v1 1/3] bpf: Check fastcall contract for helper and kfunc stack buffers

[Kumar Kartikeya Dwivedi]

When the verifier inlines a bpf_fastcall helper or kfunc, it removes the
spill/fill pairs that clang emitted around the call and lowers the
subprogram's stack depth so that their slots are no longer part of the
frame. This is only safe if nothing else accesses those slots or any slot
below them. check_fastcall_stack_contract() enforces this: an access to
that region from outside a fastcall pattern disables the rewrite for the
subprogram.

Stack loads and stores call this check, but stack buffers passed to
helpers and kfuncs are validated by check_stack_range_initialized(),
which does not. If such a buffer is the only other access to the region,
the rewrite is still applied, the JIT reserves a frame that does not
contain the buffer, and the helper reads or writes live kernel stack
below the program's frame. For example, without CAP_PERFMON:

	*(u64 *)(r10 - 8) = r1;
	call bpf_get_smp_processor_id;
	r1 = *(u64 *)(r10 - 8);
	r2 = 0;
	r3 = r10;
	r3 += -64;
	r4 = 8;
	call bpf_skb_load_bytes;

is accepted with a stack depth of 0 instead of 64.

Some buffers got the check indirectly. For constant-sized outputs in raw
mode, mark_raw_stack() marks the buffer initialized through ordinary
byte stores, which call the check. Commit 5da4a9f26fca ("bpf: Preserve
stack initialization for generic output buffers") limited raw mode to
programs that may read uninitialized stack, so outputs of programs
without CAP_PERFMON lost the check, as in the example above. Buffers
that never used raw mode have lacked the check since fastcall support
was added: helper inputs such as a map key whose only store is a fastcall
spill, variable-sized outputs, and buffers at a variable stack offset.

Call check_fastcall_stack_contract() from check_stack_range_initialized()
for every nonempty access, before the raw-mode return. Use the frame that
owns the buffer, so that a callee passing a pointer into its caller's
stack disables the caller's rewrite. Zero-sized accesses touch no stack
and leave the rewrite enabled. Variable-offset stack reads now get the
check through check_stack_range_initialized(), so drop the separate call
in check_stack_read_var_off().

Fixes: 5b5f51bff1b6 ("bpf: no_caller_saved_registers attribute for helper calls")
Fixes: 5da4a9f26fca ("bpf: Preserve stack initialization for generic output buffers")
Link: https://lore.kernel.org/bpf/85f9995a43edb70c76615280e103dc5325742e88330f36c97962ee35f17e3e32@mail.kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
 kernel/bpf/verifier.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a7c9e2d8965d..1d9defa231d1 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4138,7 +4138,6 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg
 				  dst_regno);
 	if (err)
 		return err;
-	check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off);
 	return 0;
 }
 
@@ -7041,6 +7040,13 @@ static int check_stack_range_initialized(
 		max_off = reg_smax(reg) + off;
 	}
 
+	/*
+	 * Helpers and kfuncs can access stack slots without going through the
+	 * regular stack read/write paths, including when raw mode is disabled.
+	 */
+	if (access_size)
+		check_fastcall_stack_contract(env, state, env->insn_idx, min_off);
+
 	/* Unprivileged outputs retain each byte's initialization state. */
 	if (raw_mode) {
 		meta->arg_raw_mem.size[arg_slot] = access_size;
-- 
2.53.0

Re: [PATCH bpf-next v1 1/3] bpf: Check fastcall contract for helper and kfunc stack buffers

[Alexei Starovoitov]

On Wed, Sep 23, 2026 at 10:41 AM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
> +	/*
> +	 * Helpers and kfuncs can access stack slots without going through the
> +	 * regular stack read/write paths, including when raw mode is disabled.
> +	 */
> +	if (access_size)
> +		check_fastcall_stack_contract(env, state, env->insn_idx, min_off);

can we do it once in check_stack_access_within_bounds() ?
Both check_mem_access() and check_stack_range_initialized() call it
first, and it already has the frame that owns the slot and min_off.
Then the other four check_fastcall_stack_contract() calls can go
and patch 2 is not needed.

[PATCH bpf-next v1 2/3] bpf: Check fastcall contract of the frame a stack read targets

[Kumar Kartikeya Dwivedi]

check_stack_read_fixed_off() applies the fastcall contract check to the
current frame, even when the pointer targets a caller's stack. A callee
can therefore read a caller's fastcall spill slot without disabling the
caller's rewrite:

	caller:
	r1 = 1;
	*(u64 *)(r10 - 8) = r1;
	call bpf_get_smp_processor_id;
	r1 = *(u64 *)(r10 - 8);
	r1 = r10;
	r1 += -8;
	call callee;

	callee:
	r0 = *(u64 *)(r1 + 0);

The caller's spill and fill are removed and its stack depth drops to 0,
but the verifier still tracks the slot as holding the spilled constant.
The callee's load then reads kernel stack outside the caller's frame,
while the verifier assumes r0 is 1.

Check the contract of reg_state, the frame that owns the slot, as the
stack write paths already do.

Fixes: 5b5f51bff1b6 ("bpf: no_caller_saved_registers attribute for helper calls")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
 kernel/bpf/verifier.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1d9defa231d1..b38c4c7562db 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3951,7 +3951,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 	reg = &reg_state->stack[spi].spilled_ptr;
 
 	mark_stack_slot_scratched(env, spi);
-	check_fastcall_stack_contract(env, state, env->insn_idx, off);
+	check_fastcall_stack_contract(env, reg_state, env->insn_idx, off);
 
 	/*
 	 * Refine the in-progress load record's origin to the source stack slot.
-- 
2.53.0

[PATCH bpf-next v1 3/3] selftests/bpf: Test fastcall rewrite with indirect stack accesses

[Kumar Kartikeya Dwivedi]

Add tests where a helper stack buffer, or a load through a pointer into
another frame, overlaps the slots of a fastcall spill/fill pair. The
rewrite must not be applied in these cases, so check that the spill and
fill remain in the translated program and that the final stack depth
still covers the accessed slots. Cover:

 - a constant-sized uninitialized output without CAP_PERFMON;
 - the same output in the caller's stack, passed to a helper by a callee;
 - a helper input whose only initialization is the fastcall spill;
 - a callee load from the caller's fastcall spill slot.

Also add a zero-sized buffer, for which the rewrite must still be
applied.

The tests only load the programs and inspect the verifier log and the
translated instructions. Do not run them: without the fixes, the
verifier accepts programs that access memory outside their stack frame.

Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
 .../bpf/progs/verifier_bpf_fastcall.c         | 161 ++++++++++++++++++
 1 file changed, 161 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
index a73b837553fb..5f54f542a622 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
@@ -502,6 +502,167 @@ __naked void bad_helper_write(void)
 	: __clobber_all);
 }
 
+/*
+ * Load these programs without running them: removing the fastcall spills must
+ * not shrink the stack below an output buffer that has no explicit accesses.
+ */
+SEC("tc")
+__log_level(4)
+__msg_unpriv("subprog 0 (helper_uninit_stack) main {{.*}} stack 64")
+__xlated_unpriv("*(u64 *)(r10 -8) = r1")
+__xlated_unpriv("...")
+__xlated_unpriv("r1 = *(u64 *)(r10 -8)")
+__caps_unpriv(CAP_BPF | CAP_NET_ADMIN)
+__success_unpriv
+__naked void helper_uninit_stack(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = r1;"
+	"call %[bpf_get_smp_processor_id];"
+	"r1 = *(u64 *)(r10 - 8);"
+	"r2 = 0;"
+	"r3 = r10;"
+	"r3 += -64;"
+	"r4 = 8;"
+	"call %[bpf_skb_load_bytes];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_smp_processor_id),
+	  __imm(bpf_skb_load_bytes)
+	: __clobber_all);
+}
+
+static __used __naked void helper_uninit_stack_callee(void)
+{
+	asm volatile (
+	"r3 = r2;"
+	"r2 = 0;"
+	"r4 = 8;"
+	"call %[bpf_skb_load_bytes];"
+	"exit;"
+	:
+	: __imm(bpf_skb_load_bytes)
+	: __clobber_all);
+}
+
+SEC("tc")
+__log_level(4)
+__msg_unpriv("subprog 0 (helper_uninit_stack_caller) main {{.*}} stack 64")
+__xlated_unpriv("*(u64 *)(r10 -8) = r1")
+__xlated_unpriv("...")
+__xlated_unpriv("r1 = *(u64 *)(r10 -8)")
+__caps_unpriv(CAP_BPF | CAP_NET_ADMIN)
+__success_unpriv
+__naked void helper_uninit_stack_caller(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = r1;"
+	"call %[bpf_get_smp_processor_id];"
+	"r1 = *(u64 *)(r10 - 8);"
+	"r2 = r10;"
+	"r2 += -64;"
+	"call helper_uninit_stack_callee;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_smp_processor_id)
+	: __clobber_all);
+}
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, __u64);
+} fastcall_map SEC(".maps");
+
+SEC("tc")
+__log_level(4)
+__msg("subprog 0 (helper_reads_fastcall_spill) main {{.*}} stack 8")
+__xlated("*(u64 *)(r10 -8) = r1")
+__xlated("...")
+__xlated("r1 = *(u64 *)(r10 -8)")
+__success
+__naked void helper_reads_fastcall_spill(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"call %[bpf_get_smp_processor_id];"
+	"r1 = *(u64 *)(r10 - 8);"
+	"r1 = %[fastcall_map] ll;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"call %[bpf_map_lookup_elem];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_smp_processor_id),
+	  __imm(bpf_map_lookup_elem),
+	  __imm_addr(fastcall_map)
+	: __clobber_all);
+}
+
+static __used __naked void read_caller_stack_callee(void)
+{
+	asm volatile (
+	"r0 = *(u64 *)(r1 + 0);"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("raw_tp")
+__log_level(4)
+__msg("subprog 0 (callee_reads_fastcall_spill) main {{.*}} stack 8")
+__xlated("*(u64 *)(r10 -8) = r1")
+__xlated("...")
+__xlated("r1 = *(u64 *)(r10 -8)")
+__success
+__naked void callee_reads_fastcall_spill(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"call %[bpf_get_smp_processor_id];"
+	"r1 = *(u64 *)(r10 - 8);"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call read_caller_stack_callee;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_smp_processor_id)
+	: __clobber_all);
+}
+
+SEC("raw_tp")
+__arch_x86_64
+__log_level(4)
+__msg("subprog 0 (helper_zero_size_buffer) main {{.*}} stack 0")
+__xlated("0: r1 = 1")
+__xlated("1: r0 =")
+__success
+__naked void helper_zero_size_buffer(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"*(u64 *)(r10 - 8) = r1;"
+	"call %[bpf_get_smp_processor_id];"
+	"r1 = *(u64 *)(r10 - 8);"
+	"r1 = r10;"
+	"r1 += -64;"
+	"r2 = 0;"
+	"r3 = 0;"
+	"call %[bpf_probe_read_kernel];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_get_smp_processor_id),
+	  __imm(bpf_probe_read_kernel)
+	: __clobber_all);
+}
+
 SEC("raw_tp")
 __arch_x86_64
 /* main, not patched */
-- 
2.53.0

Re: [PATCH bpf-next v1 3/3] selftests/bpf: Test fastcall rewrite with indirect stack accesses

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]

> diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
> index a73b837553fb..5f54f542a622 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
> @@ -502,6 +502,167 @@ __naked void bad_helper_write(void)
>  	: __clobber_all);
>  }
>
> +/*
> + * Load these programs without running them: removing the fastcall spills must
> + * not shrink the stack below an output buffer that has no explicit accesses.
> + */
> +SEC("tc")
> +__log_level(4)
> +__msg_unpriv("subprog 0 (helper_uninit_stack) main {{.*}} stack 64")

This isn't a bug, but would it read better to scope this comment to the
first two tests and give each of the remaining cases its own one-liner,
the way the jump-pattern tests further down in this file do? As written
the comment says the stack must not shrink, which is the opposite of
what helper_zero_size_buffer asserts (it expects "stack 0").

The comment accurately describes helper_uninit_stack and
helper_uninit_stack_caller, but helper_reads_fastcall_spill is a map key
whose only initialization is the fastcall spill,
callee_reads_fastcall_spill is a load from the caller's frame, and
helper_zero_size_buffer checks that the rewrite is still applied and the
stack does shrink to 0.

> +__xlated_unpriv("*(u64 *)(r10 -8) = r1")
> +__xlated_unpriv("...")
> +__xlated_unpriv("r1 = *(u64 *)(r10 -8)")
> +__caps_unpriv(CAP_BPF | CAP_NET_ADMIN)
> +__success_unpriv
> +__naked void helper_uninit_stack(void)

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35840155529