Re: [PATCH bpf-next v3 6/7] bpf,x86: Fix exception unwinding with outgoing stack arguments

["Kumar Kartikeya Dwivedi" <memxor@gmail.com>]

On Sun May 17, 2026 at 6:55 AM CEST, Yonghong Song wrote:
>
>
> On 5/16/26 5:59 PM, Alexei Starovoitov wrote:
>> On Fri May 15, 2026 at 3:51 PM PDT, Yonghong Song wrote:
>>> When a main program with exception_boundary has outgoing stack
>>> arguments (e.g. from calling subprogs with >5 args), bpf_throw() fails
>>> to correctly restore callee-saved registers, causing a kernel crash.
>>>
>>> The x86 JIT allocates the outgoing stack arg area below the
>>> callee-saved registers via 'sub rsp, outgoing_rsp' in the prologue.
>>> When bpf_throw() unwinds, it captures the main program's sp (which
>>> includes this outgoing area) and passes it to the exception callback.
>>> The callback gets rsp and rbp, followed by pop_callee_regs, but rsp
>>> points into the outgoing arg area rather than the callee-saved
>>> registers, so the pops restore garbage values. Returning to the
>>> kernel with corrupted callee-saved registers causes a crash.
>>>
>>> Fix this by passing the main program's outgoing_rsp as the 4th
>>> argument to the exception callback. The callback adjusts rsp with
>>> 'add rsp, rcx' before popping callee-saved registers, correctly
>>> skipping the outgoing arg area. When outgoing_rsp is 0 (the common
>>> case), this is a no-op.
>>>
>>> Fixes: 324c3ca6eed6 ("bpf,x86: Implement JIT support for stack arguments")
>>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>>> ---
>>>   arch/x86/net/bpf_jit_comp.c | 9 ++++++++-
>>>   include/linux/bpf.h         | 3 ++-
>>>   kernel/bpf/fixups.c         | 1 +
>>>   kernel/bpf/helpers.c        | 2 +-
>>>   4 files changed, 12 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
>>> index ceefefb4da21..f4fdceedaad7 100644
>>> --- a/arch/x86/net/bpf_jit_comp.c
>>> +++ b/arch/x86/net/bpf_jit_comp.c
>>> @@ -557,10 +557,15 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
>>>   			/* Keep the same instruction layout. */
>>>   			emit_nops(&prog, 3);     /* nop3 */
>>>   	}
>>> -	/* Exception callback receives FP as third parameter */
>>> +	/*
>>> +	 * Exception callback receives:
>>> +	 *   rsi = main program's SP, rdx = main program's FP,
>>> +	 *   rcx = main program's outgoing stack arg area size
>>> +	 */
>>>   	if (is_exception_cb) {
>>>   		EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */
>>>   		EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */
>>> +		EMIT3(0x48, 0x01, 0xCC); /* add rsp, rcx */
>> Maybe let's do it on C side like:
>> bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_adjust, ctx.bp, 0);
>
> This sounds better!
>
>> Avoids the need to use 'rcx'.
>>
>>>   		/* The main frame must have exception_boundary as true, so we
>>>   		 * first restore those callee-saved regs from stack, before
>>>   		 * reusing the stack frame.
>>> @@ -1789,6 +1794,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
>>>   	 * Arg 6 goes into r9 register, not on stack.
>>>   	 */
>>>   	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
>>> +	if (bpf_prog->aux->exception_boundary)
>>> +		bpf_prog->aux->stack_arg_adjust = outgoing_rsp;
>>>   	emit_sub_rsp(&prog, outgoing_rsp);
>>>
>>>   	if (arena_vm_start)
>>> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
>>> index 242f9597d9ab..2a1616c769a9 100644
>>> --- a/include/linux/bpf.h
>>> +++ b/include/linux/bpf.h
>>> @@ -1735,7 +1735,8 @@ struct bpf_prog_aux {
>>>   	int cgroup_atype; /* enum cgroup_bpf_attach_type */
>>>   	struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
>>>   	char name[BPF_OBJ_NAME_LEN];
>>> -	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
>>> +	u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64 stack_arg_adjust, u64);
>> no need to change this.
>
> Indeed, with the above 'ctx.sp + ctx.aux->stack_arg_adjust',
> the 4th argument does not need any change.
>
>>> +	u16 stack_arg_adjust;
>> this one is still needed, but maybe let's call it stack_arg_sp_adjust?
>
> Okay. Will use the stack_arg_sp_adjust.
>
>>
>> Looking at arch/arm64/net/bpf_jit_comp.c:590
>> it doesn't use SP, so should it fine.
>>
>> and arm64 seems to work already?
>
> I will take a look at arm64 as well.
>

Please also remove guards on tests to allow them to run on arm64, once it is
handled properly.