Re: [PATCH bpf-next v3 08/11] bpf,x86: Implement JIT support for stack arguments

[Yonghong Song <yonghong.song@linux.dev>]

On 4/5/26 9:54 PM, Alexei Starovoitov wrote:
> On Sun, Apr 5, 2026 at 9:14 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>>
>>
>> On 4/5/26 1:36 PM, Alexei Starovoitov wrote:
>>> On Sun, Apr 5, 2026 at 10:26 AM Yonghong Song <yonghong.song@linux.dev> wrote:
>>>> Add x86_64 JIT support for BPF functions and kfuncs with more than
>>>> 5 arguments. The extra arguments are passed through a stack area
>>>> addressed by register r12 (BPF_REG_STACK_ARG_BASE) in BPF bytecode,
>>>> which the JIT translates to RBP-relative accesses in native code.
>>>>
>>>> The JIT follows the native x86_64 calling convention for stack
>>>> argument placement. Incoming stack args from the caller sit above
>>>> the callee's frame pointer at [rbp + 16], [rbp + 24], etc., exactly
>>>> where x86_64 expects them after CALL + PUSH RBP. Only the outgoing
>>>> stack arg area is allocated below the program stack in the prologue.
>>>>
>>>> The native x86_64 stack layout for a function with incoming and
>>>> outgoing stack args:
>>>>
>>>>     high address
>>>>     ┌─────────────────────────┐
>>>>     │ incoming stack arg N    │  [rbp + 16 + (N-1)*8]  (from caller)
>>>>     │ ...                     │
>>>>     │ incoming stack arg 1    │  [rbp + 16]
>>>>     ├─────────────────────────┤
>>>>     │ return address          │  [rbp + 8]
>>>>     │ saved rbp               │  [rbp]
>>>>     ├─────────────────────────┤
>>>>     │ BPF program stack       │  (stack_depth bytes)
>>>>     ├─────────────────────────┤
>>>>     │ outgoing stack arg 1    │  [rbp - prog_stack_depth - outgoing_depth]
>>>>     │ ...                     │   (written via r12-relative STX/ST)
>>>>     │ outgoing stack arg M    │  [rbp - prog_stack_depth - 8]
>>>>     ├─────────────────────────┤
>>>>     │ callee-saved regs ...   │  (pushed after sub rsp)
>>>>     └─────────────────────────┘  rsp
>>>>     low address
>>>>
>>>> BPF r12-relative offsets are translated to native RBP-relative
>>>> offsets with two formulas:
>>>>     - Incoming args (load: -off <= incoming_depth):
>>>>         native_off = 8 - bpf_off  →  [rbp + 16 + ...]
>>>>     - Outgoing args (store: -off > incoming_depth):
>>>>         native_off = -(bpf_prog_stack + stack_arg_depth + 8) - bpf_off
>>>>
>>>> Since callee-saved registers are pushed below the outgoing area,
>>>> outgoing args are not at [rsp] at call time. Therefore, for both BPF-to-BPF
>>>> calls and kfunc calls, outgoing args are explicitly pushed from the
>>>> outgoing area onto the stack before CALL and rsp is restored after return.
>>>>
>>>> For kfunc calls specifically, arg 6 is loaded into R9 and args 7+
>>>> are pushed onto the native stack, per the x86_64 calling convention.
>>>>
>>>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>>>> ---
>>>>    arch/x86/net/bpf_jit_comp.c | 135 ++++++++++++++++++++++++++++++++++--
>>>>    1 file changed, 129 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
>>>> index 32864dbc2c4e..206f342a0ca0 100644
>>>> --- a/arch/x86/net/bpf_jit_comp.c
>>>> +++ b/arch/x86/net/bpf_jit_comp.c
>>>> @@ -390,6 +390,28 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
>>>>           *pprog = prog;
>>>>    }
>>>>
>>>> +/* Push stack args from [rbp + outgoing_base + (k - 1) * 8] in reverse order. */
>>>> +static int push_stack_args(u8 **pprog, s32 outgoing_base, int from, int to)
>>>> +{
>>>> +       u8 *prog = *pprog;
>>>> +       int k, bytes = 0;
>>>> +       s32 off;
>>>> +
>>>> +       for (k = from; k >= to; k--) {
>>>> +               off = outgoing_base + (k - 1) * 8;
>>>> +               /* push qword [rbp + off] */
>>>> +               if (is_imm8(off)) {
>>>> +                       EMIT3(0xFF, 0x75, off);
>>>> +                       bytes += 3;
>>>> +               } else {
>>>> +                       EMIT2_off32(0xFF, 0xB5, off);
>>>> +                       bytes += 6;
>>>> +               }
>>>> +       }
>>> This is not any better than v1.
>>> It is still a copy.
>>> As I said earlier:
>>> https://lore.kernel.org/bpf/CAADnVQ+5Aqxpk1bTw47xZQ5E0HOtf0-HHjmDFHaay7CDJ-7aKQ@mail.gmail.com/
>>> It has to be zero overhead. Copy pasting:
>>>
>>> "
>>> bpf calling convention for 6+ args needs to match x86.
>>> With an exception of 6th arg.
>>> All bpf insn need to remain as-is when calling another bpf prog
>>> or kfunc. There should be no additional moves.
>>> JIT should only special case 6th arg and convert bpf's STX [r12-N], src_reg
>>> into 'mov r9, src_reg', since r9 is used to pass 6th argument on x86.
>>> The rest of STX needs to be jitted pretty much as-is
>>> with a twist that bpf's r12 becomes %rbp on x86.
>>> And similar things in the callee.
>>> Instead of LDX [r12+N] it will be a 'mov dst_reg, r9' where r9 is x86's r9.
>>> Other LDX from [r12+M] will remain as-is, but r12->%rbp.
>>> On arm64 more of the STX/LDX insns become native 'mov'-s
>>> because arm64 has more registers for arguments.
>>> "
>>>
>>> Remapping in earlier patches is unnecessary.
>>> These STX [r12-N], src_reg emitted by LLVM will be JITed as-is into
>>> store of src_reg into %rbp-M slot.
>>> Only shift by 8 bytes is necessary for N to become M.
>>> where STX of 6th argument becomes 'mov' from one register to x86's r9.
>> Okay, I will do the following jit stack layout:
>>
>>     incoming stack arg N -> 1
>>     return adderss
>>     saved rbp
>>     BPF program stack
>>     tail call cnt <== if tail call reachable
>>     callee-saved regs
>>     r9 <== if priv_frame_ptr is not null
>>     outgoing stack arg M -> 1
>>     call ...
>>     undo stack of outgoing stack arg + r9
> It looks like you're trying to preserve r9 as an auxiliary register.
> If it's in the way, rewrite JIT handling. The size of the diff
> doesn't matter.

I actually will put r9 (priv_frame_ptr) into the stack. The following
is stack layout:

    incoming stack arg N -> 1
    return adderss
    saved rbp
    BPF program stack
    tail call cnt <== if tail call reachable
    callee-saved regs
    r9 <== if priv_frame_ptr is not null
    outgoing stack arg M -> 1

> r9 should be the 6th argument.