Re: [PATCH bpf-next] bpf,x86: do RSB balance for trampoline

[Menglong Dong <menglong.dong@linux.dev>]

On 2025/11/5 15:13, Menglong Dong wrote:
> On 2025/11/5 10:12, Alexei Starovoitov wrote:
> > On Tue, Nov 4, 2025 at 5:30 PM Menglong Dong <menglong.dong@linux.dev> wrote:
> > >
> > > On 2025/11/5 02:56, Alexei Starovoitov wrote:
> > > > On Tue, Nov 4, 2025 at 2:49 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > > > >
> > > > > In origin call case, we skip the "rip" directly before we return, which
> > > > > break the RSB, as we have twice "call", but only once "ret".
> > > >
> > > > RSB meaning return stack buffer?
> > > >
> > > > and by "breaks RSB" you mean it makes the cpu less efficient?
> > >
> > > Yeah, I mean it makes the cpu less efficient. The RSB is used
> > > for the branch predicting, and it will push the "rip" to its hardware
> > > stack on "call", and pop it from the stack on "ret". In the origin
> > > call case, there are twice "call" but once "ret", will break its
> > > balance.
> > 
> > Yes. I'm aware, but your "mov [rbp + 8], rax" screws it up as well,
> > since RSB has to be updated/invalidated by this store.
> > The behavior depends on the microarchitecture, of course.
> > I think:
> > add rsp, 8
> > ret
> > will only screw up the return prediction, but won't invalidate RSB.
> > 
> > > Similar things happen in "return_to_handler" in ftrace_64.S,
> > > which has once "call", but twice "ret". And it pretend a "call"
> > > to make it balance.
> > 
> > This makes more sense to me. Let's try that approach instead
> > of messing with the return address on stack?
> 
> The way here is similar to the "return_to_handler". For the ftrace,
> the origin stack before the "ret" of the traced function is:
> 
>     POS:
>     rip   ---> return_to_handler
> 
> And the exit of the traced function will jump to return_to_handler.
> In return_to_handler, it will query the real "rip" of the traced function
> and the it call a internal function:
> 
>     call .Ldo_rop
> 
> And the stack now is:
> 
>     POS:
>     rip   ----> the address after "call .Ldo_rop", which is a "int3"
> 
> in the .Ldo_rop, it will modify the rip to the real rip to make
> it like this:
> 
>     POS:
>     rip   ---> real rip
> 
> And it return. Take the target function "foo" for example, the logic
> of it is:
> 
>     call foo -> call ftrace_caller -> return ftrace_caller ->
>     return return_to_handler -> call Ldo_rop -> return foo
> 
> As you can see, the call and return address for ".Ldo_rop" is
> also messed up. So I think it works here too. Compared with
> a messed "return address", a missed return maybe have
> better influence?
> 
> And the whole logic for us is:
> 
>     call foo -> call trampoline -> call origin ->
>     return origin -> return POS -> return foo

The "return POS" will miss the RSB, but the later return
will hit it.

The origin logic is:

     call foo -> call trampoline -> call origin ->
     return origin -> return foo

The "return foo" and all the later return will miss the RBS.

Hmm......Not sure if I understand it correctly.

>