[PATCH bpf-next v1 1/6] bpf, x86: Account for the r9 push in exception boundary program

[Kumar Kartikeya Dwivedi <memxor@gmail.com>]

A program with a private stack pushes r9, which holds the private frame
pointer, around every call. The exception callback of an exception
boundary program resets the stack pointer to the value the boundary frame
had at the call it was thrown from, plus stack_arg_sp_adjust, and pops
the callee-saved registers from there. With a private stack that value
is 8 bytes short, so the registers are restored from the wrong slots and
the program returns to the kernel with corrupted callee-saved registers.
Include the push in stack_arg_sp_adjust.

A tracing program with 64 bytes or more of stack runs on a private
stack, so any such program that calls bpf_throw() is affected. The two
private stack exception selftests run such a program; they pass only
because nothing observes the restored registers after the return into
the kernel. The case was found while reworking how the callback restores
the callee-saved registers.

Fixes: 7d1cd70d4b16 ("bpf, x86: Support private stack in jit")
Cc: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
---
 arch/x86/net/bpf_jit_comp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 72a2a0a89e03..ea97e068efda 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2625,7 +2625,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	 */
 	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
 	if (bpf_prog->aux->exception_boundary)
-		bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
+		/* With a private stack every call site also has r9 pushed. */
+		bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp + (priv_frame_ptr ? 8 : 0);
 	emit_sub_rsp(&prog, outgoing_rsp);
 
 	if (arena_vm_start)
-- 
2.53.0