[PATCH v5 0/6] powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

[PATCH v5 0/6] powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

This patch series enables support for two BPF JIT features
on powerpc64. The first three patches target support for
tail calls with subprogram combinations. The first patch
supports realignment of tail_call_cnt offset in stack frame.
Implementation details are provided in the commit messages.

The last three patches add support for BPF exceptions. An
architecture-specific stack walker is implemented to assist
with stack walk during exceptions.

All selftests related to tailcalls and exceptions are passing:

# ./test_progs -t tailcalls
#442/1   tailcalls/tailcall_1:OK
#442/2   tailcalls/tailcall_2:OK
#442/3   tailcalls/tailcall_3:OK
#442/4   tailcalls/tailcall_4:OK
#442/5   tailcalls/tailcall_5:OK
#442/6   tailcalls/tailcall_6:OK
#442/7   tailcalls/tailcall_bpf2bpf_1:OK
#442/8   tailcalls/tailcall_bpf2bpf_2:OK
#442/9   tailcalls/tailcall_bpf2bpf_3:OK
#442/10  tailcalls/tailcall_bpf2bpf_4:OK
#442/11  tailcalls/tailcall_bpf2bpf_5:OK
#442/12  tailcalls/tailcall_bpf2bpf_6:OK
#442/13  tailcalls/tailcall_bpf2bpf_fentry:OK
#442/14  tailcalls/tailcall_bpf2bpf_fexit:OK
#442/15  tailcalls/tailcall_bpf2bpf_fentry_fexit:OK
#442/16  tailcalls/tailcall_bpf2bpf_fentry_entry:OK
#442/17  tailcalls/tailcall_poke:OK
#442/18  tailcalls/tailcall_bpf2bpf_hierarchy_1:OK
#442/19  tailcalls/tailcall_bpf2bpf_hierarchy_fentry:OK
#442/20  tailcalls/tailcall_bpf2bpf_hierarchy_fexit:OK
#442/21  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_fexit:OK
#442/22  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_entry:OK
#442/23  tailcalls/tailcall_bpf2bpf_hierarchy_2:OK
#442/24  tailcalls/tailcall_bpf2bpf_hierarchy_3:OK
#442/25  tailcalls/tailcall_freplace:OK
#442/26  tailcalls/tailcall_bpf2bpf_freplace:OK
#442/27  tailcalls/tailcall_failure:OK
#442/28  tailcalls/reject_tail_call_spin_lock:OK
#442/29  tailcalls/reject_tail_call_rcu_lock:OK
#442/30  tailcalls/reject_tail_call_preempt_lock:OK
#442/31  tailcalls/reject_tail_call_ref:OK
#442     tailcalls:OK
Summary: 1/31 PASSED, 0 SKIPPED, 0 FAILED

# ./test_progs -t exceptions
#105/1   exceptions/exception_throw_always_1:OK
#105/2   exceptions/exception_throw_always_2:OK
#105/3   exceptions/exception_throw_unwind_1:OK
#105/4   exceptions/exception_throw_unwind_2:OK
#105/5   exceptions/exception_throw_default:OK
#105/6   exceptions/exception_throw_default_value:OK
#105/7   exceptions/exception_tail_call:OK
#105/8   exceptions/exception_ext:OK
#105/9   exceptions/exception_ext_mod_cb_runtime:OK
#105/10  exceptions/exception_throw_subprog:OK
#105/11  exceptions/exception_assert_nz_gfunc:OK
#105/12  exceptions/exception_assert_zero_gfunc:OK
#105/13  exceptions/exception_assert_neg_gfunc:OK
#105/14  exceptions/exception_assert_pos_gfunc:OK
#105/15  exceptions/exception_assert_negeq_gfunc:OK
#105/16  exceptions/exception_assert_poseq_gfunc:OK
#105/17  exceptions/exception_assert_nz_gfunc_with:OK
#105/18  exceptions/exception_assert_zero_gfunc_with:OK
#105/19  exceptions/exception_assert_neg_gfunc_with:OK
#105/20  exceptions/exception_assert_pos_gfunc_with:OK
#105/21  exceptions/exception_assert_negeq_gfunc_with:OK
#105/22  exceptions/exception_assert_poseq_gfunc_with:OK
#105/23  exceptions/exception_bad_assert_nz_gfunc:OK
#105/24  exceptions/exception_bad_assert_zero_gfunc:OK
#105/25  exceptions/exception_bad_assert_neg_gfunc:OK
#105/26  exceptions/exception_bad_assert_pos_gfunc:OK
#105/27  exceptions/exception_bad_assert_negeq_gfunc:OK
#105/28  exceptions/exception_bad_assert_poseq_gfunc:OK
#105/29  exceptions/exception_bad_assert_nz_gfunc_with:OK
#105/30  exceptions/exception_bad_assert_zero_gfunc_with:OK
#105/31  exceptions/exception_bad_assert_neg_gfunc_with:OK
#105/32  exceptions/exception_bad_assert_pos_gfunc_with:OK
#105/33  exceptions/exception_bad_assert_negeq_gfunc_with:OK
#105/34  exceptions/exception_bad_assert_poseq_gfunc_with:OK
#105/35  exceptions/exception_assert_range:OK
#105/36  exceptions/exception_assert_range_with:OK
#105/37  exceptions/exception_bad_assert_range:OK
#105/38  exceptions/exception_bad_assert_range_with:OK
#105/39  exceptions/non-throwing fentry -> exception_cb:OK
#105/40  exceptions/throwing fentry -> exception_cb:OK
#105/41  exceptions/non-throwing fexit -> exception_cb:OK
#105/42  exceptions/throwing fexit -> exception_cb:OK
#105/43  exceptions/throwing extension (with custom cb) -> exception_cb:OK
#105/44  exceptions/throwing extension -> global func in exception_cb:OK
#105/45  exceptions/exception_ext_mod_cb_runtime:OK
#105/46  exceptions/throwing extension (with custom cb) -> global func in exception_cb:OK
#105/47  exceptions/exception_ext:OK
#105/48  exceptions/non-throwing fentry -> non-throwing subprog:OK
#105/49  exceptions/throwing fentry -> non-throwing subprog:OK
#105/50  exceptions/non-throwing fentry -> throwing subprog:OK
#105/51  exceptions/throwing fentry -> throwing subprog:OK
#105/52  exceptions/non-throwing fexit -> non-throwing subprog:OK
#105/53  exceptions/throwing fexit -> non-throwing subprog:OK
#105/54  exceptions/non-throwing fexit -> throwing subprog:OK
#105/55  exceptions/throwing fexit -> throwing subprog:OK
#105/56  exceptions/non-throwing fmod_ret -> non-throwing subprog:OK
#105/57  exceptions/non-throwing fmod_ret -> non-throwing global subprog:OK
#105/58  exceptions/non-throwing extension -> non-throwing subprog:OK
#105/59  exceptions/non-throwing extension -> throwing subprog:OK
#105/60  exceptions/non-throwing extension -> non-throwing subprog:OK
#105/61  exceptions/non-throwing extension -> throwing global subprog:OK
#105/62  exceptions/throwing extension -> throwing global subprog:OK
#105/63  exceptions/throwing extension -> non-throwing global subprog:OK
#105/64  exceptions/non-throwing extension -> main subprog:OK
#105/65  exceptions/throwing extension -> main subprog:OK
#105/66  exceptions/reject_exception_cb_type_1:OK
#105/67  exceptions/reject_exception_cb_type_2:OK
#105/68  exceptions/reject_exception_cb_type_3:OK
#105/69  exceptions/reject_exception_cb_type_4:OK
#105/70  exceptions/reject_async_callback_throw:OK
#105/71  exceptions/reject_with_lock:OK
#105/72  exceptions/reject_subprog_with_lock:OK
#105/73  exceptions/reject_with_rcu_read_lock:OK
#105/74  exceptions/reject_subprog_with_rcu_read_lock:OK
#105/75  exceptions/reject_with_rbtree_add_throw:OK
#105/76  exceptions/reject_with_reference:OK
#105/77  exceptions/reject_with_cb_reference:OK
#105/78  exceptions/reject_with_cb:OK
#105/79  exceptions/reject_with_subprog_reference:OK
#105/80  exceptions/reject_throwing_exception_cb:OK
#105/81  exceptions/reject_exception_cb_call_global_func:OK
#105/82  exceptions/reject_exception_cb_call_static_func:OK
#105/83  exceptions/reject_multiple_exception_cb:OK
#105/84  exceptions/reject_exception_throw_cb:OK
#105/85  exceptions/reject_exception_throw_cb_diff:OK
#105/86  exceptions/reject_set_exception_cb_bad_ret1:OK
#105/87  exceptions/reject_set_exception_cb_bad_ret2:OK
#105/88  exceptions/check_assert_eq_int_min:OK
#105/89  exceptions/check_assert_eq_int_max:OK
#105/90  exceptions/check_assert_eq_zero:OK
#105/91  exceptions/check_assert_eq_llong_min:OK
#105/92  exceptions/check_assert_eq_llong_max:OK
#105/93  exceptions/check_assert_lt_pos:OK
#105/94  exceptions/check_assert_lt_zero:OK
#105/95  exceptions/check_assert_lt_neg:OK
#105/96  exceptions/check_assert_le_pos:OK
#105/97  exceptions/check_assert_le_zero:OK
#105/98  exceptions/check_assert_le_neg:OK
#105/99  exceptions/check_assert_gt_pos:OK
#105/100 exceptions/check_assert_gt_zero:OK
#105/101 exceptions/check_assert_gt_neg:OK
#105/102 exceptions/check_assert_ge_pos:OK
#105/103 exceptions/check_assert_ge_zero:OK
#105/104 exceptions/check_assert_ge_neg:OK
#105/105 exceptions/check_assert_range_s64:OK
#105/106 exceptions/check_assert_range_u64:OK
#105/107 exceptions/check_assert_single_range_s64:OK
#105/108 exceptions/check_assert_single_range_u64:OK
#105/109 exceptions/check_assert_generic:OK
#105/110 exceptions/check_assert_with_return:OK
#105     exceptions:OK
Summary: 1/110 PASSED, 0 SKIPPED, 0 FAILED

[v1]: https://lore.kernel.org/all/20260105105212.136645-1-adubey@linux.ibm.com/
[v2]: https://lore.kernel.org/all/20260114114450.30405-1-adubey@linux.ibm.com/
[v3]: https://lore.kernel.org/all/20260122165716.10508-1-adubey@linux.ibm.com/
[v4]: https://lore.kernel.org/all/20260122211854.5508-1-adubey@linux.ibm.com/

Changes v4->v5:
	Patch comments and code refactoring
	Handle exception callback and boundary frame checks
Changes v3->v4:
        Handle bpf-ci warnings
Changes v2->v3:
        Added PPC_BCC_CONST_SHORT for short jumps of constant offset
        Optimize tailcall allocation for BPF_TRAMP_F_CALL_ORIG flag
        New helper for stack size calculation during exceptions
        Prologue JIT optimizations during non exception prog case
Changes v1->v2:
        Move tail_call_cnt to offset 0 in stack frame
        Remove trampoline NVR remapping-patch3/6

Abhishek Dubey (6):
  powerpc64/bpf: Moving tail_call_cnt to bottom of frame
  powerpc64/bpf: Support tailcalls with subprogs
  powerpc64/bpf: Avoid tailcall restore from trampoline
  powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT
  powerpc64/bpf: Support exceptions
  powerpc64/bpf: Additional NVR handling for bpf_throw

 arch/powerpc/net/bpf_jit.h        |  16 ++
 arch/powerpc/net/bpf_jit_comp.c   |  85 +++++++--
 arch/powerpc/net/bpf_jit_comp64.c | 276 +++++++++++++++++++++++++-----
 3 files changed, 315 insertions(+), 62 deletions(-)

-- 
2.48.1

[PATCH v5 1/6] powerpc64/bpf: Moving tail_call_cnt to bottom of frame

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

To support tailcalls in subprogs, tail_call_cnt needs to be on the BPF
trampoline stack frame. In a regular BPF program or subprog stack
frame, the position of tail_call_cnt is after the NVR save area
(BPF_PPC_STACK_SAVE). To avoid complex logic in deducing offset for
tail_call_cnt, it has to be kept at the same offset on the trampoline
frame as well. But doing that wastes nearly all of BPF_PPC_STACK_SAVE
bytes on the BPF trampoline stack frame as the NVR save area is not
the same for BPF trampoline and regular BPF programs. Address this by
moving tail_call_cnt to the bottom of the frame.

This change avoids the need to account for BPF_PPC_STACK_SAVE bytes
in the BPF trampoline stack frame when support for tailcalls in BPF
subprogs is added later. Also, this change makes offset calculation
of tail_call_cnt field simpler all across.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit.h        |  1 +
 arch/powerpc/net/bpf_jit_comp.c   |  6 +++---
 arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
 3 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 8334cd667bba..9f6ec00bd02e 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -24,6 +24,7 @@
 
 #define SZL			sizeof(unsigned long)
 #define BPF_INSN_SAFETY		64
+#define BPF_PPC_TAILCALL	8
 
 #define PLANT_INSTR(d, idx, instr)					      \
 	do { if (d) { (d)[idx] = instr; } idx++; } while (0)
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 5e976730b2f5..de1adc0b1157 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -604,8 +604,8 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex
 					       int func_frame_offset, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
-		int tailcallcnt_offset = 7 * 8;
+		/* See Generated stack layout */
+		int tailcallcnt_offset = BPF_PPC_TAILCALL;
 
 		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
 		EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
@@ -620,7 +620,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
 		/* See bpf_jit_stack_tailcallcnt() */
-		int tailcallcnt_offset = 7 * 8;
+		int tailcallcnt_offset = BPF_PPC_TAILCALL;
 
 		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
 		EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 1fe37128c876..296e9ea14f2e 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -20,13 +20,15 @@
 #include "bpf_jit.h"
 
 /*
- * Stack layout:
+ * Stack layout with frame:
+ * Layout when setting up our own stack frame.
+ * Note: r1 at bottom, component offsets positive wrt r1.
  * Ensure the top half (upto local_tmp_var) stays consistent
  * with our redzone usage.
  *
  *		[	prev sp		] <-------------
- *		[   nv gpr save area	] 6*8		|
  *		[    tail_call_cnt	] 8		|
+ *		[   nv gpr save area	] 6*8		|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
  *		[     frame header	] 32/112	|
@@ -36,10 +38,12 @@
 /* for gpr non volatile registers BPG_REG_6 to 10 */
 #define BPF_PPC_STACK_SAVE	(6*8)
 /* for bpf JIT code internal usage */
-#define BPF_PPC_STACK_LOCALS	32
+#define BPF_PPC_STACK_LOCALS	24
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
 #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
-				 BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
+				 BPF_PPC_STACK_LOCALS + \
+				 BPF_PPC_STACK_SAVE   + \
+				 BPF_PPC_TAILCALL)
 
 /* BPF register usage */
 #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
 }
 
 /*
+ * Stack layout with redzone:
  * When not setting up our own stackframe, the redzone (288 bytes) usage is:
+ * Note: r1 from prev frame. Component offset negative wrt r1.
  *
  *		[	prev sp		] <-------------
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
- *		[   nv gpr save area	] 6*8
  *		[    tail_call_cnt	] 8
+ *		[   nv gpr save area	] 6*8
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
  */
 static int bpf_jit_stack_local(struct codegen_context *ctx)
 {
-	if (bpf_has_stack_frame(ctx))
+	if (bpf_has_stack_frame(ctx)) {
+		/* Stack layout with frame */
 		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
-	else
-		return -(BPF_PPC_STACK_SAVE + 32);
+	} else {
+		/* Stack layout with redzone */
+		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+	}
 }
 
 static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
 {
-	return bpf_jit_stack_local(ctx) + 24;
+	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
 }
 
 static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
@@ -115,7 +124,7 @@ static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
 	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
 		return (bpf_has_stack_frame(ctx) ?
 			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
-				- (8 * (32 - reg));
+				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
 
 	pr_err("BPF JIT is asking about unknown registers");
 	BUG();
@@ -145,7 +154,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 	if (ctx->seen & SEEN_TAILCALL) {
 		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
 		/* this goes in the redzone */
-		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_STACK_SAVE + 8)));
+		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	} else {
 		EMIT(PPC_RAW_NOP());
 		EMIT(PPC_RAW_NOP());
-- 
2.48.1

[PATCH v5 2/6] powerpc64/bpf: Support tailcalls with subprogs

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

Enable tailcalls support in subprogs by passing tail call count as
reference instead of value. The actual tailcall count is always
maintained in the tailcall field present in the frame of main
function (also called entry function). The tailcall field in the
stack frame of subprogs contains reference to the tailcall field
in the stack frame of main BPF program. Accordingly, rename
tail_call_cnt field in the stack layout to tail_call_info.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit.h        | 13 ++++++
 arch/powerpc/net/bpf_jit_comp.c   | 63 +++++++++++++++++++++++++---
 arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++--------
 3 files changed, 122 insertions(+), 22 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 9f6ec00bd02e..56f56fdd4969 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -52,6 +52,13 @@
 		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));					\
 	} while (0)
 
+/* When constant jump offset is known prior */
+#define PPC_BCC_CONST_SHORT(cond, offset)							\
+	do {											\
+		BUILD_BUG_ON(offset < -0x8000 || offset > 0x7fff || (offset & 0x3));		\
+		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));	\
+	} while (0)
+
 /*
  * Sign-extended 32-bit immediate load
  *
@@ -73,6 +80,10 @@
 	} } while (0)
 
 #ifdef CONFIG_PPC64
+
+/* for gpr non volatile registers BPG_REG_6 to 10 */
+#define BPF_PPC_STACK_SAVE	(6 * 8)
+
 /* If dummy pass (!image), account for maximum possible instructions */
 #define PPC_LI64(d, i)		do {					      \
 	if (!image)							      \
@@ -167,6 +178,7 @@ struct codegen_context {
 	unsigned int alt_exit_addr;
 	u64 arena_vm_start;
 	u64 user_vm_start;
+	bool is_subprog;
 };
 
 #define bpf_to_ppc(r)	(ctx->b2p[r])
@@ -206,6 +218,7 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass
 			  struct codegen_context *ctx, int insn_idx,
 			  int jmp_off, int dst_reg, u32 code);
 
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
 #endif
 
 #endif
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index de1adc0b1157..4b18daed054a 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
 	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
 	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
+	cgctx.is_subprog = bpf_is_subprog(fp);
 
 	/* Scouting faux-generate pass 0 */
 	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
 	bpf_prog_unlock_free(fp);
 }
 
+bool bpf_jit_supports_subprog_tailcalls(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
 bool bpf_jit_supports_kfunc_call(void)
 {
 	return true;
@@ -600,15 +606,50 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
 	return 0;
 }
 
-static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_context *ctx,
-					       int func_frame_offset, int r4_off)
+/*
+ * Refer __arch_prepare_bpf_trampoline() for stack component details.
+ *
+ * The tailcall count/reference is present in caller's stack frame. The
+ * tail_call_info is saved at the same offset on the trampoline frame
+ * for the traced function (BPF subprog/callee) to fetch it.
+ */
+static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
+						int func_frame_offset,
+						int bpf_dummy_frame_size, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
 		/* See Generated stack layout */
-		int tailcallcnt_offset = BPF_PPC_TAILCALL;
+		int tailcallinfo_offset = BPF_PPC_TAILCALL;
+
+		/*
+		 * func_frame_offset =                                   ...(1)
+		 *      bpf_dummy_frame_size + trampoline_frame_size
+		 */
+		EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
+		EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
 
-		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
+		/*
+		 * Setting the tail_call_info in trampoline's frame
+		 * depending on if previous frame had value or reference.
+		 */
+		EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
+		PPC_BCC_CONST_SHORT(COND_GT, 8);
+		EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
+		/*
+		 * From ...(1) above:
+		 * trampoline_frame_bottom =                            ...(2)
+		 *      func_frame_offset - bpf_dummy_frame_size
+		 *
+		 * Using ...(2) derived above:
+		 * trampoline_tail_call_info_offset =                  ...(3)
+		 *      trampoline_frame_bottom - tailcallinfo_offset
+		 *
+		 * From ...(3):
+		 * Use trampoline_tail_call_info_offset to write reference of main's
+		 * tail_call_info in trampoline frame.
+		 */
+		EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
+								- tailcallinfo_offset));
 	} else {
 		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
 		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
@@ -714,6 +755,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	 * LR save area                 [ r0 save (64-bit)  ]   | header
 	 *                              [ r0 save (32-bit)  ]   |
 	 * dummy frame for unwind       [ back chain 1      ] --
+	 *                              [ tail_call_info    ] optional - 64-bit powerpc
 	 *                              [ padding           ] align stack frame
 	 *       r4_off                 [ r4 (tailcallcnt)  ] optional - 32-bit powerpc
 	 *       alt_lr_off             [ real lr (ool stub)] optional - actual lr
@@ -795,6 +837,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		}
 	}
 
+	/*
+	 * Save tailcall count pointer at the same offset on the
+	 * stack where subprogs expect it
+	 */
+	if ((flags & BPF_TRAMP_F_CALL_ORIG) &&
+		(flags & BPF_TRAMP_F_TAIL_CALL_CTX))
+		bpf_frame_size += BPF_PPC_TAILCALL;
+
 	/* Padding to align stack frame, if any */
 	bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
 
@@ -896,7 +946,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 
 		/* Replicate tail_call_cnt before calling the original BPF prog */
 		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
-			bpf_trampoline_setup_tail_call_cnt(image, ctx, func_frame_offset, r4_off);
+			bpf_trampoline_setup_tail_call_info(image, ctx, func_frame_offset,
+								bpf_dummy_frame_size, r4_off);
 
 		/* Restore args */
 		bpf_trampoline_restore_args_stack(image, ctx, func_frame_offset, nr_regs, regs_off);
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 296e9ea14f2e..18da5a866447 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -26,8 +26,12 @@
  * Ensure the top half (upto local_tmp_var) stays consistent
  * with our redzone usage.
  *
+ * tail_call_info - stores tailcall count value in main program's
+ *                  frame, stores reference to tail_call_info of
+ *                  main's frame in sub-prog's frame.
+ *
  *		[	prev sp		] <-------------
- *		[    tail_call_cnt	] 8		|
+ *		[    tail_call_info	] 8		|
  *		[   nv gpr save area	] 6*8		|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
@@ -35,8 +39,6 @@
  * sp (r1) --->	[    stack pointer	] --------------
  */
 
-/* for gpr non volatile registers BPG_REG_6 to 10 */
-#define BPF_PPC_STACK_SAVE	(6*8)
 /* for bpf JIT code internal usage */
 #define BPF_PPC_STACK_LOCALS	24
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
@@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
  *		[	prev sp		] <-------------
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
- *		[    tail_call_cnt	] 8
+ *		[    tail_call_info	] 8
  *		[   nv gpr save area	] 6*8
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
@@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
 	}
 }
 
-static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
 {
 	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
 }
@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 #endif
 
 	/*
-	 * Initialize tail_call_cnt if we do tail calls.
-	 * Otherwise, put in NOPs so that it can be skipped when we are
-	 * invoked through a tail call.
+	 * Tail call count(tcc) is saved & updated only in main
+	 * program's frame and the address of tcc in main program's
+	 * frame (tcc_ptr) is saved in subprogs frame.
+	 *
+	 * Offset of tail_call_info on any frame will be interpreted
+	 * as either tcc_ptr or tcc value depending on whether it is
+	 * greater than MAX_TAIL_CALL_CNT or not.
 	 */
-	if (ctx->seen & SEEN_TAILCALL) {
+	if (!ctx->is_subprog) {
 		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
 		/* this goes in the redzone */
 		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	} else {
-		EMIT(PPC_RAW_NOP());
-		EMIT(PPC_RAW_NOP());
+		/*
+		 * if tail_call_info < MAX_TAIL_CALL_CNT
+		 *      main prog calling first subprog -> copy reference
+		 * else
+		 *      subsequent subprog calling another subprog -> directly copy content
+		 */
+		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
+		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
+		EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+		PPC_BCC_CONST_SHORT(COND_GT, 8);
+		EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2),
+								-(BPF_PPC_TAILCALL)));
+		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	}
 
 	if (bpf_has_stack_frame(ctx)) {
@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
 	EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));
 	PPC_BCC_SHORT(COND_GE, out);
 
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+	PPC_BCC_CONST_SHORT(COND_LE, 8);
+
+	/* dereference TMP_REG_1 */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
+
 	/*
-	 * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
+	 * if (tail_call_info == MAX_TAIL_CALL_CNT)
 	 *   goto out;
 	 */
-	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
 	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
-	PPC_BCC_SHORT(COND_GE, out);
+	PPC_BCC_SHORT(COND_EQ, out);
 
 	/*
-	 * tail_call_cnt++;
+	 * tail_call_info++; <- Actual value of tcc here
 	 */
 	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1));
-	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+
+	/*
+	 * Before writing updated tail_call_info, distinguish if current frame
+	 * is storing a reference to tail_call_info or actual tcc value in
+	 * tail_call_info.
+	 */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT));
+	PPC_BCC_CONST_SHORT(COND_GT, 8);
+
+	/* First get address of tail_call_info */
+	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	/* Writeback updated value to tail_call_info */
+	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0));
 
 	/* prog = array->ptrs[index]; */
 	EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));
-- 
2.48.1

[PATCH v5 3/6] powerpc64/bpf: Avoid tailcall restore from trampoline

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

Back propagation of tailcall count is no longer needed for
powerpc64 due to use of reference, which updates the tailcall
count in the tail_call_info field in the frame of the main
program only.

Back propagation is still required for 32-bit powerpc.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit_comp.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 4b18daed054a..f8769d785123 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -659,14 +659,11 @@ static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_conte
 static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx,
 						 int func_frame_offset, int r4_off)
 {
-	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
-		int tailcallcnt_offset = BPF_PPC_TAILCALL;
-
-		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
-	} else {
-		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
+	if (IS_ENABLED(CONFIG_PPC32)) {
+		/*
+		 * Restore tailcall for 32-bit powerpc
+		 * See bpf_jit_stack_offsetof() and BPF_PPC_TC
+		 */
 		EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
 	}
 }
-- 
2.48.1

[PATCH v5 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

This function is used by bpf_throw() to unwind the stack
until frame of exception-boundary during BPF exception
handling.

This function is necessary to support BPF exceptions on
PowerPC.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit_comp64.c | 41 +++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 18da5a866447..1bba068f2c4c 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -247,6 +247,47 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
 	bpf_jit_build_fentry_stubs(image, ctx);
 }
 
+/*
+ * arch_bpf_stack_walk() - BPF stack walker for PowerPC
+ *
+ * Based on arch_stack_walk() from stacktrace.c.
+ * PowerPC uses stack frames rather than stack pointers. See [1] for
+ * the equivalence between frame pointers and stack pointers.
+ * Additional reference at [2].
+ * TODO: refactor with arch_stack_walk()
+ *
+ * [1]: https://lore.kernel.org/all/20200220115141.2707-1-mpe@ellerman.id.au/
+ * [2]: https://lore.kernel.org/bpf/20260122211854.5508-5-adubey@linux.ibm.com/
+ */
+
+void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64), void *cookie)
+{
+	// callback processing always in current context
+	unsigned long sp = current_stack_frame();
+
+	for (;;) {
+		unsigned long *stack = (unsigned long *) sp;
+		unsigned long ip;
+
+		if (!validate_sp(sp, current))
+			return;
+
+		ip = stack[STACK_FRAME_LR_SAVE];
+		if (!ip)
+			break;
+
+		/*
+		 * consume_fn common code expects stack pointer in third
+		 * argument. There is no sp in ppc64, rather pass frame
+		 * pointer(named sp here).
+		 */
+		if (ip && !consume_fn(cookie, ip, sp, sp))
+			break;
+
+		sp = stack[0];
+	}
+}
+
 int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func)
 {
 	unsigned long func_addr = func ? ppc_function_entry((void *)func) : 0;
-- 
2.48.1

[PATCH v5 5/6] powerpc64/bpf: Support exceptions

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

The modified prologue/epilogue generation code now
enables exception-callback to use the stack frame of
the program marked as exception boundary, where callee
saved registers are stored.

As per ppc64 ABIv2 documentation[1], r14-r31 are callee
saved registers. BPF programs on ppc64 already saves
r26-r31 registers. Saving the remaining set of callee
saved registers(r14-r25) is handled in the next patch.

[1] https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit.h        |  2 ++
 arch/powerpc/net/bpf_jit_comp.c   |  7 ++++
 arch/powerpc/net/bpf_jit_comp64.c | 59 +++++++++++++++++++++----------
 3 files changed, 50 insertions(+), 18 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 56f56fdd4969..82bbf63f0e57 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -179,6 +179,8 @@ struct codegen_context {
 	u64 arena_vm_start;
 	u64 user_vm_start;
 	bool is_subprog;
+	bool exception_boundary;
+	bool exception_cb;
 };
 
 #define bpf_to_ppc(r)	(ctx->b2p[r])
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index f8769d785123..3a2e6fa3ac7c 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
 	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
 	cgctx.is_subprog = bpf_is_subprog(fp);
+	cgctx.exception_boundary = fp->aux->exception_boundary;
+	cgctx.exception_cb = fp->aux->exception_cb;
 
 	/* Scouting faux-generate pass 0 */
 	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)
 	bpf_prog_unlock_free(fp);
 }
 
+bool bpf_jit_supports_exceptions(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
 bool bpf_jit_supports_subprog_tailcalls(void)
 {
 	return IS_ENABLED(CONFIG_PPC64);
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 1bba068f2c4c..db121b1404fe 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
 	 * - the bpf program uses its stack area
 	 * The latter condition is deduced from the usage of BPF_REG_FP
 	 */
-	return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP));
+	return ctx->seen & SEEN_FUNC ||
+	       bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) ||
+	       ctx->exception_cb;
 }
 
 /*
@@ -161,8 +163,13 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
 		/* this goes in the redzone */
 		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
-	} else {
+	} else if (!ctx->exception_cb) {
 		/*
+		 * Tailcall jitting for non exception_cb progs only.
+		 * exception_cb won't require tail_call_info to be setup.
+		 *
+		 * tail_call_info interpretation logic:
+		 *
 		 * if tail_call_info < MAX_TAIL_CALL_CNT
 		 *      main prog calling first subprog -> copy reference
 		 * else
@@ -177,8 +184,12 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	}
 
-	if (bpf_has_stack_frame(ctx)) {
+	if (bpf_has_stack_frame(ctx) && !ctx->exception_cb) {
 		/*
+		 * exception_cb uses boundary frame after stack walk.
+		 * It can simply use redzone, this optimization reduces
+		 * stack walk loop by one level.
+		 *
 		 * We need a stack frame, but we don't necessarily need to
 		 * save/restore LR unless we call other functions
 		 */
@@ -190,23 +201,35 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
 	}
 
-	/*
-	 * Back up non-volatile regs -- BPF registers 6-10
-	 * If we haven't created our own stack frame, we save these
-	 * in the protected zone below the previous stack frame
-	 */
-	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
-			EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
+	if (!ctx->exception_cb) {
+		/*
+		 * Back up non-volatile regs -- BPF registers 6-10
+		 * If we haven't created our own stack frame, we save these
+		 * in the protected zone below the previous stack frame
+		 */
+		for (i = BPF_REG_6; i <= BPF_REG_10; i++)
+			if (ctx->exception_boundary || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+				EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
+					bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
 
-	if (ctx->arena_vm_start)
-		EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
+		if (ctx->exception_boundary || ctx->arena_vm_start)
+			EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
 				 bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
+	} else {
+		/*
+		 * Exception callback receives Frame Pointer of boundary
+		 * program(main prog) as third arg
+		 */
+		EMIT(PPC_RAW_MR(_R1, _R5));
+	}
 
-	/* Setup frame pointer to point to the bpf stack area */
+	/*
+	 * Exception_cb not restricted from using stack area or arena.
+	 * Setup frame pointer to point to the bpf stack area
+	 */
 	if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
 		EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
-				STACK_FRAME_MIN_SIZE + ctx->stack_size));
+			STACK_FRAME_MIN_SIZE + ctx->stack_size));
 
 	if (ctx->arena_vm_start)
 		PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);
@@ -218,17 +241,17 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 
 	/* Restore NVRs */
 	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+		if (ctx->exception_cb || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
 			EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
 
-	if (ctx->arena_vm_start)
+	if (ctx->exception_cb || ctx->arena_vm_start)
 		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
 				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
 	/* Tear down our stack frame */
 	if (bpf_has_stack_frame(ctx)) {
 		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
-		if (ctx->seen & SEEN_FUNC) {
+		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
 			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
 			EMIT(PPC_RAW_MTLR(_R0));
 		}
-- 
2.48.1

[PATCH v5 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

The bpf_throw() function never returns, if it has clobbered
any callee-saved register, those will remain clobbered. The
prologue must take care of saving all callee-saved registers
in the frame of exception boundary program. Later these
additional non volatile registers R14-R25 along with other
NVRs are restored back in the epilogue of exception callback.

To achieve above objective, the frame size is determined
dynamically to accommodate additional non volatile registers
in exception boundary's frame.
For non-exception boundary program, the frame size remains
optimal. The additional instructions to save & restore r14-r25
registers are emitted only during exception boundary and
exception callback program respectively.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit_comp64.c | 89 ++++++++++++++++++++++++++++---
 1 file changed, 81 insertions(+), 8 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index db121b1404fe..17de8b53a962 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -32,21 +32,37 @@
  *
  *		[	prev sp		] <-------------
  *		[    tail_call_info	] 8		|
- *		[   nv gpr save area	] 6*8		|
+ *		[   nv gpr save area	] 6*8 + (12*8)	|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
  *		[     frame header	] 32/112	|
  * sp (r1) --->	[    stack pointer	] --------------
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
  */
 
 /* for bpf JIT code internal usage */
 #define BPF_PPC_STACK_LOCALS	24
+/*
+ * for additional non volatile registers(r14-r25) to be saved
+ * at exception boundary
+ */
+#define BPF_PPC_EXC_STACK_SAVE (12*8)
+
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
 #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
 				 BPF_PPC_STACK_LOCALS + \
 				 BPF_PPC_STACK_SAVE   + \
 				 BPF_PPC_TAILCALL)
 
+/*
+ * same as BPF_PPC_STACKFRAME with save area for additional
+ * non volatile registers saved at exception boundary.
+ * This is quad-word aligned.
+ */
+#define BPF_PPC_EXC_STACKFRAME (BPF_PPC_STACKFRAME + BPF_PPC_EXC_STACK_SAVE)
+
 /* BPF register usage */
 #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
 #define TMP_REG_2	(MAX_BPF_JIT_REG + 1)
@@ -88,10 +104,16 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
 	 * - we call other functions (kernel helpers), or
 	 * - the bpf program uses its stack area
 	 * The latter condition is deduced from the usage of BPF_REG_FP
+	 *
+	 * bpf_throw() leads to exception callback from a BPF (sub)program.
+	 * The (sub)program is always marked as SEEN_FUNC, creating a stack
+	 * frame. The exception callback uses the frame of the exception
+	 * boundary, so the exception boundary program must have a frame.
 	 */
 	return ctx->seen & SEEN_FUNC ||
 	       bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) ||
-	       ctx->exception_cb;
+	       ctx->exception_cb ||
+	       ctx->exception_boundary;
 }
 
 /*
@@ -103,9 +125,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
  *		[    tail_call_info	] 8
- *		[   nv gpr save area	] 6*8
+ *		[   nv gpr save area	] 6*8 + (12*8)
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
  */
 static int bpf_jit_stack_local(struct codegen_context *ctx)
 {
@@ -114,7 +139,12 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
 		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
 	} else {
 		/* Stack layout with redzone */
-		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+		return -(BPF_PPC_TAILCALL
+			+BPF_PPC_STACK_SAVE
+			+(ctx->exception_boundary || ctx->exception_cb ?
+						BPF_PPC_EXC_STACK_SAVE : 0)
+			+BPF_PPC_STACK_LOCALS
+			);
 	}
 }
 
@@ -125,9 +155,19 @@ int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
 
 static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
 {
-	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
+	int min_valid_nvreg = BPF_PPC_NVR_MIN;
+	/* Default frame size for all cases except exception boundary */
+	int frame_nvr_size = BPF_PPC_STACKFRAME;
+
+	/* Consider all nv regs for handling exceptions */
+	if (ctx->exception_boundary || ctx->exception_cb) {
+		min_valid_nvreg = _R14;
+		frame_nvr_size = BPF_PPC_EXC_STACKFRAME;
+	}
+
+	if (reg >= min_valid_nvreg && reg < 32)
 		return (bpf_has_stack_frame(ctx) ?
-			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
+			(frame_nvr_size + ctx->stack_size) : 0)
 				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
 
 	pr_err("BPF JIT is asking about unknown registers");
@@ -138,6 +178,17 @@ void bpf_jit_realloc_regs(struct codegen_context *ctx)
 {
 }
 
+/*
+ * For exception boundary & exception_cb progs:
+ *     return increased size to accommodate additional NVRs.
+ */
+static int bpf_jit_stack_size(struct codegen_context *ctx)
+{
+	return ctx->exception_boundary || ctx->exception_cb ?
+					BPF_PPC_EXC_STACKFRAME :
+					BPF_PPC_STACKFRAME;
+}
+
 void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 {
 	int i;
@@ -198,7 +249,19 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
 		}
 
-		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
+		EMIT(PPC_RAW_STDU(_R1, _R1,
+				-(bpf_jit_stack_size(ctx) + ctx->stack_size)));
+	}
+
+	/*
+	 * Program acting as exception boundary pushes R14..R25 in addition to
+	 * BPF callee-saved non volatile registers. Exception callback uses
+	 * the boundary program's stack frame, recover additionally saved
+	 * registers in epilogue of exception callback.
+	 */
+	if (ctx->exception_boundary) {
+		for (i = _R14; i <= _R25; i++)
+			EMIT(PPC_RAW_STD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
 	}
 
 	if (!ctx->exception_cb) {
@@ -248,9 +311,19 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
 				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
+	if (ctx->exception_cb) {
+		/*
+		 * Recover additionally saved non volatile registers from stack
+		 * frame of exception boundary program.
+		 */
+		for (i = _R14; i <= _R25; i++)
+			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
+	}
+
 	/* Tear down our stack frame */
 	if (bpf_has_stack_frame(ctx)) {
-		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
+		EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_jit_stack_size(ctx) + ctx->stack_size));
+
 		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
 			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
 			EMIT(PPC_RAW_MTLR(_R0));
-- 
2.48.1

Re: [PATCH v5 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3170 bytes --]

> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index db121b1404fe..17de8b53a962 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c

[ ... ]

> +/*
> + * For exception boundary & exception_cb progs:
> + *     return increased size to accommodate additional NVRs.
> + */
> +static int bpf_jit_stack_size(struct codegen_context *ctx)
> +{
> +	return ctx->exception_boundary || ctx->exception_cb ?
> +					BPF_PPC_EXC_STACKFRAME :
> +					BPF_PPC_STACKFRAME;
> +}

[ ... ]

> @@ -198,7 +249,19 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
>  			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
>  		}
>
> -		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
> +		EMIT(PPC_RAW_STDU(_R1, _R1,
> +				-(bpf_jit_stack_size(ctx) + ctx->stack_size)));
> +	}

[ ... ]

> @@ -248,9 +311,19 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx)
>  		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
>  				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
>
> +	if (ctx->exception_cb) {
> +		/*
> +		 * Recover additionally saved non volatile registers from stack
> +		 * frame of exception boundary program.
> +		 */
> +		for (i = _R14; i <= _R25; i++)
> +			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
> +	}
> +
>  	/* Tear down our stack frame */
>  	if (bpf_has_stack_frame(ctx)) {
> -		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
> +		EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_jit_stack_size(ctx) + ctx->stack_size));

For the exception callback case, doesn't this use the wrong stack_size?

In bpf_jit_build_prologue(), the exception callback does not create its own
frame - it sets R1 to the boundary's frame pointer via EMIT(PPC_RAW_MR(_R1,
_R5)). However here in the epilogue, the frame teardown uses ctx->stack_size
which is the callback's stack depth, not the boundary's.

The boundary program creates its frame with:
    STDU R1, R1, -(bpf_jit_stack_size(boundary_ctx) + boundary_stack_size)

But the callback tears it down with:
    ADDI R1, R1, bpf_jit_stack_size(callback_ctx) + callback_stack_size

Since each BPF subprogram can have a different stack_depth (set per-subprog
in kernel/bpf/verifier.c via func[i]->aux->stack_depth), if the exception
callback has a different stack depth than the boundary, R1 will end up at
the wrong location after this ADDI.

After the wrong R1 adjustment, the LR restore below loads from R1+16 which
is now pointing to the wrong memory location, potentially causing a return
to an invalid address.

Would it be correct to store the boundary's stack_size somewhere accessible
to the callback, or ensure the callback uses the same stack layout as the
boundary?

> +
>  		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
>  			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
>  			EMIT(PPC_RAW_MTLR(_R0));


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21311936056

Re: [PATCH v5 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[Hari Bathini]

On 24/01/26 1:53 pm, bot+bpf-ci@kernel.org wrote:
>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
>> index db121b1404fe..17de8b53a962 100644
>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> 
> [ ... ]
> 
>> +/*
>> + * For exception boundary & exception_cb progs:
>> + *     return increased size to accommodate additional NVRs.
>> + */
>> +static int bpf_jit_stack_size(struct codegen_context *ctx)
>> +{
>> +	return ctx->exception_boundary || ctx->exception_cb ?
>> +					BPF_PPC_EXC_STACKFRAME :
>> +					BPF_PPC_STACKFRAME;
>> +}
> 
> [ ... ]
> 
>> @@ -198,7 +249,19 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
>>   			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
>>   		}
>>
>> -		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
>> +		EMIT(PPC_RAW_STDU(_R1, _R1,
>> +				-(bpf_jit_stack_size(ctx) + ctx->stack_size)));
>> +	}
> 
> [ ... ]
> 
>> @@ -248,9 +311,19 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx)
>>   		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
>>   				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
>>
>> +	if (ctx->exception_cb) {
>> +		/*
>> +		 * Recover additionally saved non volatile registers from stack
>> +		 * frame of exception boundary program.
>> +		 */
>> +		for (i = _R14; i <= _R25; i++)
>> +			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
>> +	}
>> +
>>   	/* Tear down our stack frame */
>>   	if (bpf_has_stack_frame(ctx)) {
>> -		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
>> +		EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_jit_stack_size(ctx) + ctx->stack_size));
> 
> For the exception callback case, doesn't this use the wrong stack_size?
> 
> In bpf_jit_build_prologue(), the exception callback does not create its own
> frame - it sets R1 to the boundary's frame pointer via EMIT(PPC_RAW_MR(_R1,
> _R5)). However here in the epilogue, the frame teardown uses ctx->stack_size
> which is the callback's stack depth, not the boundary's.
> 
> The boundary program creates its frame with:
>      STDU R1, R1, -(bpf_jit_stack_size(boundary_ctx) + boundary_stack_size)
> 
> But the callback tears it down with:
>      ADDI R1, R1, bpf_jit_stack_size(callback_ctx) + callback_stack_size
> 
> Since each BPF subprogram can have a different stack_depth (set per-subprog
> in kernel/bpf/verifier.c via func[i]->aux->stack_depth), if the exception
> callback has a different stack depth than the boundary, R1 will end up at
> the wrong location after this ADDI.
> 
> After the wrong R1 adjustment, the LR restore below loads from R1+16 which
> is now pointing to the wrong memory location, potentially causing a return
> to an invalid address.
> 
> Would it be correct to store the boundary's stack_size somewhere accessible
> to the callback, or ensure the callback uses the same stack layout as the
> boundary?

That is a good catch! Jitting the rollback of the stack frame of
exception boundary program in exception callback program is tricky
as stack_depth of exception boundary program is not readily available
while jitting exception callback program...

"ld r1, 0(r1)" instruction can be used instead of "addi r1, r1, size"
form for tearing down the stack.

The other issue is calculating the offset of non volatile registers
to restore them from the stack. To reliably restore, move one frame
above exception boundary frame and restore NVRs with redzone offset
without relying on stack_depth of exception boundary program.

Also, as exception callback may also use stack, setup frame for
exception callback and restore BPF JIT NVRs in its epilogue.

In essence, something like the below is needed on top of your patch..

---

diff --git a/arch/powerpc/net/bpf_jit_comp64.c 
b/arch/powerpc/net/bpf_jit_comp64.c
index b1a3945ccc9f..15342d76f96f 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -32,7 +32,8 @@
   *
   *		[	prev sp		] <-------------
   *		[    tail_call_info	] 8		|
- *		[   nv gpr save area	] 6*8 + (12*8)	|
+ *		[   nv gpr save area	] 6*8		|
+ *		[ addl. nv gpr save area] (12*8)	| <--- only in exception boundary 
program
   *		[    local_tmp_var	] 24		|
   * fp (r31) -->	[   ebpf stack space	] upto 512	|
   *		[     frame header	] 32/112	|
@@ -125,7 +126,8 @@ static inline bool bpf_has_stack_frame(struct 
codegen_context *ctx)
   *		[	  ...       	] 		|
   * sp (r1) --->	[    stack pointer	] --------------
   *		[    tail_call_info	] 8
- *		[   nv gpr save area	] 6*8 + (12*8)
+ *		[   nv gpr save area	] 6*8
+ *		[ addl. nv gpr save area] (12*8) <--- Only in case of exception 
boundary
   *		[    local_tmp_var	] 24
   *		[   unused red zone	] 224
   *
@@ -139,12 +141,9 @@ static int bpf_jit_stack_local(struct 
codegen_context *ctx)
  		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
  	} else {
  		/* Stack layout with redzone */
-		return -(BPF_PPC_TAILCALL
-			+BPF_PPC_STACK_SAVE
-			+(ctx->exception_boundary || ctx->exception_cb ?
-						BPF_PPC_EXC_STACK_SAVE : 0)
-			+BPF_PPC_STACK_LOCALS
-			);
+		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE +
+			 (ctx->exception_boundary ? BPF_PPC_EXC_STACK_SAVE : 0) +
+			 BPF_PPC_STACK_LOCALS);
  	}
  }

@@ -153,20 +152,27 @@ int bpf_jit_stack_tailcallinfo_offset(struct 
codegen_context *ctx)
  	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + 
BPF_PPC_STACK_SAVE;
  }

-static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
+static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg, 
bool use_redzone)
  {
  	int min_valid_nvreg = BPF_PPC_NVR_MIN;
  	/* Default frame size for all cases except exception boundary */
  	int frame_nvr_size = BPF_PPC_STACKFRAME;

+	/*
+	 * When use_redzone is true, return offset of the NVR on redzone
+	 * Else, return offset based on whether stack is setup or not.
+	 */
+	if (!use_redzone)
+		use_redzone = bpf_has_stack_frame(ctx);
+
  	/* Consider all nv regs for handling exceptions */
-	if (ctx->exception_boundary || ctx->exception_cb) {
+	if (ctx->exception_boundary) {
  		min_valid_nvreg = _R14;
  		frame_nvr_size = BPF_PPC_EXC_STACKFRAME;
  	}

  	if (reg >= min_valid_nvreg && reg < 32)
-		return (bpf_has_stack_frame(ctx) ?
+		return (!use_redzone ?
  			(frame_nvr_size + ctx->stack_size) : 0)
  				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;

@@ -179,14 +185,12 @@ void bpf_jit_realloc_regs(struct codegen_context *ctx)
  }

  /*
- * For exception boundary & exception_cb progs:
- *     return increased size to accommodate additional NVRs.
+ * For exception boundary program, return increased size to
+ * accommodate additional NVRs.
   */
  static int bpf_jit_stack_size(struct codegen_context *ctx)
  {
-	return ctx->exception_boundary || ctx->exception_cb ?
-					BPF_PPC_EXC_STACKFRAME :
-					BPF_PPC_STACKFRAME;
+	return ctx->exception_boundary ? BPF_PPC_EXC_STACKFRAME : 
BPF_PPC_STACKFRAME;
  }

  void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
@@ -235,12 +239,30 @@ void bpf_jit_build_prologue(u32 *image, struct 
codegen_context *ctx)
  		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
  	}

-	if (bpf_has_stack_frame(ctx) && !ctx->exception_cb) {
+	if (ctx->exception_cb) {
+		/*
+		 * Exception callback receives frame pointer of exception boundary 
program as
+		 * third argument but reliable offset calculation for NVRs is only 
possible
+		 * on the redzone because stack_depth of exception boundary is not known
+		 * while jitting the exception_cb program. So, rollback to an sp above
+		 * exception boundary frame and restore the additional NVRs using redzone
+		 * offset. The regular BPF JIT NVRs can be restored now, or in the 
epilogue
+		 * of exception_cb like any other bpf prog by reusing the BPF JIT 
NVRs part
+		 * of exception_boundary stack frame. The latter is preferred as 
exception
+		 * cb may also clobber the BPF JIT NVRs.
+		 */
+		EMIT(PPC_RAW_LD(_R1, _R5, 0));
+
+		/*
+		 * Recover additionally saved non volatile registers from stack
+		 * frame of exception boundary program.
+		 */
+		for (i = _R14; i <= _R25; i++)
+			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i, true)));
+	}
+
+	if (bpf_has_stack_frame(ctx)) {
  		/*
-		 * exception_cb uses boundary frame after stack walk.
-		 * It can simply use redzone, this optimization reduces
-		 * stack walk loop by one level.
-		 *
  		 * We need a stack frame, but we don't necessarily need to
  		 * save/restore LR unless we call other functions
  		 */
@@ -257,11 +279,11 @@ void bpf_jit_build_prologue(u32 *image, struct 
codegen_context *ctx)
  	 * Program acting as exception boundary pushes R14..R25 in addition to
  	 * BPF callee-saved non volatile registers. Exception callback uses
  	 * the boundary program's stack frame, recover additionally saved
-	 * registers in epilogue of exception callback.
+	 * registers in prologue of exception callback.
  	 */
  	if (ctx->exception_boundary) {
  		for (i = _R14; i <= _R25; i++)
-			EMIT(PPC_RAW_STD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
+			EMIT(PPC_RAW_STD(i, _R1, bpf_jit_stack_offsetof(ctx, i, false)));
  	}

  	if (!ctx->exception_cb) {
@@ -273,17 +295,11 @@ void bpf_jit_build_prologue(u32 *image, struct 
codegen_context *ctx)
  		for (i = BPF_REG_6; i <= BPF_REG_10; i++)
  			if (ctx->exception_boundary || bpf_is_seen_register(ctx, 
bpf_to_ppc(i)))
  				EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
-					bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
+					bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i), false)));

  		if (ctx->exception_boundary || ctx->arena_vm_start)
  			EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
-				 bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
-	} else {
-		/*
-		 * Exception callback receives Frame Pointer of boundary
-		 * program(main prog) as third arg
-		 */
-		EMIT(PPC_RAW_MR(_R1, _R5));
+				 bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START), false)));
  	}

  	/*
@@ -305,20 +321,12 @@ static void bpf_jit_emit_common_epilogue(u32 
*image, struct codegen_context *ctx
  	/* Restore NVRs */
  	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
  		if (ctx->exception_cb || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
-			EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, 
bpf_to_ppc(i))));
+			EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1,
+					bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i), false)));

  	if (ctx->exception_cb || ctx->arena_vm_start)
  		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
-				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
-
-	if (ctx->exception_cb) {
-		/*
-		 * Recover additionally saved non volatile registers from stack
-		 * frame of exception boundary program.
-		 */
-		for (i = _R14; i <= _R25; i++)
-			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
-	}
+				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START), false)));

  	/* Tear down our stack frame */
  	if (bpf_has_stack_frame(ctx)) {

- Hari

Re: [PATCH v5 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[Hari Bathini]

On 27/01/26 12:00 am, Hari Bathini wrote:
> 
> 
> On 24/01/26 1:53 pm, bot+bpf-ci@kernel.org wrote:
>>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/ 
>>> bpf_jit_comp64.c
>>> index db121b1404fe..17de8b53a962 100644
>>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
>>
>> [ ... ]
>>
>>> +/*
>>> + * For exception boundary & exception_cb progs:
>>> + *     return increased size to accommodate additional NVRs.
>>> + */
>>> +static int bpf_jit_stack_size(struct codegen_context *ctx)
>>> +{
>>> +    return ctx->exception_boundary || ctx->exception_cb ?
>>> +                    BPF_PPC_EXC_STACKFRAME :
>>> +                    BPF_PPC_STACKFRAME;
>>> +}
>>
>> [ ... ]
>>
>>> @@ -198,7 +249,19 @@ void bpf_jit_build_prologue(u32 *image, struct 
>>> codegen_context *ctx)
>>>               EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
>>>           }
>>>
>>> -        EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx- 
>>> >stack_size)));
>>> +        EMIT(PPC_RAW_STDU(_R1, _R1,
>>> +                -(bpf_jit_stack_size(ctx) + ctx->stack_size)));
>>> +    }
>>
>> [ ... ]
>>
>>> @@ -248,9 +311,19 @@ static void bpf_jit_emit_common_epilogue(u32 
>>> *image, struct codegen_context *ctx)
>>>           EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
>>>                   bpf_jit_stack_offsetof(ctx, 
>>> bpf_to_ppc(ARENA_VM_START))));
>>>
>>> +    if (ctx->exception_cb) {
>>> +        /*
>>> +         * Recover additionally saved non volatile registers from stack
>>> +         * frame of exception boundary program.
>>> +         */
>>> +        for (i = _R14; i <= _R25; i++)
>>> +            EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
>>> +    }
>>> +
>>>       /* Tear down our stack frame */
>>>       if (bpf_has_stack_frame(ctx)) {
>>> -        EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx- 
>>> >stack_size));
>>> +        EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_jit_stack_size(ctx) + ctx- 
>>> >stack_size));
>>
>> For the exception callback case, doesn't this use the wrong stack_size?
>>
>> In bpf_jit_build_prologue(), the exception callback does not create 
>> its own
>> frame - it sets R1 to the boundary's frame pointer via 
>> EMIT(PPC_RAW_MR(_R1,
>> _R5)). However here in the epilogue, the frame teardown uses ctx- 
>> >stack_size
>> which is the callback's stack depth, not the boundary's.
>>
>> The boundary program creates its frame with:
>>      STDU R1, R1, -(bpf_jit_stack_size(boundary_ctx) + 
>> boundary_stack_size)
>>
>> But the callback tears it down with:
>>      ADDI R1, R1, bpf_jit_stack_size(callback_ctx) + callback_stack_size
>>
>> Since each BPF subprogram can have a different stack_depth (set per- 
>> subprog
>> in kernel/bpf/verifier.c via func[i]->aux->stack_depth), if the exception
>> callback has a different stack depth than the boundary, R1 will end up at
>> the wrong location after this ADDI.
>>
>> After the wrong R1 adjustment, the LR restore below loads from R1+16 
>> which
>> is now pointing to the wrong memory location, potentially causing a 
>> return
>> to an invalid address.
>>
>> Would it be correct to store the boundary's stack_size somewhere 
>> accessible
>> to the callback, or ensure the callback uses the same stack layout as the
>> boundary?
> 
> That is a good catch! Jitting the rollback of the stack frame of
> exception boundary program in exception callback program is tricky
> as stack_depth of exception boundary program is not readily available
> while jitting exception callback program...
> 
> "ld r1, 0(r1)" instruction can be used instead of "addi r1, r1, size"
> form for tearing down the stack.
> 
> The other issue is calculating the offset of non volatile registers
> to restore them from the stack. To reliably restore, move one frame
> above exception boundary frame and restore NVRs with redzone offset
> without relying on stack_depth of exception boundary program.
> 
> Also, as exception callback may also use stack, setup frame for
> exception callback and restore BPF JIT NVRs in its epilogue.
> 
> In essence, something like the below is needed on top of your patch..
> 
> ---
> 
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/ 
> bpf_jit_comp64.c
> index b1a3945ccc9f..15342d76f96f 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -32,7 +32,8 @@
>    *
>    *        [    prev sp        ] <-------------
>    *        [    tail_call_info    ] 8        |
> - *        [   nv gpr save area    ] 6*8 + (12*8)    |
> + *        [   nv gpr save area    ] 6*8        |
> + *        [ addl. nv gpr save area] (12*8)    | <--- only in exception 
> boundary program
>    *        [    local_tmp_var    ] 24        |
>    * fp (r31) -->    [   ebpf stack space    ] upto 512    |
>    *        [     frame header    ] 32/112    |
> @@ -125,7 +126,8 @@ static inline bool bpf_has_stack_frame(struct 
> codegen_context *ctx)
>    *        [      ...           ]         |
>    * sp (r1) --->    [    stack pointer    ] --------------
>    *        [    tail_call_info    ] 8
> - *        [   nv gpr save area    ] 6*8 + (12*8)
> + *        [   nv gpr save area    ] 6*8
> + *        [ addl. nv gpr save area] (12*8) <--- Only in case of 
> exception boundary
>    *        [    local_tmp_var    ] 24
>    *        [   unused red zone    ] 224
>    *
> @@ -139,12 +141,9 @@ static int bpf_jit_stack_local(struct 
> codegen_context *ctx)
>           return STACK_FRAME_MIN_SIZE + ctx->stack_size;
>       } else {
>           /* Stack layout with redzone */
> -        return -(BPF_PPC_TAILCALL
> -            +BPF_PPC_STACK_SAVE
> -            +(ctx->exception_boundary || ctx->exception_cb ?
> -                        BPF_PPC_EXC_STACK_SAVE : 0)
> -            +BPF_PPC_STACK_LOCALS
> -            );
> +        return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE +
> +             (ctx->exception_boundary ? BPF_PPC_EXC_STACK_SAVE : 0) +
> +             BPF_PPC_STACK_LOCALS);
>       }
>   }
> 
> @@ -153,20 +152,27 @@ int bpf_jit_stack_tailcallinfo_offset(struct 
> codegen_context *ctx)
>       return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + 
> BPF_PPC_STACK_SAVE;
>   }
> 
> -static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
> +static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg, 
> bool use_redzone)
>   {
>       int min_valid_nvreg = BPF_PPC_NVR_MIN;
>       /* Default frame size for all cases except exception boundary */
>       int frame_nvr_size = BPF_PPC_STACKFRAME;
> 
> +    /*
> +     * When use_redzone is true, return offset of the NVR on redzone
> +     * Else, return offset based on whether stack is setup or not.
> +     */
> +    if (!use_redzone)
> +        use_redzone = bpf_has_stack_frame(ctx);
> +
>       /* Consider all nv regs for handling exceptions */
> -    if (ctx->exception_boundary || ctx->exception_cb) {
> +    if (ctx->exception_boundary) {
>           min_valid_nvreg = _R14;
>           frame_nvr_size = BPF_PPC_EXC_STACKFRAME;
>       }
> 

>       if (reg >= min_valid_nvreg && reg < 32)

On a closely look, min_valid_nvreg has to be _R14 for exception_cb to
satisfy the above reg condition check...

> -        return (bpf_has_stack_frame(ctx) ?
> +        return (!use_redzone ?
>               (frame_nvr_size + ctx->stack_size) : 0)
>                   - (8 * (32 - reg)) - BPF_PPC_TAILCALL;
> 
> @@ -179,14 +185,12 @@ void bpf_jit_realloc_regs(struct codegen_context 
> *ctx)
>   }
> 
>   /*
> - * For exception boundary & exception_cb progs:
> - *     return increased size to accommodate additional NVRs.
> + * For exception boundary program, return increased size to
> + * accommodate additional NVRs.
>    */
>   static int bpf_jit_stack_size(struct codegen_context *ctx)
>   {
> -    return ctx->exception_boundary || ctx->exception_cb ?
> -                    BPF_PPC_EXC_STACKFRAME :
> -                    BPF_PPC_STACKFRAME;
> +    return ctx->exception_boundary ? BPF_PPC_EXC_STACKFRAME : 
> BPF_PPC_STACKFRAME;
>   }
> 
>   void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
> @@ -235,12 +239,30 @@ void bpf_jit_build_prologue(u32 *image, struct 
> codegen_context *ctx)
>           EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - 
> (BPF_PPC_TAILCALL)));
>       }
> 
> -    if (bpf_has_stack_frame(ctx) && !ctx->exception_cb) {
> +    if (ctx->exception_cb) {
> +        /*
> +         * Exception callback receives frame pointer of exception 
> boundary program as
> +         * third argument but reliable offset calculation for NVRs is 
> only possible
> +         * on the redzone because stack_depth of exception boundary is 
> not known
> +         * while jitting the exception_cb program. So, rollback to an 
> sp above
> +         * exception boundary frame and restore the additional NVRs 
> using redzone
> +         * offset. The regular BPF JIT NVRs can be restored now, or in 
> the epilogue
> +         * of exception_cb like any other bpf prog by reusing the BPF 
> JIT NVRs part
> +         * of exception_boundary stack frame. The latter is preferred 
> as exception
> +         * cb may also clobber the BPF JIT NVRs.
> +         */
> +        EMIT(PPC_RAW_LD(_R1, _R5, 0));
> +
> +        /*
> +         * Recover additionally saved non volatile registers from stack
> +         * frame of exception boundary program.
> +         */
> +        for (i = _R14; i <= _R25; i++)
> +            EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i, 
> true)));
> +    }
> +
> +    if (bpf_has_stack_frame(ctx)) {
>           /*
> -         * exception_cb uses boundary frame after stack walk.
> -         * It can simply use redzone, this optimization reduces
> -         * stack walk loop by one level.
> -         *
>            * We need a stack frame, but we don't necessarily need to
>            * save/restore LR unless we call other functions
>            */
> @@ -257,11 +279,11 @@ void bpf_jit_build_prologue(u32 *image, struct 
> codegen_context *ctx)
>        * Program acting as exception boundary pushes R14..R25 in 
> addition to
>        * BPF callee-saved non volatile registers. Exception callback uses
>        * the boundary program's stack frame, recover additionally saved
> -     * registers in epilogue of exception callback.
> +     * registers in prologue of exception callback.
>        */
>       if (ctx->exception_boundary) {
>           for (i = _R14; i <= _R25; i++)
> -            EMIT(PPC_RAW_STD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
> +            EMIT(PPC_RAW_STD(i, _R1, bpf_jit_stack_offsetof(ctx, i, 
> false)));
>       }
> 
>       if (!ctx->exception_cb) {
> @@ -273,17 +295,11 @@ void bpf_jit_build_prologue(u32 *image, struct 
> codegen_context *ctx)
>           for (i = BPF_REG_6; i <= BPF_REG_10; i++)
>               if (ctx->exception_boundary || bpf_is_seen_register(ctx, 
> bpf_to_ppc(i)))
>                   EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
> -                    bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
> +                    bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i), false)));
> 
>           if (ctx->exception_boundary || ctx->arena_vm_start)
>               EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
> -                 bpf_jit_stack_offsetof(ctx, 
> bpf_to_ppc(ARENA_VM_START))));
> -    } else {
> -        /*
> -         * Exception callback receives Frame Pointer of boundary
> -         * program(main prog) as third arg
> -         */
> -        EMIT(PPC_RAW_MR(_R1, _R5));
> +                 bpf_jit_stack_offsetof(ctx, 
> bpf_to_ppc(ARENA_VM_START), false)));
>       }
> 
>       /*
> @@ -305,20 +321,12 @@ static void bpf_jit_emit_common_epilogue(u32 
> *image, struct codegen_context *ctx
>       /* Restore NVRs */
>       for (i = BPF_REG_6; i <= BPF_REG_10; i++)
>           if (ctx->exception_cb || bpf_is_seen_register(ctx, 
> bpf_to_ppc(i)))
> -            EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, 
> bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
> +            EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1,
> +                    bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i), false)));
> 
>       if (ctx->exception_cb || ctx->arena_vm_start)
>           EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
> -                bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
> -
> -    if (ctx->exception_cb) {
> -        /*
> -         * Recover additionally saved non volatile registers from stack
> -         * frame of exception boundary program.
> -         */
> -        for (i = _R14; i <= _R25; i++)
> -            EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
> -    }
> +                bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START), 
> false)));
> 
>       /* Tear down our stack frame */
>       if (bpf_has_stack_frame(ctx)) {

Also, if all exception selftests are passing, it is likely that neither
exception boundary nor exception callback is using BPF stack.
Need to have a test case that tests different BPF stack size for
exception boundary and exception callback..

- Hari

Re: [PATCH v5 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[Hari Bathini]

On 24/01/26 1:53 pm, bot+bpf-ci@kernel.org wrote:
>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
>> index db121b1404fe..17de8b53a962 100644
>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> 
> [ ... ]
> 
>> +/*
>> + * For exception boundary & exception_cb progs:
>> + *     return increased size to accommodate additional NVRs.
>> + */
>> +static int bpf_jit_stack_size(struct codegen_context *ctx)
>> +{
>> +	return ctx->exception_boundary || ctx->exception_cb ?
>> +					BPF_PPC_EXC_STACKFRAME :
>> +					BPF_PPC_STACKFRAME;
>> +}
> 
> [ ... ]
> 
>> @@ -198,7 +249,19 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
>>   			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
>>   		}
>>
>> -		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
>> +		EMIT(PPC_RAW_STDU(_R1, _R1,
>> +				-(bpf_jit_stack_size(ctx) + ctx->stack_size)));
>> +	}
> 
> [ ... ]
> 
>> @@ -248,9 +311,19 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx)
>>   		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
>>   				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
>>
>> +	if (ctx->exception_cb) {
>> +		/*
>> +		 * Recover additionally saved non volatile registers from stack
>> +		 * frame of exception boundary program.
>> +		 */
>> +		for (i = _R14; i <= _R25; i++)
>> +			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
>> +	}
>> +
>>   	/* Tear down our stack frame */
>>   	if (bpf_has_stack_frame(ctx)) {
>> -		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
>> +		EMIT(PPC_RAW_ADDI(_R1, _R1, bpf_jit_stack_size(ctx) + ctx->stack_size));
> 
> For the exception callback case, doesn't this use the wrong stack_size?
> 
> In bpf_jit_build_prologue(), the exception callback does not create its own
> frame - it sets R1 to the boundary's frame pointer via EMIT(PPC_RAW_MR(_R1,
> _R5)). However here in the epilogue, the frame teardown uses ctx->stack_size
> which is the callback's stack depth, not the boundary's.
> 
> The boundary program creates its frame with:
>      STDU R1, R1, -(bpf_jit_stack_size(boundary_ctx) + boundary_stack_size)
> 
> But the callback tears it down with:
>      ADDI R1, R1, bpf_jit_stack_size(callback_ctx) + callback_stack_size
> 
> Since each BPF subprogram can have a different stack_depth (set per-subprog
> in kernel/bpf/verifier.c via func[i]->aux->stack_depth), if the exception
> callback has a different stack depth than the boundary, R1 will end up at
> the wrong location after this ADDI.
> 
> After the wrong R1 adjustment, the LR restore below loads from R1+16 which
> is now pointing to the wrong memory location, potentially causing a return
> to an invalid address.
> 
> Would it be correct to store the boundary's stack_size somewhere accessible
> to the callback, or ensure the callback uses the same stack layout as the
> boundary?
> 
>> +
>>   		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
>>   			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
>>   			EMIT(PPC_RAW_MTLR(_R0));
> 
> 
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.

I will post the fix for this issue as a follow-up.

Thanks
Hari

Re: [PATCH v5 0/6] powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

[Venkat]

> On 24 Jan 2026, at 1:22 PM, adubey@linux.ibm.com wrote:
> 
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> This patch series enables support for two BPF JIT features
> on powerpc64. The first three patches target support for
> tail calls with subprogram combinations. The first patch
> supports realignment of tail_call_cnt offset in stack frame.
> Implementation details are provided in the commit messages.
> 
> The last three patches add support for BPF exceptions. An
> architecture-specific stack walker is implemented to assist
> with stack walk during exceptions.
> 
> All selftests related to tailcalls and exceptions are passing:
> 
> # ./test_progs -t tailcalls
> #442/1   tailcalls/tailcall_1:OK
> #442/2   tailcalls/tailcall_2:OK
> #442/3   tailcalls/tailcall_3:OK
> #442/4   tailcalls/tailcall_4:OK
> #442/5   tailcalls/tailcall_5:OK
> #442/6   tailcalls/tailcall_6:OK
> #442/7   tailcalls/tailcall_bpf2bpf_1:OK
> #442/8   tailcalls/tailcall_bpf2bpf_2:OK
> #442/9   tailcalls/tailcall_bpf2bpf_3:OK
> #442/10  tailcalls/tailcall_bpf2bpf_4:OK
> #442/11  tailcalls/tailcall_bpf2bpf_5:OK
> #442/12  tailcalls/tailcall_bpf2bpf_6:OK
> #442/13  tailcalls/tailcall_bpf2bpf_fentry:OK
> #442/14  tailcalls/tailcall_bpf2bpf_fexit:OK
> #442/15  tailcalls/tailcall_bpf2bpf_fentry_fexit:OK
> #442/16  tailcalls/tailcall_bpf2bpf_fentry_entry:OK
> #442/17  tailcalls/tailcall_poke:OK
> #442/18  tailcalls/tailcall_bpf2bpf_hierarchy_1:OK
> #442/19  tailcalls/tailcall_bpf2bpf_hierarchy_fentry:OK
> #442/20  tailcalls/tailcall_bpf2bpf_hierarchy_fexit:OK
> #442/21  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_fexit:OK
> #442/22  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_entry:OK
> #442/23  tailcalls/tailcall_bpf2bpf_hierarchy_2:OK
> #442/24  tailcalls/tailcall_bpf2bpf_hierarchy_3:OK
> #442/25  tailcalls/tailcall_freplace:OK
> #442/26  tailcalls/tailcall_bpf2bpf_freplace:OK
> #442/27  tailcalls/tailcall_failure:OK
> #442/28  tailcalls/reject_tail_call_spin_lock:OK
> #442/29  tailcalls/reject_tail_call_rcu_lock:OK
> #442/30  tailcalls/reject_tail_call_preempt_lock:OK
> #442/31  tailcalls/reject_tail_call_ref:OK
> #442     tailcalls:OK
> Summary: 1/31 PASSED, 0 SKIPPED, 0 FAILED
> 
> # ./test_progs -t exceptions
> #105/1   exceptions/exception_throw_always_1:OK
> #105/2   exceptions/exception_throw_always_2:OK
> #105/3   exceptions/exception_throw_unwind_1:OK
> #105/4   exceptions/exception_throw_unwind_2:OK
> #105/5   exceptions/exception_throw_default:OK
> #105/6   exceptions/exception_throw_default_value:OK
> #105/7   exceptions/exception_tail_call:OK
> #105/8   exceptions/exception_ext:OK
> #105/9   exceptions/exception_ext_mod_cb_runtime:OK
> #105/10  exceptions/exception_throw_subprog:OK
> #105/11  exceptions/exception_assert_nz_gfunc:OK
> #105/12  exceptions/exception_assert_zero_gfunc:OK
> #105/13  exceptions/exception_assert_neg_gfunc:OK
> #105/14  exceptions/exception_assert_pos_gfunc:OK
> #105/15  exceptions/exception_assert_negeq_gfunc:OK
> #105/16  exceptions/exception_assert_poseq_gfunc:OK
> #105/17  exceptions/exception_assert_nz_gfunc_with:OK
> #105/18  exceptions/exception_assert_zero_gfunc_with:OK
> #105/19  exceptions/exception_assert_neg_gfunc_with:OK
> #105/20  exceptions/exception_assert_pos_gfunc_with:OK
> #105/21  exceptions/exception_assert_negeq_gfunc_with:OK
> #105/22  exceptions/exception_assert_poseq_gfunc_with:OK
> #105/23  exceptions/exception_bad_assert_nz_gfunc:OK
> #105/24  exceptions/exception_bad_assert_zero_gfunc:OK
> #105/25  exceptions/exception_bad_assert_neg_gfunc:OK
> #105/26  exceptions/exception_bad_assert_pos_gfunc:OK
> #105/27  exceptions/exception_bad_assert_negeq_gfunc:OK
> #105/28  exceptions/exception_bad_assert_poseq_gfunc:OK
> #105/29  exceptions/exception_bad_assert_nz_gfunc_with:OK
> #105/30  exceptions/exception_bad_assert_zero_gfunc_with:OK
> #105/31  exceptions/exception_bad_assert_neg_gfunc_with:OK
> #105/32  exceptions/exception_bad_assert_pos_gfunc_with:OK
> #105/33  exceptions/exception_bad_assert_negeq_gfunc_with:OK
> #105/34  exceptions/exception_bad_assert_poseq_gfunc_with:OK
> #105/35  exceptions/exception_assert_range:OK
> #105/36  exceptions/exception_assert_range_with:OK
> #105/37  exceptions/exception_bad_assert_range:OK
> #105/38  exceptions/exception_bad_assert_range_with:OK
> #105/39  exceptions/non-throwing fentry -> exception_cb:OK
> #105/40  exceptions/throwing fentry -> exception_cb:OK
> #105/41  exceptions/non-throwing fexit -> exception_cb:OK
> #105/42  exceptions/throwing fexit -> exception_cb:OK
> #105/43  exceptions/throwing extension (with custom cb) -> exception_cb:OK
> #105/44  exceptions/throwing extension -> global func in exception_cb:OK
> #105/45  exceptions/exception_ext_mod_cb_runtime:OK
> #105/46  exceptions/throwing extension (with custom cb) -> global func in exception_cb:OK
> #105/47  exceptions/exception_ext:OK
> #105/48  exceptions/non-throwing fentry -> non-throwing subprog:OK
> #105/49  exceptions/throwing fentry -> non-throwing subprog:OK
> #105/50  exceptions/non-throwing fentry -> throwing subprog:OK
> #105/51  exceptions/throwing fentry -> throwing subprog:OK
> #105/52  exceptions/non-throwing fexit -> non-throwing subprog:OK
> #105/53  exceptions/throwing fexit -> non-throwing subprog:OK
> #105/54  exceptions/non-throwing fexit -> throwing subprog:OK
> #105/55  exceptions/throwing fexit -> throwing subprog:OK
> #105/56  exceptions/non-throwing fmod_ret -> non-throwing subprog:OK
> #105/57  exceptions/non-throwing fmod_ret -> non-throwing global subprog:OK
> #105/58  exceptions/non-throwing extension -> non-throwing subprog:OK
> #105/59  exceptions/non-throwing extension -> throwing subprog:OK
> #105/60  exceptions/non-throwing extension -> non-throwing subprog:OK
> #105/61  exceptions/non-throwing extension -> throwing global subprog:OK
> #105/62  exceptions/throwing extension -> throwing global subprog:OK
> #105/63  exceptions/throwing extension -> non-throwing global subprog:OK
> #105/64  exceptions/non-throwing extension -> main subprog:OK
> #105/65  exceptions/throwing extension -> main subprog:OK
> #105/66  exceptions/reject_exception_cb_type_1:OK
> #105/67  exceptions/reject_exception_cb_type_2:OK
> #105/68  exceptions/reject_exception_cb_type_3:OK
> #105/69  exceptions/reject_exception_cb_type_4:OK
> #105/70  exceptions/reject_async_callback_throw:OK
> #105/71  exceptions/reject_with_lock:OK
> #105/72  exceptions/reject_subprog_with_lock:OK
> #105/73  exceptions/reject_with_rcu_read_lock:OK
> #105/74  exceptions/reject_subprog_with_rcu_read_lock:OK
> #105/75  exceptions/reject_with_rbtree_add_throw:OK
> #105/76  exceptions/reject_with_reference:OK
> #105/77  exceptions/reject_with_cb_reference:OK
> #105/78  exceptions/reject_with_cb:OK
> #105/79  exceptions/reject_with_subprog_reference:OK
> #105/80  exceptions/reject_throwing_exception_cb:OK
> #105/81  exceptions/reject_exception_cb_call_global_func:OK
> #105/82  exceptions/reject_exception_cb_call_static_func:OK
> #105/83  exceptions/reject_multiple_exception_cb:OK
> #105/84  exceptions/reject_exception_throw_cb:OK
> #105/85  exceptions/reject_exception_throw_cb_diff:OK
> #105/86  exceptions/reject_set_exception_cb_bad_ret1:OK
> #105/87  exceptions/reject_set_exception_cb_bad_ret2:OK
> #105/88  exceptions/check_assert_eq_int_min:OK
> #105/89  exceptions/check_assert_eq_int_max:OK
> #105/90  exceptions/check_assert_eq_zero:OK
> #105/91  exceptions/check_assert_eq_llong_min:OK
> #105/92  exceptions/check_assert_eq_llong_max:OK
> #105/93  exceptions/check_assert_lt_pos:OK
> #105/94  exceptions/check_assert_lt_zero:OK
> #105/95  exceptions/check_assert_lt_neg:OK
> #105/96  exceptions/check_assert_le_pos:OK
> #105/97  exceptions/check_assert_le_zero:OK
> #105/98  exceptions/check_assert_le_neg:OK
> #105/99  exceptions/check_assert_gt_pos:OK
> #105/100 exceptions/check_assert_gt_zero:OK
> #105/101 exceptions/check_assert_gt_neg:OK
> #105/102 exceptions/check_assert_ge_pos:OK
> #105/103 exceptions/check_assert_ge_zero:OK
> #105/104 exceptions/check_assert_ge_neg:OK
> #105/105 exceptions/check_assert_range_s64:OK
> #105/106 exceptions/check_assert_range_u64:OK
> #105/107 exceptions/check_assert_single_range_s64:OK
> #105/108 exceptions/check_assert_single_range_u64:OK
> #105/109 exceptions/check_assert_generic:OK
> #105/110 exceptions/check_assert_with_return:OK
> #105     exceptions:OK
> Summary: 1/110 PASSED, 0 SKIPPED, 0 FAILED
> 
> [v1]: https://lore.kernel.org/all/20260105105212.136645-1-adubey@linux.ibm.com/
> [v2]: https://lore.kernel.org/all/20260114114450.30405-1-adubey@linux.ibm.com/
> [v3]: https://lore.kernel.org/all/20260122165716.10508-1-adubey@linux.ibm.com/
> [v4]: https://lore.kernel.org/all/20260122211854.5508-1-adubey@linux.ibm.com/
> 
> Changes v4->v5:
> Patch comments and code refactoring
> Handle exception callback and boundary frame checks
> Changes v3->v4:
>        Handle bpf-ci warnings
> Changes v2->v3:
>        Added PPC_BCC_CONST_SHORT for short jumps of constant offset
>        Optimize tailcall allocation for BPF_TRAMP_F_CALL_ORIG flag
>        New helper for stack size calculation during exceptions
>        Prologue JIT optimizations during non exception prog case
> Changes v1->v2:
>        Move tail_call_cnt to offset 0 in stack frame
>        Remove trampoline NVR remapping-patch3/6
> 
> Abhishek Dubey (6):
>  powerpc64/bpf: Moving tail_call_cnt to bottom of frame
>  powerpc64/bpf: Support tailcalls with subprogs
>  powerpc64/bpf: Avoid tailcall restore from trampoline
>  powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT
>  powerpc64/bpf: Support exceptions
>  powerpc64/bpf: Additional NVR handling for bpf_throw
> 
> arch/powerpc/net/bpf_jit.h        |  16 ++
> arch/powerpc/net/bpf_jit_comp.c   |  85 +++++++--
> arch/powerpc/net/bpf_jit_comp64.c | 276 +++++++++++++++++++++++++-----
> 3 files changed, 315 insertions(+), 62 deletions(-)
> 
> -- 
> 2.48.1
> 

Tested this series, it’s working as expected. Please add below tag.

Tested-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>

Regards,
Venkat.