[PATCH v2 0/6] powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

[PATCH v2 0/6] powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

This patch series enables support for two BPF JIT features
on powerpc64. The first three patches target support for
tail calls with subprogram combinations. The first patch
supports realignment of tail_call_cnt offset in stack frame.
Implementation details are provided in the commit messages.

The last three patches add support for BPF exceptions. An
architecture-specific stack walker is implemented to assist
with stack walk during exceptions.

All selftests related to tailcalls and exceptions are passing:

# ./test_progs -t tailcalls
#442/1   tailcalls/tailcall_1:OK
#442/2   tailcalls/tailcall_2:OK
#442/3   tailcalls/tailcall_3:OK
#442/4   tailcalls/tailcall_4:OK
#442/5   tailcalls/tailcall_5:OK
#442/6   tailcalls/tailcall_6:OK
#442/7   tailcalls/tailcall_bpf2bpf_1:OK
#442/8   tailcalls/tailcall_bpf2bpf_2:OK
#442/9   tailcalls/tailcall_bpf2bpf_3:OK
#442/10  tailcalls/tailcall_bpf2bpf_4:OK
#442/11  tailcalls/tailcall_bpf2bpf_5:OK
#442/12  tailcalls/tailcall_bpf2bpf_6:OK
#442/13  tailcalls/tailcall_bpf2bpf_fentry:OK
#442/14  tailcalls/tailcall_bpf2bpf_fexit:OK
#442/15  tailcalls/tailcall_bpf2bpf_fentry_fexit:OK
#442/16  tailcalls/tailcall_bpf2bpf_fentry_entry:OK
#442/17  tailcalls/tailcall_poke:OK
#442/18  tailcalls/tailcall_bpf2bpf_hierarchy_1:OK
#442/19  tailcalls/tailcall_bpf2bpf_hierarchy_fentry:OK
#442/20  tailcalls/tailcall_bpf2bpf_hierarchy_fexit:OK
#442/21  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_fexit:OK
#442/22  tailcalls/tailcall_bpf2bpf_hierarchy_fentry_entry:OK
#442/23  tailcalls/tailcall_bpf2bpf_hierarchy_2:OK
#442/24  tailcalls/tailcall_bpf2bpf_hierarchy_3:OK
#442/25  tailcalls/tailcall_freplace:OK
#442/26  tailcalls/tailcall_bpf2bpf_freplace:OK
#442/27  tailcalls/tailcall_failure:OK
#442/28  tailcalls/reject_tail_call_spin_lock:OK
#442/29  tailcalls/reject_tail_call_rcu_lock:OK
#442/30  tailcalls/reject_tail_call_preempt_lock:OK
#442/31  tailcalls/reject_tail_call_ref:OK
#442     tailcalls:OK
Summary: 1/31 PASSED, 0 SKIPPED, 0 FAILED

# ./test_progs -t exceptions
#105/1   exceptions/exception_throw_always_1:OK
#105/2   exceptions/exception_throw_always_2:OK
#105/3   exceptions/exception_throw_unwind_1:OK
#105/4   exceptions/exception_throw_unwind_2:OK
#105/5   exceptions/exception_throw_default:OK
#105/6   exceptions/exception_throw_default_value:OK
#105/7   exceptions/exception_tail_call:OK
#105/8   exceptions/exception_ext:OK
#105/9   exceptions/exception_ext_mod_cb_runtime:OK
#105/10  exceptions/exception_throw_subprog:OK
#105/11  exceptions/exception_assert_nz_gfunc:OK
#105/12  exceptions/exception_assert_zero_gfunc:OK
#105/13  exceptions/exception_assert_neg_gfunc:OK
#105/14  exceptions/exception_assert_pos_gfunc:OK
#105/15  exceptions/exception_assert_negeq_gfunc:OK
#105/16  exceptions/exception_assert_poseq_gfunc:OK
#105/17  exceptions/exception_assert_nz_gfunc_with:OK
#105/18  exceptions/exception_assert_zero_gfunc_with:OK
#105/19  exceptions/exception_assert_neg_gfunc_with:OK
#105/20  exceptions/exception_assert_pos_gfunc_with:OK
#105/21  exceptions/exception_assert_negeq_gfunc_with:OK
#105/22  exceptions/exception_assert_poseq_gfunc_with:OK
#105/23  exceptions/exception_bad_assert_nz_gfunc:OK
#105/24  exceptions/exception_bad_assert_zero_gfunc:OK
#105/25  exceptions/exception_bad_assert_neg_gfunc:OK
#105/26  exceptions/exception_bad_assert_pos_gfunc:OK
#105/27  exceptions/exception_bad_assert_negeq_gfunc:OK
#105/28  exceptions/exception_bad_assert_poseq_gfunc:OK
#105/29  exceptions/exception_bad_assert_nz_gfunc_with:OK
#105/30  exceptions/exception_bad_assert_zero_gfunc_with:OK
#105/31  exceptions/exception_bad_assert_neg_gfunc_with:OK
#105/32  exceptions/exception_bad_assert_pos_gfunc_with:OK
#105/33  exceptions/exception_bad_assert_negeq_gfunc_with:OK
#105/34  exceptions/exception_bad_assert_poseq_gfunc_with:OK
#105/35  exceptions/exception_assert_range:OK
#105/36  exceptions/exception_assert_range_with:OK
#105/37  exceptions/exception_bad_assert_range:OK
#105/38  exceptions/exception_bad_assert_range_with:OK
#105/39  exceptions/non-throwing fentry -> exception_cb:OK
#105/40  exceptions/throwing fentry -> exception_cb:OK
#105/41  exceptions/non-throwing fexit -> exception_cb:OK
#105/42  exceptions/throwing fexit -> exception_cb:OK
#105/43  exceptions/throwing extension (with custom cb) -> exception_cb:OK
#105/44  exceptions/throwing extension -> global func in exception_cb:OK
#105/45  exceptions/exception_ext_mod_cb_runtime:OK
#105/46  exceptions/throwing extension (with custom cb) -> global func in exception_cb:OK
#105/47  exceptions/exception_ext:OK
#105/48  exceptions/non-throwing fentry -> non-throwing subprog:OK
#105/49  exceptions/throwing fentry -> non-throwing subprog:OK
#105/50  exceptions/non-throwing fentry -> throwing subprog:OK
#105/51  exceptions/throwing fentry -> throwing subprog:OK
#105/52  exceptions/non-throwing fexit -> non-throwing subprog:OK
#105/53  exceptions/throwing fexit -> non-throwing subprog:OK
#105/54  exceptions/non-throwing fexit -> throwing subprog:OK
#105/55  exceptions/throwing fexit -> throwing subprog:OK
#105/56  exceptions/non-throwing fmod_ret -> non-throwing subprog:OK
#105/57  exceptions/non-throwing fmod_ret -> non-throwing global subprog:OK
#105/58  exceptions/non-throwing extension -> non-throwing subprog:OK
#105/59  exceptions/non-throwing extension -> throwing subprog:OK
#105/60  exceptions/non-throwing extension -> non-throwing subprog:OK
#105/61  exceptions/non-throwing extension -> throwing global subprog:OK
#105/62  exceptions/throwing extension -> throwing global subprog:OK
#105/63  exceptions/throwing extension -> non-throwing global subprog:OK
#105/64  exceptions/non-throwing extension -> main subprog:OK
#105/65  exceptions/throwing extension -> main subprog:OK
#105/66  exceptions/reject_exception_cb_type_1:OK
#105/67  exceptions/reject_exception_cb_type_2:OK
#105/68  exceptions/reject_exception_cb_type_3:OK
#105/69  exceptions/reject_exception_cb_type_4:OK
#105/70  exceptions/reject_async_callback_throw:OK
#105/71  exceptions/reject_with_lock:OK
#105/72  exceptions/reject_subprog_with_lock:OK
#105/73  exceptions/reject_with_rcu_read_lock:OK
#105/74  exceptions/reject_subprog_with_rcu_read_lock:OK
#105/75  exceptions/reject_with_rbtree_add_throw:OK
#105/76  exceptions/reject_with_reference:OK
#105/77  exceptions/reject_with_cb_reference:OK
#105/78  exceptions/reject_with_cb:OK
#105/79  exceptions/reject_with_subprog_reference:OK
#105/80  exceptions/reject_throwing_exception_cb:OK
#105/81  exceptions/reject_exception_cb_call_global_func:OK
#105/82  exceptions/reject_exception_cb_call_static_func:OK
#105/83  exceptions/reject_multiple_exception_cb:OK
#105/84  exceptions/reject_exception_throw_cb:OK
#105/85  exceptions/reject_exception_throw_cb_diff:OK
#105/86  exceptions/reject_set_exception_cb_bad_ret1:OK
#105/87  exceptions/reject_set_exception_cb_bad_ret2:OK
#105/88  exceptions/check_assert_eq_int_min:OK
#105/89  exceptions/check_assert_eq_int_max:OK
#105/90  exceptions/check_assert_eq_zero:OK
#105/91  exceptions/check_assert_eq_llong_min:OK
#105/92  exceptions/check_assert_eq_llong_max:OK
#105/93  exceptions/check_assert_lt_pos:OK
#105/94  exceptions/check_assert_lt_zero:OK
#105/95  exceptions/check_assert_lt_neg:OK
#105/96  exceptions/check_assert_le_pos:OK
#105/97  exceptions/check_assert_le_zero:OK
#105/98  exceptions/check_assert_le_neg:OK
#105/99  exceptions/check_assert_gt_pos:OK
#105/100 exceptions/check_assert_gt_zero:OK
#105/101 exceptions/check_assert_gt_neg:OK
#105/102 exceptions/check_assert_ge_pos:OK
#105/103 exceptions/check_assert_ge_zero:OK
#105/104 exceptions/check_assert_ge_neg:OK
#105/105 exceptions/check_assert_range_s64:OK
#105/106 exceptions/check_assert_range_u64:OK
#105/107 exceptions/check_assert_single_range_s64:OK
#105/108 exceptions/check_assert_single_range_u64:OK
#105/109 exceptions/check_assert_generic:OK
#105/110 exceptions/check_assert_with_return:OK
#105     exceptions:OK
Summary: 1/110 PASSED, 0 SKIPPED, 0 FAILED

[v1]: https://lore.kernel.org/all/20260105105212.136645-1-adubey@linux.ibm.com/

Changes v1->v2:
        Move tail_call_cnt to offset 0 in stack frame
        Remove trampoline NVR remapping-patch3/6[v1]

Abhishek Dubey (6):
  powerpc64/bpf: Move tail_call_cnt to bottom of stack frame
  powerpc64/bpf: Support tailcalls with subprogs
  powerpc64/bpf: Tailcall handling with trampolines
  powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT
  powerpc64/bpf: Support exceptions
  powerpc64/bpf: Additional NVR handling for bpf_throw

 arch/powerpc/net/bpf_jit.h        |  18 ++-
 arch/powerpc/net/bpf_jit_comp.c   |  98 ++++++++----
 arch/powerpc/net/bpf_jit_comp64.c | 240 ++++++++++++++++++++++++------
 3 files changed, 280 insertions(+), 76 deletions(-)

-- 
2.48.1

[PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

In the conventional stack frame, the position of tail_call_cnt
is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the
offset of tail_call_cnt in the trampoline frame is after the
stack alignment padding. BPF JIT logic could become complex
when dealing with frame-sensitive offset calculation of
tail_call_cnt. Having the same offset in both frames is the
desired objective.

The trampoline frame does not have a BPF_PPC_STACK_SAVE area.
Introducing it leads to under-utilization of extra memory meant
only for the offset alignment of tail_call_cnt.
Another challenge is the variable alignment padding sitting at
the bottom of the trampoline frame, which requires additional
handling to compute tail_call_cnt offset.

This patch addresses the above issues by moving tail_call_cnt
to the bottom of the stack frame at offset 0 for both types
of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE
in trampoline frame, and a common offset computation for
tail_call_cnt serves both frames.

The changes in this patch are required by the third patch in the
series, where the 'reference to tail_call_info' of the main frame
is copied into the trampoline frame from the previous frame.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit.h        |  4 ++++
 arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
 2 files changed, 24 insertions(+), 11 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 8334cd667bba..45d419c0ee73 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -72,6 +72,10 @@
 	} } while (0)
 
 #ifdef CONFIG_PPC64
+
+/* for tailcall counter */
+#define BPF_PPC_TAILCALL        8
+
 /* If dummy pass (!image), account for maximum possible instructions */
 #define PPC_LI64(d, i)		do {					      \
 	if (!image)							      \
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 1fe37128c876..39061cd742c1 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -20,13 +20,15 @@
 #include "bpf_jit.h"
 
 /*
- * Stack layout:
+ * Stack layout 1:
+ * Layout when setting up our own stack frame.
+ * Note: r1 at bottom, component offsets positive wrt r1.
  * Ensure the top half (upto local_tmp_var) stays consistent
  * with our redzone usage.
  *
  *		[	prev sp		] <-------------
- *		[   nv gpr save area	] 6*8		|
  *		[    tail_call_cnt	] 8		|
+ *		[   nv gpr save area	] 6*8		|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
  *		[     frame header	] 32/112	|
@@ -36,10 +38,12 @@
 /* for gpr non volatile registers BPG_REG_6 to 10 */
 #define BPF_PPC_STACK_SAVE	(6*8)
 /* for bpf JIT code internal usage */
-#define BPF_PPC_STACK_LOCALS	32
+#define BPF_PPC_STACK_LOCALS	24
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
 #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
-				 BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
+				 BPF_PPC_STACK_LOCALS + \
+				 BPF_PPC_STACK_SAVE   + \
+				 BPF_PPC_TAILCALL)
 
 /* BPF register usage */
 #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
@@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
 }
 
 /*
+ * Stack layout 2:
  * When not setting up our own stackframe, the redzone (288 bytes) usage is:
+ * Note: r1 from prev frame. Component offset negative wrt r1.
  *
  *		[	prev sp		] <-------------
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
- *		[   nv gpr save area	] 6*8
  *		[    tail_call_cnt	] 8
+ *		[   nv gpr save area	] 6*8
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
  */
 static int bpf_jit_stack_local(struct codegen_context *ctx)
 {
-	if (bpf_has_stack_frame(ctx))
+	if (bpf_has_stack_frame(ctx)) {
+		/* Stack layout 1 */
 		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
-	else
-		return -(BPF_PPC_STACK_SAVE + 32);
+	} else {
+		/* Stack layout 2 */
+		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+	}
 }
 
 static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
 {
-	return bpf_jit_stack_local(ctx) + 24;
+	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
 }
 
 static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
@@ -115,7 +124,7 @@ static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
 	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
 		return (bpf_has_stack_frame(ctx) ?
 			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
-				- (8 * (32 - reg));
+				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
 
 	pr_err("BPF JIT is asking about unknown registers");
 	BUG();
@@ -145,7 +154,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 	if (ctx->seen & SEEN_TAILCALL) {
 		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
 		/* this goes in the redzone */
-		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_STACK_SAVE + 8)));
+		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	} else {
 		EMIT(PPC_RAW_NOP());
 		EMIT(PPC_RAW_NOP());
-- 
2.48.1

[PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

Enabling tailcalls with subprog combinations by referencing
method. The actual tailcall count is always maintained in the
tail_call_info variable present in the frame of main function
(also called entry function). The tail_call_info variables in
the frames of all other subprog contains reference to the
tail_call_info present in frame of main function.

Dynamic resolution interprets the tail_call_info either as
value or reference depending on the context of active frame
while tailcall is invoked.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit.h        | 12 +++++-
 arch/powerpc/net/bpf_jit_comp.c   | 10 ++++-
 arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++--------
 3 files changed, 70 insertions(+), 20 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 45d419c0ee73..5d735bc5e6bd 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -51,6 +51,12 @@
 		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));					\
 	} while (0)
 
+/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */
+#define PPC_COND_BRANCH(cond, dest)                                           \
+	do {								      \
+		long offset = (long)(dest) - CTX_NIA(ctx);		      \
+		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));    \
+	} while (0)
 /*
  * Sign-extended 32-bit immediate load
  *
@@ -75,6 +81,8 @@
 
 /* for tailcall counter */
 #define BPF_PPC_TAILCALL        8
+/* for gpr non volatile registers BPG_REG_6 to 10 */
+#define BPF_PPC_STACK_SAVE      (6*8)
 
 /* If dummy pass (!image), account for maximum possible instructions */
 #define PPC_LI64(d, i)		do {					      \
@@ -170,6 +178,7 @@ struct codegen_context {
 	unsigned int alt_exit_addr;
 	u64 arena_vm_start;
 	u64 user_vm_start;
+	bool is_subprog;
 };
 
 #define bpf_to_ppc(r)	(ctx->b2p[r])
@@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
 void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx);
 void bpf_jit_realloc_regs(struct codegen_context *ctx);
 int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
-
 int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass,
 			  struct codegen_context *ctx, int insn_idx,
 			  int jmp_off, int dst_reg, u32 code);
-
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
 #endif
 
 #endif
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 5e976730b2f5..069a8822c30d 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
 	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
 	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
+	cgctx.is_subprog = bpf_is_subprog(fp);
 
 	/* Scouting faux-generate pass 0 */
 	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
 	bpf_prog_unlock_free(fp);
 }
 
+bool bpf_jit_supports_subprog_tailcalls(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
 bool bpf_jit_supports_kfunc_call(void)
 {
 	return true;
@@ -604,7 +610,7 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex
 					       int func_frame_offset, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
+		/* See bpf_jit_stack_tailcallinfo_offset() */
 		int tailcallcnt_offset = 7 * 8;
 
 		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
@@ -619,7 +625,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont
 						 int func_frame_offset, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallcnt() */
+		/* See bpf_jit_stack_tailcallinfo_offset() */
 		int tailcallcnt_offset = 7 * 8;
 
 		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 39061cd742c1..cebf81fbd59f 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -26,8 +26,12 @@
  * Ensure the top half (upto local_tmp_var) stays consistent
  * with our redzone usage.
  *
+ * tail_call_info - stores tailcall count value in main program's
+ *                  frame, stores reference to tail_call_info of
+ *                  main's frame in sub-prog's frame.
+ *
  *		[	prev sp		] <-------------
- *		[    tail_call_cnt	] 8		|
+ *		[    tail_call_info	] 8		|
  *		[   nv gpr save area	] 6*8		|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
@@ -35,8 +39,6 @@
  * sp (r1) --->	[    stack pointer	] --------------
  */
 
-/* for gpr non volatile registers BPG_REG_6 to 10 */
-#define BPF_PPC_STACK_SAVE	(6*8)
 /* for bpf JIT code internal usage */
 #define BPF_PPC_STACK_LOCALS	24
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
@@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
  *		[	prev sp		] <-------------
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
- *		[    tail_call_cnt	] 8
+ *		[    tail_call_info	] 8
  *		[   nv gpr save area	] 6*8
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
@@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
 	}
 }
 
-static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
+int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
 {
 	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
 }
@@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 #endif
 
 	/*
-	 * Initialize tail_call_cnt if we do tail calls.
-	 * Otherwise, put in NOPs so that it can be skipped when we are
-	 * invoked through a tail call.
+	 * Tail call count(tcc) is saved & updated only in main
+	 * program's frame and the address of tcc in main program's
+	 * frame (tcc_ptr) is saved in subprogs frame.
+	 *
+	 * Offset of tail_call_info on any frame will be interpreted
+	 * as either tcc_ptr or tcc value depending on whether it is
+	 * greater than MAX_TAIL_CALL_CNT or not.
 	 */
-	if (ctx->seen & SEEN_TAILCALL) {
+	if (!ctx->is_subprog) {
 		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
 		/* this goes in the redzone */
 		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	} else {
-		EMIT(PPC_RAW_NOP());
-		EMIT(PPC_RAW_NOP());
+		/*
+		 * if tail_call_info < MAX_TAIL_CALL_CNT
+		 *     main prog calling first subprog -> copy reference
+		 * else
+		 *     subsequent subprog calling another subprog -> directly copy content
+		 */
+		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
+		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
+		EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+		EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2),
+							-(BPF_PPC_TAILCALL)));
+		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
 	}
 
 	if (bpf_has_stack_frame(ctx)) {
@@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
 	EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));
 	PPC_BCC_SHORT(COND_GE, out);
 
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
+	PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);
+
+	/* dereference TMP_REG_1 */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
+
 	/*
-	 * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
+	 * if (tail_call_info == MAX_TAIL_CALL_CNT)
 	 *   goto out;
 	 */
-	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
 	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
-	PPC_BCC_SHORT(COND_GE, out);
+	PPC_COND_BRANCH(COND_EQ, out);
 
 	/*
-	 * tail_call_cnt++;
+	 * tail_call_info++; <- Actual value of tcc here
 	 */
 	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1));
-	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+
+	/*
+	 * Before writing updated tail_call_info, distinguish if current frame
+	 * is storing a reference to tail_call_info or actual tcc value in
+	 * tail_call_info.
+	 */
+	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT));
+	PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+
+	/* First get address of tail_call_info */
+	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
+	/* Writeback updated value to tail_call_info */
+	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0));
 
 	/* prog = array->ptrs[index]; */
 	EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));
-- 
2.48.1

[PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

The trampoline mechanism sets up its own stack frame and
an additional dummy frame. We need to have additional JIT
instructions handling tailcall dereferencing in the
trampoline's context.

We don't add the two stack frames pointed above, rather
add space for tail_call_info at bottom in trampoline frame
for ppc64. This makes the trampoline's frame consistent with
layout of all other frames wrt tail_call_info offset.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++-----------
 1 file changed, 56 insertions(+), 27 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 069a8822c30d..e3088cf089d1 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
 	return 0;
 }
 
-static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_context *ctx,
-					       int func_frame_offset, int r4_off)
-{
-	if (IS_ENABLED(CONFIG_PPC64)) {
-		/* See bpf_jit_stack_tailcallinfo_offset() */
-		int tailcallcnt_offset = 7 * 8;
-
-		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
-	} else {
-		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
-		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
-	}
-}
+/*
+ * Refer the label 'Generated stack layout' in this file for actual stack
+ * layout during trampoline invocation.
+ *
+ * Refer __arch_prepare_bpf_trampoline() for stack component details.
+ *
+ * The tailcall count/reference is present in caller's stack frame. Its required
+ * to copy the content of tail_call_info before calling the actual function
+ * to which the trampoline is attached.
+ *
+ */
 
-static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx,
-						 int func_frame_offset, int r4_off)
+static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
+					       int func_frame_offset,
+					       int bpf_dummy_frame_size, int r4_off)
 {
 	if (IS_ENABLED(CONFIG_PPC64)) {
 		/* See bpf_jit_stack_tailcallinfo_offset() */
-		int tailcallcnt_offset = 7 * 8;
+		int tailcallinfo_offset = BPF_PPC_TAILCALL;
+		/*
+		 * func_frame_offset =                                   ...(1)
+		 *     bpf_dummy_frame_size + trampoline_frame_size
+		 */
+		EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
+		EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
+
+		/*
+		 * Setting the tail_call_info in trampoline's frame
+		 * depending on if previous frame had value or reference.
+		 */
+		EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
+		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
+		EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
+		/*
+		 * From ...(1) above:
+		 * trampoline_frame_bottom =                            ...(2)
+		 *     func_frame_offset - bpf_dummy_frame_size
+		 *
+		 * Using ...(2) derived above:
+		 *  trampoline_tail_call_info_offset =                  ...(3)
+		 *      trampoline_frame_bottom - tailcallinfo_offset
+		 *
+		 * From ...(3):
+		 * Use trampoline_tail_call_info_offset to write reference of main's
+		 * tail_call_info in trampoline frame.
+		 */
+		EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
+					- tailcallinfo_offset));
 
-		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
-		EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
 	} else {
 		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
-		EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
+		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
 	}
 }
 
@@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	 * LR save area                 [ r0 save (64-bit)  ]   | header
 	 *                              [ r0 save (32-bit)  ]   |
 	 * dummy frame for unwind       [ back chain 1      ] --
+	 *                              [ tail_call_info    ] non optional - 64-bit powerpc
 	 *                              [ padding           ] align stack frame
 	 *       r4_off                 [ r4 (tailcallcnt)  ] optional - 32-bit powerpc
 	 *       alt_lr_off             [ real lr (ool stub)] optional - actual lr
@@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		}
 	}
 
-	/* Padding to align stack frame, if any */
-	bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
+	if (!(bpf_frame_size % (2 * SZL))) {
+		/* Stack is 16-byte aligned */
+		/* Room for padding followed by 64-bit tail_call_info */
+		bpf_frame_size += SZL + BPF_PPC_TAILCALL;
+	} else {
+		/* Room for 64-bit tail_call_info */
+		bpf_frame_size += BPF_PPC_TAILCALL;
+	}
 
 	/* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */
 	bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64;
@@ -902,7 +934,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 
 		/* Replicate tail_call_cnt before calling the original BPF prog */
 		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
-			bpf_trampoline_setup_tail_call_cnt(image, ctx, func_frame_offset, r4_off);
+			bpf_trampoline_setup_tail_call_info(image, ctx, func_frame_offset,
+							   bpf_dummy_frame_size, r4_off);
 
 		/* Restore args */
 		bpf_trampoline_restore_args_stack(image, ctx, func_frame_offset, nr_regs, regs_off);
@@ -917,10 +950,6 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		/* Store return value for bpf prog to access */
 		EMIT(PPC_RAW_STL(_R3, _R1, retval_off));
 
-		/* Restore updated tail_call_cnt */
-		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
-			bpf_trampoline_restore_tail_call_cnt(image, ctx, func_frame_offset, r4_off);
-
 		/* Reserve space to patch branch instruction to skip fexit progs */
 		if (ro_image) /* image is NULL for dummy pass */
 			im->ip_after_call = &((u32 *)ro_image)[ctx->idx];
-- 
2.48.1

[PATCH v2 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

This function is used by bpf_throw() to unwind the stack
until frame of exception-boundary during BPF exception
handling.

This function is necessary to support BPF exceptions on
PowerPC.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index cebf81fbd59f..ec58395f74f7 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
 	bpf_jit_build_fentry_stubs(image, ctx);
 }
 
+void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64), void *cookie)
+{
+	// callback processing always in current context
+	unsigned long fp = current_stack_frame();
+
+	for (;;) {
+		unsigned long *frame = (unsigned long *) fp;
+		unsigned long ip;
+
+		if (!validate_sp(fp, current))
+			return;
+
+		ip = frame[STACK_FRAME_LR_SAVE];
+		if (!ip)
+			break;
+
+		/*
+		 * consume_fn common code expects stack pointer(sp) in third
+		 * argument. There is no sp in ppc64, rather pass frame
+		 * pointer.
+		 */
+		if (ip && !consume_fn(cookie, ip, fp, fp))
+			break;
+
+		fp = frame[0];
+	}
+}
+
 int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func)
 {
 	unsigned long func_addr = func ? ppc_function_entry((void *)func) : 0;
-- 
2.48.1

[PATCH v2 5/6] powerpc64/bpf: Support exceptions

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

The modified prologue/epilogue generation code now
enables exception-callback to use the stack frame of
the program marked as exception boundary, where callee
saved registers are stored.

As per ppc64 ABIv2 documentation[1], r14-r31 are callee
saved registers. BPF programs on ppc64 already saves
r26-r31 registers. Saving the remaining set of callee
saved registers(r14-r25) is handled in the next patch.

[1] https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit.h        |  2 ++
 arch/powerpc/net/bpf_jit_comp.c   |  7 ++++
 arch/powerpc/net/bpf_jit_comp64.c | 53 +++++++++++++++++++++----------
 3 files changed, 45 insertions(+), 17 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 5d735bc5e6bd..fb548ae5d143 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -179,6 +179,8 @@ struct codegen_context {
 	u64 arena_vm_start;
 	u64 user_vm_start;
 	bool is_subprog;
+	bool exception_boundary;
+	bool exception_cb;
 };
 
 #define bpf_to_ppc(r)	(ctx->b2p[r])
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index e3088cf089d1..26991940d36e 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
 	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
 	cgctx.is_subprog = bpf_is_subprog(fp);
+	cgctx.exception_boundary = fp->aux->exception_boundary;
+	cgctx.exception_cb = fp->aux->exception_cb;
 
 	/* Scouting faux-generate pass 0 */
 	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
@@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)
 	bpf_prog_unlock_free(fp);
 }
 
+bool bpf_jit_supports_exceptions(void)
+{
+	return IS_ENABLED(CONFIG_PPC64);
+}
+
 bool bpf_jit_supports_subprog_tailcalls(void)
 {
 	return IS_ENABLED(CONFIG_PPC64);
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index ec58395f74f7..a6083dd9786c 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
 	 * - the bpf program uses its stack area
 	 * The latter condition is deduced from the usage of BPF_REG_FP
 	 */
-	return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP));
+	return ctx->seen & SEEN_FUNC ||
+	       bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) ||
+	       ctx->exception_cb;
 }
 
 /*
@@ -190,23 +192,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
 	}
 
-	/*
-	 * Back up non-volatile regs -- BPF registers 6-10
-	 * If we haven't created our own stack frame, we save these
-	 * in the protected zone below the previous stack frame
-	 */
-	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
-			EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
+	if (!ctx->exception_cb) {
+		/*
+		 * Back up non-volatile regs -- BPF registers 6-10
+		 * If we haven't created our own stack frame, we save these
+		 * in the protected zone below the previous stack frame
+		 */
+		for (i = BPF_REG_6; i <= BPF_REG_10; i++)
+			if (ctx->exception_boundary || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+				EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
+					bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
 
-	if (ctx->arena_vm_start)
-		EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
+		if (ctx->exception_boundary || ctx->arena_vm_start)
+			EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
 				 bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
-	/* Setup frame pointer to point to the bpf stack area */
-	if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
-		EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
+		/* Setup frame pointer to point to the bpf stack area */
+		if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
+			EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
 				STACK_FRAME_MIN_SIZE + ctx->stack_size));
+	} else {
+		/*
+		 * Exception callback receives Frame Pointer of main
+		 * program as third arg
+		 */
+		EMIT(PPC_RAW_MR(_R1, _R5));
+	}
 
 	if (ctx->arena_vm_start)
 		PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);
@@ -218,17 +229,25 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 
 	/* Restore NVRs */
 	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
-		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+		if (ctx->exception_cb || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
 			EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
 
-	if (ctx->arena_vm_start)
+	if (ctx->exception_cb || ctx->arena_vm_start)
 		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
 				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
+	if (ctx->exception_cb) {
+		/*
+		 * LR value from boundary-frame is received as second parameter
+		 * in exception callback.
+		 */
+		EMIT(PPC_RAW_MTLR(_R4));
+	}
+
 	/* Tear down our stack frame */
 	if (bpf_has_stack_frame(ctx)) {
 		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
-		if (ctx->seen & SEEN_FUNC) {
+		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
 			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
 			EMIT(PPC_RAW_MTLR(_R0));
 		}
-- 
2.48.1

[PATCH v2 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[adubey]

From: Abhishek Dubey <adubey@linux.ibm.com>

The bpf_throw() function never returns, if it has
clobbered any callee-saved register, those will
remain clobbered. The prologue must take care of
saving all callee-saved registers in the frame of
exception boundary program. Later these additional
non volatile registers R14-R25 along with other
NVRs are restored back in the epilogue of exception
callback.

To achieve above objective the frame size is
determined dynamically to accommodate additional
non volatile registers in exception boundary's frame.
For non-exception boundary program, the frame size
remains optimal. The additional instructions to
save & restore r14-r25 registers are emitted only during
exception boundary and exception callback respectively.

Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
---
 arch/powerpc/net/bpf_jit_comp64.c | 70 +++++++++++++++++++++++++++----
 1 file changed, 63 insertions(+), 7 deletions(-)

diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index a6083dd9786c..941e0818c9ec 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -32,21 +32,37 @@
  *
  *		[	prev sp		] <-------------
  *		[    tail_call_info	] 8		|
- *		[   nv gpr save area	] 6*8		|
+ *		[   nv gpr save area	] 6*8 + (12*8)	|
  *		[    local_tmp_var	] 24		|
  * fp (r31) -->	[   ebpf stack space	] upto 512	|
  *		[     frame header	] 32/112	|
  * sp (r1) --->	[    stack pointer	] --------------
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
  */
 
 /* for bpf JIT code internal usage */
 #define BPF_PPC_STACK_LOCALS	24
+/*
+ * for additional non volatile registers(r14-r25) to be saved
+ * at exception boundary
+ */
+#define BPF_PPC_EXC_STACK_SAVE (12*8)
+
 /* stack frame excluding BPF stack, ensure this is quadword aligned */
 #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
 				 BPF_PPC_STACK_LOCALS + \
 				 BPF_PPC_STACK_SAVE   + \
 				 BPF_PPC_TAILCALL)
 
+/*
+ * same as BPF_PPC_STACKFRAME with save area for additional
+ * non volatile registers saved at exception boundary.
+ * This is quad-word aligned.
+ */
+#define BPF_PPC_EXC_STACKFRAME (BPF_PPC_STACKFRAME + BPF_PPC_EXC_STACK_SAVE)
+
 /* BPF register usage */
 #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
 #define TMP_REG_2	(MAX_BPF_JIT_REG + 1)
@@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
  *		[	  ...       	] 		|
  * sp (r1) --->	[    stack pointer	] --------------
  *		[    tail_call_info	] 8
- *		[   nv gpr save area	] 6*8
+ *		[   nv gpr save area	] 6*8 + (12*8)
  *		[    local_tmp_var	] 24
  *		[   unused red zone	] 224
+ *
+ * Additional (12*8) in 'nv gpr save area' only in case of
+ * exception boundary.
  */
 static int bpf_jit_stack_local(struct codegen_context *ctx)
 {
@@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
 		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
 	} else {
 		/* Stack layout 2 */
-		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
+		return -(BPF_PPC_TAILCALL
+			+ BPF_PPC_STACK_SAVE
+			+ (ctx->exception_boundary || ctx->exception_cb ?
+							BPF_PPC_EXC_STACK_SAVE:0)
+			+ BPF_PPC_STACK_LOCALS);
 	}
 }
 
@@ -125,9 +148,19 @@ int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
 
 static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
 {
-	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
+	int min_valid_nvreg = BPF_PPC_NVR_MIN;
+	/* Default frame size for all cases except exception boundary */
+	int frame_nvr_size = BPF_PPC_STACKFRAME;
+
+	/* Consider all nv regs for handling exceptions */
+	if (ctx->exception_boundary || ctx->exception_cb) {
+		min_valid_nvreg = _R14;
+		frame_nvr_size = BPF_PPC_EXC_STACKFRAME;
+	}
+
+	if (reg >= min_valid_nvreg && reg < 32)
 		return (bpf_has_stack_frame(ctx) ?
-			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
+			(frame_nvr_size + ctx->stack_size) : 0)
 				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
 
 	pr_err("BPF JIT is asking about unknown registers");
@@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
 			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
 		}
 
-		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
+		int stack_expand = ctx->exception_boundary || ctx->exception_cb ?
+					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
+		EMIT(PPC_RAW_STDU(_R1, _R1, -(stack_expand + ctx->stack_size)));
+	}
+
+	/*
+	 * Program acting as exception boundary pushes R14..R25 in addition to
+	 * BPF callee-saved non volatile registers. Exception callback uses
+	 * the boundary program's stack frame, recover additionally saved
+	 * registers in epilogue of exception callback.
+	 */
+	if (ctx->exception_boundary) {
+		for (i = _R14; i <= _R25; i++)
+			EMIT(PPC_RAW_STD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
 	}
 
 	if (!ctx->exception_cb) {
@@ -237,6 +283,13 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
 
 	if (ctx->exception_cb) {
+		/*
+		 * Recover additionally saved non volatile registers from stack
+		 * frame of exception boundary program.
+		 */
+		for (i = _R14; i <= _R25; i++)
+			EMIT(PPC_RAW_LD(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
+
 		/*
 		 * LR value from boundary-frame is received as second parameter
 		 * in exception callback.
@@ -246,7 +299,10 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
 
 	/* Tear down our stack frame */
 	if (bpf_has_stack_frame(ctx)) {
-		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
+		int stack_shrink = ctx->exception_cb || ctx->exception_boundary ?
+					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
+		EMIT(PPC_RAW_ADDI(_R1, _R1, stack_shrink + ctx->stack_size));
+
 		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
 			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
 			EMIT(PPC_RAW_MTLR(_R0));
-- 
2.48.1

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 8334cd667bba..45d419c0ee73 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h

[ ... ]

> +/* for tailcall counter */
> +#define BPF_PPC_TAILCALL        8

This new constant defines tail_call_cnt offset as 8 bytes from prev sp.

> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 1fe37128c876..39061cd742c1 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c

[ ... ]

> static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
> {
>-	return bpf_jit_stack_local(ctx) + 24;
>+	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
> }

The tail_call_cnt offset changes from +24 to +72 (24 + 48) relative to
bpf_jit_stack_local().

There appears to be an inconsistency with the trampoline code in
bpf_jit_comp.c. The function bpf_trampoline_setup_tail_call_cnt() and
bpf_trampoline_restore_tail_call_cnt() use a hardcoded offset:

    int tailcallcnt_offset = 7 * 8;  /* = 56 */

The comment says "See bpf_jit_stack_tailcallcnt()" but after this patch,
bpf_jit_stack_tailcallcnt() returns an offset that corresponds to 8 bytes
from prev sp (BPF_PPC_TAILCALL), not 56 bytes. When BPF_TRAMP_F_TAIL_CALL_CTX
is set, will the trampoline read/write to the wrong stack location?

Note: This appears to be fixed later in the same series by commit 6889ec3d3e10
("powerpc64/bpf: Tailcall handling with trampolines") which updates the
trampoline code to use BPF_PPC_TAILCALL instead of the hardcoded 7 * 8.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]

> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index 069a8822c30d..e3088cf089d1 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c

[ ... ]

> +static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
> +					       int func_frame_offset,
> +					       int bpf_dummy_frame_size, int r4_off)
> +{
> +	if (IS_ENABLED(CONFIG_PPC64)) {
> +		/* See bpf_jit_stack_tailcallinfo_offset() */
> +		int tailcallinfo_offset = BPF_PPC_TAILCALL;
> +		/*
> +		 * func_frame_offset =                                   ...(1)
> +		 *     bpf_dummy_frame_size + trampoline_frame_size
> +		 */
> +		EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
> +		EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
                              ^^^^

Should this LD instruction be an ADDI instead? Looking at the stack frame
setup, after the two STLU instructions in __arch_prepare_bpf_trampoline():

  STLU _R1, _R1, -bpf_dummy_frame_size  ->  stores original _R1 at dummy frame
  STLU _R1, _R1, -bpf_frame_size        ->  stores dummy frame ptr at tramp frame

From the trampoline frame, the back chain stored in the dummy frame at offset
bpf_frame_size points to the caller's frame. But func_frame_offset equals
(bpf_dummy_frame_size + bpf_frame_size), which points into the caller's frame
itself. Loading from that address returns the caller's back chain (pointing
to the grandparent frame), not a pointer to the caller's frame.

This appears to cause the subsequent load at [_R4 - 8] to read the grandparent's
tail_call_info rather than the caller's. Would loading from [_R1 + bpf_frame_size]
give the correct pointer to the caller's frame?

For comparison, the subprogram prologue in bpf_jit_build_prologue() uses
[_R1 + 0] to get the back chain pointing to its caller, which seems like
the right pattern.

> +
> +		/*
> +		 * Setting the tail_call_info in trampoline's frame
> +		 * depending on if previous frame had value or reference.
> +		 */
> +		EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
> +		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
> +		EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

[Christophe Leroy (CS GROUP)]

Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> Enabling tailcalls with subprog combinations by referencing
> method. The actual tailcall count is always maintained in the
> tail_call_info variable present in the frame of main function
> (also called entry function). The tail_call_info variables in
> the frames of all other subprog contains reference to the
> tail_call_info present in frame of main function.
> 
> Dynamic resolution interprets the tail_call_info either as
> value or reference depending on the context of active frame
> while tailcall is invoked.
> 
> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
> ---
>   arch/powerpc/net/bpf_jit.h        | 12 +++++-
>   arch/powerpc/net/bpf_jit_comp.c   | 10 ++++-
>   arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++--------
>   3 files changed, 70 insertions(+), 20 deletions(-)
> 
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 45d419c0ee73..5d735bc5e6bd 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -51,6 +51,12 @@
>   		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));					\
>   	} while (0)
>   
> +/* Same as PPC_BCC_SHORT, except valid dest is known prior to call. */
> +#define PPC_COND_BRANCH(cond, dest)                                           \
> +	do {								      \
> +		long offset = (long)(dest) - CTX_NIA(ctx);		      \
> +		EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | (offset & 0xfffc));    \
> +	} while (0)

I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the 
verification out. Now we will have two macros doing the exact same thing 
with one handling failure case and one ignoring failure case. There is a 
big risk that one day or another someone will use the wrong macro.

Could you change bpf_jit_build_prologue() to return an int add use 
PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?


>   /*
>    * Sign-extended 32-bit immediate load
>    *
> @@ -75,6 +81,8 @@
>   
>   /* for tailcall counter */
>   #define BPF_PPC_TAILCALL        8
> +/* for gpr non volatile registers BPG_REG_6 to 10 */
> +#define BPF_PPC_STACK_SAVE      (6*8)

Add spaces before and after the *

>   
>   /* If dummy pass (!image), account for maximum possible instructions */
>   #define PPC_LI64(d, i)		do {					      \
> @@ -170,6 +178,7 @@ struct codegen_context {
>   	unsigned int alt_exit_addr;
>   	u64 arena_vm_start;
>   	u64 user_vm_start;
> +	bool is_subprog;
>   };
>   
>   #define bpf_to_ppc(r)	(ctx->b2p[r])
> @@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
>   void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context *ctx);
>   void bpf_jit_realloc_regs(struct codegen_context *ctx);
>   int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
> -
>   int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass,
>   			  struct codegen_context *ctx, int insn_idx,
>   			  int jmp_off, int dst_reg, u32 code);
> -

Not sure why this patch needs to remove those blank lines here and above.

> +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
>   #endif
>   
>   #endif
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index 5e976730b2f5..069a8822c30d 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>   	cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
>   	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
>   	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
> +	cgctx.is_subprog = bpf_is_subprog(fp);
>   
>   	/* Scouting faux-generate pass 0 */
>   	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
> @@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
>   	bpf_prog_unlock_free(fp);
>   }
>   
> +bool bpf_jit_supports_subprog_tailcalls(void)
> +{
> +	return IS_ENABLED(CONFIG_PPC64);
> +}
> +
>   bool bpf_jit_supports_kfunc_call(void)
>   {
>   	return true;
> @@ -604,7 +610,7 @@ static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex
>   					       int func_frame_offset, int r4_off)
>   {
>   	if (IS_ENABLED(CONFIG_PPC64)) {
> -		/* See bpf_jit_stack_tailcallcnt() */
> +		/* See bpf_jit_stack_tailcallinfo_offset() */
>   		int tailcallcnt_offset = 7 * 8;
>   
>   		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
> @@ -619,7 +625,7 @@ static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont
>   						 int func_frame_offset, int r4_off)
>   {
>   	if (IS_ENABLED(CONFIG_PPC64)) {
> -		/* See bpf_jit_stack_tailcallcnt() */
> +		/* See bpf_jit_stack_tailcallinfo_offset() */
>   		int tailcallcnt_offset = 7 * 8;
>   
>   		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 39061cd742c1..cebf81fbd59f 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -26,8 +26,12 @@
>    * Ensure the top half (upto local_tmp_var) stays consistent
>    * with our redzone usage.
>    *
> + * tail_call_info - stores tailcall count value in main program's
> + *                  frame, stores reference to tail_call_info of
> + *                  main's frame in sub-prog's frame.
> + *
>    *		[	prev sp		] <-------------
> - *		[    tail_call_cnt	] 8		|
> + *		[    tail_call_info	] 8		|
>    *		[   nv gpr save area	] 6*8		|
>    *		[    local_tmp_var	] 24		|
>    * fp (r31) -->	[   ebpf stack space	] upto 512	|
> @@ -35,8 +39,6 @@
>    * sp (r1) --->	[    stack pointer	] --------------
>    */
>   
> -/* for gpr non volatile registers BPG_REG_6 to 10 */
> -#define BPF_PPC_STACK_SAVE	(6*8)
>   /* for bpf JIT code internal usage */
>   #define BPF_PPC_STACK_LOCALS	24
>   /* stack frame excluding BPF stack, ensure this is quadword aligned */
> @@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
>    *		[	prev sp		] <-------------
>    *		[	  ...       	] 		|
>    * sp (r1) --->	[    stack pointer	] --------------
> - *		[    tail_call_cnt	] 8
> + *		[    tail_call_info	] 8
>    *		[   nv gpr save area	] 6*8
>    *		[    local_tmp_var	] 24
>    *		[   unused red zone	] 224
> @@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
>   	}
>   }
>   
> -static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
> +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
>   {
>   	return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE;
>   }
> @@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
>   #endif
>   
>   	/*
> -	 * Initialize tail_call_cnt if we do tail calls.
> -	 * Otherwise, put in NOPs so that it can be skipped when we are
> -	 * invoked through a tail call.
> +	 * Tail call count(tcc) is saved & updated only in main
> +	 * program's frame and the address of tcc in main program's
> +	 * frame (tcc_ptr) is saved in subprogs frame.
> +	 *
> +	 * Offset of tail_call_info on any frame will be interpreted
> +	 * as either tcc_ptr or tcc value depending on whether it is
> +	 * greater than MAX_TAIL_CALL_CNT or not.
>   	 */
> -	if (ctx->seen & SEEN_TAILCALL) {
> +	if (!ctx->is_subprog) {
>   		EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
>   		/* this goes in the redzone */
>   		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
>   	} else {
> -		EMIT(PPC_RAW_NOP());
> -		EMIT(PPC_RAW_NOP());
> +		/*
> +		 * if tail_call_info < MAX_TAIL_CALL_CNT
> +		 *     main prog calling first subprog -> copy reference
> +		 * else
> +		 *     subsequent subprog calling another subprog -> directly copy content
> +		 */
> +		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
> +		EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
> +		EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
> +		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
> +		EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2),
> +							-(BPF_PPC_TAILCALL)));
> +		EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_TAILCALL)));
>   	}
>   
>   	if (bpf_has_stack_frame(ctx)) {
> @@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
>   	EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));
>   	PPC_BCC_SHORT(COND_GE, out);
>   
> +	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
> +	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
> +	PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);
> +
> +	/* dereference TMP_REG_1 */
> +	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
> +
>   	/*
> -	 * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
> +	 * if (tail_call_info == MAX_TAIL_CALL_CNT)
>   	 *   goto out;
>   	 */
> -	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
>   	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
> -	PPC_BCC_SHORT(COND_GE, out);
> +	PPC_COND_BRANCH(COND_EQ, out);
>   
>   	/*
> -	 * tail_call_cnt++;
> +	 * tail_call_info++; <- Actual value of tcc here
>   	 */
>   	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1));
> -	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
> +
> +	/*
> +	 * Before writing updated tail_call_info, distinguish if current frame
> +	 * is storing a reference to tail_call_info or actual tcc value in
> +	 * tail_call_info.
> +	 */
> +	EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
> +	EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT));
> +	PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
> +
> +	/* First get address of tail_call_info */
> +	EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, bpf_jit_stack_tailcallinfo_offset(ctx)));
> +	/* Writeback updated value to tail_call_info */
> +	EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0));
>   
>   	/* prog = array->ptrs[index]; */
>   	EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));

Re: [PATCH v2 0/6] powerpc64/bpf: Support tailcalls with subprogs & BPF exceptions

[Christophe Leroy (CS GROUP)]

Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> This patch series enables support for two BPF JIT features
> on powerpc64. The first three patches target support for
> tail calls with subprogram combinations. The first patch
> supports realignment of tail_call_cnt offset in stack frame.
> Implementation details are provided in the commit messages.

Did you consider doing it also for powerpc32 ?

Christophe

Re: [PATCH v2 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2940 bytes --]

> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index a6083dd9786c..941e0818c9ec 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c

[ ... ]

> @@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
>  *		[	  ...       	] 		|
>  * sp (r1) --->	[    stack pointer	] --------------
>  *		[    tail_call_info	] 8
> - *		[   nv gpr save area	] 6*8
> + *		[   nv gpr save area	] 6*8 + (12*8)
>  *		[    local_tmp_var	] 24
>  *		[   unused red zone	] 224
> + *
> + * Additional (12*8) in 'nv gpr save area' only in case of
> + * exception boundary.
>  */
>  static int bpf_jit_stack_local(struct codegen_context *ctx)
>  {
> @@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
>  		return STACK_FRAME_MIN_SIZE + ctx->stack_size;

Does this need to account for BPF_PPC_EXC_STACK_SAVE when
exception_boundary or exception_cb is true?

In the else branch below, the exception case is handled, but the
bpf_has_stack_frame() case returns the same value regardless of
exception_boundary. However, bpf_jit_build_prologue() creates a larger
frame using BPF_PPC_EXC_STACKFRAME when exception_boundary is true.

This inconsistency appears to cause bpf_jit_stack_tailcallinfo_offset()
to return an incorrect offset when both bpf_has_stack_frame() returns
true AND exception_boundary is true. The offset would be off by 96 bytes
(BPF_PPC_EXC_STACK_SAVE).

For example, if a program uses bpf_throw (setting exception_boundary=true)
and also calls helper functions (setting SEEN_FUNC, making
bpf_has_stack_frame() return true), and uses tail calls, the tail call
handling in bpf_jit_emit_tail_call() would access the wrong stack location.

>  	} else {
>  		/* Stack layout 2 */
> -		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
> +		return -(BPF_PPC_TAILCALL
> +			+ BPF_PPC_STACK_SAVE
> +			+ (ctx->exception_boundary || ctx->exception_cb ?
> +							BPF_PPC_EXC_STACK_SAVE:0)
> +			+ BPF_PPC_STACK_LOCALS);
>  	}
>  }

[ ... ]

> @@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
>  			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
>  		}
>
> -		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
> +		int stack_expand = ctx->exception_boundary || ctx->exception_cb ?
> +					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
> +		EMIT(PPC_RAW_STDU(_R1, _R1, -(stack_expand + ctx->stack_size)));

The stack frame expansion here uses BPF_PPC_EXC_STACKFRAME for exception
cases, but bpf_jit_stack_local() in the bpf_has_stack_frame() branch does
not account for this larger frame size.

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/20993216190

Re: [PATCH v2 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT

[Christophe Leroy (CS GROUP)]

Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> This function is used by bpf_throw() to unwind the stack
> until frame of exception-boundary during BPF exception
> handling.
> 
> This function is necessary to support BPF exceptions on
> PowerPC.
> 
> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
> ---
>   arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++
>   1 file changed, 28 insertions(+)
> 
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index cebf81fbd59f..ec58395f74f7 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
>   	bpf_jit_build_fentry_stubs(image, ctx);
>   }
>   
> +void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64), void *cookie)
> +{
> +	// callback processing always in current context
> +	unsigned long fp = current_stack_frame();
> +
> +	for (;;) {
> +		unsigned long *frame = (unsigned long *) fp;
> +		unsigned long ip;
> +
> +		if (!validate_sp(fp, current))
> +			return;
> +
> +		ip = frame[STACK_FRAME_LR_SAVE];
> +		if (!ip)
> +			break;
> +
> +		/*
> +		 * consume_fn common code expects stack pointer(sp) in third
> +		 * argument. There is no sp in ppc64, rather pass frame
> +		 * pointer.
> +		 */
> +		if (ip && !consume_fn(cookie, ip, fp, fp))
> +			break;
> +
> +		fp = frame[0];
> +	}
> +}
> +

This fonction looks very close to arch_stack_walk(). Would it be 
possible to refactor and have a common part used by both functions, like 
ARM64 for instance ?

>   int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func)
>   {
>   	unsigned long func_addr = func ? ppc_function_entry((void *)func) : 0;

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

[kernel test robot]

Hi,

kernel test robot noticed the following build errors:

[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes linus/master v6.19-rc5 next-20260114]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/adubey-linux-ibm-com/powerpc64-bpf-Move-tail_call_cnt-to-bottom-of-stack-frame/20260114-195044
base:   https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
patch link:    https://lore.kernel.org/r/20260114114450.30405-4-adubey%40linux.ibm.com
patch subject: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines
config: powerpc-randconfig-002-20260114 (https://download.01.org/0day-ci/archive/20260115/202601150350.ZftaCBVV-lkp@intel.com/config)
compiler: powerpc-linux-gcc (GCC) 12.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260115/202601150350.ZftaCBVV-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601150350.ZftaCBVV-lkp@intel.com/

All errors (new ones prefixed by >>):

   arch/powerpc/net/bpf_jit_comp.c: In function 'bpf_trampoline_setup_tail_call_info':
>> arch/powerpc/net/bpf_jit_comp.c:644:43: error: 'BPF_PPC_TAILCALL' undeclared (first use in this function); did you mean 'BPF_TAIL_CALL'?
     644 |                 int tailcallinfo_offset = BPF_PPC_TAILCALL;
         |                                           ^~~~~~~~~~~~~~~~
         |                                           BPF_TAIL_CALL
   arch/powerpc/net/bpf_jit_comp.c:644:43: note: each undeclared identifier is reported only once for each function it appears in
   arch/powerpc/net/bpf_jit_comp.c: In function '__arch_prepare_bpf_trampoline':
   arch/powerpc/net/bpf_jit_comp.c:850:41: error: 'BPF_PPC_TAILCALL' undeclared (first use in this function); did you mean 'BPF_TAIL_CALL'?
     850 |                 bpf_frame_size += SZL + BPF_PPC_TAILCALL;
         |                                         ^~~~~~~~~~~~~~~~
         |                                         BPF_TAIL_CALL


vim +644 arch/powerpc/net/bpf_jit_comp.c

   625	
   626	/*
   627	 * Refer the label 'Generated stack layout' in this file for actual stack
   628	 * layout during trampoline invocation.
   629	 *
   630	 * Refer __arch_prepare_bpf_trampoline() for stack component details.
   631	 *
   632	 * The tailcall count/reference is present in caller's stack frame. Its required
   633	 * to copy the content of tail_call_info before calling the actual function
   634	 * to which the trampoline is attached.
   635	 *
   636	 */
   637	
   638	static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
   639						       int func_frame_offset,
   640						       int bpf_dummy_frame_size, int r4_off)
   641	{
   642		if (IS_ENABLED(CONFIG_PPC64)) {
   643			/* See bpf_jit_stack_tailcallinfo_offset() */
 > 644			int tailcallinfo_offset = BPF_PPC_TAILCALL;
   645			/*
   646			 * func_frame_offset =                                   ...(1)
   647			 *     bpf_dummy_frame_size + trampoline_frame_size
   648			 */
   649			EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
   650			EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
   651	
   652			/*
   653			 * Setting the tail_call_info in trampoline's frame
   654			 * depending on if previous frame had value or reference.
   655			 */
   656			EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
   657			PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
   658			EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
   659			/*
   660			 * From ...(1) above:
   661			 * trampoline_frame_bottom =                            ...(2)
   662			 *     func_frame_offset - bpf_dummy_frame_size
   663			 *
   664			 * Using ...(2) derived above:
   665			 *  trampoline_tail_call_info_offset =                  ...(3)
   666			 *      trampoline_frame_bottom - tailcallinfo_offset
   667			 *
   668			 * From ...(3):
   669			 * Use trampoline_tail_call_info_offset to write reference of main's
   670			 * tail_call_info in trampoline frame.
   671			 */
   672			EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
   673						- tailcallinfo_offset));
   674	
   675		} else {
   676			/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
   677			EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
   678		}
   679	}
   680	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

[Christophe Leroy (CS GROUP)]

Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> In the conventional stack frame, the position of tail_call_cnt
> is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the
> offset of tail_call_cnt in the trampoline frame is after the
> stack alignment padding. BPF JIT logic could become complex
> when dealing with frame-sensitive offset calculation of
> tail_call_cnt. Having the same offset in both frames is the
> desired objective.
> 
> The trampoline frame does not have a BPF_PPC_STACK_SAVE area.
> Introducing it leads to under-utilization of extra memory meant
> only for the offset alignment of tail_call_cnt.
> Another challenge is the variable alignment padding sitting at
> the bottom of the trampoline frame, which requires additional
> handling to compute tail_call_cnt offset.
> 
> This patch addresses the above issues by moving tail_call_cnt
> to the bottom of the stack frame at offset 0 for both types
> of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE
> in trampoline frame, and a common offset computation for
> tail_call_cnt serves both frames.
> 
> The changes in this patch are required by the third patch in the
> series, where the 'reference to tail_call_info' of the main frame
> is copied into the trampoline frame from the previous frame.
> 
> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
> ---
>   arch/powerpc/net/bpf_jit.h        |  4 ++++
>   arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
>   2 files changed, 24 insertions(+), 11 deletions(-)
> 
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 8334cd667bba..45d419c0ee73 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -72,6 +72,10 @@
>   	} } while (0)
>   
>   #ifdef CONFIG_PPC64
> +
> +/* for tailcall counter */
> +#define BPF_PPC_TAILCALL        8

This needs to be defined outside of CONFIG_PPC64 ifdef because from 
patch 3 it is used in bpf_jit_comp.c which is also built on powerpc32.

> +
>   /* If dummy pass (!image), account for maximum possible instructions */
>   #define PPC_LI64(d, i)		do {					      \
>   	if (!image)							      \

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

[Hari Bathini]

On 14/01/26 6:33 pm, adubey wrote:
> On 2026-01-14 17:57, Christophe Leroy (CS GROUP) wrote:
>> Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
>>> From: Abhishek Dubey <adubey@linux.ibm.com>
>>>
>>> Enabling tailcalls with subprog combinations by referencing
>>> method. The actual tailcall count is always maintained in the
>>> tail_call_info variable present in the frame of main function
>>> (also called entry function). The tail_call_info variables in
>>> the frames of all other subprog contains reference to the
>>> tail_call_info present in frame of main function.
>>>
>>> Dynamic resolution interprets the tail_call_info either as
>>> value or reference depending on the context of active frame
>>> while tailcall is invoked.
>>>
>>> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
>>> ---
>>>   arch/powerpc/net/bpf_jit.h        | 12 +++++-
>>>   arch/powerpc/net/bpf_jit_comp.c   | 10 ++++-
>>>   arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++++++++++--------
>>>   3 files changed, 70 insertions(+), 20 deletions(-)
>>>
>>> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
>>> index 45d419c0ee73..5d735bc5e6bd 100644
>>> --- a/arch/powerpc/net/bpf_jit.h
>>> +++ b/arch/powerpc/net/bpf_jit.h
>>> @@ -51,6 +51,12 @@
>>>           EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
>>> (offset & 0xfffc));                    \
>>>       } while (0)
>>>   +/* Same as PPC_BCC_SHORT, except valid dest is known prior to 
>>> call. */
>>> +#define PPC_COND_BRANCH(cond, dest)         \
>>> +    do {                                      \
>>> +        long offset = (long)(dest) - CTX_NIA(ctx);              \
>>> +        EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
>>> (offset & 0xfffc));    \
>>> +    } while (0)
>>
>> I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the
>> verification out. Now we will have two macros doing the exact same
>> thing with one handling failure case and one ignoring failure case.
>> There is a big risk that one day or another someone will use the wrong
>> macro.
>>
>> Could you change bpf_jit_build_prologue() to return an int add use
>> PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?
> I implemented exactly same change in bpf_jit_build_prologue(). But, 
> during internal review, @HariBathini suggested
> to have separate macro with a caution note.
> 
> @Hari please suggest here!

Not just about the change of return type but the check seems like an
overkill for cases where the offset is known and within branch range.
How about using BUILD_BUG_ON() to avoid unecessary checks and
return type change for places where the branch offset is known
and is a constant?

>>
>>>   /*
>>>    * Sign-extended 32-bit immediate load
>>>    *
>>> @@ -75,6 +81,8 @@
>>>     /* for tailcall counter */
>>>   #define BPF_PPC_TAILCALL        8
>>> +/* for gpr non volatile registers BPG_REG_6 to 10 */
>>> +#define BPF_PPC_STACK_SAVE      (6*8)
>>
>> Add spaces before and after the *
>>
>>>     /* If dummy pass (!image), account for maximum possible 
>>> instructions */
>>>   #define PPC_LI64(d, i)        do {                          \
>>> @@ -170,6 +178,7 @@ struct codegen_context {
>>>       unsigned int alt_exit_addr;
>>>       u64 arena_vm_start;
>>>       u64 user_vm_start;
>>> +    bool is_subprog;
>>>   };
>>>     #define bpf_to_ppc(r)    (ctx->b2p[r])
>>> @@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct 
>>> codegen_context *ctx);
>>>   void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context 
>>> *ctx);
>>>   void bpf_jit_realloc_regs(struct codegen_context *ctx);
>>>   int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, 
>>> int tmp_reg, long exit_addr);
>>> -
>>>   int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 
>>> *fimage, int pass,
>>>                 struct codegen_context *ctx, int insn_idx,
>>>                 int jmp_off, int dst_reg, u32 code);
>>> -
>>
>> Not sure why this patch needs to remove those blank lines here and above.
>>
>>> +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
>>>   #endif
>>>     #endif
>>> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/ 
>>> bpf_jit_comp.c
>>> index 5e976730b2f5..069a8822c30d 100644
>>> --- a/arch/powerpc/net/bpf_jit_comp.c
>>> +++ b/arch/powerpc/net/bpf_jit_comp.c
>>> @@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct 
>>> bpf_prog *fp)
>>>       cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
>>>       cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux- 
>>> >arena);
>>>       cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
>>> +    cgctx.is_subprog = bpf_is_subprog(fp);
>>>         /* Scouting faux-generate pass 0 */
>>>       if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
>>> @@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
>>>       bpf_prog_unlock_free(fp);
>>>   }
>>>   +bool bpf_jit_supports_subprog_tailcalls(void)
>>> +{
>>> +    return IS_ENABLED(CONFIG_PPC64);
>>> +}
>>> +
>>>   bool bpf_jit_supports_kfunc_call(void)
>>>   {
>>>       return true;
>>> @@ -604,7 +610,7 @@ static void 
>>> bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex
>>>                              int func_frame_offset, int r4_off)
>>>   {
>>>       if (IS_ENABLED(CONFIG_PPC64)) {
>>> -        /* See bpf_jit_stack_tailcallcnt() */
>>> +        /* See bpf_jit_stack_tailcallinfo_offset() */
>>>           int tailcallcnt_offset = 7 * 8;
>>>             EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - 
>>> tailcallcnt_offset));
>>> @@ -619,7 +625,7 @@ static void 
>>> bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont
>>>                            int func_frame_offset, int r4_off)
>>>   {
>>>       if (IS_ENABLED(CONFIG_PPC64)) {
>>> -        /* See bpf_jit_stack_tailcallcnt() */
>>> +        /* See bpf_jit_stack_tailcallinfo_offset() */
>>>           int tailcallcnt_offset = 7 * 8;
>>>             EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
>>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/ 
>>> bpf_jit_comp64.c
>>> index 39061cd742c1..cebf81fbd59f 100644
>>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
>>> @@ -26,8 +26,12 @@
>>>    * Ensure the top half (upto local_tmp_var) stays consistent
>>>    * with our redzone usage.
>>>    *
>>> + * tail_call_info - stores tailcall count value in main program's
>>> + *                  frame, stores reference to tail_call_info of
>>> + *                  main's frame in sub-prog's frame.
>>> + *
>>>    *        [    prev sp        ] <-------------
>>> - *        [    tail_call_cnt    ] 8        |
>>> + *        [    tail_call_info    ] 8        |
>>>    *        [   nv gpr save area    ] 6*8        |
>>>    *        [    local_tmp_var    ] 24        |
>>>    * fp (r31) -->    [   ebpf stack space    ] upto 512    |
>>> @@ -35,8 +39,6 @@
>>>    * sp (r1) --->    [    stack pointer    ] --------------
>>>    */
>>>   -/* for gpr non volatile registers BPG_REG_6 to 10 */
>>> -#define BPF_PPC_STACK_SAVE    (6*8)
>>>   /* for bpf JIT code internal usage */
>>>   #define BPF_PPC_STACK_LOCALS    24
>>>   /* stack frame excluding BPF stack, ensure this is quadword aligned */
>>> @@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct 
>>> codegen_context *ctx)
>>>    *        [    prev sp        ] <-------------
>>>    *        [      ...           ]         |
>>>    * sp (r1) --->    [    stack pointer    ] --------------
>>> - *        [    tail_call_cnt    ] 8
>>> + *        [    tail_call_info    ] 8
>>>    *        [   nv gpr save area    ] 6*8
>>>    *        [    local_tmp_var    ] 24
>>>    *        [   unused red zone    ] 224
>>> @@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct 
>>> codegen_context *ctx)
>>>       }
>>>   }
>>>   -static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
>>> +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
>>>   {
>>>       return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + 
>>> BPF_PPC_STACK_SAVE;
>>>   }
>>> @@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct 
>>> codegen_context *ctx)
>>>   #endif
>>>         /*
>>> -     * Initialize tail_call_cnt if we do tail calls.
>>> -     * Otherwise, put in NOPs so that it can be skipped when we are
>>> -     * invoked through a tail call.
>>> +     * Tail call count(tcc) is saved & updated only in main
>>> +     * program's frame and the address of tcc in main program's
>>> +     * frame (tcc_ptr) is saved in subprogs frame.
>>> +     *
>>> +     * Offset of tail_call_info on any frame will be interpreted
>>> +     * as either tcc_ptr or tcc value depending on whether it is
>>> +     * greater than MAX_TAIL_CALL_CNT or not.
>>>        */
>>> -    if (ctx->seen & SEEN_TAILCALL) {
>>> +    if (!ctx->is_subprog) {
>>>           EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
>>>           /* this goes in the redzone */
>>>           EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - 
>>> (BPF_PPC_TAILCALL)));
>>>       } else {
>>> -        EMIT(PPC_RAW_NOP());
>>> -        EMIT(PPC_RAW_NOP());
>>> +        /*
>>> +         * if tail_call_info < MAX_TAIL_CALL_CNT
>>> +         *     main prog calling first subprog -> copy reference
>>> +         * else
>>> +         *     subsequent subprog calling another subprog -> 
>>> directly copy content
>>> +         */
>>> +        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
>>> +        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), 
>>> bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
>>> +        EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
>>> +        PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
>>> +        EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2),
>>> +                            -(BPF_PPC_TAILCALL)));
>>> +        EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - 
>>> (BPF_PPC_TAILCALL)));
>>>       }
>>>         if (bpf_has_stack_frame(ctx)) {
>>> @@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, 
>>> struct codegen_context *ctx, u32 o
>>>       EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));
>>>       PPC_BCC_SHORT(COND_GE, out);
>>>   +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, 
>>> bpf_jit_stack_tailcallinfo_offset(ctx)));
>>> +    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
>>> +    PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);
>>> +
>>> +    /* dereference TMP_REG_1 */
>>> +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
>>> +
>>>       /*
>>> -     * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
>>> +     * if (tail_call_info == MAX_TAIL_CALL_CNT)
>>>        *   goto out;
>>>        */
>>> -    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, 
>>> bpf_jit_stack_tailcallcnt(ctx)));
>>>       EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
>>> -    PPC_BCC_SHORT(COND_GE, out);
>>> +    PPC_COND_BRANCH(COND_EQ, out);
>>>         /*
>>> -     * tail_call_cnt++;
>>> +     * tail_call_info++; <- Actual value of tcc here
>>>        */
>>>       EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 
>>> 1));
>>> -    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, 
>>> bpf_jit_stack_tailcallcnt(ctx)));
>>> +
>>> +    /*
>>> +     * Before writing updated tail_call_info, distinguish if current 
>>> frame
>>> +     * is storing a reference to tail_call_info or actual tcc value in
>>> +     * tail_call_info.
>>> +     */
>>> +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 
>>> bpf_jit_stack_tailcallinfo_offset(ctx)));
>>> +    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT));
>>> +    PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
>>> +
>>> +    /* First get address of tail_call_info */
>>> +    EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, 
>>> bpf_jit_stack_tailcallinfo_offset(ctx)));
>>> +    /* Writeback updated value to tail_call_info */
>>> +    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 0));
>>>         /* prog = array->ptrs[index]; */
>>>       EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));
> -Abhishek

Re: [PATCH v2 4/6] powerpc64/bpf: Add arch_bpf_stack_walk() for BPF JIT

[Hari Bathini]

On 14/01/26 6:50 pm, adubey wrote:
> On 2026-01-14 18:07, Christophe Leroy (CS GROUP) wrote:
>> Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
>>> From: Abhishek Dubey <adubey@linux.ibm.com>
>>>
>>> This function is used by bpf_throw() to unwind the stack
>>> until frame of exception-boundary during BPF exception
>>> handling.
>>>
>>> This function is necessary to support BPF exceptions on
>>> PowerPC.
>>>
>>> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
>>> ---
>>>   arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++++++
>>>   1 file changed, 28 insertions(+)
>>>
>>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/ 
>>> bpf_jit_comp64.c
>>> index cebf81fbd59f..ec58395f74f7 100644
>>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
>>> @@ -247,6 +247,34 @@ void bpf_jit_build_epilogue(u32 *image, struct 
>>> codegen_context *ctx)
>>>       bpf_jit_build_fentry_stubs(image, ctx);
>>>   }
>>>   +void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, 
>>> u64), void *cookie)
>>> +{
>>> +    // callback processing always in current context
>>> +    unsigned long fp = current_stack_frame();
>>> +
>>> +    for (;;) {
>>> +        unsigned long *frame = (unsigned long *) fp;
>>> +        unsigned long ip;
>>> +
>>> +        if (!validate_sp(fp, current))
>>> +            return;
>>> +
>>> +        ip = frame[STACK_FRAME_LR_SAVE];
>>> +        if (!ip)
>>> +            break;
>>> +
>>> +        /*
>>> +         * consume_fn common code expects stack pointer(sp) in third
>>> +         * argument. There is no sp in ppc64, rather pass frame
>>> +         * pointer.
>>> +         */
>>> +        if (ip && !consume_fn(cookie, ip, fp, fp))
>>> +            break;
>>> +
>>> +        fp = frame[0];
>>> +    }
>>> +}
>>> +
>>
>> This fonction looks very close to arch_stack_walk(). Would it be
>> possible to refactor and have a common part used by both functions,
>> like ARM64 for instance ?
> Yes, its inspired from arch_stack_walk(). consume_entry() have different 
> parameter count in both cases.
> If merged, it need additional handling to identify which call_back to 
> invoke.
> Also, we need to define arch-specific weak function 
> arch_bpf_stack_walk(), so renaming of arch_stack_walk is needed on merge.
> Stack walker logic with "bpf" name might be confusing when used at other 
> places. Thoughts?

Not sure what you mean by renaming of arch_stack_walk is needed on
merge but refactoring does not have to change API signature or any
common code for that matter..

- Hari

Re: [PATCH v2 5/6] powerpc64/bpf: Support exceptions

[Hari Bathini]

On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> The modified prologue/epilogue generation code now
> enables exception-callback to use the stack frame of
> the program marked as exception boundary, where callee
> saved registers are stored.
> 
> As per ppc64 ABIv2 documentation[1], r14-r31 are callee
> saved registers. BPF programs on ppc64 already saves
> r26-r31 registers. Saving the remaining set of callee
> saved registers(r14-r25) is handled in the next patch.
> 
> [1] https://ftp.rtems.org/pub/rtems/people/sebh/ABI64BitOpenPOWERv1.1_16July2015_pub.pdf
> 
> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
> ---
>   arch/powerpc/net/bpf_jit.h        |  2 ++
>   arch/powerpc/net/bpf_jit_comp.c   |  7 ++++
>   arch/powerpc/net/bpf_jit_comp64.c | 53 +++++++++++++++++++++----------
>   3 files changed, 45 insertions(+), 17 deletions(-)
> 
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 5d735bc5e6bd..fb548ae5d143 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -179,6 +179,8 @@ struct codegen_context {
>   	u64 arena_vm_start;
>   	u64 user_vm_start;
>   	bool is_subprog;
> +	bool exception_boundary;
> +	bool exception_cb;
>   };
>   
>   #define bpf_to_ppc(r)	(ctx->b2p[r])
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index e3088cf089d1..26991940d36e 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>   	cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
>   	cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
>   	cgctx.is_subprog = bpf_is_subprog(fp);
> +	cgctx.exception_boundary = fp->aux->exception_boundary;
> +	cgctx.exception_cb = fp->aux->exception_cb;
>   
>   	/* Scouting faux-generate pass 0 */
>   	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
> @@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)
>   	bpf_prog_unlock_free(fp);
>   }
>   
> +bool bpf_jit_supports_exceptions(void)
> +{
> +	return IS_ENABLED(CONFIG_PPC64);
> +}
> +
>   bool bpf_jit_supports_subprog_tailcalls(void)
>   {
>   	return IS_ENABLED(CONFIG_PPC64);
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index ec58395f74f7..a6083dd9786c 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
>   	 * - the bpf program uses its stack area
>   	 * The latter condition is deduced from the usage of BPF_REG_FP
>   	 */
> -	return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP));
> +	return ctx->seen & SEEN_FUNC ||
> +	       bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) ||
> +	       ctx->exception_cb;
>   }
>   
>   /*
> @@ -190,23 +192,32 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
>   		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
>   	}
>   
> -	/*
> -	 * Back up non-volatile regs -- BPF registers 6-10
> -	 * If we haven't created our own stack frame, we save these
> -	 * in the protected zone below the previous stack frame
> -	 */
> -	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
> -		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
> -			EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
> +	if (!ctx->exception_cb) {
> +		/*
> +		 * Back up non-volatile regs -- BPF registers 6-10
> +		 * If we haven't created our own stack frame, we save these
> +		 * in the protected zone below the previous stack frame
> +		 */
> +		for (i = BPF_REG_6; i <= BPF_REG_10; i++)
> +			if (ctx->exception_boundary || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
> +				EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
> +					bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
>   
> -	if (ctx->arena_vm_start)
> -		EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
> +		if (ctx->exception_boundary || ctx->arena_vm_start)
> +			EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
>   				 bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
>   
> -	/* Setup frame pointer to point to the bpf stack area */
> -	if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
> -		EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
> +		/* Setup frame pointer to point to the bpf stack area */
> +		if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
> +			EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
>   				STACK_FRAME_MIN_SIZE + ctx->stack_size));
> +	} else {
> +		/*
> +		 * Exception callback receives Frame Pointer of main
> +		 * program as third arg
> +		 */
> +		EMIT(PPC_RAW_MR(_R1, _R5));
> +	}
>   
>   	if (ctx->arena_vm_start)
>   		PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);
> @@ -218,17 +229,25 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
>   
>   	/* Restore NVRs */
>   	for (i = BPF_REG_6; i <= BPF_REG_10; i++)
> -		if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
> +		if (ctx->exception_cb || bpf_is_seen_register(ctx, bpf_to_ppc(i)))
>   			EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
>   
> -	if (ctx->arena_vm_start)
> +	if (ctx->exception_cb || ctx->arena_vm_start)
>   		EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
>   				bpf_jit_stack_offsetof(ctx, bpf_to_ppc(ARENA_VM_START))));
>   

> +	if (ctx->exception_cb) {
> +		/*
> +		 * LR value from boundary-frame is received as second parameter
> +		 * in exception callback.
> +		 */
> +		EMIT(PPC_RAW_MTLR(_R4));
> +	}
> +

No. Both second and third parameter of exception_cb() are stack pointer
(or frame pointer, if you prefer that) and not LR.
The above hunk is wrong. It still worked because of mtlr
instruction below that restored LR from PPC_LR_STKOFF offset
on the stack. Please drop the above hunk.

>   	/* Tear down our stack frame */
>   	if (bpf_has_stack_frame(ctx)) {
>   		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
> -		if (ctx->seen & SEEN_FUNC) {
> +		if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
>   			EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
>   			EMIT(PPC_RAW_MTLR(_R0));
>   		}

- Hari

Re: [PATCH v2 5/6] powerpc64/bpf: Support exceptions

[Hari Bathini]

On 16/01/26 12:17 pm, adubey wrote:
> On 2026-01-16 12:14, adubey wrote:
>> On 2026-01-16 11:57, Hari Bathini wrote:
>>> On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
>>>> From: Abhishek Dubey <adubey@linux.ibm.com>
>>>>
>>>> The modified prologue/epilogue generation code now
>>>> enables exception-callback to use the stack frame of
>>>> the program marked as exception boundary, where callee
>>>> saved registers are stored.
>>>>
>>>> As per ppc64 ABIv2 documentation[1], r14-r31 are callee
>>>> saved registers. BPF programs on ppc64 already saves
>>>> r26-r31 registers. Saving the remaining set of callee
>>>> saved registers(r14-r25) is handled in the next patch.
>>>>
>>>> [1] https://ftp.rtems.org/pub/rtems/people/sebh/ 
>>>> ABI64BitOpenPOWERv1.1_16July2015_pub.pdf
>>>>
>>>> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
>>>> ---
>>>>   arch/powerpc/net/bpf_jit.h        |  2 ++
>>>>   arch/powerpc/net/bpf_jit_comp.c   |  7 ++++
>>>>   arch/powerpc/net/bpf_jit_comp64.c | 53 ++++++++++++++++++++ 
>>>> +----------
>>>>   3 files changed, 45 insertions(+), 17 deletions(-)
>>>>
>>>> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
>>>> index 5d735bc5e6bd..fb548ae5d143 100644
>>>> --- a/arch/powerpc/net/bpf_jit.h
>>>> +++ b/arch/powerpc/net/bpf_jit.h
>>>> @@ -179,6 +179,8 @@ struct codegen_context {
>>>>       u64 arena_vm_start;
>>>>       u64 user_vm_start;
>>>>       bool is_subprog;
>>>> +    bool exception_boundary;
>>>> +    bool exception_cb;
>>>>   };
>>>>     #define bpf_to_ppc(r)    (ctx->b2p[r])
>>>> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/ 
>>>> bpf_jit_comp.c
>>>> index e3088cf089d1..26991940d36e 100644
>>>> --- a/arch/powerpc/net/bpf_jit_comp.c
>>>> +++ b/arch/powerpc/net/bpf_jit_comp.c
>>>> @@ -207,6 +207,8 @@ struct bpf_prog *bpf_int_jit_compile(struct 
>>>> bpf_prog *fp)
>>>>       cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux- 
>>>> >arena);
>>>>       cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux- 
>>>> >arena);
>>>>       cgctx.is_subprog = bpf_is_subprog(fp);
>>>> +    cgctx.exception_boundary = fp->aux->exception_boundary;
>>>> +    cgctx.exception_cb = fp->aux->exception_cb;
>>>>         /* Scouting faux-generate pass 0 */
>>>>       if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, 
>>>> false)) {
>>>> @@ -436,6 +438,11 @@ void bpf_jit_free(struct bpf_prog *fp)
>>>>       bpf_prog_unlock_free(fp);
>>>>   }
>>>>   +bool bpf_jit_supports_exceptions(void)
>>>> +{
>>>> +    return IS_ENABLED(CONFIG_PPC64);
>>>> +}
>>>> +
>>>>   bool bpf_jit_supports_subprog_tailcalls(void)
>>>>   {
>>>>       return IS_ENABLED(CONFIG_PPC64);
>>>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/ 
>>>> bpf_jit_comp64.c
>>>> index ec58395f74f7..a6083dd9786c 100644
>>>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>>>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
>>>> @@ -89,7 +89,9 @@ static inline bool bpf_has_stack_frame(struct 
>>>> codegen_context *ctx)
>>>>        * - the bpf program uses its stack area
>>>>        * The latter condition is deduced from the usage of BPF_REG_FP
>>>>        */
>>>> -    return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, 
>>>> bpf_to_ppc(BPF_REG_FP));
>>>> +    return ctx->seen & SEEN_FUNC ||
>>>> +           bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)) ||
>>>> +           ctx->exception_cb;
>>>>   }
>>>>     /*
>>>> @@ -190,23 +192,32 @@ void bpf_jit_build_prologue(u32 *image, struct 
>>>> codegen_context *ctx)
>>>>           EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx- 
>>>> >stack_size)));
>>>>       }
>>>>   -    /*
>>>> -     * Back up non-volatile regs -- BPF registers 6-10
>>>> -     * If we haven't created our own stack frame, we save these
>>>> -     * in the protected zone below the previous stack frame
>>>> -     */
>>>> -    for (i = BPF_REG_6; i <= BPF_REG_10; i++)
>>>> -        if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
>>>> -            EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, 
>>>> bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
>>>> +    if (!ctx->exception_cb) {
>>>> +        /*
>>>> +         * Back up non-volatile regs -- BPF registers 6-10
>>>> +         * If we haven't created our own stack frame, we save these
>>>> +         * in the protected zone below the previous stack frame
>>>> +         */
>>>> +        for (i = BPF_REG_6; i <= BPF_REG_10; i++)
>>>> +            if (ctx->exception_boundary || 
>>>> bpf_is_seen_register(ctx, bpf_to_ppc(i)))
>>>> +                EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1,
>>>> +                    bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
>>>>   -    if (ctx->arena_vm_start)
>>>> -        EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
>>>> +        if (ctx->exception_boundary || ctx->arena_vm_start)
>>>> +            EMIT(PPC_RAW_STD(bpf_to_ppc(ARENA_VM_START), _R1,
>>>>                    bpf_jit_stack_offsetof(ctx, 
>>>> bpf_to_ppc(ARENA_VM_START))));
>>>>   -    /* Setup frame pointer to point to the bpf stack area */
>>>> -    if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
>>>> -        EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
>>>> +        /* Setup frame pointer to point to the bpf stack area */
>>>> +        if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
>>>> +            EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
>>>>                   STACK_FRAME_MIN_SIZE + ctx->stack_size));
>>>> +    } else {
>>>> +        /*
>>>> +         * Exception callback receives Frame Pointer of main
>>>> +         * program as third arg
>>>> +         */
>>>> +        EMIT(PPC_RAW_MR(_R1, _R5));
>>>> +    }
>>>>         if (ctx->arena_vm_start)
>>>>           PPC_LI64(bpf_to_ppc(ARENA_VM_START), ctx->arena_vm_start);
>>>> @@ -218,17 +229,25 @@ static void bpf_jit_emit_common_epilogue(u32 
>>>> *image, struct codegen_context *ctx
>>>>         /* Restore NVRs */
>>>>       for (i = BPF_REG_6; i <= BPF_REG_10; i++)
>>>> -        if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
>>>> +        if (ctx->exception_cb || bpf_is_seen_register(ctx, 
>>>> bpf_to_ppc(i)))
>>>>               EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, 
>>>> bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
>>>>   -    if (ctx->arena_vm_start)
>>>> +    if (ctx->exception_cb || ctx->arena_vm_start)
>>>>           EMIT(PPC_RAW_LD(bpf_to_ppc(ARENA_VM_START), _R1,
>>>>                   bpf_jit_stack_offsetof(ctx, 
>>>> bpf_to_ppc(ARENA_VM_START))));
>>>>
>>>
>>>> +    if (ctx->exception_cb) {
>>>> +        /*
>>>> +         * LR value from boundary-frame is received as second 
>>>> parameter
>>>> +         * in exception callback.
>>>> +         */
>>>> +        EMIT(PPC_RAW_MTLR(_R4));
>>>> +    }
>>>> +
>>>
>>> No. Both second and third parameter of exception_cb() are stack pointer
>>> (or frame pointer, if you prefer that) and not LR.
>>> The above hunk is wrong. It still worked because of mtlr
>>> instruction below that restored LR from PPC_LR_STKOFF offset
>>> on the stack. Please drop the above hunk.
>> Okay, but from patch 3/6:
> My mistake, its patch 4/6.

Please note the consume_fn() is not exception_cb(). It is
bpf_stack_walker() that helps with the stack walk to be precise.
You can find the exception_cb() call in bpf_throw() kfunc as:

   ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);

>>
>> ip = frame[STACK_FRAME_LR_SAVE];
>> ...
>> if (ip && !consume_fn(cookie, ip, fp, fp))
>> we send cookie as first, LR as second parameter and fp as third & 
>> fourth both.
>> Default callback expects 4 parameter, so we copy fp in both.
>>
>>>
>>>>       /* Tear down our stack frame */
>>>>       if (bpf_has_stack_frame(ctx)) {
>>>>           EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx- 
>>>> >stack_size));
>>>> -        if (ctx->seen & SEEN_FUNC) {
>>>> +        if (ctx->seen & SEEN_FUNC || ctx->exception_cb) {
>>>>               EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
>>>>               EMIT(PPC_RAW_MTLR(_R0));
>>>>           }
>>>
- Hari

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

[Christophe Leroy (CS GROUP)]

Le 16/01/2026 à 05:50, Hari Bathini a écrit :

Not received this mail that Hari is reponding to.

> 
> 
> On 14/01/26 6:33 pm, adubey wrote:
>> On 2026-01-14 17:57, Christophe Leroy (CS GROUP) wrote:
>>> Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
>>>> From: Abhishek Dubey <adubey@linux.ibm.com>
>>>>
>>>> Enabling tailcalls with subprog combinations by referencing
>>>> method. The actual tailcall count is always maintained in the
>>>> tail_call_info variable present in the frame of main function
>>>> (also called entry function). The tail_call_info variables in
>>>> the frames of all other subprog contains reference to the
>>>> tail_call_info present in frame of main function.
>>>>
>>>> Dynamic resolution interprets the tail_call_info either as
>>>> value or reference depending on the context of active frame
>>>> while tailcall is invoked.
>>>>
>>>> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
>>>> ---
>>>>   arch/powerpc/net/bpf_jit.h        | 12 +++++-
>>>>   arch/powerpc/net/bpf_jit_comp.c   | 10 ++++-
>>>>   arch/powerpc/net/bpf_jit_comp64.c | 68 ++++++++++++++++++++++ 
>>>> +--------
>>>>   3 files changed, 70 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
>>>> index 45d419c0ee73..5d735bc5e6bd 100644
>>>> --- a/arch/powerpc/net/bpf_jit.h
>>>> +++ b/arch/powerpc/net/bpf_jit.h
>>>> @@ -51,6 +51,12 @@
>>>>           EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
>>>> (offset & 0xfffc));                    \
>>>>       } while (0)
>>>>   +/* Same as PPC_BCC_SHORT, except valid dest is known prior to 
>>>> call. */
>>>> +#define PPC_COND_BRANCH(cond, dest)         \
>>>> +    do {                                      \
>>>> +        long offset = (long)(dest) - CTX_NIA(ctx);              \
>>>> +        EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
>>>> (offset & 0xfffc));    \
>>>> +    } while (0)
>>>
>>> I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the
>>> verification out. Now we will have two macros doing the exact same
>>> thing with one handling failure case and one ignoring failure case.
>>> There is a big risk that one day or another someone will use the wrong
>>> macro.
>>>
>>> Could you change bpf_jit_build_prologue() to return an int add use
>>> PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?
>> I implemented exactly same change in bpf_jit_build_prologue(). But, 
>> during internal review, @HariBathini suggested
>> to have separate macro with a caution note.
>>
>> @Hari please suggest here!
> 
> Not just about the change of return type but the check seems like an
> overkill for cases where the offset is known and within branch range.
> How about using BUILD_BUG_ON() to avoid unecessary checks and
> return type change for places where the branch offset is known
> and is a constant?

When offset is a constant known at build time, checks are eliminated by 
gcc at build, see exemple below from disasembly of bpf_jit_comp32.o, 
there are no checks.

                                 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
     36d8:       3c 80 41 81     lis     r4,16769
                                 EMIT(PPC_RAW_CMPLW(src_reg, _R0));
     36dc:       81 3f 00 04     lwz     r9,4(r31)
                                 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
     36e0:       60 84 00 10     ori     r4,r4,16
                                 EMIT(PPC_RAW_CMPLW(src_reg, _R0));
     36e4:       39 29 00 01     addi    r9,r9,1
                                 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
     36e8:       55 23 10 3a     slwi    r3,r9,2
                                 EMIT(PPC_RAW_CMPLW(src_reg, _R0));
     36ec:       91 3f 00 04     stw     r9,4(r31)
                                 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
     36f0:       7c 97 19 2e     stwx    r4,r23,r3
                                 EMIT(PPC_RAW_LI(dst_reg, 0));
     36f4:       55 49 a9 94     rlwinm  r9,r10,21,6,10
                                 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
     36f8:       80 9f 00 04     lwz     r4,4(r31)
                                 EMIT(PPC_RAW_LI(dst_reg, 0));
     36fc:       65 29 38 00     oris    r9,r9,14336
                                 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
     3700:       38 84 00 01     addi    r4,r4,1
                                 EMIT(PPC_RAW_LI(dst_reg, 0));
     3704:       54 83 10 3a     slwi    r3,r4,2
                                 PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
     3708:       90 9f 00 04     stw     r4,4(r31)
                                 EMIT(PPC_RAW_LI(dst_reg, 0));
     370c:       7d 37 19 2e     stwx    r9,r23,r3


> 
>>>
>>>>   /*
>>>>    * Sign-extended 32-bit immediate load
>>>>    *
>>>> @@ -75,6 +81,8 @@
>>>>     /* for tailcall counter */
>>>>   #define BPF_PPC_TAILCALL        8
>>>> +/* for gpr non volatile registers BPG_REG_6 to 10 */
>>>> +#define BPF_PPC_STACK_SAVE      (6*8)
>>>
>>> Add spaces before and after the *
>>>
>>>>     /* If dummy pass (!image), account for maximum possible 
>>>> instructions */
>>>>   #define PPC_LI64(d, i)        do {                          \
>>>> @@ -170,6 +178,7 @@ struct codegen_context {
>>>>       unsigned int alt_exit_addr;
>>>>       u64 arena_vm_start;
>>>>       u64 user_vm_start;
>>>> +    bool is_subprog;
>>>>   };
>>>>     #define bpf_to_ppc(r)    (ctx->b2p[r])
>>>> @@ -204,11 +213,10 @@ void bpf_jit_build_epilogue(u32 *image, struct 
>>>> codegen_context *ctx);
>>>>   void bpf_jit_build_fentry_stubs(u32 *image, struct codegen_context 
>>>> *ctx);
>>>>   void bpf_jit_realloc_regs(struct codegen_context *ctx);
>>>>   int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context 
>>>> *ctx, int tmp_reg, long exit_addr);
>>>> -
>>>>   int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 
>>>> *fimage, int pass,
>>>>                 struct codegen_context *ctx, int insn_idx,
>>>>                 int jmp_off, int dst_reg, u32 code);
>>>> -
>>>
>>> Not sure why this patch needs to remove those blank lines here and 
>>> above.
>>>
>>>> +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx);
>>>>   #endif
>>>>     #endif
>>>> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/ 
>>>> bpf_jit_comp.c
>>>> index 5e976730b2f5..069a8822c30d 100644
>>>> --- a/arch/powerpc/net/bpf_jit_comp.c
>>>> +++ b/arch/powerpc/net/bpf_jit_comp.c
>>>> @@ -206,6 +206,7 @@ struct bpf_prog *bpf_int_jit_compile(struct 
>>>> bpf_prog *fp)
>>>>       cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
>>>>       cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux- 
>>>> >arena);
>>>>       cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux- 
>>>> >arena);
>>>> +    cgctx.is_subprog = bpf_is_subprog(fp);
>>>>         /* Scouting faux-generate pass 0 */
>>>>       if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, 
>>>> false)) {
>>>> @@ -435,6 +436,11 @@ void bpf_jit_free(struct bpf_prog *fp)
>>>>       bpf_prog_unlock_free(fp);
>>>>   }
>>>>   +bool bpf_jit_supports_subprog_tailcalls(void)
>>>> +{
>>>> +    return IS_ENABLED(CONFIG_PPC64);
>>>> +}
>>>> +
>>>>   bool bpf_jit_supports_kfunc_call(void)
>>>>   {
>>>>       return true;
>>>> @@ -604,7 +610,7 @@ static void 
>>>> bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_contex
>>>>                              int func_frame_offset, int r4_off)
>>>>   {
>>>>       if (IS_ENABLED(CONFIG_PPC64)) {
>>>> -        /* See bpf_jit_stack_tailcallcnt() */
>>>> +        /* See bpf_jit_stack_tailcallinfo_offset() */
>>>>           int tailcallcnt_offset = 7 * 8;
>>>>             EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - 
>>>> tailcallcnt_offset));
>>>> @@ -619,7 +625,7 @@ static void 
>>>> bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_cont
>>>>                            int func_frame_offset, int r4_off)
>>>>   {
>>>>       if (IS_ENABLED(CONFIG_PPC64)) {
>>>> -        /* See bpf_jit_stack_tailcallcnt() */
>>>> +        /* See bpf_jit_stack_tailcallinfo_offset() */
>>>>           int tailcallcnt_offset = 7 * 8;
>>>>             EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
>>>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/ 
>>>> bpf_jit_comp64.c
>>>> index 39061cd742c1..cebf81fbd59f 100644
>>>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>>>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
>>>> @@ -26,8 +26,12 @@
>>>>    * Ensure the top half (upto local_tmp_var) stays consistent
>>>>    * with our redzone usage.
>>>>    *
>>>> + * tail_call_info - stores tailcall count value in main program's
>>>> + *                  frame, stores reference to tail_call_info of
>>>> + *                  main's frame in sub-prog's frame.
>>>> + *
>>>>    *        [    prev sp        ] <-------------
>>>> - *        [    tail_call_cnt    ] 8        |
>>>> + *        [    tail_call_info    ] 8        |
>>>>    *        [   nv gpr save area    ] 6*8        |
>>>>    *        [    local_tmp_var    ] 24        |
>>>>    * fp (r31) -->    [   ebpf stack space    ] upto 512    |
>>>> @@ -35,8 +39,6 @@
>>>>    * sp (r1) --->    [    stack pointer    ] --------------
>>>>    */
>>>>   -/* for gpr non volatile registers BPG_REG_6 to 10 */
>>>> -#define BPF_PPC_STACK_SAVE    (6*8)
>>>>   /* for bpf JIT code internal usage */
>>>>   #define BPF_PPC_STACK_LOCALS    24
>>>>   /* stack frame excluding BPF stack, ensure this is quadword 
>>>> aligned */
>>>> @@ -98,7 +100,7 @@ static inline bool bpf_has_stack_frame(struct 
>>>> codegen_context *ctx)
>>>>    *        [    prev sp        ] <-------------
>>>>    *        [      ...           ]         |
>>>>    * sp (r1) --->    [    stack pointer    ] --------------
>>>> - *        [    tail_call_cnt    ] 8
>>>> + *        [    tail_call_info    ] 8
>>>>    *        [   nv gpr save area    ] 6*8
>>>>    *        [    local_tmp_var    ] 24
>>>>    *        [   unused red zone    ] 224
>>>> @@ -114,7 +116,7 @@ static int bpf_jit_stack_local(struct 
>>>> codegen_context *ctx)
>>>>       }
>>>>   }
>>>>   -static int bpf_jit_stack_tailcallcnt(struct codegen_context *ctx)
>>>> +int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
>>>>   {
>>>>       return bpf_jit_stack_local(ctx) + BPF_PPC_STACK_LOCALS + 
>>>> BPF_PPC_STACK_SAVE;
>>>>   }
>>>> @@ -147,17 +149,32 @@ void bpf_jit_build_prologue(u32 *image, struct 
>>>> codegen_context *ctx)
>>>>   #endif
>>>>         /*
>>>> -     * Initialize tail_call_cnt if we do tail calls.
>>>> -     * Otherwise, put in NOPs so that it can be skipped when we are
>>>> -     * invoked through a tail call.
>>>> +     * Tail call count(tcc) is saved & updated only in main
>>>> +     * program's frame and the address of tcc in main program's
>>>> +     * frame (tcc_ptr) is saved in subprogs frame.
>>>> +     *
>>>> +     * Offset of tail_call_info on any frame will be interpreted
>>>> +     * as either tcc_ptr or tcc value depending on whether it is
>>>> +     * greater than MAX_TAIL_CALL_CNT or not.
>>>>        */
>>>> -    if (ctx->seen & SEEN_TAILCALL) {
>>>> +    if (!ctx->is_subprog) {
>>>>           EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
>>>>           /* this goes in the redzone */
>>>>           EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - 
>>>> (BPF_PPC_TAILCALL)));
>>>>       } else {
>>>> -        EMIT(PPC_RAW_NOP());
>>>> -        EMIT(PPC_RAW_NOP());
>>>> +        /*
>>>> +         * if tail_call_info < MAX_TAIL_CALL_CNT
>>>> +         *     main prog calling first subprog -> copy reference
>>>> +         * else
>>>> +         *     subsequent subprog calling another subprog -> 
>>>> directly copy content
>>>> +         */
>>>> +        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 0));
>>>> +        EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), 
>>>> bpf_to_ppc(TMP_REG_2), -(BPF_PPC_TAILCALL)));
>>>> +        EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), 
>>>> MAX_TAIL_CALL_CNT));
>>>> +        PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
>>>> +        EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), 
>>>> bpf_to_ppc(TMP_REG_2),
>>>> +                            -(BPF_PPC_TAILCALL)));
>>>> +        EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, - 
>>>> (BPF_PPC_TAILCALL)));
>>>>       }
>>>>         if (bpf_has_stack_frame(ctx)) {
>>>> @@ -352,19 +369,38 @@ static int bpf_jit_emit_tail_call(u32 *image, 
>>>> struct codegen_context *ctx, u32 o
>>>>       EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));
>>>>       PPC_BCC_SHORT(COND_GE, out);
>>>>   +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, 
>>>> bpf_jit_stack_tailcallinfo_offset(ctx)));
>>>> +    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
>>>> +    PPC_COND_BRANCH(COND_LE, CTX_NIA(ctx) + 8);
>>>> +
>>>> +    /* dereference TMP_REG_1 */
>>>> +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 0));
>>>> +
>>>>       /*
>>>> -     * if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
>>>> +     * if (tail_call_info == MAX_TAIL_CALL_CNT)
>>>>        *   goto out;
>>>>        */
>>>> -    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, 
>>>> bpf_jit_stack_tailcallcnt(ctx)));
>>>>       EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
>>>> -    PPC_BCC_SHORT(COND_GE, out);
>>>> +    PPC_COND_BRANCH(COND_EQ, out);
>>>>         /*
>>>> -     * tail_call_cnt++;
>>>> +     * tail_call_info++; <- Actual value of tcc here
>>>>        */
>>>>       EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), 
>>>> bpf_to_ppc(TMP_REG_1), 1));
>>>> -    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, 
>>>> bpf_jit_stack_tailcallcnt(ctx)));
>>>> +
>>>> +    /*
>>>> +     * Before writing updated tail_call_info, distinguish if 
>>>> current frame
>>>> +     * is storing a reference to tail_call_info or actual tcc value in
>>>> +     * tail_call_info.
>>>> +     */
>>>> +    EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_2), _R1, 
>>>> bpf_jit_stack_tailcallinfo_offset(ctx)));
>>>> +    EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_2), MAX_TAIL_CALL_CNT));
>>>> +    PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
>>>> +
>>>> +    /* First get address of tail_call_info */
>>>> +    EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_2), _R1, 
>>>> bpf_jit_stack_tailcallinfo_offset(ctx)));
>>>> +    /* Writeback updated value to tail_call_info */
>>>> +    EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_2), 
>>>> 0));
>>>>         /* prog = array->ptrs[index]; */
>>>>       EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));
>> -Abhishek
> 

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

[Hari Bathini]

On 16/01/26 1:19 pm, Christophe Leroy (CS GROUP) wrote:
> 
> 
> Le 16/01/2026 à 05:50, Hari Bathini a écrit :
> 
> Not received this mail that Hari is reponding to.

That is weird.

>>
>>
>> On 14/01/26 6:33 pm, adubey wrote:
>>> On 2026-01-14 17:57, Christophe Leroy (CS GROUP) wrote:
>>>> Le 14/01/2026 à 12:44, adubey@linux.ibm.com a écrit :
>>>>> From: Abhishek Dubey <adubey@linux.ibm.com>
>>>>>
>>>>> Enabling tailcalls with subprog combinations by referencing
>>>>> method. The actual tailcall count is always maintained in the
>>>>> tail_call_info variable present in the frame of main function
>>>>> (also called entry function). The tail_call_info variables in
>>>>> the frames of all other subprog contains reference to the
>>>>> tail_call_info present in frame of main function.
>>>>>
>>>>> Dynamic resolution interprets the tail_call_info either as
>>>>> value or reference depending on the context of active frame
>>>>> while tailcall is invoked.
>>>>>
>>>>> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
>>>>> ---
>>>>>   arch/powerpc/net/bpf_jit.h        | 12 +++++-
>>>>>   arch/powerpc/net/bpf_jit_comp.c   | 10 ++++-
>>>>>   arch/powerpc/net/bpf_jit_comp64.c | 68 ++++++++++++++++++++++ 
>>>>> +--------
>>>>>   3 files changed, 70 insertions(+), 20 deletions(-)
>>>>>
>>>>> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
>>>>> index 45d419c0ee73..5d735bc5e6bd 100644
>>>>> --- a/arch/powerpc/net/bpf_jit.h
>>>>> +++ b/arch/powerpc/net/bpf_jit.h
>>>>> @@ -51,6 +51,12 @@
>>>>>           EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
>>>>> (offset & 0xfffc));                    \
>>>>>       } while (0)
>>>>>   +/* Same as PPC_BCC_SHORT, except valid dest is known prior to 
>>>>> call. */
>>>>> +#define PPC_COND_BRANCH(cond, dest)         \
>>>>> +    do {                                      \
>>>>> +        long offset = (long)(dest) - CTX_NIA(ctx);              \
>>>>> +        EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
>>>>> (offset & 0xfffc));    \
>>>>> +    } while (0)
>>>>
>>>> I don't like the idea of duplicating PPC_BCC_SHORT() to just kick the
>>>> verification out. Now we will have two macros doing the exact same
>>>> thing with one handling failure case and one ignoring failure case.
>>>> There is a big risk that one day or another someone will use the wrong
>>>> macro.
>>>>
>>>> Could you change bpf_jit_build_prologue() to return an int add use
>>>> PPC_BCC_SHORT() instead of that new PPC_COND_BRANCH() ?
>>> I implemented exactly same change in bpf_jit_build_prologue(). But, 
>>> during internal review, @HariBathini suggested
>>> to have separate macro with a caution note.
>>>
>>> @Hari please suggest here!
>>
>> Not just about the change of return type but the check seems like an
>> overkill for cases where the offset is known and within branch range.
>> How about using BUILD_BUG_ON() to avoid unecessary checks and
>> return type change for places where the branch offset is known
>> and is a constant?
> 
> When offset is a constant known at build time, checks are eliminated by 
> gcc at build, see exemple below from disasembly of bpf_jit_comp32.o, 
> there are no checks.
> 
>                                  PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 
> 4);
>      36d8:       3c 80 41 81     lis     r4,16769
>                                  EMIT(PPC_RAW_CMPLW(src_reg, _R0));
>      36dc:       81 3f 00 04     lwz     r9,4(r31)
>                                  PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 
> 4);
>      36e0:       60 84 00 10     ori     r4,r4,16
>                                  EMIT(PPC_RAW_CMPLW(src_reg, _R0));
>      36e4:       39 29 00 01     addi    r9,r9,1
>                                  PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 
> 4);
>      36e8:       55 23 10 3a     slwi    r3,r9,2
>                                  EMIT(PPC_RAW_CMPLW(src_reg, _R0));
>      36ec:       91 3f 00 04     stw     r9,4(r31)
>                                  PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 
> 4);
>      36f0:       7c 97 19 2e     stwx    r4,r23,r3
>                                  EMIT(PPC_RAW_LI(dst_reg, 0));
>      36f4:       55 49 a9 94     rlwinm  r9,r10,21,6,10
>                                  PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 
> 4);
>      36f8:       80 9f 00 04     lwz     r4,4(r31)
>                                  EMIT(PPC_RAW_LI(dst_reg, 0));
>      36fc:       65 29 38 00     oris    r9,r9,14336
>                                  PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 
> 4);
>      3700:       38 84 00 01     addi    r4,r4,1
>                                  EMIT(PPC_RAW_LI(dst_reg, 0));
>      3704:       54 83 10 3a     slwi    r3,r4,2
>                                  PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 
> 4);
>      3708:       90 9f 00 04     stw     r4,4(r31)
>                                  EMIT(PPC_RAW_LI(dst_reg, 0));
>      370c:       7d 37 19 2e     stwx    r9,r23,r3
> 

Interesting. I do see is_offset_in_cond_branch_range() in action with
constant offsets too, on ppc64 compile at least. fwiw, I had this
optimized version in mind for constant offset:

   #define PPC_BCC_CONST_SHORT(cond, offset) 
                     \
         do { 
                     \
                 BUILD_BUG_ON(offset < -0x8000 || offset > 0x7fff || 
(offset & 0x3));        \
                 EMIT(PPC_INST_BRANCH_COND | (((cond) & 0x3ff) << 16) | 
(offset & 0xfffc));  \
         } while (0)

With that, something like:

     PPC_BCC_SHORT(COND_NE, (ctx->idx + 3) * 4);

becomes

     PPC_BCC_CONST_SHORT(COND_NE, 12);

- Hari

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

[Hari Bathini]

On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> In the conventional stack frame, the position of tail_call_cnt
> is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the
> offset of tail_call_cnt in the trampoline frame is after the
> stack alignment padding. BPF JIT logic could become complex
> when dealing with frame-sensitive offset calculation of
> tail_call_cnt. Having the same offset in both frames is the
> desired objective.
> 
> The trampoline frame does not have a BPF_PPC_STACK_SAVE area.
> Introducing it leads to under-utilization of extra memory meant
> only for the offset alignment of tail_call_cnt.
> Another challenge is the variable alignment padding sitting at
> the bottom of the trampoline frame, which requires additional
> handling to compute tail_call_cnt offset.
> 
> This patch addresses the above issues by moving tail_call_cnt
> to the bottom of the stack frame at offset 0 for both types
> of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE
> in trampoline frame, and a common offset computation for
> tail_call_cnt serves both frames.
> 
> The changes in this patch are required by the third patch in the
> series, where the 'reference to tail_call_info' of the main frame
> is copied into the trampoline frame from the previous frame.
> 
> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
> ---
>   arch/powerpc/net/bpf_jit.h        |  4 ++++
>   arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
>   2 files changed, 24 insertions(+), 11 deletions(-)
> 
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 8334cd667bba..45d419c0ee73 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -72,6 +72,10 @@
>   	} } while (0)
>   
>   #ifdef CONFIG_PPC64
> +
> +/* for tailcall counter */
> +#define BPF_PPC_TAILCALL        8
> +
>   /* If dummy pass (!image), account for maximum possible instructions */
>   #define PPC_LI64(d, i)		do {					      \
>   	if (!image)							      \
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 1fe37128c876..39061cd742c1 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -20,13 +20,15 @@
>   #include "bpf_jit.h"
>   
>   /*
> - * Stack layout:
> + * Stack layout 1:
> + * Layout when setting up our own stack frame.
> + * Note: r1 at bottom, component offsets positive wrt r1.
>    * Ensure the top half (upto local_tmp_var) stays consistent
>    * with our redzone usage.
>    *
>    *		[	prev sp		] <-------------
> - *		[   nv gpr save area	] 6*8		|
>    *		[    tail_call_cnt	] 8		|
> + *		[   nv gpr save area	] 6*8		|
>    *		[    local_tmp_var	] 24		|
>    * fp (r31) -->	[   ebpf stack space	] upto 512	|
>    *		[     frame header	] 32/112	|
> @@ -36,10 +38,12 @@
>   /* for gpr non volatile registers BPG_REG_6 to 10 */
>   #define BPF_PPC_STACK_SAVE	(6*8)
>   /* for bpf JIT code internal usage */
> -#define BPF_PPC_STACK_LOCALS	32
> +#define BPF_PPC_STACK_LOCALS	24
>   /* stack frame excluding BPF stack, ensure this is quadword aligned */
>   #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
> -				 BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
> +				 BPF_PPC_STACK_LOCALS + \
> +				 BPF_PPC_STACK_SAVE   + \
> +				 BPF_PPC_TAILCALL)
>   
>   /* BPF register usage */
>   #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
> @@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
>   }
>   

>   /*
> + * Stack layout 2:
>    * When not setting up our own stackframe, the redzone (288 bytes) usage is:
> + * Note: r1 from prev frame. Component offset negative wrt r1.
>    *
>    *		[	prev sp		] <-------------
>    *		[	  ...       	] 		|
>    * sp (r1) --->	[    stack pointer	] --------------
> - *		[   nv gpr save area	] 6*8
>    *		[    tail_call_cnt	] 8
> + *		[   nv gpr save area	] 6*8
>    *		[    local_tmp_var	] 24
>    *		[   unused red zone	] 224
>    */

Calling it stack layout 1 & 2 is inappropriate. The stack layout
is essentially the same. It just goes to show things with reference
to r1 when stack is setup explicitly vs when redzone is being used...

- Hari

Re: [PATCH v2 2/6] powerpc64/bpf: Support tailcalls with subprogs

[Hari Bathini]

On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
> From: Abhishek Dubey<adubey@linux.ibm.com>
> 
> Enabling tailcalls with subprog combinations by referencing
> method. The actual tailcall count is always maintained in the
> tail_call_info variable present in the frame of main function
> (also called entry function). The tail_call_info variables in
> the frames of all other subprog contains reference to the
> tail_call_info present in frame of main function.

I would rephrase:

"The tail_call_info variables in the frames of all other subprog
contains reference to the tail_call_info present in frame of
main function."

to:

"The tail_call_info field in the stack frame of subprogs contains
reference to the tail_call_info field in the stack frame of main
BPF program."

- Hari

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

[Hari Bathini]

On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> The trampoline mechanism sets up its own stack frame and
> an additional dummy frame. We need to have additional JIT
> instructions handling tailcall dereferencing in the
> trampoline's context.
> 
> We don't add the two stack frames pointed above, rather
> add space for tail_call_info at bottom in trampoline frame
> for ppc64. This makes the trampoline's frame consistent with
> layout of all other frames wrt tail_call_info offset.
> 
> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
> ---
>   arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++-----------
>   1 file changed, 56 insertions(+), 27 deletions(-)
> 
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index 069a8822c30d..e3088cf089d1 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
>   	return 0;
>   }
>   
> -static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct codegen_context *ctx,
> -					       int func_frame_offset, int r4_off)
> -{
> -	if (IS_ENABLED(CONFIG_PPC64)) {
> -		/* See bpf_jit_stack_tailcallinfo_offset() */
> -		int tailcallcnt_offset = 7 * 8;
> -
> -		EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
> -		EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
> -	} else {
> -		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
> -		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
> -	}
> -}
> +/*
> + * Refer the label 'Generated stack layout' in this file for actual stack
> + * layout during trampoline invocation.
> + *
> + * Refer __arch_prepare_bpf_trampoline() for stack component details.
> + *
> + * The tailcall count/reference is present in caller's stack frame. Its required
> + * to copy the content of tail_call_info before calling the actual function
> + * to which the trampoline is attached.
> + *
> + */
>   
> -static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct codegen_context *ctx,
> -						 int func_frame_offset, int r4_off)
> +static void bpf_trampoline_setup_tail_call_info(u32 *image, struct codegen_context *ctx,
> +					       int func_frame_offset,
> +					       int bpf_dummy_frame_size, int r4_off)
>   {
>   	if (IS_ENABLED(CONFIG_PPC64)) {
>   		/* See bpf_jit_stack_tailcallinfo_offset() */

> -		int tailcallcnt_offset = 7 * 8;
> +		int tailcallinfo_offset = BPF_PPC_TAILCALL;

This offset update should have been part of patch#1

> +		/*
> +		 * func_frame_offset =                                   ...(1)
> +		 *     bpf_dummy_frame_size + trampoline_frame_size
> +		 */
> +		EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
> +		EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
> +
> +		/*
> +		 * Setting the tail_call_info in trampoline's frame
> +		 * depending on if previous frame had value or reference.
> +		 */
> +		EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
> +		PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
> +		EMIT(PPC_RAW_ADDI(_R3, _R4, bpf_jit_stack_tailcallinfo_offset(ctx)));
> +		/*
> +		 * From ...(1) above:
> +		 * trampoline_frame_bottom =                            ...(2)
> +		 *     func_frame_offset - bpf_dummy_frame_size
> +		 *
> +		 * Using ...(2) derived above:
> +		 *  trampoline_tail_call_info_offset =                  ...(3)
> +		 *      trampoline_frame_bottom - tailcallinfo_offset
> +		 *
> +		 * From ...(3):
> +		 * Use trampoline_tail_call_info_offset to write reference of main's
> +		 * tail_call_info in trampoline frame.
> +		 */
> +		EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - bpf_dummy_frame_size)
> +					- tailcallinfo_offset));
>   
> -		EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
> -		EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - tailcallcnt_offset));
>   	} else {
>   		/* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
> -		EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
> +		EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
>   	}
>   }
>   
> @@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
>   	 * LR save area                 [ r0 save (64-bit)  ]   | header
>   	 *                              [ r0 save (32-bit)  ]   |
>   	 * dummy frame for unwind       [ back chain 1      ] --
> +	 *                              [ tail_call_info    ] non optional - 64-bit powerpc
>   	 *                              [ padding           ] align stack frame
>   	 *       r4_off                 [ r4 (tailcallcnt)  ] optional - 32-bit powerpc
>   	 *       alt_lr_off             [ real lr (ool stub)] optional - actual lr

> @@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
>   		}
>   	}
>   
> -	/* Padding to align stack frame, if any */
> -	bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
> +	if (!(bpf_frame_size % (2 * SZL))) {
> +		/* Stack is 16-byte aligned */
> +		/* Room for padding followed by 64-bit tail_call_info */
> +		bpf_frame_size += SZL + BPF_PPC_TAILCALL;
> +	} else {
> +		/* Room for 64-bit tail_call_info */
> +		bpf_frame_size += BPF_PPC_TAILCALL;
> +	}
>   
>   	/* Dummy frame size for proper unwind - includes 64-bytes red zone for 64-bit powerpc */
>   	bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64;

This change assumes the size is at least 8-byte aligned which is
true today but better skip that assumption by not touching the
padding part. The above hunk could simply be:

diff --git a/arch/powerpc/net/bpf_jit_comp.c 
b/arch/powerpc/net/bpf_jit_comp.c
index 5e976730b2f5..266cc6f17dcc 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -795,6 +795,10 @@ static int __arch_prepare_bpf_trampoline(struct 
bpf_tramp_image *im, void *rw_im
                 }
         }

+       /* Save tailcall count pointer at the same offset on the stack 
where subprogs expect it */
+       if ((flags & BPF_TRAMP_F_CALL_ORIG) && (flags & 
BPF_TRAMP_F_TAIL_CALL_CTX))
+               bpf_frame_size += SZL;
+
         /* Padding to align stack frame, if any */
         bpf_frame_size = round_up(bpf_frame_size, SZL * 2);


Patch#2 is not complete without this change. Please fold this patch
into patch#2 itself.

- Hari

Re: [PATCH v2 3/6] powerpc64/bpf: Tailcall handling with trampolines

[Hari Bathini]

On 17/01/26 4:09 pm, Hari Bathini wrote:
> 
> 
> On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
>> From: Abhishek Dubey <adubey@linux.ibm.com>
>>
>> The trampoline mechanism sets up its own stack frame and
>> an additional dummy frame. We need to have additional JIT
>> instructions handling tailcall dereferencing in the
>> trampoline's context.
>>
>> We don't add the two stack frames pointed above, rather
>> add space for tail_call_info at bottom in trampoline frame
>> for ppc64. This makes the trampoline's frame consistent with
>> layout of all other frames wrt tail_call_info offset.
>>
>> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
>> ---
>>   arch/powerpc/net/bpf_jit_comp.c | 83 ++++++++++++++++++++++-----------
>>   1 file changed, 56 insertions(+), 27 deletions(-)
>>
>> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/ 
>> bpf_jit_comp.c
>> index 069a8822c30d..e3088cf089d1 100644
>> --- a/arch/powerpc/net/bpf_jit_comp.c
>> +++ b/arch/powerpc/net/bpf_jit_comp.c
>> @@ -606,33 +606,58 @@ static int invoke_bpf_mod_ret(u32 *image, u32 
>> *ro_image, struct codegen_context
>>       return 0;
>>   }
>> -static void bpf_trampoline_setup_tail_call_cnt(u32 *image, struct 
>> codegen_context *ctx,
>> -                           int func_frame_offset, int r4_off)
>> -{
>> -    if (IS_ENABLED(CONFIG_PPC64)) {
>> -        /* See bpf_jit_stack_tailcallinfo_offset() */
>> -        int tailcallcnt_offset = 7 * 8;
>> -
>> -        EMIT(PPC_RAW_LL(_R3, _R1, func_frame_offset - 
>> tailcallcnt_offset));
>> -        EMIT(PPC_RAW_STL(_R3, _R1, -tailcallcnt_offset));
>> -    } else {
>> -        /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
>> -        EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
>> -    }
>> -}
>> +/*
>> + * Refer the label 'Generated stack layout' in this file for actual 
>> stack
>> + * layout during trampoline invocation.
>> + *
>> + * Refer __arch_prepare_bpf_trampoline() for stack component details.
>> + *
>> + * The tailcall count/reference is present in caller's stack frame. 
>> Its required
>> + * to copy the content of tail_call_info before calling the actual 
>> function
>> + * to which the trampoline is attached.
>> + *
>> + */
>> -static void bpf_trampoline_restore_tail_call_cnt(u32 *image, struct 
>> codegen_context *ctx,
>> -                         int func_frame_offset, int r4_off)
>> +static void bpf_trampoline_setup_tail_call_info(u32 *image, struct 
>> codegen_context *ctx,
>> +                           int func_frame_offset,
>> +                           int bpf_dummy_frame_size, int r4_off)
>>   {
>>       if (IS_ENABLED(CONFIG_PPC64)) {
>>           /* See bpf_jit_stack_tailcallinfo_offset() */
> 
>> -        int tailcallcnt_offset = 7 * 8;
>> +        int tailcallinfo_offset = BPF_PPC_TAILCALL;
> 
> This offset update should have been part of patch#1
> 
>> +        /*
>> +         * func_frame_offset =                                   ...(1)
>> +         *     bpf_dummy_frame_size + trampoline_frame_size
>> +         */
>> +        EMIT(PPC_RAW_LD(_R4, _R1, func_frame_offset));
>> +        EMIT(PPC_RAW_LD(_R3, _R4, -tailcallinfo_offset));
>> +
>> +        /*
>> +         * Setting the tail_call_info in trampoline's frame
>> +         * depending on if previous frame had value or reference.
>> +         */
>> +        EMIT(PPC_RAW_CMPLWI(_R3, MAX_TAIL_CALL_CNT));
>> +        PPC_COND_BRANCH(COND_GT, CTX_NIA(ctx) + 8);
>> +        EMIT(PPC_RAW_ADDI(_R3, _R4, 
>> bpf_jit_stack_tailcallinfo_offset(ctx)));
>> +        /*
>> +         * From ...(1) above:
>> +         * trampoline_frame_bottom =                            ...(2)
>> +         *     func_frame_offset - bpf_dummy_frame_size
>> +         *
>> +         * Using ...(2) derived above:
>> +         *  trampoline_tail_call_info_offset =                  ...(3)
>> +         *      trampoline_frame_bottom - tailcallinfo_offset
>> +         *
>> +         * From ...(3):
>> +         * Use trampoline_tail_call_info_offset to write reference of 
>> main's
>> +         * tail_call_info in trampoline frame.
>> +         */
>> +        EMIT(PPC_RAW_STL(_R3, _R1, (func_frame_offset - 
>> bpf_dummy_frame_size)
>> +                    - tailcallinfo_offset));
>> -        EMIT(PPC_RAW_LL(_R3, _R1, -tailcallcnt_offset));
>> -        EMIT(PPC_RAW_STL(_R3, _R1, func_frame_offset - 
>> tailcallcnt_offset));
>>       } else {
>>           /* See bpf_jit_stack_offsetof() and BPF_PPC_TC */
>> -        EMIT(PPC_RAW_STL(_R4, _R1, r4_off));
>> +        EMIT(PPC_RAW_LL(_R4, _R1, r4_off));
>>       }
>>   }
>> @@ -720,6 +745,7 @@ static int __arch_prepare_bpf_trampoline(struct 
>> bpf_tramp_image *im, void *rw_im
>>        * LR save area                 [ r0 save (64-bit)  ]   | header
>>        *                              [ r0 save (32-bit)  ]   |
>>        * dummy frame for unwind       [ back chain 1      ] --

>> +     *                              [ tail_call_info    ] non 
>> optional - 64-bit powerpc

Also, why non-optional? This can be optional on BPF_TRAMP_F_CALL_ORIG
and BPF_TRAMP_F_TAIL_CALL_CTX flags?

>>        *                              [ padding           ] align 
>> stack frame
>>        *       r4_off                 [ r4 (tailcallcnt)  ] optional - 
>> 32-bit powerpc
>>        *       alt_lr_off             [ real lr (ool stub)] optional - 
>> actual lr
> 
>> @@ -801,8 +827,14 @@ static int __arch_prepare_bpf_trampoline(struct 
>> bpf_tramp_image *im, void *rw_im
>>           }
>>       }
>> -    /* Padding to align stack frame, if any */
>> -    bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
>> +    if (!(bpf_frame_size % (2 * SZL))) {
>> +        /* Stack is 16-byte aligned */
>> +        /* Room for padding followed by 64-bit tail_call_info */
>> +        bpf_frame_size += SZL + BPF_PPC_TAILCALL;
>> +    } else {
>> +        /* Room for 64-bit tail_call_info */
>> +        bpf_frame_size += BPF_PPC_TAILCALL;
>> +    }
>>       /* Dummy frame size for proper unwind - includes 64-bytes red 
>> zone for 64-bit powerpc */
>>       bpf_dummy_frame_size = STACK_FRAME_MIN_SIZE + 64;
> 
> This change assumes the size is at least 8-byte aligned which is
> true today but better skip that assumption by not touching the
> padding part. The above hunk could simply be:
> 
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/ 
> bpf_jit_comp.c
> index 5e976730b2f5..266cc6f17dcc 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -795,6 +795,10 @@ static int __arch_prepare_bpf_trampoline(struct 
> bpf_tramp_image *im, void *rw_im
>                  }
>          }
> 
> +       /* Save tailcall count pointer at the same offset on the stack 
> where subprogs expect it */
> +       if ((flags & BPF_TRAMP_F_CALL_ORIG) && (flags & 
> BPF_TRAMP_F_TAIL_CALL_CTX))
> +               bpf_frame_size += SZL;
> +
>          /* Padding to align stack frame, if any */
>          bpf_frame_size = round_up(bpf_frame_size, SZL * 2);
> 
> 
> Patch#2 is not complete without this change. Please fold this patch
> into patch#2 itself.
> 

- Hari

Re: [PATCH v2 6/6] powerpc64/bpf: Additional NVR handling for bpf_throw

[Hari Bathini]

On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
> From: Abhishek Dubey <adubey@linux.ibm.com>
> 
> The bpf_throw() function never returns, if it has
> clobbered any callee-saved register, those will
> remain clobbered. The prologue must take care of
> saving all callee-saved registers in the frame of
> exception boundary program. Later these additional
> non volatile registers R14-R25 along with other
> NVRs are restored back in the epilogue of exception
> callback.
> 
> To achieve above objective the frame size is
> determined dynamically to accommodate additional
> non volatile registers in exception boundary's frame.
> For non-exception boundary program, the frame size
> remains optimal. The additional instructions to
> save & restore r14-r25 registers are emitted only during
> exception boundary and exception callback respectively.
> 
> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
> ---
>   arch/powerpc/net/bpf_jit_comp64.c | 70 +++++++++++++++++++++++++++----
>   1 file changed, 63 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index a6083dd9786c..941e0818c9ec 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -32,21 +32,37 @@
>    *
>    *		[	prev sp		] <-------------
>    *		[    tail_call_info	] 8		|
> - *		[   nv gpr save area	] 6*8		|
> + *		[   nv gpr save area	] 6*8 + (12*8)	|
>    *		[    local_tmp_var	] 24		|
>    * fp (r31) -->	[   ebpf stack space	] upto 512	|
>    *		[     frame header	] 32/112	|
>    * sp (r1) --->	[    stack pointer	] --------------
> + *
> + * Additional (12*8) in 'nv gpr save area' only in case of
> + * exception boundary.
>    */
>   
>   /* for bpf JIT code internal usage */
>   #define BPF_PPC_STACK_LOCALS	24
> +/*
> + * for additional non volatile registers(r14-r25) to be saved
> + * at exception boundary
> + */
> +#define BPF_PPC_EXC_STACK_SAVE (12*8)
> +
>   /* stack frame excluding BPF stack, ensure this is quadword aligned */
>   #define BPF_PPC_STACKFRAME	(STACK_FRAME_MIN_SIZE + \
>   				 BPF_PPC_STACK_LOCALS + \
>   				 BPF_PPC_STACK_SAVE   + \
>   				 BPF_PPC_TAILCALL)
>   
> +/*
> + * same as BPF_PPC_STACKFRAME with save area for additional
> + * non volatile registers saved at exception boundary.
> + * This is quad-word aligned.
> + */
> +#define BPF_PPC_EXC_STACKFRAME (BPF_PPC_STACKFRAME + BPF_PPC_EXC_STACK_SAVE)
> +
>   /* BPF register usage */
>   #define TMP_REG_1	(MAX_BPF_JIT_REG + 0)
>   #define TMP_REG_2	(MAX_BPF_JIT_REG + 1)
> @@ -103,9 +119,12 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
>    *		[	  ...       	] 		|
>    * sp (r1) --->	[    stack pointer	] --------------
>    *		[    tail_call_info	] 8
> - *		[   nv gpr save area	] 6*8
> + *		[   nv gpr save area	] 6*8 + (12*8)
>    *		[    local_tmp_var	] 24
>    *		[   unused red zone	] 224
> + *
> + * Additional (12*8) in 'nv gpr save area' only in case of
> + * exception boundary.
>    */
>   static int bpf_jit_stack_local(struct codegen_context *ctx)
>   {
> @@ -114,7 +133,11 @@ static int bpf_jit_stack_local(struct codegen_context *ctx)
>   		return STACK_FRAME_MIN_SIZE + ctx->stack_size;
>   	} else {
>   		/* Stack layout 2 */
> -		return -(BPF_PPC_TAILCALL + BPF_PPC_STACK_SAVE + BPF_PPC_STACK_LOCALS);
> +		return -(BPF_PPC_TAILCALL
> +			+ BPF_PPC_STACK_SAVE
> +			+ (ctx->exception_boundary || ctx->exception_cb ?
> +							BPF_PPC_EXC_STACK_SAVE:0)
> +			+ BPF_PPC_STACK_LOCALS);
>   	}
>   }
>   
> @@ -125,9 +148,19 @@ int bpf_jit_stack_tailcallinfo_offset(struct codegen_context *ctx)
>   
>   static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
>   {
> -	if (reg >= BPF_PPC_NVR_MIN && reg < 32)
> +	int min_valid_nvreg = BPF_PPC_NVR_MIN;
> +	/* Default frame size for all cases except exception boundary */
> +	int frame_nvr_size = BPF_PPC_STACKFRAME;
> +
> +	/* Consider all nv regs for handling exceptions */
> +	if (ctx->exception_boundary || ctx->exception_cb) {
> +		min_valid_nvreg = _R14;
> +		frame_nvr_size = BPF_PPC_EXC_STACKFRAME;
> +	}
> +
> +	if (reg >= min_valid_nvreg && reg < 32)
>   		return (bpf_has_stack_frame(ctx) ?
> -			(BPF_PPC_STACKFRAME + ctx->stack_size) : 0)
> +			(frame_nvr_size + ctx->stack_size) : 0)
>   				- (8 * (32 - reg)) - BPF_PPC_TAILCALL;
>   
>   	pr_err("BPF JIT is asking about unknown registers");
> @@ -189,7 +222,20 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
>   			EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
>   		}
>   

> -		EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
> +		int stack_expand = ctx->exception_boundary || ctx->exception_cb ?
> +					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
> +		EMIT(PPC_RAW_STDU(_R1, _R1, -(stack_expand + ctx->stack_size)));

[...]

> -		EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
> +		int stack_shrink = ctx->exception_cb || ctx->exception_boundary ?
> +					BPF_PPC_EXC_STACKFRAME : BPF_PPC_STACKFRAME;
> +		EMIT(PPC_RAW_ADDI(_R1, _R1, stack_shrink + ctx->stack_size));
> +

An inline helper bpf_jit_stack_size() defined to return the stack
size in both prologue and epilogue while setting up and tearing
down the stack should be more elegant.


Also, IIUC, the JIT code to handle tailcall info is irrelevant for
all subprogs of a BPF program with seen_exception. JIT code in the
prologue for tailcall count handling can be skipped for exception_cb
at least?

- Hari

Re: [PATCH v2 1/6] powerpc64/bpf: Move tail_call_cnt to bottom of stack frame

[Hari Bathini]

On 17/01/26 4:11 pm, adubey wrote:
> On 2026-01-17 15:41, Hari Bathini wrote:
>> On 14/01/26 5:14 pm, adubey@linux.ibm.com wrote:
>>> From: Abhishek Dubey <adubey@linux.ibm.com>
>>>
>>> In the conventional stack frame, the position of tail_call_cnt
>>> is after the NVR save area (BPF_PPC_STACK_SAVE). Whereas, the
>>> offset of tail_call_cnt in the trampoline frame is after the
>>> stack alignment padding. BPF JIT logic could become complex
>>> when dealing with frame-sensitive offset calculation of
>>> tail_call_cnt. Having the same offset in both frames is the
>>> desired objective.
>>>
>>> The trampoline frame does not have a BPF_PPC_STACK_SAVE area.
>>> Introducing it leads to under-utilization of extra memory meant
>>> only for the offset alignment of tail_call_cnt.
>>> Another challenge is the variable alignment padding sitting at
>>> the bottom of the trampoline frame, which requires additional
>>> handling to compute tail_call_cnt offset.
>>>
>>> This patch addresses the above issues by moving tail_call_cnt
>>> to the bottom of the stack frame at offset 0 for both types
>>> of frames. This saves additional bytes required by BPF_PPC_STACK_SAVE
>>> in trampoline frame, and a common offset computation for
>>> tail_call_cnt serves both frames.
>>>
>>> The changes in this patch are required by the third patch in the
>>> series, where the 'reference to tail_call_info' of the main frame
>>> is copied into the trampoline frame from the previous frame.
>>>
>>> Signed-off-by: Abhishek Dubey <adubey@linux.ibm.com>
>>> ---
>>>   arch/powerpc/net/bpf_jit.h        |  4 ++++
>>>   arch/powerpc/net/bpf_jit_comp64.c | 31 ++++++++++++++++++++-----------
>>>   2 files changed, 24 insertions(+), 11 deletions(-)
>>>
>>> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
>>> index 8334cd667bba..45d419c0ee73 100644
>>> --- a/arch/powerpc/net/bpf_jit.h
>>> +++ b/arch/powerpc/net/bpf_jit.h
>>> @@ -72,6 +72,10 @@
>>>       } } while (0)
>>>     #ifdef CONFIG_PPC64
>>> +
>>> +/* for tailcall counter */
>>> +#define BPF_PPC_TAILCALL        8
>>> +
>>>   /* If dummy pass (!image), account for maximum possible 
>>> instructions */
>>>   #define PPC_LI64(d, i)        do {                          \
>>>       if (!image)                                  \
>>> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/ 
>>> bpf_jit_comp64.c
>>> index 1fe37128c876..39061cd742c1 100644
>>> --- a/arch/powerpc/net/bpf_jit_comp64.c
>>> +++ b/arch/powerpc/net/bpf_jit_comp64.c
>>> @@ -20,13 +20,15 @@
>>>   #include "bpf_jit.h"
>>>     /*
>>> - * Stack layout:
>>> + * Stack layout 1:
>>> + * Layout when setting up our own stack frame.
>>> + * Note: r1 at bottom, component offsets positive wrt r1.
>>>    * Ensure the top half (upto local_tmp_var) stays consistent
>>>    * with our redzone usage.
>>>    *
>>>    *        [    prev sp        ] <-------------
>>> - *        [   nv gpr save area    ] 6*8        |
>>>    *        [    tail_call_cnt    ] 8        |
>>> + *        [   nv gpr save area    ] 6*8        |
>>>    *        [    local_tmp_var    ] 24        |
>>>    * fp (r31) -->    [   ebpf stack space    ] upto 512    |
>>>    *        [     frame header    ] 32/112    |
>>> @@ -36,10 +38,12 @@
>>>   /* for gpr non volatile registers BPG_REG_6 to 10 */
>>>   #define BPF_PPC_STACK_SAVE    (6*8)
>>>   /* for bpf JIT code internal usage */
>>> -#define BPF_PPC_STACK_LOCALS    32
>>> +#define BPF_PPC_STACK_LOCALS    24
>>>   /* stack frame excluding BPF stack, ensure this is quadword aligned */
>>>   #define BPF_PPC_STACKFRAME    (STACK_FRAME_MIN_SIZE + \
>>> -                 BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
>>> +                 BPF_PPC_STACK_LOCALS + \
>>> +                 BPF_PPC_STACK_SAVE   + \
>>> +                 BPF_PPC_TAILCALL)
>>>     /* BPF register usage */
>>>   #define TMP_REG_1    (MAX_BPF_JIT_REG + 0)
>>> @@ -87,27 +91,32 @@ static inline bool bpf_has_stack_frame(struct 
>>> codegen_context *ctx)
>>>   }
>>>
>>
>>>   /*
>>> + * Stack layout 2:
>>>    * When not setting up our own stackframe, the redzone (288 bytes) 
>>> usage is:
>>> + * Note: r1 from prev frame. Component offset negative wrt r1.
>>>    *
>>>    *        [    prev sp        ] <-------------
>>>    *        [      ...           ]         |
>>>    * sp (r1) --->    [    stack pointer    ] --------------
>>> - *        [   nv gpr save area    ] 6*8
>>>    *        [    tail_call_cnt    ] 8
>>> + *        [   nv gpr save area    ] 6*8
>>>    *        [    local_tmp_var    ] 24
>>>    *        [   unused red zone    ] 224
>>>    */
>>
>> Calling it stack layout 1 & 2 is inappropriate. The stack layout
>> is essentially the same. It just goes to show things with reference
>> to r1 when stack is setup explicitly vs when redzone is being used...
> Agree. I am using it as labels to refer in comment. Any better suggestions?
I think the comments could refer to has stack frame vs Redzone case..

- Hari